DINO is Also a Semantic Guider: Exploiting Class-aware Affinity for Weakly Supervised Semantic Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-23 09:51:26

Abstract: Weakly supervised semantic segmentation (WSSS) using image-level labels is a challenging task, with relying on Class Activation Map (CAM) to derive segmentation supervision. Although many efficient single-stage solutions have been proposed, their performance is hindered by the inherent ambiguity of CAM. This paper introduces a new approach, dubbed ECA, to Exploit the self-supervised Vision Transformer, DINO, inducing the Class-aware semantic Affinity to overcome this limitation. Specifically, we introduce a Semantic Affinity Exploitation module (SAE). It establishes the class-agnostic affinity graph through the self-attention of DINO. Using the highly activated patches on CAMs as “seeds”, we propagate them across the affinity graph and yield the Class-aware Affinity Region Map (CARM) as supplementary semantic guidance. Moreover, the selection of reliable “seeds” is crucial to the CARM generation. Inspired by the observed CAM inconsistency between the global and local views, we develop a CAM Correspondence Enhancement module (CCE) to encourage dense local-to-global CAM correspondences, advancing high-fidelity CAM for seed selection in SAE. Our experimental results demonstrate that ECA effectively improves the model's object pattern understanding. Remarkably, it outperforms state-of-the-art alternatives on the PASCAL VOC 2012 and MS COCO 2014 datasets, achieving 90.1% upper bound performance compared to its fully supervised counterpart.

Main paper: 
 introduction: 
 Semantic segmentation is a fundamental computer vision task that assigns a class for every pixel in an image. Despite the substantial progress in deep-learning-based segmentation models, their performance heavily relies on large-scale pixel-level annotations. Weakly supervised semantic segmentation (WSSS) aims to address this shortcoming by training segmentation models using weak and costeffective labels, such as sparse points [4] , bounding boxes [10, 21] , and image-level labels [24, 29, 30] . Among these WSSS tasks, the most rewarding yet challenging one is segmentation using only image-level labels because of its minimum information capacity. This paper also falls within the domain of training a segmentation model using only image-level labels.Prevalent WSSS with image-level labels commonly follow a multi-stage pipeline: 1) generating pixel pseudo-labels from Class The CAMs are from the prior art, AFA [24] . The proposed ECA's Class-aware Affinity Region Maps (CARM) from DINO can produce more accurate pseudo-labels.Activation Maps (CAM) [38] ; 2) training an affinity refinement network [1, 2] with dense Conditional Random Field (CRF) to refine the pseudo-labels; 3) training segmentation networks [6, 31] using the refined pseudo-labels. This pipeline necessitates training multiple models, thus complicating WSSS and diminishing the training efficiency. To address this issue, some recent works attempted to devise efficient single-stage solutions based on end-to-end Convolutional Neural Network (CNN) [3, 34] and Vision Transformer (ViT) architectures [24, 25] .However, compared to the pseudo-labels refined by training a dedicated refinement network in the multi-stage WSSS, those directly generated by the CAMs in single-stage methods exhibit inferior quality. This dramatically impairs segmentation supervision, resulting in subpar segmentation outcomes. One critical reason is the inevitable ambiguity of CAM that stems from the gap between the classification and localization tasks. As illustrated in Figure 1 , CAM tends to cover non-target regions (i.e., over-activations) or only the most discriminative regions (i.e., underactivations). Despite some post-processing modules integrated in recent single-stage works [24, 33] , which introduce local RGB and position information to alleviate this problem, they still struggle to produce high-quality pseudo-labels. We can clearly observe that the flaws existing in CAMs still remain in the produced final pseudolabels, indicating the inadequacy of leveraging RGB and position information to calibrate them.Recently, the self-supervised Vision Transformer (ViT) model [11] , DINO [5, 22] , has shown the capability of naturally modeling pairwise affinity between two image patches without using any annotations. This property allows for the grouping of patches with similar semantic properties. Although these grouped components are "class-agnostic" and independent of predefined categories (e.g., head and body rather than person and car), the coverage of each component is semantically precise and complete, and the patches within the same component have strong affinity. This merit can reveal the regions that CAM is difficult to cover accurately. Conversely, CAM cannot produce accurate object coverage, but it can provide class and localization information. Based on this insight, our objective is to leverage the CAM's class and localization priors to induce the task-required class-aware affinity from DINO as the semantic guidance to overcome the above issue.This paper proposes a new framework, ECA, to improve the performance of single-stage WSSS through the exploitation of DINO. Specifically, we propose a DINO-guided Semantic Affinity Exploitation module (SAE) to generate Class-aware Affinity Region Map (CARM), offering the complementary semantic guidance of CAM. For each image, SAE comprises two steps motivated by the vanilla "Seed and Expansion" pipeline. The first step, "selection of initial seeds", involves selecting top 𝑘% activations on the CAM to serve as the initial seeds. The second step, "generation of class affinity region map", entails constructing a class-agnostic affinity graph by measuring pairwise similarities among the patch keys derived from DINO [5] . The seeds are then propagated on the affinity graph, yielding the CARM of each labeled class for supervision. Empirical evidence shows that the generated CARM effectively mitigates the detrimental impact of CAM ambiguity on segmentation supervision, leading to notable performance improvement. Furthermore, to select better initial seeds, we introduce the CAM Correspondence Enhancement (CCE). Inspired by the inconsistency of CAM under different views [29] , CCE encourages local-to-global correspondences of CAM, providing high-fidelity CAM for the seed selection. Overall, our contributions are summarized as follows:• We propose a Semantic Affinity Exploitation module (SAE) to fully exploit the potential of semantic affinity existing in DINO. It produces powerful complementary semantic guidance, i.e., Class-aware Affinity Region Map (CARM), to boost the single-stage WSSS performance.• We propose a CAM Correspondence Enhancement module (CCE) to promote the dense local-to-global correspondences of CAM. It is proved that CCE promotes the object activation completeness and the robustness of CAM, which benefits the seed selection in SAE module.• Building upon SAE and CCE, the proposed ECA can significantly outperform single-stage WSSS competitors on the PASCAL VOC 2012 and MS COCO 2014 dataset, and achieve comparable performance with multi-stage methods.To the best of our knowledge, ECA is the first work that derives semantic affinity of DINO for WSSS. 
related work 2.1 single-stage wsss: 
 Compared with the multi-stage WSSS pipeline, single-stage solutions learn a segmentation decoder using the generated pseudolabels in an end-to-end manner. Due to the lack of satisfactory pseudo-labels, the performance of single-stage WSSS models lags behind multi-stage models. To address this limitation, many studies focused on optimizing CAM pseudo-labels [3, 24, 33] . For instance, RRM [35] employed Conditional Random Field (CRF) to produce the refined label as supervision for segmentation. AFA [24] introduced a Pixel-Adaptive Refinement (PAR) module to refine the CAM pseudo-labels effectively. Additionally, it explores the use of Transformer with an affinity learning loss to enforce the model to learn object-related attention, resulting in better pseudo-labels for segmentation. TSCD [33] introduced a CAM-based self-distillation task and proposes a Variation-aware Refine Module to achieve better segmentation. However, we observe that these strategies mainly depend on the RGB and position priors of pixels to refine CAM pseudo-labels, making them difficult to calibrate low-quality pseudo-labels that are severely impaired by CAM ambiguity. In this work, we use CAM to induce the class-aware affinity map from the self-supervised ViT, DINO, as the complementary guidance. This significantly enhances the performance of single-stage WSSS.Self-supervised learning aims to learn visual representations from unlabeled images by solving pretext tasks [15] . In this manner, their pretrained foundation models enable to outperform their supervised counterparts when transferring to downstream tasks. DINO [5] , a self-supervised ViT model, has been observed that it automatically emerges underlying class-agnostic semantic properties, which is not explicitly present in any supervised ViT or CNN [5, 40] .Leveraging the unique property of DINO, several studies proposed to detect a single salient object from each image using a graph constructed with DINO's patch tokens [26, 28] . In contrast, WSSS aims to segment single or multiple objects with specific semantics for each image. The proposed ECA integrates the CAM's class and position priors to induce each labeled class's class-aware affinity region via the constructed class-agnostic affinity graph from DINO. This perspective has not been studied in previous WSSS works.  
class activation map in transformers: 
 Initially, CAM is introduced for CNN to localize the discriminative regions for specific classes [38] . Because of its effectiveness and simplicity, CAM has been prevalently employed to generate the initial pseudo-labels for WSSS. For Transformer-based models, CAM is generated in a similar manner as that on CNN. Given an image, its patch tokens are treated as a feature map F ∈ R D×HW , where HW denotes the spatial dimension, i.e., the number of patches N, and D denotes the channel dimension. Subsequently, it is sent to a Global Max-Pooling layer with a classification layer to retrieve the classification score. In the above process, given C classes, the global view CAM of class 𝑐, denoted as M 𝑐 ∈ R HW , is generated by weighting and summing the feature map F using the classification weights W ∈ R D×C :••• … crop CAMs 𝓛 𝐜𝐥𝐬 DINO (ViT)••• ••• ••• ••• ! " "!EQUATIONwhere ReLU function is used to filter out negative activations. Then, min-max normalization is applied to rescale M 𝑐 to [0, 1]. For the background (class 0) CAM, denoted as M 0 , the value of patch at position 𝑖 is defined as:EQUATIONwhere M 𝑐 (𝑖) is the CAM activation value of class 𝑐 at patch 𝑖. Note that the background CAM is only adopted in Semantic Affinity Exploitation (Equation 3). During the pseudo-label generation stage, a hard background threshold 𝛼 is set to separate the foreground objects and background.Selection of Initial CAM Seeds. The CAM can provide the class and localization priors to induce the class-aware regions from DINO. Thereinto, the value of CAM can be regarded as confidence scores to select reliable initial seeds of its labeled class. However, some regions are often activated in multiple classes and may erroneously dominate the high-confident activations in multiple classes, producing incorrect seeds for subsequent processes. To this end, for each image's CAM, i.e., M ∈ R (C+1) ×HW , we deploy a mask I cam = argmax 𝑐 (M) to ensure that the position of each image patch is exclusively activated in one class during seed selection. Then, the initial CAM seeds of its labeled class 𝑐 are those patches within the top 𝑘% of nonzero activations:EQUATIONwhere Q 𝑐 represents the collection that records the position of class 𝑐's initial seeds, and ⊙ denotes the Hadamard product. In the above selection process, hyperparameter 𝑘 controls the percent of patches considered, where a smaller 𝑘 indicates higher confidence for selected initial seeds. Construction of Class-agnostic Affinity Graph. DINO [5] demonstrates that its self-supervised ViT can learn a perceptual grouping of image patches. Therefore, we leverage this merit to model the pair-wise affinity between any two image patches, which is noticeable when using the keys from its self-attention heads [26, 27] . For each image, given the keys of N patches from the last self-attention block of the frozen DINO, we construct an undirected class-agnostic affinity graph G aff ∈ R N×N . The affinity between patch 𝑖 and patch 𝑗, denoted as G aff (𝑖, 𝑗), can be calculated by:EQUATIONwhere 𝒌 𝑖 and 𝒌 𝑗 are the key of the patch at spatial position 𝑖, 𝑗 ∈ [0, N -1]. A higher G aff (𝑖, 𝑗) indicates stronger affinity between patch 𝑖 and patch 𝑗, implying that these two patches tend to belong to the same labeled class. Moreover, since the class token interacts with patches to represent the image, the attention between class token and patches can reveal valuable foreground object cues that facilitate the filtering of background patches. Thus, similarly to the construction of G aff , we establish the foreground-cue map for each image, denoted as I fg ∈ R 1×N , by measuring the similarity between the query of the class token and the keys of N patches. For the patch at spatial position 𝑖 ∈ [0, N -1], its value of I fg (𝑖) can be calculated as:EQUATIONwhere 𝒒 denotes the query of the class token, while 𝒌 𝑖 represents the key of the patch 𝑖. A higher value of I fg (𝑖) indicates a stronger affinity between the class token and patch 𝑖, suggesting that patch 𝑖 is more likely to be part of the foreground.Note that the DINO model used in the above process is always kept frozen, serving only for inference purposes. This ensures that the DINO model does not affect too much training efficiency. We exclude the Multi-head Self-attention mechanism in the above construction of G aff and I fg (Equation 4 and 5) for brevity. In actual implementation, the query and keys from each self-attention head are concatenated together to derive G aff and I fg .Generation of Class-aware Affinity Region Map. For each image, its initial CAM seeds are used to induce G aff to generate its corresponding Class-aware Affinity Region Map (CARM). Assuming that the image is labeled as 𝑐, the seeds from its Q 𝑐 first propagate along the rows of G aff , constructing the binary adjacency vectors A based on a threshold 𝛾. Formally, for the seed at position 𝑖 ∈ Q 𝑐 , the above process can be formulated as:EQUATIONSubsequently, the binary adjacency vectors A ∈ R | Q 𝑐 | ×N are aggregated with the foreground-cue map I fg to construct the original CARM of class 𝑐, denoted as M 𝑐 :EQUATIONwhere the value of M 𝑐 ∈ R 1×N intuitively represents the degree of each patch from a graph perspective. A patch with a higher degree in M 𝑐 suggests that it is more likely to belong to class 𝑐. Finally, we reshape M 𝑐 to the size of H×W and applied min-max normalization to rescale its values.Calibration of Class-aware Affinity Region Map. After generating CARM M for each image, it can be utilized as the semantic guidance for WSSS, following a similar manner as CAM. However, our early experiment indicates that CARM is susceptible to erroneous initial CAM seed selection owing to the over-activation issue, i.e., identifying the background patch as that of target objects. This situation will lead to imprecise propagation on the class-agnostic affinity graph and produce noisy CARM, which in turn degrades the final segmentation performance.To tackle this issue, we introduce the momentum prototype for each class to calibrate CARM. The motivation is that the momentum prototypes, updated by the tokens across the entire datasets, can serve as more robust class-representative tokens compared to the tokens from image instances. Consequently, we compute the similarity between image tokens and the prototypes to suppress unreliable CARM regions. Specifically, the momentum prototypes P are updated by the tokens corresponding to the seeds from Q. For each class 𝑐, its prototype P 𝑐 can be updated as:EQUATIONwhere t 𝑖 denotes the patch token of the selected seed at position 𝑖, 𝜏 is momentum for the prototype update, and | • | represents the collection cardinality. Then, given an image labeled as class 𝑐 with its feature map F ∈ R D×HW and prototype P 𝑐 ∈ R 1×D , we calculate the similarity map S 𝑐 ∈ R HW , where the value of the patch at position 𝑖, denoted as S 𝑐 (𝑖), can be calculated as :EQUATIONFinally, the similarity map S 𝑐 is reshape to H × W and the image's calibrated CARM of class 𝑐 will be:EQUATIONAfter calibration process, the CARM pseudo-label is produced as the complementary supervision of the segment decoder and the guidance of the semantic affinity relations in self-attention through affinity learning [24] .In the SAE module, due to the CAM ambiguity, the selected seeds sometimes fail to cover all components of target objects or be assigned to incorrect collections of Q. This phenomenon is exacerbated when the images have incomplete or local views, as noted by Wang et al. [29] . To alleviate the CAM inconsistency under the global and local views, CAM Correspondence Enhancement (CCE) is introduced to enhance the object activation completeness and the robustness of CAM for better seed selection. The objective of CCE is to align the CAM activations between the global view and multiple local views through establishing dense correspondences. Intuitively, if a pair of patches in the global view exhibits highly similar activations, the corresponding pairs of activations within the local views will be encouraged stronger similarity. Otherwise, the pair of activations in the local view will be encouraged to exhibit weaker similarity. The local views are generated by segmenting objects from the global view and applying re-augmentation to accentuate the discrepancy from the global view. To reduce the computational load, we utilize the CAM pseudolabel of the global view to establish the hard pairwise correlations (i.e., the activation similarity between two patches assigned with the same pseudo-label is considered as 1, otherwise it is 0). Following previous studies [24, 25] , we begin by filtering uncertain regions of CAMs to produce CAM pseudo-labels. Next, the patches that share the same pixel pseudo-label in the global view are identified as positive pairs, whereas those with different labels are designated as negative pairs. As illustrated in Figure 3 , the objective of CCE is two-fold: maximizing the similarity between patch activations belonging to positive pairs, and meanwhile, minimizing the similarity for those in negative pairs. This proposed correspondence loss is formulated as follows:EQUATIONwhere M 𝑛 (: , 1 : 𝑖) ∈ R C denotes the non-background CAM activation of patch 𝑖 at the local view 𝑙, and (R + 𝑙 , R - 𝑙 ) represent the collections of positive and negative pairs within each local view 𝑙, determined by the corresponding global CAM pseudo-label, respectively. Across all local views, from 𝑙 = 1 to 𝑁 , Jensen-Shannon divergence D is employed to measure the discrepancy between the activations of two patches as the training loss to promote the CAM's correspondence.As shown in Figure 2 , except the classification loss L cls and the joint segmentation loss of CAM and CARM pseudo-labels L seg and L seg , the proposed ECA also employ the auxiliary affinity learning loss L aff [24] to transfer the knowledge from CARM to the semantic affinity relations in multi-head self-attention. Following the common practice, these CAM and CARM pseudo-labels are refined by the online post-processing module, i.e., PAR [24] . The Multi-label Soft Margin Loss is adopted for L cls , and the Cross Entropy is adopted for both L seg and L seg . The optimization objective of ECA is the weighted sum of these loss terms:EQUATIONwhere 𝜆 𝑖 , 𝑖 ∈ {1, 2, 3} are the weights used to rescale the losses of different learning objectives. 
experiments 4.1 setup: 
 Datasets and evaluation metric. We conduct our experiments on PASCAL VOC 2012 [13] and MS COCO 2014 datasets [19] [20] . The initial learning rate is set as 6×10 -5 with a polynomial decay scheduler. The weight decay factor is set as 0.01. For the input images, they are cropped to 512 × 512, and applied augmentation strategies including random scaling with a range of [0.5, 2.0] and random horizontal flipping. The batch size is set as 8. The background threshold 𝛼 is set to 0.5 by default.The other hyperparameter settings of PAR and auxiliary affinity learning loss (Section 3.4) are followed by [24] . For the experiments on PASCAL VOC 2012, we train the network for 18k iterations with 2k iterations warmed up for the classifiers. These images of the local view for the CCE module are cropped to 256 × 256. They are re-augmented by random color distortion and horizontal flipping. By default, the loss weights of (𝜆 1 , 𝜆 2 , 𝜆 3 ) are all set as 0.1, 𝑘 in Equation 3 is set as 0.3, and the affinity threshold 𝛾 in Equation 6 is set as 0.15. For the experiments on MS COCO 2014, we train the network for 80k iterations. The loss weights of (𝜆 1 , 𝜆 2 , 𝜆 3 ) are set as (0.1, 0.05, 0.1) and 𝑘 is set as 0.15, while other settings remain the same. In the inference stage, following the common practice in WSSS, we adopt multi-scale inference and CRF post-processing [6] to obtain the final segmentation results.Comparison to State-of-the-art. We report the segmentation performance of ECA on PASCAL VOC 2012 and MS COCO 2014 in Table 1 . ECA achieves 70.9% (+1.1%), 71.4% (+0.9%), and 42.9% (+1.6%) mIoU on PASCAL VOC 2012 val, test and COCO val set, respectively, outperforming previous single-stage methods. Moreover, ECA demonstrates competitive performance compared to multi-stage methods, while significantly reducing the training cost. Notably, on the more challenging MS COCO 2014 dataset, our single-stage ECA even outperforms many recent multi-stage WSSS methods, demonstrating its robustness in more complex scenarios.Fully-Supervised Counterparts. The single-stage competitors presented in Table 1 adopt various backbones. For a fair comparison, we examine the performance of their fully-supervised counterparts on the VOC 2012 val set. The results in Table 2 demonstrate that ECA significantly boosts the utilization of image-level annotations for semantic segmentation. ECA improves by +4.6% compared to methods using the same MiT-B1 backbone and by +3.4% compared to recent state-of-the-art work, i.e.., ToCo [25] .Per-category segmentation performance. The per-category segmentation results on PASCAL VOC val set is tabulated in Table 3 . We can see the proposed ECA achieves remarkable performance improvement in most semantic classes (16 of 21) . We can see the proposed ECA outperforms prior-arts over 5% in many classes. Especially, in the class of aeroplane, bicycle, cat, train, which are susceptible to the over-activation issue, ECA improve their performance by +7.3%, +7.2%, +6.6%, +8.4%, respectively. [35] , 1Stage [3] , AA&LR [36] , and AFA baseline [24] . The improvement over 5% are marked in color .Qualitative Results. In Figure 5 , we visualize the segmentation results of our ECA on the VOC 2012 val and MS COCO 2014 val sets, and compare them with the recent state-of-the-art methods, i.e., AFA [24] and ToCo [25] . We can clearly observe that the proposed ECA demonstrates a significant improvement over AFA and ToCo. ECA effectively produces segmentation results that align more precisely with object boundaries, which are close to the ground-truths. Particularly, we can find that both AFA and ToCo produce many false-positive predictions caused by the pseudo-labels suffering from the semantic ambiguity problem of CAM. With the DINO as the semantic guider for segmentation, ECA can overcome this problem well. Also, we present the CAM visualization in Figure 5 .We can observe that ToCo activates more non-target background regions and fails to activate some less discriminative regions. In contrast, the proposed ECA can achieve better object activations and provide high-quality CAM for seed selection and pseudo-label generation.Ablation. We investigate the impact of each component in our proposed ECA. The results of the ablation studies are reported in improves the CAM and segmentation performance to 63.2% and 65.0%, respectively. Next, we validate the effectiveness of joint supervision (i.e., L seg and L seg ) for the segmentation decoder. By replacing CAM with CARM to generate pseudo-labels as guidance, we observe improvements of 3.4% and 2.1% in CAM and segmentation performance, respectively. When we jointly use the CAM and CARM pseudo-labels, they can further improve the segmentation performance to 68.4%. We speculate that CAM and CARM serve complementary roles, where erroneous gradient updates caused by one pseudo-label are calibrated by the other, thereby enhancing the robustness and accuracy of the segmentation model. Furthermore, the CCE module significantly enhances the CAM performance with an increase of 2.3%, demonstrating that the local-to-global CAM correspondence benefits the robustness of CAM performance. With better CAM, which generates better pseudo-labels and seeds in SAE module, the segmentation performance further promotes to 69.4%, leading to the state-of-the-art.ViT Architecture in SAE. The motivation of the SAE module is using the powerful class-agnostic affinity from DINO to yield ToCo 5 . We can observe that the ViT fully supervised on ImageNet-1k cannot generate effective CARMs for semantic guidance, leading to catastrophic segmentation performance. Compared with other self-supervised ViT models [7, 22] , we can find that DINO is the best choice for SAE. Further, we test the patch size of DINO and its ViT architecture. The results show that a small patch size of 8 is suitable for producing CARM, archiving the best segmentation performance, while a larger architecture (i.e., ViT-B) would not benefit our proposed ECA.ECA ImageAnalysis of CARM. In Figure 6a , we visualize the effect of CARM calibration in SAE. We show some challenging cases where the initial seeds involve erroneous localizations, subsequently leading to inferior pseudo-labels. After CARM calibration, we can find that numerous noise labels are successfully calibrated, thus improving the quality of these pseudo-labels. In Figure 6b , we demonstrate that CARM (after calibration) adeptly captures the foreground and background object information, enabling clear identification of the foreground objects. Notably, it's intriguing to observe that the background CARM exhibits strong cues of foreground objects. In single-class cases, the background CARM alone is competent in providing high-quality pseudo-labels for the segmentation decoder.Initial Seed Selection. We utilize top 𝑘 activations of CAM as the initial seeds to exploit the class-aware affinity region for each labeled class. SAE with a smaller 𝑘 select fewer but more reliable seeds. Conversely, SAE with a larger 𝑘 select more seeds that cover broader object regions, thus benefiting the completeness of CARM. Table 6a shows 𝑘 = 0.3 is the best choice but other values smaller than 0.3 can also yield favorable performance.Size of Local View. The CCE module aligns CAM activations between global and local views to enhance the performance and reliability of CAM for better seed selection and CAM pseudo-labels.In Table 6b , we report the impact of local view size in CCE. It shows that the crop size of 256 and 384 both achieve favorable performance. Considering the computational load of establishing dense correlation pairs in CCE, we choose the crop size of 256 in our experiments by default. 
conclusion: 
 This paper explores the distinct property of DINO to extract semantic guidance for single-stage WSSS, aiming to overcome the limitation of using CAM pseudo-labels. Specifically, highly-activated patches are selected as the initial seeds to induce Class-aware Affinity Region Map for each labeled class through the class-agnostic affinity map established by DINO. The CARM captures precise classspecific region affinities, thus offering high-fidelity pseudo-labels for both the segmentation decoder and affinity learning processes, enhancing the final segmentation outcomes. Moreover, motivated by the inconsistency between the global and local views, we propose CAM Correspondence Enhancement to align the CAM of global and local views through the dense correspondences, improving the performance of CAM for seed selection and the CAM pseudo-label generation. On PASCAL VOC 2012 and MS COCO 2014 datasets, our method achieves new state-of-the-art performance in single-stage WSSS and even outperforms numerous multi-stage WSSS methodologies on MS COCO 2014. From a broader view, the proposed method provides a novel perspective that leverages the powerful affinity modeling capability of DINO to excavate semantic guidance to overcome the CAM limitations.  
 
 Appendix:  


Decision: The contributions offer reasonable novelty. State-of-the-art results.

Reviews: review: paper_summary: paper_summary: The article proposes a novel approach to generate semantic guidance using the self-supervised nature of DINO model to address the limitations of CAM pseudo-tagging when using only image-level tags for semantic segmentation. Specifically, this paper proposes semantic affinity exploitation module to establish class-agnostic affinity graph by self-attention of DINO. Further, the class-aware affinity region map is yielded as supplementary semantic guidance. The paper also proposes CCE to encourage dense local-to-global CAM correspondences.

Experiments are conducted on VOC and COCO to validate the proposed methods.
strength_weakness: strength_weakness: 1. The article designs a Semantic Affinity Exploitation module that can efficiently utilize the semantic affinity of DINO to generate high-quality Class-Aware Affinity Region Mapping (CARM), thus providing high-fidelity pseudo-labels for the segmentation decoder and affinity learning process.

2. The article proposes a CAM Correspondence Enhancement module that can improve the performance of CAM by establishing a dense correspondence between global and local perspectives, thus improving seed selection and CAM pseudo-label generation.

3. New state-of-the-art performance was achieved for the article's approach on both PASCAL VOC and MS COCO datasets.
limitations: limitations: 1. There is an insufficient introduction of DINO, which is important in the paper.

2. ECA does not go far enough in explaining the workings of the DINO model and the limitations of CAM and could have further elaborated on the connection and motivation for combining the two.

3. The analysis of the method's time complexity and computational resource consumption is not comprehensive enough, and consideration could be given to include an analysis and discussion of this aspect.
paper_summary: paper_summary: This paper introduces a new approach, dubbed ECA, to Exploit the self-supervised Vision Transformer, DINO, inducing the Class-aware semantic Affinity to overcome this limitation. Specifically, It introduces a Semantic Affinity Exploitation module (SAE). It establishes the class-agnostic affinity graph through the self-attention of DINO. Using the highly activated patches on CAMs as “seeds”, it propagates them across the affinity graph and yield the Class-aware Affinity Region Map (CARM) as supplementary semantic guidance. Experimental results demonstrate that ECA improves the model's object pattern understanding.
strength_weakness: strength_weakness: * Novelty: The ECA framework introduces a novel Semantic Affinity Exploitation (SAE) module that leverages the self-supervised Vision Transformer DINO to generate Class-aware Affinity Region Maps (CARM) as complementary semantic guidance. This innovative approach effectively addresses the limitations of traditional Class Activation Maps (CAMs) in WSSS.
* Theoretical Approach: By incorporating DINO to induce class-aware affinity from CAMs, ECA bridges the gap between class and localization tasks, providing more accurate pseudo-labels for segmentation supervision. This theoretical approach enhances the quality of segmentation outcomes and overcomes the ambiguity inherent in CAMs.
* Technical Correctness: The ECA framework demonstrates technical correctness by effectively utilizing the semantic properties of DINO to guide the segmentation process. The proposed SAE module and CAM Correspondence Enhancement (CCE) module contribute to improving the quality of pseudo-labels and segmentation results, showcasing the technical soundness of the approach.
* Adequate Evaluation: The paper provides thorough empirical evidence and ablation studies to validate the effectiveness of the ECA framework. Comparative results with state-of-the-art methods such as AFA and ToCo demonstrate significant improvements in segmentation accuracy and object boundary alignment. The evaluation methodology ensures the robustness and reliability of the proposed approach.
*  Clarity: The paper effectively communicates the proposed framework, modules, and experimental results in a clear and concise manner. The methodology, results, and implications are well-explained, making it easy for readers to understand the contributions and significance of the ECA framework.
* Applications: The ECA framework has practical applications in various domains requiring semantic segmentation, especially when dealing with limited annotation resources. The improved performance in single-stage WSSS using image-level labels opens up possibilities for more efficient and accurate segmentation tasks in real-world applications.
limitations: limitations: * While the proposed ECA framework introduces innovative modules like Semantic Affinity Exploitation (SAE) and CAM Correspondence Enhancement (CCE) to enhance segmentation performance, the fundamental concept of leveraging semantic affinity from DINO for segmentation guidance has been explored in prior works.
* The paper could benefit from a more extensive evaluation to validate the generalizability and robustness of the proposed ECA framework 2. While the empirical results on the PASCAL VOC dataset demonstrate significant improvements in per-category segmentation performance, a broader evaluation on diverse datasets or real-world scenarios would strengthen the paper's contributions and practical applicability.
paper_summary: paper_summary: This paper tackles the problem of weakly supervised semantic segmentation. A single-stage framework named ECA is proposed to fully exploit the potential of semantic affinity in DINO. Furthermore, a CCE module is designed to promote the dense local-to-global correspondences of CAM. Experimental results on benchmark datasets show that the proposed method outperforms previous methods.
strength_weakness: strength_weakness: 1. This paper innovatively explores the utilization of DINO as a Semantic Guider for weakly supervised semantic segmentation.
2. This paper demonstrates improved performance compared to existing one-stage methods.
3. The rationale behind each of the new components (e.g., SAE, CCE) is intriguing.
4. The experiments and ablation studies are comprehensive and provide strong support for the author's claims.
limitations: limitations: 1. The abstract involves multiple modules, but they lack clear logical connections, making it difficult to read.
2. The methods section lacks an overview of the entire framework, including details on the overall data flow.
3. The loss function L_{align} in Figures 2 and 3 is not introduced in the main text and is not part of the overall optimization objective in Equation 12. What does the red text "CAE" in Figure 2 represent?
4. The notation needs to be consistent throughout the text. For example, should there be a multiplication sign between the spatial dimensions HW? What is the relationship between N and F in line 228?
5. Why are the results for ECA in Tables 3 and 1 inconsistent while the results for other methods are consistent?
6. Are the results for ToCo in Table 1 consistent with their published results? They reported better results using ImageNet pre-training. Can the proposed ECA also benefit from the pre-training dataset used by ToCo?
7. Compared to the two-stage approach, the single-stage approach has the advantage of training efficiency. However, does it have an advantage in terms of training time complexity compared to other single-stage methods?
8. In Equation 12, the hyperparameters for the loss functions are set very low, making L_{cls}​ significantly dominant. What is the reason for this? Does simply assigning it a smaller weight affect performance?
paper_summary: paper_summary: The manuscript exploits the potential of semantic affinity existing in DINO and proposes a CAM Correspondence Enhancement module (CCE) to promote the dense local-to-global correspondences of CAM.
strength_weakness: strength_weakness: 1. The manuscript introduces the DINO foundation model to CAM optimization.
2. The CAM is calibrated by the prototype method.
3. The local view is introduced to improve the performance.
limitations: limitations: The problems include: 
1. “Foundation Model Assisted Weakly Supervised Semantic Segmentation”[1] has been proposed. The idea of using large model’s class-agnostic affinity graph is incremental.
2. Please discuss the computation cost of the proposed method. 
3. The performance is worse than the reference [2], about 72.7 and 72.8.
4.  CAE in Fig 2 is error.
5. “Qualitative Results. In Figure 5” in page 6 should be Figure 4.

References:
[1] Yang X, Gong X. Foundation model assisted weakly supervised semantic segmentation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 523-532.
[2] Chen T, Yao Y, Huang X, et al. Spatial Structure Constraints for Weakly Supervised Semantic Segmentation[J]. IEEE Transactions on Image Processing, 2024.