Title: CG-SSL: Concept-Guided Self-Supervised Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representation Learning

Curriculum Learning

Self-supervised Learning

Vision Foundation Models

Vision Transformers
Abstract: Humans understand visual scenes by first capturing a global impression and then refining this understanding into distinct, object-like components. Inspired by this process, we introduce \textbf{C}oncept-\textbf{G}uided \textbf{S}elf-\textbf{S}upervised \textbf{L}earning (CG-SSL), a novel framework that brings structure and interpretability to representation learning through a curriculum of three training phases: (1) global scene encoding, (2) discovery of visual concepts via tokenised cross-attention, and (3) alignment of these concepts across views.
Unlike traditional SSL methods, which simply enforce similarity between multiple augmented views of the same image, CG-SSL accounts for the fact that these views may highlight different parts of an object or scene. To address this, our method establishes explicit correspondences between views and aligns the representations of meaningful image regions. At its core, CG-SSL augments standard SSL with a lightweight decoder that learns and refines concept tokens via cross-attention with patch features. The concept tokens are trained using masked concept distillation and a feature-space reconstruction objective. A final alignment stage enforces view consistency by geometrically matching concept regions under heavy augmentation, enabling more compact, robust, and disentangled representations of scene regions. 
Across multiple backbone sizes, CG-SSL achieves state-of-the-art results on image segmentation benchmarks using $k$-NN and linear probes, substantially outperforming prior methods and approaching, or even surpassing, the performance of leading SSL models trained on over $100\times$ more data. Code and pretrained models will be released.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes \"Concept Guided Self-Supervised Learning\", which is a technique for training a vision model without labels to produce generally useful representations, both for summary tasks (e.g. classification, retrieval), and dense tasks (e.g. semantic segmentation). Similar in spirit to techniques such as iBot and DINOv2, CG-SSL places additional constraints on the model, namely that the patch tokens more coherently represent semantically related regions in the image. DINOv2, being the preeminent SSL method, doesn't explicitly enforce that tokens that should have semantically similar representations, actually do. CG-SSL enforces this coherence through cross attention from a fixed number of queries to the dense features, and matching those to unmasked momentum self-features, and then also reconstruction via a weighted sum of the spatially broadcast concept queries. Additionally, they employ a stage 3 that encourages views of the same objects/regions to be spatially equivariant by aligning and then matching in a stage 3 training curriculum.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThe proposed method appears to produce cleaner feature maps compared to contemporary methods.\n\nConcept regions to appear to map to coherent regions.\n\n### Weaknesses\n\nAblations are performed against DINO and iBot, however, DINOv2 is considered a much stronger baseline. How does CG-SSL's classifier ability compare to DINOv2?\n\nCG-SSL is strongly better than CAPI for ViT-S, however, for ViT-B and larger, CAPI is arguably stronger.\n\nD2V2 is only presented in acronym form, without citation. I might have guessed this was some shorthand for DINOv2, however, both D2V2 and DINOv2 are present in Figure 1.\n\nQUESTIONS:\nThe DINOv2 [1] paper claims the following ADE20k linear probe mIoUs:\n- ViT-S/14: 44.3\n- ViT-B/14: 47.3\n- ViT-L/14: 47.7\n\nhowever, in this paper, it is claimed that DINOv2 achieves the following:\n- ViT-S/14: *\n- ViT-B/14: *\n- ViT-L/14: 39.0\n\nCan you please explain this discrepancy?\n\n\nIn E.3, the following is stated: \"In the main ImageNet experiments, we set N = 4 as the dataset is less complex.\" How does the number of concept tokens affect the generalizability of the method, given that one dataset may have far fewer concepts than another?\n\n[1] https://arxiv.org/pdf/2304.07193\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-17"}

comments: SUMMARY:
The authors propose a new self-supervised learning (SSL) method for visual pretraining. Unlike existing methods utilizing image-level or patch-level consistency given some well-design augmentations, they utilize concept-level or object-level consistency. It seems to be inspired both from Object-Centric Learning and self-supervised segmentation, with some adaptations.

STRENGTHS AND WEAKNESSES:
**Strengths**
- Well-motivated
- Clear presentation
- Novel design

**Weaknesses**
- The training details can be very important to such SSL, but the authors did not provide much of them. Should include a section in the appendix to expose your "secrete recipe".
- Plain (but quite okay) analysis in the concepts learned: more in-depth analysis can give readers more impactful insights.
- Lacks intuitive comparisons between your method and the baselines.
- Figure/Table captions should be clearer: Like in Figure 2, it is better to tell readers meanings of all the notations; In Table 3, should briefly clarify what are the meanings for each subtable, instead of telling readers to jump to the appendix.
- The numbers of some equations are missing, like the one between Line 162 and 163.

QUESTIONS:
**Q1**
The attention defined between Line 162 and 163: Why do the authors not choose the competitive cross attention, i.e., Slot Attention, from object-centric learning? What are the possible effects, both in performance and explanability like in Figure 5?

**Q2**
What are the details training costs of your method and of some representative baselines? Like training time, number of iterations, memory consumption.

LIMITATIONS:
No. The authors did not provide discussions on their limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new self-supervised approach (CG-SSL) to learning visual representations that aims to more fine-grained representations by learning to group patches into regions and compare these across views. The proposed approach consists of 3 training phases.\n\nIn the first phase, they aim to learn global representations by using an earlier technique called iBot that learns by comparing the patch level embeddings from a teacher network that observes the full image and a student network that takes in a masked version.\n\nIn the second phase, they learn to group regions by training a cross-attention transformer that uses learned concept tokens to attent to image features and extract N regions. This phases uses 2 training objectives: 1) concept level distillation uses the iBot loss but between each concept token separately (again comparing student and teacher embeddings) 2) feature reconstruction loss uses a MONet, SIMONe style multi-object reconstruction setup where they predict patch features and a mask for each spatial location from each concept. These masks and patch features predictions are then combined to reconstruct the full image level patch features and compared to the true patch features from the teacher.\n\nIn the final third phase, they aim to improve learned representations by aligning concept representations across two views of the same image. To do that, they get a mask for each concept by thresholding the output of teacher network's mask logits. Then transform these masks using the known transformation between views. These masks are then used to pool the patches from each view to form concept representations for each view, which are then compared across views using the usual iBot style loss.\n\nOn evaluations, CG-SSL leads to some improvement most on segmentation while performing on-par or slightly better on classification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Overall well written and easy to follow\n- Although the individual components are not entirely novel, combines various ideas from the literature in a novel way\n- Good performance improvement on segmentation compared to baseline\n- Good ablation study\n\nWeaknesses:\n\n- Needs a more thorough evaluation to make the case for the proposed approach stronger\n  - There are various papers in the literature that explore similar ideas around splitting the image into multiple regions\n  - It would be good to compare to at least some of these (like DORA, Venkataramanan et al.,\"Is ImageNet worth 1 Video? Learning Strong Image Encoders from 1 Long Unlabelled Video,\")\n\nQUESTIONS:\n- Are the phases optimised jointly? Line 205 says phase 1 and 2 are optimized together. Line 226 says that phase 3 loss is added only after convergence of phase 1 and 2.\n- Concept level distillation loss in eqn. 4 is applied on each concept separately, right? Might be good to make this explicit.\n- For feature reconstruction loss, why not use the attention weights from each concept to patches for masks? Did the authors try this? Does this lead to worse results?\n- For evals, why not use attentive probing as well? This could be useful to compare to previous work.\n- It'd be nice to see the learned masks. Currently, only attention maps are plotted in Figure 5. How about a similar figure for masks?\n- What is CAPI + LVD 142M. This is probably explained in the appendix but would be useful to mention in main text.\n- line 175: \"Given that\" should be removed?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers self-supervised learning. Its approach is inspired by human perception: humans first grasp the global structure of a scene, then organize the visual input into coherent groups that they are then able to track in changing conditions.\n\nSimilarly, the proposed self-supervised learning method is split into three phases. The first phase captures glabal image features with the existing iBot approach which aligns cls and patch level embeddings produced ba a student-teacher pair of networks for two augmented views. cls token features are matched across the two views, while the patch token features are matched within the same view. \n\nIn the second phase, the student and the teacher network process the same augmented image. A trainable matrix C is initialized to encode N concepts. These concepts interact with student/teacher features with a cross attention mechanism. Token representations in C serve as queries, while transformer block outputs serve as keys and values. This allows each token to iteratively refine its representation of concepts within an image. Student and teacher networks are further aligned with a cross-entropy loss between its concept sets. An additional objective is used to ensure that teacher's patch-level features may be reconstructed and concepts capture all regions of a given image. This is done by broadcasting each token representation over a 2D spatial grid and calculating a corresponding confidence map . The confidence indicates the level of alignment between the concept and image regions. Broadcasted features are multiplied with confidences and fed to a reconstruction loss.\n\nIn the final phase, binary masks for concepts within a reference image are extracted with a teacher network. Lightly augmented images are created and crop metadata is used to obtain appropriate concepts masks for novel views . Concepts are aligned across vies with a cross-entropy loss.\n\nExperiments are don on ImageNet-1k, Pascal VOC, Visual Genome and MS-Coco.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The topic of self-supervised learning is important\n2) Good experimental results demonstrate coherent representation and good performance in classification and dense prediction, as well as robustness to occlusions.\n3) Authors provide the code with the submission\n\nWeaknesses:\n1) The paper is difficult to follow and poorly written. The method could be explained more concisely. E.g., the sentence in lines 175-176 does not really make sense on its own or connect to the previous and following sentences. It is not clear if there is one or two concept matrices C in the second phase.\n2) The results could also be better presented and contextualized. It is not clear in the tables which was the training data used in each of the experiments. Why does the method not outperform some of the exisitng self-supervised approaches etc.\n3) Additional ablations could be useful, especially those that explore phase scheduling. How long is each phase performed individually? How are the losses balanced? Is the training sensitive to loss weighting?\n\nQUESTIONS:\n1) Would an increase in the number of tokens lead to more low-grained scene understanding?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}