CG-SSL: Concept-Guided Self-Supervised Learning
Authors: Sara Atito
Josef Kittler
Imran Razzak
Muhammad Awais
Venue: NeurIPS 2025 poster ()
Keywords: Representation Learning
Curriculum Learning
Self-supervised Learning
Vision Foundation Models
Vision Transformers
Primary area: deep_learning
TL;DR: We introduce CG-SSL, a concept-guided self-supervised learning framework that aligns meaningful image regions across views, achieving state-of-the-art performance on dense prediction tasks.

Abstract: Humans understand visual scenes by first capturing a global impression and then refining this understanding into distinct, object-like components. Inspired by this process, we introduce \textbf{C}oncept-\textbf{G}uided \textbf{S}elf-\textbf{S}upervised \textbf{L}earning (CG-SSL), a novel framework that brings structure and interpretability to representation learning through a curriculum of three training phases: (1) global scene encoding, (2) discovery of visual concepts via tokenised cross-attention, and (3) alignment of these concepts across views.
Unlike traditional SSL methods, which simply enforce similarity between multiple augmented views of the same image, CG-SSL accounts for the fact that these views may highlight different parts of an object or scene. To address this, our method establishes explicit correspondences between views and aligns the representations of meaningful image regions. At its core, CG-SSL augments standard SSL with a lightweight decoder that learns and refines concept tokens via cross-attention with patch features. The concept tokens are trained using masked concept distillation and a feature-space reconstruction objective. A final alignment stage enforces view consistency by geometrically matching concept regions under heavy augmentation, enabling more compact, robust, and disentangled representations of scene regions. 
Across multiple backbone sizes, CG-SSL achieves state-of-the-art results on image segmentation benchmarks using $k$-NN and linear probes, substantially outperforming prior methods and approaching, or even surpassing, the performance of leading SSL models trained on over $100\times$ more data. Code and pretrained models will be released.

CG-SSL: Concept-Guided Self-Supervised Learning
Sara Atito .1.2
Josef Kittler²
Imran Razzak
Muhammad Awais
Surrey Institute for Crople-Centred AI, University of Surrey, UK CCntte for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK 3 Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
sara ati atito@surrey ac, ac.uk, imrrnn.inn.. www..iaaaa.azzaanmmaaaaa.te wwwzaaammzuaii..ccccomm acc.ae j kkittlerdsurrey,
muhammad, awaisdsurrey ac uk
Abstract
Humans understand visual scenes by first capturing global impression and then refining this understanding into distinct, object-like components. Inspired by this process, introduce Joncept-Guided Self-Supervised Learning (CG-SSL), novel framework that brings structure and interpretability to representation learning through curriculum of three training phases: (1) global scene encoding, (2) discovery of visual concepts via tokenised cross-attention, and (3) alignment these concepts across views. Unlike traditional SSL methods, which simply enforce similarity between multiple augmented views of the same image, CG-SSL accounts for the fact that these views may highlight different parts of an object or scene. To address this, our method establishes explicit correspondences between views and aligns the representations of meaningful image regions. At its core, CG-SSL augments standard SSL with a lightweight decoder that learns and refines concept tokens via cross-attention with patch features. The concept tokens are trained using masked concept distillation and feature-space reconstruction objective. A final alignment stage enforces view consistency by geometrically matching concept regions under heavy augmentation, enabling more compact, robust, and disentangled representations of scene regions. Across multiple backbone sizes, CG- SSL achieves state-of-the-arr results on image segmentation benchmarks using k- NN and linear probes, substantially outperforming prior methods and approaching. surpassing, the performance of leading SSL models trained over x00x more data. Code and pretrained models will be relcased.
1 Introduction
Deep neural networks have achieved huge success across a wide range of computer vision tasks, but this success has come at the cost of increasingly large and expensive labeled datasets. Self-supervised learning (SSL) offers scalable alternative, enabling models to learn generalisable representations from unlabeled data. SSL has shown promise not only in mainstream tasks such as image classification
and segmentation 2 3. but also in domains where annotations are limited costly, such as medical imaging 4 5 6) satellite imagery [7]. underwater exploration [ and beyond.
Within the SSL landscape,
families of approaches have proven particularly effective. Generative methods, such as masked image modeling (MIM) 99] 10 12 13], train models to reconstruct occluded patches of input images. These methods particularly effective learning spatially grounded features and fine-grained textures. However, they often fall short when it comes to capturing high-level semantic information, typically underperforming on transfer tasks that require robust, discriminative features [4]. In practice, such models require extensive fine-tuning and
Project Lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ADE20K-
ADE 20K Linear
Pascal
70
Consistence
V
25
30
55
20
25
50
55
22
20
50
VI1-5
VIT-B
VIT-
VIT-5
VIT-B
VIT-L
VIT
VIT-B
VIT-
VIT-B
VIT
(a)
NC 591
1507
D2V2
MAE
CAPI
CO:53L
(LVD-582X))
CAPI
(LVD-13N))
(b)
Figure 1: (a) CG-SSL overview: The model segments an image into N coherent "concept" regions. Original image is passed through the teacher to obtain attention masks. Using the known geometry of the crops, these masks are accurately mapped augmented views 01, 02. Tokens from each region are aggregated and consistency loss is applied between corresponding regions. (b) We evaluate frozen representations using k-NN and linear probes. CG-SSI. outperforms prior baselines, scales well with model size, and in some cases surpasses models pre-trained on much larger datasets.
fail to generalise well with simple linear probes. In contrast, invariance- based methods [T5] 16
17 18 19 focus on aligning the representations of different augmented views of the same image. These approaches produce features that are generally more transferable and better suited to global classification tasks. Nonetheless, they rely on a critical assumption that the applied augmentations do not alter the semantic content of the scene. In cluttered complex visual environments, this assumption frequently breaks down. Moreover, by compressing the image into single vector representation, these methods tend to discard spatial information, making them poorly suited for tasks that require localisation or fine-grained reasoning, such as object detection and segmentation.
Recent works have attempted bridge the gap between these paradigms. Methods like iBoT [16 20 and DINOv2 [2]] integrate token-level predictions and masked modeling objectives to enhance spatial awareness. While these models demonstrate strong performance and scalability, they continue to treat image patches largely as independent units and do not explicitly encourage the formation of coherent visual entities. As such, they fall short of modeling the structured, hierarchical nature of visual perception observed in humans, where coherent regions are grouped [2 233.
have made progress toward structured visual representations. However, many of these methods depend on pre-trained models or heuristic objectives, which limits their scalability and hinders end-to-end learning.
Finally, recent advances in region- and object-centric SSL 224 25 26
To address these limitations, we propose Concept-Guided Self- Supervuee Learning (CG-SSL), a framework that explicitly discovers, tracks, and aligns coherent visual regions "referred to as concepts" across augmented views. Rather than relying on implicit alignment through crop overlap, CG-SSL learns set of concept tokens dedicated embeddings that attend to image patches via cross-attention. These tokens are refined using two auxiliary objectives: masked concept distillation loss that promotes consistency, and feature-space reconstruction loss that encourages diversity and completeness. In the final phase, CG-SSL tracks these discovered concepts across augmented views by leveraging known geometric transformations. Using reference image (a lightly augmented version of the original no cropping) as a stable anchor, we project attention masks into each augmented view and aggregate the corresponding features to enforce cross- view consistency. This structured alignment enables CG-SSL to learn stable and interpretable representations without any supervision.
By combining global understanding with region-level structure and view ciw-ccossistent alignment, CG-SSL moves beyond existing limitations in SSL by producing representations that are not only transferable but also interpretable and spatially coherent. We demonstrate that CG-SSL achieves state-of-the--rt (SOTA) performance on image segmentation benchmarks using only simple probes, and approaches or surpasses models trained on orders of magnitude more data.
2 Related Works
SSI. methods largely fall into two broad families: generative approaches that reconstruct input signals, and invariance-base based approaches that learn representations by enforcing consistency across views.

---

As our work aligns more closely with the latter, we focus here on anvarianee-based methods, which we organise into three levels of granularity; image-level, patch-level, and region-level.
Image-ILeve SSL., A key milestone in SSL was instance discrimination 28 where each image is treated its own class. This formulation paved the way for contrastive approaches such InfoNCE [29]. SimCLR [9) and MoCo [17]. which encourage agreement between views of the same image while contrasting them against others. These methods achieved solid results on downstream tasks, establishing cross-view consistency as central principle in SSL. Subsequent work focused on preventing collapse. Negative sampling [3) remains one solution, though it often relies on large batch sizes. Alternatives include momentum encoders [3], clustering- -based objectives [8) and entropy- -based regularisation [3). all of which stabilise learning without explicit negatives.
While these methods vary in their learning objectives, they share a common assumption: the image treated holistic unit. However, this global perspective overlooks the rich spatial structure of natural scenes. To address this, recent efforts have shifted toward learning at the patch or region level, where spatial relationships and part-whole semantics play a central role.
Patch-Level SSL. Pixel- and patch-level SSL extends instance discrimination from global image embeddings to local features within the spatial grid 33
[6 20 21]. These methods aim to learn dense representations by enforcing local consistency within and/or across augmented views. The primary distinction among them lies in how positive patch correspondences are defined. Some works 33 28
34
35
36
use contrastive losses (c.g., InfoNCE) where matching pairs are drawn from the same spatial location in the student and teacher outputs, while others R5 36) rely on nearest-neighhor patch alignment in feature space across views. Another class of methods, including iBOT (20) and DINOv2 21, extend global cross-entropy objectives the patch level, matching representations of masked and unmasked views in studentteacher setup.
These patch-level objectives yield strong performance on dense prediction tasks. However, they still treat each patch independently, without explicit modeling of spatial coherence across regions. As a result, they may fail capture structured patterns like compositional regions, which motivates the transition to region-level methods that learn to group patches into coherent regions/concepts.
SSL. While image- and patch-level SSL has seen great success, recent work
has shifted toward learning structured visual representations by grouping regions or objects within images. However, many of these methods are primarily designed for unsupervised segmentation or object discovery, rather than general- purppose SSL. For example, PICIE (37), STEGO (38], and SegDiscover [39] cluster dense pixel features to enable semantic segmentation, often relying hand-tuned spatial priors or post-processing Other approaches like SlotCon [24 25), CrOC (26).
and CriBo [2 aim to learn object-centric representations, but often depend on clustering external region grouping, limiting scalability.
In contrast, CG-SSL is designed as general-purpor self-supervised framework. It learns to discover and align coherent visual concepts fully end-to-end manner, without pretraining, offline clustering, or handcrafted proposals. Concept tokens emerge via attention-based grouping, and cross-view consistency is enforced through geometric alignment of discovered regions.
3 Methodology
We propose CG-SSL framework, which follows three-phase curriculum: (1) capturing the global image gist, (2) segmenting the scene into coherent regions, and (3) aligning these regions across viewpoints. Each phase builds the previous one, progressively enriching the learned representations. Training is staged, with each phase introducing new objectives while retaining earlier ones which enables the model to develop from broad understanding fine-grained, aligned representations, resulting in robust and interpretable SSL. An overview and results are shown in Figure I
3.1 Phase I: Global Representation Learning
Human perception begins with an immediate grasp of the global structure of a scene 40 41 Cognitive psychology refers to this as global precedence, where low spatial frequency information helps the visual system quickly form a "gist" of scene before identifying individual components.
3

---

The content of gist usually includes a conceptual understanding of a scene, e.g. birthday party, the spatial layout of the environment, and the identity of few objects 42
In machine vision, SSL has paralleled this phase through invariance-based methods, which aim to learn global representations by making the model recognise that different views of the same image correspond to the same underlying content. While this first phase of global feature learning is crucial, also well-established and heavily explored subfield of SSL. Rather than proposing new formulation, we adopt iBoT [20] a successor to DINO [8], because of its strong empirical results and architectural simplicity. iBoT aligns both image-level cls token and patch-level embeddings produced by a studentteacher pair of Vision Transformers (ViTs).
Let E,,&; E, denote the student and teacher encoders, and p! pletel patchh Pl plpatcal their respective projection heads for the cls and patch streams. From a single image x E # xx we draw two stochastic augmentations v1, 02 ~ T(x) E R typically h 224, that channel dimensions are dropped for simplicity. The teacher receives the fully visible view 222 whereas the student is given 11 after randomly masking subset of its patches, denoted 01. The projected probability vectors are
224. Note
[cla] softmax Plotol (elotn (61)) a 8,01
plclol (c/(h)) clola)
[els] 2,00
softmax
TI
and analogously [natch] [patch] for patch tokens are computed. Here т, and are temperature parameters and clols| is running mean (centering) of the cls features, which prevents collapse.
a a
The student is updated by minimising the cross entropy between its predictions and those of the teacher. The cls token features are matched across views and the patch token features are matched within the same view.
Lele = o [c1c] The overall iBoT objective is the weighted sum: LiBet a1 Liassl + 02 C patchi) where a1, 02 > 0 balance the class- and patch-level signals. The teacher parameters are updated via an exponential moving average (EMA) of the student: 0, +- 20, + (1 A)0s, with À following cosine
dfsu log (1.1)) Lpatch -o log ds
|patch]
|attch dt,r2
[patch] (2) log 2,002
schedule from 0.996 to during training 15
3.2 Phase 2: Discovering Visual Concepts
While the global representations learned in Phase 1 capture overall scene identity, many downstream tasks demand finer, object-level understanding. Cognitive research shows that humans naturally organise visual input into coherent groups based principles such similarity, proximity, and continuity, a process known as perceptual grouping or segmentation [43] 44. Inspired by this, we propose decompose each image into N distinct regions. Each region is represented by dedicated concept token, as illustrated in Figure 2 which serves as a compact representation of that visual component. Several elements of this phase are informed by insights from the object-centric learning literature 45 46) and are designed to promote the emergence of structured representations.
Given an augmented view u, the student and teacher ViT encoders produce patch-level features:
F, Elpatehh)
F, = (v) E R
Here S S2 x S,, denotes the total number of image patches, where S, = h/p and S,, "/p correspond to the number of patches along the height and width dimensions, respectively, given patch stride of p. represents the embedding dimension of the ViT backbone.
A learnable matrix CO E N is used to initialise N concept queries. each intended to capture distinct, high-level visual component. These concept tokens interact with the student patch features F, via lightweight clustering module C, composed of L Transformer decoder blocks. Within each block, cross-atee attentionn mechanisms enable the concept tokens attend and integrate information from the patch-level features. The cross- atteenattnnion update at the E-th layer is given by:
Z ++1 MHA(Q = Zz, K V F.) Zz.
Ł= 0.0,..,
Z CO
where MHA denotes multi-hean attention. This iterative process produces the final set of concept representations Z z E NxD Each decoder block also includes LayerNorm and feed-forward sublayers, though these components are omitted from the equation for brevity.

---

11.00
k v
C
Lept]
D
f
Transtomme Decoder Block
X
Brondcast Decoder
L cpt
m
ccpt]
C
00
Lrcc
Figure 2: Overview of Phase 2. Given patch-level features from masked student view, set of learnable concept tokens iteratively interact with the patch grid via a lightweight clustering module to produce coherent region representations. The training guided by two complementary objectives: masked concept distillation, enforced by the loss Cept, and feature- space reconstruction, guided by frec, both encouraging the emergence of meaningful and consistent region groupings.
Two design choices require clarification. First, stacking multiple cross-attention layers, L, allows each token iteratively refine its region representation, querying patches based on increasingly informed prototypes. This results sharper, more coherent region boundaries. Second, although many works apply entropy-based regularisation (e.g., Sinkhorn) to prevent token collapse, we observed no such issue with the concept tokens, thanks to Phase spatially diverse features. We show in the ablations that adding Sinkhorn modestly reducing performance,
omit it for simplicity.
To ensure the concept tokens are both meaningful and useful, we introduce two auxiliary objectives: concept-level distillation and feature- space reconstruction.
Concept-level Distillation. The first objective builds directly on the asymmetric masking strategy employed in Phase 1 and leverages to guide concept-level learning. Given that the student network receives masked view of the image, while the teacher processes the full, unmasked version. As a result, the student is encouraged to infer the structure of missing regions based on limited input, while aligning with the teacher's holistic understanding. The concept token sets from the student and teacher, denoted as Ze,v and Ze respectively, are projected through projection head and compared using cross-entropy loss:
plppt/ (Zs,r)
p plpptl
(Zr,v) plop)((Z,.))) clepti
leptl softmax
[ept] a
softmax
(4)
Ts
Tt
where pl(pt) and P. p(spt)
denote the student and teacher projection heads for the concept tokens and c/ept] is running mean of concept features. The concept-level distillation loss is then defined as:
[cpt] log [cpt] a a 2 5.22
[ept] log [cpt]
Leps = -
(5)
Unlike the patch alignment objective in Phase this loss encourages each concept token to capture the global structure of its corresponding region, despite partial masking. As such, it serves high-level signal that promotes consistency and the emergence of coherent region representations.
Feature-space Reconstruction. The second objective ensures that each concept token captures sufficient information reconstruct the teacher's patch-level features. To achieve this, lightweight spatial broadcast decoder 'D adapted from 47]. Each concept token is first broadcasted over 2D spatial grid of size S. S,, and then augmented with learned positional embeddings. This results in tensor of shape S₂ x S,, x D, which is processed by a shared CNN (with weights shared across all concepts) produce per-concept outputs of shape S, S,, x (D+ 1).
Each por-con output split into two components: (i) a reconstructed patch feature map f e S x5, x and (ii) spatial confidence map 7" E S.S,, where n {1,... N) indexes the n-th concept. To integrate the reconstructions from all concepts, we perform a softmax over the N confidence maps 7 ....,,?? V at each spatial location. This yields normalised attention weights n for each concept:
T
m" = softmax
E IR S.S,
for n I,...,N,
(6)
5

---

Marl
Mr
My
gep
1
PA teps)
P tops]
Mudt
Mry
Ma
D
E
0
Teacher Forward
May
(n Mall
M
3 Ma
E7
2/1
V1
0/2
(a)
(b)
(c)
Figure 3: Phase Overview. Given reference image 2ref, the model generates N attention masks Mye, each capturing a learned concept (left). These masks are geometrically projected onto two augmented views 01 U2, producing Ma, and Ms, (middle). Patch tokens are then aggregated per concept using the projected masks, yielding N concept vectors. These are passed through projection head and aligned via cross-entropy loss enforce concept-level consistency across views (right). Masks overlaid on images for visualisation only.
where the softmax computed independently each spatial location across the N concepts, and is a temperature parameter controlling the sharpness of the distribution.
The final reconstructed feature map is obtained by computing weighted sum of the per-concept reconstructions: F > m" @ f xyyx) where @ denotes element-wiss multiplication
broadcasted over the feature dimension D. Finally, the reconstruction loss is defined as:
free
Fr.
This objective encourages the concept tokens to distribute their representational capacity over different spatial regions, ensuring non-redundant and semantically rich decomposition of the scene.
The overall loss for Phase 2 combines both objectives: Lphase2 = ax + ag with weights 03, a4 0. These terms are optimised jointly with Phase 1.
3.3 Phase 3: View-Invariant Alignment of Discovered Concepts
Once humans form abstract representations of concepts scene, they are able recognise these same entities under different lighting conditions, perspectives, spatial configurations. This capacity, often referred as view-invariant recognition, allows us perceive an object the same despite visual variations that arise naturally in everyday environments [48 49].
Following the discovery of visual concepts in Phase ability by maintaining consistent concept representations across diverse augmented views. While Phase 2 clusters spatial features into fixed number of tokens, there no guarantee that the same concept appears in every augmented view, nor that it will be positioned or ordered consistently. Therefore, trying to match across views, especially under strong augmentations, can be unreliable. To address this, we design a guided alignment mechanism that uses a stable reference image as an anchor for transferring concept regions multiple views via geometric transformations.
aim equip our model with similar
Given an image r, we generate a lightly augmented reference view with no cropping, Iret- The teacher processes Fref produce myef, which is then thresholded to obtain binary masks M (0, 1] S. where each mask indicates the spatial support of a concept.
Using crop metadata, we project the Mat into the coordinate grids of U2, obtaining M.,, M (0, 1 SxN then perform masked feature aggregation to obtain pooled concept features:
Σ., Misja
FiJ s.ve
E [1,2]
(8)
Mida + Ens

---

These per-concept vectors are projected via a shared head d plape] and compared across views using cross-entropy loss:
Cgpp
Σ
alli
log a!n!
+ a log a
nEV
where V includes concept indices visible in both views. To ensure stability, we delay applying Egep until concept tokens from Phase 2 have sufficiently converged. Full derivations, including mask transfer, visibility conditions, and alignment schedules, are provided the Appendix.
The overall loss in Phase 3 combines all preceding objectives: LeG-ssI. Ca + 4r2Cpatch azLept + arbraa + Crp.. For simplicity, we set a; for I,...,5.
ADE-20K
Pascal- VOC
Cityscapes
Model
K-NN
Linear
h-NN
Lincar
K--N
Lincar
ViT-Small
SiT
14.9
20.6
35.8
45,4
MC-SSL
18.2
22.2
54.5
61.4
Dino
15.5
18.5
36.3
41.6
iBot
19.3
23.6
48.1
56.4
CG-SSL
25.8
28.9
60.8
66.7
34.2
39.1
CAPI (LVD-1422M)
20.6
25.8
50.5
59.4
31.3
36.5
VIT
Base
MAE
17.3
23.1
46.8
55.6
Data2Vec 2.0
16.8
21.2
47.4
48.9
MC-SSL
23.3
27.2
60.8
65.1
iBot
22.9
27.3
57.1
65.4
CG-SSL
28.3
31.5
62.7
69.0
35.2
40.0
CAPI (LVD-142M)
28.3
33.6
61.8
70.3
36.9
41.9
ViT-Large
MAE
21.3
27.4
53.7
61.5
32.8
38.5
Data2 Voc 2.0
24.2
27.6
57.5
58.1
32.8
38.2
iBot
26.0
30.7
60.2
68.8
35.7
39.8
CAPI
29.2
34.4
60.7
69.7
35.6
41.7
CG-SSL
30.5
35.6
64.0
70.6
36.3
45,5
DINOv2 (LVD-142M)
34.0
39.0
63.0
72.8
42.0
46.8
CAPI (LVD-142M)
32.1
37.2
63.8
72.7
38.9
44.3
Table 1: Comparison with state-of-the-ard methods semantie segmentation using frozen features.
We report k-NN and linear probe performance. CG-SSL consistently outperforms prior approaches, highlighting the quality of its learned representations.
4 Experiments
We evaluate the effectiveness of CG-SSI. framework through series of experiments. We begin by outlining our experimental setup, including datasets, architectures, and implementation details. Next, we report quantitative and qualitative results on tasks involving whole-image understanding and dense prediction. Finally, conduct ablation studies to validate key components our method.
4.1 Experimental Setup
Our main models are pretrained on ImageNet- - 1K without labels. For abla- tion studies, we pretrain on a combination of dense datasets, namely PASCAL VOC [50), Visual Genome [51] and MS-COCO (52), which together provide approximately 170K diverse images.
Pretraining Dataset:
Implementation Details: We use ViT backbone with patch size 16 x 16, following standard ViT-S/16.
ViT-B/16, and ViT-L/16. We use N 4 concept tokens and the clustering module C consists of
L= 4 transformer decoder blocks. The output of C, along with the encoder's [CLS] and [patch] tokens, are passed through a shared projection head comprising two linear layers with 2048 units and GELU activations, followed by 356-dimensional bottleneck. The output is 12-normalised and projected into an 8192-dimensional embedding space. All models are trained using the AdamW optimiser with cosine learning rate schedule and an effective batch size of 256 distributed across 8 GPUs. We train ViT-S for 800 epochs, ViT-B for 500 epochs, and ViT-I. for 300 epochs. Further training and ablation details are included in the Appendix.
7

---

VIT-S/66
VIT-816
V/T
VIT
CG.SSL
iBor
KINN
74.7
75.2
Lunear
77.7
773
KNN
76.8
77.1
Linear
7540
79.5
iBor
Pascal
89.4
VGenomee
30.3
COCO
57.4
Pascal
90.1
VGennoe
30.4
COCO
58.2
CAPRLVD-142M)
71.5
79.6
(a) Multi-class Linear Classification
(b) Multi-label Linear Classification
Table 2: Image classification results employing ViT-S/16 and ViT-B/16 backbones.
4.2 Results
Dense Image Understanding. A core motivation behind CG-SSL is to produce semantically rich and spatially localised representations via patch-level features. To evaluate the quality of these representations, we conduct experiments on dense prediction tasks, specifically semantic segmentation. We follow the k-NN and linear probe evaluation protocols from (53) and report mloU on three standard benchmarks: ADE20K [54]. PASCAL VOC [55], and Cityscapes [56], refer to Table
Notably, CG-SSL achieves high K-NN performance across all datasets, often matching or even exceeding the performance of strong baselines under linear probing. This is particularly impressive, as k-NN relies purely on raw feature similarity without any additional training, making it a direct measure of the intrinsic quality of the learned representations. For example, Pascal VOC, CG-SSL outperforms prior SOTA methods pre-trained on 100x more data, i.c. CAPI, by a wide margin of over 10 mloU in the k-NN setting for the ViT-S backhone. This substantial boost under non-parametric evaluation setup demonstrates that CG-SSL learns highly discriminative and well- structured representations suitable for spatial reasoning and region- based understanding.
Image Classification. We evaluate CG-SSL using a standard linear probing setup on multi-class and multi-label classification tasks. As shown in Table while CG-SSL shows slightly lower performance than iBOT on ImageNet (the pretraining dataset), it consistently outperforms iBOT on other datasets such Pascal, COCO, and Visual Genome. This is expected representation and dense localisation, which may slightly reduce image-level discrimination on seen data but yields stronger generalisation unseen domains.
method balances both global
Qualitative Comparison Dense Feature Representations via PCA. We adopt the qualitative feature analysis proposed in CAPI 533 applying PCA to the dense output features. As shown in Figure we visualise the first three principal components as an RGB composite and compare CG- SSL against SOTA vision models employing ViT-L backbone, except for I-JEPA, which uses VIT-H. Among all methods, CG-SSL produces some of the most discriminative and spatially coherent feature maps. The visualisations reveal clear emergence of object boundaries with minimal noise in uniform regions. Compared to CAPI and DINOv2, CG-SSL features appear less noisy. Moreover, in contrast to masked image modeling (MIM) methods, CG-SSL focuses more effectively on semantically meaningful regions the image. More visualisations are in Appendix.
Image
MAE
I-Jepa
D2V2
DI-
CAPI
CG-SSL
NOv2+r
Figure 4: Comparison of feature visualisations produced by CG-SSL. VIT-L/16 and SOTA methods on 560-pixel resolution images. We apply PCA to the dense feature maps. The first column displays composite RGB image formed from the first three principal components.
As further qualitative assesment, we visualise the attention maps produced by our model on randomly selected images outside the ImageNet-IRe IK dataset (Figure 5). Each map highlights distinct concept,
8

---

showing the model's ability to localise semantically meaningful regions, even under occlusion. For instance, the second image, the model accurately identifies the visible part of the cow despite being partially obscured by person.
Figure 5: Attention maps for learned concepts using ViT-L/16 pretrained on ImageNet-IK.
(a) Correspondence between two augmented views of same image.
(a) Correspondence between two different images from class. Figure 6: Top 12 patch correspondences extracted by CG-SSL with ViT-L/16.
Sparse Correspondence. We tackle a sparse correspondence task in which patches from two images belonging to the same semantic class are expected to matched. To assess performance, we visualise the top 12 correspondences with the highest self-attention scores, obtained from ViT-L/16 model pre-trained on ImageNet-1K using our CG-SSL framework. The image pairs are sampled from the ImageNet validation set.
Figure 6 presents representative examples of such correspondences. In Figure pp. CG-SSL exhibits near-pen perfect correspondence matching between two augmented views of the same image, accurately aligning nearly all patch pairs.
Moreover, as depicted in Figure Gp. CG-SSI. effectively establishes meaningful correspondences across two different images of the same class, despite substantial differences in texture, color, pose,

---

and background context. These results underscore the robustness and generalisation capabilities of the learned representations, demonstrating their suitability for tne-grained, patch-level retrieval tasks.
ADE
Pascal
ADE
Frozen
Pascal
ADE
Pascal
ADE
Pascal
DINO
15.6
35.7
Kmecnss
23.1
54.8
23.8
55.3
23.9
55.9
CG-S8L
iBot
CG SSL
Stage 1.2]
21,4
18.8
53.4
47.2
Ours
FFizzen
22.9
Learnable
54.6
24,0
24.1
24.3
56.1
55.9
55.7
2
4
24.0
24.3
56.1
56.0
Stage 1,2,3]
56.1
SlotAtin
19.8
52.2
23.8
55.5
8
23.8
55.9
Ours
24.3
56.1
(a)
(b)
(c)
(d)
K
ADE
Pascal
0.1
23.8
55.9
0.3
24.1
56.0
ADE
Pascal
Shared
ADE
Pascal
patch
ADE
Puscal
Pixels
23.8
53.6
24.3
56.1
24.3
56.1
0.5
24.3
56.1
Features
24.3
$6.1
X
23.4
49.8
X
24.1
56.0
0.8
23.7
55.1
23,3
55.0
(f)
(g)
(h)
(e)
Table 3: Ablation study of the design choices in our framework. We report linear performance on image segmentation tasks. We highlight the default setting in gray, and bold the best-performing solution. An in-depth analysis of these results is provided in the Appendix.
4.3 Ablation Studies
We conduct a comprehensive set of ablations show the impact of key design choices in CG- SSL, including (a) incremental component build-up across training phases, (b) learnable vs. frozen clustering modules, (c) varying the number of concept tokens, (d) decoder depth in the clustering module, (c) reconstruction temperature, (f) choice of reconstruction target, (g) projection head sharing between [CLS] and concept tokens, and (h) retaining patch-level loss in later phases. Results are reported in Tables Bh. For detailed analysis and additional ablations, see Appendix.
5 Discussion and Concluding Remarks
We introduced CG-SSL, a structured self-supervised framework that discovers, organises, and aligns semantic concepts across views without requiring manual supervision pretrained models. Through three-phase curriculum, global representation learning, concept discovery, and view- invariant aligmment, CG-SSL bridges the gap between global discriminative features and spatially coherent representations. Our experiments shows that CG-SSL achicves strong performance on both classification and segmentation tasks, outperforming several baselines and even rivaling models trained on substantially larger datasets. Notably, the use of concept tokens enables interpretability and spatial localisation that are difficult achieve with conventional invariance- based methods.
Limitations and Future Work. While CG-SSL provides structured and interpretable represen- tations, the number of concept tokens fixed across images, which may be suboptimal for scenes with varying complexity. Looking forward, we aim to explore adaptive mechanisms for determining the number of concept tokens per image and to extend CG-SSL to video, enabling temporal concept tracking. Importantly, this work was developed by a group with modest computational resources. As was the case with DINOv2, which achieved significant performance gains by scaling and optimising iBOT, we believe CG-SSL has similar potential to benefit from scaling and extensive hyperparameter tuning. We hope the community can build upon this foundation and explore its full capacity.
Acknowledgment
This research was supported by the UKRI AHRC CoSTAR National Lab for Creative Industries R&D as part of the Creative AI Futures
10

---

References
[1] Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14502-14511, 2022.
[2] Srinivasa Rao Nandam, Sara Atito, Zhenhua Feng, Josef Kittler, and Muhammad Awais. Enhanced weakly supervised few-shot classification & segmentation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[3] Srinivasa Rao Nandam, Sara Atito, Zhenhua Feng. Josef Kittler, and Muhammad Awais. Text augmented correlation transformer for few-shot classification & segmentation. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25357-25366, 2025.
[4] Sara Atito, Syed Muhammad Anwar, Muhammad Awais, and Josef Kittler. Sb-ssl: Slice-based self-supervised transformers for knee abnormality classification from mri. In Workshop on Medicul Image Learning with Limited and Noisy Data, pages 86-95. Springer, 2022.
[5] Rayan Krishman, Pranav Rajpurkar, and Eric Topol. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering, 6(12):1 1411133622 1352, 2022.
[6] Umar Marikkar, Sara Atito, Muhammad Awais, and Adam Mahdi. Lt-vit: A vision transformer for multi-label chest x-ray classification. In 2023 IEEE International Conference on Image Processing (ICIP), pages 3565-2569. IEFE, 2023.
[7] Yi Wang, Conrad M Albrecht, Nassim Ait Ali Braham, Lichao Mou, and Xiao Xiang Zhu. supe-supe learning in remote sensing: A review. IEEE Geoscience and Remote Sensing Magazine, 10(4):213-247, 2022.
[8] Xuewen Yang. Xing Zhang, Nan Wang, Guoling Xin, and Wenjie Hu. Underwater self- supervised depth estimation. Neurocomputing, 51 514:362-374 -00022--3,34000 2022.
(9] Sara Atito, Muhammed Awais, Srinivasa Nandam, and Josef Kittler. Gmml is all you need. In 2023 IEEE International Conference on Image Processing (ICIP), pages 2125-2129. IEEE, 2023.
[10] Sara Atito, Muhammad Awais, and Josef Kittler. SiT: Self-supervised vision transformer. arXiv preprint arXiv:2 104.03602, 2021.
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[12] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2022.
[13] Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In European conference on computer vision, pages 108-124, Springer, 2022.
[14] Yann LeCun. A path towards autonomous machine intelligence. 2022.
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural 2020.
information processing systems,
[16] Sara Atito, Muhammad Awais, Ammarah Farooq, Zhenhua Feng, and Josef Kittler. Mc-ssl0, Towards multi-concept self-supervised learning. arXiv preprint arXiv:2111.15344 2021.
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.

---

[18] Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[19] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[20] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
[21] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yo-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[22] Robert Desimone, John Duncan, et al. Neural mechanisms of selective visual attention. Anmual review of neuroscience, 18(1):193-22) 1995.
[23] James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3):415 434, 2012.
[24] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised vi- sual representation learning with semantic grouping. Advances in neural information processing systems, 35:16423- 16438, 2022.
[25] Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, and XIAOJUAN QL An image is worth k slots: Data-efficient scaling of self- visual pre-training openreview.ner, 2025.
[26] Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Yean-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7000-7009, 2023.
[27] Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar, Pean-Philippe Thiran, and Tinne Tuytelaars. Cribo: Self-supervised learning via cross-image object-level bootstrapping. arXiv preprint arXiv:2310.07855 2023.
[28) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-parametric instance discrimination. In Proceedings of the IÉEE conference on computer vision and pattern recognition, pages 3733-37423 2018
[29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.0077334 2018
[30] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CV international conference on computer vision, pages 9640-9649, 2021.
[31] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
[32] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European conference computer vision, pages 456-473. Springer, 2022.
[33] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 3024-3033, 2021.
12

---

[34] Jiantao Wu, Shentong Mo. Sara Atito, Zhenhua Feng, Josef Kittler, Syed Sameed Husain, and Muhammad Awais. Masked momentum contrastive learning for semantic understanding by observation. In 2024 IEEE International Conference on Image Processing (ICIP), pages 263-269,, IEEE, 2024.
[35] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient sulf-supe vision transformers for representation learning, arXiv preprint arXiv:2106.09785, 2021.
[36] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. Advances in Neural Information Processing Systems, 35:8799-8810, 2022.
[37] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16794-16804 2021.
[38] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint vrXiv:2203.084// 2022.
[39] Haiyang Huang. Zhi Chen, and Cynthia Rudin. Segdiscover: Visual concept discovery via unsupervised semantic segmentation. arXiv preprint arXiv:2204. /0926, 2022.
[40] David Navon. Forest before trees: The precedence of global features in visual perception. Cognitive psychology. 93))3533-33,, 1977.
[41] Aude Oliva and Antonio Torralba. Building the gist of scene: The role of global image features recognition. Progress in bruin research, 155:23-36, 2006.
[42] George L Malcolm, Iris IA Groen, and Chris Baker. Making sense of real-world scenes. Trends in cognitive sciences, 20(1 (1I1)::3 1):843-866, 2016.
[43] Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychology, 12(1))97-136, 1980.
[44] Max Wertheimer. Untersuchungen zur lehre von der gestalt ii Psychologische Forschung. 4(1:3001-350, 1923.
[45] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances neural information processing systems, 33:1 1125-- 2020.
[46] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, al. Bridging the gap to real-world object-centric learning, arXiv preprint arXiv:2209. 14860, 2022.
[47] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint
2019.
[48] James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-434, 2012.
[49] Nikos Logothetis, Jonas Pauls, and Tomaso Poggio. Shape representation in the inferior temporal cortex of monkeys. Current Biology, 5(5):552-563, 1995.
[50] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-11) 2015.
[51] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1)332-7,, 2017.
13

---

[52] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[53] Timothée Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Cluster and predict latents patches for improved masked image modeling. arXiv preprint arXiv:2502.08769 2025.
[54] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[55] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338 2010.
[56] Marius Cordts, Mohamed Omran, Sehastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper aims improve self self-supervised learning (SSL) by addressing the limitations of current SSL methods, introducing concept-guided approach that aligns visual concepts across views enhance representation quality.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included a limitation section at the end of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
No theoretical assumptions made.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All implementation and training details are provided in Section 4.1 Addition- ally, code and pretrained models will be made publicly available to facilitate reproducibility and ease of use.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code and pretrained models will made publicly available to facilitate reproducibility and ease of use. Additionally, code will be submitted with the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 http:///wwaualcl.uaaa///////aaaa Ctte///ooeeeteieeeeeeeeeee
wwwiissssiiniinnnnooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We use standard benchmarks with pre-defined train/val/test splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high computational cost of training our full models, we report results from single run. While we acknowledge the importance of reporting variance, our primary findings are based on consistent evaluation protocols and strong performance margins across benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are mentioned in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: the NeurlPS Code of Ethics are respected.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While our work focuses on improving visual representation learning from natural imageswith potential benefits for downstream applications such as medical imaging, environmental monitoring, and accessibility, we also acknowledge possible negative societal impacts. These include misuse in surveillance or biased decision-makin if applied without fairness considerations. Although a detailed analysis is beyond the scope of this paper, we bricfly discuss these issues in the supplementary material.
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our code builds upon publicly available repositories licensed under CC-BY 4.0. All such sources are clearly credited within the codebase wherever they are used, in accordance with the license terms.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.papersstt wwppppsipiaaia thcode com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM was only used for enhancing the writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httt:p///ceerriooccccc///oerrrer for what should or should not be described.
21

---

Technical Appendices and Supplementary Material
A Additional Details For Phase 3: View-Invariant Alignment
Once the model learns to decompose an image into coherent concepts (Phase 2), the next step is to ensure that these concepts are consistently recognised across different augmented views. Human vision naturally achieves this through view-invariant recognition, maintaining stable representations of entities under transformations such as cropping, flipping, or color variation 488 49
However, directly matching concept tokens across views is unreliable due to their permutation invariant and view-specific nature. To address this, design guided alignment mechanism that uses a stable reference image as an anchor for transferring concept regions to multiple views via geometric transformations.
For each input image x E HRW we generate:
A reference view Eee E R hx via light augmentations (no cropping), fed unmasked to the teacher.
Two strongly augmented views v1, 02 T(x) k each associated with crop parame- ters:
((() = (=)
(K) w
f Crap
(K)
flip(")) h {1,2}
(10)
The teacher processes Teef to produce concept attention masks Mee {0, 1 where See Stef. X Stef,y is the number of patches in the reference view and N is the number of concept tokens.
Mask Transfer via Geometry. Let (i,j) denote a patch coordinate in the reference grid. We define its normalized center as:
i + 0.5
j+ 0.5
(11)
S
Seetyy
This patch is visible in view Up if:
2
20 (K)
+
(kk)
crop
(()) yo
v
< yo
<(k)
+ -k)
(12)
where the crop parameters are normalized by the original image size H W. For visible patches, compute the projected patch index (i', in view un:
11 - 20 <(k)
(k)
i' =
S Sune
j
8 S
(13)
x(k) Wenap
(k+
Herap
horizontal flipping is applied (flip 1), then 1' Sou My E {0, 50 for concept n is defined as:
1'. The transferred binary mask
M Mari if Eq. holds, otherwise
M..s'a .
(14)
0.
Masked Feature Aggregation. Let F Fa S. x denote the student's patch features for view vn- We compute aggregated concept features by masked average pooling over patches:
Mas'' oi En Ue 2.
(15)
6,89
Evs Ma. o +
Only concepts that are visible in both i1 and U2 (i.e., with non-zero denominator in both) are aligned.
22

---

Concept-Level Alignment Loss. Each aggregated concept feature s, is passed through shared projection head pisptl, and softmax-normalizer
plspt) (x
[t]
softmax
(16)
Ts
The group-level consistency loss is defined as cross-entropy across matching concepts:
Cgpp
Σ nEV
a
8,02 log
a!!! +
a!! log a!!!
(17)
where iss the set concept indices visible in both views. This loss enforces consistent representa- tions for matched concepts under geometric variation.
Training Schedule. To ensure stable alignment, we delay the application of Egpp until the model has learned coherent concept masks from Phase 2.
Ciotal ai Glaas] + azLipateh) + a3Cept + asCres + asCarp
(18)
B Experimental Setup
Implementation Details. We VIT backbone with patch size 16 16, following standard VIT- S/16, ViT-B/16, and ViT-L/16 configurations with embedding dimensions 384, 768, and 1024, respectively. Each encoder outputs sequence of patch tokens and global [CLS] token.
For the clustering module C, we use N concept tokens and consists L = transformer decoder blocks. The output of C, along with the encoder's [CLS] token and [patch] tokens, is passed through a shared projection head comprising two linear layers with 2048 units and GELU activations, followed by -i6-dimnnsionall bottleneck. The output is L2- 12-normalised and projected into an 8192-dimensional embedding space.
The broadcast decoder D. adapted from [7], reconstructs teacher patch features from the concept tokens. Each token is broadcasted to the patch grid size, augmented with learned positional encodings. and processed with a shared MLP to produce patch-wise reconstructions f" and unnormalised attention weights >"; the final reconstruction is computed using
weights with temperature 0.5. The decoder output dimension D matches the backbone's 's embedding size.
For the reference image in Phase 3, a denser set of patch tokens, enhancing the quality of concept representations. Since there is no backpropagation from the teacher model, the increased resolution introduces minimal computational overhead.
higher-resolution input of 448 448, resulting in
All models trained using the AdamW optimiser with cosine learning rate schedule. We largely follow the iBOT pre-training hyperparameters, with the exception of increasing the masking ratio from 30% 40%. We trained the models with effective batch size of 256 distributed across 8 GPUs on VIT-S for 800 epochs, ViT-B for 600 epochs, and ViT-L for 400 epochs.
For VIT-S/16, Phase 1 is applied for the first 50% of the training epochs, followed by Stages 1+2 for the next 10%, and the full pipeline (Stages 1+2+3) for the final 40%. These phase proportions are based on empirical intuition and have not been exhaustively tuned. For VIT-B/16 and ViT-L/16. we initialise from iBOT-pret prettaind weights to avoid redundant computation and reduce energy consumption. To ensure fair comparison, we also continue training the original iBOT models under our schedule. However, we observe that their performance degrades with further training when using the original iBOT objectives.
Time and Memory Requirements. CG-SSL takes approximately minutes per epoch to pre-train a ViT- model using 8 GPUs with an effective batch size of 256. Compared to iBOT, our method takes about 6.5 minutes longer per epoch and uses roughly 4.6 GB more GPU memory under the same training setup. While CG-SSL is approximately epoch per hour slower, the added compute
23

---

is a reasonable trade-off for the improved spatial consistency, interpretability, and robustness of the learned representations.
C Visualisations
Qualitative Comparison of Dense Feature Representations via PCA. We adopt the qualitative feature analysis methodology proposed in CAPI [53]. applying PCA to the dense output features, E, e.laatchl (z), extracted across all images, As shown in Figure 7 the second column visualises the first three principal components as an RGB composite, while the subsequent six columns show each of the first six components individually.
Each component highlights distinct semantic regions, revealing that CG-SSL encodes meaningful visual concepts. Notably, the components differentiate object parts from the background, indicating the model's ability to disentangle structured elements of the scene. This visualisation underscores the spatially localised and interpretable nature of CG-SSL's learned representations.
Pattern Layout of CLS vs Concept Tokens. For this experiment, we used the MSCOCO dataset, known for its visually rich and dense scenes. We extracted CLS token features and concept token features using a ViT-S/16 model pre-trained on ImageNet-IK. The clustering results, based on MSCOCO images, are shown Figure S and Figure 9 respectively. By examining the top clusters, we observed several notable patterns.
CLS token features tend capture the global context or overall "scene gist" of an image. This is evident in Figure 8 where images are grouped based on their broader visual setting. For instance, airplanes in the sky are clustered together, while those on the ground form a separate group. These clusters reflect how CLS tokens prioritise high-level scene understanding.
In contrast, concept token features focus more localised and object-specific information. For visualisation clarity, we overlay the attention maps of each concept token on the images, highlighted in red. As shown in Figure unlike CLS tokens, background elements are less influential these features, leading to tighter focus on object-centric regions. For example, airplanes are clustered irrespective of whether they are flying or on the ground, and elephants are grouped across varying backgrounds. Even when only partial object visibility is present, concept tokens effectively capture the object identity.
particularly interesting observation the emergence of fine-grained clusters. In the last row of Figure 9 we distinet cluster composed entirely of hands from different images, activities, and viewpoints. This suggests that concept tokens are not only able to localise objects but also capture consistent semantic parts across varied visual contexts.
24

---

Figure 7: Visualisation of the features produced by CG-SSL ViT-L/16 applied to images at 560 pixel resolution, Images are randomely selected from validation set of ImageNet- 1K dataset.
25

---

DON'T TOP
STOP
STOP
BELIEN
STOP
STOP
STOP
STOP
Figure 8: Visualisation for pattern layout of [CLS] token.
26

---

STOP
STOP
GORE
FOR
TRANT
ONE WAY
S
STOP ME
STOP Afl
STOP
Repert
Lindley AV
STOP
STOP
TOP
2.
STOP
TOP
STO
NO THRU TRAFFIC CARE
STRIMNC
EL--WAY
Figure 9: Visualisation for pattern layout of concept tokens.
27

---

D Detailed Ablation Analysis
We conduct comprehensive ablation studies to systematically analyse the influence of various design choices on our model's performance. Given resource constraints, we pretrain our model using selected combination of dense datasets, namely PASCAL VOC, MS-COCO, and Visual Genome, which contains approximately 170K diverse samples. Our training employs the ViT-S/16 architecture for a total of 500 epochs. Although utilising 10% of ImageNet for ablation studies might seem more conventional choice, we deliberately opt for these datasets due to their inherent complexity. characterised by crowded scenes and multiple objects, rather than the dominant, centrally located subjects typically found in ImageNet images. This selection enables to rigorously evaluate our model's 's capability to extract meaningful representations from complex and cluttered visual scenes, an important aspect often underexplored current self-supervised learning research. To ensure fair comparison, we pretrain our baseline method, iBot, under identical conditions using the same dataset combination.
Our initial goal was to conduct the entire study exclusively on these challenging datasets. However, given the community's emphasis on ImageNet benchmarks, include Net-eNet-based comparisons to provide a clearer context for our findings. While our approach already outperforms ffae-of-thhe-a methods, we believe that conducting ablation studies directly on ImageNet could further enhance our results.
All numbers mean intersection-over-uiiie mloU ADE20K / backbone with a frozen linear probe, as in Table 3 of the main paper.
Pascal- VOC using the ViT-S/16
D.1 Curriculum Build-up Across the Three Phases
As shown in Table 4 while DINO and iBOT offer strong global representations, they fall short on dense prediction tasks, reaching only 15.6/35.7 and 18.8 8848/7 mloU on ADE20K and Pascal, respectively. Adding Phase 2 improves the structure of learned features by grouping spatial regions into object-like concepts, which raises performance to 21.4/53.4 Phase 3 further aligns these concepts across views, enforcing spatial consistency and yielding the final boost to 24.3/56.1. This progression shows the importance of both discovering region-level structure and ensuring it transfers across viewpoints.
Model / Phases
ADE
Pascal
DINO baseline
15.6
35.7
iBOT baseline
18.8
47.2
CG-SSL (Stage 1 + 2)
21.4
53.4
CG-SSL (1 3)
24.3
56.1
Table 4: Effect of progressively adding the three curriculum phases,
D.2 Clustering Strategy for Concept Tokens
We explored several alternatives the learnable clustering module in Phase 2 as shown in Table S Surprisingly, even a simple k-means clustering on the reference image, followed by geometric alignment, achieved 1/54.8 on ADE20K/Pascal. This highlights the strength of Phase 3. However, this setup adds nontrivial computational overhead due to the per-image clustering step, making it less scalable.
When we used our clustering module but froze it after initial training, performance dropped slightly (22.9/54.6), indicating that allowing the grouping mechanism to remain adaptive is important for maintaining accuracy.
Slot Attention, despite being learnable, resulted in a marked drop (19.8/52.2). Investigation revealed that tokens collapsed to the same spatial location across images, even with Sinkhorn normalisation, leading to low-quality groupings. Nonetheless, this setup still outperformed iBOT (18.5 (1888/4.22.. suggesting that cross-view matching Phase provides consistent benefits, even when the reference regions are poorly formed.
28

---

Clustering module
ADE
Pascal
K-means (Frozen)
23.1
54.8
CG-SSL (Frozen clustering Module)
22.9
54.6
Slot-Attention (learnable)
19.8
52.2
CG-SSL (ours)
24.3
56.1
Table Impact of alternative cluster assignment mechanisms.
D.3 Number of Concept Tokens N
We evaluated the model's robustness to different numbers of concept tokens N as shown in Table 6 Performance remains stable across a range of values, with peak N In the main ImageNet experiments, set N 4 as the dataset is less complex.
ADE
Pascal
2
23.8
55.3
4
24.1
55.9
6
24.3
56.1
8
24.0
55.7
12
23.8
55.5
Table Sensitivity the number of concept tokens.
D.4 Depth L of the Clustering Module
We evaluate the impact of the number of transformer decoder blocks (L) in the clustering module. While using L = yields the best performance, overall results are relatively stable across different number of blocks.
L
ADE
Pascal
23.9
55.9
2
24.0
56.0
24.3
56.1
8
23.8
55.9
Table 7: Effect of clustering module depth.
D.5 Sharing the Projection Head
We investigate whether the projection head should be shared between the [CLS] token and the concept tokens. Empirically, sharing improves performance, suggesting that unified embedding space may benefit alignment. However, we believe that this design choice requires deeper analysis. For instance, could partial sharing. e.g., sharing early layers while keeping the output branches independent, offer a better trade-off? Should the concept tokens use sower-dimensional output, given that their role is capture only concepts, while the [CLS] token must also encode broader context and inter-region relations?
D.6 Temperature x in Softmax Fusion
We found that a moderate temperature (x = 0.5) yields the best balance between sharpness and stability.
29

---

Shared
ADE
Pascal
24.3
56.1
X
23.4
49.8
Table 8: Effect of sharing projection head between [CLS] and concept- tokens.
ADE
Pascal
0.1
23.8
55.9
0.3
24.1
56.0
0.5
24.3
56.1
0.8
23.7
55.1
1.0
23.3
55.0
Table Influence of the temperature parameter in mask fusion.
D.7
Reconstruction Target
We compare reconstructing raw pixels vs. teacher features. As expected, the latter performs better, which is highlighting the importance of semantically guided reconstruction over low-level signal recovery.
Target
ADE
Pascal
Pixels (RGB)
23.8
55.6
Teacher features
24.3
56.1
Table 10: Pixel-space vs. feature-space reconstruction.
D.8 Retaining the Patch-Level Loss [patch)]
We ablate whether retaining Phase I's patch-level loss in later stages beneficial. Removing results in modest drop in accuracy, indicating that fine- fine-grained supervision from patch features complements concept-level grouping and alignment.
Separately, we believe that a more systematic investigation into the weighting of individual loss components Q1.....5 would be valuable, may uncover better trade-offs across the different training objectives.
D.9 Backbone Scaling
Finally, we verify the robustness of CG-SSL across different backbones, including ViT-S, VIT-B, and VIT-L. Performance scales consistently, with ViT-L achieving the strongest results across both datasets.
30

---

Ppttch
ADE
Pascal
24.3
56.1
24.1
56.0
Table 11: Effect of retaining the Patch-Level Loss [patch]]
ADE
Pascal
VIT-S/16
24.3
56.1
ViT-B/16
26.1
58.6
ViT-L/16
28.4
59.0
Table 12: Backbone Scaling.
31


Reviews:
[{"review_id": "c6nyRySIJr", "reviewer": "Reviewer_YAiC", "summary": "This paper considers self-supervised learning. Its approach is inspired by human perception: humans first grasp the global structure of a scene, then organize the visual input into coherent groups that they are then able to track in changing conditions.\n\nSimilarly, the proposed self-supervised learning method is split into three phases. The first phase captures glabal image features with the existing iBot approach which aligns cls and patch level embeddings produced ba a student-teacher pair of networks for two augmented views. cls token features are matched across the two views, while the patch token features are matched within the same view. \n\nIn the second phase, the student and the teacher network process the same augmented image. A trainable matrix C is initialized to encode N concepts. These concepts interact with student/teacher features with a cross attention mechanism. Token representations in C serve as queries, while transformer block outputs serve as keys and values. This allows each token to iteratively refine its representation of concepts within an image. Student and teacher networks are further aligned with a cross-entropy loss between its concept sets. An additional objective is used to ensure that teacher's patch-level features may be reconstructed and concepts capture all regions of a given image. This is done by broadcasting each token representation over a 2D spatial grid and calculating a corresponding confidence map . The confidence indicates the level of alignment between the concept and image regions. Broadcasted features are multiplied with confidences and fed to a reconstruction loss.\n\nIn the final phase, binary masks for concepts within a reference image are extracted with a teacher network. Lightly augmented images are created and crop metadata is used to obtain appropriate concepts masks for novel views . Concepts are aligned across vies with a cross-entropy loss.\n\nExperiments are don on ImageNet-1k, Pascal VOC, Visual Genome and MS-Coco.", "questions": "1) Would an increase in the number of tokens lead to more low-grained scene understanding?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The topic of self-supervised learning is important\n2) Good experimental results demonstrate coherent representation and good performance in classification and dense prediction, as well as robustness to occlusions.\n3) Authors provide the code with the submission\n\nWeaknesses:\n1) The paper is difficult to follow and poorly written. The method could be explained more concisely. E.g., the sentence in lines 175-176 does not really make sense on its own or connect to the previous and following sentences. It is not clear if there is one or two concept matrices C in the second phase.\n2) The results could also be better presented and contextualized. It is not clear in the tables which was the training data used in each of the experiments. Why does the method not outperform some of the exisitng self-supervised approaches etc.\n3) Additional ablations could be useful, especially those that explore phase scheduling. How long is each phase performed individually? How are the losses balanced? Is the training sensitive to loss weighting?", "was_revised": true, "final_justification": "I have read the other reviews and authors' rebuttal. I find that my concerns were sufficiently addressed and therefore raise my score by 1.", "rebuttal": "> The paper is difficult to follow and poorly written. \n\nThank you for your feedback. We appreciate your effort in understanding the full scope of our method despite the density of the presentation. Your summary shows a strong grasp of the key ideas, and we are grateful for the time you took to engage deeply with the paper.\n\nWe recognise that certain parts of the manuscript could have been communicated more clearly. In trying to provide both the technical rigor and the motivation behind each design choice, some explanations may have become complex. Based on your suggestion, we would like to offer a simplified summary of the method that highlights the core intuition:\n\nExisting self-supervised methods like iBOT and DINOv2 are limited by two key assumptions:\n1. treating each patch token as an independent concept and aligning them directly, and \n2. relying on one-to-one patch matching between masked and unmasked views. \n \nOur method addresses these as follows:\n- Stage 2 [*To address limitation 1*] introduces concept tokens that group related patches into coherent regions and align their features, moving beyond token-level reasoning.\n\n- Stage 3 [*To address limitation 2*] aligns these discovered concepts across different views using geometric consistency, instead of relying on token-to-token matches from the same view.\n\nWhile addressing these limitations is important, our primary motivation is to propose a more principled and structured approach to self-supervised learning, one that is inspired by how humans perceive and organise visual input. This is why we emphasised the motivations for each phase alongside the technical details.\n\nWe will revise the final version of the paper to improve clarity and narrative flow, simplifying the exposition while preserving the core contributions. Thanks again for the feedback.\n\n$~$\n \n> The method could be explained more concisely. E.g., the sentence in lines 175-176 does not really make sense on its own or connect to the previous and following sentences. \n\nThank you for pointing this out. You are absolutely right. Line 176 contains a grammatical error, it is a sentence fragment rather than a complete sentence. \n\nHere is the intended version of the passage, with improved clarity and structure:\n*Given that the student network receives a masked view of the image, while the teacher processes the full, unmasked version, the student is encouraged to infer the structure of missing regions based on limited input, while aligning with the teacher's holistic understanding.*\n\nWe will revise this section accordingly and also go through the entire manuscript to ensure it is free from grammatical errors and improves overall readability.\n\n$~$\n \n> It is not clear if there is one or two concept matrices C in the second phase.\n\nThere is only one learnable clustering module $\\mathcal{C}$, used by the student. The teacher uses a copy of this module, updated via exponential moving average, as is standard in student-teacher frameworks. \n\n$~$\n \n> The results could also be better presented and contextualized. It is not clear in the tables which was the training data used in each of the experiments. \n\nAs mentioned in the Experimental Setup section (lines 236–238), our main results are based on models pre-trained on ImageNet-1K (1.2 million images). Unless otherwise noted, all reported comparisons are under this setting. The only exceptions are methods explicitly tagged with \"LVD-142M\", which indicates that they were trained on a significantly larger (142 Million images), curated, in house dataset from Meta.\n\nWe will revise the captions and text to make this distinction more obvious and ensure that readers can easily interpret the results in the proper context.\n\nAdditionally, while space limitations led us to move many qualitative analyses and supporting details to the supplementary material, we agree with the reviewer that certain clarifications should have been included in the main text. In the final version, we will ensure the main text clearly conveys this context, with the supplement used only for additional support.\n\n \n$~$\n \n> Why does the method not outperform some of the exisitng self-supervised approaches etc. \n\n\nThank you for the question. We would like to clarify that in a fair comparison, i.e., when all methods are pre-trained on the same dataset (ImageNet-1K), our method outperforms prior work on dense tasks, as shown in Table 1. These tasks benefit from spatially structured and fine-grained representations, which CG-SSL is explicitly designed to learn.\n\nWe also included results for DINOv2 and CAPI, both of which are trained on LVD-142M, a dataset that is over 100 times larger than ImageNet-1K. Despite this massive scale difference, CG-SSL achieves comparable or even better performance in certain settings, which we believe demonstrates the strength and scalability of our approach.\n\nIn Table 2, our method also surpasses iBOT on multi-label classification, a setting that requires richer, more localised representations, another strength of our framework. On single-label classification (e.g., standard ImageNet evaluation), our performance is on par with SOTA models pre-trained on ImageNet-1K.\n\nWe believe this pattern is expected, datasets like ImageNet or LVD are structured such that a single dominant object often defines the label. Models optimised for global invariance naturally excel in these settings. However, when evaluated on tasks that require dense understanding or multi-object reasoning, such models tend to underperform, whereas CG-SSL shows significant advantages, as evidenced by Table 1.\n\n$~$\n \n> Additional ablations could be useful, especially those that explore phase scheduling. How long is each phase performed individually? How are the losses balanced? Is the training sensitive to loss weighting?\n\n>> How long is each phase performed individually?\n\nWe provide the phase scheduling details in Appendix B (Experimental Setup -> Implementation Details). That said, the schedule itself is based on empirical intuition rather than extensive tuning. We agree that further exploration of scheduling strategies could be valuable.\n\n>> How are the losses balanced? Is the training sensitive to loss weighting?\n\nAs noted in line 229 of the main manuscript, we set all loss weights ($\\alpha_1$ - $\\alpha_5$) to 1 for simplicity. We recognise that this uniform weighting is not necessarily optimal, different downstream tasks may benefit from emphasising certain objectives. For example, increasing $\\alpha_1$ would put more focus on global representations, which may improve image-level classification performance.\n\nWe believe there is substantial room for improvement through careful tuning of loss weights and other hyperparameters. However, our goal in this work is to introduce and validate a principled learning framework, not to perform exhaustive tuning. Even without fine-grained hyperparameter optimisation, CG-SSL shows strong performance across both global and dense prediction tasks, highlighting the robustness and generality of the method.\n\nThat said, we absolutely agree with the reviewer that the model could benefit from further tuning, especially around phase scheduling and loss balancing. Due to limited computational resources, we focused our ablation studies on validating the core components of our method, and were unable to explore the full range of parameter choices. This can be a promising direction for future work.\n\n\n\n$~$\n \n> Would an increase in the number of tokens lead to more low-grained scene understanding?\n\nWe addressed this in our ablation study (Table 3d), where we varied the number of concept tokens. This experiment was conducted on complex, dense datasets (Pascal, COCO, Visual Genome), which differ significantly from ImageNet in structure and scene complexity.\n\nWe observed that 6 concept tokens gave the best performance on these datasets. For ImageNet pretraining, we opted for 4 tokens, as scenes in ImageNet are typically simpler and dominated by a single object.\n\nInterestingly, on the dense datasets, increasing the number of concept tokens beyond 6 led to a slight drop in performance. *Our hypothesis* is that too many tokens cause the model to over-segment the scene, breaking coherent regions into smaller, fragmented parts. This may dilute semantic meaning, reduce alignment consistency across views, and ultimately weaken training signals. Visualisations of the concept maps seem to support this interpretation, as they show more scattered regions when the token count is too high. So while increasing token count may intuitively suggest finer-grained understanding, in practice it introduces trade-offs that can hurt performance when not matched to scene complexity.\n\nWe believe this trade-off between granularity and semantic coherence is an important direction for future exploration, potentially including adaptive token allocation based on scene complexity.\n\n$~$\n \n--------------------------------\n\n$~$\n \nWe sincerely thank the reviewer for the thoughtful and insightful questions, which not only helped clarify key aspects of our work but also highlighted valuable directions for future research.\n\nIf there are any remaining questions or points needing clarification, we would be happy to elaborate further. Otherwise, we hope our responses have addressed your concerns and we would kindly appreciate your consideration in reflecting this in your overall score."}, {"review_id": "BbOwBuuI8c", "reviewer": "Reviewer_VvJo", "summary": "This paper presents a new self-supervised approach (CG-SSL) to learning visual representations that aims to more fine-grained representations by learning to group patches into regions and compare these across views. The proposed approach consists of 3 training phases.\n\nIn the first phase, they aim to learn global representations by using an earlier technique called iBot that learns by comparing the patch level embeddings from a teacher network that observes the full image and a student network that takes in a masked version.\n\nIn the second phase, they learn to group regions by training a cross-attention transformer that uses learned concept tokens to attent to image features and extract N regions. This phases uses 2 training objectives: 1) concept level distillation uses the iBot loss but between each concept token separately (again comparing student and teacher embeddings) 2) feature reconstruction loss uses a MONet, SIMONe style multi-object reconstruction setup where they predict patch features and a mask for each spatial location from each concept. These masks and patch features predictions are then combined to reconstruct the full image level patch features and compared to the true patch features from the teacher.\n\nIn the final third phase, they aim to improve learned representations by aligning concept representations across two views of the same image. To do that, they get a mask for each concept by thresholding the output of teacher network's mask logits. Then transform these masks using the known transformation between views. These masks are then used to pool the patches from each view to form concept representations for each view, which are then compared across views using the usual iBot style loss.\n\nOn evaluations, CG-SSL leads to some improvement most on segmentation while performing on-par or slightly better on classification.", "questions": "- Are the phases optimised jointly? Line 205 says phase 1 and 2 are optimized together. Line 226 says that phase 3 loss is added only after convergence of phase 1 and 2.\n- Concept level distillation loss in eqn. 4 is applied on each concept separately, right? Might be good to make this explicit.\n- For feature reconstruction loss, why not use the attention weights from each concept to patches for masks? Did the authors try this? Does this lead to worse results?\n- For evals, why not use attentive probing as well? This could be useful to compare to previous work.\n- It'd be nice to see the learned masks. Currently, only attention maps are plotted in Figure 5. How about a similar figure for masks?\n- What is CAPI + LVD 142M. This is probably explained in the appendix but would be useful to mention in main text.\n- line 175: \"Given that\" should be removed?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Overall well written and easy to follow\n- Although the individual components are not entirely novel, combines various ideas from the literature in a novel way\n- Good performance improvement on segmentation compared to baseline\n- Good ablation study\n\nWeaknesses:\n\n- Needs a more thorough evaluation to make the case for the proposed approach stronger\n  - There are various papers in the literature that explore similar ideas around splitting the image into multiple regions\n  - It would be good to compare to at least some of these (like DORA, Venkataramanan et al.,\"Is ImageNet worth 1 Video? Learning Strong Image Encoders from 1 Long Unlabelled Video,\")", "was_revised": true, "final_justification": "The authors have addressed some of my concerns during the rebuttal so I raise my score to Accept.", "rebuttal": "> Needs a more thorough evaluation to make the case for the proposed approach stronger\nThere are various papers in the literature that explore similar ideas around splitting the image into multiple regions\nIt would be good to compare to at least some of these (like DORA, Venkataramanan et al.,\"Is ImageNet worth 1 Video? Learning Strong Image Encoders from 1 Long Unlabelled Video,\")\n\n\nThank you for this comment. We appreciate the pointer to related work and agree that a broader contextualisation would strengthen the paper.\n\nTo clarify, DORA derives concepts from multi-head attention maps, which are inherently shaped by the underlying learning objective. Since DORA is built on DINO, the resulting attention maps tend to focus heavily on the dominant object in the image, as also illustrated in Figure 3 of the DINO [1] paper, where heads attend to overlapping parts of the main dominant concept.\n\nIn contrast, CG-SSL introduces explicit concept tokens that interact with the image through cross-attention. These tokens are refined to represent coherent regions across the entire image, not just the dominant object. Moreover, our method is designed to explicitly align these structured representations across views, going beyond patch-level matching or implicit clustering.\n\nThat said, we fully recognise the conceptual connection in moving beyond flat patch-wise learning. We will include a more detailed discussion of related methods like DORA in the related work section to better position CG-SSL within the landscape of structured SSL approaches.\n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" ICCV 2021.\n\n$~$\n\n> Are the phases optimised jointly? Line 205 says phase 1 and 2 are optimized together. Line 226 says that phase 3 loss is added only after convergence of phase 1 and 2.\n\nAs clarified in Appendix B (Experimental Setup -> Implementation Details), the training follows a curriculum learning approach:\n\n- The model is initially trained with Stage 1 objectives to learn global representations of the image.\n- Once a basic understanding of image-level structure is achieved, we add Stage 2 objectives to introduce concept-level reasoning through clustering.\n- Finally, after the model has learned to group patches into coherent regions, we introduce Stage 3, which aligns these learned concepts across different augmented views.\n\nThe motivation for this staged approach is to progressively build structure into the learning process. If we introduce Stage 3 too early, i.e. before concept tokens have learned to represent coherent regions, then the alignment across views would be matching corresponding spatial regions, not semantically meaningful concepts. This would not fully leverage the goal of Stage 3, which is to align high-level, meaningful representations across views.  \n\nTo improve clarity, we will explicitly include this training schedule and motivation in the main paper, not just the appendix. Thank you for pointing this out.\n\n$~$\n\n> Concept level distillation loss in eqn. 4 is applied on each concept separately, right? Might be good to make this explicit.\n\nYes, that is correct, the loss is applied independently to each concept token. Thank you for pointing this out, we will make this detail explicit in the revised manuscript.\n\n$~$\n\n> For feature reconstruction loss, why not use the attention weights from each concept to patches for masks? Did the authors try this? Does this lead to worse results?\n\nWe did not explore this idea. Leveraging attention maps could indeed be beneficial and may help improve convergence. We truly appreciate the suggestion and believe it is certainly worth investigating in future work. Thank you for bringing it to our attention.\n\n$~$\n\n> For evals, why not use attentive probing as well? This could be useful to compare to previous work.\n\nWe did the bare minimum in terms of evaluation, just using standard protocols and plugging in our weights. That said, we agree that attentive probing could be useful, especially since our method focuses on improving patch-level representations. It is definitely worth trying.\n \n$~$\n\n> It'd be nice to see the learned masks. Currently, only attention maps are plotted in Figure 5. How about a similar figure for masks?\n\nAs they are very close to each others, we chose to show only one to avoid redundancy. However, this ties nicely with your earlier point of using attention maps from clustering module directly to guide the reconstruction in the decoder.\n\n$~$\n\n> What is CAPI + LVD 142M. This is probably explained in the appendix but would be useful to mention in main text.\n\nCAPI (LVD-142M) refers to the CAPI model pre-trained on LVD, the 142 million images in-house dataset curated by Meta proposed in DINOv2 paper. We will clarify this in the main paper to avoid confusion.\n\n$~$\n\n> line 175: \"Given that\" should be removed?\n\nYes, you are absolutely right. This was also noted by Reviewer YAiC. The sentence will be corrected in the final version for clarity.\n\n$~$\n\n-----------------\n\n$~$\n\nWe sincerely appreciate the reviewer's in-depth analysis and thoughtful suggestions, which reflect a strong understanding of our framework. Your comments opened up several promising directions for refinement and future exploration. We will incorporate as many of your suggestions as possible in the revised version and follow up on others in future work. If our responses have addressed your concerns, we kindly ask you to consider raising your score. Thank you again for your valuable feedback."}, {"review_id": "vUQLq1WhGx", "reviewer": "Reviewer_pNLd", "summary": "The authors propose a new self-supervised learning (SSL) method for visual pretraining. Unlike existing methods utilizing image-level or patch-level consistency given some well-design augmentations, they utilize concept-level or object-level consistency. It seems to be inspired both from Object-Centric Learning and self-supervised segmentation, with some adaptations.", "questions": "**Q1**\nThe attention defined between Line 162 and 163: Why do the authors not choose the competitive cross attention, i.e., Slot Attention, from object-centric learning? What are the possible effects, both in performance and explanability like in Figure 5?\n\n**Q2**\nWhat are the details training costs of your method and of some representative baselines? Like training time, number of iterations, memory consumption.", "limitations": "No. The authors did not provide discussions on their limitations.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Well-motivated\n- Clear presentation\n- Novel design\n\n**Weaknesses**\n- The training details can be very important to such SSL, but the authors did not provide much of them. Should include a section in the appendix to expose your \"secrete recipe\".\n- Plain (but quite okay) analysis in the concepts learned: more in-depth analysis can give readers more impactful insights.\n- Lacks intuitive comparisons between your method and the baselines.\n- Figure/Table captions should be clearer: Like in Figure 2, it is better to tell readers meanings of all the notations; In Table 3, should briefly clarify what are the meanings for each subtable, instead of telling readers to jump to the appendix.\n- The numbers of some equations are missing, like the one between Line 162 and 163.", "was_revised": true, "final_justification": "Thank the authors for their efforts. I would keep my rating.", "rebuttal": "> The training details can be very important to such SSL, but the authors did not provide much of them. Should include a section in the appendix to expose your \"secrete recipe\".\n\nThank you for this comment. We do provide an expanded version of the experimental setup in the supplementary material, but we agree that it could be more detailed. That said, there is not really a \"secret recipe\" behind our results, our method relies on a relatively straightforward implementation without heavy engineering.\n\nWe fully agree that further ablations and hyperparameter tuning (e.g. loss weighting, phase durations, learning rates) would be valuable for understanding and possibly improving performance. Due to limited computational resources, we were unable to explore these in depth. Nonetheless, we believe our results already show strong promise, and we are confident that with better tuning, the performance of CG-SSL could be improved even further.\n\n$~$\n\n> Plain (but quite okay) analysis in the concepts learned: more in-depth analysis can give readers more impactful insights.\n\nWe included a range of qualitative visualisations in Appendix D beside the one in the main paper, such as:\n\n- Sparse correspondences across views (Figure 7a) and between different images of the same class (Figure 7b), highlighting strong sensitivity to fine image details. For instance, Figure 7b shows that even in cluttered scenes, the model aligns fine-grained objects like cucumbers with high precision.\n\n- In Figure 9, we show that CG-SSL can discover consistent concepts regardless of context or background, which is often not the case in global representation that favor dominant objects and association of it with the background.\n\n- Figure 9 also shows that the concept tokens learns consistent representation of small, non-dominant concepts accross images (e.g. hands), a level of granularity rarely observed in global representation.\n\nThese results collectively support that CG-SSL is learning robust and semantically meaningful local representations. That said, we agree that additional probing and quantitative concept evaluation could offer further insights, and we plan to explore this in future work.\n\n$~$\n\n> Figure/Table captions should be clearer: Like in Figure 2, it is better to tell readers meanings of all the notations; In Table 3, should briefly clarify what are the meanings for each subtable, instead of telling readers to jump to the appendix.\n\nYou are absolutely right. Due to space constraints, we had to offload some details to the appendix, but this has impacted clarity. We will revise the figure and table captions in the final version to make them more self-contained and readable without needing to jump back and forth.\n\n$~$\n\n> The numbers of some equations are missing, like the one between Line 162 and 163.\n\nThank you for catching that. We will double-check the numbering and ensure that all equations are properly referenced and numbered in the final version.\n\n$~$\n \n> The attention defined between Line 162 and 163: Why do the authors not choose the competitive cross attention, i.e., Slot Attention, from object-centric learning? What are the possible effects, both in performance and explanability like in Figure 5?\n\nWe appreciate this insightful question. We did try Slot Attention, and included results in the ablation study. However, we found that the method quickly collapsed during end-to-end training, failing to assign distinct concepts. This is a known issue with Slot Attention as reported here [1]. We did not invest deeply in tuning it further, but agree that with proper regularisation or scheduling, it might not collapse and can be a valid alternative. \n\n[1] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\" arXiv preprint arXiv:2209.14860 (2022).\n\n$~$\n\n> What are the details training costs of your method and of some representative baselines? Like training time, number of iterations, memory consumption.\n\nOur method is built on top of iBOT, and naturally adds some computational overhead due to the concept token module and reconstruction decoder, we reported the exact overheads in Appendix B (B\nExperimental Setup -> Time and Memory Requirements) as follows:\n\n*Time and Memory Requirements. CG-SSL takes approximately 24 minutes per epoch to pre-train\na ViT-B/16 model using 8 GPUs with an effective batch size of 256. Compared to iBOT, our method\ntakes about 6.5 minutes longer per epoch and uses roughly 4.6 GB more GPU memory under the\nsame training setup. While CG-SSL is approximately one epoch per hour slower, the added compute\nis a reasonable trade-off for the improved spatial consistency, interpretability, and robustness of the\nlearned representations.*\n\nHowever, to have a fair comparison with iBoT, we continued the training of iBot to match same number of epochs as CG-SSL. Interestingly, the performance slightly decreased instead of enhancing with more training.\n\n $~$\n\n> Limitations: No. The authors did not provide discussions on their limitations.\n\nWe just want to point that we have provided the limitations under section 5 Discussion and Concluding Remarks.\n\n$~$\n\n----------------------------\n\n$~$\n\nWe sincerely thank the reviewer for their thoughtful and detailed feedback. Your questions and suggestions have been both insightful and actionable. We will incorporate your points to strengthen the clarity, depth, and completeness of the final version. If you feel your concerns have been addressed, we kindly ask you to consider raising your score. Thank you again for your valuable input."}, {"review_id": "HABiKbORC8", "reviewer": "Reviewer_eBBN", "summary": "The paper proposes \"Concept Guided Self-Supervised Learning\", which is a technique for training a vision model without labels to produce generally useful representations, both for summary tasks (e.g. classification, retrieval), and dense tasks (e.g. semantic segmentation). Similar in spirit to techniques such as iBot and DINOv2, CG-SSL places additional constraints on the model, namely that the patch tokens more coherently represent semantically related regions in the image. DINOv2, being the preeminent SSL method, doesn't explicitly enforce that tokens that should have semantically similar representations, actually do. CG-SSL enforces this coherence through cross attention from a fixed number of queries to the dense features, and matching those to unmasked momentum self-features, and then also reconstruction via a weighted sum of the spatially broadcast concept queries. Additionally, they employ a stage 3 that encourages views of the same objects/regions to be spatially equivariant by aligning and then matching in a stage 3 training curriculum.", "questions": "The DINOv2 [1] paper claims the following ADE20k linear probe mIoUs:\n- ViT-S/14: 44.3\n- ViT-B/14: 47.3\n- ViT-L/14: 47.7\n\nhowever, in this paper, it is claimed that DINOv2 achieves the following:\n- ViT-S/14: *\n- ViT-B/14: *\n- ViT-L/14: 39.0\n\nCan you please explain this discrepancy?\n\n\nIn E.3, the following is stated: \"In the main ImageNet experiments, we set N = 4 as the dataset is less complex.\" How does the number of concept tokens affect the generalizability of the method, given that one dataset may have far fewer concepts than another?\n\n[1] https://arxiv.org/pdf/2304.07193", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThe proposed method appears to produce cleaner feature maps compared to contemporary methods.\n\nConcept regions to appear to map to coherent regions.\n\n### Weaknesses\n\nAblations are performed against DINO and iBot, however, DINOv2 is considered a much stronger baseline. How does CG-SSL's classifier ability compare to DINOv2?\n\nCG-SSL is strongly better than CAPI for ViT-S, however, for ViT-B and larger, CAPI is arguably stronger.\n\nD2V2 is only presented in acronym form, without citation. I might have guessed this was some shorthand for DINOv2, however, both D2V2 and DINOv2 are present in Figure 1.", "was_revised": true, "final_justification": "Considering the rebuttal to my review, as well as additional information in the reviews and rebuttals of the other reviewers, I am updating my recommendation from 3->4.", "rebuttal": "> Ablations are performed against DINO and iBot, however, DINOv2 is considered a much stronger baseline. How does CG-SSL's classifier ability compare to DINOv2?\n\nIt's crucial to note upfront that DINOv2 is trained on LVD-142M, a large-scale, in-house, curated dataset that is over 100 times larger than ImageNet-1K, which we use for pretraining. Due to this vast difference in training data and computational scale, a direct comparison is not entirely fair.\n\nThat said, we still include DINOv2 in our results to give context. Despite the scale gap, CG-SSL achieves competitive performance in several settings, highlighting the strength and efficiency of our method. In some dense prediction tasks, CG-SSL even approaches or outperforms DINOv2, without access to massive data or aggressive tuning.\n\nIt's also worth mentioning that while DINOv2 performs exceptionally well, its core learning mechanism remains similar to iBOT, as acknowledged in its own paper and stated in the DINOv2 abstract *Most of the technical contributions aim at accelerating and stabilizing the training at scale*. Much of its performance gain is attributed to engineering improvements, scale, and data curation, rather than a shift in learning principles.\n\nOur focus in CG-SSL is to push forward the conceptual design of self-supervised learning by introducing structure, region-level reasoning, and explicit view consistency rather than engineering advantages or scaling tricks. From an academic research standpoint, advancing conceptual understanding often precedes optimisation. While optimisation is undoubtedly valuable, especially for deployment and benchmarking, we believe foundational contributions should not be overshadowed by resource-intensive scaling.\n \n$~$\n\n> CG-SSL is strongly better than CAPI for ViT-S, however, for ViT-B and larger, CAPI is arguably stronger.\n\nOur main comparisons with CAPI are conducted on ImageNet-1K, where CG-SSL consistently outperforms CAPI across all relevant configurations. It's particularly noteworthy that our method surpasses CAPI on ViT-S while using 100 times less data. We view this as **a strong indicator** of CG-SSL's generalisation capabilities and efficiency rather than a point of critithisim. \n\nFor ViT-B and ViT-L backbones, our experiments were constrained by limited computational resources. We were unable to perform any hyperparameter tuning or run multiple trials, which would be ideal for maximising performance at scale. That said, the current results already show CG-SSL to be highly competitive, and we are confident that with modest tuning, it has the potential to match or exceed CAPI even while pre-trained on 100 times more data.\n\nWhile this is an aspirational claim, we believe the existing evidence supports the promise of CG-SSL as a scalable and conceptually robust framework.\n\n$~$\n\n> D2V2 is only presented in acronym form, without citation. I might have guessed this was some shorthand for DINOv2, however, both D2V2 and DINOv2 are present in Figure 1.\n\nApologies for the confusion. D2V2 refers to Data2Vec2, not DINOv2. We will add this and the proper citation and clarification to avoid any ambiguity.\n\n$~$\n\n> The DINOv2 [1] paper claims the following ADE20k linear probe mIoUs:\nViT-S/14: 44.3\nViT-B/14: 47.3\nViT-L/14: 47.7\nhowever, in this paper, it is claimed that DINOv2 achieves the following:\nViT-S/14: *\nViT-B/14: *\nViT-L/14: 39.0\nCan you please explain this discrepancy?\n\nThe numbers we report for DINOv2 are taken from the CAPI paper, which uses a different evaluation setup. Specifically, as noted in Section 7.4 of the DINOv2 paper, DINOv2's ADE20k results are based on linear probes trained at $512\\times512$ resolution, whereas the CAPI paper use the standard $224 \\times 224$ resolution.\n\nFor fairness and consistency, we follow the CAPI evaluation protocol across all benchmarks.  \n\n$~$\n\n> In E.3, the following is stated: \"In the main ImageNet experiments, we set N = 4 as the dataset is less complex.\" How does the number of concept tokens affect the generalizability of the method, given that one dataset may have far fewer concepts than another?\n\nWe appreciate this insightful question. Indeed, the number of concept tokens N could significantly affect model generalisation, especially across datasets of varying complexity. Our choice of  N=4 for ImageNet was driven by intuition and limited GPU resources. We agree that this may not be optimal and that further investigation could reveal more effective configurations.\n\nWe view this as an important direction for future work. A deeper analysis of the relationship between dataset complexity and optimal token count could lead to more adaptive or data-driven strategies, further enhancing CG-SSL's performance across domains.\n\n$~$\n\n------------------\n\n$~$\n\nWe thank the reviewer for the insightful feedback. Your comments helped surface important distinctions. We hope our clarifications have addressed your concerns. If everything is now clear, we would sincerely appreciate your consideration in updating your score to reflect these points."}]

Author rebuttal:

Meta-review: This paper presents a new self-supervised approach to learning visual representations with more fine-grained representations by learning to group patches into regions and compare these across views. The proposed approach consists of 3 training phases. On evaluations, the approach leads to improvements on segmentation and classification tasks. The reviewers are positive and recommend accepting the paper after rebuttal, in which most of the concerns were properly addressed. The AC agrees with the decision to accept the paper.
Decision: Accept (poster)