Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Authors: Thalles Silva
Helio Pedrini
Adín Ramírez Rivera
Venue: ICML 2025 poster (2025)
Keywords: self-supervised learning
computer vision
representation learning
non-parametric SSL
Primary area: deep_learning->selfsupervised_learning
TL;DR: A new SSL algorithm to learn visual features from unlabeled data based on a non-parametric pre-training strategy.

Abstract: We present Self-Organizing Visual Prototypes (SOP), a new training technique for unsupervised visual feature learning. Unlike existing prototypical self-supervised learning (SSL) methods that rely on a single prototype to encode all relevant features of a hidden cluster in the data, we propose the SOP strategy. In this strategy, a prototype is represented by many semantically similar representations, or support embeddings (SEs), each containing a complementary set of features that together better characterize their region in space and maximize training performance. We reaffirm the feasibility of non-parametric SSL by introducing novel non-parametric adaptations of two loss functions that implement the SOP strategy. Notably, we introduce the SOP Masked Image Modeling (SOP-MIM) task, where masked representations are reconstructed from the perspective of multiple non-parametric local SEs. We comprehensively evaluate the representations learned using the SOP strategy on a range of benchmarks, including retrieval, linear evaluation, fine-tuning, and object detection. Our pre-trained encoders achieve state-of-the-art performance on many retrieval benchmarks and demonstrate increasing performance gains with more complex encoders.

Self-Organizing
Visual Prototypes for Non-Parametric Representation Learning
Thalles Silva
Helio Pedrini
Adín Ramírez Rivera?
Abstract
80
We present Orga-Orggg Visual Prototypes (SOP), a new training technique for unsupervised visual feature learning. Unlike existing prototypi- cal self-supervised learning (SSL) methods that rely on single prototype to encode all relevant features of a hidden cluster in the data, we pro- pose the SOP strategy. In this strategy. a prototype is represented by many semantically similar rep- resentations, support embeddings (SEs), each containing complementary set of features that to- gether better characterize their region in space and maximize training performance. We reaffirm the feasibility of non-parametric SSL by introducing novel non- parametric adaptations of two loss func- tions that implement the SOP strategy. Notably, we introduce the SOP Masked Image Modeling (SOP-MIM) task, where masked representations reconstructed from the perspective of multiple pa-aaremetii local SEs. We comprehensively evaluate the representations learned using the SOP strategy on range of benchmarks, including retrieval, linear evaluation, fine-tuning, and ob- ject detection. Our pre-trained encoders achieve state-of-the-ar performance
SOP
ImageNet k-NN Top-1
iBOT
78
MaSSL
DINO
76
74
25
50
100
200
400
Number of parameters (Millions)
Figure I. k-NN top-1 accuracy on ImageNet.
of pre-defined prototypes that are presumed to represent hidden clusters in the data. When presented with multiple views as input, these methods assume that prototypes must produce consistent predictions across views of the same image during pre-training. To avoid ill-posed solutions, methods enforce an equipartition constraint (Asano et al., 2019) that uniformly assigns a batch of images to prototypes using techniques such as centering (Caron et al., 2020) and BunkhornKKnop (Cuturi, 2013). these regularizers are removed, pre-training collapses to trivial solutions.
many retrieval benchmarks and demonstrate increasing perfor- mance gains with more complex encoders. Code: https://githee com/ stwallessassspp...
1. Introduction
Despite undeniable success, this framework places exces- sive importance on the prototypes' representations, which need to encode comprehensive set of relevant features for consistent and accurate predictions across views. Current methods (Caron et al., 2020; Zhou et al., 2022; Caron et al., 2021; Oquab et al., 2023) report their best performance when the number of prototypes > the number of actual classes C' in the data. However, addi- tion to increased computational and memory requirements, argue that over-clustering is suboptimal approach to fully exploring the feature space under the weak supervision of SSL pre-training. Specifically, a large K translates into fewer images per prototype, facilitating the optimization problem by biasing simpler features, which may limit the complexity the learned features.
State-of-le the-art art self-super learning (SSL) methods (Zhou et al., 2022; Oquab et al., 2023) for computer vi- sion rely on prototype learning with a multiview training strategy using embeddinb architectures (JEA). These methods have a similar framework: they learn a large set
much larger than
Computing, University of Campinas, Campinas-SP, Brazil *Department of Informatics, Univer- sity of Oslo, Oslo, Norway. Correspondence to: Thalles Silva <thaales silva @ students.ic.uu www.ci...nnnpbbrr.. br>, Helio Pedrini <helio@ ic www.uuiiicab....mmm Adín Ramírez Rivera <adinr @ no..
'Institute
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Self-Organizing Visual Prototypes for Non-Paramitric Representation Learning
We hypothesize that when a large number of prototypes is employed, they may fail to capture the essential features needed to fully describe the hidden clusters within the data.
We present Self-Organizing Prototypes, an alternative ap- proach prototypical SSL. We optimize views based on the soft similarity perspective of Self-Organizing Proto- types in the space of mon-parametric representations.
Each cluster can be conceptualized as a "region" of salient features in space. If a prototype underrepresents its under- lying region, the interactions with views may not be strong enough to pull representations towards it. Furthermore, over- clustering does not ensure comprehensive coverage of the feature space. Even with uniform distribution on the hyper- sphere (Wang& & Isola, 2020), gaps may still exist between prototypes, resulting in features that cannot be effectively attracted to any prototype due to insufficient supervision.
We propose the novel SOP-MIM pretext task to learn fine-grained features by reconstructing local-level masked representations based on non non-parametric SOPs composed of patch-level feature as local supports. We show that SOP-MIM improves downstream performance dense prediction tasks compared to existing solutions.
Our work reaffirms the feasibility of non-parametric SSL., where we avoid learning prototypes from random weights. We show that our method is stable, does not require ex- tra regularizers to avoid mode collapse, is extensible to many pretext tasks such as MIM, does not require over- clustering, cf. Table 10, and produces transferable repre- sentations. Moreover, we show that SOP's performance improves as the model architecture is scaled.
Motivated by these limitations, we deviate from the pop- ular prototypical learning paradigm and propose non- parametric approach that optimizes the feature space us- ing Self-Orgen Self-Organizing Prototypes (SOP). Our proposal orga- nizes the feature space by exploiting random local structures within the data. Intuitively, an SOP can be viewed as data structure, such as a graph, representing a local region in the feature space. Each SOP consists of a set of non con-ammmttticc support embeddings (SEs), which reside in close proximity and share semantic characteristics. Each support embedding predicts the degree of similarity (or likelihood) of view to its "region" in latent space (SOP), based on its distinct feature set. Then, SEs combine their individual votes to produce the final likelihood of a view to their SOP, Unlike prototypical SSL, SOPs are inherently stochastic. Instead of optimizing a set of prototypes that move in space during pre- pre-training. SOPs randomly select and optimize regions in the data.
2. Methodology
We argue that the common approach of learning prototypes as bottleneck feature vectors to represent salient regions in feature space is suboptimal due the lack of strong super- vision and the inductive biases that promote over-clustering in SSL. The lack of a strong supervisory signal promotes under-optimized regions in the latent space, which are typi- cally modeled as clusters represented by prototypes. Regu- lar prototypical SSL contrasts different views against these prototypes using point-to-point comparisons, resulting in unstable comparisons during the learning process. Instead, we propose optimizing views over local structures called Self- Seff-Organizing Prototypes (SOPs). We define an SOP as set consisting of an anchor prototype and its k closest neighbors, which we call support embeddings (SEs). Unlike prototypical SSL, SOPs leverage local structures within the data and optimize regions in the feature space (via SEs) rather than individual points, as sources of representative features. SOPs can be viewed as augmented prototypes. However, rather than relying on single prototype, SOPs group features from different SEs, which collectively repre- sent the region in space more accurately, cf. Fig. 2.
Unlike regular methods (Caron et al., 2020; Zhou et al., 2022), SOPs are dynamic (not constrained to region in space) and fully cover the space of features through built-in randomization. SOPs explore their surroundings to boot- strap SEs, which in turn augment the feature set used to represent their region in space. By comparison, SOPs can be viewed dynamic, feature-enriched versions of regular prototypes. While prototypical SSL optimizes relationships between views and prototypes (discrete points in space), our method optimizes views over SOPs, i.e., local "regions" in space containing multiple SEs that cover a larger area of the same region, offering better adaptation. We demonstrate that our dynamic SOP structure alleviates the over-clustering problem inherent in prototypical SSL (cf. Table 10). To en- sure that our learned representations contain information beyond the class level, we adapt the Masked Image Mod- cling (MIM) pretext task using the SOP strategy and intro- duce a novel pretext task where masked representations are trained to agree with corresponding non-masee masked ones from the perspective of multiple local-level supports.
Notation. Let X be an image dataset and x ~ X a uni- formly random observation. We denote by x" the w-th augmented version of æ, referred to as a view of æ, where the superscript indexes the views V. To create views, we use random transformation function such that x" t(x). For simplicity, we consider the case where V 2. How- ever, we explore multiple view scenarios in the main ex- periments. We denote by fa a Vision Transformer (ViT) (Dosovitskiy et al., 2020) encoder with parameters that receives a view and produces a matrix of representation vec- tors Z' = (xx) E LLxd where L and d are the number
Our contributions are threefold:
2

---

Self-Organizing Visual Prototypes for Non-Parametrie Representation Learning
xx
0
View
0.8
Memory
C1
sdos
[1.00....
View
0
0
Sup. Fimin
00.10...
[0.0.8...
Y
[0.0.4....
10.0.1....
[...0.800]
[..00.7.0]
0,001
Anchar
0.....0.1
0.7....7]
Q1
Sup. Eanh:
20
C4
Figure 2. First, we select a set of random anchors A {a:} (colored squares with patterns) from a set of representations kept in memory (gray sphere). Second, each anchor selects support embeddings (SEs) (colored diamonds) their nearest neighbors (2 in this illustration). Each anchor a; and their SEs form an SOP, representing a hidden structure within the data (shaded colored region). Note that given embodding may belong more than one SOP simultancously. D (K+++) 1x with labels Y E (k++))) representing the intercomnections between SEs and anchors. Intuitivoly, cach SOP contains set that estimate the degree similarity between views and SOPs. Then, SEs combine their votes to produce final score for each view, resulting in similarity distributions optimized be consistent across SOPs.
together, SOPs can be lincarly arranged a dataset
of patch tokens and feature dimensionality respectively, such that Z" = { contains patch representations where first element Zo E Idd is classification or [CLS] token embedding and the remaining ZI.L. elements are patch embeddings from image Instead of learning class- and paich-level discrete features (or prototypes)
The cluster assignment prediction task aims to learn em- beddings that covary w.r.. a set of learnable prototypes Rxdd The optimization follows
L (eLs1,0 0 =
Σ x
pICLS] (zt) log (P) (zā)), (I)
previous work did (Caron et al., 2020; Zhou et al., 2022), we define memory containers Eç E R Ne xd and Ep E R N xd to store [CLS] and patch embeddings from previous itera- tions. Intuitively, the memories act as subsets of the training data features for global and local representations. For details regarding practical experiments, cf. Appendix A.
where PICtS] (u) = al(u (T)), a is the softmax func- tion and ( is the cosine similarity. Basically, P |CLs)] (-) linear layer, parameterized by f, that maps the views' vec- tor embeddings z" into K pseudo-categories assigning each representation a soft distribution (prediction) that describe its membership probabilities to all prototypes.
Next, we introduce the basic functionality of current proto- typical SSL methods and highlight the two commonly used loss functions. Then, we present SOP, emphasize its main differences and advantages.
This objective can viewed from a pseudo cluutt-cluttrrigg per- spective, where each prototype represents a pseudo-class, In both parametric and non-parametric approaches, each prototype estimates the likelihood of view to itself. Ex- isting solutions typically employ a single prototype, either learnable (Caron et al., 2021) or non-parametric (Silva et al., 2024), to represent region in the representation space.
2.1. Learning Representations with Prototypical SSL
Current prototypical SSL methods (Oquab et al., 2023; Zhou et al., 2022) have a nearly identical framework composed of two pretext tasks: (i) cluster assignment prediction over class-level embeddings and (ii) token-level embedding re- construction or Masked Image Modeling (MIM). Usually, each task is learned with different set of trainable parame- ters.
2.2. Self-Organizing Prototypes
The under-representation problem in prototypical SSL oc- curs when a prototype lacks the necessary features to de- scribe its region in space. We propose to solve this problem by increasing the information redundancy of a prototype by
3

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
augmenting its feature set through Self-Organiznng Proto- types (SOP).
Now, we can compute the probability distributions for each view as PICIsi (u) = a((u, D² where Y are soft contributions that sum up to one and encode the contribu- tions of each SE. Compared to the parametric loss (1), our approach swaps the learnable prototypes 0 for non- parametric SOPs. Note that the matrix multiplication be- tween the probability distribution a(.) and soft contributions Y. represents the weighted combination of the SEs predic- tions within each SOP.
SOPs explore random local structures in the latent space containing different embeddings that share semantic char- acteristics about their region. These local structures are used to bootstrap support embeddings (SEs), leveraged as a source of representative features. This approach assumes that embeddings in a vicinity contain enough information to describe their region in the feature space. Ideally, each SE within an SOP contributes complementary features that may not be present or sufficiently emphasized in single prototype.
Finally, we minimize the non-parametric version of C ficiss.e e (I), as
LiCLs = Σ x X
P (CLS) (z0)² log (p (z²)) (2)
1.2.1. NON-PARAMETRIC SUPPORT EMBEDDING SELECTION
2.3. Self-Organizing Prototypes for Masked Image Modeling
An important consideration is how to bootstrap SEs while maintaining semantics. If SEs do not share semantic char- acteristics, their contributions may be noisy, potentially harming the learned features. Inspired by recent work on non-parametric SSL (Silva et al., 2024), we use a memory Ec to store representations from previously processed im- ages during training. From the memory Ec, we perform three main operations: store, sample, and search.
The MIM task has been extensively explored by Zhou et al. (2022) and Oquab et al. (2023) from the parametric perspective. The task aims produce consistent predic- tions between reconstructed patch embeddings and their corresponding uncorrupted representations w.r.. set of learnable discrete local-level features. The goal is to train an online local-level tokenizer & by randomly mask- ing a portion of the patch token representations æ = {aili_o using a binary mask m E {0, 1) such that a { : (1 m.)ar, + m.e (Masx] }4t is corrupted ver- sion of the input image æ, and exasx is a learnable token. The corrupted input è is fed to the encoder 2 = f() and reconstructed from the uncorrupted version following
To build an SOP, first, we sample a subset of anchor representations A = {a:) C Ec from memory. Second, we search for SEs through spherical k-Nearest Neighbors with anchors A as centroids such that D = arg m ((A, )) where the arg max- operator returns the anchors and the set of the top-k closest neighbors of each anchor. At this point, D can be viewed as a dataset con- taining K SOPs, each containing k + 1 members, i.e., anchor d; and k SEs. Note that this definition allows an SE ej E E to belong to more than one SOP, cf. Fig. 2.
L
Lpatch & = Σ
my myppa ppalch
(z+)² log
phucch (p(tcc(()))) (3)
where, similar to C LicLsI.0 (1), ppacch() (.) is a linear layer that computes the probability distributions w.r.t. learnable discrete features & by soft assigning the patch tokens to K distinct discretized representations. Note that the loss Spuihh (3) skips the [CLS] ] token øg. and optimizes differ- ent versions of the same image view, where one is masked.
Given the potential uncertainty in the unsupervised K-NN se- lection algorithm, it is reasonable to expect that SEs (within an SOP) contribute differently. Naturally, SEs that share more features with an anchor should have stronger influ- ence than those more distinct. Based on that observation, we introduce Y to model the contributions of SEs within their SOPs. Intuitively, we can think of an SOP as an acyclic directed graph (DAG) containing k + 1 nodes (an anchor plus SEs) and 1 edges. All edges point to the sink anchor node, and the strength of the edges represents the contribu- tions of SEs towards its SOP, cf. Fig. 2. A naive strategy would assign equal contribution to each SE, i..., Y ex- pressed as a one-hot vector. We show in Section 4 that such strategy is suboptimal, likely due to false positives from the k-NN selection. Instead, we propose y E Y, to model the soft contribution of an SE as the embedding similarity score with anchors, i.e., (e,, a.) for j E 1.... K(k+1 ([)+ and i {0, 1, ...... K}. This strategy minimizes potential noise arising from false positives from the SE selection algorithm.
We propose a new version of the MIM pretext task based on a non- non-parametric strategy. Instead of learning a set of discrete features (online tokenizer), we obtain the proba- bility distributions pretch by exploring relationships be- tween semantically similar patch embeddings in the space of -arammemtt representations using Self-Organizing Prototypes, cf. Section 2.2.
We start by randomly sampling K anchor patch discrete tokens À = { C Ep as the roots for our SOPs. Then, each anchor selects le nearest patch token representations to become SEs. In conjunction, anchors and SEs form patch- level SOPs, and D = arg max ((A, Ex/ )) is the dataset
4

---

Self-Organizing Visual Prototypes for Non-Parametrie Representation Learning
containing a of SOPs. Note that Ep be seen non-parametric offline tokenizer.
Table 1. K-NN and Linear probing, semi- and full fine-tuning on ImageNet-IM:
Similarly to Section 2.2, we obtain the patch-level probabil- ity distributions, in non-parametric form, as ppachh (v)=
METHOD
ARCH
EP.
k-NN
LIN.
1%
10%
100%
(( (u, Y, and optimize
ESVIT
SWIN-T/14
300
77.0
78.7
IBOT
SWIN-T/14
300
76.2
79.3
SOP
SWIN-T/14
300
77.2
79.4
DEIT
VIT-S/16
800
79.3
79.8
Cpach
my Ppacc (2t)² log (prach (2))) (4)
DINO
VIT-S/16
800
74.5
77.0
60.3
74.3
82.0
IROT
VIT-S/16
800
75.2
77.9
61.9
75.1
82.3
MASSL
VIT-S/16
800
75.1
77.8
where we remove the learnable discrete tokens & in favor of non-parametric embeddings Ep and introduce the SEs' soft contributions through Y.
SOP
VIT-S/16
800
75.3
77.9
62.1
75.1
82.3
DEIT
VIT-B/16
400
81.0
81.8
75.6
81.4
MoCo-v3
VIT-B/16
400
76.7
NNCLR
VIT-B/16
1000
76.5
The proposed SOP-MIM task (4) encourages the network to reconstruct the missing patches so that local-level SEs from multiple SOPs produce consistent predictions between reconstructed and original embeddings.
DINO
VIT-B/16
400
76.1
78.2
64.4
76.3
83.6
IBOT
VIT-B/16
400
77.1
79.5
68.5
78.1
84.0
MASSL
VIT-B/16
400
77.2
79.6
SOP
VIT-B/16
400
78.2
79.9
69.5
78.4
84.2
IBOT
VIT-L/16
250
78.0
81.0
84.8
I-JEPA
VIT-L/16
600
77.5
The final loss a convex combination the two losses, SSP X (cLa] + X2Cpachh By default, A X2 =1
VIT-H/14
300
79.3
SOP
VIT-I/I6
250
79.2
81.2
84.9
3. Main Experiments
Table 2. Object detection and instance segmentation on COCO and semantic segm ADE20K. ViT-B encoders.
We begin by assessing the quality of the pre-trained repre- sentations across a range of downstream tasks, adhering to the experimental protocol outlined by Zhou et al. (2022). Subsequently, we justify the choices our architecture by ablating the methods' main components. For more details, please ef. Appendix A.
METHOD
DET. AP"
ISEG. APM
SEG! MIOU
SEG MIOU
SUP.
49.8
43.2
35.4
46.6
BEIT
50.1
43.5
27.4
45.8
DINO
50.1
43.4
34.5
46.8
IBOT
51.2
44.2
38.3
50.0
3.1. Linear Evaluation on ImageNet
SOP
51.4
44.3
38.7
50.6
k-NN and Linear Probing. In Table 1, we evaluate the linear transferability of the representations learned with SOP using two protocols: (1) non- non-parametric k-NN and (2) linear models. For the k-NN estimator, we tested different values of & E 10, 20, 100, 200 and reported the best results. For linear probing, we used pre-trained encoders as feature extractors and trained a linear layer on top of the frozen features. On the k-NN benchmark, SOPs improve over existing methods across all choices of backbones. For ViT-L (307 million parameters), improves over iBOT by + +1.2% top-1 accuracy, reaching 79.2%, which is similar to I-JEPA's ViT-H (632 million parameters) linear top-1 accuracy of 79.3% (Assran et al., 2023). These results suggest strong performance of SOP's off-the-hhelf features. Additionally, we report performance values for the supervised baseline DeiT (Touvron al., 2021), as well as for the strong SwinT (Liu et al., 2021) baseline ESVIT (Li et al., 2022).
is similar to existing solutions, more complex backbones, such as ViT-B/L and SwinT, produce larger performance gains. These gains primarily shown in the k-NN evalua- tion, suggesting strong boost in the off-the-shed represen- tational power for retrieval tasks, cf. Section 3.5.
3.2. Semi-Supervised Fine- Fine-Tuning on ImageNet
In Table 1, we measure SOP's representation capacity to learn tasks using limited set of labeled examples. Follow- ing the unsupervised pre-train, supervised fine-tune proto- col, we report top-1 accuracy using 1 10% of ImageNet- IM labeled images. We observe that SOP's performance improves and surpasses competing methods as more com- plex encoders are used. We observed performance gap (+1.0%) between SOP and iBOT in smaller data regimes, such as the 1% labeled data. As the fraction of annotated data increases, performances tend to level out. Following previous work (Chen et al., 2020b), we fine-tuned the pre- trained encoders for 1000 epochs from the first layer of the projection head, ef. Appendix
We observed an interesting performance scaling when train- ing ViTs with the SOP algorithm. As we increased the com- plexity of the VIT backbones, the performance gains were greater than those observed for competing methods. In Ta- ble while SOP* performance using the VIT-S backbone
5

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Table Transfer learning by fine-tuning SSL methods on smaller datasets. Top-1 accuracy for VIT-B encoders.
Table 4. Video object segmentation on DAVIS 2017. We report mean region similarity Jn. and mean contour-based accuracy Fm..
METHOD DATA
ARCH.
(I&F)
Im
Fm
METHOD
C10
C100
INAT18
1NAT19
FLWRS
CARS
Sup.
RAND
99.0
90.8
73.2
77.7
98.4
92.1
IN-IK
IN-1K I/D/Y
VIT-S/8
66.0
63.9
68.1
BEIT
99.0
90.1
72.3
79.2
98.0
94.2
STM
RN50
81.8
79.2
84.3
DINO
99.1
91.7
72.6
78.6
98.8
93.0
Self-Sup.
IBTT
99.2
92.2
74.6
79.6
98.9
94.3
CT
VLOG
RN50
48.7
46.4
50.0
SOP
99.3
92.4
74.6
79.7
99.0
94.5
MAST
YT-VOS
RNI8
65.5
63.3
67.6
STC
KINETICS
RN18
67.6
64.8
70.2
DINO
IN-IK
VIT-S/16
61.8
60.2
63.4
IN-IK
VIT-B/16
62.3
60.7
63.9
3.3. Dense Prediction Tasks
IBOT
IN-1K
VIT-S/16
61.8
60.4
63.2
IN-1K
VIT-B/16
62.7
61.7
63.7
To address wide range of downstream tasks such as object detection, segmentation, and classification, optimal fixed- size representation should balance coarse and fine-grat fine-grained features. To evaluate the effectiveness of SOP in dense pre- diction tasks, we consider three downstream evaluations: (1) object detection, (2) semantic, and (3) instance segmenta- tion. Complementary to the linear evaluations in Section 3.1, the learned representations must encode information beyond the class level, including the object's 's localization, shape, and ability to discriminate among different instances.
SOP
IN-IK
VIT-B/I6
63.3
61.7
65.0
CIFAR-10/100 (Krizhevsky & Hinton, 2009), iNaturalist 2018/2019 (Van Horn et al., 2018), Oxford 102 Flower (Nils- back & Zisserman, 2008), and Stanford Cars (Krause et al., 2013). SOP pre-trained encoders demonstrate strong down- stream performance on fine-tuning protocols, surpassing competitors 5 out of 6 datasets with modest gains. We hypothesize that the extended fine-tuning regime of 1000 epochs, as per Zhou et al.'s (2022) protocol, leads most methods to achieve similar performance levels, indicating a saturation point.
Object Detection and Instance Segmentation on COCO. In Table 2, the first and second columns present the AP and AP" metrics for various SSL methods on the COCO dataset (Lin al., 2014), with Mask R- R-CNN (He et al., 2017) as the task layer. The entire network is fine-tuned for 12 epochs, following the protocol outlined by Zhou et al. (2022). The representations learned using the SOP strategy show modest improvements of +0.2 in APh and +0.1 in APM over iBOT for object detection and instance segmentation, respectively.
3.5. Image Retrieval
Image retrieval. To assess the image retrieval properties of SOP's representations, we consider the revisited Oxford and Paris image retrieval datasets (Radenović al., 2018). Each dataset is divided into three sets of increasing difficulty. We use SOP's frozen encoders as feature extractors and apply k-NN classification on the frozen features. In Table 5, we report Mean Average Precision (mAP) for the Medium (M) and Hard (H) splits. Features pre-trained with SOP significantly outperform current state-of-the-art methods, improving mAP performance by up to +3.2 on the Hard split of both benchmarks. We also include results from supervised retriival-specific baseline (Revaud et al., 2019).
Semantic Segmentation on ADE20K. In Table 2, the third and fourth columns report the mean intersection over union (mloU) for semantic segmentation on the ADE20K dataset (Zhou et al., 2017). Following Zhou et al. (2022), we con- sider two protocols: (1) linear probing and (2) fine-tuning. For linear probing, the patch tokens from the pre-trained SOP encoder are kept fixed, and only a linear model is trained on top of the frozen features. In the fine-tuning protocol, the task layer UPerNet (Xiao et al., 2018) is used, and we fine-tune all the parameters of the network. In both scenarios, SOP's pre- pre-trained representations improved upon iBOT's strong baselines by +0.4 and +0.6 mloU, re- spectively, and further increased the gap to the supervised baselines by +3.3 and +5.0 mloU, respectively.
Video instance segmentation. In Table 4, we use frozen patch tokens from SOP's pre-trained encoders to perform video scene segmentation using a nearest neighbor classifier between consecutive frames. As we do not update any addi- tional parameters, this evaluation is particularly valuable for assessing the fine-grained downstream capabilities of SOP's frozen features, which are learned through reconstruction using our proposed SOP-MIM task (4). We compare the performance of SOP to existing SSL methods and to su- pervised ViT-S/8 model trained on ImageNet- IM. SOP's features surpass the iBOT baseline by up to +1.3 on mean contour accuracy Fu.
3.4. Transfer Learning
In Table 3, we study transfer learning tasks using SOP pre-trained encoders as initialization to perform fine- tuning on several classification tasks using smaller datasets. We report top-1 accuracy for six datasets including
6

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Table Image retrieval. mAP using features.
Table 7. Parametric vs, non-parametric tokenizers. trained DALL-E encoder.
A: pre-
ROx
RPar
Method
LICLE)
k-NN
Lin.
METHOD
ARCH.
EPO.
M
H
M
H
iBOT
V
69.1
74.2
SUP.
RN101
100
49.8
18.5
74.0
52.1
X
9.5
29.8
DINO
VIT-B/16
400
37.4
13.7
63.5
35.6
BEIT
A
X
6.9
23.5
IBOT
VIT-B/16
400
36.8
14.3
64.1
36.6
DINO
V
67.9
72.5
MASSL
VIT-B/16
400
39.3
14.1
65.8
38.1
BEIT+DINO
V
48.0
62.7
SOP
VIT-B/16
400
42.7
17.5
67.3
41.3
SOP
V
70.0
74.3
16.8
30.2
X
68.9
72.8
Table Robustness against background changes. ViT-B en-
enders.
Table 8. Multiple SEs on pretext tasks. k-NN top-1 accuracy.
BACKGROUND CHANGES
CLEAN
A: Support Embeddings [CLS]
OF
MS
MR
MN
NF
OBB
OBT
IN-9
[MIM]
2
4
8
16
IBOT
91.9
89.7
81.9
79.7
54.7
17.6
20.4
96.8
MASSI.
91.0
90.2
83.0
80.4
53.4
15.8
23.7
97.6
69.3
69.6
69.5
70.0
69.5
2
69.7
69.3
69.6
69.4
SOP
93.3
91.4
85.6
83.1
55.8
19.9
22.8
97.1
4
69.5
69.2
69.6
69.4
69.3
3.6. Robustness
Table 9, Modeling SEs' contributions in the SOP algorithm.
We evaluate the performance of SOP's pre-trained en- coders on robustness test containing seven variations of foreground/background mixing and masking using the ImageNet-9 dataset (Xiao et al., 2020). Results for ViT-B encoders are presented in Table 6. Our method significantly outperforms competitors in six of the seven background changes with notable gains in most of the categories: Only- FG (OF) +2.3, Mixed-Rand (MR) +2.6, Mixed-Next (MN) +2.7, and Only-BG-B (OBB) +233.
METHOD
SOFT
ONE HoT
K-NN
70.0
69.7
On the number of support embeddings. In Table 8, we examine the impact of the number of support embeddings (SEs) within an SOP on each of the proposed loss functions. Our findings indicate that the global loss (2), which operates on [CLS] tokens, benefits from the inclusion of SEs. Per- formance improves as the number of SEs per SOP increases, peaking at eight, after which performance starts decrease. Conversely, for the local SOP-MIM loss (4), Table 8 shows that multiple SEs neither improve degrade performance. The empirical results in Table 10 suggest a meaningful per- formance gain from the incorporation of multiple SEs. We hypothesize that such gains come from the ability of SEs to better adapt to local structures in the latent space, resulting in enhancement the prototypes' features, which in turn benefits the SSL view optimization process.
4. Ablations
To understand why the SOP strategy learns useful visual rep- resentations from unsupervised data, we examine its main components and the rationale for selecting the optimal set of hyperparameters. Unless otherwise specified, ablations were conducted using ViT-S encoders pre-trained for 300 epochs without multicrop augmentation.
Online vs. non-parametric tokenizers. In Table 7, we compare the performance of methods using online and pre- trained tokenizers vs. our non-parametric approach. We ablate the effect of each loss function, (2) and (4). Op- timizing both loss functions, C [CCS)] + L [MIM), our method achieves k-NN 1 accuracy gain of 0.9% over iBOT. Minimizing only the LICLS] J objective, similar to DINO (Caron et al., 2021), SOP's k--N performance im- proves by 1.0%. Lastly, optimizing only the L. [MTM] loss, SOP's non parametric strategy (4) outperforms the para- metric counterpart iBOT by 7.3% accuracy points in k-NN, indicating that the proposed SOP-MIM learns faster and contributes more to the final representation.
On the contribution of support embeddings. As de- scribed in Section 2.2.1, SEs are selected as the closest embeddings to the SOP's anchor using spherical k-NN. When combining the individual scores of SEs within an SOP, the contribution of each SE to a view proportional to its distance the anchor. Consequently. SEs closer to the anchor have stronger influence on the view member- ship calculation than those farther away. In Table 9, we explore an alternative approach to modeling SE's contribu- tions. Instead of using the distance to the SOPs' anchors as the contribution weight, we assign one-hot distribution to

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
scribes framework where the relationship between vari- ables can be derived from the data without assuming any parametric form (Sanborn et al., 2024). Wu et al. (2018) proposed a non-parametric alternative to the parametric soft- max classifier to solve unsupervised classification problems at the instance level using Noise Contrastive Estimation (NCE) to approximate the full softmax. Subsequent work by He et al. (2020) and Chen al. (2021) builds upon this idea but uses augmented versions of the same image (views) as positives. He et al. (2020) employed memory bank to sample negative pairs and optimized variation of the NCE loss, termed the InfoNCE (Oord et al., 2018). Chen et al. (2020a) avoided an external memory by exploring in- in-batch representations to sample negatives. Similarly, Dwibedi et al. (2021) optimized the InfoNCE using different images as positive pairs. For each input image, the most similar representation in memory is taken positive and the rest of the representations in memory are deemed as negatives. Recently, Silva et al. (2024) proposed a non-parametric ap- proach for clustering- is that views of an image should produce similar predic- tion patterns when compared to representations of similar images stored in memory.
Table 10. Learning multiple SOPs. I-NN top-1 accuracy.
METHOD
1024
2048
4096
8192
16384
IBOT
67.6
67.9
68.3
69.1
1
68.8
SOP
69.5
69.7
70.0
69.9
69.7
each SE, meaning that SEs within an SOP contribute equally when predicting the views' membership to an SOP. Our proposal demonstrates robustness to both methods, with slight preference for our soft-contribution approach.
On the number of Self-Organizing Prototypes. The SOP strategy optimizes random regions in the represen- tation space by exploring feature locality. In Table 10, we examine how the number of SOPs affects the learned rep- resentations. The results indicate that our method is robust to varied number of SOPs. Notably, unlike prototypical SSL, which requires many prototypes, our method does not require a large number of SOPs to produce transferable off-the-shelf features. With 1024 SOPs (akin to 1024 proto- types in (Zhou et al., 2022)), we report 69.5% top-1 k-NN accuracy on ImageNet-IM. IM, which is 1.9% above iBOT's respective performance (67.6% (67..%)..o Moreover, the difference between optimizing 1024 SOPs vs. the best configuration (4096) is only 0.5%. For iBOT, a similar comparison yields a larger performance discrepancy of 1.5%. These results strongly suggest that SOPs alleviates the necessity and is- sues associated with over-clustering in prototypical SSL,
SSL. The primary assumption
SOP. Different from previous approaches, the proposed SOP strategy learns image embeddings by considering the viewpoints of many semantically similar support embed- dings (SEs) from different images representing regions of salient features in the data. Each SOP represents an adapt- able latent prototype in the data. Each SE encodes its own aspects of an SOP, and when combined, enriches the SOP's features. Our method does not require negative sampling and does not optimize the NCE or InfoNCE objectives. Moreover, our method is a general framework, and under a strict configuration, is equivalent to the framework of Silva et al. (2024). Additionally, we propose the novel SOP-MIM (Section 2.3) loss, where the reconstruction task is based on the viewpoints of local-level SEs representing different image patches in a non- non- parametric space.
5. Related Work
Clustering and Representation Learning. Combining clustering and deep learning has been promising approach for unsupervised visual representation learning. Caron et al. (2018; 2019); Van Gansbeke et al. (2020) incorporated clas- sic methods such as k-Means and k-NN in a deep unsu- pervised learning framework for visual features. Asano et al. (2020) proposed a self-labeling unsupervised method as an instance of the optimal transport problem. Caron et al. (2020) proposed mini-batch version of the Sinkhomn Knopp algorithm (Cuturi, 2013) to optimize cluster assign- ments between views of an image. Silva & Ramírez Rivera (2022) followed the clustering idea using SGD. Caron et al. (2021) scaled previous ideas to ViTs (Dosovitskiy et al., 2020). Inspired by modern NLP methods (Devlin et al., 2019), Zhou et al. (2022) investigated the masked image modeling (MIM) pretext task, also studied by Bao et al. (2021). These methods require special regularization tech- niques, such as centering, sharpening, and Sinkhorn-Knopp. to avoid ill-posed states.
6. Conclusions
We presented Belf-Onganizing Prototypes, a novel SSL pre- training strategy to learn effective representations from un- labeled images. SOP addresses the problem of underrep- resented prototypes in SSL by enhancing the feature set of prototypes through multiple support embeddings that re- side in a semantically similar region in the -araameettiic space of features. Our method avoids learning prototypes and presents two novel non parametric pretext tasks that are stable to train and do not require extra regularization to avoid collapsed solutions. We showed that training SSL methods with the SOP strategy alleviates the problems asso- ciated with over-clustering present in the majority of current
paramtrte SSL. The term non- non-maraetttic does not imply learning systems without parameters. Instead, it de-
8

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
state-of-the-ard prototypical SSL. Our comprehensive bench- marking showed that SOP's visual representations perform well in many downstream tasks such as object detection, instance and semantic segmentation, image retrieval, and linear probing. Additional improvements such as hyper- parameter tuning, extra regularizers, and scaling techniques, as studied by Oquab et al. (2023), can potentially improve SOP's performance and are left for future work.
data. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 2959- 2968, 2019.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2020.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging Properties in Self-Supervised Vision Transformers. IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 9650-9660, 2021.
Acknowledgements
The computations were performed in part on resources pro- vided by Sigma2-the the National Infrastructure for High Per- formance Computing and Data Storage in Norway-througge Project NN8104K, This work was funded part by the Re- search Council of Norway, through its Centre for Research- based Innovation funding scheme (grant no. 309439), and Consortium Partners. This study was financed in part by the Coordenação de Aperfeigoamento de Pessoal de Nível Superior-Brasse (CAPES)Finaany) Code 001.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual repre- sentations. In Inter. Conf. Mach. Learn. (ICML), 2020a.
Chen, T., Kormblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi- supervised learners. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 33, pp. 22243-22255, 2020b.
Chen, X.. Xie, S., and He, K. An Empirical Study of Training Self-Supervised Vision Transformers. In IEEE Inter. Conf. Comput. Vis. (ICCV), 2021.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 26, pp. 2292-2300, 2013.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 4171-4186. Association for Computational Linguistics, June 2019.
References
Asano, Y. M., Rupprecht, C., and Vedaldi, A. Self-labelling via simultaneous clustering and representation learning. In Inter. Conf. Learn. Represent. (ICLR), 2019.
Asano, Y. M., Rupprecht, C., and Vedaldi, A. Self-labelling via simultaneous clustering and representation learning. In Inter. Conf. Learn. Represent. (ICLR), 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. An Image is Worth 16x16
Assran, M.. Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with joint-embedding predictive architecture. In IEEE/CVF Inter. Conf. Comput. Vis. Put- 2023.
Words: Transformers for Image Recognition at Scale. In
Inter. Conf. Learn. Represent. (ICLR), 2020.
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman, A. With a little help from my friends: Nearest- neighbor contrastive learning of visual representations. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 9588-9597, October 2021.
tern Recog. (CVPR), pp.
Bao, H., Dong, L., Piao, S., and Wei, F. BETT: BERT Pre- Training of Image Transformers. In Inter. Conf. Learn. Represent. (ICLR), 2021.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask R-CNN. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 2961-2969, 2017.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In European Conf. Comput. Vis. (ECCV), pp. 132-149, 2018.
He, K. Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 9729-9738, 2020.
Caron, M., Bojanowski, P., Mairal, J., and Joulin, A. Un- supervised pre-training of image features on non-curated

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 554-561, 2013.
Silva, T. and Ramírez Rivera, A. Representation learning via consistent assignment of views clusters. In IEEE Inter. Symp. Applied Comput. Intell. Inf. (SACT), pp. 987-994, 2022. ISBN 9781450387132. doi: 10.1 (0.1444/3477314 3507267.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. 2009.
Silva, T., Pedrini, H., and Rivera, A. R. Learning from mem- ory: Non- con-parametric memory augmented self-supervised learning of visual features. In Inter. Conf. Mach. Learn. (ICML), pp. 1-17, July 2024.
Li, C.. Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., and Gao, J. Efficient Self-supervised Vision Transformers for Representation Learning. In Inter. Conf. Learn. Represent. (ICLR), 2022.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention. In Inter. Conf. Mach. Learn. (ICML), pp. 10347-10357. PMLR, 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common Objects in Context. In European Conf. Comput. Vis. (ECCV), pp. 740-755. Springer, 2014.
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proes- mans, M.. and Van Gool, L. Scan: Learning to classify images without labels. In European Conf. Comput. Vis. (ECCV), pp. 268-285. Springer, 2020.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 10012-10022, 2021.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pp. $769-8775 2018.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Wang. T. and Isola, P. Understanding contrastive represen- tation learning through alignment and uniformity on the hypersphere. In Inter. Conf. Learn. Represent. (ICLR), pp. 9929 9939. PMLR, 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation Learn- ing with Contrastive Predictive Coding. arXiv preprint arXiv:1807, 00744, 2018
Oquab M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.-YY, Xu, H., Sharma, V., Li, S.-W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I,, Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning Robust Visual Features without Supervision. Trans. Mach. Learn. Res., 2023.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised fea- ture learning via non-parametric instance discrimination. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 3733-3742, 2018.
Xiao, K., Engstrom, L., Iyyas, A., and Madry, A. Noise signal: The role of image backgrounds in object recogni- tion. In Inter. Conf. Learn. Represent. (ICLR), 2020.
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified Perceptual Parsing for Scene Understanding. In European Conf. Comput. Vis. (ECCV), pp. 418 1188434, 2018.
Radenović, F., Iscen, A., Tolias, G., Avrithis, Y., and Chum, O. Revisiting Oxford and Paris: Large-scale image re- trieval benchmarking. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 5706-5715, 2018
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene Parsing through ADE20K Dataset. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 633 641, 2017.
Revaud, J., Almazán, J., Rezende, R. S., and Souza, C. R. d. Learning with average precision: Training image retrieval with listwise loss. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 5107-5116, 2019.
Zhou, J., Wei, C., Wang. H., Shen, W., Xie, C., Yuille, A., and Kong, T. iBOT: Image BERT Pre- neeiTriinng with On- line Tokenizer. In Inter. Conf. Learn. Represent. (ICLR), 2022.
Sanborn, S., Mathe, J., Papillon, M., Buracas, D., Lille- mark, H. J., Shewmake, C., Bertics, A., Pennec, X., and Miolane, N. Beyond euclid: An illustrated guide to mod- ern machine learning with geometric, topological, and algebraic structures. arXiv preprint arXiv:2407.09468, 2024.
10

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
A. Implementation Details
For the main experiments in Section we train SSL encoders using the SOP strategy with three Vision Transformer architectures: ViT-Small, ViT-Base, and ViT-Large, with 21, 85, and 307 million parameters, respectively. Additionally, we experiment with Swin-Transformer backbone containing 28 million parameters. Following previous methods (Caron et al., 2021; Zhou al., 2022), create 12 views of the same image each training iteration. Views indexed from {0, 1} have shape x² E R 222222xx and views indexed from
{2, 3, 4, ..., 11) have shape x² E I996
The memories Ec E IR Nexd and Ep E RR xd store vector representations from global and local patches, respectively. We use Ne = 65536, N2 = 8192, and set the feature dimensionality to = 256. At each iteration, we update each memory following a FIFO (first-in first-out) strategy. For Ec, we select the [CLS] token representation from one of the global views and insert it into one end of Ec. For Eg,, we randomly pick one of the local patch embeddings using a uniform distribution and insert it into one end of Ep.
For the non non-parametric support embedding (SE) selection algorithm (2), uniformly sample K 4096 anchors. Each anchor selects an additional k- = 8 neighbors, resulting in total of 9 SEs per SOP. Refer to Section 4 for additional context on the optimal number of support embeddings. After selecting SEs, we create the pseudo-dataset D E p where K is the number of anchors, & is the number of SEs, and d = 256 is the feature vector dimensionality. Likewise, the soft contributions Y E (k++))))
For the SOP-MIM loss (4), sample K- 512 anchors. As shown in Table 8, the SOP-MIM loss does seem benefit from multiple SEs. Thus, and labels have shapes D R. Rxd and Y IR k
single SE (the anchor itself), to represent local SOP. Consequently, the pscudo-dataset
In practice, for the global loss (2), given K 4096 anchors and k: support embeddings, the pseudo-dataset D has shape 26666 Likewise, for the SOP-MIM local loss (4), with K = 512, the pscudo-dataset D has shape 251
A.1. Py Torch Style Pseudo-code
memory.
class Memory (nn. :
def _init__ (self, K, num_anchors, num_SEs, dim, smoothing-O. 1)
self.K =
self.num anchors num anchors
self.num SEs num SEs
self.smoothing smoothing
self.memory = F. norrmalize (randn (dim, K), dim=0)
self.labels arange (0, num_anchers)
self.labels self.labels hetplssrreeattttteeeeeeeee seee mmun_83s, dim=0) unsqueeze (1)
def
idx (self):
indices = multinomial (ones (self.K), aolf.num_anchora roplacement-Falee
return indices Indices.unsqueeze
unsqueeze (3)
forward (self, s_embeds, t_embeds):
s_embeds: [N x dim]
t_embbas (M x dim] "**
labels self. labels smooth self, smoothing
memory self.memon
anchor_ide self. sele____ccccccccrrrr__ anchor_embeds take_along_din (memory, indicces-anchor_idx dim=l) # [dim x mum_anchors
top-k nearest neighbours similarities anchor anchor_embedd. embeds.t memory
scores, indices = topk

---

Self-Organizing Visual Prototypes for Non-Paramettie Representation Learning
similarities, k=self.nun_ses, dim=-1)
get emebddings of anchors and support embeddings
indices indices. flatten () unsqueeze (0)
embeddings take_ make_along dim (memcry, indices, dim=1) # [dim num_ num_anchors]
amooth_valur = smooth (self.nur_anchoon
1.0)
exp_labels full(
(labels. shape [0], self num_anchors), smooth_value)
anchors contributions sum to across all SOPs exp_labels. scatterr labels, Full_1ike (exp_ labela 1.0 smooth)
logits s_embeds | embedding ngs s_temp
[N SkS num_anchors] (N
logits embeds embeddings temp
SEs anchors
probs softmax is logits, dim=-1) @ exp labels $ [N x num anchors]
t_probs softmax (t_logits, dim=-1) 0 exp_labels
[N x nun_anchors]
return ppcbs, t_probs
main.py
f():: student encoder
g(.): teacher encoder
K: memory size
0: embedding dimension
6: sequence length (number of patches)
number of views
batch size
class_ _aso_r_emrry__oeee = Memory (K=65536, num_anchors=109, nun SEa=8, dim=256)
patch_memery Memory (K-8192, nun_anchors=512, num_$ dim=256)
for images in loader:
student and teacher branches
s_out, t_out = fimagess) (images (:2]) $ [VNN x DJ, [2*N L, DJ
s_cls, s_patch s_out t_eis, p_tch t_u
cls s_probs, t_probs cls_mem_out
_em__mmm______m mem_mmmtttt__eee out = class_memory cls, t_cls)
global [CLS] loss
probs s ...poosschunkk chunk (V)
probs = t_probs. chunk (2) there are two global views
cls_ce cross_entropy (s_probs, t_probs)
SOF-MIM patch loss
patch_mem_out patch_memory (s_patchss__phh t_patch)
sp_probs, tp_probs patch_mem_cnt
s9_probs ue.probs.ccunk (2)
tp_probs = chunkk (2)
patch_ce ce = cross entropy (sp_pr tp probs, masks)
loss = cls
patch_ce
optimizer.zero_geee zero_grad
loss. loss.backwar backward optimizer.step.
update memories
update_mem (class_memory, t_cls)
update_ment memooy, t_patch)
12

---

Self-Organizing Visual Prototypes for Non-Paramettic Representation Learning
Table B.I. Training time and memory: report top-1 k-NN accuracy on ImageNet- training time (hours), memory (gigabytes) for SSL methods using VIT-S/16 backbones.
100 EPOCHS
300 EPOCHS
800 EPOCHS
k-NN
TIME
K--N
TIME
k-NN
TIME
MEM
DINO
69.7
24.2H
72.8
72.6H
74.5
180.0H
15.4GB
IBOT
71.5
24.3H
74.6
73.3H
75.2
193.4H
19.5GB
MASSL
72.7
24.2H
74.7
72.44
75.1
177.3H
15.1GB
SOP
72.8
24.411
74.7
73.3H
75.2
193.5H
19.4GB
B. Extended Experiments
B.1. Time and Computing Trade-off
In Table B, we explore trade-offs between parametric and non-parametric SSL. Following the protocol from (Silva et al., 2024), we report the training time and memory requirements for SOP and existing solutions. The main difference between iBOT/DINO and SOP the absence of learnable prototypes in SOP. Instead, SOP employs two memory components, Ec and Ep. to store [CLS] and patch-level representations, respectively. In contrast, iBOT learns two separate sets of prototypes: for [CLS] tokens and second for patch-level tokens trained with MIM. From resource perspective, learning the prototypes requires extra memory to store gradients for updating the prototypes during the backward pass. SOP on the other hand, updates the prototypes following simpler FIFO strategy. Despite this, the general computing time and memory requirements for pre-training SOP on ImageNet- MM are very similar to those of iBOT.
B.2. Semi-Supervised Evaluations with Frozen Features
In Table 1, we assessed the supervipe performance of SSL methods using the unsupervised pre-train and supervised fine-tune paradigm. Additionally, Table B.2, we compare the performance multiple SSL methods on semi-supervised task setup using frozen, off-the-shelf features on the ImageNet dataset. We report k-NN top-1 accuracy for the best- performing value of k E (10, 20, 100, 200} using the data splits provided by Chen et al. (2020a).
SOP 's performance significantly improves model complexity increases. For ViT-S backbones, SOP performs comparably iBOT in both data regimes. However, with the more complex ViT-B and Swin'T backbones, the performance gap between SOP and its competitors widens significantly, with gains of +2.3 and +1.4 for ViT-B in data regimes of 1-10% labels, respectively.
We emphasize the still substantial gap between supervised methods (Touvron et al., 2021) and unsupervised methods on retrieval-based tasks. Specifically, for low data regimes, existing gap suggests that current SSL methods still have room for improvement.
Table B.2. Semi-supervised evaluations with frozen features on ImageNet-IM: report k-NN top-1 accuracy using 1-10% labels. reference, we include results from supervised DeiT (Touvron et al., 2021).
METHOD
ARCH.
1 1
10%
Supervised
DEIT
VIT-S/16 VIT-B/16
77.3 80.2
78.7
DEIT
80.9
Self-supervised
DINO
VIT-S/16
61.3
69.1
VIT-B/16
71.0
IBOT
VIT-S/16
62.3
70.1
VIT-B/16
66.3
72.9
SWINT-14
64.2
71.5
SOP
VrT-S/16
62.2
70.3
VIT-B/16
68.6
74.3
SWINT-14
65.3
72.3
13

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
B.3. Dense Prediction Tasks
In Table B.3, we provide additional metrics for object detection, instance segmentation, and semantic segmentation evaluations using SOP's ViT-B pre-trained backbone. For object detection and instance segmentation, use the Cascade Mask R-CNN the task layer and the COCO dataset (Lin et al., 2014). In addition the metrics reported in Table 2, include AP 50 and AP75 for object detection, and AP and AP75 for instance segmentation.
For semantic segmentation on ADE20K (Zhou et al., 2017), we follow the protocol from Zhou et al. (2022) and consider two scenarios: (I) training linear layer on top of the frozen encoder, and (2) using UPerNet as the task layer.
Table B.3. Additional results for object detection, instance segmentation, and semantic segmentation using ViT-B encoders.
DET. & INST. SEG. w/ CASCADE MASK R R-CNN
SEG. W/ LIN.
SEG. w/ UPERNET
METHOD
AP
AP
AP7s
AP"
AP50
AP%
MlOU
MACC
MIOU
MACC
SUP.
49.8
69.6
53.8
43.2
66.6
46.5
35.4
44.6
46.6
57.0
DINO
50.1
68.5
54.6
43.5
66.2
47.1
27.4
35.5
45.8
55.9
IBOT
51.2
70.8
55.5
44.2
67.8
47.7
38.3
48.0
50.0
SOP
51.4
70.9
55.5
44.3
68.0
47.8
38.7
48.1
50.6
60.5
C. Extended Ablations
C.1. Multiple Tasks Improve the Learned Representations.
As explained in Section 2.2, the SOP strategy first samples a subset of anchors A = {a C Ec, where each anchor can be seen as hidden local cluster within the data. Next, each anchor a; selects additional representatives (support embeddings) using the k-Nearest Neighbor method. Thus, each SOP is represented by its anchor a; along with k additional support embeddings ej. as determined by the N--N criterion.
This process be repeated multiple times within each training iteration. In Table C.1, we report the effects of this strategy for each of the pretext tasks described in Section 2.2.1 and Section 2.3. We observe positive trend the number of SOP tasks performed per training iteration increases. Moreover, Table C.1 suggests that both global and local tasks benefit from this strategy.
Table C.I. The effect the number independent pretext tasks per iteration.
# OF TASKS [CLS]
[MIM]
2
4
68.0
68.5
68.6
69.6
70.0
69.7
69.7
69.8
70.0
C.2. Learning Global-Level Features: [CLS] vs. Average Patch Embeddings.
In Table C.2, investigate common strategies for learning class-level representations with VIT backbones. We compare (i) the default approach, which uses dedicated [CLS] token leam global information, with (ii) alternative that averages the patch-level embeddings. For SSL pre-training with SOP, the default [CLS] token strategy yields slightly better k-NN performance.
Table C.2, Global- representations as [CLS] vs AVG, patch visual embeddings.
[CLS]
AVG. PATCH
K-NN 70.0
67.8
14

---

Self-Organizing Visual Prototypes for Non-Parametrie Representation Learning
RAND BLOCK
70
Top-1
60
50
50
100
150
200
250
300
Epochs
Figure C.I. Blockwise vs. Random masking.
Table C.3. Does the number of local-level anchors matter?
Table C.4. The effect of the momentum hyper-parameter on the teacher encoder.
K
256
1024
k-NN
69.0
70.0
69.5
W
.992
.994->1
.996->
K-NN
69.2
70.0
69.6
C.3. The Masking Strategy
In Figure C.1, we compare two masking strategies for the SOP-MIM task: blockwise and random masking. The blockwise approach follows the iterative technique of Bao et al. (2021), where, at each iteration, a randomly sized and shaped block of the image is masked until the desired masking ratio is reached. In contrast, the random masking strategy independently masks patches according to a specified ratio. We use masking ratio 0.3 (30%) for blockwise masking and 0.7 (70%) for random masking. Figure C.1 shows that the blockwise strategy consistently yields higher top-1 k-NN accuracy
C.4. Does the Number of Local-Level Anchors Matter?
Similar to the global [CL.]] task, SOP-MIM samples subset of patch-level anchors À from the memory Ep, which stores local embeddings from previous iterations. Each anchor represents a local structure in the embedding space, grouping patch-level representations that share semantic features. In Table C.3, we examine how the number of sampled local anchors affects the N-NN performance of the learned representations. Overall, SOP demonstrates robustness across wide range of sampling sizes.
C.5. The Momentum Encoder
As is standard practice in SSL, SOPs trained with two sibling encoders in teacher-student setup. The student encoder is updated via gradient descent, while the teacher encoder is updated using a moving average of the student's weights: Φ, = m + (1 - m where 4, and 4, denote the weights of the student and teacher encoders, respectively. This framework can also be interpreted from a distillation perspective, where the teacher distills knowledge from previous iterations into the student. Here, m controls the flow of distillation between the two encoders and follows a cosine schedule. In Table C.4, we study the effect the hyperparameter m on the downstream performance of the learned representations.
C.6. [CLS] and Patch Memory Sizes
[CLS] Memory Size: Table C.5, we ablate the effect of memory size No on the learned representations. We observe an inverse U-shaped relationship between memory size No and downstream k--N performance: increasing No improves
15

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Table C.5. Increasing the memory size No henefits the learned representations.
Table C.6. Increasing the memory size Np benefits the learned representations.
Ne
8192
16384
32768
65536
98304
N
1024
2048
4096
8192
16384
67.0
67.7
68.5
70.0
69.7
68.3
68.7
69.2
70.0
69.1
Table C.7. Anchor selection strategies Sega-Organ Prototypes (SOPs). compare selecting anchors randomly at iteration (default SOP) with keeping anchors fixed during pre-training. Fixed anchor selection leads pre-training collapse.
FIXED ANCHORS
RANDOM ANCHORS
70.0
performance up to point, after which further increases lead to diminishing returns. For this experiment, the patch memory size fixed N₂- = 8192.
Patch Memory Size: In Table C.6, we ablate the effect of patch memory size N, on the learned representations. Similar Table C.5, we observe an inverse U-shaped curve between N, and downstream k-NN performance, suggesting optimal memory size at N₂ 8192. The [CLS] memory size is fixed No 65536.
C.7. Collapse Analysis
To assess the importance of randomization as a regularizer prevent training collapse, we explore two algorithmic variations for selecting anchors for SOPs. As described in Section 2.2, the default SOP strategy selects anchors uniformly random. In Table C.7, we compare our default strategy with alternative anchor selection method in which anchors are randomly selected at the beginning of training and kept fixed throughout. Keeping anchors fixed during training corrupts the learned features and leads to a collapse, suggesting that randomization in anchor selection is a key ingredient for avoiding collapse.
C.8. t-SNE Feature Visualization.
To qualitatively evaluate the features learned using our proposed non parametric SOP strategy, Fig. C.2 and Fig. C.3 present t-SNE visualizations of features on the CIFAR-10/100 datasets. For both datasets, we compare the feature spaces learned by our method (left) and by iBOT (right), using pre-trained ViT-B encoders.
16

---

Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Figure C.2, E-SNE visualizations on CIFAR-10 with SSL pre-trained ViT-Base feature extractors: SOP (left) vs. iBOT (right).
Figure C.3. t-SNE visualization on CIFAR 100 using SSL pre-trained ViT-Base encoders as feature extractors. Qualitative results for SOP (left) and iBOT (right).
17


Reviews:
[{"review_id": "FWGXGtw3zi", "reviewer": "Reviewer_2CgY", "summary": "This paper introduces  a novel technique for self-supervised learning,  Self-Organizing Visual Prototypes (SOP), to address limitations in existing prototypical self-supervised learning (SSL) methods. \nSOP enhances the feature set of prototypes by using multiple semantically similar support embeddings (SEs) to collectively represent regions in the feature space, rather than relying on a single prototype per cluster. This approach is implemented through a non-parametric framework that avoids over-clustering and mode collapse, resulting in stable and transferable representations.  The method is evaluated on the full range of standard self-supervised benchmarks, comparing against most expected methodologies and achieving good performance improvements across the board.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's fair assessment of our work. We would like to highlight key points from the review, including our \"extensive experimental verification\" and the accurate summary that captures the core innovation of our method. The review emphasizes our method's ability to mitigate over-clustering and its consistent performance across a wide range of benchmarks. We acknowledge the constructive feedback and address each point in turn.\n\n> Training time and memory usage of SOP and baseline methods are not compared. [...] In particular, the efficiency of k-NN search [...].\n\nRegarding the training time and memory usage of SOP against baselines, we kindly refer the reader to Section B.1 and Table B.1 in the appendix. In Section B.1, we discuss the trade-offs and differences between parametric and non-parametric approaches. To address the reviewer’s point, we found that the computing and memory requirements of SOP are similar to existing approaches such as iBOT (Table B.1). As pointed out, our method requires running k-NN to bootstrap the support embeddings. **On the other hand, our method does not have the large learnable prototypical layer present in clustering-based methods such as DINO and iBOT, which requires extra computing and memory to compute/store gradients in the backward pass**. Our method stores 256-d representations in two separate memories E_C and E_p of sizes 65536 and 8192, which are updated following an efficient FIFO strategy.\n\n> Transfer learning on smaller datasets is not very meaningful and can be placed in supplementary materials.\n\nWe acknowledge the reviewer's point and share their conclusion regarding the saturation of these benchmarks, as indicated in line 326, column 1. Following the reviewer's recommendation, we will relocate Table 3 (Transfer learning on smaller datasets) to the supplementary material and insert Table B.1 (Training time and memory trade-off) into the main text.\n\n> W1 Figure 2 is not very easy to understand, and it is difficult to make a good connection between the pseudo code and Figure 2\n\nFollowing the reviewer's suggestion, we have updated Figure 2 with supplementary labels to improve clarity. The revised figure can be found in Figure 2 (https://shorturl.at/hlvGW).\n\n> W2 The motivation of SOP-MIM is clear [...] it is just an incremental work of MIM. [...].\n\nThe majority of work on SSL that optimizes an MIM task focuses on learnable tokens, such as iBOT, BEiT, and DINOv2. Instead, **our work presents an alternative approach**. We show that a non-parametric strategy also works well for MIM, opening new possibilities for further development.\n\n> W3 Is the key to the improvement of this paper due to the increase in the number of anchor points? [...] current method performs best when the number of prototypes K≫C is much larger than the number of actual categories C [...], but the method proposed in this paper also requires larger prototypes.\n\nThanks for the feedback. We argue that the SOP strategy is a key factor contributing to the performance of our method. As shown in Table 8, overall performance improves with an increase in the number of support embeddings (SEs). This supports our claim that the SOP strategy, which involves bootstrapping semantically similar embeddings to better represent a random region in the feature space, is more effective than learning a single prototype for that region, as prototypical methods do.\n\nRegarding the number of anchors, we used a default value of K=4096. Although this may seem large, Table 10 demonstrates that even with K=1024 (nearly the same number of classes as in the ImageNet dataset), **our method still surpasses iBOT, which learns K=8192 prototypes**. Additionally, the performance difference between using K=1024 and K=4096 is minimal (only 0.5%). In contrast, iBOT shows a significantly larger gap (1.5%) when comparing similar settings, as detailed in line 397. These results corroborate our claims about overclustering and how the SOP strategy mitigates it.", "claims_and_evidence": "The paper's claims regarding the effectiveness of Self-Organizing Visual Prototypes (SOP) are well-supported by evidence. SOP's ability to mitigate over-clustering is demonstrated through strong performance with fewer prototypes compared to existing methods.", "methods_and_evaluation_criteria": "The proposed SOP method and evaluation criteria are well-suited for the problem of unsupervised visual representation learning. SOP addresses limitations of existing prototype-based methods through its non-parametric approach and multiple support embeddings.", "theoretical_claims": "The paper does not present formal proofs or extensive theoretical claims that require rigorous mathematical verification.", "experimental_designs_or_analyses": "This paper has conducted extensive experimental verification on ImageNet, including linear evaluation, semi-supervised fine-tuning, fine-tuning, and dense prediction on COCO and ADE20K, as well as image retrieval and video segmentation tasks, proving the effectiveness of the method. And ablation experiments were performed on the shutdown components mentioned in the method.\n\nHowever, I think there are still some points that can be improved:\n\n1. The training time and memory usage of SOP and baseline methods are not compared. Non-parametric memory management may increase the computational burden and affect practicality. In particular, the efficiency of k-NN search under large memory is not discussed, and there may be scalability issues.\n\n2. Transfer learning on smaller datasets is not very meaningful and can be placed in supplementary materials.", "relation_to_broader_scientific_literature": "The key contributions of the paper SOP are closely related to several important trends and findings in the broader scientific literature on self-supervised learning (SSL), representation learning, and computer vision.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\nS1 This paper introduces Self-Organizing Prototypes (SOPs), replacing fixed prototypes with dynamically constructed support embeddings (SEs) in non-parametric prototype design. This challenges the traditional assumption in prototypical SSL and enhances feature diversity beyond methods like iBOT and SwAV.\n\nS2 The paper proposes SOP-MIM, an extension of masked image modeling (MIM) that leverages non-parametric patch-level SEs instead of learned discrete tokens (e.g., BEiT). This reduces parametric bottlenecks and offers a novel perspective on representation learning.\n\nWeaknesses:\nW1 Figure 2 is not very easy to understand, and it is difficult to make a good connection between the pseudo code and Figure 2\n\nW2 The motivation of SOP-MIM is clear and easy to understand, but it is just an incremental work of MIM. Maybe I misunderstood it. The author can explain it further.\n\nW3 Is the key to the improvement of this paper due to the increase in the number of anchor points? As mentioned in the introduction, the current method performs best when the number of prototypes K≫C is much larger than the number of actual categories C in the data, but the method proposed in this paper also requires larger prototypes.", "other_comments_or_suggestions": "I think the clarity of the writing can be improved.\nThe term “prototype” is used ambiguously—sometimes referring to traditional parametric prototypes (e.g., iBOT) and other times to SOP anchors.", "questions_for_authors": "see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Ff6tOgwvQ", "reviewer": "Reviewer_Rjpz", "summary": "This paper proposes a new self-supervised learning approach that leverages multiple anchors and support embeddings to infer soft label for self-supervised learning. Extensive experiments demonstrate that the proposed method achieves favorable performance across various downstream tasks.\n\n** Update after rebuttal **\n\nThe performance advantages and novelty of this paper are not evident compared to existing SSL works. Thus I decide to maintain my rating.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her careful assessment and constructive feedback, which helped us clarify key concepts about our work, improve the readability of the paper, and strengthen our arguments. Below, we address each of the reviewer's points in turn.\n\n> Figure 2 lacks some illustration details [...] add explanations [...] for squares, circles, and different colors [...].\n\nWe thank the reviewer for the constructive feedback and apologize for the lack of clarity in Figure 2. To improve readability and aid reader comprehension, we have added text labels for each symbol in Figure 2. Please refer to [Figure 2](https://shorturl.at/hlvGW). The updated figure now includes labels below each core component, clarifying the meanings of squares, circles, and different colors.\n\n> What are the advantages of SOP over learnable tokens? no discussion of trade-offs [...]\n\nIn the paper, we highlight the following advantages of our method over existing prototypical SSL approaches:\n\n1- Our work reaffirms the feasibility of non-parametric SSL. Our goal is not to discourage parametric clustering-based approaches. Instead, we sought to demonstrate that our proposed non-parametric method is a viable alternative, a different path that can contribute to the scientific discussion and development of visual unsupervised learning.\n\n2- Our method avoids learning prototypes from random weights and exhibits several key benefits:\n- Stability: The method is inherently stable.\n- No Extra Regularizers: It does not require additional regularizers to prevent mode collapse (Section 2.2.1).\n- Extensibility: It is extensible to various pretext tasks such as MIM (Section 2.3 and Table 7).\n- No Over-Clustering: It does not necessitate over-clustering (Section 4, Table 10).\n- Transferable Representations: The SOP pre-training strategy produces representations that are transferable across many different tasks, with varying degrees of difficulty (Tables 1 to 7).\n\n3- Moreover, we show that SOP’s performance increases as we scale the model architecture (Section 3.1 and Table 1).\n\n> [...] the proposed method shows only marginal improvements over previous baselines, with some results improving by less than one percentage point [...].\n\nWe acknowledge the reviewer's concern regarding the magnitude of improvements. However, **we believe that scoring well across a wide range of downstream tasks demonstrates the robustness, stability, and effectiveness of the proposed approach, which is particularly challenging in SSL.**\n\nAs recognized, we conducted thorough evaluations on various downstream tasks, including classification, retrieval, robustness, fine-tuning, transfer learning, object detection, and segmentation. While some improvements may seem marginal, we would like to highlight several non-trivial improvements:\n\n- Image Retrieval (Table 5): mAP improved by +3.2.\n- Robustness Against Background Changes (Table 6):\n   - OnlyFG (OF) +2.3\n   - Mixed-Rand (MR) +2.6\n   - Mixed-Next (MN) +2.7\n   - Only-BG-B (OBB) +2.3\n- kNN with ViT-Large backbone (Table 1): +1.2%\n- Semi-Supervised Fine-Tuning with 1% of Labeled Data (Table 1): +1.0%\n\nAdditionally, in Table B.2 (appendix), we report an improvement gain of +2.3 in semi-supervised learning (low data regime) on ImageNet using frozen features.\n\n**These results collectively demonstrate that our method provides non-trivial improvements and is effective across various tasks and settings.**\n\n> In previous SSL methods, a view means an augmentation of the input image [...] in the context of SOP, the meaning of “view” is not clear. [...], this description needs more clarity.\n\nWe apologize for any confusion regarding the term \"view.\" In this work, views are augmented versions of an image, similar to existing SSL methods. Based on the reviewer's feedback, we have improved the readability of [Figure 2] \n (https://shorturl.at/hlvGW).\n\nAdditionally, we define a view $x^v$ as an \"augmented version of an image\" in the Notation paragraph (page 094, column 2), and we use multiple views (multicrop) during optimization. The misunderstanding may have arisen from a part of the text where we intuitively suggest viewing SOPs as augmented prototypes. This interpretation emphasizes that, rather than relying on a single prototype, SOPs group features among different SEs, collectively representing the region in space more accurately.\n\n> Some typo issues [...].\n\nThanks, we have corrected the typo issues and the format of double quotes.\n\n> [...] does SOP has higher computational efficiency compared to previous methods? The inclusion of an efficiency comparison is recommended.\n\nWe agree with the reviewer on the importance of an efficiency comparison. Please refer to Table B.1 (appendix), where we compare the computing and time resources of our approach against existing prototypical SSL methods. Our findings indicate that the computational and memory requirements of SOP are similar to those of existing approaches such as iBOT.", "claims_and_evidence": "The paper’s writing is poor, and not easy to follow.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "See Other Strengths And Weaknesses.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n\n- The paper conducts thorough experiments on various downstream tasks.\n- This paper proposes self-organizing visual prototypes as a non-parametric prototypical SSL technique.\n\n\n\nWeaknesses\n\n- Figure 2 lacks some illustration details. It is recommended to add explanations below the figure clarifying the meanings of squares, circles, and different colors to improve readability and aid reader comprehension.\n\n- What are the advantages of SOP over learnable tokens? There is no discussion regarding the trade-offs between non-parametric and parametric methods. \n\n- The authors conduct experiments on various downstream tasks. However, according to the experimental results, the proposed method shows only marginal improvements over previous baselines, with some results improving by less than one percentage point, which may fail to convincingly demonstrate the superiority of the proposed method.\n\n- In previous self-supervised learning methods, a view typically means an augmentation of the input image. However, in the context of SOP,  the specific meaning of “view” is not clear. While the authors mention that “each support embedding predicts the degree of similarity (or likelihood) of a view to its ”region” in latent space (SOP)”, this description needs more clarity.", "other_comments_or_suggestions": "Some typo issues. The format of double quotes is wrong, e.g., line 076 “region”.", "questions_for_authors": "As SOP is a non-parametric prototypical learning, does it have higher computational efficiency compared to previous methods? The inclusion of an efficiency comparison is recommended.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LSx8xjfbLB", "reviewer": "Reviewer_KMMU", "summary": "1-1. The authors proposed a non-parametric design called Self-organizing visual embeddings (SOLVE) in the field of self-supervised learning. Specifically, SOVE assumes the presence of multiple anchors that together represent a single concept, combining them to form a single soft label for self-supervised learning. Additionally, it presents a non-parametric design for masked image modeling loss, intended for joint use during training.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for his/her careful reading and constructive feedback. The insights have significantly contributed to improving our manuscript. Below, we address each point in turn.\n\n>2-1 underrepresentation and overclustering [...] no evidence [...]\n\nWe argue that over-clustering and underrepresentation of prototypes are related. As noted in lines 036 and 052 (col 2), due to the weak supervision in SSL, prototypical methods tend to learn a large number of prototypes, dividing the feature space into many smaller sections, each with a prototype as its centroid. Consequently, each section contains fewer mages, biasing the prototypes towards learning simpler features, potentially leading to an underrepresentation of its region.\n\nTo support our claim, we conducted an ablation study on different values of K (number of SOPs). If our claim is correct, our method should perform well with a small number of SOPs. In Table 10 and line 394, using K=1024 anchors (similar to ImageNet classes), the SOP strategy outperforms iBOT, which learns K=8192 prototypes. Furthermore, the performance difference between K=1024 and our default value of K=4096 is minimal (0.5%). In contrast, iBOT exhibits a 3x larger discrepancy (1.5%) for the same comparison; DINO [2] reports an even greater difference (1.9%).\n\n>2-2.[...] no difference one-hot vs. soft approach [...]\n\nIn Table 9, we compared two strategies for modeling the contribution of SEs towards their SOPs. We reached the same conclusion as the reviewer and demonstrated that our method is robust to both strategies (line 377, column 2).\nWe believe the \"soft assignment\" approach has a slight advantage because it handles uncertainty better, which is crucial in SSL. Intuitively, we can relate to the \"label smoothing\" regularizer commonly used in semi-supervised setups, which improves generalization by softening ground-truth labels, reducing noise and preventing overfitting.\n\n>2-3. no performance difference for C=K=100 in [1] [...]\n\nIf we understand correctly, the reviewer is asking why SwAV [1] does not report a significant performance difference between learning K=1000 and K=3000 prototypes, which might contradict our claim. We believe there are two reasons. First, the difference between K=1000 and K=3000 may not be large enough to highlight the discrepancy. Second, performance-wise, iBOT and DINO are more effective than SwAV. In fact, DINO is the successor of SwAV (same main author), and **one of the reasons why it performs better is over-clustering**. This is demonstrated in the DINO paper [2], page 16, where there is a significant difference in performance (1.9%) from learning K=1024 to K=65536 prototypes, supporting our claim. The same pattern occurs in iBOT, (cf. Table 10). We agree that the reference to [1] should be removed from this context.\n\n[2] Emerging Properties in Self-Supervised Vision Transformers (ICCV)\n\n>3-1-1. Nc and Np not defined.\n\nThanks for pointing this inconsistency. Specifically, $N_C$ and $N_p$ are the sizes of the memories $E_C$ and $E_p$. \n\n>3-1-2. [...] not possible to infer Y.\n\nThank you for the feedback. The manuscript now includes the shape of $Y$ in the text. Please note that the shape of $Y \\in \\mathbb{R}^{K(k+1) \\times K}$ is defined in Figure 2's caption, and its practical size is detailed in Sec. A, appendix. Please refer to line 193 for an explanation of $Y$, which encodes the contributions of SEs, towards their SOPs. \n\n>5-1.\"complex backbones, larger performance gains\" no intuition [...]\n\nThank you. The manuscript has been updated with an intuitive explanation. We believe the performance gain from larger backbones is linked to a higher efficiency of the SOP mechanism, which provides richer features to optimize views. Since SOPs use support embeddings (SEs) to represent their region in latent space, complex backbones can better utilize SEs' features, resulting in larger gains (cf. Table 1, line 264).\n\n>10-W1. overall improvements [...] seem marginal.\n\nDue to space limitations, we kindly refer the reviewer to our third response to reviewer Rjpz, where we discuss this issue in detail.\n\n>10-W2. Table B.1, advantages [...] not clearly evident\n\nWhile we did not claim advantages in computing and memory, Table B.1 shows that our non-parametric approach does not incur additional computational/time requirements and can be trained on commodity hardware. Please, note that MaSSL does not use a MIM loss, reducing computing/time resources.\n\n>12-1. When comparing the case where [...]\n\nThe fact that a support embedding (SE) may belong to more than one SOP is a consequence of the k-NN algorithm. Each anchor selects the k closest embeddings as SEs, allowing distinct anchors to select the same SE. To address the reviewer's concerns, we tested a [Non-shared SEs] (https://shorturl.at/iqdmT) version, where an SE cannot participate in more than one SOP. Results show no practical difference, likely due to the small number of such cases.", "claims_and_evidence": "2-1. In the introduction, Lines 56-63, the issues of underrepresentation and overclustering are mentioned, but there is no supporting evidence to demonstrate whether the proposed method addresses these issues.\n\n2-2. In Table 9, there appears to be no significant difference between the one-hot approach and the proposed soft approach. In Line 210, the effectiveness of the soft approach is explained as 'potential noise due to false positives from the SE selection algorithm,' but there is no supporting evidence provided for this claim.\n\n2-3. According to Table 12 of [1] mentioned in the claim on Line 41, there seems to be no significant performance difference even when the number of prototypes $C=K=1000$, compared to the $1000=C <K=3000$ used by the authors of [1]. However, there is no explanation provided for this.\n\n[1] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (NeurIPS’20)", "methods_and_evaluation_criteria": "3-1. The explanation of the proposed method is difficult to read. For example,\n\n3-1-1. In Line 138, $N_c$ and $N_p$ are not defined.\n\n3-1-2. In Line 197, there is a sentence starting with \"we introduce $Y$,\" but it is only possible to infer what $Y$ is, its size, and its meaning from the following context.", "theoretical_claims": "4-1. The theoretical proof is not provided in the paper.", "experimental_designs_or_analyses": "5-1. In Line 269, it is mentioned that \"more complex backbones produced larger performance gains,\" but the intuition behind this statement is not provided.", "relation_to_broader_scientific_literature": "7-1. Self-supervised representation learning is an actively researched field. While various methods have been proposed, representation collapse remains a critical issue that needs to be addressed. This work proposes a new solution to this problem.", "essential_references_not_discussed": "8-1. It seems that there are no essential references that have not been discussed.", "other_strengths_and_weaknesses": "10-S1. The non-parametric approach to masked image modeling appears to be a reasonable method.\n\n10-W1. Looking at the tables provided in the main experiments, aside from image retrieval, the overall performance improvement compared to other self-supervised methods (e.g., iBOT) seems marginal.\n\n10-W2. Upon reviewing Table B.1, the advantages in terms of train time, memory, and performance over the existing method, MaSSL, are not clearly evident.", "other_comments_or_suggestions": "None.", "questions_for_authors": "12-1. When comparing the case where \"allows a SE to belong to more than one SOP\" (mentioned in Line 190) is applied versus when it is not, is there a performance gain? Or was this method used simply because it is a simpler way to assign soft probabilities?\n\n12-2. Please also refer to questions 2, 3, 5, and 10.\n\n---\n\n**After Rebuttal**\n\nWhile the method proposed by the authors appears to be a valid approach, I remain unconvinced by the claimed advantages based on the current experimental results and the rebuttal. Therefore, I would prefer to keep my score unchanged at this point.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALO1WTi607", "reviewer": "Reviewer_P59u", "summary": "This paper introduces ​Self-Organizing Prototypes (SOP), a non-parametric self-supervised learning (SSL) framework designed to address limitations in traditional prototypical SSL methods. Conventional approaches rely on static, learnable prototypes to represent clusters in unlabeled data, often leading to over-clustering and incomplete feature space coverage. SOP innovates by dynamically aggregating ​multiple support embeddings (SEs)—semantically similar representations stored in memory banks—to form enriched, adaptive prototypes. These SEs collectively capture complementary features within local regions of the feature space, enabling broader and more robust representations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s careful assessment and constructive feedback on our work. Your insights have helped us improving the manuscript. Below, we address each point in turn and provide additional visualizations and tables through anonymous links as requested.\n\n> [...] paper does not demonstrate that SOP achieves comparable performance with fewer prototypes [...]\n\nDue to space limitations, we kindly refer the reviewer to Table 10, and to our first response to reviewer KMMU, where we address a similar question.\n\n>  Besides, there is no t-SNE/UMAP visualization [...].\n\nAs suggested, we provide additional t-SNE visualizations of features using pre-trained SOP and iBOT encoders.\n- [tSNE Cifar10](https://shorturl.at/W9T2h)\n- [tSNE Cifar100](https://shorturl.at/R8H9q)\n\n> The FIFO memory might propagate noise [...] lack of explanation on how the model avoids collapse.\n\nWe agree with the reviewer on the importance of discussing collapse avoidance. We hypothesize that one key aspect contributing to the stability of our method is the built-in randomization of our SOPs. As quoted in line 67 and 82: “Our proposal organizes the feature space by exploring random local structures within the data”.\n\nWe agree that the FIFO updates might introduce biases where newer representations are located at one end of the memory while older ones are located at the other end. **However, the SOP's built-in randomization, which selects random anchors at each iteration, breaks this “recency correlation” in the FIFO memory, preventing the bias from affecting the loss function.**\n\nTo support our hypothesis, we conducted a [Collapse Analysis](https://shorturl.at/StDnj) comparing the effectiveness of our method when SOPs are randomly created at each iteration (default behavior) versus a version where SOPs are fixed after the initial random selection, similar to learnable prototypes. As shown, fixing the SOPs lead to collapse. \n\n>  [...] does not ablate the SOP-MIM [...] to isolate the contribution of the losses.\n\nWe kindly refer the reviewer to Section 4: Online vs. Non-Parametric Tokenizers and Table 7, where we ablate and isolate the contribution of each proposed loss function, as suggested. In summary, training with $L_{CLS}$ alone produces good representations and surpasses competirors in $k$-NN and linear probing. Similarly, training with the non-parametric $L_{MIM}$ alone improves over iBOT (MIM alone) by $+7.3\\%$ ($k$-NN).\n\n> [...] memory bank $E_C/E_p$ acts as a [...] non-differentiable parameter store. [...] a semantic distinction [..] does not explain how SOP’s design differs [...]\n\nWe respectfully offer a different perspective from the reviewer on this point. We believe that learning prototypes from scratch and optimizing a memory of non-parametric representations that self-organize, have important differences, which we highlight below:\n\n1- The optimization task does not involve gradients to update prototypes.\n\n2- In prototypical SSL, centroids are initialized at random locations and move in the space to find an optimal position. The SOP strategy optimizes random regions in the feature space using semantically related embeddings to represent their regions.\n\n3- Our method does not require additional terms in the loss function to avoid collapse, which is a critical component in current prototypical SSL.\n\nRegarding differences in optimization dynamics, we discuss how the proposed approach differs from prototypical SSL in terms of learning fixed prototypes vs. dynamic SOPs, lines 035, 082. About generalization, we discuss the issue of over-clustering that current methods employ and how it can hurt the learned representations (lines 052, 064, and Table 10).\n\n> [...] memory size [...] is not tested. [...]\n\nWe thank the reviewer for pointing out this flaw. Please refer to [Memory Size ablations](https://shorturl.at/xYg9a) where we ablated the sizes of the $\\texttt{[CLS]}$ memory $N_c$ and the size of the patch memory $N_p$. We observed a similar reversed “U” shape in both components. In general, increasing the memory size is beneficial up to a certain point, $N_c=65536$ and $N_p=8192$ for the class and patch memories, respectively.\n\n> [...] there is no analysis of prototype collapse [...].\n\nWe thank the reviewer for the thoughtful suggestion. However, in our setup, it doesn’t make sense to analyze the prototypes since our working memory is different from the traditional prototypes in existing methods. The memory that we use to compute the SOPs is ever changing due to its FIFO property. Thus, SOPs are dynamic (not fixed) data structures. They are randomly created at each iteration from random locations in the feature space. Hence, we don’t have access to them as we would have access to prototypes in regular prototypical SSL. Please refer to [Collapse Analysis](https://shorturl.at/StDnj) where we show that randomly creating SOPs is the key to avoid collapse.", "claims_and_evidence": "The claims in the submission are ​partially supported by empirical evidence but lack rigorous validation for several critical assertions.\n1. \"SOP alleviates over-clustering in prototypical SSL\". The paper does not demonstrate that SOP achieves comparable/better performance with ​fewer prototypes than baselines (e.g., iBOT). Over-clustering is defined as using K≫C, but SOP’s default K=4,096 anchors are not justified as \"non-overclustered.\" Besides, there is no t-SNE/UMAP visualization or entropy metric to quantify whether SOP prototypes cover the feature space more comprehensively than parametric prototypes.\n2. \"SOP is stable and requires no explicit regularization (e.g., Sinkhorn-Knopp).\" The FIFO-based memory might propagate noise during early training stages, but no experiments address this. Besides, there is a lack of explanation on how the model can avoid training collapse.\n3. \"SOP-MIM improves local feature learning for dense tasks.\" The paper does not ablate the SOP-MIM component to isolate its contribution. For example, training with only $L_{[CLS]}$ versus full $L_{SOP}$.\n4. \"Non-parametric SSL is feasible without learnable prototypes.\" While SOP avoids explicit prototype parameters, the memory bank $E_C$/$E_P$, effectively acts as a dynamic, non-differentiable parameter store. This is a semantic distinction rather than a fundamental shift. Besides, this paper does not explain how SOP’s non-parametric design differs from parametric methods in terms of optimization dynamics or generalization.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "1. The impact of memory size ($N_C$ =65,536, $N_p$=8,192) is not tested. Larger/smaller memory banks could reveal performance trade-offs.\n2. The paper uses K=4,096 anchors for SOP—similar to iBOT’s prototype count (K=1,024–10,000). No experiments show SOP achieving comparable performance with fewer prototypes.\n3. While SOP claims to avoid collapse without regularization (e.g., Sinkhorn-Knopp), there is no analysis of prototype collapse (e.g., percentage of \"dead\" prototypes) or comparison to parametric methods.", "relation_to_broader_scientific_literature": "The paper bridges non-parametric contrastive learning (e.g., MoCo, NNCLR) and prototypical SSL (e.g., SwAV, iBOT) by introducing dynamic self-organizing prototypes (SOPs) constructed from memory-based support embeddings, extending masked image modeling (MIM) to a non-parametric regime while addressing over-clustering and scalability limitations of prior prototype-based methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary

The paper introduces Self-Organizing Visual Prototypes (SOP), a non-parametric self-supervised learning (SSL) technique that improves the feature set of prototypes by using multiple semantically similar support embeddings (SEs) to represent regions in the feature space. This approach avoids over-clustering and mode collapse, resulting in stable and transferable representations. The method is evaluated on standard self-supervised benchmarks and shows favorable performance across various tasks. The authors also propose a non-parametric design for masked image modeling loss, intended for joint use during training.

## Strengths
- Introduces Self-Organizing Prototypes (SOPs) replacing fixed prototypes with dynamically constructed support embeddings.
- Challenges traditional assumptions in prototypical SSL, enhancing feature diversity.
- Proposes SOP-MIM, an extension of MIM using non-parametric patch-level SEs.
- Conducts thorough experiments on various downstream tasks.

## Weaknesses
 -Figure 2 is challenging to connect with pseudo code and lacks illustrations for better readability.
- SOP-MIM is an incremental work of MIM.
- The current method performs best when prototypes are larger than actual categories.
- No discussion on trade-offs between non-parametric and parametric methods.
- Experimental results show marginal improvements over previous baselines.

## Conclusions
Based on the reviews, the author feedback and a short postrebuttal discussion, the paper has been improved during the rebuttal period addressing some concerns of the reviewers and could be accepted in ICML if there is room in the program. Mainly, experiments  are very interesting showing interesting results although they are difficult to visualize the most outperforming. In addtion, there is no clear difference between equation 1 and 2, only theta deletion from one equation to the other. Therefore, the paper should included all the comments from the reviewers if the paper is finally accepted.
Decision: Accept (poster)