Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
Authors: Huaiyuan Qin
Muli Yang
Siyuan Hu
Peng Hu
Yu Zhang
Chen Gong
Hongyuan Zhu
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects or semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, we demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce effectiveness, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover’s Distance (EMD) as an estimator to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate our findings across a range of settings, highlighting their robustness and applicability on diverse data sources.

Published in Transactions on Machine Learning Research (09/2025)
Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
Huaiyuan Qin
Sinhy@@@r.a-star..dee
Institute for Infocomm Research (PR), A SSARR, Singopore
Muli Yang
gongml@@h2r.a-ttrr.mm edw.aarrredd.....
Institute for Infocomm Research (FR), SSARR, Singapore
Siyuan Hu National University of Singapore, Singopore
siyuanhu@nus.cdu.s
Peng Hu
penghu.ml@gmail.com
Sichuan University, China
Yu Zhang
edu.co
Southeast University, China
Chen Gong
chen.gong@ssjtu.etu.cr
Shanghai Jiaotong University, China
Hongyuan Zhu"
zhuh@i2r. 1ee.00i2.assstoe star edu.sg
Institute for Infocomm Research (FR), A A*SAR,, Singopore
Reviewed on OpenReview: https: // openreview. net/ per/formm? id= 4NCC3YYAO
Abstract
Self-supervieed learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce offectivenoss, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover's Distance (EMD) an estimator measure mutual information between views, finding that moderate FMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate findings across range of settings, highlighting their robustness and applicability on diverse data sources.
1 Introduction
Humans can effortlessly recognize objects across different viewpoints and contexts. A cat lounging on a couch remains a cat, whether seen from the side or above. This identity-inearrinn consistency has inspired the design of self-supervised learning (SSL) methods in computer vision, which leverage cross-view consistency as a supervision signal (Jing Tian, 2020; He al., 2020; Chen al., 2020b; 2021; Grill et al., 2020; Caron
* "Corresponding author.

---

Published in Transactions on Machine Learning Research (09/2025)
(a) Random Crops on Iconic Data
(b) Random Crops on Non-iconic Data
Figure 1: Visualization of random crops on iconic data (e.g. ImageNet (Deng et al., 2009)) and non-iconic data (e.g. COCO (Lin et al., 2014)). For iconic data, different views of the same image maintain instance consistency. However, for non-iconic data, different views may capture entirely different object instances, leading to the breakdown of such consistency.
et al., 2020; 2021; Oquab et al., 2023; Siméoni et al., 2025). SST. has emerged as an effective approach for learning visual representations from unlabcled data by aligning different views of the same image, relying on the instance consistency paradigm (Wu et al., 2018), which considers each image as a separate class. In this paradigm, different augmentations of an image, such cropping, rotation, or color jittering, are treated as positive pairs, while augmentations from other images serve as negatives (He et al., 2020; Chen et al., 2020a). The goal is to learn representations that capture essential semantic information from common instance while discarding irrelevant variations.
This instance consistency paradigm works remarkably well for iconic datascts, where images typically feature a single, dominant object, ensuring that different views naturally share the same semantic content of the common instance. However, training on such iconic datasets like ImageNet. (Deng et al., 2009) poses challenges in scalability, due to the requirement of intensive data collection and cleaning. In contrast, non-iconic datasets, such as COCO (Lin et al., 2014) and OpenImages (Kuznetsova et al., 2020), are casier to collect but introduce a fundamental challenge: these non-iconic datasets often feature complex scenes with multiple objects and diverse backgrounds (Van Gansbeke et al., 2021; Selvaraju et al., 2021; Zhu et al., 2023; Chuang et al., 2022; Stegmüller et al., 2023; Mishra et al., 2022), leading to the facts that two augmented views from the same image may not guarantee to contain the same object or share consistent semantic information, as illustrated in Figure 1. Surprisingly, despite the breakdown of instance consistency on non-iconic data, SSL methods can still achieve competitive performance, reported in Van Gansbeke et al. (2021), Mishra et al. (2022) and Zhu et al. (2023). This challenges the conventional assumption that positive pairs must always share instance semantics, raising a critical research question: Is instance consistency a strict requirement for sclf-supervised Icarning?
To address this, we investigate the following two key aspects: (1) How does SSL perform under different levels of instance consistency? We systematically evaluate whether SSL can still function effectively when positive pairs contain minimal shared instance semantics. Our configurations range from overlapping views with shared instance and background patterns, to entirely non-overlapping views with limited shared foreground content. We also explore configurations where only background information is shared or one view contains foreground while the other contains only background. Surprisingly, our results reveal that strict instance consistency is less essential in SSL than previously assumed. To further explore this observation, we study: (2) How much diversity between positive pairs is beneficial, and when does it become detrimental? We observe that incrcasing the diversity between positive pairs, such as enforcing zero overlapping or using smaller crop scales, could encourage the model to discover more fine-grained visual consistencies, particularly benefiting classification and dense prediction tasks. However, excessively increasing the diversity in between can hinder the effectiveness, leading to performance drop in downstream evaluations. This suggests that an optimal range of the shared information exists, where the balance between the consistency and diversity in positive pairs plays a significant role for effective SSL.
To quantify this balance, we adopt Earth Mover's Distance (EMD) as metric to mcasure the view diversity. Our analysis reveals that moderate EMD values correlate with improved SSL performance, providing a useful estimator for guiding positive pair selection in future SSL framework design. Finally, we validate our findings
2

---

Published in Transactions on Machine Learning Research (09/2025)
above across range of settings, including multiple SSL methods, various training datasets, and broad range of downstream evaluation tasks, demonstrating the practical applicability of our insights for effective SSL across diverse application scenarios.
In short, our main contributions are as follows:
1. Revisiting the Necessity of Instance Consistency: We empirically demonstrate that strict instance consistency is not essential for effective SSL, as models can leverage broader contextual cues even when positive pairs contain minimal shared instance semantics. Meanwhile, we show that increasing diversity between positive pairs can encourage the discovery of fine-graincn visual consistencies, enhaneing SSL's effectiveness. However, excessive diversity can hinder learning, suggesting the existence of optimal range for view diversity.
Earth Mover's Distance as an Estimator for Optimal View Diversity: We adopt Earth Mover's Distance (EMD) as an estimator to quantify mutual information between positive pairs, finding it to be predictive measure of view diversity for effective SSL.
3. Validation Across Diverse Methods, Datascts and Tasks: We validate our findings across diverse settings, while also demonstrating the robustness and generalizability of our proposed EMD- based diversity estimator on various data sources.
2 Related Work
Self-supervised Learning. Self-supervised learning (SSL) has emerged as a powerful technique for learning rich visual representations from unlabeled data (Jing & Tian, 2020), which have demonstrated impressive performance improvements across various downstream tasks (Caron et al., 2020; Grill et al., 2020; Oquab et al., 2023; Darcet et al., 2023; Henaff, 2020; Luo et al., 2023; Li et al., 2020; Ma et al., 2022; Bardes et al., 2021; Chen et al., 2020a; 2023a;b; Siméoni et al., 2025).
Early SSL efforts primarily focused on pretext tasks that leverage image-level or spatial-level structures to create supervisory signals. These include jigsaw puzzle solving (Noroozi & Favaro, 2016), where a network learns to rearrange shuflled patches into the correct spatial configuration; RotNet (Gidaris et al., 2018), which trains models to predict the rotation angle applied to an image; and video-based SSL methods (Jayaraman & Grauman, 2016), which exploit temporal coherence to learn feature representations from adjacent video frames. While foundational, these methods are often limited in scalability and generalization.
Recent SSL methods can be broadly categorized into three main groups: (1) Contrastive learning based SSLs (Chen et al., 2020a;b; 2021), are designed to optimize representations by maximizing the similarity between augmented views of the same image while minimizing similarity with other images. These methods typically rely on the assumption that each image represents single object- centrtc entity, making them well-suited for iconic datascts like ImageNet (Deng et al., 2009). (2) Self-distillation based SSLs (Grill et al., 2020; Caron et al., 2021; Oquab et al., 2023; Caron et al., 2020; Siméoni et al., 2025), remove the need for explicit negative pairs by focusing on consistency between teacher and student networks. These approaches also need to learn by aligning representations across augmented views, making them effective in object-centric contexts but challenging to extend to complex, multi-object scenes. (3) Reconstruction-
SSLs (He et al., 2022; Xie et al., 2022b), aim to reconstruct masked parts of the image, leveraging spatial context to learn representations. These techniques inherently focus on local structures, making them suitable for tasks requiring spatial feature prescrvation. Both contrastivo-based based and distillation-basel SSLs primarily rely on the instance consistency (Wu et al., 2018), where each image is treated as separate class.
SSL on Non-iconic Data. Recent work has explored extending SSL to non-iconic datascts (Lin et al., 2014; Kuznetsova et al., 2020), which pose unique challenges due to complex scenes with multiple objects. Approaches like Zhao et al. (2021), Liu et al. (2020), Stegmüller et al. (2023), Chen et al. (2023b) and Wang et al. (2021) attempt to align dense features across views within a single image. However, these methods often require on manual feature- or image-level matching, which is challenging in complex scenes. Other tochniques handle non-iconic data by pre-processing images into object-centric patches, using supervised (Mishra et. al.,
3

---

Published in Transactions on Machine Learning Research (09/2025)
2022; Selvaraju et al., 2021) unsupervised (Zhu et al., 2023; Peng et al., 2022) object discovery methods, allowing traditional SST, frameworks to operate effectively on these pseunl-ddjjcct-cmnric images. Alternative methods have introduced new loss functions designed to handle noisy or inconsistent semantics in varied views (Chuang et al., 2022). Mcanwhile, studics on SSL with natural images (Goyal et al., 2021; 2022) suggest that random cropping remains broadly effective, with Van Gansbeke et al. (2021) offering empirical evidence for its applicability. In parallel, Purushwalkam Gupta (2020) conduct an early investigation into SSL invariances and dataset biases, finding that models trained on non-iconic data perform worse than those trained on iconic data. However, deeper analysis of view consistency and diversity remains under-explored, which is essential to fully leverage SSL's potential.
Data Augmentation in SSL. Data augmentation plays a fundamental role in SSL, providing the super- vision signal to learn meaningful representations by capturing invariance across augmentations. Experiments in SimCLR (Chen et al., 2020a) have established a detailed ablation studics on augmentation strategics, concluding that applying diverse transformations to positive pairs improves representation learning. These findings have since become the standard practice in modern SSL methods (He et al., 2020; Caron et al., 2021; 2020; Grill et al., 2020), which typically employ loss function that pushes together representations of augmented views. More recently, Morningstar et al. (2024) propose a unified SSL framework, showing that augmentation diversity plays a critical role in the success of recent SST, methods. Our work investigates how augmentation-inte induuied view diversity affects SSL when positive pairs contain minimal shared semantics.
Earth Mover's Distance. Earth Mover's Distance (EMD) is widely used in computer vision as a metric to quantify structural similarity between distributions. Initially applied in tasks such as color and texture-based image retrieval (Rubner et al., 2000) and visual tracking (Schulter et al., 2017; Zhao et al., 2008; Li, 2013), EMD has demonstrated effectiveness in capturing relationships between complex structural palterns. More recently, EMD has been employed to few-shot classification tasks (Zhang et al., 2020; Xie et al., 2022a) to measure structural distances between image representations. In the context of SSL,, Sclf-EMD (Liu et al., 2020) utilizes EMD to align dense feature embeddings in non-iconic datasets such as COCO, preserving spatial structure in feature maps to improve object detection. Unlike prior work that applies EMD in settings with rich supervised semantic information, our study introduces EMD in fully self-supervised setting, using it to estimate the view diversity, thereby providing insights for future SSL design.
3 Preliminary
In this section, we provide an overview of MoCo-v2 (Chen et al., 2020b) and DINO (Caron et al., 2021), 1, two widely used SSL frameworks that serve as the foundation of our study. Our work specifically focuses on SSLs with instance consistency (Wu et al., 2018), where cach image is treated as a separate class. Both MoCo-v2 and DINO implicitly rely on this assumption, which we seek to extend to more diverse data sources to investigate the necessity of instance consistency in SSL.
MoCo-v2 (Chen et al., 2020b) employs memory bank to store large number of negative samples, ensur- ing smooth updates with momentum for better consistency. It learns feature representations using the InfoNCE (Oord et al., 2018) loss:
exp (q k+/T)
by - llg exp (g - -k//))//)) + Ev exp (q kg-k=/
where T is the temperature, is the encoded query, k4 is the positive key, and k_ represents the negative koys. Note that q and 44 are two augmented views from the same image.
DINO (Caron et al., 2021) uses a teacher-student self-distillation framework, where the model learns categorical distributions from the [CLS] token of two augmented views from the same image. The teacher 0, and the student 82 share the same architecture, and the teacher parameters are updated with the Exponential Moving Average (EMA) of the student parameters. The knowledge is distilled from teacher 0, to student 0, by minimizing the cross-cntropy loss:
Liass] - -P|CLS] (v)T log p[CLS)] (u):
(2)
4

---

Published in Transactions on Machine Learning Research (09/2025)
Viewl
View2
Larger Crop
Scale Diversity
GT Inst.
Image from COCO
Smaller Crop
Random Crop
Zero Spatial Ovlp.
Inst. vs Bg
Only Bg
Smaller Crop!
Instance Diversity
Figure Overview of positive pairs from different configurations. This figure illustrates various positive pair configs proposed in our experiments, categorized into Instance Diversity and Scale Diversity. The Instance Diversity category varies the level of instance consistency between positive pairs to investigate its necessity, while the Scale Diversity category varies crop scales to evaluate the impact of diversity between positive pairs.
where and two augmented views from the same image, and Po is the probability distribution of network 0.
4 Delving into Instance Consistency in SSL
In this section, we conduct a comprehensive ablation study to investigate the effectiveness of SSL when mnstince-onsissencyy is not guaranteed. By systematically adjusting crop configurations, we analyze whether SSL can still function effectively when positive pairs contain minimal shared instance semantics. Our findings reveal that the strict instance consistency plays less important. role in SST than previously assumed, while view diversity plays crucial role in enhancing SSL performance. To quantify this balance, we then introduce Earth Mover's Distance (EMD) an estimator for quantifying diversity between auginented views, demonstrating its alignment with experimental results and its potential as predictive measure for optimizing future SST. augmentation design. Finally, we validate our findings across diverse settings to evaluate the robustness and generalizability of SSL on a wider range of data sources. Detailed descriptions of the pre-training and rewmstteem fine-tuning datasels, along with the experimental selups, are provided in Appendix A.
4.1 How Does SSL React to Different Levels of Instance Consistency?
Traditional SSL methods (Chen et al., 2020b; Caron et al., 2021; Chen et al., 2020a) are often under the instance consistency paradigm to treat each image as a separate class (Wu et al., 2018) and are pre-trained on iconic data for ensuring invariant semantic features are shared across different views of the same object. However, for non-iconic data, with complex scenes containing multiple diverse objects and varied backgrounds, random crops of the same image may contain entirely different objects background elements, leading to the breakdown of such instance consistency, of pet expo might feature a cat in one view and a dog in the other. Given this, it remains unclear to what extent SSL performance is alfected by different levels of instance consistency. While SSL methods like MoCo-v2 (Chen et al., 2020b) and DINO (Caron et al., 2021) can achieve strong performance on non-iconic data (Van Gansbeke et al., 2021; Mishra et al., 2022; Zhu et al., 2023), is essential investigate whether this holds consistently across different levels of instance consistency. This leads to our first question: How does SSL perform under different levels of instance consistency?
illustrated in Figure 2. For example, two crops from an image
5

---

Published in Transactions on Machine Learning Research (09/2025)
COCO
ImageNet-
Config
CIFAR-10
CIFAR-100
DID
Pets
STL-10
CIFAR-10
CIPAR-100
DTD
Peta
STL-10
Baseline
70.90
47.03
38.40
71.85
39.20
43.85
Lower Round
32.72 -38-38
12.40
-30.8
7:29
-30.3
7.28
-30.12
27.59
-44.2
32.64
-m.32
12.22
6.65
-2241
7.00
-30.7
28.36
-40.00
Spatial Ovlp.
71.75 10.60
48.15
-1.43
41.65
18:26
39.53
1.11
74.42
19.56
71.72
13.77
51.95
1mm
16.01
10.m
46,70
19.45
76.69
11.04
Inst. us Bg
76.20 +8.00
51.77
41.12
+9.72
10.58)
-3.00
74.75
+3.01
75.71
+3.06
33,21
-4.07
12.77
+0.01
45.11
+1.00
77.38 +3.07
Only Bg
72.13
49:47
41.91
39:20
73.91
73.73
50.62
43.40
45.03
76.56
Larger Crop
67.02
43.64
33.24
29.05
60.00
06.72
34.63
36.33
72.94
Smaller Crop
71.36 +9.60
48.28
-1.25
41.76
+3.30
39.81
-1.41
73.69
+4.86
74.97
-3.43
51.50
+5.05
44.63
+0.59
45.90
+2.05
76.92
+1.01
Smaller Crop!
70.34 -0.00
47.26
10.20
40.43
+2.13
36.44
-1.00
72.26
-0.12
67.72
-1.1
50.21
+1.07
40.96
+2.70
40.37
-2.0
75.65
+0.34
Table 1: Classification results with MoCo-v2 (Chen et al., 2020b) pre-trained on COCO (Lin et al., 2014) and ImageNet-100 (Deng et al., 2009). We freeze the pre-trained weights of the SSL backbone and train a supervised linear classifier to evaluate the learned representations on five classification benchmarks (Krizhevsky et al., 2009a;b; Cimpoi et al., 2014; Parkhi et al., 2012; Coates et al., 2011). All configurations are pre-trained and linear fine-tuned for 100 epochs to ensure fair comparison. Performance gaps relative to the baseline configuration are indicated as superscripts. Smaller Crop denotes to Smaller Crop with Zero Spatial Overlap configuration.
To probe further, we empirically conduct a series of ablation experiments with MoCo-v2 and DINO pre-trained on various data sources, while systematically varying the shared instance semantics between positive pairs. We then evaluate the learned representations on classification and dense prediction tasks to assess the necessity of instance consistency on SSL performance.
Experiment Setup. We conduct controlled experiments to analyze the impact of different levels of instance consistency between the positive pair (V1, V2), obtained from augmented views?. Given target instances with ground-truth bounding boxes denoted as U -== box; we introduce the following five primary configuratioss::
1) Completely Random Crop. Two views V1 and V2 are randomly cropped from the same image without any spatial constraint. This configuration replicates the default setup in multi-view SSL methods (i.e. MoCo-v2 and DINO), which serves as the Baseline of the comparison.
2) Zero Spatial Overlap. Views are sampled with no spatial overlap to test the reliance of SSLs on the instance consistency from the shared spatial regions.
ToU(v1, v2)= =0
3) Instance us Bg. To further reduce the instance consistency in positive pairs in the former config (two views may be partially cropped one same instance), we sample V1 around foreground instance, while V2 contains only background information, ensuring no overlap with any instance object. Here Bg. refers to Background for simplicity.
LoU v2) =
(i ToU(v1, box,) > 0.8) A (Vj, ToU(v2, box,) < 0.1)
4) Only Bg. To completely remove possible instance consistency, two views are randomly sampled purely from background regions, ensuring no foreground instances are included.
loU(vi,v2) =0
Wi, LoU(Vi box.) 0.1 A IoU( box.) 0.1
5) Lower Bound. Each view is sampled from entirely different: images, minimizing any possible consistency within positive pairs. This configuration serves as the lower-bound comparison to evaluate how SSL performs when no mutual information exists within positive pairs.
Detailed descriptions on the implementations of these configs are provided in Appendix A.1.
We follow the settings from Chen al. (2020a) to use the RandonResizedCrop in PyTorch with the scaling (0.2, and the output size of 224 x 224.
We collectively refer to these five configurations as the Instance Diversity category, as illustrated in Figure 2.

---

Published in Transactions on Machine Learning Research (09/2025)
COCO
ImageNee--100
Config
VOC-0712
DOTA-v1.0
VOC-0712
DOTA-v1.0
random init.
53.58
31.59
53.38
31.59
Lower Bound
70.54 -2.78
47.94
-8.44
69.97
-3.94
48.96
-0.34
Baseline
73.32
54.38
73.91
53.30
Spatial Ovlp.
74.55
55.84
74.87
56.23
Inst. us Bg
74.15 +0.83
55.47
+1.09
74.35 +0.44
56,65
Only Bg
74.73
55.34
74.43
56,65
Larger Crop
72.14 -1.18
52.09
-2.29
73.40
-0.51
54.06
11.24
Smaller Crop
74.58
+11.6
55.52
+11.4
74.49
+0.68
56.26
+0.00
Smaller Crop!
73.90
54.68
74.09
55.63
Table Object detection results with MoCo-v2 (Chen et al., 2020b) pre-trained on COCO (Lin et al., 2014) and ImageNet-100 (Deng et al., 2009). We evaluate the learned representations on VOC (Everingham et al,, 2010) and DOTA (Xia et al., 2018) for object detection. All configs pre-trained for 100 epochs for fair comparison. Random Init. refers to the backbone being randomly initialized during downstream fine-tuning.
Results. Table presents the performance the proposed configurations on classification tasks (Krizhevsky et al., 2009a;b; Coates et al., 2011; Parkhi al., 2012; Cimpoi al., 2014), while Table presents results on object detection tasks (Everingham et al., 2010; Xia et al., 2018). As expected, the Lower Bownd configuration yields the lowest performance, highlighting the importance of shared information between positive pairs for effecive SSL. Surprisingly, notable finding is that all other configurations (Zero Spatial Overlap, Instance vs Bg, and Only Bg) outperform the baseline configuration across classification and object detection evaluations, indicating that SSL can still effectively learn representations without strict instance consistency, even surpassing the default settings.
Discussion. In contrast to prior beliefs (Selvaraju et al., 2021; Chuang et al., 2022), which suggests SSL should rely heavily on object-centric iconic data with strong consistent semantics in positive pairs, our findings reveal that strict instance consistency is not essential. Existing SSL methods still learn meaningful representations when positive pairs are in the absence of strict instance consistency, as long as both views are sampled from the same image. This suggests that SSLs are capable of leveraging broader contextual cues beyond instance consistency, including shared background patterns, consistent camera viewpoints, and general color style, aligning with observations Van Gansbeke el. al. (2021). These findings highlight the potential of SSL on non-iconic data, expanding the range of a wider applicable data sources.
Remark. The above controlled experiments aim investigate the necessity of the instance consistency assumption, especially when pre-training on non-iconic data, where such consistency may not naturally hold. To this end, the designed ground-truth-based view generation config is solely intended for controlled experimental setups, allowing us to systematically assess how varying levels of instance consistency between positive pairs affect SSL performance. These configs are not designed to serve as data augmentation pipelines for practical SSL pre-training use.
Additionally, certain configs, like Zero Spatial Overlap, may not fully eliminate instance consistency, non-overlapping regions of large object instance may still share the same semantic information. However, we would like to emphasize that these configs only serve to stepwise reduce the degree of semantic consistency, achieving by firstly removing the spatial shared overlapping content between augmented views. In practice, it offers a controlled setup to study the impact of different levels of instance consistency on SSL performance.
Takeaway
In contrast to prior beliefs of instance consistency (Selvaraju et al., 2021; Chuang al., 2022), our experiments empirically show that SSL can learn meaningful representations even when positive pairs contain minimal shared instance semantics.
7

---

Published in Transactions on Machine Learning Research (09/2025)
4.2 How Much View Diversity is Beneficial?
As explored in Section 4.1, instance consistency appears to be less critical for effective SSL. Meanwhile, while our configurations reduce the instance consistency, they simultaneously increase diversity and reduce rodundancy between positive pairs, yct still serve as a valid, and even better supervision signal. This raises a new question: How much diversity between positive pairs is beneficial, and when does become detrimental? This aligns with findings from Tian et al. (2020), which suggest that higher diversity between views can enhance SSL performance.
To further validate this hypothesis, conduct a set of orthogonal experiments focusing on the diversity between positive pairs, specifically by varying crop scales. According to the Law of Large Numbers, smaller crop scales naturally reduce the likelihood of overlapping regions between views, while larger crops increase spatial redundancy. Additionally, smaller crops inherently capture less information per view, potentially increasing the diversity within positive pairs. The following experiments evaluate how different levels of view diversity impact SSL representation learning.
Experiment Setup. We introduce three primary configurations', which complement the previous configs in Section 4.1 to regulate the diversity between positive pairs by systematically varying crop scales.
1) Smaller Crop. This configuration applics a smaller crop scale to reduce the area captured by each view. The smaller region minimizes shared information increase diversity between positive pairs.
2) Larger Crop. Larger crop scales increase the area captured by each view, creating larger overlap to preserve more shared information, thereby reducing the diversity.
3) Smaller Crop with Zero Spatial Overlap. To maximize the diversity, this configuration combines the smaller crop with the zero spatial overlap constraint, ensuring no overlapping spatial overlapping within positive pairs. This setting enforces the lowest shared information, allowing us to evaluate SSL'' ability to learn from highly diverse positive pairs.
All configurations maintain a consistent output size of 224 x 224, aligning with the settings in Section 4.1. Detailed descriptions and ablation studies on the selection of crop scales are provided in Appendix A.1.
Results. Table 1 presents the classification results of varying crop scales, and Table 2 presents the detection results. The findings confirm our hypothesis that increasing diversity between positive pairs in SSL can enhance downstream performance: configurations with higher diversity (i.e. Smaller Crop) consistently outperform the baseline. Conversely, the Larger Crop configuration, which reduces diversity by preserving more shared information, leads to a significant performance drop, suggesting that excessive redundancy between views can hinder SSL effectiveness. Interestingly, while Smaller Crop and Zero Spatial Overlap individually boost performance, their combination, Smaller Crop with Zero Spatial Overlap, does not yield additional gains and instead results in a slight performance drop.
Discussion. These findings highlight the importance of striking balance in view diversity for effective SSL. Increasing view diversity through configurations like, Smaller Crop and Zero Spatial Overlap, effectively reduces mutual information, encouraging the model to discover more fine-grained visual consistencies, thus enhancing SSL performance. However, the observed performance drop when combining these two configs suggests that excessive diversity can be detrimental, as minimizing shared information beyond a certain threshold hinders the model's ability to learn meaningful representations.
This aligns with prior study (Tian et al., 2020) that descríbes a U-shaped relationship between mutual information and downstream performance indicating that while reducing redundancy is beneficial, completely eliminating shared information can degrade SSL's effectiveness. These insights imply the existence of an optimal range for view diversity, while mutual information is minimized yet remains sufficient for effective SSL learning. This highlights the need for a quantitative estimator to evaluate and balance shared information between views, guiding the augnentation process loward optimal performance, which is to be elaborated in Section 4.3.
We collectively refer to these three configurations as the Scale Diversity category, as illustrated Figure
8

---

Published in Transactions on Machine Learning Research (09/2025)
Takeaway
Our findings empirically show that increasing the diversity positive pairs encourages the discovery of more fine-grained visual consistencies SSL, thus enhancing downstream performance. However, excessive diversity may degrade SSL's effectiveness.
4.3 Earth Mover's Distance Diversity Estimator
Experiments in Sections 4.1 and 4.2 show that view diversity between positive pairs plays crucial role in SSL performance. To quantify this diversity and give an estimation of the effectiveness of different view augmentations, we adopt Earth Mover's Distance (EMD) as an estimator to measure the shared information between positive pairs. By evaluating the similarity between augmented views within positive pairs, EMD provides a robust estimation of augmentation quality before model pre-training. This enables a systematic approach to optimize the positive pair selection for improving SSL effectiveness.
Background. To accurately measure the distance or the similarity between two views, we require metric that account for spatial variations in the possible data sources of SSL. To accommodate non-iconic data containing, where multiple objects or complex scene environments appear across different views, the simple L2 distance metric is unsuitable due to its reliance on strict spatial aligument. Furthermore, previous experiments Section 4.1 show that SSL can perform effective feature learning without strict instance consistency, suggesting that the used view similarity should not simply focus on the pixel or the image level, which needs to further explore in the feature space. Therefore, we adopt Earth Mover's Distance (EMD) to automatically identify correspondences between views based on their visual features.
Earth Mover's Distance (Rubner et al., 2000; Zhang et al., 2020) quantifies the distance between two distributions by computing the minimum cost needed to transform one distribution into another, making it a well-established formulation of the optimal transport problem (OTP). In our case, EMD measures the distance between the given feature maps of two augmented views X, RNNxD, where N denotes the number of feature vectors in each feature map and D represents the feature dimension. More details on the definition and computation of EMD are provided in Appendix A.3.
Implementations. As shown in Figure 3, to com- pute the EMD-based similarity score, we follow the settings in Zhang et al. (2020) to employ two strate- gies for generating feature vectors from two views in the positive pair. In both strategies, we first gener- ate two augmented views of each image and extract their features using pre-trained ResNet-50 (He et al., 2016). To account for potential scale differences be- tween augmented views, each strategy uses distinct cropping patterns:
(I) Grid-based EMD
1) Grid-based: Each view is divided into uniform grids, with grid factors of and 3. Each grid cell serves as separate patch, which is then passed through pre-trained model to generate the feature vector.
(2) Sumpling-based EMD
2) Sampling- based:: Each view is randomly sam pled into 9 patches, varying the sizes and as- pect ratios to introduce scale diversity. Each sampled patch is resized with the input size of 84 before being processed by the pre-trained model to produce its corresponding feature vector.
Figure 3: Two strategies for EMD-based simi- larity score. Different line widths between patches represent the mass transported under the OT plan used in EMD computation. Thicker lines correspond to higher transport mass, indicating stronger corre- spondence between the two patches.
9

---

Published in Transactions on Machine Learning Research (09/2025)
Results. To validate the effectiveness of Earth Mover's Distance (EMD) estimator for assessing similarity between augmented views, we compute the EMD similarity score for all the proposed configurations in Sections 4.1 and 4.2. Figure 4 reveals a clear reverse-U U relationship between the EMD score and downstream accuracy, evaluated using the two proposed cropping strategics. Configurations with moderate EMD scores (ranging from 3 to 4) consistently yield the highest performance. This suggests that when the diversity between positive pairs within an optimal range, the shared mutual information between views remains sufficient for effective feature learning, leading to improved downstream performance. In contrast, configurations with either very high (above or very low EMD scores (below 2) exhibit drop in downstream accuracy, indicating that extreme overlap or excessive diversity between views can hinder SSL's ability to learn meaningful feature representations. Plots for SSL pre-trained on ImageNet (Deng et al., 2009) are provided in Appendix B.5.
Discussion. Our analysis demonstrates that Earth Mover's Distance (EMD) is an effective estimator of view diversity SSL, providing an approach to quantify the mutual information between augmented views, By leveraging EMD, we can evaluate and regulate view diversity ensure remains within an optimal range for effective SSL training. This insight suggests that EMD can serve as valuable measure for guiding augmentation strategics in future SSL framework design, allowing the prediction of the effectiveness of positive pair selection to enhance reanstream performance.
Suggestions for Positive Pair Sclection. Our experimental results suggest potential improvement for posilive pair selection in SSL: pre-calculating EMD scores before pre-training. Specifically, our findings indicate that the optimal EMD range lies between the baseline score (upper bound) and that Smaller Crop' (lower bound), where the latter can also represent positive pairs with excessive diversity. This insight offers a promising alternative the conventional random cropping approach in SSL. Our experiment results with Zero Spatial Overlap and Smaller Crop configurations also support the effectiveness of this strategy.
Takeaway 3
Our findings reveal that Earth Mover's Distance (EMD) effectively quantifies mutual information between positive pairs, making a predictive mcasure of view divorsity for effective SSL.
4.4 Validation Across Diverse Settings
We have demonstrated the generalizability of our findings on both iconic and non-iconic data, validating their effectiveness on both classification and object detection tasks. Building on these results, we provide additional evaluations to further assess the robustness of our insights across diverse application scenarios.
Diverse SSL Methods. To ensure our findings are not limited to contrastive learning, we further expand our analysis to the DINO (Caron et al., 2021) framework shown in Figures 4c and 4d. These results demon- strate that our insights apply beyond contrastive-based methods, generalizing to broader range of instance consistency-basel SSL approaches. Numerical results and further analysis are provided in Appendix B.1.
Diverse Tasks & Experimental Settings. We further validate our findings by extending experiments lo more downstream tasks, including instance segmentation and depth prediction, assessing the generalizability of our insights across different learning objectives. Additionally, we examine various experimental scttings, such as
tuning, extensive dataset transfer scenarios, and extended pre-training durations, comprehensively evaluate our findings under diverse training conditions. Detailed results and analysis are provided in Appendices B.2 and B.3.
Takeaway
Our validation experiments confirm the generalizability of our findings across range of settings, while highlighting the adaptability of the proposed EMD diversity estimator on various data sources.
4EMD in Figure are scaled by 10 for better visualization.
10

---

Published in Transactions on Machine Learning Research (09/2025)
56
75
56-
74-
001A mAk (16)
74
DOTA mAP (%)
73
72
VDC
72
90
VDC
49
DOTA
1
VOC
48
DOTA
2
3
similarity
Gri-baaed LMD similanity
Sampling similarity
Sampling-asee EMD similarity
(a) EMD similarity versus detection accuracy with MoCo-v2 (Chen et al., 2020b) pre-trained on COCO.
c990.20 700
63-
53
+
DTAR-2
43765-100
Pets
50
ST-10
0nd based 3M simiarry
Ord based similrrt
Giddl BMD simslrity
O:dd b3se SMOsimbrriy
C19813 To PM
2200
Fola Aoc
23-
12
3-
DPAR-32
C1901-200
5T-30
33
Sosplig-based EMO
EMD
Sampiighised EMD
Samziin-tooe EMD
Samsingttsse EMD sinianty
(b) EMD similarity versus classification accuracy with MoCo-v2 (Chen al., 2020b) pre-trained on COCO.
65
75
75
010 mAbe 186
70
DOTA mAP (86)
MAP
MAP
85
65
60-
50
60
50
55
55
VDC
Grid-bate hiid-bssed EMD similarity
Grid-based EMD simillaity
Sampling-baser EMD similarity
Sampling-assed EMD similanty
(c) EMD similarity versus detection accuracy with DINO (Caron al., 2021) pre-trained on COCO.
73-
31033395 Sec 794
CPA0120 B003
63-
52
CHAN.11
Grid
73-
02
22
Somoltg-oowd kinlbrtty
Semstig-hooed Kinleitty
Sensing-toee siiletty
Sunsing-twwe cillanty
(d) EMD similarity versus classification accuracy with DINO (Caron et al., 2021) pre-trained on COCO.
Figure& 4: EMD similarity versus detection and classification accuracy. The similarity scores between views are plotted against object detection results (a), (c) and classification results (b), (d). Baseline configuration is highlighted for reference. EMD scores are scaled by factor of 10 for better visualization. Across all settings, the results exhibit clear reverso-U curve, supporting the hypothesis that an optimal range of view diversity exists for effective SSL.
11

---

Published in Transactions on Machine Learning Research (09/2025)
5 Conclusion
In this paper, we investigate critical rescarch question: Is instance consistency strict requirement for self-supervised learning (SSL)? To explore this, we systematically analyze the effectiveness of SSL when instance consistency is not guaranteed. Our findings reveal that SSL can still learn meaningful representations even when positive pairs contain minimal shared instance semantics, suggesting that strict instance consistency is not essential for effective SSL Iearning. Furthermore, our analysis further reveals that increasing diversity between positive pairs, such enforcing zero overlapping or using smaller crop scales, enhance performance across various downstream tasks. However, excessive diversity is found to reduce effectiveness, indicating the existence of an optimal range for view diversity. To quantify this diversity, we adopt Earth Mover's Distance (EMD) as metric to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing a useful estimator for guiding positive pair selection in future SSL framework design. We validate our findings across a range of settings, highlighting the practical applicability of our insights for effective SSL across diverse application scenarios.
Broader Impacts. While our findings highlight the flexibility of SSL without strict instance consistency, particularly on non-icomic data, it may also lead to unreliable representations if applied in sensitive domains such healthcare autonomous driving without proper validation. Furthermore, training on large-scale, unfiltered Internet data may amplify socictal and demographic biases due to undre-eoprrinntatinn or skewed content. We therefore recommend domain-specific validation and bias auditing when applying our findings Lo real-world applications.
This research is supported by the EDB Space Technology Development Programme under Project $22-19016-STDP and the National Natural Science Foundation of China under Grant 62576089. The authors sincercly thank the Reviewers and Action Editor for their valuable and constructive feedback.
Acknowledgements.
12

---

Published in Transactions on Machine Learning Research (09/2025)
References
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianc--imaaraacccccvvaaaa regularization for self-supervised learning. arXiv preprint arXiv:2105.0 04906, 2021.
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaplive bins. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4009 -018,, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervisod learning of visual features by contrasting cluster assignments. In Adv. Newral Inform. Process. Syst., pp- 9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CV international conference computer vision, pp. (650-9660, 2021.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwci Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.071: 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and GeofTrey Hinton. simple framework for contrastive learning of visual representations. In ICML, pp. 1597 1607, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXi::2003.040 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVT international conference on computer vision, pp. (640-9649, 2021.
Yingyi Chen, Xi Shen, Yahui Lin, Qinghua Tao, and Johan AK Suykens. Jigsaw-vit: Learning jigsaw puzzles in vision transformer. Pallern Recognition Letlers, 166:53 60, 2023a.
Zihan Chen, Hongyuan Zhu, Hao Cheng, Siya Mi, Yu Zhang, and Xin Geng. Lpel: Localized prominence contrastive learning for self-supervised dense visual pre-training Pottern Recognition, 135:109185, 2023b.
Ching- Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. In IEEE Conf. Comput. Vis. Pullern Recog., pp. 16670-16681, 2022.
Mircea Cimpoi, Subhransu Maji, Tasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 3606 3613, 2014.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and bench- mark. https://github. cww/ /pen/ http://onmmmmammmmmggetttaaaaa
2020.
Marco Cuturi. Sinkhorn distances: Lightspeed compulation of optimal transport. Advances in newral information processing systems, 26, 2013.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint 16588 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In IEFE Conf. Comput. Vis. Pattern Recog., pp. 248 255. Teee, 2009.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. Learn. Represent. 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson; Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9588-95888 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. Inl. J. Compul. Vis., 88(2):303 338, 2010.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
2018.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of visual features in the wild. arXiv preprini xrx::221030009 0198888 2021.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision. arxiv preprint orXiv:2202.08360 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Adv. Neural Inform. Process. Syst., pp. 21271-21284, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFE Conj. Compul. Vis. Pottern Recog,, pp. 770-770077 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Int. Conf. Comput. Vis., pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conf. Comput. Vis. Pottern Recog., pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conf. Comput. Vis. Pottern Recog., pp. 16000-16009, June 2022.
Olivier Henaff. Data-cfficient image recognition with contrastive predictive coding. In ICML, pp. 4182-4192, 2020.
Dinesh Jayaraman and Kristen Grauman. Slow and steady feature analysis: higher order temporal coherence in video. In Proceedings of the TEFE Conference on Computer Vision and Poltern Recográtion, pp. 2016.
Longlong Jing and Yingli Tian. Sclf-supervisod visual feature learning with deep neural networks: A survey. IEEE Trans. Pattern Anol. Mach. Intell., 2020.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009a. URL httt:://ww..ee www.sooorotooo.oo edu/-kriz/cifar html.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). 2009b. URL httt:://ww.cce.m ca. toronto. www.oto..ooooeeeeetee www.:/tt..dde/eeii//ciiaaar.....
14

---

Published in Transactions on Machine Learning Research (09/2025)
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. Inl. J. Compul. Vis., 128(1):1956 1981, 2020.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervisod representations. ar preprint urXiv:2005.0400 2020.
Peihua Li. Tensor-sift based earth mover's distance for contour tracking. Journal of mathematical imaging and vision, 46:44-68 65, 2013.
Zhenyu Li. Donocular-Depthe Estimation-Tollooaeeeo 2022.
Monocular depth estimation toolbox.
httpp:///ggthuu..com///yyy/
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In Eur. Conf. Comput. Vis., volume 8693 of Lecture Notes in Computer Science, pp. 740 755. Springer, 2014.
Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv preprinl arXiv:2011.130 2020.
Zhenfei Luo, Yixiang Dong, Qinghua Zheng, Huan Liu, and Minnan Luo. Dual-channel graph contrastive learning for self-supervised graph-level representation learning. Paltern Recognition, 139:109448, 2023.
Xin Ma, Xiaoqiang Zhou, Huaibo Huang, Gengyun Jia, Zhenhua Chai, and Xiaolin Wei. Contrastive attention network with dense field ostimation for face completion. Pattern Recognition, 124:108465, 2022.
Shlok Kumar Mishra, Anshul Shah, Ankan Bansal, Janit K Aujaria, Abhyuday Narayan Jagannatha, Abhishek Sharma, David Jacobs, and Dilip Krishnan. Object-aware cropping for self-supervised learning. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openrewe// net//
forum?id=WXggJN77ABB
Warren Morningstar, Alex Bijamov, Chris Duvarney, Luke Friedman, Neha Kalibhat, Luyang Liu, Philip Mansfield, Renan Rojas-Gomez, Karan Singhal, Bradley Green, et al. Augmentations vs algorithms: What. works in self-supervised learning. arXiv preprint arXiv:2403.00 052660 2024.
Théo Moutakanni, Maxime Oquab, Mare Szafraniec, Maria Vakalopoulou, and Piotr Bojanowski. You don't need domain-specific data augmentations when scaling self-supervised learning. Advances in Neural Information Processing Systems, 37:116106- 116125,, 2024.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 3XXi::8077007//3 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szalraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0713 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IBEE Conf. Compul. Vis. Pallern Recog., pp. 3498 1108-,55,,44 2012.
Xiangyu Peng, Kai Wang, Zheng Zhu, Mang Wang, and Yang You. Crafting better contrastive views for siamese representation learning. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 16031-16040, 2022.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervisod learning: Invariances, augmentations and dataset biases. Advonces in Neural Information Processing Systems, 33:3407-3415 2020.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster CNN: Towards real-time object detection with region proposal networks. In Adv. Neural Inform. Process. Syst. pp. 91-99, 2015.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 40:99- 121, 2000.
Samuel Schulter, Paul Vernaza, Wongun Choi, and Manmohan Chandraker. Deep network flow for multi- object tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognátion, pp- 6951 6960, 2017.
Ramprasoath R. Sclvaraju, Karan Desai, Justin Johnson, and Nikhil Naík. Casting your model: Learning to localize improves self-supervised representations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 11058 11067, 2021.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pallern analysis and machine intelligence, 22(8):888 905, 2000.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Aasion-ECCA 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khali- dov, Mare Szafranice, Seungeun Yi, Michaöl Ramamonjisoa, et al. Dinov3. arXiv preprint Xrv:22508 10104, 2025.
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-35) -4443338,,,,, 1967.
Hari Sowrirajan, Jingbo Yang, Andrew Ng, and Pranav Rajpurkar. Moco pretraining improves representation and transferability of chest x-ray models. In Medical Imaging with Deep Learning, pp- 728-744. PMLR, 2021.
Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pottern Recognition, pp. 7000-7009, 2023.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Adv. Neural Inform. Process. Syst., pp. 6827-688 6839, 2020.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. Int. J. Comput. Vis., 104(2):154-171, 2013.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue V Gool. Revisiting contrastive methods for unsupervised learning of visual representations. In Adv. Neural Inform. Process. Syst., pp. 16238 16250, 2021.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 3024-3033, 2021.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CTI conference computer vision and pattern recognition, pp. 3124-3134 2023.
Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. Tn Adv. Newral Inform. Process. Syst., pp. 22682 22694, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non pan-aaarmmiic instance discrimination. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 3733 3742, 2018.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in acrial images. In Proceedings of the IEEE conference compuler vision and pollern recognilion, pp. 3974 0983,,93, 2018.
Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pattern Recognition (CVPR), pp. 7972-7981, June 2022a.
Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han. Oriented r-cnn for object detection. In Proceedings of the IEEE/CV /CVF international conference on computer vision, pp. 5520-3529, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653-9636699 2022b.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 12203 12213, 2020.
Qi Zhao, Zhi Yang, and Hai Tao. Differential earth mover's distance with its applications to visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(2):274 287, 2008.
Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. Self-supervised visual representations learning by contrastive mask prediction. In Int. Conf. Comput. Vis., pp. 10160-10169, 2021.
Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, al. Mmrotate: rotated object detection benchmark using pytorch. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 7331-7334, 2022.
Ke Zhu, He, and Jianxin Wu. Coarse is better? a new pipeline towards self-supervised learning with uncurated images. arXiv preprint Xrxiv:2306. 04244, 2023.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Contents
1 Introduction
1
2 Related Work
3
3 Preliminary
4
4 Delving into Instance Consistency in SSL
5
4.1 How Does SSL React to Different Levels of Instance Consistency?
5
4.2 How Much View Diversity is Beneficial?
8
4.3
Earth Mover's Distance as Diversity Estimator
4.4
Validation Across Diverse Settings
10
Conclusion
12
A Implementation Details
19
A.1 Pre-training Sctup
19
A.2 Downstream Fine-tuning Setup
20
A.3 EMD-based Similarity Score Computation
20
B Additional Experiment Results
22
B.1 Validation Across Diverse SSL Methods
22
B.2 Validation Across Diverse Tasks
................................
23
B.3 Validation Across Diverse Experimental Settings
................................
23
B.4 Additional Ablation Studies
25
B.5 Additional Results on EMD-based Estimator
26
B.6 Additional Results on Medical Imaging Domain
................................
26
B.7 Additional Results on Time Cost. for EMD Computation
28
B.8 Ablation on Impact of Memory Bank in MoCo-v2.
28
C Further Discussions
29
C.1 Gains Observed and Their Significance
29
C.2 Applicability to Reconstruction-basse based SSLs
29
C.3 Clarifying Scope and Relationship Prior Work
30
Future Work
30
18

---

Published in Transactions on Machine Learning Research (09/2025)
A Implementation Details
A.1 Pre-training Setup
Dataset. We conduct SSL pre-training two datasets: COCO for non-iconic data and ImageNet-100
for object-centric data. COCO (Lin et al., 2014) is large non-iconic dataset with 118k training images containing approximately 896k labeled objects, averaging 7 objects per image. In contrast, ImageNet- 100 is a subset of the object-centric dataset ImageNet- (Deng et al., 2009), consisting of 100 randomly selected classes, with 128k images in total. The selection ensures aligument with the number of training samples in COCO for fair comparisons. The specific ImageNet- classes used our experiments are listed in Table A. All images of both datasets are used for SSL pre-training in our experiments.
List of ImageNet-100 classes
n01 n01443537
n01484850
n01514668
n01518878
101531178
n01532829
n015375444
m01580077
n01582220
n01601694
n01608432
n01632458
n01665541
n01669191
n01704323
n01728920
n01729977
n01753581
n01756291
n01797886
n01807496
n01824575
n01843065
n01847000
n01871265
n01872401
n01873310
n01950731
n01968807
n01978287
n01983481
n01985128
n02002724
102009912
n02011460
n02017213
n02018207
n02018795
n02088364
n02088632
n02090622
n02090721
n02091032
002091467
n02092002
n02093859
n02096437
n02097047
n02097209
n02100236
n02101388
n02105855
n02110627
n02110806
nO21 13624
n021 13978
n021 14548
n021 14855
102116738
n02130308
n02137549
n02165106 105
n02174001
n02177972
n02281788
n02319095
n02364673
n02413577
n02417914
n02442845
n02443114
n02444819
n02447366
n02480495
n02481823
n02493793
n02640242
n02643566
n02655020
n02727426
n02776631
n02782093
n02797295
n02804414
n02823428
n02834397
n02865351
n02869837
n02871522
n02877765
n02883205
102917067
m02927161
a02939185
n02948072
n02965783
n02966687
n02977058
a02992529
n02999410
Table A: List of classes from ImageNet-100. ImageNet-1le 1K dataset (Deng et al., 2009).
These classes are randomly sampled from the original
Setup. All models are pre-trained from scratch for 100 epochs. For the backbone, we use ResNet-50 (He et al., 2016) in MoCo-v2 (Chen et al., 2020b), and ViT-S (Dosovitskly et al., 2021) with the patch size of 16 in DINO (Caron et al., 2021). Specifically, for MoCo-v2, we set the batch size as 256 and the learning rate as 0.3 with the SGD optimizer. For DINO, set the batch size 256 and the learning rate as 0.0005 with the AdamW optimizer. All other training hyper-parameters follow the original set settings in their respective implementations. All pre-training experiments are conducted on NVIDIA RIX A6000 GPUs.
Implementations. ments.
We provide the implementation details for the proposed configs in our ablation experi-
For the Instance Diversity category, we need to utilize the locations of object instances in the given image. We use the GT annotations provided in COCO as the reference to obtain this object instance information. However, unlike COCO, which includes GT bounding-box annotations, we nced self-identify the locations of foreground instances in ImageNeet1000. We adopt two unsupervised approaches to generate pseudo masks for object instances: MaskCut (Wang et al., 2023), and Selective Search (Uijlings et al., 2013).
MaskCut (MC) is introduced in CutLER (Wang et al., 2023), which combines Normalized Cuts (NCut) (Shi & Malik, 2000) and Conditional Random Fields (CRFs) (Krähenbühl & Koltun, 2011) to discover multiple object instance masks without any supervision. We adopt the official implementation of MaskCut to obtain pscudo masks for ImageNet-100. Selective Search (SS) (Uijlings et al., 2013) is a classic unsupervised object proposal generation method, which leverages color similarity, texture similarity, region size, and fit between regions identify object candidates. We the object proposals generated by SoCo's (Wei et al., 2021)
19

---

NVIDIA RTX A6000 GPUs. We list the setups for Published in Transactions on Machine Learning Research (09/2025)
official implementation. To reduce noise and exclude tiny instances, we limit the maximum number of objects to 3 per image for both approaches. The pseudo masks generated by these methods are used to implement. the configs in ablation experiments in the Instance Diversity category. A discussion of this two approaches is provided in Appendix B.4.
For the Scale Diversity category, we carefully adjust crop scales within the pre-training dataset. For COCO, we use the average object instance size derived from dataset annotations, resulting in a scaling range of s= (0.08, 0.4) for Smaller Crop. Meanwhile, we apply a scaling range of (0.4, 1.0), which doubles the scale in the default setting for Larger Crop. Considering the object scale difference in two datasets, for ImageNet-100. we apply a scaling range of (0.18, 0.9) for Smaller Crop, and = (0.4, 1.0) for Larger Crop. The crop scales are selected based on the ablation studies in Appendix B.4.
A.2 Downstream Fine-tuning Setup
Dataset. We evaluate the pre-trained models a board range of downstream evaluation tasks including classification, object detection, instance segmentation and depth prediction. For object detection, we use PASCAL VOC-0712 (Everingham et al., 2010) for general object detection, and DOTA-v1.0 (Xia et al., 2018) for aerial object detection. For classification, we utilize five small-scale classification datasets: CIFAR- 10 (Krizhevsky et al., 2009a), CIFAR-100 (Krizhevsky et al., 2009b), DTD (Cimpoi et al., 2014), Oxford Pots (Parkhi et al., 2012), and STI-110 (Coates et al., 2011). Additionally, COCO (Lin al., 2014) included for the in in-distribution evaluation on object detection and instance segmentation tasks. We also include depth prediction NYUd (Silberman et al., 2012) to demonstrate the generalizability of our findings to 3D downstream tasks.
Setup. All downstream experiments are conducted on downstream tasks as follows:
Object Detection: Evaluations are performed using MMDetection (Chen et al., 2019) and MMRo- tate (Zhou et al., 2022). Specifically, Faster R-CNN (Ren al., 2015) with the 24k iteration schedule is used for PASCAL VOC general object detection. Oriented R-CNN (Xie et al., 2021) with the 1x schedule is used for DOTA aerial object detection, and Mask R-CNN (He et al., 2017) with the 1x schedule is used for COCO object detection and instance segmentation. The batch size is set to 2 per GPU, with other hyper-parameters following the default settings. We report the mean Average Precision (mAP) as the evaluation metric for object detection.
Classification: We freeze the pre-trained SSL backbone and train a supervised linear classifier on top of it to perform the classification evaluation. For MoCo-v2, we follow the evaluation protocol in Peng et al. (2022), the linear classifier is trained for 100 epochs with initial learning rate of 10.0, reduced by factor of 0.1 at the 60th and 80th epochs. For DINO, we follow Caron et al. (2021) to train the linear classifier for 100 epochs with an initial learning rate of 0.001, optimizing by SGD using a cosine annealing schedule. The batch size set to 512 across all five datasets for both cases. We report the Top-1 Accuracy as the evaluation metric for classification.
Depth Prediction:
Danponlaa-Deepp--hs-tatttte Toolbbox (Li, 2022) based on MMSegmentation (Contributors, 2020). Adabins (Bhat et al., 2021) with 2x schedule is used for NYUd depth prediction. The batch size is set to 8 per GPU, with other dyper-parameters following the default settings. We report the RMSE as the evaluation metric for depth prediction.
Evaluations are performed using
A.3 EMD-based Similarity Score Computation
We provide more details of the definition and computation of EMDD-based similarity below.
Definition. Earth Mover's Distance (Rubner et al., 2000; Zhang et al., 2020) quantifies the distance between two distributions by computing the minimum cost needed to transform one distribution into another, which has the form of the well-studied optimal transport problem (OTP). ). In our case, EMD measures the distance between the given feature maps of the two augmented views X, Y NXD where N denotes the number of
20

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm A: Procedure for EMD--based Similarity Score
Data: Two augmented views X, Y E RVXD.
Result: Similarity score 5(X,Y)
Flatten feature maps into local feature representations
X, + [x (x
Define the overall transportation polytope
(ss.d) = {PE E x P1 s, pT1 1 = d}:
Compute the cost matrix
Cg+ + x
Solve optimal transport via Sinkhorn-Knopp iteration
= arg mit C) hh());
Compute the similarity score
8(X, Y) - (P.1 C);
vectors in each feature map and is the feature dimension. We first flatten these maps into two sets of local feature representations X' = (x:li = 1,2, .., N) and y {yyli = 1.2,..,,N N), where each x; and y, represents local vector at specific spatial location in the given view.
The EMD between these two feature maps is then defined as the minimum "transport cost" required to transfer units from "suppliers" in X' to "demanders" in V, where each supplier x; has 8, units to transport, and each demander y, requires dj units. The roles of suppliers and demanders can be switched without affecting the total transportation cost. The overall transportation polytope can formulated as follows:
U(s,d) = (P T P = 1=
(A)
Here 1 E R represents the all-ones vectors, while and d are vectorized forms of (s:) and {dz}, respectively. These vectors are also referred to the marginal weights of matrix P across its rows and columns. We then define the cost matrix Cij to represent the cost per unit transported from supplier node x; to demander node yj according to their cosine distances as:
x y,
Cjj=
=1
(B)
|x,
||,
With this notation, we can define the EMD as:
OT d) = min (P.C)
(C)
PeU((..)
where OT(s, d) is the total transportation cost and (. denotes the Frobenius dot product of two matrices.
Computation Details. To find the optimal assignment matrix Pa, we consider the OTP as Linear Programming problem by using -oahhonn-Koop iteration (Sinkhorn & Knopp, 1967; Cuturi, 2013), which introduces a entropy constraint term h:
P aig (P,C) - - [)(p), (eU(s
(D)
where h(P) is the regularization of the entropy of the assignments, and À is constant typer-parameter to control the intensity of regularization term. After repeating T times iterations (T 10 in case), the approximate optimal assignment p" can be obtained.
For our case, high values of P indicate a low transport cost from X; to y,, allowing maximum unit transfer, which suggests that x; and y, have similar features, thus potentially sharing more meaningful mutual information. Therefore, we can compute the similarity score S between the feature representations within two augmented views as:
S( (P, C)
(F)
21

---

Published in Transactions on Machine Learning Research (09/2025)
COCO
ImageNet-100
Config
CIFAR-10
CIFAR-100
DTD
Pets
STL-10
CIFAR-10
CIFAR-100
DTD
Pets
STL-10
Baseline
74.21
49.84
48.21
81.79
77.54
53.88
57.57
Lower Bound
28.25 +4..0
12.16
-0n.0
8.19
-41.5
7.63
-40.0
24.65
-97.44
28.68
-45.0
11.07
-4.00
6.49
-47.00
8.77
-10.00
26,00)
-56.00
Sputial Ovlp.
76.92 -am
52.92
51.67
63.45
52.38
17
84.14
78.57
11
55.03
1.
57.14
1816
59.14
1..0
86.56 11.08
Trst. as Bg
77.92
53.28
53.12
52.22
83.62
80.24
56.20
57.87
64.05
86.01
Only Bg
75.85
-1.44
52.88
-3.01
54.26
79.73
56.86
61.22
86.36 +1.70
Larger Crop
70.95
45.03
-4.1
48.19
-1.05
38.05
-10.4
78.44
-9.00
75.31
-2:20
49.43
-4.45
52.29
-1.70
53.58
-3.00
81.24
-5.08
Smaller Crop
76.19 1.m
52.04
1.30
52.19
53.96
1
83.51
1.72
79.84 13:00
56.31
1..4
58.35
14:36
06.
19.36
86.91 1.:0
Smaller Crop!
74.26 10.0
48.94
49.10
48.73
82.44
76.51
-1.03
52.82
52.98
-1:01
56.58
84.09
Table B: Classification results with DINO (Caron et al., 2021) pre-trained on COCO (Lin et al., 2014) and ImageNee--100 (Deng et al., 2009). We freeze the pre-trained weights of the SSL backbone and train a supervised linear classifier to evaluate the learned representations on five classification benchmarks (Krizhevsky et al., 2009a;b; Cimpoi al., 2014; Parkhi et al., 2012; Coates et al., 2011). All configurations are pre-trained and linear fine-tuned for 100 epochs ensure fair comparison. Performance gaps relative to the baseline configuration are indicated as superscripts. Smaller Crop denotes to Smaller Crop with Zero Spatial Overlap configuration.
COCO
ImageNet-100
Config
VOC-0712
DOTA-v1.0
VOC-0712
DOTA-v1.0
Random Init.
58.62
46.96
58.62
46.96
Lower Bound
53.30 -21.62
44.03
-18.70
53.28
-22.32
44.27
-19.78
Baseline
74.92
62.73
75.60
64.03
Spatial Ovlp.
76.40 -1.48
64.08
+3.35
77.06
+1.48
66.98
+2.95
Inst. Bg
76.25 -1.33
64.55
+1.82
77.80
+2.20
66.49
+2.46
Only Bg
76.31
-1.38
64.05
+1.02
76.76
+1.16
65.45 +1.42
Larger Crop
73.43 -1.49
62.27
74.45
-1.15
63.18
Smaller Crop
76.12
64.29
76,81
+1.21
65.72
+1.00
Smaller Crop!
74.06
-0.86
62.49
-0.24
75.35
-0.25
64.31 +0.28
Table C: Object detection results with DINO (Caron et al., 2021) pre-trained on COCO (Lin et al., 2014) and Image:Net-100 (Deng et al., 2009). We evaluate the learned representations on VOC (Everingham et al., 2010) and DOTA (Xia et al., 2018) for object detection. All configs are pre-trained for 100 epochs for fair comparison. Random Init. refers to the backbone being randomly initialized during downstream fine-tuning.
where 1 1 C' denotes the cosine similarity between two local feature vectors.
We provide the pscudo code for computing the Earth Mover's Distance (EMD)-based similarity score between two augmented views in Algorithm A.
B Additional Experiment Results
In this section, we present additional experiment results under diverse settings to further validate the generalizability of our findings.
B.1 Validation Across Diverse SSL Methods
To assess the broader applicability of our findings, we extend the ablation experiments the effectiveness of instance consistency to another SSL framework, DINO (Caron et al., 2021). Unlike contrastive learning- based methods MoCo-v2 (Chen et al., 2020b), DINO employs a self-distillation approach while still relying on instance consistency during knowledge distillation, which trcats diffcrent views of the same image as positive pairs. For fair comparison, we adopt the same experiment setup as MoCo-v2: pre-training on COCO ImageNet-100 dataset and fine-tuning for classification and object detection evaluations.
Results. As shown in Tables B and C, results on DINO align closely with those observed on MoCo-v2: increasing diversity between positive pairs consistently enhances baseline performance, while excessive diversity
22

---

Published in Transactions on Machine Learning Research (09/2025)
COCO Object Detection
COCO Instance Segmentation
Config
Pre-trained
Aph
AP
AP%
APm
APoo
AP75
Bascline
COCO
52.95
37.39
31.28
50.10
33.33
Spatial Ovlp. =
COCO
35.19 40.87
53.51
40.58
38.35
40.08
31.82
40.54
50.82
40.72
34.19 40.80
Smaller Crop
COCO
35.07 40.45
53.31
40.36
38.09
40.70
31.71
+0.45
50.52
+0.42
33.98 +0.0s
Smaller Crop!
COCO
34.78
+0.16
+0.10 53.05
37.87
+0.48
31.46
+0.18
30.28
+0.38
33.76 +0.43
Baseline
ImageNet-100
34.80
53.29
37.71
31.59
50.53
33.95
Spatial Ovlp. =
ImageNet-100
35.09
10.29
53.56
10.27
38.12
10.41
32.00
10.41
51.01
10.48
34.28
10.88
Smaller Crop
ImageNet-100
35.08 10.28
53.53
10.24
38.05
10.84
31.80
10.21
50.74
10.21
34.11 10.10
Smaller Crop!
ImageNet-100
34.87 +0.07
53.33
+0.04
37.87
+0.16
31.66
+0.07
50.35
+0.02
34.05 +0.10
Table D: Object detection and instance segmentation results with MoCo-v2 (Chen et al., 2020b) pre-trained on COCO (Lin et al., 2014) and ImageNet-100 (Deng et al., 2009). We evaluate the learned representations on COCO (Lin et al., 2014) for object detection and instance segmentation.
COCO
ImageNet-100
Config
NYUd RMSE +
Random Init.
0.7467
0.7467
Lower Bound
0.6564 +0.1081
0.6859 +0.1357
Baseline
0.5483
0,5502
Spatial Ovlp.
0.5154 0.0820
0.6221
Inst. Bg
0.5149
0.0884
0.5157
Only Bg
0.5183
-0.0300
0.5189
Larger Crop
0.5626
0.5593
Smaller Crop
0.5199
0.5114
Smaller Crop!
0.5596
0.5571
+0.0009
Table E: Depth prediction results with MoCo-v2 (Chen et al., 2020b) pre-trained on COCO (Lin et al., 2014) and ImageNet-100 (Deng et al., 2009). We evaluate the learned representations on NYUd (Silberman et al., 2012) for depth prediction.
yields no additional improvements. These findings extend the applicability of our conclusions from contrastive SSL. to broader range of SSL. methods under the instance consistency paradigm.
B.2 Validation Across Diverse Tasks
To validate our findings more downstream tasks, we evaluate the pre-trained models on COCO for object detection and instance segmentation and NYUd for depth prediction mentioned in Appendix A.2, with MoCo-v2 framework pre-trained on COCO and ImageNet-100.
Results. As presented in Table D, results on COCO object detection and instance segmentation indicate that using smaller crop scales and zero overlapping continues to enhance baseline performance, with no added benefit from combining the two configs. Depth prediction results in Table also align closely with the observations in classification and object detection tasks, exhibiting a clear U-curve as shown in Figure A duc to the use of RMSE metric.
B.3 Validation Across Diverse Experimental Settings
To validate our findings with diverse training conditions, we examine the pre-trained models with various experimental settings as follows:
Tuning, Typically, downstream evaluations involve full fine-tuning of the backbone adapt the pro-trained model for task-specific performance. To better isolate and preserve the learned representations from SSL pre-training, we evaluate models under a frozen-backbon setting, where only the task-specific head is fine-tuned while backbone weights remain fixed. This allows to more
Full-Tuning vs
23

---

Published in Transactions on Machine Learning Research (09/2025)
NYUd (COCO pre trained)
NYUd (COCO pre traincd)) pre-trained)
NYUd (NN-100 pre-trained)
0.65
0.65
NYUd RMSE
0.60
0.60
0.55
0.55
2
6
Grid-based EMD similarity
Sampling based EMD similarity
Figure A: EMD similarity versus depth prediction results. The similarity scores between views are plotted against depth prediction results. Note that RMSE is used for evaluation metric, thus results exhibit a clear U-curve.
COCO
ImageNeet-100
Config
VOC-0712
DOTA-v1.0
VOC-0712
DOTA-v1.0
random init.
33.85
26.24
33.85
26.24
Lower Bound
18.63
14.63
19.65
15.42
Baseline
59.28
41.20
60.36
42.54
Spatial Ovlp. =
62.35 -3.07
43.85
+2.85
62.39
+2.0s
44.04
+1.50
Only Bg
63.16
44.05
63.41
13.05
44.18
11.04
Smaller Crop
62.68
42.39
63.16
43.91
1:88
Smaller Crop
60.25 -0.97
41.30
60.96 +0.60
42.89
Table F: Object detection results under frozen-backbone tuming with MoCo-v2 (Chen al., 2020b) pre-trained on COCO (Lin et al., 2014) and ImageNet-100 (Deng et al., 2009).
directly observe the impact of instance consistency and diversity between positive pairs from pre-training. In this sctup, as shown in Table F, we find similar trends the full-tuning setting: using smaller crops and zero overlapping outperform the baseline, with no added gain from combining both. The performance gaps between baseline and other configurations are more pronounced in this setup, reinforcing that the observed performance improvements are due to different positive pair selection during SSL pre-training rather than downstream adaptation. This further supports our findings of the necessity of instance consistency and diversity ween positive pairs for effective SSL.
Transfer us In-Distribution Evaluation. In previous discussions, we primarily evaluate our findings using transfer learning tasks, where models are pre-trained on one dataset. and fine-tuned on a different. downstream one. To determine whether our insights hold for in-distribution tasks, where pre-training and evaluation on the same dataset, we conduct experiments where both pre-training and evaluation are performed on COCO (Lin al., 2014). As stated in Appendix B.2, results in Table D indicate that using smaller crop scales and zero overlapping continues to enhance baseline performance, with added benefit from combining the two configurations. This pattern mirrors the observation in transfer learning tasks and aligns with EMD measurement, reinforcing that our findings on instance consistency and view diversity are robust across both transfer and in-distribution scenarios.
Short Long Epochs. Initial experiments pre-training duration of 100 epochs, relatively short period for SSL pro-training, which often performs longer duration to ensure effective training. To verify
Config
100 epochs
200 epochs
400 epochs
Baseline
73.32
76.34
78.67
Smaller Crop
74.58 +1.26
77.52
+1.18
79.75 +1.08
Spatial Ovlp.
0 74.90
+1.58
77.90
+1.50
79.99 +1.32
Table G: Object detection results with extended training epochs.
24

---

Published in Transactions on Machine Learning Research (09/2025)
COCO
ImageNet-100
Config
VOC-0712
DOTA-vLO
VOC- 0712
DOTA-vI.0
GT
MC
SS
GT
MC
SS
MC
58
MC
SS
Baseline
73.32
54.38
73.91
55.30
Inst. us Bg 74.15
74.12
74.18
35.47
55.08
55.56
74.35
74.30
56.65
55.94
Only Rg
74.73
74.52
74.66
55.34
55.32
55.41
74.43
74.44
56.65
56.37
Table H: Object detection results with different pscudo mask generation methods with MoCo- v2 (Chen et al., 2020b) pre-trained on COCO (Lin et al., 2014) and ImageNet-100 (Deng et al., 2009). MC and SS refer to the pseudo mask generation methods MaskCut (Wang et al., 2023) and Selective Scarch (Tijlings al., 2013), respectively.
Crop Scale
VOC-0712
DOTA-v1.0
Baseline
73.91
55.30
(0.18,0.9)
74.49
56,26
00.16,0.8)
73.97
55.66
(0.14,0.7)
73.98
55.34
00.1
74.06
55.23
(..1.
74.04
55.74
= (0.08, 0.4)
73.84
55.51
Table I: Object detection results with varied crop scales with MoCo-v2 (Chen et al., 2020b) pre- trained on ImageNet-100 (Deng et al., 2009). The Bascline config uses a scaling range of s = (0.2, 1.0). Optimal performance is achieved with a scaling range of s = (0.18, 0.9).
whether our findings hold with more epochs, we extend the pre-training duration to 200 and 400 epochs. As shown in Table G, with more training epochs, using smaller crop scales and enforcing zero overlapping consistently enhance performance over the baseline. This reinforces that our observations regarding instance consistency and view diversity remain consistent across different training durations.
B.4 Additional Ablation Studies
We provide additional ablation studies regarding the implementation details for proposed configs as follows.
Ablation Studies on Pscudo Masks. We conduct ablation experiments comparing two pscudo mask generation methods, as shown in Table H. Rather than relying on explicit bounding box or mask accuracy, we evaluate pseudo-mash quality via their impact on downstream SSL performance. Specifically, the experiments are conducted with MoCo-v2 framework and evaluated on VOC-0712 and DOTA-v1.0 under three mask sources: Ground Truth (GT), MaskCut (MC), and Selective Scarch (SS).
As the results show, both pseudo-mask methods closely track the performance of ground-truth masks across diverse configurations and datasets. This supports our claim that the generated pseudo masks are sufficiently reliable to approximate the real instance locations for the purpose of our controlled analysis. We adopt MC in all our experiments.
Ablation Studics on Crop Scales. To optimize downstream task performance, we conduct the ablation experiments on the selection of crop scales, focusing specifically on the scaling range used in Smaller Crop configuration. The scale used in Larger Crop is fixed to (0.4, 1.0), which doubles the default setting.
For COCO, we derive the scaling range directly from the average object instance size provided in the datasct annotations, resulting in a range of s (0.08, 0.4). For ImageNeet1000 we vary the scaling range incrementally from 0.08 to 0.2, with a step size of 0.02, to systematically observe changes in downstream performance for object detection. The experiments are conducted with MoCo-v2 framework and evaluated on VOC-0712 and DOTA-v1.0. As shown in Table I, applying the scaling range of (0.18, 0.9) achieves optimal results for both downstream tasks. Based on these results, we adopt this scaling range in all our experiments.
25

---

Published in Transactions on Machine Learning Research (09/202))
75
75
75
74
74-
74
m&P (80
VOC mdp (50
73
73-
73
72
72-
72
ResNet-50
71
VIT-S VTTB
71-
VIT-DINO-S
RosNet-101
VIT-DINO-8
3
2
Grid-based similarity
similrity
Grid-based similarity
75
75
75
74
74
74
VOC MAP (8)
VOC MAP (86)
73
73
73-
72
72
72
71
Nes-et RosNet-101
71
71
VIT-DINO-S VIT-DINO-8
VITB
2
Sampling-bssed simillaiity
Sampling-baser EMD similarity
Sampling-basse similarity
Figure B: EMD plots with different feature encoders. The similarity scores between views are plotted against object detection results. EMD scores are computed with six different feature encoders: ResNet-50, ResNet-101, ViT-S, ViT-B, VIT-DINO-S, and ViT-DINO-B. Baseline configuration highlighted for reference.
B.5 Additional Results on EMD-based Estimator
We provide additional validation results to support the reliability of the EMD-based similarity score estimator of view diversity.
We use pre-trained ResNet-50 (He et al., 2016) and ViT-S (Dosovitskiy et al., 2021) to extract features for computing the EMD-based score for MoCo-v2 and DINO, respectively. Figure C presents the relationships between EMD scores with object detection accuracy and classification accuracy for models pre-trained on ImageNet- 100 (Deng et al., 2009). Across all tasks, our results consistently reveal a clear reverse-U curve under the two proposed cropping strategios, reinforcing our findings that EMD serve as effective estimator of view diversity across different data sources.
Furthermore, we conduct a cross-architecture validation experiment by computing EMD using six different feature encoders: ResNet-50, ResNet-101, ViT-S, ViT-B, VIT-DINO-S, and VIT-DINO-B. As illustrated in Figure B, all plots exhibit a clear and consistent reverse-U U curve under the two proposed cropping strategies, indicating that a moderate level of view diversity, as estimated EMD, correlates robustly with optimal downstream performance, regardless of the encoder used. This consistency across architectures further emphasizes the generalizability of our findings.
B.6 Additional Results on Medical Imaging Domain
We provide additional validation results on medical imaging domain further evaluate the gencralizability of our findings. Specifically, we conduct. the proposed controlled experiments on the NIH Chest. X-ray dataset (Wang et al., 2017) with MoCo-v2. Following Sowrirajan et al. (2021), explore two settings for SSL pre-training: (1) training from scratch, and (2) training with ImageNet- weights, which are known to provide stronger representations and initializations for downstream medical tasks.
We evaluate the performance of the pre- pre-trained models with the disease classification task on the same dataset. To maintain consistency with standard classification evaluation, we filter out samples with multiple or missing labels, and report the Top-1 classification accuracy. As shown in Table J, the trends observed the medical domain align with those from natural image datasets: Increasing diversity between positive pairs consistently
26

---

Published in Transactions on Machine Learning Research (09/2025)
75
75
56
56-
74
D00 mAR (%)
74
DOTA (S)
-4
73
34
52-
71
71
70
VOC
DOTA
70
VOC
DOTA
Crid-based similarity
Grid-based EMD similrity
Sampling-based similarity
Sampling-based EMD similarity
(a) EMD similarity detection accuracy with MoCo-v2 (Chen al., 2020b) pre-trained ImageNet-100.
C14b30 1.66 (N)
C TO
15 Top
53-
based END
based 3HD
based
1/ddlaassd EHD similatt
A45 (00
42
92-
Top3
DMA-32
C1963-100
DTD
Fels
5T-30
5
7
Samplig-avved EWO simvvityy
Samolty dssd ENO amiootty
Somolighhoed EMD aibbonty
Samsing-taoe EMD smibett
Semsingttwee EMD cernrity
(b) EMD similarity versus classification accuracy with MoCo-v2 (Chen et al., 2020b) pre-trained ImageNet-100.
75
75
mccm (N)
70-
001A mAbeY 186
60
70
DOTA mAk (N
80
65
55
65
55
60
60
VOC
DOTA
4
VOC
DOTA
similaiity
Grid-based EMD similanity
Sampling based similarity
based EMD similanty
(c) EMD similarity versus detection accuracy with DINO (Caron et al., 2021) pre-trained on ImageNet-100.
60
300
DT0
63
73
63-
CH 10 To
D90530
53-
37
4 D7A--1
C1M4-200
ST-30
3
3
3
3
3
3
Samoling bosol 3R0 simiarit
Sams bzol 3RD similarty
Samsktg basll 3MD simlaaly
Samskig busol 5MD smilanty
Sansing bosel seitani
(d) EMD similarity versus classification accuracy with DINO (Caron el al., 2021) pre-trained on ImageNet-100.
Figure C: EMD similarity versus detection and classification accuracy. The similarity scores between views are plotted against object detection and classification results. Baseline configuration highlighted for reference.
27

---

Published in Transactions on Machine Learning Research (09/2025)
Config
Baseline
Lower Bound Spatial Ovlp. 0 Inst. Bg Only Bg
Larger Crop Smaller Crop Smaller Crop!
w/o pre-train
31.8
27.8
33.9
33.9 12.
33.7
29.1
33.0
30.5
with pre-train
39.7
29.3
10.4
41.7 12.
41.9 1..
42.0 19.4
35.7
-40
41.2 m..
38.1 -1.6
Table J: Classification results with MoCo-v2 (Chen et al., 2020b) pre-trained on NIH Chest. X-ray (Wang et al., 2017). We freeze the pre-trained weights of the SSL backbone and train a supervised lincar classifier to evaluate the learned representations on the same datasct following Sowrirajan al. (2021).
improves downstream performance; excessive diversity, however, yields no additional improvements. These results reinforce the robustness of our findings and suggest their applicability extends beyond natural images to specialized domains such as medical imaging.
B.7 Additional Results on Time Cost for EMD Computation
We provide additional results to quantify the computational cost of computing EMD and clarify its feasibility as an offline metric for guiding optimal positive pair selection in SSL pre-training.
Robustness to Data Fraction. To reduce the cost of full-dataset computation, we first examine the stability of EMD scores when being computed on only small subset of the whole dataset. As shown in Table K, even with just 0.1% of the data, EMD scores remain consistent and are well-separated across different configs. This confirms that EMD can be reliably estimated with a minimal fraction of data, significantly reducing the computation burden.
Data Fraction +
100%
50%
20%
10%
1%
0.1%
EMD (Bascline)
0.6876
0.6878
0.6889
0.6880
0. 0.6869
0.6872
EMD (Spatial Ovip. = 0)
0.4329
0,4326
0.4330
0.4337
0.4328
0,4332
Table K: Grid-based EMD similarity scores using ResNet-50 encoder pre-trained on COCO.
Wall-Clock Time Comparison. We then compare the time required to compute EMD (with both full and 0.1% data) against the time required for standard SSL pre-training using MoCo-v2 on COCO as shown in Table L. Considering evaluating one SSL setting typically requires 100 epochs of pre-training, even computing EMD on the full dataset is already 10x faster, and using 0.1% data yields speed-up of over 10,000x. This confirms that EMD introduces negligible overhead, especially when used as a one-time offline estimator.
EMD 100% data
EMD SSL pre-training epoch
SSL pre-training 100 epochs
0.1% data
Time (mins)
10.17
0.01
1.07
106.67
Table L: Wall-clock time cost comparison for EMD computation. Time are measured with the batch size of 256 on 8 NVIDIA RTX A6000 GPUs and AMD EPYC 7543 32-Core CPU, with EMD solver from OpenCV.
These two observations confirm that EMD is both effective and efficient for quantifying view diversity, making it a practical tool for guiding positive pair selection in real-world SSL pipelines.
B.8 Ablation on Impact of Memory Bank in MoCo-v2
We conduct additional ablation experiments to analyze the impact memory bank in MoCo-v2. Specifically, the MoCo-v2 models are pro-trained on COCO across varying memory bank sizes, and evaluated on VOC-0712 for object detection. As shown in Figure D, models pre-trained on COCO generally fail to benefit from larger memory banks, in contrast to the trend observed ImageNet, reported in Figure 3 of He et al. (2020).
28

---

Published in Transactions on Machine Learning Research (09/202))
74.50
VOC mAp (%)
74.25
74.00
73.75
Baseline
73.50
Spatial Ovip.=0
Smaller Crop
256
4096
65536
Memory Bank Size K (log-scale)
Figure D: Effect of memory bank size with MoCo-v2 pre-trained on COCO.
Notably, we observe that increasing view diversity within the optimal EMD range enhances the effectiveness of utilizing larger memory banks, while excessive diversity remains ineffective. This observation further demonstrates the broader applicability of our findings in understanding SSL mechanisms on diverse data sources.
C
Further Discussions
C.1
Gains Observed and Their Significance
Our experiments highlight consistent and meaningful performance improvements across different positive pair selection configurations in SSL. While modifying views in augmentation perspective naturally yiclds relatively modest gains, as reported in prior works Peng et al. (2022) and Van Gansbeke el al. (2021), our results demonstrate that targeted control of view diversity produces improvements with practical significance. Specifically, our proposed configurations yield:
1.5% mAP gain on VOC detection;
over 3x higher gains on COCO detection than reported in Peng et al. (2022);
and 3-5% accuracy improvement on classification tasks.
These improvements remain consistent even under longer training durations, shown in Table G, reinforcing the robustness of our findings. The consistent performance gap across datasets and tasks demonstrates the importance of view diversity control in SSL, especially for non-iconic or complex data sources.
C.2 Applicability to Reconstruction-baseed SSLs
Our study specifically focuses on investigating the necessity of instance-level consistency in SSLs, which serves as core assumption cantrast and distillation-based SSLs, such as MoCo-v2 and DINO. These methods explicitly treat each image separate class and primarily rely on the consistency between augmented views of the same image to learn meaningful representations.
In contrast, reconstruction--asse SSLs like MAE (He et al., 2022) and SimMIM (Xie et al., 2022b) are designed with fundamentally different objective: to reconstruct the masked parts of an image, without requiring or benefiting from carefully designed view augmentations. As such, they do not rely on instance-level consistency and thus fall outside the scope of our study.
For this reason, we focus our validation on contrastive- and distillation-basdd methods, which are the most representative paradigms aligned with research objective.
29

---

Published in Transactions on Machine Learning Research (09/2025)
C.3 Clarifying Scope and Relationship to Prior Work
We clarify the distinctions between our contributions and related prior works below.
Comparison with Tian et al. (2020). While both works observe a reverse---shaped relationship between view diversity and SSL downstream performance, our study addresses a fundamentally different research question and offers several novel contributions:
1) Different Research Focus: Instance Consistency us. View Informativeness. InfoMin (Tian el al., 2020) investigates the informativeness of views and hypothesizes that optimal pair of views should share minimal but sufficient mutual information for learning useful representations. In contrast, our work focuses on investigating the assumption of instance-level consislency. This is particularly relevant for non-iconic data, where two augmented views from the same image may not guarantee to contain the same object or share consistent semantic information.
2) Broader and More Realistic Evaluation Settings. InfoMin evaluates view informativeness solely on iconic data, where the consistency between object instances in different crops is largely guaranteed. We go beyond this by performing controlled experiments on both iconic and non-iconic datasets, across multiple SSL frameworks. This broadens the practical relevance of our findings and systematically studies the effectiveness of SST. across real-world data sources.
3) From Qualitative Observation to Quantitative Estimation via EMD. InfoMin proposes qualitative hypothesis about the existence of an optimal sweet point in view diversity, but does not offer concrete metric to quantify it. We go step further by introducing EMD as a quantitative and generalizable tool to estimate view diversity and identify this sweet spot reliably across diverse datasets, SSL methods, and downstream tasks.
4) Temporal Relevance and Novel Insights. InfoMin is grounded earlier SSL scttings based on iconic data and instance-consistett views. Our work offers new insights in the current context, where non-iconic, uncurated, and largo-scale web datasets are increasingly used for SSL pro-training. In such settings, ensuring instance consistency is no longer practical, and we demonstrate that SSL can still remain effective, thereby providing an important and timely contribution to modern representation learning.
Comparison with Moutakanni et al. (2024). This work demonstrates that data augmentations may become less critical in large-data-scale SSL settings. While we agree with this observation, we emphasize that the focus of our work is fundamentally different. Specifically, our study investigates the assumption of instance consistency in SSLs. Specifically, we aim to understand how the degree of instance consistency affects the quality of the obtained representations, particularly when pre-training on non-iconic data where instance-level semantic alignment is not guaranteed.
Rather than studying the necessity of augmentation at scale, our work investigates how modern SSL frameworks generalize to broader and more realistic training data sources, including uncurated and diverse scenarios. In this sense, our analysis complements the discussion in Moutakanni et al. (2024) by targeting a different axis of the SSL landscape.
D Future Work
Our study provides an initial investigation into the role of instance consistency and diversity between positive pairs in SSL. We highlight several directions for future research:
Broader Evaluation Across SSL Methods. While we validate our findings on contrastive (Chen el al., 2020b) and distillation- based (Caron et al., 2021) SSL methods, future work could examine whether similar trends hold for other paradigms such as SimCLR (Chen et al., 2020a), SwAV (Caron et al., 2020), BYOL (Grill et al., 2020), and NNCLR (Dwibedi et al., 2021). Understanding how view diversity influences learning across these different objectives could further generalize our insights.
30

---

Published in Transactions on Machine Learning Research (09/2025)
Scaling to Larger Pre-training Datasets. Our experiments primarily use moderate-sized datasets such COCO (Tin et al., 2014) and ImageNet-100 (Deng et. al., 2009). Investigating whether the observed consistency-ddivrsity trade-off holds on larger and more diverse datasets like Openlmages (Kuznetsova et al., 2020) would validate scalability and offer insights into SSL behavior under more realistic data regimes.
Toward Theoretical Insights. While our findings are grounded in extensive empirical analysis, a theo- retical understanding of how mutual information, instance consistency, and view diversity interact in SSL remains an open question. Developing such theoretical framework could deepen our understanding of view-based supervision and guide the design of more adaptive learnable augmentation strategics.
31


Reviews:
[{"review_id": "dpY0VHmHjF", "reviewer": "Reviewer_xSmj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors conduct comprehensive experiments involving two SSL frameworks (MoCo-v2 and DINO), datasets (both iconic and non-iconic), and diverse downstream tasks (classification, detection, segmentation, depth estimation).\n2. The authors introduce EMD as an estimator of view diversity and offer guidance for SSL training.\n3. The empirical results provide clear implications for visual SSL involving non-iconic data.\n\nWeakness:\n1. The reported performance differences are relatively modest (Table 1 “Smaller Crop” & Table 2, typically 1-3% absolute), and no statistical significance tests or confidence intervals are provided. Prior augmentation-focused studies such as Object-aware cropping [1] achieve much larger margins (e.g. +8.8mAP on OpenImages), and SimCLR’s augmentation [2] ablations routinely yield more than 5 point swings. Without confidence intervals or significance tests, it is difficult to judge whether the observed gains are robust.\n2. The authors did not explain the potential circularity in the EMD computation. EMD is computed on features from a pre‑trained encoder (ResNet-50). Earlier optimal‑transport SSL such as Self‑EMD performs distance computation within the training loop to avoid external feature bias [3]. Reliance on an external backbone can over‑estimate alignment quality, particularly if that backbone is architecturally similar to the model under evaluation. The observed empirical phenomena (e.g. U-shaped relationship) may not hold in general if we switch the encoder for EMD. For example, as reported in the DINO paper, a supervised ViT does not attend well to objects in presence of clutter both qualitatively and quantitatively.\n3. The SSL methodology considered in the paper is limited. Adaptive mask-modeling approaches are straightforward solutions for non-iconic data, including MAE [4], SimMIM [5], AdaMAE [6]. They could eschew instance matching altogether while achieving competitive results. Excluding these paradigms weakens the generality of the conclusions.\n4. The sensitivity and dependency to pseudo-mask quality is unaddressed. The ImageNet-100 experiments depend on the masks generated by MaskCut. For example, CutLER [7] shows that segmentation noise in such masks substantially affects downstream detection performance. An analysis of alternative mask generators or noise levels would clarify the robustness of the experiments.\n\nReferences:\n\n[1] Object-Aware Cropping for Self-Supervised Learning. 2021. Shlok Mishra, Anshul Shah, Ankan Bansal, Abhyuday Jagannatha, Janit Anjaria, Abhishek Sharma, David Jacobs, Dilip Krishnan.\n\n[2] A Simple Framework for Contrastive Learning of Visual Representations. 2020. Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton.\n\n[3] Self-EMD: Self-Supervised Object Detection without ImageNet. 2020. Songtao Liu, Zeming Li, Jian Sun.\n\n[4] Masked Autoencoders Are Scalable Vision Learners. 2021. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick.\n\n[5] SimMIM: A Simple Framework for Masked Image Modeling. 2021. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, Han Hu.\n\n[6] AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders. 2022. Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami, Mehdi Nikkhah, Motilal Agrawal, Vishal M. Patel.\n\n[7] Cut and Learn for Unsupervised Object Detection and Instance Segmentation. 2023. Xudong Wang, Rohit Girdhar, Stella X. Yu, Ishan Misra.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically revisits the instance‑consistency assumption in visual self‑supervised learning (SSL) that two augmented views of the same image must represent the same semantic instance. Through extensive experimentation, the authors empirically demonstrate that the instance‑consistency is not necessary to achieve strong downstream performance. The experiments cover two leading SSL frameworks (MoCo-v2 and DINO) and evaluate performance across diverse downstream vision tasks, including image classification, object detection, semantic segmentation, and depth estimation. The authors design controlled experiments that qualitatively vary the degree of instance-consistency by manipulating three factors: crop scale, spatial overlap between views, and the presence or absence of foreground objects.\n\nTo quantify the view diversity, the authors then propose Earth Mover’s Distance (EMD) as an estimator to measure mutual information between positive pairs. By systematically varying crop scales, the study explores how different degrees of view diversity impact downstream task performance. Their findings highlight an inverted-U relationship between EMD and linear probing accuracy on the COCO dataset. Empirical evidence thus suggests that moderate levels of view diversity (i.e., moderate EMD scores) yield optimal downstream results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend the following major changes:\n1. Weakness-1: Could you report confidence intervals or statistical tests for the key results?\n2. Weakness-2: Could you discuss more on the potential bias introduced by using an external pre-trained encoder for EMD?\n\nAdditionally, I suggest some minor changes:\n1. Weakness-4: The authors can evaluate sensitivity to mask quality by varying pseudo-mask generators or parameters.\n2. In the current draft the backbone choice for the MoCo-v2 runs that feed Table 1 is only stated deep in Appendix A.1:\n   > “For the backbone, we use ResNet-50 (He et al., 2016) in MoCo-v2 …”\n\n   To avoid any ambiguity for readers who may not consult the appendix, I recommend adding this information in one of the following places: (1) Table 1 caption, or (2) somewhere in section 4.1.\n3. The authors can explain Table 1 more clearly. Add two‑to‑three lines in section 4.2 interpreting the mixed results of Smaller Crop†.", "broader_impact_concerns": "No immediate ethical concerns are apparent.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V0oU6ffe6p", "reviewer": "Reviewer_HCEj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- Experiments cover two pre-training objectives and several downstream tasks (classification, fine-grained classification, detection, instance segmentation, depth estimation).\n- The presentation is easy to understand.\n\n#### Weaknesses\n- Literature review is sparse on methods before 2020. For example, jigsaw puzzles (https://arxiv.org/abs/1603.09246, ECCV 2016), RotNet (https://arxiv.org/abs/1803.07728, ICLR 2018), SSL methods using videos (eg. CVPR 2016 slow and steady feature analysis https://openaccess.thecvf.com/content_cvpr_2016/papers/Jayaraman_Slow_and_Steady_CVPR_2016_paper.pdf). \n- An experiment that selects positive pairs based purely on the Earth mover distance is expected after the paragraph in page 9 \"suggestions for positive pair selection\". This experiment would help complete the experiments in this manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript analysis instance consistency based SSL methods (like MoCo-v2 and DINO) with regards to their depends on having true view-consistent positive pairs. This is particularly relevant in non-iconic datasets (in-the-wild datasets) where images contain multiple objects and two randomly cropped image regions (conventionally regarded as a positive pair) will likely not capture the same instance and thus not be a true positive pair.\n\nThe manuscript quantifies the extent to which a positive pair are similar using the earth mover distance. They use various cropping methods to get different data points on this spectrum (from highly consistent positive pairs with low earth mover distance to less consistent ones with high earth mover distance). Experiments show that too high view consistency and too low view consistency are bad for SSL pre-training. A sweet spot exists.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please expand the literature survey to include classic methods such as but not limited to the ones above.\n- Discuss the metric being used in table 2\n- Several of the strengths of the paper come from the sections in the supplementary. Why not use the full 12 pages?\n- Table 2 middle block (\"Baseline\", \"Spatial Ovlp. = 0\", \"Inst. vs Bg\", \"only bg\") hasn't been referred to in the text. One option is to refer to these in section 4.1 \"Results\".\n- Page 7 \"Smaller Crop with Zero Spatial Overlap) consistently outperform the baseline\" is probably a typo. The last row in in Table 1 has -4.13 for CIFAR 10 (ImageNet-100) for example.", "broader_impact_concerns": "This manuscript does not raise broader impact concerns beyond those of the SSL methods that it explores.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pPzK9yxNNi", "reviewer": "Reviewer_vJZ6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper shows that self-supervised models do not need two crops of the same object to learn strong features, overturning a decade-old design heuristic.\n\n- Five crop policies, two datasets (COCO, ImageNet-100), two SSL paradigms (MoCo-v2, DINO), two backbones (ResNet-50, ViT-S/16) and four downstream tasks (classification, detection, segmentation, depth) make the conclusions relativly strong.\n\n- EMD is introduced as a simple quantitative proxy for \"just-right\" view diversity, letting users measure rather than guess their augmentations.\n\n- The paper is well structured; figures intuitively illustrate the crop regimes and the reverse-U effect. Code snippets for the EMD calculation are promised, aiding reproducibility.\n\n-  Relaxing the instance-consistency requirement means large, uncurated photo collections or even surveillance video frames, can be used without costly object-centric filtering.\n\nWeaknesses\n\n- Several diversity setups rely on ground-truth or pseudo bounding boxes, adding extra annotations or engineering to an ostensibly unsupervised study.\n\n- All data are natural images; the findings are untested on more homogeneous or safety-critical domains (e.g., chest X-rays) where foreground boundaries are diffuse and background statistics differ markedly.\n\n- Key curves are reported with single runs; confidence intervals or significance tests are missing, making it hard to gauge robustness.\n\n- The manuscript omits wall-clock cost for computing EMD over large datasets, leaving open whether the \"offline tuning\" step is negligible or burdensome.\n\n- The paper does not cover reconstruction-based, masked-prediction or multimodal SSL frameworks; it is unclear whether the same diversity sweet spot applies there.\n\n- The Wasserstein distance is measured with a frozen ResNet-50; correlations might change with backbone scale or patch granularity, an aspect not explored.\n\n- Source code does not seem to be provided", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper questions the instance-consistency assumption that dominates modern image-based self-supervised learning. Instead of insisting that two augmented views must depict the same object, the authors provide a systematic analysis of how much shared content is actually needed to learn useful representations. Their key contributions are:\n\n1.  Using COCO and ImageNet-100, the paper shows that positive pairs containing only background pixels or even coming from disjoint image regions, can match or outperform the standard random-crop baseline on classification, detection, segmentation and depth tasks. \n\n2. As the visual overlap between the two views decreases, accuracy first improves and then collapses, revealing an optimal moderate diversity regime rather than the extremes of identical or totally unrelated crops. \n\n3.  The paper quantifies diversity with the 1-Wasserstein distance\n$W_1\\bigl(P_{\\text{view}\\,1},\\,P_{\\text{view}\\,2}\\bigr)$,\nand shows that values around 0.3–0.4 consistently coincide with the performance peak across 10 crop configurations, two SSL frameworks (MoCo-v2, DINO) and five downstream datasets. \n\n4. The phenomena hold for both convolutional (ResNet-50) and transformer (ViT-S/16) backbones, and transfer from image-level classification to dense prediction tasks without retraining the backbone. \n\n5. The authors propose a simple, task-agnostic crop-policy-tuning recipe: measure $W_1$ offline for candidate augmentations, select those in the mid-range band, and skip costly manual curation of object-centric images.\n\nThis indicates a possible relaxation of a central design constraint in SSL and suggest a principled knob for controlling view diversity, and demonstrate that robust representations can emerge even when positive pairs do not share a clearly delineated object.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you include an experiment on a medical imaging dataset such as NIH Chest X-ray 14 or CheXpert, using the provided lung or pathology bounding boxes.\nIt would be interest to tests whether the reverse, U diversity curve and the EMD \"sweet-spot\" generalise when foreground boundaries are weak and the global appearance is highly uniform.\n\n2. For the principal results: Baseline, Peak, and Extreme-diversity points in Fig. 4, could you run each configuration at least three times and report mean $\\pm$ standard deviation; add paired $t$-tests or bootstrapped confidence intervals where appropriate.\n\n3. Could you provide wall-clock time and GPU/CPU specs for computing Wasserstein distances on COCO-train2017 with your 9-patch sampler, and compare it to one epoch of SSL pre-training.\n\n4. Please state how many images rely on ground-truth boxes versus pseudo-boxes, describe the pseudo-box generator’s mAP, and discuss whether those boxes are strictly necessary for reproducing the main findings.\n\n5. Release code and pretrained weights. Publishing the crop samplers, EMD scripts and checkpoint zoo would markedly ease adoption and further validation.\n\n6. Could you re-compute the EMD–accuracy correlation with a larger encoder (e.g.\\ ResNet-101 or ViT-B/16) or with different feature-grid resolutions to check whether the 0.3-0.4 optimum is scale-agnostic.\n\n7.  Could you briefly speculate, supported by at least one pilot result, on whether masked-prediction or reconstruction-based SSL benefits from the same diversity regime, or explain why it might differ.", "broader_impact_concerns": "By relaxing the instance-consistency assumption, practitioners might transfer the method to medical, autonomous-driving or security domains without verifying that the learned features remain reliable when foreground-background statistics differ. Mis-calibration here could yield brittle models and harmful decisions. The paper should explicitly warn against blind deployment and suggest domain-specific validation.\n\nTraining on unfiltered Internet images can encode societal or demographic biases (e.g. under-representation of certain ethnicities). Because the method reduces the need for manual curation, the risk of latent bias propagation may rise. A bias audit protocol or mitigation advice would strengthen the Broader-Impact section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KYk2x02XaZ", "reviewer": "Reviewer_oinq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The evaluations are diverse in many crucial aspects: pretraining dataset, SSL objective/pipeline, architecture, and downstream dataset tasks.\n\n- The paper is overall well-written, and the message is clearly conveyed to the reader.\n\n## MAJOR concerns\n\na) **Fine-grained classification datasets**: The reported results on the classification datasets do not correspond to the standard fine-grained classification benchmarks, such as CUB, Car, Dog, Flower, see citations below. The textures dataset (DTD) is certainly not in the category of fine-grained classification. refs [1-4].\n\n\nb) **Data determines the scope of the analysis**. While I see some benefit from smaller-scale analysis on the sensitivity of the chosen augmentations, this analysis is heavily dependent on the dataset (number of samples, quality of data, number of semantic categories). The authors must state this as a limitation/scope of their experimental analysis. In [6], it is recently shown that the augmentations are due to the lack of data, and when scaling from Imagenet1k to imagenet21k (>10M samples), augmentations become less relevant. I do still agree on the presented way of separating iconic vs non-iconic datasets.\n\n\nc) Crop scale ranges in the Scale Diversity category: I would at least expect the authors to share the final values of the crop scales used in Section 4.2, where they refer to smaller and larger crops. This must be in the main text.\n\nd) EMD. There is not enough detailed description in the main manuscript for the EMD calculations, which is supposed to be a major contribution of this work. For me, it’s not quite clean how this computation is done. How many samples are used? How many random augmentations per image? The algorithm only shows what’s happening once N features of dimension D are available from 2 views.\n\n\ne) Connections, differences, and overlap with proximal work [7]. This work does not sufficiently describe the overlap with the existing work. How are the estimations of mutual information related compared to [7]? It is not enough to write our results align with [7]. If both manuscripts explore a similar topic, why conduct similar experiments and end up in similar findings 4 years later? This is a major concern for me. The u-shaped behaviour in Figure 4 has also been shown in [7].\n\nf) Limited predictive power due to supervised ResNet-50 for EMD. The proposed EMD metric uses a supervised metric, which cancels out the idea of estimating the mutual info between views for self-supervised learning. An off-the-shelf feature extractor, such as dinov2 trained on large-scale and diverse unlabelled data, would make more sense here, in my view.\n\nh) Section/paragraph: “Diverse Tasks & Experimental Settings.” In which experiments is this paragraph referring to? Are the authors discussing results that are related to the main text or supplementary? Cross-referencing is missing here.\n\n\n### MINOR concerns \n\n- Table 2: Which metric is used for object detection? \n\n- As far as I understand, the supplementary material should have been submitted as a separate file and not at the end of the main manuscript.\n\n- Results on ImageNet-1k in the supplementary. “ Results for SSL pre-trained on ImageNet (Deng et al., 2009) are provided in the supplementary materials.” Where are these results precisely?\n\nThe description of MOCO and DINO is trivial. Better to remove this part or write a self-sufficient preliminary explanation in the supplementary.\n\n## References\n\n[1] Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The caltech-ucsd birds-200-2011 dataset. \n\n[2] Krause, J., Stark, M., Deng, J., & Fei-Fei, L. (2013). 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops (pp. 554-561).\n\n[3] Dataset, E. (2011). Novel datasets for fine-grained image categorization. In First workshop on fine grained visual categorization, CVPR. Citeseer. Citeseer. Citeseer (Vol. 5, No. 1, p. 2).\n\n[4] Nilsback, M. E., & Zisserman, A. (2008, December). Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing (pp. 722-729). IEEE.\n\n[5] Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021). With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9588-9597).\n\n[6] Moutakanni, T., Oquab, M., Szafraniec, M., Vakalopoulou, M., & Bojanowski, P. (2024, December). You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning. In NeurIPS 2024-The Thirty-Eighth Annual Conference on Neural Information Processing Systems.\n\n[7] Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., & Isola, P. (2020). What makes for good views for contrastive learning?. Advances in neural information processing systems, 33, 6827-6839.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work conducts a comprehensive experimental evaluation on the creation of positive views in visual SSL, precisely: 1) Instance Diversity of views is controlled using the bounding boxes (ground truth or based on prediction from pretrained neural net object detectors), 2) Scale Diversity consistns of a group of exprimental that explicitly controlled the cropping scale range in the randomresize crop transformation. Finally, using a supervised, pre-trained ResNet50, the authors attempt to quantify a measure of mutual information based on the way the views are generated.  Similar to [7], a u-shaped behaviour is observed, meaning that view diversity can be enhanced to some extent while leading to superior downstream performance (object detection + classification) compared to the current augmentation pipeline, but not to a large extent.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Apart from my main concerns and specifically d,e,f:\n- Nearest-Neighbor Contrastive Learning would have been a valuable addition to the study, especially in conjunction with the MOCO baseline, which utilizes a memory bank.\n\n- In general, there are a lot of things to improve in the manuscript, for instance, sec 3 is trivially explained in my humble opinion and can be replaced by more experiments (as I wrote in minor concerns)\n\n- To repeat my main concerns concerning EMD, the presentation of EMD as an alternative to the infonce mutual info estimation in [7] must be clearly explained. The differences in the papers, scopes and investigation should be crystal clear in the next version of the manuscript. On top the EMD section must be more meticulously written and the reliance to a supervised models must be tested with an label-free pretrained model. A pseudocode for EMD would probably be more useful compared to the algorithm in its current form where many details are missing for other researchers to plug their pipeline and test it.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)