Beyond Random Augmentations: Pretraining with Hard Views
Authors: Fabio Ferreira
Ivo Rapant
Jörg K.H. Franke
Frank Hutter
Venue: ICLR 2025 Poster (2025)
Keywords: Self-Supervised Learning
Data Augmentation
Pretraining
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Hard View Pretraining, a learning-free self-supervised method generating challenging samples based on the model’s state, improving multiple baselines by 1% linear eval. accuracy, including a 0.6% boost for DINO ViT-B/16, reaching 78.80%.

Abstract: Self-Supervised Learning (SSL) methods typically rely on random image augmentations, or views, to make models invariant to different transformations. We hypothesize that the efficacy of pretraining pipelines based on conventional random view sampling can be enhanced by explicitly selecting views that benefit the learning progress. A simple yet effective approach is to select hard views that yield a higher loss. In this paper, we propose Hard View Pretraining (HVP), a learning-free strategy that extends random view generation by exposing models to more challenging samples during SSL pretraining. HVP encompasses the following iterative steps: 1) randomly sample multiple views and forward each view through the pretrained model, 2) create pairs of two views and compute their loss, 3) adversarially select the pair yielding the highest loss according to the current model state, and 4) perform a backward pass with the selected pair. In contrast to existing hard view literature, we are the first to demonstrate hard view pretraining's effectiveness at scale, particularly training on the full ImageNet-1k dataset, and evaluating across multiple SSL methods, Convolutional Networks, and Vision Transformers. As a result, HVP sets a new state-of-the-art on DINO ViT-B/16, reaching 78.8% linear evaluation accuracy (a 0.6% improvement) and consistent gains of 1% for both 100 and 300 epoch pretraining, with similar improvements across transfer tasks in DINO, SimSiam, iBOT, and SimCLR.

Published as a conference paper at ICLR 2025
BEYOND RANDOM AUGMENTATIONS: PRETRAINING WITH HARD VIEWS
Fabio Ferreira'
Ivo Rapant" University of Freiburg
Jörg K.H. Franke
University of Freiburg
University of Freiburg
Frank Hutter
ELLIS Institute Tübingen & University of Freiburg
ABSTRACT
Self-Supervised Learning (SSL) methods typically rely on random image augmentations, or views, to make models invariant different transforma- tions. We hypothesize that the efficacy of pretraining pipclines based on conventional random view sampling can be enhanced by explicitly selecting views that benefit the learning progress. A simple yet effective approach is to select hard views that yield a higher loss. In this paper, we propose Hand View Pretraining (HVP), learning fereenng-free free strategy that extends random view generation by expasing models to more challenging samples during SSL pretraining. HVP encompasses the following iterative steps: 1) randomly sample multiple views and forward each view through the pretrained model, 2) create pairs of two views and compute their loss, 3) adversarially select the pair yielding the highest loss according to the current model state, and 4) perform a backward pass with the selected pair. In contrast existing hard view literature, we are the first to demonstrate hard view pretraining 's effectiveness scale, particularly training on the full ImageNNet-I1 dataset, and evaluating across multiple SST. methods, ConvNets, and ViTs. As result, HVP sets a new state-of-the-ar on DIÑO FiT-B/16, reaching 78.8% linear evaluation accuracy (a 0.6% improvement) and consistent gains of 1% for both 100 and 300 epoch pretraining, with similar improvements across transfer tasks in DINO, SimSiam, iBOT, and SimCLR.
INTRODUCTION
Learning effective and generalizable visual representations in Self-Supervisod Learning (SSL) has been approached in various ways. Many SSL methods can be broadly categorized into generative and discriminative approaches (Chen et al.) 2020a).. Generative methods focus generating image inputs, while discriminative methods, particularly contrastive learning (Hadsell et al.) 2006 He et al. 22200), aim at learning latent representations which similar image views are located closely, and dissimilar ones distantly.
Such views are generated by applying a sequence of (randomly sampled) image transfor- mations and usually composed of geometric (cropping, rotation, etc.) and appearance (color distortion, blurring, etc.) transformations. Prior work (Chen et al. 2020aa Wu et al. 2020 Purushwalkam & Gupta 2020; Wagner al.) 2022: Tian al.] 2020b) has identified random resized crop (RRC), which randomly crops the image and resizes il back to a fixed size, as well as color distortion as critical transformations for effective representation learn- ing. However, despite this finding and to our best knowledge, little research has gone into identifying more effective ways for generating views improve performance.
Existing SSL approaches that attempt to control the hardness of views include adversarial (Shi et al. 2022 Tamkin et al.) 2021 or cooperative (Hou et al. 2023) techniques. For
"Equal contribution. Correspondence to: ferreira@es.mmi-feiiburggee

---

Published as a conference paper at ICLR 2025
Modi
Los
Alvasara
Serda 501
(a)
(b)
Figure 1: (a) HVP first samples N views, pairs them, and adversarially selects the hardest pair, i.e. the one with the worst loss according to the current model state. (b) Examples (left) and sampled views (right) after transformations. Hard pairs selected by HVP are shown with solid frame.
instance, Tian et al. (2020b) use mutual information theory to adversarially learn view generators. These methods offer valuable insights into optimizing views but often introduce additional complexity, such as requiring additional models or significant changes to the SSL pipeline, limiting their practicality in statc-of-tho-art models where resource constraints are alrcady concern. Similar to what we propose, [Koyigit et al. (2023) introduce a hard view sampling strategy but targeting the acceleration of pretraining. Moreover, their approach requires luning hyperparametens like learning rate and augmentation magnitude, which can introduce confounding variables. Furthermore, neither Kocyigit et al. (2023) nor Tian et al. (2020b) validate their methods larger datasets such ImageNet- k (Deng et al. 2009 or across larger model architectures, limiting their scalability and applicability.
Building on these observations, we propose Hard View Pretruining (HVV), a fully learning- free, casy-to-integrate approach designed to improve standard pretraining methods without the need for additional model training or complex modifications. Our method leverages the current model state to select challenging samples during pretraining by adversarially sampling pairs of views and selecting the pair that yields the highest loss according to the model's current state (see Fig. Lal. Unlike previous approaches, HVP requires no hyperparameter tuning and demonstrates scalability to large datasets like ImageNet-1lk, offering an efficient and practical solution for improving SSL pretraining. To the best of our knowledge, we are the first to demonstrate the effectiveness of hard view sampling strategy at scale, particularly on modern architectures like Vision Transformers (ViTs) and training on the full ImageNet dataset. Our approach not only integrates seamlessly with recent state-of-the-ar SSL methods but also showcases consistent improvements across both convolutional architectures and ViTs, validating its robustness and scalability.
Overall, our contributions be summarized follows:
We propose Hard View Pretrainóng (HVP), an easy-to-use method complementing SSL by extending the common random view generation to automatically expose the model to harder samples during pretraining. HVP simply requires the ability compute sample-wise losses;
We demonstrate the effectiveness and compatibility of our approach using ImageNet- 1k pretraining across four popular SSL methods that cover a diverse range of discriminative objectives such SimSiam (Chen & He, 2021). DINO (Caron ct al. 2021). iBOT (Zhou et al., 2021), and SimCLR (Chen et al. 2020a):
HVP achicves a new statc-of-the-ar result on DINO VIT-B/16. improving over the officially reported baseline of 78.2% linear evaluation accuracy by reaching 78.8% (400 epochs). HVP also consistently improves all other baselines by an average of 1% in linear evaluation on ImageNet across 100 and 300 epoch-pretratee pretraining runs;
We show similar improvements on diverse set of transfer tasks, including finetun- ing, object detection, and segmentation, and present insights into the underlying mechanisms and robustness of HVP.
2

---

Published as a conference paper at ICLR 2025
We make PyTorch Paszke et al 2019) code, models, and all used hyperparameters publicly available under httpp::/ggiaa //githbb. hommautoml/hyy
RELATED WORK
2.1 DISCRIMINATIVE SELF-SUPERVISED LEARNING
The core idea behind the discriminative learning framework (Chen et. al. 2020a) is to learn image representations by contrasting positive pairs (two views of the same image) against negative pairs (two views of different images) (Hadsell al. 2006). To work well in practice
and to prevent model collapse, contrastive learning mothods often require large number of
negative samples (Wu ct al.] 2018: van den Oord al. 2018: Chen al. 2020a He al.
2020
Tian et al. 2020a Chen et al. 2020b) stored in memory banks (Wu ct al. 2018 He
el al. 2020) or, for instance, in the case of CLC,,, implicitly
large batches (Chen el al.
2020a Non-contrastive approaches, such as BYOL (Grill et al. 2020 SimSiam (Chen &
He 2021), DINO (Caron et al. 2021 and others (Zbontar et al., 2021 Caron et al.) 2020
Ermolov et al. 2021), can only positive pairs without causing model collapse but rely on other techniques, such as Siamese architectures, whitening of embeddings, clustering, maximizing the entropy of the embeddings, momentum encoders, and more.
2.2 OPTIMIZING FOR HARD VIEWS IN SSL
benefits, the realm of learning task-specific augmentation policies based on data has seen quick development (Cubuk et al.
Due to its performance-improviine
2019 Ho al.) 2019
Lin et al.) 2019 Zhang et al. 2020 Hataya et al.] 2020] Hou et al.) 2023 Müller & Hutter 2021 However, these approaches do not include the random resize crop operation in their scarch spaces, limiting the control of view hardness. Similar to us, Kocyigit et al. (2023) uses the current model state for selecting hard views. However, their approach requires controlling learning hyperparameters, while mostly training on a smaller version of ImageNet and NesNets (He et al. 2016) only. We offer a more complete analysis of hard view pretraining, demonstrating the benefits without potential confounding factors such as hyperparameter adjustments. We also focus on performance rather than pretraining speedups and employ higher budgets (longer pretraining, larger batch sizes). on the full ImageNet dataset and both ResNets and ViTs (Dosovitskiy et al. 2020 ). Other works utilize additional networks for view generation, such as [Tmmkin et al. (2021); Shi et al. (2022); Tian et al. (2020b) (adversarial view generators), Peng et al. (2022) (localization network for semantic awareness). Li el al. (2024) (pretrained generative models to enhance augmentation quality), and Han et al (2023) (leveraging pretrained GAN in SimCLR- only setting). However, unlike HVP, all these methods add non-trivial complexity to the training pipeline by requiring learning auxiliary and adversarial components or are often limited in their applicability across different SSL frameworks (e.g., by requiring negative view pairs).
METHOD
3.1 SELF-SUPERVISED LEARNING FRAMEWORK
In this section, we introduce our approach, which is also depicted in Algorithm | Many different sclf-supervised discriminative learning (He al. (2020) objectives exist, each char- acterized by variations stemming from design choices, such as by the use of positive and negative samples or asymmetry in the encoder-projector network structure. For simplicity of exposure, we will introduce 2021), but we do note that our method can be used with any other discriminative SSL objective that allows the computation of sample-wise losses.
approach based on the SimSiam objective (Chen & He
SimSiam works as follows. Assume a given set images D, an image augmentation dis- tribution T. a minibatch of M images x = {x,) sampled uniformly from D, and two sets of randomly sampled image augmentations A and B sampled from T. SimSiam applies
3

---

Published as a conference paper at ICLR 2025
A and B to each image in resulting in x and x". . Both augmented sets of views subsequently projected into an embedding space with zd = =f((((x))) and h = fo(x") where fo represents an encoder (or backbone) and 9o projector network. SimSiam then
minimizes the following objective:
[()) = (D(ZA, D( D(z),11))
(1)
where D denotes the negative cosine similarity function. Intuitively, when optimizing 0, the embeddings of the two augmented views are attracted to each other.
3.2 PRETRAINING WITH HARD VIEWS
We now formalize how expose the model to more challenging views during pretraining. In a mutshell, Hard View Pretraining extends the random view generation by sampling adversarially harder views during pretraining. Instead of having two sets of augmentations A and B, we now sample N sets of augmentations, denoted as A = {A1, A set A, is sampled from the image augmentation distribution T, and applied to each image in x, resulting in N augmented sets of views xA1, x AAAN Similarly, we obtain N sets of emboddings z A1, AA AA, ........,2 ,2 AN and predictions h Ai, hh...... AA through the encoder and projector networks. We then define a new objective function that seeks to find the pair (I²" xx,") ) of a given image Ia that yields the highest loss:
Each
(x x211 = arg max C(0)).e.
k.
(2)
2 (D(= A441 + (( ,na)))
arg max k
where L( simply denotes a sample-wise variant of Eq. 1
Overall, first generate N aug- mented views for each image x; in the minibatch. Then, we forward these augmented views through the networks and create all combinatori- ally possible (N) pairs of augmented images. Subsequently, we use Eq. 2 to compute the sample-wise loss for each pair. We then select all pairs that yielded the highest loss to form the new hard minibatch of augmented sets Ax. and x <!t,, dis- card the other pairs and use the hard minibatch for optimization. As shown in Algorithm [ we repeat this process in each training iteration.
A Algorihhm 1 Pretraining with Hard Views
1: Input: Number of views N 2 2, batch size M,
2: augmentation distribution T, model
3: for each H; in the sampled batch { do
4: Sample N augmentations: A = (In ~ TJ
5: Create augmented views: x {ka (x::) )
6: Forward all views through
7: Create all (2) view pairs An x A1 k #
8: Add hard pair (n.At. .x. Ai. that maximizes
Eq. to the new batch with only hard pairs
9: Proceed with standard SSL training
10: Repeat for all batches
11: return Pretrained model
Intuitively, our approach introduces a more challenging learning scenario in which the model encouraged to learn more discriminate features by being exposed to harder views. In the early stage of training, the embedding space lacks a defined structure for representing similarity among views. As training progresses, our method refines the concept of similarity through exposure to views that, from the perspective of the model, remain challenging given its current state. By limiting the number of sampled views, we upper-bound the difficulty of learning to prevent tasks from becoming too difficult and hindering learning. This ensures a controlled evolution of the embedding space, where the model's perception of difficulty is continuously challenged in tandem with its growing capacity to differentiate views. Consoquently, HVP can be seen as a regularization that prevents the model from overfitting to casy views.
While we exemplified the integration of HVP with the SimSiam objective, integrating it into other contrastive methods is straightforward. The only requirement of HVP is to
4

---

Published as a conference paper at ICLR 2025
be able to compute sample-wise losses (to select the views with the highest loss). In our experiments section and in addition to SimSiam, we study the application of HVP to the objectives of DINO, iBOT, and SimCLR (see also Appendix K.1 for formal exemplary definition for the integration of HVP into SimCLR).
4 IMPLEMENTATION AND EVALUATION PROTOCOLS
4.1 IMPLEMENTATION
We now describe the technical details of approach. HVP be used with any SSL method that allows computing sample-wise losses, and the only two elements in the pipeline we adapt are: 1) the data loader (which now needs sample N views for each image) and 2) the forward pass (which now invokes select function to identify and return the hard views). The image transformation distribution taken from the basclines is left unchanged. Note, for the view selection one could simply resort to random resized crop (RRC) only and apply the rest of the operations after the hard view selection (see Section 6. for a study on the influence of appearance on the selection).
All experiments were conducted with N 4 sampled views, yielding ( 6 pairs to compare, except for DINO which uses 10 views (2 global, 8 local heads) per default. For DINO, we apply HVP to both global and local heads but to remain tractable, we upper- bound the number of total pair comparisons to 128. SimCLR uses positive and negative samples, Following the simplicity of HVP, we do not alter its objective, which naturally leads to selecting hard views that adversarial to positive and "cooperative" to negative views. For iBOT, we use the original objective as defined by the authors with global views only.
4.2 EVALUATION PROTOCOLS
100 epochs
300 epochs
Method Arch.
We now describe the protocols used to evaluate the performance in our main results section. In self-supervised learn- ing, it is common to assess pretraining performance with the linear evaluation protocol by training linear classifier on top of frozen features or finetuning the features on downstream tasks. Our gen- eral procedure is to follow the baseline methods closely as possible, including hyperparameters and code bases (if re- ported). It is common to RRC and horizontal flips during training and re- port the test accuracy on central crops. Due to the sensitivity of hyperparame- ters, and as done by Caron et al. (2021). we also report the quality of features with simple weighted nearest neighbor classifier (k-NN).
Lin.
k-NN
Lin.
k-NN
DINO
VIT-S/16
73.52
68.80
75.48
72.62
+ HVP
VIT-S/16
74.67
70.72
76.56
73.65
Impr.
+1.15
+1.92
+1.08
+1.03
DINO
RN50
71.93
66.28
75.25
69.53
+ HVP
RN50
72.87
67.33
75.65
70.05
+0.94
+1.05
+0.40
+0.52
SimSiam
RN50
68.20
57.47
70.35
61.40
+ HVP
RN50
68.98
58.97
70.90
62.97
+0.78
+1.50
+0.55
+1.57
SimCLR RN50
63.37
52.83
65.50
55.65
HVP RN50
65.33
54.76
67.30
56.80
+1.96
+ +1.93
+1.80
+1.15
IBOT
VIT-S/16
69.55
62.93
72.76
66.92
+ HVP
VIT-S/16
70.27
62.75
73.99
67.16
+0.73
-0.18
+1.23
+0.24
Table 1: Average top-1 linear and K-NN classifica- tion accuracy on the ImageNet validation set for 100 and 300-epoch pretrainings across 3 seeds.
5 MAIN RESULTS
Here, we discuss our main results on image classification, object detection, and segmentation tasks. All results are self-reproduced using the original baseline code and hyperparameters
(see Appendix Section A for details).
5.1 EVALUATIONS ON IMAGENET
We report the top-1 validation accuracy on frozen features, as well as the k-NN classifier performance, in Table 1 For DINO, additionally compare ResNet-50 (He et al. 2016)
5

---

Published as a conference paper at ICLR 2025
CIFAR10
CIFAR100
Flowers102
iNat 21
Food101
Method
Arch.
Lin.
F.T.
Lin.
F.T.
Lin.
F.T.
Lin.
F.T.
Lin.
F.T.
SimSiam
RN50
82.60
95.50
54.20
77.20
34.27
56,40
32.50
60.30
65.70
83.90
+ HVP
RN50
84.40
96.10
57.10
78.20
38.37
58.90
33.90
60.90
67.10
84.70
Impr.
+1.80
+0.60
+2290
+1.00
+4.10
+2.50
+1.40
+0.60
+1.40
+0.80
DINO VIT-S/16
94.53
98.53
80.63
87.90
91.10
93.20
46.93
53.97
73.30
87.50
+ HVP VIT-S/16
95.13
98.65
81.27
88.23
92.07
93.60
49.03
54.16
74.13
87.91
Impr.
+0.60
+0.12
+-0.63
+-0.33
+00.77
+-0.40
+2.10
+-0.19
+-0.83
+-0.41
Table 2: HVP compares favorably against models trained without it when fine-tuned (F.T.) to or linearly evaluated (Lin.) other datasets (averaged over 3 seeds; 100-ep, preraining).
against the VIT-S/16 (Dosovisskiy et al., 2020) architecture. We point out that both meth- ods, vanilla, and vanilla+ always receive the same number of data samples for training. Our method compares favorably against all baselines with an increased performance of approximately 1% on average for 100 and 300 epoch pretraining, showing the benefit of sampling hard views.
Due to limited computing resources, we run the majority of pretrainings in this paper for 100 epochs and 300 epochs (200 epochs for SimCLR) and batch sizes of 512 (100 epoch) or 1024 (200 & 300 epoch trainings), respectively. This choice is in line with a strategy that favors the evaluation of a diverse and larger set of baselines over the evaluation of a less diverse and smaller set and underpins the broad applicability of HVP. We primarily ran our experiments with SxNVIDIA GeForce RTX 2080 Ti nodes, with which the pretraining and linear evaluation duration ranged from ~3.5 to ~5 days. While HVP requires roughly 2x the training time of the DINO baseline (see Appendix [ for further discussion on the time complexity of HVP), this investment translates directly into improved performance.
Rather than prioritizing efficiency, our focus was on achieving state-of-the-art results, high- lighting HVP's flexibility and robustness across training durations. To showcase this point, we explored the scalability of HVP with larger models and extended training schedules and achieved new state-of-the-art result of 78.8% accuracy in linear evaluation, im- proving over the officially reported baseline of 78.2% on DINO ViT-B/1C (400 epochs). For k-NN classification, the same model similarly surpassed the DINO baseline by 0.85%, reaching 76.95% compared to 76.1%. These results demonstrate the scalability of our method larger models and longer pretraining schedules. We emphasize that HVP insensitive to the baseline hyperparameters and simply reusing the default ones consistently resulted in improvements in the reported magnitudes across all experiments.
5.2 TRANSFER TO OTHER DATASETS AND TASKS
We now report the transferability of features learned with HVP. For all experiments here, we use our 100-epoch ImageNet pretrained iBOT and DINO VIT-S/16 models, respectively.
5.2.1 LINEAR EVALUATION AND FINETUNING
In Table 2 we apply both the linear evaluation (Lin.) and finetuning (F.T.)
Method Arch.
OD
IS
100
300
100
300
protocols our models across a diverse
IOTT
VIT-S/16
66.13
66.80
63.10
63.63
set of datasets consisting of CIFAR10 Krizhevsky| 2009). CIFAR100, Flowers102 Nilsback & Zisserman 2008). Food101 Bossard et al. (2014))) and iNaturalist 2021 NNaturalist 2021 competition dataset). Our results show that the improvements achieved by sampling hard views that observed so far also transfer to other datasets.
НVP
VIT-S/16
66.50
67.13
63.50
61,23
Impr.
+0.37
+0.33
+0.40
+0.60
DINO
VIT-S/16
65.90
66.60
62.83
63.63
+ HVP Impr.
VIT-S/16
66.37
67.00
63.37
64.00
+0.47
+0.40
+..53
+...7
Table 3: Object Detection (OD) and In- stance Segmentation (IS) AP50 performance on COCO for 100/300 epoch pretraining.

---

Published as a conference paper at ICLR 2025
Porcontage pair also
Comparing IoU
2.5
Sndiarr nour muthed
Sincion
2.0
1o.ddctee on in.r0/ee
Deraky
0
20
100
1.0
Spoch
E0
Figure 2: Left: over 40% the cases, the adversarially selected view pair has also the lowest Intersection over Union throughout SimSiam++HV? pretraining. We attribute the carly spike to the random initialization of the embedding. Right: HVP (blue) shows shift to smaller IoU values over standard pretraining (orange). Both results are based on seeds.
5.2.2 OBJECT DETECTION AND INSTANCE SEGMENTATION
For object detection and instance segmentation, we use the COCO (Lin et al. 2014) dataset with Cascade Mask R-CNN (Cai & Vasconcelos) 2019 He et al. 2017]. Table 3 where we report the AP50 performance, shows that the features learned with HVP also transfer favorably to these tasks and outperform the iBOT and DINO bascline with 100-ep. and 300-ep. pretraining. More details and performance results on this task are provided in Appendix H.1
EMPIRICAL ANALYSIS OF HVP
In this section, we discuss studies designed to shed light on the mechanisms underlying HVP. We address the following questions: 1) "Which pattern can be observed that underlies the hard view selection?" and 2) "What are the effects on empowering the adversary?". For all experiments conducted here, use our 100-epoch, ImageNett-pretranne Sim models with four sampled views. In Appendix [.I we also analyze whether we can infer "manual" augmentation policy from the following observed patterns.
6.1
Q1: WHICH PATTERNS CAN BE OBSERVED WITH HVP?
When visually studying examples and the views selected by HVP in Figures 1b and (in the appendix), we notice that both geometric and appearance characteristics seem to be the brightness difference between the views of the first rows in Fig. lb We also see generally higher training loss (Fig. e in the appendix) indicative of an increased task difficulty.
exploited, for instance,
6.1.1 LOGGING AUGMENTATION DATA
To assess these observations, we logged relevant hyperparameter data during SimSiam train- ing with HVP. The logs include for each view the sampled geometric and appearance param- eters from the data augmentation operations (such as the height/wii /width of the crops or the brightness; see Section E in the appendix for more details), well the loss and whether the view was selected. As evaluated metrics, we chose the Intersection over Union (IoU), Relative Distance (normalized distance of the center points views), color distortion distance (the Euclidean distance between all four color distortion parameters), and the individual color distortion parameters.

---

Published as a conference paper at ICLR 2025
Comparinn Average IDU without
Percentage pair IDU pair
0.50
Sindiamindu metrid
0.48
14.0
Cootlameee mettrd
0.46
setecte on aneraee
0.49
0.43
Smale
Sirciam hour miched
0.40
0.38
0.34
20
80
100
20
100
Epech
Figure 3: Left: The average IoU of view pairs selected by SimSiam+HVP (blue) compared against the default SimSiam training (green). Right: Using static color augmentation for all pairs before the selection increases the dependency on the IoU.
6.1.2 IMPORTANCE OF AUGMENTATION METRICS
Given 300k such samples, we then used fANOVA (Hutter et al., 2014) to determine how predictive these metrics are. This resulted in the metric with the highest predictive capacity on the loss being the ToU, explaining 15% of the variance in performance, followed by brightness with 5% (for more details see Fig. in the appendix). The importance of ToU in HVP is further underpinned by the following observation: the fraction of view pairs selected by HVP, which are also the pairs with the lowest IoU among all six pairs (N=4), is over 40% (random: ~16.7%) during training. Morcover, when using HVP, shift to smaller IoU values can be observed when comparing against standard pretraining (see Fig.
6.1.3 TAKING A CLOSER LOOK AT THE INTERSECTION OVER UNION
We also examined the IoU value over the course of training in Fig. 3 (left). An observable pattern is that the IoU value with HVP (Fig. (left) in green) smaller and varies more when compared against training without HVP (blue). We believe this is due to the sample- wise and stateful nature of the adversarial selection as HVP chooses different IoU values between varying samples and model states.
Lastly, we assessed the effect of the color augmentation on the pair selection. For this study, sampled set of color augmentations (as opposed to one for each view) per iteration and applied it to all views. We apply sampled data augmentations to each view only after identifying the hardest pair. As we show in Fig. (right), the fraction of selected pairs that are also the hardest pairs slightly increases in this case. One possible explanation for this is that it reflects the non-negligible role of color variation between views (as shown previously with the importance analysis), where HVP is given less leverage to increase hardness through static appearance and instead, depends more on leveraging the loU, Another key obscrvation is that HVP often chooses view pairs that incorporate zooming in and out or an increased distance between the views (sce last row of Fig. lbb.
6.2 Q2: WHAT ARE THE EFFECTS OF STRONG ADVERSARIES?
is well known that adversarial learning can suffer from algorithmic instability (Xing et al. 2021 e.g. by giving an adversary too much capacity. Here, we further explore the space of adversarial capacity for pretraining with hard views by adapting and varying HVP hy- perparameters in order to gain better understanding of their impact and robustness discriminative learning. Additionally, we report further results on learning an adversary in A Appendix G.T
8

---

Published as a conference paper at ICLR 2025
63.0
Smswn
Smdd method
Imnageee: Latea Svelusdin Aec (5a
68.5
0.08
C12
05.0
67
4/
67.5
Imagelez
ex.0
C12
Smoam
40.00
Sindmm-De melhed
0.50
0.15
0.20
0.25
0:30
8:55
0-40
0.5
1.0
2.0 2.5
Kandon ResiosiCrup Lower Sempling Ronge
Chogliter Stre
Figure 4: With HVP, SimSiam appears more robust to augmentation hyperparameter vari- ation. We show this for values indicate the lower value of the sampling range and for CJ the intensity of the color cues. Results averaged over two seeds and SimSiam defaults are RRC=0.2 and CJ=0.55
(left) and ColorJitter (right). For RRC, the
6.2.1 ROBUSTNESS TO AUGMENTATION HYPERPARAMETERS
In assessing the robustness of HVP to augmentation hyperparameters, we conducted an analysis focusing two primary augmentation operations: RandomBResizedCrop (RRC) and ColorJitter (CJ). Our findings depicted Fig. suggest that HVP enhances the robustness of SSL methods, like SimSiam, against variations in these hyperparameters. Note, when varying one operation, either RRC or CJ, we maintained the other operation at its default configuration. In both settings, we observe less performance degradation with extreme augmentation values and overall smaller degradation rates for HVP. We believe that this robustness stems from hard view pretraining which inadvertently equips the model to handle stronger augmentations.
6.2.2 INCREASING THE NUMBER OF VIEWS
In our initial experiments, explored variations in the number of sampled views N with SimSiam and HVP. As can be seen in Fig. 7 in the appendix, while N = 8 views still outperform the baseline in terms of linear evaluation accuracy, it is slightly worse than using N = 4 views (-0.05% for 100 epochs pretraining on linear evaluation). We interpret this result as the existence of a "sweet spot" in setting the number of views, where, in the limit, a higher number of views corresponds to approximating a powerful adversarial learner, capable of choosing very hard and unfavorable learning tasks that lead to model collapse and performance deterioration. We experimented with such an adversarial learner and report results in Appendix G.T
7 CONCLUSION
We presented HVP, data augmentation and learning strategy for Self-Supervised Learning designed to challenge pretrained models with harder samples. This straight ward method allows pushing the effectiveness of the traditional random view generation in SSL. When combined with methods like DINO, SimSiam, iBOT, and SimCLR, HVP con- sistently showcased improvements of 1% on average in linear evaluation and diverse set of transfer tasks. HVP achieved a new state-of-the-an result of 78.8% linear evaluation accu- racy on DINO VIT-B/16, 0.6% improvement over the previous baseline. This illustrates the scalability and effectiveness of our approach for larger pretraining sottings. With grow- ing models, there is an increasing demand for more data to effectively train them. Synthetic data generation offers viable solution by enhancing the quantity and diversity of training data. Data augmentation techniques like HVP play a crucial role in this process by creating challenging views, which can serve as synthetic data. All in all, HVP holds promise for sce- narios where one seeks to push absolute performance or explore making models less sensitive to hyperparameters, thereby strengthening them for various downstream applications.
9

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
Frank Hutter acknowledges the financial support of the Hector Foundation. We also ac- knowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the au- (h) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Moreover, we acknowledge funding by the Deutsche Forschungsge- meinschaft (DFG, German Research Foundation) under grant number 417962828, by the state of Baden-Württenn Württemberg through bwHPC and the German Research Foundation (DFG) FUGG, well as the Gauss Center for Supercomputing eV
through grant INST www.gauss-centre..o for funding this project by providing computing time on the GCS supercomputer JUWELS at Jülich Supercomputing Center (JSC). We also acknowlodge Robert Bosch GmbH for financial support and the computing time made available on the high-ppeffrmance computer NHROKIT Compute Cluster at the NHR Center NHR@KIT.
Funded by the European Union
REFERENCES
L. Bossard, M. Guillaumin, and L. Van Gool. Food-101 mining discriminative components with random forests. In Proc. of ECCV'14. 2014.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: High quality object detection and instance segmentation. IEEE transactions on pattern analysis and machine intelligence, 43(5):1483 1498, 2019.
M. Caron, L. Misra, J. Mairal, Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proc. of NewrlPS 200 2020.
M. Caron, H. Touvron, 1. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proc. of ICCV'21, pp. 9630 9640, 2021.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection voolbox and benchmark. ar preprint urxiv:1906 0006007155 2019.
T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In Proc. of ICML '20 pp. 1597-1607, 2020a.
X. Chen and K. He. Exploring simple siamese representation learning. In Proc. of CVPR'21, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.0429) 042997, 2020b.
E. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q Le. Autoaugment: Learning augmen- tation strategies from data. In Proc. of CVPR 19, pp. 113 113-123, 2019.
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. ImageNet: Large-Scale 2009.
Hierarchical Image Database. In Proc. of CVPR '9, pp.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint orxii:120100 11929, 2020.
10

---

Published as a conference paper at ICLR 2025
A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe. Whitening for self-supervised repre- sentation learning. In Proc. of ICML '2, pp. 3015-3024, 2021.
J. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, pp. 1189 1232, 2001.
J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Ávila Pires, 2. Daniel Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your latent: A new approach to self-supervised learning. In Proc. of NeurIPS'20, 2020.
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In Proc. of CVPR 06, pp. 1735-1742, 2006.
L. Han, S. Han, S. Sudalairaj, C. Loh, R. Dangovski, F. Deng, P. Agrawal, D. Metaxas, L. Karlinsky, T.-W. Weng, el al. Constructive assimilation: Boosting contrastive learning performance through view generation strategies. arXiv preprint ar Xrx::22304. X00622344000001, 2023.
R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. In Proc. of ECCV'20, pp. 1-16, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. of CVPR'IG, pp. 770 778, 2016.
K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. In Proc. of ICCV'17, pp. 2980-2963 2988, 2017.
K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. of CVPR'20, pp. 9726 9735, 2020.
D. Ho, E. Liang, X. Chen, 1. Stoica, and P. Abbeel. Population based augmentation: Elfi- cient learning of augmentation policy schedules. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 2731-2741, 2019.
C. Hou, J. Zhang, and T. Zhou. When to learn what: Model-adaptive data augmentation curriculum. CoRR, abs/ /2022409, 2023.
F. Hutter, H. Hoos, and K. Leyton-Browt Brown. An efficient approach for assessing hyperparam- cter importance. In Proc. of ICML (C, pp. 754-762, 2014.
iNaturalist 2021 competition datasct. iNaturalist 2021 competition dataset. http://iiiieddaaatttcooooopo hatpa/iitttttaaeeeeeeeee httpr//p///eeee/atte////000 2021.
https: //
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukeuoglu. Spatial transformer net- works. In Proc. of NeurlPS'15, pp. 2017 2022,, 2015.
M. Taha. Kogyigit, T. M. Hospedales, and H. Bilen. Accelerating self-supervised learning via efficient training strategies. In Proc. of WACV, pp. 5643-5654 IEFF, 2023.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University Toronto, 2009.
X. Li, Y. Yang, X. Li, J. Wu, Y. Yu, B. Ghanem, and M. Zhang. Genview: Enhancing view quality with pretrained generative model for self-super learning. In ECCV, pp. 306-325 2024.
C. Lin, M. Guo, C. Li, X. Yuan, W. Wu, J. Yan, D. Lin, and W. Ouyang, Online hyper- parameter learning for autemmttmmnnnioonn strategy. In Proc. of ICCV'19, pp. 6578 -587, 2019.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zitnick. Microsoft. COCO: common objects in context. In Proc. of ECCV'14, pp. 740-755. 2014.
11

---

Published as a conference paper at ICLR 2025
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proc. of ICLR'IS 19, 2019.
S. Müller and F. Hutter. Trivialaugment: Tuning-free yet state-oftthe-at data augmenta- tion. In Proc. of ICCV'21, pp. 774 782, 2021.
M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proc. of ICVGIP'0R 08, pp. 722-729, 2008.
A. Paszke, S. Gross, F. Massa, A. Lerer, et al PyTorch: An imperative style, high- performance deep learning library. In Proc. of NeurlPS'19 pp. 8024 8035, 2019.
X. Peng, K. Wang, Z. Zhu, M. Wang, and Y. You. Crafting better contrastive views for siamese representation ion learning. In Proc. of CVPR 22, pp- 16031-16040, 2022.
S. Purushwalkam and A. Gupta. Demystifying contrastive sclf-supervised learning: Invari- ances, augmentations and dataset biases. In Proc. of NewrIPS 20, 2020.
Y. Shi, N. Siddharth, P. H. S. Torr, and A. R. Kosiorek. Adversarial masking for self- supervised learning. Proc. of ICML 22, volume 162, pp. 20026 20040, 2022.
A. Tamkin, M. Wu, and N. D. Goodman. Viewmaker networks: Learning views for unsu- pervised reprosentation learning. In Proc. of ICLR 21, 2021.
Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding, In Proc. of ECCV'20, pp. 776-794 694, 2020a.
Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? In Proc. of NeurIPS 2020b.
A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807 10000887.33748, 2018.
D. Wagner, F. Ferreira, D. Stoll, R. T. Schirrmeister, S. Müller, and F. Hutter. On the importance of hyperparameters and data augmentation for self-supervised learning. In Pre- Training Workshop the International Conference for Machine Learning (ICML), 2022. URL //ttps://icm cc/ vtt.tiall/002/206655
M. Wu, C. Zhuang, M. Mosse, D. Yamins, and N. D. Goodman. On mutual information in contrastive learning for visual representations. arXii:2005.13143 Cvs, 2020.
Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance-level discrimination. In Proc. of CVPR'1 18, 2018.
Y. Xing, Q Song, and G. Cheng. On the algorithmic stability of adversarial training. In Proc. of NeurIPS'21, pp. (6523-265332 2021.
Y. You, I. Gitman, and B. Ginsburg, Large batch training of convolutional networks. arXiv preprint orXiv: 170.003888 2017.
J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Proc. of ICML 21, pp. 12310-12320, 2021.
X. Zhang, Q. Wang, J. Zhang, and Z. Zhong. Adversarial AutoAugment. In Proc. of ICLR '00 2020.
J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. L. Yuille, and T. Kong. ibot: Image BERT pre-training with online tokenizer. In Proc. of TCML, 22, 2021.
12

---

Published as a conference paper at ICLR 2025
APPENDIX
A REPRODUCIBILITY STATEMENT
We provide complete code, environment installation instructions. hyperparame- ter settings and model chockpoints here: https://anonymous serrennsciennerr/ gestraining-har-vieeee
For transparency, we outline all hyperparameters, data splits, and evaluation protocols in detail in Section | Most experiments were run across multiple seeds, and we report average results to account for variability. Information regarding the required computational resources is discussed in Section J below.
B EXAMPLES SAMPLED BY HVP
Figure 5: We depict row-wise ten example images from the ImageNet train set along with their sampled views with a finished, 100-epoch trained SimSiam ResNet50 model. Left: original image with the overlaid randomly sampled crops (colored dashed rectangles). Right: All views after applying resizing and appearance augmentations. The pair that is selected adversarially by HVP is highlighted in solid lines, eg. View I and View 4 in the first row.
13

---

Published as a conference paper at ICLR 2025
C
TraINING Loss
Training Comparison
DIND HVP
3-
20
100
#poch
Figure 6: The training loss over 100 epochs. Comparing the DINO vanilla method with DINO + HVP. The spike and drop in the loss curve of DINO is caused by freezing the last layer in the first epoch which was proposed by the authors as a strategy to enhance downstream performance. For HVP we only because HVP exposes the model to hard views from the beginning training (i.c. the loss is immediately maximized).
drop and no spike. We believe this is
D EFFECT OF MORE VIEWS ON LINEAR EVALUATION PERFORMANCE
Effect of number of views on linear evaluation
100 Epochs
200 Epochs
76.5
70.0
09.5
ImageMet
68.0
67.5
67.0
SimSiaas (taseline)
Sim. HVS (4 views:
Sim. HVS (8 views]
SimSiam (baseline) Sim. + HVS (4 views)
Sim. + HVS (8 views)
Figure 7: Setting the number of views too high can result in performance deterioration. This shows that diminishing returns exist, likely because the adversary becomes strong, resulting in too hard learning lask.
E ASSESSING THE IMPORTANCE OF METRICS WITH FANOVA
To assess the importance of various metrics on the training loss, we apply FANOVA Hutter et al. (2014) on data that we logged during training with HVP. We used 300k samples that contain the following sampled parameters from the geometric and appearance data augmentation operations for each view: all random resized crop parameters (height and width of the original image, coordinates of crop corners and height and width of the crop), all Colorjitter (color distortion) strengths (brightness, contrast, saturation, hue), grayscale on/off, Gaussian blurring onfoff, horizontal flip on/off, loss, and if the crop was selected
14

---

Published as a conference paper at ICLR 2025
ToU Policy Type
SimSiam
DINO
Baseline (B)
68.20
73.50
(++ragee0.3300.33
-0.80
-1.47
(033.33 -alt.
+0.10
-0.45
B #+rage(0..400...
+00.55
-0.40
B (0..e((0.4-0.6) (1.4-0.6)+alll +all.
+.0.55
-0.20
B range(0.1-0.5
-33.95
-1.50
B B+range(1.0-0.)
+0007
Table 4: Top-1 lin. eval. accuracies for the manual IoU policy.
not. The metrics chose Intersection over Union (IoU), Relative Distance (sample- wise normalized distance of the center points of crop pairs), color distortion distance (the Euclidean distance between all four color distortion operation parameters, i.c. brightness, contrast, saturation, huc), and the individual color distortion parameters of the Colorjitter operation. As can be seen in Fig. the metric with the highest predictive capacity on the loss is the IoU with an importance of 15.2% followed by brightness with 5.1%. The relative distance has an importance of 3.3%, the Colorjitter distance 2. 2.3%, the contrast 1.6%, the saturation 1.4%, hue 0.6%, and all parameters jointly 1.7%.
Importance of Metrics When Applying HVS
mozorancc
Coloritterr Bightresa
Menrics
Figure Application of [ANOVA Hutter al. (2014) on logged training data to determine metrics with high predictive capacity the training loss.
F ADDITIONAL EMPIRICAL ANALYSIS
F.1 CAN A MANUAL AUGM. POLICY BE DERIVED?
Since harder pretraining tasks seem beneficial according to observations made in Q1, natural question arises: can we mimic the adversarial selection with a manually scripted augmentation policy? Such policy would replace HVP and lower the computational cost. Since the ToU plays an important role, below, we study several ways to construct simple manual augmentation policy based on IoU.
F.1.1 DERIVING AN AUGMENTATION POLICY
We implemented the following rejection sampling algorithm in the augmentation pipeline: we linearly approximate the loU values from Fig. (left; in blue) with start and end values of 0.30 and 0.35 (ignoring the dip in the early phase). For each iteration, the pairs exceed the loU value and if so, we reject the pair and re-sample a new pair. This ensures that only pairs are sampled that entail minimum task difficulty (by means
then check
15

---

Published as a conference paper at ICLR 2025
of small enough IoU). We varied different hyperparameters, e.g., IoU start/end ranges, inverse schedules, and alternating between the IoU schedule and the standard augmentation Training both SimSiam and DINO models for 100 epochs yielded performance drops or insignificant improvements (see Table 4 These results indicate that using a manual policy based on metries in pixel space such the ToU is non-trivial. Additionally, these results show that transferring such policy from SimSiam to DINO does not work well, possibly due to additional variations in the augmentation pipeline such as multi-crop. In contrast, we believe that HVP is effective and transfers well since it 1) operates on a similarity level of latent embeddings that may be decorrelated from the pixel space and 2) has access to the current model state.
F.1.2 ASSESSING THE DIFFICULTY OF PREDICTING THE PAIR SELECTED BY HVP
We further validate the previous result and assess an upper limit on the performance for predicting the hardest pair. By fitting gradient boosting classification tree Friedman (2001), we predict the selected view pair conditioned on all SimSiam hyperparameter log data from Q1 except for the flag that indicates whether view was selected. As training and test data, we used the logs from two seeds (each 300k samples) and the logs from a third seed, respectively. We also tuned hyperparameters on train train/valide /aalid splits and applied a 5-fold CV. However, the resulting average test performance in all scenarios never exceeded 40%, indicating that it is indced challenging to predict the hardness of views basod on parameter- level data. This further supports our hypothesis that deriving policy for controlling and increasing hardness based on geometric and appearance parameters is non-trivial and that such a policy must function per-sample basis and have access to the current model state as in HVP.
G LEARNING VIEW GENERATION
G.1 ADVERSARIAL LEARNER
In this experiment, we explore adversarially learning a network to output the transforma- tion matrix for view generation. We use Spatial Transformer Network (STN) Jaderberg lt al. (2015) to allow generating views by producing 6D transformation matrices (allowing translating, rotating, shearing, scaling, affine transformations, and combinations thereof) in a differentiable way since most common augmentations are not off-the-shelf differentiable. We optimize the STN jointly with the DINO objective and ViT-tiny. We train it alongside the actual pretrained network using the same (inverted) objective. For our experiments, we use DINO with multi-crop, i.e. 2 global and 8 local heads. As STN we use small CNN fol- lowed by a linear layer for outputting the 10*6D transformation matrices. In this scenario, use ViT-tiny/1n 16 with 300 epoch pretraining on CIFAR10 with batch size of 256. All other hyperparameters are identical to the ones reported in the DINO paper.
Figure visualizes the procedure. The STN takes the raw image input and generates number of transformation matrices that are applied to transform the image input into views. These views are then passed to the DINO training pipeline. Both networks are trained jointly with the same loss function. DINO is trained with its original contrastive objective, where the STN is trained by inverting the gradient after the DINO during backpropagation.
The STN, without using auxiliary losses, starts zooming in and generating single-color views. To counteract this behavior, we experimented with different penalties on the transformation matrices produced by the STN, For instance, in order to limit the zooming pattern, we can use the determinants of the sub-matrices of the transformation matrix to penalize based on the area calculated and apply a regression loss (e.g. MSE). We refer to this type of penalty as Theta Crops Penalty (TCP). Additionally, we also restrict its parameters to stay within sphere with different parameters for local and global crops. Next to determinant-based penalty losses, we also experimented with other penalty functions such as the weighted MSE between the identity and the current transformation matrix or penalties based on histograms of the input image and generated views after applying the transformation. To avoid strong uni-dimensional scaling behavior, we also implemented restricting scaling in symmetric
16

---

Published as a conference paper at ICLR 2025
maxinize
augnn
Iuput
STN
Crops
Views
DINO
Loss
Penalty
miniiize
Figure 9:
Illustration of adversarial learning with Spatial Transformer Network (STN)
jointly with contrastive learning (here: DINO).
way (i.e. applied both x and y dimensions) and refer this as scule-sym.. We report our best results in Table 5 which are all TCP-based. As can be seen, no setting is able to outperform the baseline. Our best score was achieved with pranslatio--cale-symmmeen which is very similar to random cropping. When removing the symmetries in scaling, the performance drops further. Removing constraint adds one transformation parameter and therefore one dimension. This be scen giving more capacity to the adversarial learner which in turn can make the task significantly harder. Similarly, when adding rotation, the performance drops further and in part drastically. This is on the penalties not being fully able to restrict the output of the STN. On the other, the task of extracting useful information from two differently rotated crops is even harder, and learning spatial invariance becomes too challenging. All in all, we experienced two modes: either the STN is too restricted, leading to static output (i.e. independent of image content, the the STN has too much freedom, resulting in extremely difficult tasks. See Fig. 10 for an example of the former behavior.
hand due to the
STN would produce constant transformation matricos)
orig
global 1
global 2
local 1
local 2
local 3
local 4
local 5
local 6
local 7
local 8
orig
global
global 2
local 1
local 2
local 3
local 4
local 5
local 6
local 7
local 8
orig
global
global 2
local 1
local 2
local 3
local 4
local 5
local 6
local 7
local 8
Figure 10: Example for static outpul behavior of the STN.
Mode
Penalty
Lin.
F.T.
baseline
86.1
92.7
translationn-caale-ssy.
TCP
83.7
90.3
translation-scale
TCP
82.8
89.7
rotation-traanslttio
TCP
56.7
rotation-trranllttionsscl
TCP
31.7
rolation-tranllaioo-vvalssss
TCP
77.6
affine
TCP
78.3
83.5
Table 5: Linear evaluation and finetuning classification performance on CIFAR10. Top-1 accuracy on the validation set of CIFAR10 for our best results reported with different STN transformation modes.
17

---

Published as a conference paper at ICLR 2025
G.2 COOPERATIVE LEARNER
To investigate the effect of cooperalive, i.e. easy pair selection, we conducted a small experiment. Instead of selecting the pair yielding the worst loss, we inverted the objective and selected the pair with the best loss. As expected, this led to model collapses with linear eval. performance of 0.1%. This result in line with previous findings that highlight. the importance of strong augmentations in CL.
H ADDITIONAL, RESULTS
H.1 ОвJЕCТ DETECTION AND INSTANCE SEGMENTATION
Here we provide more detailed results on object detection and instance segmentation, shown
Tab. G and Tab. respectively. We followed iBOT's default configurations which
employed Cascade Mask R-CNN as the task layer.
Method
Arch.
Object Detection
Instance Segmentation
Apb
AP
AP 75
Apm
APm 50
AP75
50
75
iBOT
VIT-S/16
47.00
66.13
50.63
40.67
63.10
43.37
HVP
VIT-S/16
47.27
66.50
50.90
40.90
63.50
43.83
Improvement
+0.27
+0.37
+0.27
+0.23
+0.40
+0.47
DINO
ViT-S/16
46.50
65.90
50.30
40.43
62.83
43.27
+ HVP
VIT-S/16
47.07
66.37
50.63
40.80
63.37
43.87
Improvement
+0.57
+0.47
+0.33
+0.37
+0.53
+0.60
Table 6: Additional object detection and instance segmentation results on the COCO dataset. The ViT-S/16 models were pretrained for 100 epochs.
Method
Arch.
Object Detection
Instance Segmentation
Apb
APb
APh
Apm
APm
APm
60
50
iBOT
ViT-S/16
47.60
66.80
51.33
41.10
63.63
44.07
HVP
ViT-S/16
48.03
67.13
51.73
41.50
64.23
44.40
Improvement
+0.43
+0.33
+0.40
+0.40
+0.60
+0.33
DINO
ViT-S/16
47.27
66.60
51.00
41.00
63.63
44.03
+ HVP
VIT-S/16
47.50
67.00
51.33
41.17
64.00
44.13
Improvement
+0.23
+0.40
+0.33
+0.17
+0.37
+0.10
Table 7: Additional object detection and instance segmentation results on the COCO dataset. The VIT-S/16 models were pretrained for 300 epochs.
I HYPERPARAMETERS
L1 EVALUATIONS ON IMAGENET
L11.1 DINO
For DINO, we report the VIT pretraining hyperparameters Table g (VIT-S) and Table (VTT-B) which are the original ones reported by the authors. Note, for HVP we limit the total number of comparisons to 128 across all heads. Lincar evaluation is executed for
9
100 epochs and use learning rate of 0.00075, SGD optimizer (AdamW Loshchilov & Hutter (2019) during pretraining), a batch size of 1024, a momentum of 0.9, and no weight decay as reported by the authors).
18

---

Published as a conference paper at ICLR 2025
Hyperparameter
Value
Hyperparameter
Value
architecture
vit-small
epochs:
100
img-size
224
warmup-epochs:
10
patch-size
16
freeze-last-layer:
out-dim
65536
Ir:
0.0005
norm-last-layer
true
min-lr:
1.0e-06
momentum-teadee
0.996
optimizer:
AdamW
use-bn-in-head
false
weight-decay:
0.04
leacher-lemp
0.04
weightt-lecay-entt
0.4
warmuuptteacher-teem
0.04
global-crops-ccalee
0.4. 1.0
varmuuptteccher--mm--pooosss
0
global-crops-sinn 224
fp16
true
local-crops-munilee number:
batch-size
512
local-crops-scatl
0.05, 0.4
clip-grad
3.0
local-crops-siec:
96
drop-path-rate
0.1
Table 8: Pretraining ImageNet hyperparameters for the runs with DINO VYT-S/16. For
300 epochs, we use a batch size of 1024.
Hyperparameter
Value
Hyperparameter
Value
architecture
vit-base
epochs:
400
img-size
224
warmup-epochs:
10
patch-size
16
freczo-last-layer:
3
out-dim
65536
Ir:
0.00075
norm-last-layer
true
min-lr:
2.0e-06
momentum-teaeee
0.996
optimizer:
AdamW
use-bn-in-head
false
weight-decay:
0.04
teacher-temp
0.07
weight-decay-end:
0.4
warmuppteaccher-temp
0.04
global-crops-scale:
0.25, 1.0
warmunp-teachee-temp-pppeh
50
global-crops-size 224
fp16
false
local-cropp seealcrops-numblee number:
10
batch-size
1024
local-crops-scaler
0.05, 0.25
clip-grad
0.3
local-cross-sizc:
96
drop-path-rate
0.1
Table 9: Pretraining TmageNet hyperparameters for the runs with DINO VIT-B/16.
1.1.2 SIMSIAM
In Table 10 we report the ResNet-550 pretraining hyperparameters. Linear evaluation is executed for 90 epochs (as reported by the SimSiam authors) and we use a learning rate of 0.1, LARS optimizer You et al. (2017), a batch size of 4096, and no weight decay.
I.1.3 SIMCLR
We report the ResNet-50 pretraining hyperparameters for SimCLR in Table 11 Linear evaluation is executed for 90 epochs with learning rate of 0.1, SGD optimizer, batch size of 4096, and no weight decay.
1.2 TRANSFER TO OTHER DATASETS AND TASKS
For linear evaluation on the transfer datasets, we simply used the same hyperparametens for linear evaluation ImageNet (DINO and SimSiam respectively). For finetuning DINO ViT-S/16, we used the hyperparameters reported in Table 12 and for SimSiam ResNet-50 used the hyperparameters in Table 13
19

---

Published as a conference paper at ICLR 2025
Hyperparameter
Value
architecture
resnet50
batch-size
512
blur-prob
0.5
crops-scale
0.2, 1.0
crop-size
224
featuro-dimension
2048
epochs
100
fix-pred-Ir
true
Ir
0.05
momentum
0.9
predictor-dimension
512
weight-decay
0.0001
optimizer
SGD
Table 10:
Pretraining ImageNet hyperparameters for the runs with SimSiam. For 300
epochs, a batch size of 1024.
Hyperparameter
Value
architecture
resnet50
proj-hidden-dim
2048
out-dim
128
use-bn-in-head
true
batch-size
4096
optim
LARS
0.3
sqrt-Ir
false
momentum
0.9
weight-decay
le-4
epochs
100
warmup-epochs
10
zero-init-residual
true
Table 11
Hyperparameter
CIFAR10
CIFAR100
Flowers102
iNat 21
Food101
7.5e-6
7.5e-6
5e-5
5e-5
5e-5
weight-decay
0.05
0.05
0.05
0.05
0.05
optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
epochs
300
300
300
100
100
batch-size
512
512
512
512
512
Table 12: Finetuning hyperparameterss for DINO ViT-S/16.
Hyperparameter CIFAR10
CIFAR100 Flowers102
iNat 21
Food101
7.5c-6
50-6
5c-4
7c-5
5c-6
weight-decay
0.05
0.05
0.05
0.05
0.05
optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
epochs
300
300
300
100
100
batch-size
512
512
512
512
512
Table 13: Finetuning hyperparameters for SimSiam and ResNet-50.
20

---

Published as a conference paper at ICLR 2025
1.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
Our experiments utilized Open MMLab's detection library Chen et al. (2019) for object
delection and instance segmentation COCO Lin el al. 2014 default configuration.
We lollowed iBOT's
Obj. Det. & Inst.
Segm. COCO
Hyperparameter
Value
epochs
12
batch-size
32
0.02
Table 14: Hyperparameters object detection and instance segmentation COCO.
COMPUTATIONAL, OVERHEAD OF HVP
The additional forward passes that HVP introduces for the selection phase increase the time complexity of the individual baseline methods. Several possible approaches exist. to mitigate this overhead, one which is to alternate between the vanilla and HVP training step. We measured the overhead factors for different alternating frequencies (i.e., after how many training steps we use the hard views from HVP; we refer to this as step) for SimSiam, Below, we propose additional ways that
DINO, and iBOT which we report in Table 15 may allow using hard views more efficiently.
SimSiam (RN50)
DINO (VIT-S/16)
DINO (RN50)
iBOT (VIT-S/16)
Step
1 (HVP)
x1.64
x2.21
x2.01
x2.13
x1.38
x1.61
x1.56
x1.57
3
x1.32
x1.43
x1.42
x1.38
x1.29
x1.34
x1.35
x1.29
Table 15: Slowdown factors for HVP and the alternating training method, where step refers to the interval at which HVP applied during training (i.c., step=1 refers to full HVP and step=3 indicates that HVP is approx. used 33% of the training time). Measurements averages across 3 seeds.
As an alternative to the alternating training, we also experimented with 50% image resolu- tion reduction for the selection phase but observed that the final performance was negatively affected by it or that baseline performance could not be improved.
The details of hardware and software used for this analysis are: one single compute node with 8 NVIDIA RTX 2080 Ti, AMD EPYC 7502 (32-Core Processor), 512GB RAM, Ubuntu 22.04.3 LTS, PyTorch 2.0.1, CUDA 11.8. For DINO's global and 8 local views (default), applying HVP with nviews=2 sampled for cach original view results in 4 global and 16 local views. Since considering combinations would yield over 77k unique comparisons (0)x x (26).. to remain tractable, we limit the nunber of total comparisons to 64. For training experiments that exceed the limit of 8x RTX 2080 Ti, we apply gradient accumulation.
While technically there be memory overhead with HVP, with the number of sampled views chosen in this paper, the backward pass of the methods that compute gradients only for the sclected view pair still consumes more memory than the selection part of HVP (even for 8 sampled viows in SimSiam). Note, that selection and the backward computation are never executed at the same time but sequentially.
We emphasize that further ways exist to optimize HVP's efficiency which remain to be explored. For instance:
21

---

Published as a conference paper at ICLR 2025
using embeddings of views from "carlier" layers in the networks
using 4/8 bit low-precision for the view selection or
using one GPU just for creating embeddings and selecting the hardest views while the remaining GPUs are used for learning or
other approaches derive manual augmentation policies
bypassing forwarding of similar pairs.
HARD VIEW PRETRAINING OBJECTIVES K
K.1
SIMCLR
In this section, we are going to introduce the application of HVP with the SimCLR objective. Assume given set of images D, an image augmentation distribution T, minibatch of M images { sampled uniformly from D, and two sets of randomly sampled image augmentations A (t, ~ T==1 and B sampled from T. We apply A and B to each image in x resulting in x and 8 Both augmented sets of views are subsequently projected into embedding space with 2 {x({o(x)) and 2 gx((o(x where so represents encoder (or backbone) and go a projector network. Contrastive learning algorithms then minimize the following objective function:
C(T, x:0) = log exp(sim (z, ,2 //T//
Σijn exp(sim(22 2 (2/22/)/))
where denotes a temperature parameter and sim a similarity function that is often chosen cosine similarity. Intuitively, when optimizing 6, embeddings of two augmented views of the same image are attracted to each other while embeddings of different images are pushed further away from each other.
To further enhance the training process, we introduce a modification to the loss function where instead of having two sets of augmentations A and B, we now have 'N" sets of augmentations, denoted as A = Ag.... An}. Each set A; is sampled from the image augmentation distribution T, and applied each image in x, resulting in 'N' augmented sets of views A x Av.
Similarly, obtain N sets of embeddings AA, 2 ,...,,, ....,,AA through the encoder and pro- jector networks defined as:
- = -=(x((((xx))) i= 1,2,......
We then define a new objective function that seeks to find the pair of augmented images that yield the highest loss. The modified loss function is defined as:
Lmax (T, x, max C(T, x; 0) kIik
where
C(T,x;0) log exp(sim(z)) (z 2,11 )/T) Eidn exp(sim (2 .2,11 (/1)/T)
and k. E (1, N} and i,j
For each iteration, we evaluate all possible view pairs and contrast each view against every other example in the mini-batch. Intuitively, the pair that yields the highest loss is selected, which is the pair that at the same time minimizes the numerator and maximizes the de- nominator in the above equation. In other words, the hardest pair is the one, that has the lowest similarity with another augmented view of itself and the lowest dissimilarity with all other examples.
22


Reviews:
[{"review_id": "tbkH5dJGSU", "reviewer": "Reviewer_xunC", "summary": "This paper proposes Hard View Pretraining (HVP), an approach for improving self-supervised learning (SSL) by selecting challenging, high-loss views during pretraining. Unlike conventional SSL methods that rely on random augmentations, HVP samples multiple views for each input, computes the pairwise loss, and selects the view pair with the highest loss. This adversarial selection of views enables the model to learn more robust representations by iteratively exposing it to more difficult training examples. The method integrates seamlessly with popular SSL frameworks, including DINO, SimSiam, iBOT, and SimCLR, demonstrating consistent performance gains across various models and transfer tasks. HVP achieves state-of-the-art results on the DINO ViT-B/16 model with a 0.6% improvement in linear evaluation accuracy (78.8%) and demonstrates its scalability across different architectures and datasets.", "questions": "1. How does HVP address the potential issue of false pairs in hard view selection? Were any additional measures considered for identifying or filtering these pairs?\n2. Could the authors further clarify HVP’s unique advantages compared to existing view-construction methods and provide relevant experimental comparisons?\n3. Given HVP’s reliance on loss maximization, which may limit its use in tasks with complex pairwise losses (e.g., MAE or relational consistency tasks like Relational KD), could the authors discuss potential adaptations to extend HVP’s applicability?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- HVP introduces a straightforward, loss-based hard view selection mechanism that enhances SSL training without requiring additional components or extensive hyperparameter tuning. This simplicity makes it highly practical for integration into existing SSL pipelines.\n- The paper presents extensive experiments across different SSL frameworks, model architectures (e.g., CNNs and Vision Transformers), and datasets. This broad evaluation supports the generalizability of HVP and its effectiveness in improving SSL methods.\n- The method is demonstrated on large-scale datasets such as ImageNet and COCO, where it consistently outperforms baselines, particularly in challenging settings. HVP’s strong performance on both linear evaluation and transfer tasks, including object detection and segmentation, highlights its robustness and adaptability to diverse downstream applications.", "weaknesses": "1. HVP’s reliance on high-loss pair selection may result in false positive pairs (i.e., views from different instances within the same image) being chosen, which could hinder representation learning. The paper does not clearly address whether the current HVP method can effectively avoid or mitigate this issue.\n2. The related work section does not thoroughly discuss other existing view construction methods such as [1,2,3,4] nor does it compare HVP with these methods experimentally. The absence of experimental comparisons with these methods limits the paper’s ability to demonstrate HVP’s distinct advantages in SSL.\n3. HVP’s reliance on loss maximization may limit its effectiveness in tasks where pairwise loss is not a straightforward metric, such as pixel-level reconstruction tasks (e.g., Masked Autoencoders[5]) or relational consistency tasks like Relational Knowledge Distillation[6]. The paper could explore adaptations to broaden HVP’s applicability in such contexts.\n\n[1]Tamkin, Alex, et al. “Viewmaker networks: Learning views for unsupervised representation learning.” CVPR 2020.\n\n[2] Peng, Xiangyu, et al. \"Crafting better contrastive views for siamese representation learning.\" CVPR 2022.\n\n[3] Han, Ligong, et al. \"Constructive assimilation: Boosting contrastive learning performance through view generation strategies.\" arXiv:2304.00601.\n\n[4]Li, Xiaojie, et al. “GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learning.” ECCV. 2024.\n\n[5] He, Kaiming, et al. “Masked autoencoders are scalable vision learners.” CVPR. 2022.\n\n[5]Zheng, Mingkai, et al. “Weak Augmentation Guided Relational Self-Supervised Learning.” TPAMI 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnCF8AykTO", "reviewer": "Reviewer_mBDk", "summary": "The main content of this article is to introduce a new self supervised learning (SSL) pre training method called Hard View Pretraining (HVP). The core idea of this method is to enhance the learning performance of the model by selecting views with higher difficulty (i.e. views that generate higher loss). The HVP strategy includes the following iterative steps:\n\n1. Randomly sample multiple views and propagate each view forward through a pre trained model.\n\n2. Create a pair of two views and calculate their loss.\n\n3. Adversarially select the view pairing that generates the highest loss based on the current model state.\n\n4. Perform backpropagation on the selected view pairing.\n\nIn addition, the author also explores the computational cost of HVP, its integration with existing methods, and how to optimize the efficiency of HVP.", "questions": "Please refer to weakness for the details.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative approach: A new self supervised learning pre training method HVP has been proposed, which improves the model's generalization ability by selecting difficult views. This is a novel research direction.\n\n2. Wide applicability: The HVP method is not only applicable to one SSL method, but can be integrated into various popular SSL frameworks such as SimSiam, DINO, iBOT, and SimCLR, demonstrating good compatibility.\n\n3. Significant performance improvement: HVP has shown better performance than existing methods on multiple datasets and tasks, particularly achieving new optimal accuracy on DINO ViT-B/16.", "weaknesses": "Although this article proposes a promising self supervised learning pre training method HVP and demonstrates its effectiveness on multiple tasks, there are also some potential shortcomings:\n\n1. Computational cost: The HVP method requires additional forward propagation to select the most difficult view pairs, which may increase the computational cost of training, especially on large-scale datasets and complex models. Please try to compare the computational cost of proposed approach and existing ones for a more comprehensive performance evaluation.\n\n2. Hyperparameter adjustment: Although HVP does not require adjusting too many hyperparameters, some adjustments may still be needed in determining the number of views and selecting the most difficult view pairs, which may require additional experimental and computational resources.\n\n3. Validation of generalization ability: Although the article demonstrates the effectiveness of HVP on multiple datasets and tasks, further validation is needed for its generalization ability on a wider range of tasks and datasets.\n\n4. Theoretical analysis: The article mainly focuses on experimental results, and the theoretical analysis behind why HVP is effective may not be in-depth enough, especially in understanding how the model learns useful features from difficult views. Maybe a proof of underlying theory (e.g., how HVP affects the geometry of the learned feature space, or providing theoretical bounds on the expected improvement from using hard views) will be helpful.\n\n5. Memory overhead: In some cases, HVP may increase memory overhead, especially when dealing with a large number of views and complex models, which may limit its application in resource constrained environments.\n\n6. Stability of opponent models: When exploring the capacity of opponent models, the article mentions the issue of algorithm stability, indicating that in some cases, HVP may be affected by model crashes. Please conduct a systematic study of how different adversarial strengths affect training stability and performance.\n\n7. Dependence on existing processes: Although HVP can be integrated into existing SSL frameworks, its effectiveness may depend on specific data augmentation distributions and model architectures, which may limit its applicability in different settings.\n\n8. Diversity of experimental design: Although the article provides a wide range of experiments to validate the effectiveness of HVP, more experiments may be needed to explore the performance of HVP under different conditions, such as different learning rates, batch sizes, and training period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkgdAisRgu", "reviewer": "Reviewer_AeuA", "summary": "The authors have proposed and validated an augmentation  method to train SSL models. The methods finds hardest views (HVP) based on loss in SSL to refine the learned representations. Thorough investigation has been done with DINO, SimSiam, SimCLR. \nThe method is simple and easy to incorporate in existing SSL pipelines. Transfer learning experiments have been done, many datasets are used.", "questions": "Follow strengths and weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written, so many experiments have been done. \nThe method is simple and easy to incorporate in existing SSL pipelines, maybe seen as a plug-n-play method.", "weaknesses": "When we choose hardest views based on loss value, then it is certainly encouraging few augmentation strategies over others which defies the purpose of randomness of augmentations. \nSo, it may results performance improvement on unseen example (validation set) of the dataset ion which model is trained however, generalizability of model become questionable in reference to  domain adaptation. \n\nInitially the model parameters are not effective, therefore, the higher loss may not be a good indicator of hard view. \n\nAs per Table 1, it is evident that longer pretraining is improving the performance even with original methods. Now, computation analysis suggests HVP approximately require 2x time of computation than original method (simSiam, simCLR, DINO iBOT), respectively. It conveys if original methods are pretrained twice number of epochs then they consume same computation resources like HVP. So, maybe, a fair comparison from computation perspective to compare HVP vs Original methods might require pretraining double. Like DINO with 100/300 epochs improves.  \n\nRandom crop is dominant augmentation to receive the hard view as per loss, it introduces , i) information loss in visual concept, ii) spatial characteristics are randomly changed. Thus, it is important to understand whether hypothesis behind HVP stand without random crop. \nThe nonlinear transforms such as MPD and LCM two very recent augmentations without crop. It would be good to comment on such augmentations. \nMPD: Möbius Transform for Mitigating Perspective Distortions in Representation Learning\nLCM: Log Conformal Maps for Robust Representation Learning to Mitigate Perspective Distortion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4peMHHPSLb", "reviewer": "Reviewer_k3VL", "summary": "The paper introduces Hard View Pretraining (HVP), a novel self-supervised learning method aimed at improving pretraining pipelines by selecting hard views. Traditional SSL methods rely on random augmentations to create image views for model training. The authors hypothesize that by selecting the hardest views (those yielding higher loss), the learning process can be improved, resulting in better model performance. The method can be seamlessly combined with existing method like DIMO, SimSiam, iBOT and SimCLR, and the results show the effectiveness across many architectures and datasets.", "questions": "- To leverage the computed embeddings from all views, one can take the average embedding and identify the one with the largest distance from it. This selected embedding can then be contrasted with either the other individual embeddings or their average. This might ensure that the additional computation from data augmentation is utilized effectively. Would adopting this strategy improve the method?\n    \n- In Appendix C, the training loss of DINO is notably high during the early stages. Could HVP be applied only in the later stages of training instead of throughout the entire process? If so, would it still enhance performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Good writing and clarity. The motivation is well-articulated, and the differences from prior work are clearly outlined. Overall, the writing is strong and clear.\n    \n- Simplicity. The approach is straightforward to implement and adaptable to a variety of architectures.", "weaknesses": "- The related works in Sec 2.2 are not comprehensive. There might be missing references for view selection. For example, Tian et al. [1] has a thorough discussion on the topic of \"What Are the Optimal Views for Contrastive Learning?\" Similar to your paper, Tian et al.  [1] studies the input view for contrastive learning. Moreover, Peng et al. [2] propose to generate contrastive views which could avoid most false positives (i.e., object vs. background). Similar to your paper, Peng et a. [2] studies the view selection. Thus, to clarity the distinction from the prior research, it is suggested to discuss the relation with the semantic-aware view selection in [2].\n    \n- Compuational overhead. The data augmentation significantly increases computational demands. Table 15 shows that HVP slows down the training of all models, despite efforts to use the hard view more efficiently. Please ensure a fair comparison under the same training budget or runtime. For example, the baseline (DINO) could be trained longer to match the computational budget of DINO+HVP.\n   \n    \n- Minors. The authors report achieving a state-of-the-art 78.8% linear probing accuracy on ImageNet. To ensure a fair and comprehensive comparison, please include the results of DINO+HVP using the ViT-B/16 architecture (400 epochs) in the main table. Typo: the legend of figure 6 should be DINO+HVP rather than DINO+HVS.\n\n- Minors. Lack of theoretical analysis: The method could be framed within the context of regularization techniques for self-supervised learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)