Probabilistic Self-supervised Representation Learning via Scoring Rules Minimization
Authors: Amirhossein Vahidi
Simon Schosser
Lisa Wimmer
Yawei Li
Bernd Bischl
Eyke Hüllermeier
Mina Rezaei
Venue: ICLR 2024 poster (2024)
Keywords: Self-supervised Learning
Probablistic Machine Learning
Proper Scoring Rule
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel probabilistic self-supervised learning via scoring rule minimization (ProSMin) to enhance representation quality and mitigate collapsing representations.

Abstract: %
Self-supervised learning methods have shown promising results across a wide range of tasks in computer vision, natural language processing, and multimodal analysis. However, self-supervised approaches come with a notable limitation, dimensional collapse, where a model doesn't fully utilize its capacity to encode information optimally. Motivated by this, we propose ProSMin, a novel probabilistic self-supervised learning approach that leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations. Our proposed approach involves two neural networks, the online network and the target network, which collaborate and learn the diverse distribution of representations from each other through probabilistic knowledge distillation. The two networks are trained via our new loss function based on proper scoring rules. We provide a theoretical justification for ProSMin and demonstrate its modified scoring rule. This insight validates the method's optimization process and contributes to its robustness and effectiveness in improving representation quality. We evaluate our probabilistic model on various downstream tasks, such as in-distribution generalization, out-of-distribution detection, dataset corruption, low-shot learning, and transfer learning. Our method achieves superior accuracy and calibration, outperforming the self-supervised baseline in a variety of experiments on large datasets such as ImageNet-O and ImageNet-C. ProSMin thus demonstrates its scalability and real-world applicability. Our code is publicly available: https://github.com/amirvhd/SSL-sore-rule.

Published as a conference paper at ICLR 2024
PROBABILISTIC SELF-SUPERVISED LEARNING VIA SCORING RULES MINIMIZATION
Amirhossein V Simon Schosser" Lisa Wimmer": Yawei Li
Bernd B 2
Eyke Hüllermeier Mina Rezaeia.6
Department of Statistics LMU Munich, Munich, Germany Munich Center for Machine Learning, Munich, Germany "Welloome Sanger Institute, Cambridge, UK Institute of Informatics LMU Munich, Munich, Germany
(av13@sanger.ae ac. mina rezae @ttat uni-mue -ei--muucchendde)
ABSTRACT
Self-supervised learning methods have shown promising results across wide range of tasks in computer vision, natural language processing, and multimodal analysis. However, self-supervised approaches come with notable limitation, dimensional collapse, where a model doesn't it fully utilize its capacity to encode information optimally. Motivated by this, we propose ProSMin, a novel probabilistic self- supervised learning approach that leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations, Our proposed approach involves two neural networks, the online network and the target network, which collaborate and learn the diverse distribution of representations from each other through probabilistic knowledge distillation. The two networks are trained via our new loss function based on proper scoring rules. We provide a theoretical justification for ProSMin and demonstrate its modified scoring rule. This insight validates the method's optimization process and contributes to its robustness and effectiveness in improving representation quality. We evaluate our probabilistic model on various downstream tasks, such as in-distribution generalization, out-of- distribution detection, dataset corruption, low-shot learning. and transfer learning. Our method achieves superior accuracy and calibration, outperforming the self- supervised baseline variety of experiments large datasets such ImageNet- O and ImageNet-C: ProSMin thus demonstrates its scalability and real-world applicability. Our code is publicly available: https:/githuu..com// www/amirvwd/SSLL...... rule.
INTRODUCTION
Self-supervised learning (SSL) is one of the most promising approaches for learning representations
from a limited set of labeled data and has achieved outstanding results in several domains and
applications, including natural language processing (NLP: [Devlin et al. (2018); Brown al. (2020) :
Saggau et al. (202))). computer vision (Chen et al. 2020 Bardes et al. 2021 Grill et al. 2020
Rezaei al., 2023a), multimodal learning (Radford al. 2021 Li et al. 2022b Shi al. 2022).
and bioinformatics (Gündüz et al., 2023 Rezaei et al. 2023b). However, collapsing representations one of the significant drawbacks to current SSL methods, which the representations converge to
limited set of points in the embedding space.
The dimensional collapse SSL occurs due to excessive distortion caused by strong data augmen- tation, making the duplicates dissimilar to the originals collapsing certain dimensions, as well as network overparameterization which leads to dower-dimensional solutions (Jing et al., (022). Strong augmentation introduce when the contrastive covariance matrix is not positive semidefinite, while overparameterization biases the network towards flatter minima, hindering information encoding in dimensions even for
variance than present in the data distribution, causing collapse

---

Published as a conference paper at ICLR 2024
Network
fs
59
Expant tilly
J(0) S(P.
ft 36
#
P/
=g(=;)
Figure 1: Illustration of proposed probabilistic self-supervised learning via scoring rule mini- mization namely ProSMin. Given a batch X of input samples, two different augmented samples z and ø' are taken by an online network with 0 and target network with parameters respectively. Our objective is to minimize the proposed scoring rule between Po, Ps-
similar positive pairs. Understanding and addressing these mechanisms is crucial for enhancing the effectiveness of self-supervised learning and preventing undesirable dimensional collapses.
Collapsing representations lead to reduced representation quality, impaired generalization to new tasks domains, and limited capacity to handle variations the data (Jing et al. (002)). Most recent studies addressed this problem with contrastive learning through effective augmentation (Bai
al. 2022), negative sample strategies (Kalantidis et al. 2020), ensemble approaches (Vahidi et al.
2023
Ruan et al. 2022). regularization techniques (Li et al. 2022a Rezaci et al. 2023a), removing
correlations in the feature space (Bardes et al. 2022). clustering- based approaches (Assran et al.
2022
2021 and self-distillation (Caron al. 2021 Grill al. 2020).
In this paper, we formulate the collapsing self-supervised representtionooooo problem through probabilistic machine learning, aiming to provide comprehensive and nuanced solution that addresses the limitations of deterministic approaches. This integration allows us to offer predictive distributions alongside their associated point predictions (Ho et al.) 2020 Nichol & Dhariwal (021). This pivotal shift offers promising avenues for achieving representation reliability and superior generalization capabilities in supervisd learning scenarios.
Specifically, we propose ProSMin, a novel probabilistic self-supervised learning method that mini- mizes scoring rule during pretext task learning. We motivate ProSMin by formulating knowledge distillation (KD) in probabilistic manner. As shown in Figure [ our proposed method involves two deep neural networks, an online and target network, each learning different representation of input samples. The online network maps input samples representation of the online encoder part. We train the online network such manner that samples from its output distribution predict the target network's 's representation of the input. The loss realized by this prediction is expressed via modified scoring rule, which incentivizes the recovery of the true distribution. Our contributions are:
probability distribution inferred from the
second augmented view
We introduce a novel probabilistic definition of robust representation for self-supervised learning. The probabilistic definition provides deeper understanding of the quality and trustworthiness of the learned representations guiding subsequent tasks.
Our probabilistic approach effectively mitigates collapsing representations by encouraging the online and target networks explore diverse range of representations, thus avoiding convergence to a limited set, which results a more comprehensive representation space that better encapsulates the intricacies of the data distribution.
We discuss a rigorous theoretical foundation for our proposed algorithm. This theoretical insight not only underscores the robustness of our approach but also provides principled explanation for its effectiveness in improving representation quality.
Through extensive empirical analysis, we validate the effectiveness of our approach in diverse scenarios. Our method achieves competitive predictive performance and calibration on various tasks such as in-distribution (IND), outto--distribution (OOD), and corrupted datasets, demonstrating generalization capabilities. Moreover, demonstrate the superi-
2

---

Published as a conference paper at ICLR 2024
ority of our method in semi-supervised and low-shot learning scenarios. Our framework establishes a superior trade-off between predictive performance and robustness when com- pared to deterministic baselines. This outcome is particularly notable on large-scale datasets such as ImageNet-O and ImageNet-C, underscoring the scalability and effectiveness of our method in real-world, high-dimensional settings.
2 BACKGROUND AND RELATED WORK
Self-supervised methods are designed to tackle unsupervised problems by training on a pretext task that utilizes the data itself to generate labels, effectively employing supervised methods to solve unsupervised problems (Grill et al. 20200] Chen et al. 2020 Jang et al. 2023 Caron al.. 202T ZZou ct al. 2021 Zbontar et al., 202 Bardes et al. 2022 Chen et al. (2021). The resulting representations learned from the pretext task can serve as a foundation for downstream supervised tasks, such as image classification or object detection. Alternatively, the extracted representation can be directly utilized for downstream applications, such as detecting anomalies and OOD data (Tran al., (2022). Recent studies (Oquab et al.] 2023 Zhou et al. (021)) provided evidence that performing self--uperrvsed pretext task learning on a large-scale and diverse dataset can extract features that are effective across different image distributions and tasks without the need for fine-tuning. Following this, we introduce a novel probabilistic self-supervised framework aiming to learn robust representation over parameters using self-distillation and by minimizing a scoring rule.
Self-distillation is a variant of knowledge distillation (Hinton et al. 2015) in which a larger model (teacher) is used to distill knowledge into smaller model (student) of the same architecture (Caron et al. 2021: Zhou et al. 2021). Given an input sample æ. the student network fo is trained on the soft labels provided by the teacher network fe. Self-distillation combines self-supervised learning with knowledge distillation and was introduced by DINO (Caron et al.) 202T). The two networks share the same architecture but take different augmentations of the input sample and output different representation vectors. The knowledge is distilled from the teacher network fe to student fe by minimizing the cross-entropy between the respective representation vectors. The parameters 0 of the student network are obtained from an exponential moving average (EMA) of the parameters & of the teacher network, thus reducing computational cost by confining backpropagation to the teacher network. In this paper, we present novel approach to knowledge distillation aimed at training network probabilistic manner using parametrization trick (Kingma et al. 2015) and novel scoring rule objective function.
A scoring rule is a function used to evaluate the accuracy of a probabilistic prediction (Gneiting & Raftery 2007: Der Kiureghian & Ditlevsen 2009 Hanselle et al. 2023). It quantifies the divergence between the predicted probability distribution and the true distribution of the event. The concept of scoring rule is fundamental to many areas of machine learning, including probabilistic classifi- cation (Parryl 2016) and decision theory (Dawid & Musio 2014)). A proper scoring rule (Gneiting & Katzfuss 2014) one that incentivizes truthful reporting of the probabilities by the forecaster, i.e. the forecaster is incentivized to report the correct probability distribution (Pacchiardi et al., 2021). The use of proper scoring rules has had significant implications in areas such as online learning IV'yugin& & Trunov| 2019), generative neural networks (Pacchiardi & Dutta 2022), and uncertainty quantification (Bengs et al. 2023 Gruber & Buettner, 2022 Sale al. 2023) This paper utilizes scoring rules and adapts them for the purpose of pretext task learning of a sulf-supe framework. This adaptation is inspired by the endeavor to infuse probabilistic learning principles into the self-supervised learning domain.
3 PROBLEM FORMULATION
Avoiding collapsing representation One of the major challenges in self-supervised learning is collapsing representations, where learned representations converge to limited set of points in the representation space. In other words, the model fails to capture the full diversity and richness of the underlying data distribution. This can lead to reduced representation quality, impaired generalization to new tasks or domains, and limited capacity to handle variations in the data. various techniques have been explored to mitigate dimensional collapse including contrastive losses (Kalantidis et al.) 2020) that separate embedding of negative examples, non non-contrastive losses (Rezaci et 2023al Bardes al. 2022) that reduce informational redundancy among embeddings, and joint embedding
3

---

Published as a conference paper at ICLR 2024
aiming to increase the average embedding entropy. In this paper, we formulate the collapsing representation problem through probabilistic lens, aiming to provide comprehensive and nuanced solution that not only addresses the limitations of deterministic approaches but also harnesses the power of uncertainty quantification and broader representation distributions.
and clustering approaches (Assran et al. 2022 2023)
Scoring rules A scoring rule (Gneiting & Raftery. 2007) is a function that evaluates how well a predicted distribution P over a random variable X aligns with the actually observed realizations æ of X. We define the los:" of predicting distribution P while observing æ as S(P, æ). Assuming that X follows some true distribution Q. the expected scoring rule measuring the loss of predicting P can be expressed as S(P, Q) & Ex-oS(P.X). A scoring rule S is proper with respect a set of distributions P if for all P. Q P it holds that S(Q,Q) Q) S S(P,Q) thus incentivizing the prediction of the true distribution Q. If the former holds with equality, i.e., the expected score S(P, Q) is uniquely minimized in Q at Q = P, then the scoring rule is called strictly proper. In practice, the expectation with respect to Q is usually replaced by an empirical mean over a finite amount of samples. We refer to the resulting scoring rule as S. There are many types of scoring rules, including we provide more
entire parameterized families of strictly proper scoring rules. In Appendix I0 details about some of the rules we use in our experiments.
4 METHOD
Consider a randomly sampled mini-batch of training data X [æi,...,, En,] where a; E RDD and transformation functions T, acting on the data. To enhance the training process, the transformation functions produce two augmented views ã a r(x) and a' a (xx for each sample in X. These augmented views are generated by sampling T, r' from the distribution of suitable data transforma- tions, such as partially masking image patches (He et al. 2022 or applying image augmentation techniques (Chen et al. 2020).
As depicted in Fig. [ the first augmented view E is fed to the encoder of online network fe that outputs a representation ye - ( This is followed by passing it subsequently through a projector ge and predictor qa. such that to _ (e())) and ze & (te)). We collect all trainable parameters of the online network in O. Similarly, the encoder of target network fe takes the second augmented view and outputs ve _ (ā)), followed by the projector network ge producing 4g - ge (we). where the trainable parameters are denoted as &. It is important to note that the predictor is applied exclusively to the online network.
In order to introduce probabilistic self-distillation training, we employ scoring rule (Gneiting & Raftery, 2007) as our loss function. accomplish this, is necessary to generate samples from the online network. However, directly sampling and a from a deterministic vector go()) is inadequate, as it prevents us from performing backpropagation effectively. One way of producing samples from neural network architecture, while still enabling backpropagation with respect to the latent representation 2, is to use the reparametrization trick (Kingma et al., 2013). Here, we assume that the output of the predictor network ze (te) follows underlying normal distribution with mean μ and standard deviation a. We generate r E N samples from the output of the linear layers following the prediction head by sampling random noise of NO, 1) for each augmented view of the i-th data point, such that the j-th sample, with j 1..., r, is given by: z). = p + o
Thus, we obtain samples 2 by shifting and scaling the random noise samples cf by the outputs (p, σ) of a neural network with trainable parameters 9, and the loss incurred by these samples can be backpropagated to update 0 during training. The online network parameters are updated by minimizing the scoring rule as follows:
== arg min ((),, (()) = S(Pe. Pc) = Beg (Po, zg].
(I)
0
'The original proposal in Gneiting & Raftery (2077) detines scoring rules in terms of a gain that is to be maximized. We adhere to the convention in deep learning of expressing the objective via loss function that we seek minimize.
4

---

Published as a conference paper at ICLR 2024
where Pg denotes the target distribution and 2g denotes the target output and Pe represents the online induced multivariate normal distribution. We define the approximation of our customized scoring rule loss for Po and P& follows:
S(Po, PE)
1
2A
|
A T(r
Σ [|=] -
(2)
1) jæt
where 2 represents the target prediction for the i-the input sample. (0,2) 2) and E (0, 1) are learn-able hyperparameters. We set À that S(Po, ze) 0. See Appendix 10.5.6 for detailed information.
By the principle of knowledge distillation. the parameters of the target network are updated through the EMA of the weights from the online network (Grill et al., 2020 Caron et al. 2021), saving the need for backpropagation and thus reducing computation time considerably.
f. (1 - at,-1, t= 1.2,.... a [0
The initial weights Eo are obtained through random initialization, while a is hyperparameter that determines the updating rate the target weights and is set to change from 0.9 to during training with cosine scheduler.
Next, we provide comprehensive description of our objective function based on scoring rules (Section 4. 4.I. For detailed understanding of scoring rules and their various variants, please refer to Section
4.1 OBJECTIVE FUNCTION
Numerous scoring rules can be decomposed into two terms. The first term is a function of zj and the realized observation zę. The second term is a function of two samples, 2j and žμ. drawn from the predicted distribution P. Our objective function is adjusted version of the former, where the two components are posed as convex combination with component weights controlled via hyperparameter X E (0. 1). We set À so that S(Po. =ę) > O. This modification of the scoring rule can be useful adjust the focus of the loss function on either part of the scoring rule. Two notable examples of scoring rules adhering to this form are the energy score and the kernel score (Gneiting & Rafferyy 2007).
With the above notation, we define the energy score as:
Sp(Po, =E) = 2 Ep, ||z; ze =e1l2] (-EPo E [||| 2z 12])
=e) S
(4)
Analogously, we write the kernel score as:
Sk(Po. =x) = Er, [k=2z, 26)] + (-2 Er, [k(2, z3)1) =: (Po) + Sko =g);
(5)
with suitable kernel function k(::)).
For simplicity, we only write S(Po: =ς) = S (P)) ) in the following for both scores.
With E (0, 1), we define the general form of our objective function as follows:
S' ze) AS' (Po. ze) + +1 2)5
4.2 THEORETICAL JUSTIFICATION
Our approach is motivated by strictly proper scoring rule, which ensures that the expected score is optimized only by the true distribution. However, our modified scoring rule gives more weight to the online network predictions during training. This deliberate focus is aimed enhancing the model's learning efficacy. By optimizing the relative importance of each component using a hyperparameter X, we encourage the model not only to fit the data but also to actively refine its predictive capabilities. Our proposed loss function (Eq. consists two key elements:
Comparison with the target network (S') ): This term measures the adherence of the model's predictions to a known benchmark, ensuring that the model remains grounded in established knowledge.
5

---

Published as a conference paper at ICLR 2024
Entropy for online network predictions (S²): This term emphasizes the internal consistency and reliability of the model's own predictions over time. This term increases the entropy of prediction and helps generalization by helping the exploration power of the model.
In its original form, the scoring rule can exhibit slight bias toward matching the target network too closely. To address this, we introduce stochasticity into the training process by increasing the weight of the second term (comparing online network predictions) in the loss function. This is not an artifact of poor design but a strategic insertion of complexity that nudges the algorithm towards more sophisticated regions of the solution space, thereby potentially locating a better optimum. Empirically, we show that an "improper" scoring rule variant can paradoxically increase accuracy.
5 IMPLEMENTATION DETAILS AND EXPERIMENTAL SETUP
Image augmentation We define a random transformation function T that applies a combination of multi-crop, horizontal flip, color jittering, and grayscale. Similar to torCn et al. (202T), we perform multi-crops with a random size from 0.8 to 1.0 of the original area and a random aspect ratio from 3/4 to 4/3 of the original aspect ratio. We define color-jittering of (0.8, 0.8, 0.8, 0.2), and Gaussian blurring with 0.5 probability and Ç = 2.0).
Deep self-supervised network architecture The online neural network is constructed from a backbone f. which can be either ViT (Dosovitskiy et al. 202I) or ResNet (He et al., 2016), and projection head g followed by a prediction q. The backbone output f is used as feature for downstream tasks. The projection consists of a 3-layer multilayer perceptron (MLP) with a hidden dimension of 2048, followed by normalizations and with K dimensions, similar to the design used in the DINO projection head. We use a predictor with two layers of MIPs with hidden dimension of 12000, with GELU nonlinearity in between. Note that ViT architectures do not use batch normalization (BN) by default, so we do not use BN in the projection or prediction when using VIT.
fully connected layer
The target network has the same backbone and projection as the online network and the target network learns through self-distillation. Similar DINO Caron et al. (202T), after the online network parameters are updated, an EMA of the online parameters (i.e., a momentum encoder) is used to update the target parameters. The EMA prevents the target parameters from being updated too quickly. After the parameters are updated, the target also receives a new centering parameter.
Optimization Our pretraining process involves training the models on the ImageNet training dataset (Deng et al. (209)) using the adamw optimizer (Loshchilov & Hutter, 70) and a batch size of 512, distributed across 8 GPUs using Nvidia Tesla A.0 with VIT-S/I6 architecture. We adopt a linear scaling rule to determine the base value of the learning rate, which is ramped up linearly during the first 30 epochs. Specifically, the learning rate is set to Ir = 0.0005 batchsize/ /256. warmup phase, we decay the learning rate using cosine schedule (Loshhhlovv & Hutter, 2017b). The weight decay also follows a cosine schedule, increasing from 0.04 to 0.4. The centering (smooth parameter) is 0.9.
/256. After the
Datasets
The datasets utilized in our experiments are as follows: The ImageNet (Deng et al. 2009
dataset with 1.28 million training images and 50,000 validation images with the size of 256 x 256 contains 1,000 classes. The ImageNet-O dataset (Srivastava et al. 2022) comprises images belonging to classes that are not present in the ImageNet-1K dataset. It is considered a challenging benchmark for evaluating the robustness of the models, as it requires models to generalize to a diverse range of visual conditions and handle variations that are not typically encountered in standard training datasets. The ImageNet-CC (Hendrycks & Dietterich] (019)) is a benchmark for evaluating the robustness of the models against common corruptions and perturbations that can occur in real- world scenarios. It consists of more than 30,000 images derived from the ImageNet dataset, with each image being corrupted in one of 15 different ways, including noise, blur, weather conditions, and digital artifacts. CIFAR-10/1
(Krizhevsky) 2009) are subsets of the tiny images dataset. Both datasets include 50,000 images for training and 10,000 validation images of size 32 x 32 with 10 and 100 classes, respectively. The Oxford 102 Flower (Nilsback & Zisserman, 2008) consists of 102 flower categories, each class including between 40 and 258 images. The images have large scale, pose, and light variations. In addition, there are categories that have large variations within the category

---

Published as a conference paper at ICLR 2024
and several very similar categories. iNaturalist-2018 (Van Horn et al. 2018) (iNat) comprises
vast collection of 675.170 training and validation images, classified into 5,089 distinct fine- ane-grained categories found in the natural world. It is worth noting that the iNat dataset exhibits significant imbalance, as the number of images varies greatly across different categories.
Tasks We evaluate the performance of ProSMin representations after self-supervised pretraining on the ImageNet on the basis of In-Domain (IND) generalization, OOD detection, semi-supervised learning, low-shot learning, corrupted dataset evaluation (see Section 6), well as transfer learning to other datasets and tasks (see Section II.
Evaluation metrics We report the prediction performance with the following metrics: Top-1 accuracy t: refers to the proportion of test observations that are correctly predicted by the model's output as belonging to the correct class. AUROC t: the area under the ROC curve represents the relationship between false-positive and false-negative rates for various classification thresholds. In this case, the positive and negative classes refer to whether an observation is in or out of given distribution, respectively, and the ROC curve is plotted as the threshold for classifying as "positive" is gradually increased. Negative log- log-likelihood (NLL) 4: measures the probability of observing the given test data given the estimated model parameters, multiplied by -1. This measure quantifies the degree to which the model's 's estimated parameters fit the test observations. Expected calibration error (ECE) J (Nacini et al. 2015): calculated as the mean absolute difference between the accuracy and confidence of the model's 's predictions, where confidence is defined as the highest posterior probability among the predicted classes. The difference is calculated across equally-spaced confidence intervals or bins and is weighted by the relative number of samples in each bin. A lower value of ECE indicates better calibration of the model. Mean Calibration Error (mCE) & is a metric used to evaluate the calibration of a classification model, similar to ECE. It is calculated as the mean of the absolute differences between the predicted and true probabilities of a given class across all classes. A lower value of mCE indicates better calibration of the model.
observation
6 RESULTS AND DISCUSSION
In-distribution generalization IND generalization (or linear evaluation) measures how well model's confidence aligns with its accuracy. To assess and compare the predictive abilities of our proposed model on in-distribution datasets, we freeze the encoder of the online network, denoted as fe. after performing unsupervised pretraining. Then, we train a supervised linear classifier using fully connected layer followed by softmax, which is placed on top of fo after removing the projection and prediction network. The desired outcome is high predictive scores and low uncertainty scores. In Table [] comprehensive comparison is presented between our approach and other self-supervised methods. The results indicate that our method outperforms all others in terms of top-1 accuracy, and calibration, as demonstrated by the lowest expected calibration error (ECE) and negative log-like ligelhoooo (NLL) scores.
The ability of model to recognize test samples from classes that were not present during training is evaluated using OOD detection, as discussed in /eng et al. (2020). We conduct experiments on IetgeNet-O (Srivastava et al., 2022) to assess the generalization of the model from IND OOD datasets, well as predict the uncertainty of the models OOD datasets. Note that evaluation is performed directly after self-supervised pretraining without a step. The y-axis of Figure 2 shows and compares the results for the OOD task in terms of AUROC. Remarkably, our method shows outstanding results for the detection of OOD samples compared to other approaches such as i-BOT, DINO, and Moco-VS V3. This outcome aligns with our expectations as we directly use the probabilistic latent representation for this task.
detection
Corrupted dataset evaluation An essential aspect of model robustness is its capability to produce precise predictions when the test data distribution changes. We examine model robustness under the context of covariate shift. Fig. 2 presents the improved performance metrics. Our method outperforms the bascline and has comparable predictive performance to the baseline for mCE.
Semi-supervised and low-shot learning on ImageNet Following the coni-supervised protocol established in Chen et al. (2020), we employ fixed 1 1% and 10% splits of labeled training data from ImageNet. In Table 2 we compare our performance against several concurrent models, including the baseline (DINO). Based on results obtained in Table 2] our approach outperforms
totte--f-theann methods in semi-supervised evaluation for both 1% and 10% scenarios. We assess our model's s efficacy on low-shot image classification task, where train logistic regression on frozen weights
7

---

Published as a conference paper at ICLR 2024
76-
20118001
76
AUROC
74
NN01300
More-11E300)
66
68 70
72
74
76
miCE
Figure 2: OOD defection and corrupted datasef evaluation. Methods with higher AUROC and lower mCE are better. Our method demonstrates outstanding performance among other approaches. We present and compare the performance of our method with others on the task of OOD detection on ImageNet-O) and corrupted datasets on ImageNet-C, in Table
with 1% and 10% labels. It's important note that this experiment was performed on frozen weights, without finetuning. Table shows our features achieved better performance compared to state-of-the-art methods.
Table I: IND Generalization (or Linear Evaluation). Top-1 accuracy, K-NN, ECE, and NLL averaged over in-distribution on test samples of the ImageNet dataset where the encoder ViT-S/16 over 800 epochs. The best score for each metric is shown in bold, and the second- best is underlined.
Method
Top-1 Ace (16)
A--N (f)
NI.I
BCB
DINO Caron al. 2021
76.8
74.5
0,919
0.015
MOCO Chen Chen ala 2021
73.2
64.7
1.152
0.027
i-BOT el al. 2021
77.9
75.2
0.918
0.013
ProSMin
78.4
0.900
ABLATION STUDY
To gain deeper understanding of the behavior and performance of our proposed method, we conducted several ablation studies explore various aspects of our approach. Specifically, investigate the following factors: different scoring rules as an objective function, the hyperparameter of our loss function (A), the number of samples used for generating latent representations, the dimension of the embedding space, the effect of the momentum hyperparameter, the impact of batch normalization (BN), and the prediction layer (PI.). We also supply ablation analysis computational efficiency and the broader impact our method in the Appendix (see Section [T). These investigations aim to provide insights and intuition regarding our approach.
Impact of different components of scoring rule We conduct a series of experiments to explore alternative scoring rules, including kernel scoring rules and various variations of energy scoring rules, for our objective function. The results, as presented in Table demonstrate that the kernel scoring
Table 2: Low-shot and semi-supervised evaluation: Top-1 accuracy (ACC), ECE, and NLL for semi- supervised on ImageNet classification using 1 and 10% training examples fine-tuning. and Low-shot results with frozen VIT features.
Methol
1%
10%
Architecture
Parameters
supervise
DINO'Caron et.l.
2021
60.3
74.3
VIT-S/16
21
i-OTT 7ioo &
2021
61.9
75.1
VIT-S/16
21
louns
I Low-shot learning
DINO
2021
64.5
VIT-S/16
i-BOT Zhoo et m. 2021
65.9
73.4
VIT-S/16
21
ProSMin [ouss]
66.1
73.8
VIT-S/16
21
8

---

Published as a conference paper at ICLR 2024
(8)
(b)
(c)
(d)
Figure 3: Study of hyperparameters our proposed ProSMin (a) X, (b) Number of samples, (c)
Momentum coefficient (a), and (d) Size of embedding obtained by 100 epochs on ImageNet.
Table 3:
Important component training. We investigate the effect of each component on the linear evaluation performance for 100 epochs. The first line shows the best combination. PL the prediction layer, BN is the batch normalization layer.
Energy (L) loes)
Kernel seore
Energy (12 loss)
BN
PL.
A
Accuncc
X
x
73.2
/
x
/
1.1
X
x
x
43.0
68.5
rule exhibits instability. Furthermore, we investigate case with L1 loss (the last row of Table with prediction layer and without batch normalization, the and a are zero, to motivate the effect of the second part of our loss function. Overall, the energy score with 8 = 1 (L1 loss) yields the best performance among the tested scoring rules. We provide a theoretical explanation for L1 in Section 10.5.3
Study of hyperparameters Figure 3a depicts the influence of the hyperparameter utilized in our
proposed loss function, which controls the impact of 5 (Po, zg) and (Po. (Po. =g) on the objective function. An ablation analysis was conducted to investigate the effect of increasing the number of samples from qo. as illustrated in Figure 3b The results demonstrate that employing four samples yields satisfactory performance. Figure showccssssss the outcomes obtained from the knowledge distillation rate. In previous approaches, the exponential moving average parameter initiated from value relatively close to (c.g., 0.996 (Grill et al. 2020). Caron et al. (202T)). However, in our case, starts from 0.9, implying a faster pace of knowledge distillation. Furthermore, we examine the impact of different sizes for the embedding vector, as presented in Fig. 3d The results obtained after 100 epochs reveal that increasing the embedding size leads improved performance. However, should be noted that larger embedding sizes necessitate additional computational resources, thus our choice of size based on the available computational capacity. Table 3 provides insights into the influence of batch normalization in the prevention of representation collapse (Grill et al., 2020). As our framework operates a probabilistic setting, the inclusion of batch normalization is unnecessary for averting collapse. Additionally, the prediction layer (go) enhances performance by facilitating improved feature extraction in online networks.
CONCLUSION
In this paper, we presented ProSMin as a novel probabilistic self-superrised framework address collapsing representation problem. Our method includes two neural networks that collaborate and learn from each other using an augmented format. Our framework is trained by minimizing proposed scoring rule objective function. We evaluated ProSMin across different tasks, including in-distribution generalization, out-of-distribution detection, dataset corruption, transfer learning, low-shot, and superviie learning. The results demonstrate that method achieves superior performance in terms of accuracy and calibration, thus showing the effectiveness of our proposed approach.

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGMENT
M. R. and B. B. were supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics Data Applications (ADA-Centers within the framework of BAYERN DIGITAL II 2003440-22-9-8... R., B. B., E. H., L. W.,Y L., and A. were supported by the German Federal Ministry of Education and Research (BMBF) Munich Center for Machine Learning (MCML). L. W. was supported by the DAAD program Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the German Federal Ministry Education and Research
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametricaly predicting view assignments with support samples. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8443-8452, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. European Conference on Computer Vision, pp. 456-473. Springer, 2022.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. supe-supe learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15619-15629, 2023.
Yalong Bai, Yifan Yang. Wei Zhang, and Tao Mei. Directional self-supervised learning for heavy image augmentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 16692-16701, 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: for self-supervised learning. International Conference on Learning Representations, ICLR, 2021.
ianaariacce-covariance regularization
Adrien Bardes, Jean Ponce, and Yann LeCun. Variance- inrtacceiivaaianee-ovvaannnn regularization for self-supervised learning. ICLR, Vicreg, 1:2, 2022.
Viktor Bengs, Byke Hüllermeier, and Willem Waegeman. On second-order scoring rules for epistemic uncertainty quantification. arXiv preprint arXiv:2301.127.:0 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 9650-96600 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Alexander Philip Dawid and Monica Musio. Theory and applications of proper scoring rules. Metron, 1292)1669-183, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
10

---

Published as a conference paper at ICLR 2024
Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? does it matter? Structural Safery, 31(2):105-112, 2009.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ACL, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
Chuanxing Geng, Sheng-jun Huang, and Songean Chen. Recent advances open set recognition: survey. IEEE transactions pattern analysis and machine intelligence, 43(10))3614-363) 2020.
Tilmann Gneiting and Matthias Katzfuss, Probabilistic forecasting. Annual Review of Statistics and Its Application, 1:125- 151, 2014.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378 2007.
nan-BBatien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21283 2020.
Sebastian Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classifica- tion and beyond. Advances in Neural Information Processing Systems,
2022. Hüseyin Anil Gündüz, Martin Binder, Xiao- Yin To, René Mreches, Bernd Bischl, Alice C McHardy, Philipp C Münch, and Mina Rezaei. A self-supervised deep learning method for data-efficient training in genomics. Communications Biology, 6(1):928, 2023.
Jonas Hanselle, Johannes Fürnkranz, and Eyke Hüllermeier. Probabilistic scoring lists for interpretable machine learning. In International Conference on Discovery Science, pp. 189-203. Springer, 2023,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https
//oeenreveww.o hwt.:/eetffoommmmeo ?immmiidHiii HiiiidddHHttgimym
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: arXiv:/503.000 1503.02533. 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Jiho Jang, Seonhoon Kim, Kiyoon Yoo, Chaerin Kong, Jangho Kim, and Nojun Kwak. Self-distileed self-supervised representation learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2829-2839, 2023.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. International Conference on Learning Representations, ICLR, 2022.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33: 21 1198-21809, 2020.
11

---

Published as a conference paper at ICLR 2024
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteriization trick. Advances in neural information processing systems, 28, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alexander C Li, Alexei A Efros, and Deepak Pathak. Understanding collapse in non-contrastive siamese representation learning. In European Conference Computer Vision, pp. 490-505. Springer, 2022a.
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, ICLR, 2017a.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017b. 20177. URL htllitts///opeeeeiiiie
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of 15, pp. 2901-2907, AAAI Press, 2015.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Maria-Elewr Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
I. Pacchiardi, R Adewoyin, P Dueben, and R Dutta. Probabilistic forecasting with generative networks via scoring rule minimization. Preprint, URL https://arxiv. org/abs/2112.082177 2021.
Lorenzo Pacchiardi and Ritabrata Dutta. Likelihood -fee inference with generative neural networks
via scoring rule minimization. arXiv preprint arXiv:2205. 157844 2022.
Matthew Parry. Linear scoring rules for probabilistic binary classification. Electronic Journal of Statistics, 10:1596-1600 2016
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. PMLR, 2021.
Mina Rezaei, Farzin Soleymani, Bernd Bischl, and Shekoofeh Azizi. Deep bregman divergence for self-supervised representations learning. Computer Vision and Image Understanding, 235:10 2023a.
Mina Rezaei, Amirhossein Vahidi, Bernd Bischl, Mengyu Wang, Tobias Elze, and Mohammad Eslami. Self-supervised learning and self-labeling framework for retina glaucoma detection. Investigative Ophthalmology & Visual Science, 64(8):352-355 2023b.
Yangjun Ruan, Saurabh Singh, Warren Morningstar, Alexander A Alemi, Sergey loffe, Ian Fischer, and Joshua V Dillon. Weighted ensemble self-supervised learning. International Conference on Learning Representations, ICLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Daniel Saggau, Mina Rezaei, Bernd Bischl, and Ilias Chalkidis. Efficient document embeddings via self-contrastive bregman divergence learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisticcs: ACL 2023, pp. 12181- 12190, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18/53/ v1/2023. 111203..indings-accl.77. URL https: sictps://aclanthottt http://aaaaatoolloooog//////////
findings acl
771
Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Second-order uncertainty quantifi- cation: distance-hased approach. arXiv preprint arXiv:2312.00995, 2023.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual
speech representation by masked multimodal cluster prediction. International Conference Learning Representations, ICLR, 2022.
Anugya Srivastava, Shriya Jain, and Mugdha Thigle. Out of distribution detection on imagenet-o. arXiv preprini arXiv:2201.093: 2022.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.0744 20,,01411, 2022.
Amirhossein Vahidi, Lisa Wimmer, Hüseyin Anil Gündüzz Bernd Bischl, Eyke Hüllermeier, and Mina Rezaei. Diversified ensemble of independent sub-networks for robust self-supervised representation learning. arXiv preprint arXiv:2308. 14705, 2023.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2018.
Vladimir V V'yugin and Vladimir Trunov. Online learning with continuous ranked probability score. In Conformal and Probabilistic Prediction and Applications, pp. 163-177. PMLR, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. ICLR, 2021.
13

---

Published as a conference paper at ICLR 2024
10 SCORING RULES
The notation employed in this study aligns with the work of Cineiting & Raftery (2007) regarding scoring rules S(P, æ), where P represents the predictive distribution and æ denotes the observed data. By assuming that X follows a true distribution Q. With this assumption, we first explain different variations of scoring rules, such as the expected scoring rule (TO.T). the proper scoring rule (T.22) the energy scoring rule (I0.3), and the kernel scoring rule (10.4).
We then discuss the unbiased estimation properties of the scoring rule 10.5 for kernel score 10.5.1 and energy score 10.5.2, which we used as the objective function in our study.
In subsection 10.3.4 establish the interchangeability of the expectation and gradient in our theoretical derivative, enabling us to derive gradients in a conventional manner. This is crucial due to the incorporation of samples within the scoring rule and the utilization of non-differential activation functions. Subsequently, in subsection 10.5.5 we introduce unbiased estimate of the gradient.
10.1 EXPECTED SCORING RULE
We define the scoring rule S(P x) as a function of the distribution P and the observation æ of X. The expected scoring rule is defined
S(P,Q) == Ex-QQS(P
10.2 PROPER SCORING RULE
A scoring rule S called proper w.rl a set of distributions P, for P,Q E P the expected score S(P, Q) is minimized in Q at Q = P. A scoring rule S is called strictly proper if there exists the unique minimum S(P. Q) S(Q, Q) for Q x P.
10.3 ENERGY SCORE
The energy score is defined by:
$ (P,x) = 2-
||X
x|6 -
x x
X XIX ~
(8)
where B E 2).
It is a strictly proper scoring rule for the class of probability measures P such that Ex~pl 1x>p|XX130 < < oo. An unbiased estimate can be obtained by replacing the expectations in s( with empirical means over draws from P.
10.4 KERNEL SCORE
Let :) be positive definite kernel. The kernel score for l defined by
S& =
x x - Ejk(X
XIX'
(9)
The kernel score is proper scoring rule for the class of probability distribution for which holds. that E Xx h X (X)') < Under the condition that the kernel Maximum Mean Discrepancy a metric, the kernel score is strictly proper. This condition is satisfied by the Gaussian kernel, which will be used in this work. Let be scalar bandwidth. We define the Gaussian kemee follows:
x
x
x(xx)) exp
212
We get an unbiased estimate by replacing the expectations in S₁ with the empirical means overdraws from P.
14

---

Published as a conference paper at ICLR 2024
10.5 UNBIASED ESTIMATORS OF THE EXPECTED SCORING RULE
Let N be the batch size and be the number of samples drawn from the indine-induced distribution.
Let Zp (z1,...,,y) be the sample drawn from the online. We recall the definition of the Expected Scoring Rule:
S(Po. P&) = Ezg (S(Po: zc)l
where 2g denotes the output of the target network while Pa represents the online induced multivariate normal distribution.
10.5.1
KERNEL SCORE
We can write the estimated kernel score follows:
xk)), - - r k ke2), at
S(Pe, P3) N r(r
Σ
jA=1ijzk
10.5.2 ENERGY SCORE
Consider B E (0,2) and N be the batch size. The Estimated Energy Score can be written as:
2
S(Po, Pε)
=E112
Σ 1.j=1
|
r(r-1)
10.5.3 PARTIAL DERIVATIVES OF THE ESTIMATED ENERGY SCORE
Let 3 E (0, 2) and let N be the batch size. Let ze be the input data for i E 1,.., N. Let E N be
the number samples. Let μ4 = f( (x;) and let of = f(() (az), where f() denotes the separated branches for mean and variance in the last layer of the online network for 1, 2. We can calculate the partial derivatives needed for stochastic gradient descent as follows. Then, the formula for the mean is:
2
S(Po, P.) pp
Ms +
of
N
Σ
+ E V of
14 + 6k
-(r-1) 1)
J.k== b
1
Mi + + 00 o/
0
14 + +
/ o8
Σ
(1) (I.) + tej
180) (zx) o2
1=
15

---

Published as a conference paper at ICLR 2024
while the formula for the variance is:
0G2
S(Po
Pt)
OG2
2
Mi
Ej
V
03 $6
[4 Mi o3 ih + EN
r(r
j,k=1,jzt
B
f0 (1) (2x+) + c/\ 10²² (x:) - 20
N d
19-1
Σ
11.0 (g) - -cx)
Volg (x:)
(x;)
1)
10.5.4 EXCHANGEABILITY OF GRADIENT AND EXPECTATION
This proof follows similar argumentation as the proof performed in Pacchiardi & Dutta (2022). We want to solve
N
1 N
0 = arg min ( ((0)
S(Po, z(2)
(10)
Finding a minimum in our network architecture is done via stochastic gradient descent (SGD) or an algorithm exploiting the same properties SGD. Recall that the scoring rule is defined as an expectation over samples from Po. Additionally, we can describe the scoring rule as some function h of independent inputs 2j 1 z Hence we can write:
S(Po: z Bej.st hh =
(11)
Let μ4, of be the estimated mean and variance, respectively, of the online network fo for the i-th input sample z,. With Pe being the distribution induced by the online network and go being its transformation, we use the reparametrization trick and get
N M
((O)
(12)
N
We now can write the derivative as follows:
N
VoJ(o) = Vo N
Σ
E ~ [Mgo( ni, of). go(d, Mi, of), z{)]
1 n
E Egg
[Veh(go(c), M4, of) go Ps, of) =[]]
72
The exchange between expectation and gradient is not trivial because of the non diffdifferentiiabiiity of functions (c.g., ReLU) within the network function go. We can still perform this step by using Theorem 5 from Pacchiardi & Dutta (0022), considering mild conditions the neural network architecture of ga-
10.5.5 UNBIASED ESTIMATE OF THE GRADIENT
Let B be a random subset (batch) the data set. Using 10.5.4, we can obtain unbiased estimates of (() using samples e/ ~ (O, 1), I...., r for each i E [I....NN:
VoJ(O)
Σ
(13)
r(r
iEB
16

---

Published as a conference paper at ICLR 2024
10.5.6 DETAILED IMPLEMENTATION
In addition to detail in Section [ it's important to mention that the target network takes two global augmented samples while the online network takes two global augmented samples and 16 local augmentations for multiccroppig samples. Then, the r in Eq. Eq. 2irefers to the number of augmentation samples multiplied by the number of samples. Furthermore, Eq. denotes of the global augmented samples.
ADDITIONAL EXPERIMENTAL. RESULTS AND ABLATION ANALYSIS
In this section, we provide additional details experiments and ablation analysis.
11.1 TRANSFER LEARNING EVALUATION
We further assess the generalization capacity of the learned representation on learning a new dataset. We followed the same transfer learning protocol explained in (Caron al. 202Ip. To this end, we evaluate the performance of our model pretrained on ImageNet to CIFAR I0/I00, an imbalanced naturalist dataset (iNat- 18), and the Flower dataset. According to results shown in Table we observe that our method provides a robust solution when transferring to the new dataset.
Table 4: Transfer to new dataset evaluation: Transfer learning by finetuming pretrained models on different datasets. We report top-1 accuracy. Self-supervised pretraining with ProSMin transfers better than supervised pretraining.
Method
CIFAR-10
CIFAR-100
iNat- 18
Flowers
DINO Caron et al. 2002
99.0
90.5
72.0
98.5
i-BOT Zhou et al. 2021
99.1
90.7
73.7
98.6
ProSMin (ours)
99.3
91.2
74.4
99.1
11.2 OUT-OF-DISTRIBUTTIO DETECTION AND CORRUPTED DATASET EVALUATION
Table compares the performance of our method for the task OOD detection and corrupted dataset on ImageNet-O) and ImageNet-C respectively.
Table 5: Out of distribution detection and corrupted dataset: AUROC for out of distribution detection for Imagent-O dataset, where higher AUROC is better, and mCE for Imagent-C C dataset where lower mCE better.
Method
AUROC (%) (1)
mCE (f)
DINO (300 epochs) Caron et al. 2022
74.0
68.1
MOCO-V3 (300 epochs) Chen et ProSMin (300 epochs)
2021
71.0
73.5
76.0
66.1
DINO (800 epochs) Caron al. 2021
76.0
66.5
ProSMin (800 epochs) Caron et al.
2021
77.1
65.8
i-BOT (800 epochs)
Caron al.
76.6
66.2
11.3 DIMENSION OF REPRESENTATION VECTOR
The dimensionality of latent representations influences the equilibrium between information preserva- tion, transferability, computational efficiency, and defense against overfitting. In Table 6 evaluate the effect of varying the output dimensionality. Based on the results shown in Table the large output dimensionality improves the performance.
17

---

Published as a conference paper at ICLR 2024
Table Analysis of dimension representation vector term Top-1 K-NN.
Method
1024
4096
16384
65536
262144
DINO Caron et al. (2021)
67.8
69.3
69.2
69.7
69.1
ProSMin (ours)
69.0
69.4
69.5
69.9
11.4 IMPACT OF PROSMIN ON MITIGATING REPRESENTATION COLLAPSE
As we explained in Chapter 3, ProSMin formulates the collapsing representation problem through probabilistic lens, aiming to provide a comprchensive and nuanced solution that not only addresses the limitations of deterministic approaches but also harnesses the power of uncertainty quantification and broader representation distributions. Specifically, we propose a novel probabilistic self-supervised learning method that minimizes scoring rule during pretext task learning. We motivate ProSMin by formulating knowledge distillation (KD) in a probabilistic manner. Through extensive empirical analysis, we validate the effectiveness of our approach compared to other self-supervised approaches proposed to prevent collapsing representation (MoCoV3 (Chen et al. 202T), DINO (Caron et al. 202I), and BYOL (Grill al., 2020).) In particular, BYOL prevents collapsing representation using batch normalization and prediction layer. For DINO, Sharpening and centering play the same role. We showed the superiority of our method compared to other approaches for collapsing representation in Tables 12 5 and as well as Figure for the task of in-domain generalization, low-shot learning, semi-supervised learning, transfer learning, OOD detection, and corrupted dataset evaluation respectively. Importantly, in Table 3 we showed that our method can achieve good performance without BN, prediction layer, centering, and sharpening which are the components of BYOL and DINO prevent collapsing representation.
11.5 ANALYSIS OF COMPUTATIONAL COST
We further assess the effectiveness of our proposed approach and compare with DINO and iBOT 7 The presented values are obtained from the data reported the DINO and iBOT papers. However, these papers do not include information regarding the total number of parameters.
Table 7: Evaluation of computational efficiency We conduct thorough analysis of the computational efficiency of novel probabilistic approach in comparison to alternative self-supervised methods. This evaluation encompasses memory utilization and computational expenditure.
Method (ViT-s)
parameters (M)
im/s
time/ 300-epochs (hr)
numher of GPUs
memory (G)
DINO'Caron 2021
21
1007
72.6
16
15.4
i-BOT Zhou 2021
21
1007
73.8
16
19.5
ProSMin (ours)
21
1007
98.0
21.1
11.6 BROADER IMPACT AND LIMITATIONS
This study has the potential to inspire new algorithms and stimulate theoretical and experimental exploration. The algorithm presented here can be used for many different probabilistic downstream tasks, including (but not limited to) uncertainty quantification, density estimation, image retrieval, probabilistic unsupervised clustering, program debugging. image generation, music analysis, and ranking. In addition, we believe that our extended concept probabilistic framework opens many interesting avenues for future development self self-supervised learning, and addresses many problems of existing models, such as avoiding representation collapse.
However, there are several limitations. One limitation of our model compared other learning methods (such supervised learning) is that self-supervised learning may require more computational resources and training time. However, considering that our proposed method does not require manual annotation, which is usually very expensive, we would argue that this trade-off is acceptable.
18

---

Published as a conference paper at ICLR 2024
15.40
12.
12.5
73.35
73.33
23.30
21.5
73.22
75.00
0.34
(a)
(b)
75.4
752
Accurer
73.0
72.8
72.6
61
72.4
6.70
0.75
3.30
0.55
3.00
0.55
4002
6303
5300
14000 12000
11000
10002
(c)
(d)
Figure 4: Study of hyperparameters of our proposed ProSMin (a) X, (b) Number of samples, (c)
Momentum coefficient (a), and (d) Size of embedding obtained by 100 epochs on ImageNet.
19


Reviews:
[{"review_id": "JNcoYiL7jQ", "reviewer": "Reviewer_QnX8", "summary": "The paper approaches the problem of learning self-supervised representation learning as a parametric probability density estimation problem in the representation space. The authors utilize an encoder to estimate the parameters of the distribution of the encoded sample similar to Kingma et al. They utilize self-distillation similar to BYOL and DINO where the distillation loss is a proper scoring rule for distributions.", "questions": "1. How do you ensure $q_\\theta$ is not an identity network w.r.t $\\mu$ with $\\sigma = 0$ since the dimensionality of $z_\\theta$ and $t_\\theta$ \n2. Since your scoring function is a variation of $L_2$ loss between $z_i$ and $z_{\\xi}$, can you show that the variational encoding actually makes a difference by perturbing $t_\\theta$ by a Gaussian noise of $\\sigma = \\epsilon$\n3. The authors claim that their method is superior in terms of preventing the collapse of representation, can you provide an experiment specific to this to highlight that the representations learned by ProSMin mitigate this problem? This is important because the authors claim that strong augmentations distort the underlying distribution, hence promoting the collapse of the representations learned. If learning the representations via a parametric distribution that utilizes the resampling trick of sampling from a Gaussian distribution, is it not effectively an augmentation in the representation space?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors present a novel perspective toward solving the problem of learning self-supervised representations as a distribution over the representation space.\nThey achieve state-of-the-art performance on imagenet-1K for linear evaluation and K-nearest neighbour search.", "weaknesses": "See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kmABznx4Dg", "reviewer": "Reviewer_pLHM", "summary": "The article focuses on the dimensional collapse problem in SSL. To address this issue, the authors propose a probabilistic approach via self-distillation to build robust representations. Detailed proofs confirm the convergence of the proposed method. The experimental results demonstrate the effectiveness of the method across various scenarios, including in-distribution, out-of-distribution, transfer learning, and more.", "questions": "1. The network structure of the proposed method is identical to BYOL. What do you think is the most important difference between these two approaches?\n2. The term ${S^2}$ in Equation 4, Equation 5 and Equation 6 is only related to $\\theta $. Why do you define the energy score as ${S^2}({P_\\theta },{z_\\xi })$?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes to learn robust feature representations through a probabilistic approach.\n2. The theoretical proofs presented in the article, along with the explanations using scoring rules, demonstrate the convergence of the proposed algorithm. \n3. The authors conduct experiments in multiple settings, and the experimental results validate the effectiveness of the proposed method.", "weaknesses": "1. There are some typos in the article, affecting the overall readability of the paper. \n- In Table 3, should \"PN\" be replaced with \"PL\"? The experimental setups in the first and last rows of Table 3 are exactly the same. Is this an error?\n- In the last sentence of the third paragraph in Chapter 4, should \"$\\mu $\" and \"$\\sigma $\" have the superscript \"i\"? As per my understanding, \"$\\mu $\" and \"$\\sigma $\" vary for each data point.\n- In the eighth line of the abstract, there is a semicolon. I believe using a comma would be more appropriate.\n\n2. The article uses a self-distillation mechanism for learning, but the description of the learning mechanism is not clear. The third paragraph of Chapter 5 contains a sentence that says, \"The target network has the same backbone and projection as the online network and it learns through self-distillation\". The target network is updated through the strategy of EMA, and I think the online network could be regarded as learning through self-distillation. Is there a mistake here, or is my understanding off? In addition, there are some mistakes in the descriptions of the distillation mechanism of DINO in the second paragraph of Chapter 2. The parameters of the teacher model, rather than the student model, are obtained through EMA.\n\n3. The association between the probabilistic method and the prevention of dimensional collapse still requires further elaboration and demonstration. In the second paragraph of Chapter 1, the authors give two possible causes for the dimensional collapse. Is the author's solution inspired by either of the two causes or did it address either of the two factors? I think this needs to be further explained. I suggest two additional experiments in the following areas:\n- Ablation experiments with the removal of \"$\\sigma $\" and the second term Equation 2. I think it is necessary to illustrate the effectiveness of the probabilistic approach.\n- Performance comparison with various baseline methods for the same feature dimensions. \n- In Figure 3d, the dimension of the largest embedding vector is 16000, which I think is not large enough and should be further increased for experiments. As far as I know, many self-supervised methods, such as Barlow Twins, are capable of not collapsing completely in a dimension of 16000. I suggest that the authors can show that their proposed method does not collapse completely in larger dimensions. It would be better if the authors could somehow demonstrate that their method improves the effective dimensions of the learned features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "74EeuILyQJ", "reviewer": "Reviewer_JrU7", "summary": "This paper propose ProSMin, a probabilistic self-supervised learning approach that can mitigate the problem of  collapsing representations in self-supervised learning. It follows the basic framework of training representations on augmented views of give images, like in contrastive learning. The core components are 1) an online network that predicts the representation as a distribution rather than a deterministic vector; 2) a target network that can be seen as an mean teacher.  Learning is accomplished based on proper scoring rules.", "questions": "Please see [Weakness]", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Give a new probabilistic modeling for self-supervised representation learning. Compared with the deterministic modeling, the probabilistic modeling effectively mitigates collapsing representations.\n2. The convergence of the proposed method is theoretically proved. The theooritical justification brings new insights on how representation quality is effectively improve\n3. Solid experiments results on in-domain case and out-of-domain case show the proposed method achieves better represnetation, and the learned scores are well calibrated.", "weaknesses": "This paper reads smooth and everything looks good to me except for one concern: The authors are trying to sell that the proposed methods avoid the collapsing problem in self-supervised representation learning, but I didn't see explict justification or evaluation on this point. \nI have no idea that representations learned with previous method, such as contrastive learning, collapsed to what extend, and no idea on how good the proposed method improved on this. Maybe the performance on in-domain and out-of-domain expereiments show the representation is better, but I cannot justify if or not this were because the collapse problem is mitigated. \nI believe this work would be definitely more technically sound If authors provide more qualititive or quantatitive results that can directly reflect the level of collapse,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)