DIETing: Self-Supervised Learning with Instance Discrimination Learns Identifiable Features

Authors: attila  juhos
alice  bizeul
patrik  reizinger
david  klindt
randall  balestriero
mark  ibrahim
julia e vogt
wieland  brendel
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-20 10:47:32

Abstract: Self-Supervised Learning (SSL) methods often consist of elaborate pipelines with hand-crafted data augmentations and computational tricks. However, it is unclear what is the provably minimal set of building blocks that ensures good downstream performance. The recently proposed instance discrimination method, coined DIET, stripped down the SSL pipeline and demonstrated how a simple SSL algorithm can work by predicting the sample index. Our work proves that DIET recovers cluster-based latent representations, while successfully identifying the correct cluster centroids in its classification head. We demonstrate the identifiability of DIET on synthetic data adhering to and violating our assumptions, revealing that the recovery of the cluster centroids is even more robust than the feature recovery.

Main paper: 
 introduction: 
 Self-Supervised Learning (SSL) methods use unlabeled datasets to learn representations by solving an auxiliary task, thus bypassing time-consuming labelling efforts. Importantly, co-occurance-based SSL relies on positive data pairs (similar samples, e.g., an original sample and a transformed/augmented one) and negative data pairs (dissimilar samples, often randomly drawn from the dataset). Contrastive and non-contrastive learning, the two prominent families of SSL methods, utilize positives and negatives differently, though they are theoretically connected [Balestriero and LeCun, 2022] . Contrastive Learning (CL) [Chen et al., 2020 , Zimmermann et al., 2021 , von Kügelgen et al., 2021 , Lyu et al., 2021 , Eastwood et al., 2023] attracts positive pairs' and repels negative pairs' representations. Non-contrastive learning [Bardes et al., 2021 , Zbontar et al., 2021 , Mialon et al., 2022] only uses positive pairs, and avoids representation collapse with strategies such as momentum encoders or covariance regularization. Unfortunately, the many actively developed Self-Supervised Learning methods with such computational tricks potentially hinder selecting the best performing and simplest SSL method for a given task. Recently, Ibrahim et al. [2024] proposed DIET, a SSL method that strips away unnecessary details by reducing the auxiliary task to a simple instance classification paradigm, and showed competitive performance on small datasets. Identifiability theory, particularly Independent Component Analysis (ICA) [Comon, 1994 , Hyvarinen et al., 2001] studies guarantees of probabilistic models to recover the ground-truth latent variables in a probabilistic latent variable model (LVM). Recent advances in nonlinear ICA theory proposed multiple self-supervised/weakly supervised models with identifiability guarantees [Hyvarinen et al., 2019 , Gresele et al., 2019 , Khemakhem et al., 2020a , Hälvä et al., 2021 , Hyvarinen and Morioka, 2016 , Khemakhem et al., 2020b , Locatello et al., 2020 , Morioka and Hyvarinen, 2023, Morioka et al., Figure 1 : DIET [Ibrahim et al., 2024] learns identifiable features: DIET learns a linear (N × d)-dimensional classification head W on top of a nonlinear encoder f through an instance discrimination objective (1). For unit-normalized f (x n ), DIET maps samples and their augmentations close to the cluster vector v c corresponding to the class as if sampled from a von Mises-Fisher (vMF) distribution, centered around the cluster vector. In case of duplicate samples, i.e., matching class labels, the corresponding rows of W will be the same, as shown for x 1 and x i with w 1 = w i 2 Identifiability guarantees for DIET This section presents our main theoretical contribution. After summarizing DIET, we introduce a mildly constrained theoretical setup, in which DIET provably recovers the correct latents. The setup is followed by the main result and a discussion on the intuition for our theoretical model.DIET [Ibrahim et al., 2024] . DIET solves an instance classification problem, where each sample x in the training dataset has a unique instance label i. Augmentations do not affect this label. We have a composite model W • f , where the backbone f produces d-dimensional representations, and a linear, bias-free classification head W that maps these representations to a logit vector equal in size to the cardinality of the training dataset. If the parameter vector corresponding to logit i is denoted as w i , then W effectively computes similarity scores (scalar products) between the w i 's and embeddings f (x). DIET trains this architecture to predict the correct instance label using multinomial regression (with f , W and temperature β as variables):L(f , W , β) = E (x,i) -ln e β⟨wi,f (x)⟩ j e β⟨wj ,f (x)⟩ .(1)Setup. For our theory, we need to formally define a latent variable model (LVM) for the data generating process (DGP) to assess the identifiability of latent factors. For this, we take a clustercentric approach, representing semantic classes by cluster vectors, similar to proxy-based metric learning [Kirchhof et al., 2022] . Then, we model the samples of a class with a von Mises-Fisher (vMF) distribution, centered around the class's cluster vector. This conditional distribution jointly models intra-class sample selection and augmentations of samples, together called intra-class variances.Our conditional does not mean that each sample pair transforms into each other via augmentations with high probability. It does mean that-since we assume an LVM on the hypersphere; i.e., all semantic concepts (color, position, etc.) correspond to a continuous latent factor-the latent manifold is connected, or equivalently, that the augmentation graph is connected, which is an assumption used in [Wang et al., 2022 , Balestriero and LeCun, 2022 , HaoChen et al., 2022] . We provide an overview of our assumptions, and defer additional details to Assums. 1C in Appx. A: Assumptions 1 (DGP with vMF samples around cluster vectors. Details omitted.). (i) There is a finite set of semantic classes C , represented by a set of unit-norm d-dimensional cluster-vectors {v c |c ∈ C } ⊆ S d-1 . The system {v c } is sufficiently large and spread out. (ii) Any sample i belongs to exactly one class c = C(i).(iii) The latent z ∈ S d-1 of our data sample with instance label i is drawn from a vMF distribution around the cluster vector v c of class c = C(i):EQUATION(iv) Sample x is generated by passing latent z through an injective generator function: x = g(z).Main result. Under Assums. 1, we prove the identifiability of both the latent representations and the cluster vectors, v c , in all four combinations of unit-normalized (i.e., when the latent space is the hypersphere, commonly used, e.g., in InfoNCE [Chen et al., 2020] ); and non-normalized (as in the original DIET paper [Ibrahim et al., 2024] ) latents, z, and weight vectors, w i . We state a concise version of our result and defer the full treatment and the proof to Thm. 1C in Appx. A: Theorem 1 (Identifiability of latents drawn from vMF around cluster vectors. Details omitted.). Let (f , W , β) globally minimize the DIET objective (1) under the following additional constraints: C3. the embeddings f (x) are unnormalized, while the w i 's are unit-normalized. Then w i identifies the cluster vector v C(i) up to an orthogonal linear transformation O: w i = Ov C(i) , for any i. Furthermore, the inferred latents z = f (x) identify the ground-truth latents z up to the same orthogonal transformation, but scaled. C4. neither the embeddings f (x) nor the w i 's are unit-normalized. Then the cluster vectors v c and the latent z are identified up to an affine linear and linear transformation, respectively. In all cases, the weight vectors belonging to samples of the same class are equal, i.e., for any i, j, C(i) = C(j) implies w i = w j .Intuition. DIET assigns a different (instance) label and a unique weight vector w i to each training sample. The cross-entropy objective is optimized if the trained neural network can distinguish between the samples. Thus, the learned representation z = f (x) should capture enough information to distinguish different samples, even from the same class. However, the weight vectors w i 's cannot be sensitive to the intra-class sample variance or the sample's instance label i (because multiple instances will usually belong to the same class). This leads to the weight vectors taking the values of the cluster vectors. As cluster vectors only capture some statistics of the conditional, feature recovery is more fine-grained than cluster identifiability. The interaction between the two is dictated by the cross-entropy loss, which is minimized if the representation z is most similar to its own assigned weight vector w i . Fig. 1 provides a visualization conveying the intuition behind Thm. 1. 
experiments: 
 In the following section, we empirically verify the claims made in Thm. 1 in the synthetic setting. We generate data samples according to Assums. 1: ground-truth latents are sampled around cluster centroids v c following a vMF distribution. Data augmentations, which share the same instance label i, are sampled from the same vMF distribution around v c . Synthetic Setup. We consider N data samples of dimensionality d generated from z ∼ p(z|v c ), sampled around a set of |C | class vectors, v c uniformly distributed across the unit hyper-sphere. We use an invertible multi-layer perceptron (MLP) to map ground truth latents to data samples. We train a classification headW = [w ⊤ i | N i=1] and an MLP encoder that maps samples to representations z ∈ R d using the DIET objective (1). While to verify Thm. 1 case C4., we do not normalize W , we do unit-normalize the weight vectors to validate Thm. 1 case C3. We verify our theoretical claims by measuring the predictability of the ground-truth z from z and v c from w i using the R 2 score on a held-out dataset. For identifiability up to orthogonal linear transformations, we train linear mappings with no intercept, assess the R 2 score and verify that the singular values of this transformation converge to one, while for identifiability up to affine linear transformations, we simply assess the predictive accuracy of a linear predictor with intercept.Table 1 : Identifiability in the synthetic setup. Mean ± standard deviation across 5 random seeds. Settings that match and violate our theoretical assumptions are ✓ and ✗ respectively. We report the R 2 score for linear mappings, z → z and w i → v c for cases with normalized (o) and not normalized (a) w i . For normalized w i , we verify that mappings z → z are orthogonal by reporting the mean absolute error between their singular values and those of an orthogonal transformation.normalized w i cases unnormalized w i R 2 o (↑) MAE o (↓) R 2 a (↑) N d |C | p(z|v c ) M. z → z w i → v c z → z w i → v c z → z w i → v c10 3 5 100 vMF(κ = 10) ✓ 98.6±0.01 99.9±0.00 0.01±0.00 0.00±0.00 99.0±0.00 99.9±0.00 10 5 5 100 vMF(κ = 10) ✓ 98.2±0.01 99.5±0.00 0.00±0.00 0.00±0.00 99.7±0.00 99.8±0.00 10 3 5 100 vMF(κ = 10) ✓ 98.6±0.01 99.9±0.00 0.01±0.00 0.00±0.00 99.0±0.00 99.9±0.00 10 3 10 100 vMF(κ = 10) ✓ 92.5±0.01 99.6±0.00 0.01±0.00 0.00±0.00 93.0±0.03 99.6±0.00 10 3 20 100 vMF(κ = 10) ✓ 70.8±0.02 97.1±0.01 0.03±0.00 0.00±0.00 81.9±0.01 99.7±0.00 10 3 5 10 vMF(κ = 10) ✓ 88.6±0.05 85.7±0.15 0.02±0.00 0.00±0.00 90.0±0.05 99.0±0.03 10 3 5 100 vMF(κ = 10) ✓ 98.6±0.01 99.9±0.01 0.01±0.00 0.00±0.00 99.0±0.00 99.9±0.00 10 3 5 1000 vMF(κ = 10) ✓ 99.3±0.00 99.9±0.00 0.00±0.00 0.00±0.00 99.2±0.00 99.9±0.00 10 3 5 100vMF(κ = 5)✓ 98.6±0.01 99.9±0.01 0.01±0.00 0.00±0.00 99.0±0.00 99.8±0.00 10 3 5 100 vMF(κ = 10) ✓ 99.0±0.00 99.9±0.00 0.00±0.00 0.00±0.00 99.1±0.00 99.9±0.00 10 3 5 100 vMF(κ = 50) ✓ 45.0±0.06 49.7±0.06 0.30±0.00 0.00±0.00 72.5±0.03 75.5±0.00 10 3 5 100 vMF(κ = 10) ✓ 98.6±0.01 99.9±0.01 0.01±0.00 0.00±0.00 99.0±0.00 99.9±0.00 10 3 5 100 Laplace (b = 1.0) ✗ 85.2±0.01 99.7±0.01 0.01±0.00 0.00±0.00 85.4±0.00 99.5±0.00 10 3 5 100 Normal (σ 2 = 1.0) ✗ 98.7±0.00 99.8±0.00 0.01±0.00 0.00±0.00 98.6±0.00 99.6±0.00Results. Tab. 1 depicts our results for synthetic experiments. For both cases, when W is and is not unit-normalized, the R 2 score for both the latents and the cluster vectors is close to 100%, except when the latent dimensionality is 20-such scalability problems are a common artifact in SSL [Zimmermann et al., 2021 , Rusak et al., 2024] . For unit-normalized W , the MAE is close to zero even in such cases. For a higher concentration of samples around v c (i.e., κ = 50) as well as a lower number of clusters (i.e., |C | = 10), the R 2 score decreases, which is also a common phenomenon, and is possibly explained by too strong augmentation overlap [Wang et al., 2022 , Rusak et al., 2024] . For a low number of clusters, high κ and a fixed number of training samples, the concentration of samples in regions surrounding centroids, v c , increases, a setting, refered to as "overly overlapping augmentations", known to be suboptimal and leading to a drop in downstream performance [Wang et al., 2022] . Our results also suggest that even under model misspecification (last two rows with non-vMF latent distributions), identifiability still holds. We provide an additional ablation study for the concentration of v c across the unit hyper-sphere in Appx. B. 
discussion: 
 Limitations. Our analysis proves the identifiability of DIET [Ibrahim et al., 2024] with a clusterbased DGP, thus providing the first such result for self-supervised parametric instance classification methods. However, our theory cannot yet explain the importance of label smoothing in DIET, noted by Ibrahim et al. [2024] , and it also remains to be seen whether such identifiability results scale for larger datasets, for which the large-dimensional classifier head in DIET in the original form is prohibitive. It also remains an issue that the vMF conditional distribution around cluster centroids jointly models intra-class sample selection and augmentations of samples, as we suspect that the supports of augmentation spaces of different samples do not overlap as much as it would be suggested by the choice of conditional. Also, we leave it for future work to investigate a formal connection to nonlinear ICA methods such as InfoNCE [Zimmermann et al., 2021] or the Generalized Contrastive Learning framework [Hyvarinen et al., 2019] .By modeling the DGP in DIET [Ibrahim et al., 2024] with a cluster-based latent variable model, we provide identifiability results for both the latent representation and the cluster vectors, which is the first of its kind for self-supervised instance discrimination methods. We also showcase this in synthetic settings, where we recover both the latents and cluster vectors even under model misspecification. We hope that our work inspires further research into investigating the theoretical guarantees of simplified but effective SSL methods like DIET.C3. the embeddings f (x) are unnormalized, while the w i 's are unit-normalized. If the system {v c |c} is diverse enough in the sense of Assum. 2, then: (a) w i = Ov C(i) , for any i ∈ I , where O is orthogonal linear;(b) h = f • g = α βO with the same orthogonal linear transformation, but scaled with α β . C4. neither the embeddings f (x) nor the rows of W are unit-normalized. Then:(a) h = f • g is linear;(b) w i identifies v C(i) up to an affine linear transformation. Furthermore, in all cases, the row vectors that belong to samples of the same class are equal, i.e., for any i, j ∈ I , C(i) = C(j) implies w i = w j .Remark. In cases C2 and C4, the cluster vectors are unnormalized and, therefore, can absorb the temperature parameter β. Thus β can be set to 1 without loss of generality. In case C3, it is f that can absorb β.Assumption 2 (Diverse data). The system {v c |c ∈ C } is said to be diverse enough, if the following |C | × 2d matrix has full column rank of 2d:EQUATIONwhere [x ⊙ y] i = x i y i is the elementwise-or Hadamard product.As long as |C | ≥ 2d, this property holds almost surely w.r.t. the Lebesgue-measure of S d-1 or any continuous probability distribution of v c ∈ S d-1 .Proof.Step 1: Deriving an equation characterizing the global optimizers of the objective.Rewriting the objective in terms of latents: we plug the expression x = g(z) into the optimization objective (6) to express the dependence in terms of the latents z:EQUATIONwhere the optimization is still over f (and not h = f • g).We note that the generator g is, by assumption, continuously invertible on the compact set S d-1 . Therefore, its image g(S d-1 ) is compact, too, and its inverse g -1 is also continuous. By Tietze's extension theorem [Wikipedia, 2024b] , g -1 can be continuously extended to a function F : R D → S d-1 . Therefore, any continuous function h :S d-1 → R d can take the role of f • g by substituting f = h • F continuous, since now f • g = h • (F • g) = h • id S d-1 = h.Hence, minimizing L z (f • g, W , β) (and by extension L(f , W , β)) for continuous f equates to minimizing L z (h, W , β) for continuous h:EQUATION)Expressing the condition for global optimality of the objective: We rewrite the objective (9) by 1) using the indicator variable δ I=i of the event {I = i} and 2) applying the law of total expectation:EQUATION)Using the properties that E A f (B) B = E A B f (B) and that E[δ I=i ] = P(I = i), we conclude that:EQUATION)By Gibbs' inequality [Wikipedia, 2024a] , the cross-entropy inside the expectation is globally minimized if and only if e β⟨wi,h(z)⟩ j∈I e β⟨wj ,h(z)⟩ = P(I = i|z), for any i ∈ I .(15)Moreover, the entire expectation is globally minimized if and only if the above equality (15) holds almost everywhere for z ∈ S d-1 . Using that instance label I is uniformly distributed, or P(I = j) = P(I = i), the likelihood of the sample being in class i can be expressed via Bayes' theorem as:EQUATION)Substituting ( 16) into (15) yields that for any i ∈ I and almost everywhere w.r.t.EQUATION)We now divide the equation ( 17) for the probability of a sample having label i with that of having label k and take the logarithm. This yields that L z (h, W , β) is globally minimized if and only ifβ⟨w i -w k , h(z)⟩ = ln p(z|I = i) p(z|I = k) (18)holds for any i, k ∈ I and almost everywhere w.r.t. z ∈ S d-1 .Plugging in the vMF distribution: Plugging the assumed conditional distribution from (4) into (18) yields the equivalent expression:EQUATIONholds for any i, k ∈ I and almost everywhere w.r.t. z ∈ S d-1 . Since h is continuous, the equation holds almost everywhere w.r.t. z if and only if it holds for all z ∈ S d-1 . Observe that if h = id| S d-1 , w i = v C(i) for any i ∈ I , and β = α, then the equation is satisfied. Thus, we can conclude that the global minimum of the cross-entropy loss is achieved.Step 2: Solving the equation for h, W and proving identifiability.We now find all solutions to prove the identifiability of the latent variables and that of the cluster vectors. Denote wi = β α w i to simplify the above equation to:EQUATIONh is injective and has full-dimensional image: We prove that h is injective. Assume that h(z 1 ) = h(z 2 ) for some z 1 , z 2 ∈ S d-1 . Plugging z 1 and z 2 into (20) and subtracting the two equations yields:EQUATIONfor any i, k. However, as the cluster vectors {v c |c} form an affine generator system, the vectors {v C(i) -v C(k) |i, k} form a generator system of R d (see Lem. 1). Therefore, ⟨y, z 1 -z 2 ⟩ = 0, for any y ∈ R d , which holds if and only if z 1 = z 2 . Hence, h is injective. By the Borsuk-Ulam theorem, for any continuous map from S d-1 to a space of dimensionality at most d -1 there exists some pair of antipodal points that are mapped to the same point. Consequently, no such function can be injective at the same time. Since h : S d-1 → R d is injective, the linear span of its image must be R d .Collapse of w i 's: We prove that wi = wk if C(i) = C(k), i.e., samples from the same cluster will have equal rows of W associated with them. Assume that C(i) = C(k) and substitute them into (20):EQUATIONHowever, we have just seen that the linear span of the image of h is R d , which implies that wi = wk . Consequently, we may abuse out notation by setting wc = wi if C(i) = c, which yields a new form for (20): Linear transformation from v a -v b to wawb : We now prove the existence of a linear map A on R d such that A(v a -v b ) = wawb for any a, b ∈ C . For this, we prove that the following mapping is well-defined:EQUATIONA : a,b∈C λ ab (v a -v b ) → a,b∈Cλ ab ( wawb ).(24)Since the system {v a -v b |a, b} is not necessarily linearly independent, we have to prove that the mapping is independent of the choice of the linear combination. More precisely if for some coefficients λ ab , λ ′ ab a,b∈CEQUATIONholds, then it should be implied that a,b∈CEQUATIONAssume that (25) holds. Then, the difference of the two sides is:EQUATIONTaking the scalar product with an arbitrary z ∈ S d-1 and using the linearity of the scalar product gives us:EQUATIONNow using (23) yields:EQUATIONHowever, the linear span of the image of h is R d , which implies that a,b∈CEQUATIONequivalent to (26). Therefore, the mapping is well-defined. The linearity of A follows trivially.h is linear: Equation ( 23) becomes: (32) This is further equivalent to ⟨y, A ⊤ h(z)⟩ = ⟨y, z⟩.EQUATION(33) Since y is arbitrary, we conclude that A ⊤ h(z) = z for any z ∈ S d-1 . Therefore A is an invertible transformation and h = (A ⊤ ) -1 is linear.Proving Thm. 1C case C4: We have shown that h is linear. Furthermore, from (31) it follows, by fixing b and defining ψ = Av b -w b , that wa = Av a + ψ, for any a ∈ C , (34) which proves case C4 of Thm. 1C.Proving Thm. 1C case C2: As a special case of the previous one, now we assume that h(z) is unit-normalized and maps S d-1 to S d-1 . That amounts to h = (A ⊤ ) -1 being linear, normpreserving, and therefore orthogonal. Consequently A is also orthogonal, h = A and (34) simplifies to β α w a = wa = Av a + ψ = h(v a ) + ψ, which proves C2 of Thm. 1C. Proving Thm. 1C case C1: We now assume that both h and w i 's are unit-normalized. Consequently, h = A is orthogonal linear and w a = α β Av a + ψ. Therefore, on one hand, the w a 's lie on a d-dimensional hypersphere of radius α β and center ψ. On the other hand, by definition, w a 's also lie on the unit hypersphere S d-1 . Since the system {w a |a ∈ C } is the bijective affine linear image of the affine generator system {v a |a ∈ C }, {w a |a ∈ C } is also an affine generator system (Lem. 1). Consequently, there could be at most one hypersphere in R d which contains all the w a 's. Hence α β = 1, ψ = 0, and w a = h(v a ), which proves C1 of Thm. 1C.Proving Thm. 1C case C3: Finally, we assume that w i 's are unit-normalized. As this is a special case of Thm. 1C C4, we know that there exists a constant vector ψ such that:EQUATIONfor any a ∈ C . We are going to prove that O = α β A is orthogonal and ψ = 0. Let O = U ⊤ ΣV be the singular value decomposition (SVD) of O. Consequently, after premultiplying with U, we receive:Uw a = ΣVv a + Uψ.(36) As orthogonal transformations U and V keep their arguments unit-normalized and {Vv a -Vv b } is still an affine generator system (Lem. 1), we may assume without the loss of generality thatEQUATIONfor any a ∈ C , where all v a 's and w a 's are unit-normalized.Let us assume that ψ ̸ = 0. In that case both sides of (37) can be scaled such that the offset ψ has unit norm. In this case w a 's are no longer on the unit hypersphere, but they instead have a mutual norm r. Assuming that the diagonal elements of Σ are σ = (σ 1 , . . . , σ d ), this is equivalent to:EQUATIONwhere [x ⊙ y] i = x i y i is the elementwise product. Eq. ( 39) is equivalent to the following:EQUATIONCollecting the equations for all a ∈ C yields:EQUATION)where D is the following |C | × (2d + 1) matrix: Using the unit-normality of v a 's, we see that (v a ⊙ v a ) ⊤ 1 d = 1. From this, it follows that the solutions are exactly the following:EQUATIONt = γ • 1 d 0 d γ + 1, where γ ∈ R.(43)Therefore, for any solution of (41) there exists γ such that:EQUATIONHowever, as the original transformation A was invertible, all singular values σ i are strictly positive and, thus, it follows that ψ = 0. Technically speaking, this is a contradiction to our initial assumption that ψ ̸ = 0. All in all, it follows that ψ = 0 is the only possibility. Therefore, (37) becomes: w a = Σv a , (46) where all v a 's and w a 's are unit-normalized. Following the same derivation yields:EQUATIONor, after collecting the equations for all a ∈ C :EQUATION)where B is theEQUATIONBy Assum. 2, B has full rank, thus, there is at most one solution to the equation Bt = 1 |C | . Due to the unit-normality of v a 's, this solution is exactly t = 1 d . However, as the singular values σ i are all positive, the only solution to σ ⊙ σ = 1 d is σ = 1 d . This is equivalent to saying that O = α β A is orthogonal.Furthermore, h = (A ⊤ ) -1 = ( β α O ⊤ ) -1 = α β O.In Tab. 2, we present additional ablation studies exploring the effect of varying the levels of concentration for v c across the unit hyper-sphere. We do not observe any significant impact on the R 2 scores from more concentrated cluster centroids v c .Table 2 : Identifiability in the synthetic setup. Mean ± standard deviation across 5 random seeds. Settings that match our theoretical assumptions are ✓. We report the R 2 score for linear mappings, z → z and w i → v c for cases with normalized (o) and unormalized (a) w i . For unormalized w i , we verify that mappings z → z are orthogonal by reporting the mean absolute error between their singular values and those of an orthogonal transformation.normalized w i cases unnormalized w i R 2 o (↑) MAE o (↓) R 2 a (↑) N d |C | p(v c ) p(z|v c ) M. z → z w i → v c z → z w i → v c z → z w i → v c10 3 5 100 Uniform vMF(κ = 10) ✓ 98.6±0.01 99.9±0.01 0.01±0.00 0.00±0.00 99.0±0.00 99.9±0.00 10 3 5 100 Laplace vMF(κ = 10) ✓ 98.7±0.00 99.5±0.00 0.01±0.00 0.00±0.00 99.1±0.00 99.8±0.00 10 3 5 100 Normal vMF(κ = 10) ✓ 98.2±0.01 99.2±0.01 0.01±0.00 0.00±0.00 99.2±0.00 99.8±0.00  
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript, especially making it more self-contained and accessible.

Reviews: review: title: title: review
main_review: main_review: The paper presents a proof and empirical validation on synthethic data that, under certain constraints, DIET (a self-supervised learning approach with sample index prediction) can recover latent representations and cluster centroids.

Admittedly, I didn't follow all details, but equations 8 and 9 in the appendix suggest that with the given constraints, the linear relationship between learned latent representations and original latent representations holds regardless the choice of f(x) (learned representations). That makes me question why f(x) is learned at all, and what the impact of learning it is. In this regard, I would have expected a more detailed analysis of architectural choices and parameters in the evaluation and in-depth discussion beyond "We also observe that for a higher concentration of samples around $v_c$ (i.e. $\kappa=50$) as well as lower number of clusters (i.e. |C|=10), identifiabilitty suffers..." (I assume $\kappa$ is the amount of samples per class).

The main body of the paper is somewhat hard to follow without the appendix, e.g., section 2 defines cases C3 and C4 in Theorem 1, while C1 and C2 are defined in the appendix (which is referred to before the theorem with "Thm. 1C in Appx. A").
title: title: Very solid, self-contained theoretical analysis with experimental verification of SSL representation formation, however, in a very simplified setup.
main_review: main_review: ## 1. Summary
The work presents a theoretical proof with additional experimental verification of the effectiveness of the self-supervised learning method DIET in a simplified, synthetic data-based setup. Specifically, the work proves the development of cluster-based latent representations under the instance-based SSL method DIET when instance augmentations and intra-class samples are drawn from von Mises-Fisher distributions.

## 2. Strengths and Weaknesses
The work addresses a significant problem related to how and which representations are formed using a type of self-supervised learning method.

The problem statement, the approach taken, the results, as well as the work's limitations, are all stated extraordinarily clearly.

A weakness is the potentially oversimplified problem setup for the theoretical analysis. However, given that it is a submission for the Extended Abstract track, I believe this work to be a promising first step. The authors fully acknowledge the limitations and propose next steps that are fully in line with my own questions regarding the work (reducing constraints and studying the effect of more complex and realistic data generation processes).

## 3. Questions

(More targeted for future work:)
How would you improve on the chosen data generation process to cover more realistic scenarios and are major changes expected?

## 4. Limitations

The work fully acknowledges its own limitations in its limitations section, primarily regarding scale and the limited complexity and introduced constraints for the theoretical study.

My main concern (which is also stated as a limitation in the work itself) is the validity of assuming a von Mises-Fisher distribution as the data-generating process, specifically the identical treatment of instance augmentations and other intra-class samples. This assumption might be an oversimplification and render the results trivial.

## 5. Ethical Concerns

None

## 6. Soundness

4 - Excellent

## 7. Presentation

4 - Excellent

## 8. Contribution

3 - Good
title: title: Interesting work that provides theoretical guarantees for a novel SSL method called DIET
main_review: main_review: This paper provides identifiability theory for a novel yet simple SSL method called DIET. 

**Pros:**
* This paper provides theoretical guarantees on the identifiability of features learned using a recently introduced SSL method called DIET. 
* The theoretical results are interesting, and are empirically verified under settings that both conform to and violate the theoretical assumptions. 
* The assumptions are clearly stated. 

I can't comment on the correctness of the theoretical results and proofs.

**Cons:**
* I feel like there are some missing pieces of important information/discussions that would help me better understand the intuitions and the results of this paper. Please see the questions/comments below. 

**Further questions/comments:**
* Lines `42 - 46`: The authors stated that the theoretical guarantees presented in this paper apply to SSL representations being used post-training (i.e., without removing the projection head/network), whereas other SSL theoretical/identifiability results apply to the representations with the projection network removed. One of the cited work is Zimmerman et al., 2021, which makes this comparison incorrect based on my understanding of that work. The theoretical guarantees in Zimmerman et al., 2021 also apply to representations being used post-training. 
* Why did the authors take a cluster-centric approach for the theory of the DIET method? What's the motivation for taking this approach?
* Assumption 1 (i): why does the set of cluster vectors {$\mathbf{v}_c$} need to be sufficiently large? What happens if there are fewer semantic classes? 
* Lines `90 - 91`: "for any $i, j, C(i) = C(j)$ implies $w_i = w_j$". Why is this the case? My understanding is that each $w_i$ corresponds to an instance with instance label $i$ whose ground truth latent $z$ is drawn from a vMF distribution around the cluster vector $v_c$ of class $c=C(i)$. Why do two instances with labels $i, j$ that have the same semantic class $c$ imply that their $w$ is identical? 
* Experimental results: why does the identifiability suffer when there's a lower number of clusters (i.e., when Assumption 1 (i) is violated)? While the authors provided justifications for this phenomenon (lines `124-127`), which includes the content-style partitioning of the latents and insufficient augmentation overlap, it is still not very clear to me why the identifiability would depend on the number of semantic classes/clusters. Further explanations on this empirical observation would be helpful.
title: title: Evaluation of the paper "DIETing: Learning Recognizable Features through Self Supervised Learning with Instance Discrimination"
main_review: main_review: This article introduces DIET, a simplified self supervised learning (SSL) method that focuses on instance recognition and claims to be able to recover recognizable cluster based latent representations. DIET has demonstrated its competitiveness by reducing the complexity of typical SSL pipelines, while also ensuring the recognizability of features. The author provides theoretical recognizability guarantees and empirical verification of synthesized data.