Self-Supervised Learning with an Information Maximization Criterion
Authors: Serdar Ozsoy
Shadi Hamdan
Sercan O Arik
Deniz Yuret
Alper Tunga Erdogan
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised learning
information maximization
representation learning
computer vision

Abstract: Self-supervised learning allows AI systems to learn effective representations from large amounts of data using tasks that do not require costly labeling. Mode collapse, i.e., the model producing identical representations for all inputs, is a central problem to many self-supervised learning approaches, making self-supervised tasks, such as matching distorted variants of the inputs, ineffective. In this article, we argue that a straightforward application of information maximization among alternative latent representations of the same input naturally solves the collapse problem and achieves competitive empirical results. We propose a self-supervised learning method, CorInfoMax, that uses a second-order statistics-based mutual information measure that reflects the level of correlation among its arguments. Maximizing this correlative information measure between alternative representations of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. An approximation of the proposed information maximization objective simplifies to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. The regularization term acts as a natural barrier against feature space degeneracy. Consequently, beyond avoiding complete output collapse to a single point, the proposed approach also prevents dimensional collapse by encouraging the spread of information across the whole feature space. Numerical experiments demonstrate that CorInfoMax achieves better or competitive performance results relative to the state-of-the-art SSL approaches.

Self-Supervised Learning with an Information Maximization Criterion
Serdar Ozsoy!.2 Shadi Hamdan Sercan Ö. Arik Deniz Yuret Alper T. Erdogan' KUIIS AI Center, Koc University, Turkey EEE Department, Koc University, Turkey 'CE Department, Koc University, Turkey *Google Cloud AI Research, Sunnyvale, CA {sozsoy19, shamdan17, dyuret, alperdogan)0ku. edu. tr soarik@google.com
Abstract
Self--upervised learning allows AI systems to learn effective representations from large amounts of data using tasks that do not require costly labeling. Mode col- lapse, i.e., the model producing identical representations for all inputs, is central problem to many sulf-supe learning approaches, making self--upervised tasks, such as matching distorted variants of the inputs, ineffective. In this article, we argue that a straightforward application of information maximization among alternative latent representations of the same input naturally solves the collapse problem and achieves competitive empirical results. We propose a self-supervised learning method, CorInfoMax, that uses a second-order statistics-based mutual information measure that reflects the level of correlation among its arguments. Maximizing this correlative information measure between alternative representa- tions of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. An approximation of the proposed information maximization objec- tive simplifies to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. The regularization term acts as a natural barrier against feature space degeneracy. Consequently, beyond avoiding complete output collapse to a single point, the proposed approach also prevents dimensional collapse by encouraging the spread of information across the whole feature space. Numerical experiments demonstrate that CorInfoMax achieves better or competitive performance results relative to the state-of-the-ar SSL approaches.
Introduction
Self-supervised learning (SSL) is an important paradigm with significant impact in artificial intelli- gence [1--]. In particular, SSL can significantly reduce the need for appensively-obtaine labeled data. Furthermore, the internal representations learned by SSL approaches are more suitable for task generalization in transfer learning applications 77. Recently, SSL. methods have even been demonstrated to achieve comparable performance to their supervised counterparts [8] y
The primary approach for SSL is to define pretext objective based on unlabeled data, whose optimization leads to rich and useful representations for downstream applications. For example, for visual processing, common way construct pretext task to obtain similar features for two different augmentations of the same input, which exploits the expected invariance of representations to particular transformations of the input. However, the corresponding optimization setting should avoid the (total) "collapse problem," which is described as generating the same feature output for all inputs. Figure (a) illustrates toy view of the total collapse problem, where all latent vector outputs converge to single point during training. We can also define its less severe version, which
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Vani [a x
Phctoo
Vali log (Rgg))
"Bit-Bong" Factior
(a)
b
Figure 1: Depictions in latent space- (a) "Total Output Collapse": all latent vectors converge to the same point, (b) "Dimensional Output Collapse": latent vectors are restricted strict subspace of the latent space, (c) Gradient dynamics of the CorInfoMax based on ( the ellipsoid surface reflects average spread pattern of latent vectors.
is referred as "dimensional collapse," where all latent vectors lie in strict subspace of the whole latent space as illustrated by Figure (b).
This article proposes an SSL approach based on the maximization of a form of mutual information among the alternative latent representations obtained from the same input. The mutual information maximization approach would serve two fundamental purposes for SSL:
(i). (Similarity) would enforce the representations obtained for the same input source be related to or dependent on each other,
(ii) (No Collapse) Since the entropy of each alternative representation is lower bounded by the mutual in- formation, maximizing mutual information ensures that latent representations have non-degenerate distributions avoiding collapse.
Therefore, the information maximization approach provides a natural solution the problems targeted by the existing SSL approaches, which are briefly surveyed in Sec. 2.2
The conventional choice for mutual information measure is Shannon Mutual Information (SMI) [10 For SSL, the fundamental limitations on the precise computation of SMI [ impose serious challenges model training. Especially when mutual information is large, require drastically
more samples (exponentially related to SMI) to obtain a reliable estimate of SMI [T3] However, the of large batch sizes improve precision may not be a favorable solution, (i) it may have an adverse impact on generalization performance 14 15 (ii) it can significantly increase memory requirements. Furthermore, estimating and optimizing SMI can significantly increase the computational burden. Finally, SMI maximization may induce nonlinear dependence between the representations corresponding the same input, where the resultant partitioning of the latent space may not be suitable as an input to a simple linear classifier.
To address these issues, we consider referred to as log-dete reg-determinant mutual information (LDMI) [16] [7]. The joint entropy measure corresponding to LDMI is the log dog-deeermnnant of the diagonally perturbed covariance matrix. The zero perturbation case boils down to the Shannon differential entropy for Gaussian distributed vectors. Unlike SMI, which reflects general dependence between its arguments, LDMI reflects linear dependence and is computationally more efficient.
mutual information measure,
In this article, propose novel SSL approach, correlative information maximization (CorInfoMax), derived from the LDMI measure. The loss function for CorInfoMax is obtained through the first-order approximation of LDMI and restricting the linear dependence to the identity map. The resulting optimization objective a Euclidean distance- distence-based loss function regularized by the log-determinant of the latent vector covariance matrix. This regularization term encourages the spreading of the feature vectors and acts as a natural barrier against the dimensional degeneracy and collapse in the feature space. Our numerical experiments confirm that CorInfoMax learns effective representations that perform well in downstream tasks.
The main contributions can be summarized as follows:

---

Based on information theoretical grounds, we propose novel framework, CorInfoMax, with an interpretable loss function that has explicit components for minimizing variance of positive samples ("the attraction factor") and for making full use of the embedding space to avoid dimensional collapse ('the big-bang factor").
We introduce a cumputationally-effiiiee approximation to LDMI that simplifies it by using an identity mapping instead of general linear mapping. This approximation directly minimizes representation variance, which is desired for self-supervised learning.
CorlnfoMax relies only on second-order statistics, does not require negative samples, and does not force embeddings to be uncorrelated.
In addition to these theoretically appealing features, the proposed CorInfoMax framework achieves either better or similar performance results compared to the state-of-the-ar SSL approaches.
The following is the organization of the article: Section provides discussion of the relevant literature related to the proposed approach. We introduce the log-determinant entropy and mutual information measures by highlighting LDMI as a measure of linear dependence in Section 3 In the same section, we derive the objective function of CorInfoMax as a variation on the LDMI mea- sure. Section $ introduces the proposed correlative information maximization-basee self-supervised learning method. Section provides the numerical experiments illustrating the performance of the proposed approach. The appendix provided in the supplementary document contains the details related to these experiments. Finally, Section 3 presents the discussion and conclusions.
2 Related work
2.1
Determinant maximization for unsupervised learning
The determinant maximization criterion utilized in our framework has been used as an effective algorithmic tool in unsupervised matrix factorization methods such as nonnegative matrix factorization
(NMF) [8 19.. simplex-structured matrix factorization (SSMF) (20). sparse component analysis (SCA)
21 bounded component analysis (BCA) 22 and polytopic matrix factorization (PMF) 24
In the generative models of these frameworks, the input data is assumed to be linear transformations of some latent vectors. Furthermore, these latent vectors are assumed to be sufficiently scattered in their domain. Maximizing the determinant of the latent covariance matrix spreads the latent vector estimates to capture this presumed scattering of the generative model samples. Similarly, the log- determinant of the latent vector covariance matrix in the CorInfoMax objective causes the spreading of latent vectors in their ambient space and, therefore, avoids collapse. According to 177, the determi- nant maximization criterion used in these matrix factorization frameworks is essentially equivalent to correlative information maximization between input and its latent factors based on the LDMI objective, where the latent vectors are constrained to lie in their domain determined by the generative model.
In the CorInfoMax approach, we utilize determinant maximization for correlative information maxi- mization among latent space vectors rather than between inputs and their latent space representations.
2.2 Handling collapse for SSL
Collapse is a central concern for SSL with the same input in computer vision, and we can categorize the existing approaches according how they handle this issue. For example, contrastive methods, such as SimCLR S and MoCo [24]. define an objective that pushes features for different inputs (negative samples) away from each other while keeping the representations corresponding the same input (positive samples) close to each other. The number and the choice of the negative samples appear as critical factors for the performance and scalability of contrastive methods. Another category of SSL is the distillation methods, such as BYOL 4 and SimSiam [25 which avoid collapse through the asymmetry of alternative encoder branches and algorithmic tuning 26 Other SSL approaches such as DeepCluster [27] SeLa (28) and SvAW [ enforce a cluster structure in the feature space to pre- vent constant output. As a different line of algorithms. decorrelated batch normalization (DBN) 29 Barlow Twins [5]. Whitening MSE (W-MSE) [50] and VICReg (6. use feature decorrelation [l] as
means to avoid information collapse. The effective whitening mechanisms used in these methods aim for the isotropic spread of information inside the feature space which also prevents "dimensional col- lapse" illustrated Figure (b). The reference 12 constructs an SSL loss function using the Hilbert- Schmidt Independence Criterion, kernel-based independence measure, to maximize the dependence
3

---

between alternative representations of the same input. Finally, B3 proposed graph representations for the embeddings positive samples and the corresponding spectral decomposition algorithm.
The proposed CorInfoMax approach is most related to the decorrelation- baased methods discussed above, mainly due to correlation-basee measures. However, unlike the decorrelation methods, CorInfoMax does not constrain latent vectors to be uncorrelated. Instead, it avoids covariance matrix degeneracy by using its log-determinant as regularizer loss function. Furthermore, the information maximization principle is more direct and explicit for the CorInfoMax algorithm.
2.3 Information maximization for unsupervised learning
The maximization of SMI has been proposed as an unsupervised learning mechanism in different but related contexts. As one of the earliest approaches, Linsker proposed maximum information transfer from input data its latent representation and showed that it is equivalent to maximizing the determinant of the output covariance under the Gaussian distribution assumption [ Around the same time frame, Becker Hinton 35 put forward a representation learning approach based on the maximization of (an approximation of) the SMI between the alternative latent vectors obtained from the same image. The most well- well-known application the Independent Component Analysis (ICA) Infomax algorithm [36] for separating independent sources from their linear combinations. The algorithm targets to maximize the mutual information between mixtures and source estimates while imposing statistical independence among outputs. The Deep Infomax approach Ai extends this idea to unsupervised feature learning by maximizing the mutual information between the input and output while matching prior distribution for the representations.
is essential underline that the proposed method clearly distinguishes itself from the Deep Infomax in 37 Our objective is not to maximize the mutual information between inputs and outputs of a deep network. Instead, we maximize the mutual information content of the alternative latent representations of the same input. From this point of view, our approach is closer to what is aimed at by [55 However, we use a different (correlative) information measure, which is computationally more efficient and induces special form of linear dependence among alternative latent representations of the same input, which may be more desirable considering the goal of generating features for linear classifier.
3 CorInfoMax as a criterion based on log-determinant mutual information
This section derives the optimization objective for the CorInfoMax framework as a variation on the LDMI measure. As described in Appendix A the LDMI between random vectors x and y is given by
((0 (x:y)
1 = (log det + =T) + log det det + +II)
RRy
4
log deet(RR,, Rxy (Ry
RRy(By + =I) Rx + eI)
log det R (Rx eI) - + =I));
where Rg and Ry are auto-covariance matrices for and y, respectively, and Rxy is their cross-covariance matrix.
There three potential advantages of using LDMI in over SMI in A.I for SSL:
It is based only on second-order statistics. On the other hand, SMI is statistic based on the joint PDFs of the argument vectors, and its accurate estimation has high sample complexity. In contrast, it is more practical to obtain (the estimates of) the auto-covariance and the cross-covariance matrices, discussed in Section 4
ii There are benefits of information maximization principle in connection with self-supervised training. The maximization of SMI of two vectors induces a general, potentially nonlinear dependence between them, whereas the maximization of LDMI increases correlation, or linear dependence. Therefore, LDDM-baaedd information maximization is expected to organize the feature space being more favorable for data-efficient and low-complexity linear or shallow supervised classifiers as targeted in SSL applications.
ii. The resulting LDMI-basee objective functions are more interpretable they involve the log- determinant the projector-spac covariance, whose maximization clearly avoids feature collapse, significant concern in SSL methods.

---

The primary motivation for obtaining an LDMI variant is to increase the similarity between the alternative latent representations of the same input by enforcing the identity transformation between them.
Let (1) and (22) represent alternative latent representations corresponding the same input. One po- tential SSL approach would be to pursue direct maximization of I (z() in 1 As discussed earlier, this would maximize the correlation between (4) and 22 therefore inducing a linear depen- dence between them. For the SSL application, we would prefer the identity mapping an arbitrary linear relationship such that the alternative latent representations corresponding to the same input concentrate in the same neighborhood of the latent space. Therefore, we modify the LDMI expression in ① to impose this constraint. For this purpose, we apply the first-order Taylor series approximation, log det(Ce C++D) log det +C)((C))T((" C- on the third term in the right side of (I. which provides
log det - R2 (R20)) + =I)
(2)) + =I)
Tr(R2 (1) R R1
11110
R (()))) + log det
2(2)
1 min F( (Azz(2) bi) + Plog(e).
E A..b
Therefore, the expression in 2 corresponds to the mean square error of the best linear (affine)
MMSE estimator of (1) from z multiplied by e-1. Similarly, if we apply the same approximation
to the fourth term on the right side of T we obtain
log det (2) R (R2(1 +sI) 02 + =I)
min F( - (A2z(1) b2) P Plog(s).
E A2.b2
In order to induce the identity mapping between (1)) and (2) we constrain A = 2 I, and b1 by O, which transforms both ( and (3) into E- E(Il 2(() (2:) + const. Therefore,
scaling the expression in by and using this modification, we obtain
(z (2) = log det + +I) + log det (R2 + +=I) 2-- Be-'E([12 -z--zz 2 (2).
(4)
where we ignored the constant terms. We refer to 4 as the stochastic CorInfoMax objective function. we propose a SSL method based on the optimization of this objective function.
In Section
4 LDMI based self-supervised learning
This section introduces the proposed correlative information maximization (CorInfoMax) approach for SSL. We start by describing the presumed setting for the pretext task, which is matching the latent representations of different augmentations of the same input in Section 4.1 Section 4.2 is the main section where we propose the correlative information maximization algorithm for SSL. Finally, discuss the implementation complexity of the proposed approach in Section
4.1 Self-supervised learning setting
We start by describing the presumed self-supervised learning setup, which is illustrated Figure
The input is a sequence of tensors [X][] E RMM J E Z}, where each index sample corre- sponds to batch of N images with dimensions M x K. As discussed in Section I the pretext task that we consider is matching the latent representations of the two augmentations of the same input. Therefore, we first apply the augmentation funccions aj (-(-) and a²(-) to obtain the transformed
versions of the input sequence, i.c., [ = a, (4) E MMMx will illustrate particular choices of augmentations in the examples provided Section 5 We note that the proposed setting can be potentially extended to multi-modal schemes where the augmentations are defined for multiple modalities of the same input, such as visual and sound.
Z}, E (1, We
We represent Siamese networks to be trained by SSL with the mappings f, w/ (Nx) for = 1,2,
where U E I MxXXN is the input, w(g) NNN represents the trainable parameters of the 4th network.
5

---

Augmentation a1(-)
DNN-1
Projector.
fi(: w DNN)
Pi Pl(Wpp)
Z
P
Imput
M
X
Augmentation-2 2
DNN-2
Projector-2
a2(+)
x²(2)
fe(: w DNN)
P2(:: w
P
Figure 2: SSL setup: consider two parallel encoder branches corresponding to two different augmentations of the same input X. Augmented views are fed into Siamese networks f followed by projector p. basically 3-layer MIP. N stands for batch size, M K for image dimension, P for feature dimension of projector output.
The outputs of the DNN for different augmented inputs are represented by Y =
[1( w(o WNN) E FRN, E (1,2), E Z, where F is the output feature dimension. To be used in self-supervised training, we project the outputs of the enconders using projector networks py (Y; 1,2, where Y is its input, and w/y is the set of trainable parameters the 9th projector.
The outputs of the projector network for different augmentations represented by Z(+)[] (Y())) 7]: w/?) E RPPN, E (1,2), Z, where P is the projector head dimension.
For the numerical experiments in the article, we consider the weight-sharing setup where both encoders use the same network weights. We provide more details in Appendix
4.2 Correlative mutual information maximization
We define the CorInfoMax approach for SSL through the optimization problem maximize WDNN, Wp
(5a)
where j(() is the sample based estimate of at the phtbatch which can be written as
2
(zz [2) = log det(R,,)) (R.,,, [1] eI) + log det Rg [1] + eT) EN
Z 1 z 11
(6)
where the rightmost term stands for the sample based estimate of the term 2- (( 2 ( in and B20 [[] and Ry [4] are the auto-covariance matrix estimates for the projector heads the batch. If the batch size is large enough, these estimates can only be obtained from the current batch samples. However, due to hardware limitations and considering the fact that intermediate batch sizes offer better accuracy [14] [1), sufficiently-lare batch sizes may not be possible to obtain reliable covariance estimates. Therefore, we adopt recursive covariance matrix estimation across batches 38 The corresponding covariance update expressions take the form
1 (0) u " (1,2). N
R (ot
[1] AR 1]
A)
where S < 1 is the forgetting factor, and ( represents the batch of mean-centralized projector outputs defined as [ Z [ مz (w) [ where (p {1,2)) represents the mean estimates for the projector outputs, updated by [1 =
+ (1 X) 2
for q {1,2). (See Appendix C for CorInfoMax pscudocode.)

---

It is informative to inspect the terms in the objective function J(z(), 2)) in 6
Minimization of N' |Z | 7 E acts as force to pull the representations of alternative augmentations toward each other, which we refer to as the aftraction factor.
+ a1) acts as a dispersion force causing non-degenerate (for & = 0) expansion of projection vectors in the P-dimensional space, which we informally refer as big-bang factor. Therefore, the covariance determinant acts as a regularization or barrier function avoiding collapse in the latent space. Consequently, it provides convincing replacement for negative samples in contrastive methods to prevent feature collapse. Appendix Killsstraee the eigenvalues of the projector covariance vector for the CIFAR-10) training, which demonstrates that the CorInfoMax criterion spreads information throughout the feature space, avoiding dimensional collapse.
Maximization of log det(RR,.)
Figure (c) illustrates the learning dynamics of the CorInfoMax approach based on (6) with a toy picture. In this figure, the ellipsoid is a representative surface for the level set of the quadratic function (z - ea 2(: (z - ()))), which reflects the spreading pattern of the latent vectors around their mean. The black arrows represent the gradient of the log (et((2g))) 260) regularization factor, which acts as force to push latent vectors away from the center of the ellipsoid (pult) and therefore corresponds the expansion force of the "big-bang" factor. In the same figure. the white arrows correspond to the gradients corresponding to the attraction factor, i.e., the Euclidean distance between two positive pairs. In summary, we can view the learning dynamics of CorInfomax SSL an expansion in the latent space, while the representations of positive samples are attracted to each other. More information on the CorrInfoMax optimization setting is provided in the Appendix M Furthermore, Appendix contains a visualization of the embeddings obtained using the CorInfoMax criterion.
4.3 Computational complexity
The main difference between our and related SSL methods the extra log-determinant computa- tion. The log determinant is typically computed using a decomposition method such the LU decomposition, whose gradient requires a matrix inversion. Both determinant and inversion have the same complexity matrix multiplication 39 i.e. (nn)) with 2 < < 3 for multiplying two n x n matrices. our experiments with GPUs, we observe an almost flat runtime cost up to 1024, which implies that the overhead due to the log det cost is insignificant in practice. We find extra cost of log determinant and its gradient to be negligible compared to the rest the model computation, whose runtime is dominated by the encoder, as confirmed by the runtime experiments reported in Appendix G
5 Experiments
5.1 Implementation details
Datasets: We perform experiments on 10, 100 440
Tiny ImageNet 411. COCO 42]. ImageNet-100 and ImageNet-1K 1K 43 datasets For ImageNet-100, we use the same subset
of ImageNet-1K 4 as related work
(See Appendix D for more details.)
The experiments consist of consecutive stages: pretraining and linear evaluation. We first perform the unsupervised pretraining of the encoder network f by applying the proposed CorInfoMax method described in Section 4.2 on the training dataset. After completing pretraining, perform the linear evaluation, a standardized protocol to evaluate the quality of the learned representations
Training Procedure:
For the linear evaluation stage, we first perform supervised training of the linear classifier using the representations obtained from the encoder network with frozen coefficients on the same training dataset. Then, we obtain the test accuracy results for the trained linear classifier based on the validation dataset.
'CorInfoMax's
source code
is publicly
available in http:://giihub..co//rddrrra
corinfomax ssl
7

---

Computational Resources: For ImageNet-100 and ImageNet-IK, we pretrain our model on up to 8 A100 Cloud GPUs. The remaining datasets are trained using single T4 and V100 Cloud GPU. Linear evaluations are performed using the same type and amount of computational resources. The details related batch sizes are described Section 5.1.3
5.1.1 Input augmentations
During the pretraining stage, two augmented versions of each input image are generated as shown in Figure During this process, each image is cropped with random size, resized to the original resolu- tion, followed by random applications of horizontal mirroring, color jittering, grayscale conversion, Gaussian blurring, and solarization. Since we the same augmentation parameters BYOL 4 and VicReg 6 each augmentation branch uses the same probability values for these randomized operations except Gaussian blurring and solarization, which different probabilities.
During the training phase the linear evaluation stage, a single augmentation of each input image is produced by random cropping and resizing followed by random horizontal flip. For the test phase of linear evaluation, we use resize and center crop augmentations, similar to We provide more details about the augmentations in Appendix E
5.1.2 Network architecture
Encoder Network: For CIFAR datasets, we use modified form of ResNet-18 architecture
48
similar to 8 33
We use standard ResNet-500 48) for Tiny ImageNet, ImageNet-100 and for ImageNet-100. In all cases, the last fully
ImageNet-1K, and also standard ResNet-18( connected layer is removed. Therefore, the encoder output size is 512 and 2048 for ResNet- 18 and ResNet-50, respectively. The encoder network shares weights between augmented branches (see
Appendix B
The output of the encoder network is fed into the projector network, in Figure The projector network p is a 3-layer MLP, with ReLU activation functions for the hidden layers and linear activation functions for the output layer. The projector dimensions
Projector Network:
2
for the CIFAR-10 dataset, 4096-4096-128 for the CIFAR-100, Tiny ImageNet and imageNet-1000 and 8192-8192-512 for the ImageNet-1K. Finally, we perform the fz-normalization on the projector
output.
Linear Classifier: For the linear evaluation phase, employ a standard linear classifier whose input is the fright-frozen encoder network's 's output.
5.1.3 Optimization
For pretraining, we use 1000 epochs with a batch size of 512 for CIFAR datasets, and 800 epochs with batch size of 1024 for Tiny ImageNet. For ImageNet-100 experiments, for ResNet-18 and 200 epochs for ResNet-50, with batch size of 1024 for both. ImageNet-1K experiments are conducted as 100 epochs with a batch size of 1536. We use the SGD optimizer with momentum of 0.9 and a weight decay of 1 x 10 -0-4. The initial learning rate is 0.5 for CIFAR datasets and Tiny ImageNet, 1.0 for ImageNet- 100, and 0.2 for ImageNet-IK. These learning rates follow the cosine decay with a linear warmup schedule.
400 epochs
We use the modified form of (6) as our objective function, where we replace 2--1 attraction coefficient, which we consider as a separate hyper-parameter. In our experiments, the diagonal perturbation is
N with the
250 for CIFAR-10, a = 2000 for ImageNet-1K, and a = 500 for Tiny ImageNet and ImageNet-100. The forgetting factor À 0.01 for all datasets except Tiny ImageNet and ImageNet- which have À 0.1. Details with coefficients of our loss function are provided in Appendix F
10- while
1000 for CIFAR-100,
For linear evaluation, the linear classifier is trained for 100 epochs with batch size of 256 for all datasets. We used the SGD optimizer with a momentum of 0.9, and without weight decay. For all datasets except ImageNet-1K. K, cosine decay schedule is utilized with initial and minimum learning rates of 0.2 and 10-3 respectively. For ImageNet- K, we use step scheduler with a starting value of 25, which is reduced by factor of 10 every 20 epoch.

---

5.2 Results
We evaluate the learned representations from the CorInfoMax pretraining by following the linear evaluation protocol explained in Sec. 5.1 Table [shows that CorInfoMax achieves state-of-the-an performance in linear classification after pretraining. Due to the limited size of the validation sets (5-10K), differences less than 0.5% are not statistically significant. The full comparison provided in Table I0 in Appendix H It is also interesting to observe the progress of the LDMI measure during the CorInfoMax training process, which is illustrated Appendix L
Table 1: Top-1 accuracies (%) under linear evaluation on different datasets. Results are reported 4 for CIFAR-10 and CIFAR-100, 33 for Tiny ImageNet, 46,
from 50 33
50 for ImageNet-100
(IN-100) in in ResNet-50, 49 for ImageNet-100 (IN-100) in ResNet-18, 25 33
for ImageNet-
(IN-1K), the case of the result of model in more than one resource, we integrate the largest score.
We bold all top results that are statistically indistinguishale.
CIFAR-10
CIFAR-100
Tiny-IN
IN-100
IN-1K
Method
ResNet-18
ResNet-18
ResNet-50
ResNet-18 ResNet-50
ResNet-50
SimCLR
3
91.80
66.83
48.12
77.04
66.5
SimSiam
25
91.40
66.04
46.76
78.72
81.6
68.1
Spectral
3
92.07
66.18
49.86
66.97
BYOL 4
92.58
70.46
80.32
78.76
69.3
W-MSE 2
91.55
66.10
69.06
MoCo-V2
51
92.94
69.89
79.28
67.4
Barlow
92.10
70.90
80.38
68.7
VICReg 92.07
68.54
79.40
68.6
CorInfoMax 93.18
71.61
54.86
80.48
82.64
69.08
We compare the semi-supervised learning performance of CorlnfoMax with VICReg 6 For fair comparison, VICReg is pretrained for 100 epochs on ImageNet-1K then semi-supervised learning performances of both models are evaluated by fine-tuning the encoders with 1(%) and 10(%) of the labeled ImageNet-IK dataset. The results are presented Table while details are provided in Appendix F.5
Table Top-1 accuracies (%) under semi-supervised classification on ImageNet-1K dataset after 100 epoch pretraining. VICReg is pretrained and evaluated using hyper-parameters reported in [6].
Method
1% of samples
10% of samples
VICReg
44.75
62.16
CorInfoMax 44.89
64.36
We experimented with transfer learning on object detection and instance segmentation with the same procedure and code of MoCo 52], and reproduce the results with the MoCo V2 [53] checkpoint. Our algorithm shows competitive performance with MoCo V2 [53], related details are provided in Appendix
Discussions and conclusions
We proposed novel SSL framework based on the correlative information maximization, CorInfoMax, which provides a natural solution for obtaining organized representations that are scattered in the latent space and avoiding collapse. Our experiments demonstrate the ofite-of-theaan performance of CorInfoMax in numerous downstream tasks. The loss function CorInfoMax does not have any significant impact training time. Future work can further improve CorInfoMax focusing on shortcomings like improved selection of hyperparameter and augmentation, a core problem in SSL. overall.

---

Acknowledgments and Disclosure of Funding
This work/research was supported by KUIS AI Center Research Award. We gratefully acknowledge the support of Google Cloud Credit Award.
References
[1] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160-167, 2008.
[2] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
[3] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[4] Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 13:2212712211 284, 2020.
[5] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 12310-12320. PMLR, 2021.
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-invarrane--cvvaia regular- ization for self-supervised learning. arXiv preprint arXiv:2105.04900 2021.
[7] Linus Eriesson, Henry Gouk, and Timothy Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 5414-5423, 2021.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1559716077. PMLR, 2020.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[10] Thomas M. Cover and Joy A. Thomas. Elements of information theory. Wiley-Interscience, 2006.
[11) Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[12] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. International Conference on Learning Representationss 2019.
[13] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual infor- mation. In International Conference on Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[14] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team, An empirical model of batch-batch training. arXiv preprint 1rXiv:1812.06..6 2018.
[15] Dominic Masters and Carlo I Luschi. Revisiting small batch training for deep neural networks. arXiv preprint 17X0011004..71,,
10

---

[16] Zhanghao Zhouyin and Ding Liu. Understanding neural networks with logarithm determinant entropy estimator. arXiv preprint arXiv:1401.3420, 2021.
[17] Alper T Erdogan. An information maximization based blind source separation approach for dependent and independent sources. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
[18] Xiao Fu, Kejun Huang, Nicholas D Sidiropoulos, and Wing- Kin Ma. Nonnegative matrix factorization for signal and data analytics: Identifiability, algorithms, and applications. IEEE Signal Process. Mag., 36(2):59 80, March 2019.
[19] Xiao Fu, Kejun Huang, Bo Yang. Wing-Kin Ma, and Nicholas D Sidiropoulos. Robust volume minimization- matrix factorization for remote sensing and document clustering. IEEE Transactions on Signal Processing, 64(23):6255-6222 6268, August 2016.
[20] Tsung- Han Chan, Wing-Kine Kin Ma. ArulMurugan Ambikapathi, and Chong- Yung Chi. A simplex volume maximization framework for hyperspectral endmember extraction. IEEE Transactions on Geoscience and Remote Sensing, 49(11):4177-4193, May 2011.
[21] Eren Babatas and Alper T Erdogan. An algorithmic framework for sparse bounded component analysis. IEEE Transactions on Signal Processing, 66(19):5194-5205 August 2018.
[22] Huseyin A Inan and Alper T Erdogan. A convolutive bounded component analysis framework for potentially nonstationary independent and/or dependent sources. IEEE Trunsactions Signal Processing, 63(1):18-30, November 2014.
[23] Gokcan Tatli and Alper T. Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing, 69:5431-5447, 2021. doi: 10. 109 12918.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1729-97335 2020.
[25] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[26] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021.
[27] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning visual features. Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[28] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.0537,, 2019.
[29] Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 791-800, 2018.
[30] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. International Conference on Machine Learning, pages 3015-3024. PMLR, 2021.
[31] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598-9608. 2021.
[32] Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems, 14:15543-15550 2021.
11

---

[33] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
[34] Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117 1988.
[35] Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces random-dot stereograms. Nature, 355(6356):161-166)) 1992.
[36] Anthony J Bell and Terrence J Sejnowski. An information-maxxiizaaiio approach to blind separation and blind deconvolution. Neural compulation, 7(6):1 129-1159, 1995.
[37] R Devon Hjelm, Alex Fedorov, Samuel Lavoie Marccii-arrc,,lonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 18888806670 2018.
[38] Justinian Rosca, Deniz Erdogmus, José Príncipe, and Simon Haykin. Independent Component Analysis and Blind Signal Separation: 6th International Conference, ICA 2006, Charleston, SC, USA, March 5-8, 2006, Proceedings. volume 3889. Springer, 2006.
[39] James R Bunch and John E Hopcroft. Triangular factorization and inversion by fast matrix multiplication. Mathematics of Computation, 28(125) 1974.
[40] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny (maggs.(2009), 2009.
[41] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3,
[42] Tsung- Yi Lin, Michael Maire, Serge J. Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. In David J. Fleet. Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13thh European Conference, Zurich, Switzelland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014. doi: 10. 20079778-3331-10660 48. URL httpp:://dae //doi wrcloi.oogg///o 1011111000073--1-10000000--- 48
[43] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. (0CCVPR.2009.22000
[44] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 19X15:1906.05849 2019.
[45] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-211 21802221999110 2020.
[46] Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. Advances in Neural Information Processing Systems, 34, 2021.
[47] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 1920-1929, 2019.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[49] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. Solo- learn: library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1-6, 2022.
12

---

Journal of [50] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-awai self-supervision. Advances in Neural Information Processing Systems, 34, 2021.
[51] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[52] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint urXiv:1911.05522. 2019.
[53] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint wrXiv:2003.04297 2020.
[54] Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear estimation. Prentice Hall, 2000.
[55] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017. doi: 10.1 N..109//CCV2221..3222
[56] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. machine learning research, 9(11), 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] As claimed the abstract and introduction, in the article we provide novel self supervised learning framework based on the informa- tion maximization principle and provide numerical experiment results reflecting its performance.
(b) Did you describe the limitations of your work? [Yes] We stated sensitivity hyperpa- rameter selections as a limitation in Section 3
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We believe that our work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Theoretical discussions in Section 3 and Section clearly provide the mathematical settings used our derivations.
(b) Did you include complete proofs of all theoretical results? [Yes] We show the details of the derivations of our proposed objective in Section B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material
URL)? [Yes] We include the Python codes of our experiments in supplementary materials and the details of the experiment results in Section and in Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provided all the details about training, its phases, hyperparameter selections in Section F and in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In Appendix, we provide information on the confidence interval of our experimental results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We reported the details about the computation resources and computation times in Section 5 and in Appendix.
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite references for all the data sets, experiment results and network models in Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include our code supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zGduO2pn7hY", "reviewer": "Reviewer_J7ne", "summary": "The paper introduced CorInfoMax, a second order statistics based mutual information method to avoid collapse problem in self supervised learning (SSL). It used log-determinant mutual information (LDMI) as the measure between data pairs with different augmentations. Mathematically, the paper gave step-by-step formulation to derivate the final form of the objective via its first-order Taylor series approximation\nExperimental results on CIFAR10/CIFAR100/ImageNet gives competitive results.\n", "questions": "As commented in strengths and weakness", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This work is novel and well intuited. Collapse issue is a well-known problem in SSL, traditionally Shannon Mutual Information (SMI) is applied to mitigate the collapse issue with several drawbacks, such as large batch requirement. The log-determinant mutual information (LDMI) naturally enforces latent distributions have non-degenerate distributions which can avoid collapse. Therefore, this method is well motivated and makes sense\n- Clean and neat formula. Step by step formula gives clean loss format in Eq (8). It clearly indicated data under different augmentation should be closed to each other and log determinant formula enforces the model from model collapse. The result is elegent.\n- This result shows state-of-the-art SSL performance. \n\nWeakness\n- Mathematical formula is too heavy in the paper. It is a bit hard for audience to follow. I would strongly recommend a more detailed inference in any appendix to show how they are coming from\n- More though experiments should be provided to demonstrate the effectiveness of the approach. For example, the paper argues the drawbacks for SMI and introduce LDMI measure as a better solution to avoid collapse. However, no evidence shown hot does it better avoids collapse. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "yIjOd57EBaf", "reviewer": "Reviewer_nwQX", "summary": "This paper extensively addresses the collapse problem by proposing second-order statistics-based mutual information measure that reflects the level of correlation among the inputs. It claims that maximizing this correlative information measure between alternative representations of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. All these are proved to simplify as a regularization term which acts as a natural barrier against feature space degeneracy. In general, the paper is well written and reasonably understandable. I found the underlying theory to be very strong, however the presented experiments aren't sufficient to show the strength of the work.", "questions": "(1) Currently, the abstract is quite long and puts a lot of unnecessary details and background information, which fades out the necessary information. It does not mention anything on the achieved results.\n\n(2) Space missing in line 201 between \"linear\" and \"(affine)\"\n\n(3) In line 330, the phrase \"objective function\" appears twice, which sounds repetitive.\n\n(4) In equation (10), the dimensionality of the projector is bottleneck because of the correlation matrix computation. Could you please comment on that?\n\n(5) In table 1, under the CIFAR-10 and ImageNet-100/ResNet-18 column, the boldfaced numbers are mistakenly written?\n\n(6) It would be good to direct the main comparison table in the supplementary in the main paper.\n\n(7) Appropriate direction to different sections of the supplementary material should be done from the main paper.\n\n(8) In figure 2, the drawing of encoder and projector could have been in different colours. I mean if they share weights, they get the same colour, otherwise if they don't share weights, they get different colours.", "limitations": "The authors have not mentioned about the limitation of the work in the paper. A possible limitation could be the consideration of only the global information and not considering the local information, which might make this method only applicable for global task, such as classification. I suspect this type of model won't work very well on the task like involving fine-grained feature understanding.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n(1) The paper is theoretically grounded on correlative information measure of representation. Additionally, it is very well reasoned, I like the way it is presented.\n\n(2) On classification downstream task, the proposed methods has obtained state-of-the-art results one some datasets. Although, I found that the results are not robust and the comparison is not complete. Please also see my experiment related comments in the weaknesses section.\n\n**Weaknesses**\n\n(1) The experiments shown are bit limited and not very robust. Currently the experiments are only done on one downstream task i.e. classification. More experiments on other tasks, such as object detection, semantic segmentation, few-shot learning, representation transferability etc could have been interesting and could have strengthen the method. Moreover, ablation of the model on different hyperparameters should have been studied to understand the strength of the model. Furthermore, standard experimental strategies are not followed, as an example, for classification tasks, self-supervised models follow \"linear\" and \"fine-tuning\" strategies, which is not followed in this work.\n\n(2) I wonder if there is any way to estimate the amount of mutual information maximized via the proposed CorInfoMax approach. It would be interesting to see some intermediate results or plots showing the evolution of estimated mutual information as the training progresses. Also it would have been interesting to see analysis on the performance behaviour of the model with evolution of mutual information. Does higher mutual information always guarantee a better trained model?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iMwiVf56Gny", "reviewer": "Reviewer_sfru", "summary": "This paper presents a self-supervised learning with an information maximization criterion among alternative latent representations of the same input that naturally prevents dimensional collapse. It considers a second-order-statistics based mutual information measure, the log-determinant mutual information (LDMI), which is equivalent to Shannon mutual information under Gaussian distribution. A further first-order approximation to the log-determinant of the sum of two matrices is used to simplify the final objective to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. Consequently it  avoids the collapse problem establishes relevance among alternative representations by increasing the linear dependence among them.  Experiments on 4 image datasets show that the proposed approach gives better results than contrastive and non-contrastive methods.  ", "questions": "The proposed method maximizes Shannon mutual information under Gaussian distribution, and a further first-oder approximation is also used. \n\nExperiments on synthetic Gaussian data set in (Poole et al. 2019, McAllester and Stratos 2020) are need to see the effect of first-oder approximation when estimating mutual information, and the result should compare with the DoE method in McAllester and Stratos' paper. \n\nFinally are there any theoretical explanations to explain why it gets better results than contrastive and non-contrastive methods? The contrastive methods are essentially a lower bound method of Shannon mutual information. ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow. However, there are several issues I'd like to address. The revised version has cleared my concerns.\n\n1. This following statement is misleading: \"A common self-supervision task is to match the latent representations that come from the distortions of the same input.\" It's true in computer vision, but not true in speech or natural language processing. \n\n2. The following statement is not quite right: \"maximizing SMI between the representations of the same input is a challenging task whose implementation would require relatively large sample sizes [11, 12].\" [11] demonstrates that when mutual information is large,  existing variational lower bounds degrade and exhibits either high bias or high variance. In fact McAllester and Stratos prove that serious statistical limitations are inherent to any lower bound method of measuring mutual information. More specifically, any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N). \n\n3. Collapse is a central concern for SSL with the same input in computer vision.\n\nReference:\nDavid McAllester and Karl Stratos, Formal Limitations on the Measurement of Mutual Information, The Twenty Third International Conference on Artificial Intelligence and Statistics, 108:875-884, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TErFwzgFeMt", "reviewer": "Reviewer_GZwK", "summary": "This paper presents a new self-supervised learning method based on the log-determinant mutual information proposed earlier. Experiments on three small size datasets and one medium size dataset show the effectiveness of the proposed method.", "questions": "1. This article only contains the results of classification tasks, and the performance of the proposed method on tasks such as detection, transfer, and semi-supervised learning also needs to be given.\n2. The classification results on ImageNet-1000K should be reported.\n3. The lines 159-162 are difficult to read. I don't understand that $h_{LD}^*\\left( {x\\left| y \\right.} \\right)$ can be treated as a measure of remaining uncertainty. I hope the author can give a detailed explanation.\n4. This paper claims that the linear property of LDMI is the key to good performance. However, from \" Self-Supervised Learning with Kernel Dependence Maximization\", we can know that the nonlinear property of the Kernel Dependence is the key to good performance. I hope the author can give in-depth analysis and comparison.\n5. Barlow-Twins is also proposed to explore the linear correlation between different augmentations of the same sample. I hope the author can give the difference and connection between the proposed method and Barlow-Twins, both experimentally and theoretically.\n6. For large-scale dataset such as ImageNet-1000, the log-det cost is significant in practice.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The experimental results are good.\n2. Overall, the article is easy to read.\n\nWeaknesses:\n1. The experimental part is relatively weak. \n2. The novel of this paper is limited. The main contribution of this paper is to directly generalize the earlier proposed log-determinant mutual information to the field of self-supervised learning.\n3. Section 3 is superfluous.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept