Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment
Authors: Dong Hoon Lee
Sungik Choi
Hyunwoo Kim
Sae-Young Chung
Venue: NeurIPS 2022 Accept (2022)
Keywords: unsupervised representation learning
pseudo-labeling
mutual information maximization
self-supervised learning

Abstract: This paper proposes Mutual Information Regularized Assignment (MIRA), a pseudo-labeling algorithm for unsupervised representation learning inspired by information maximization. We formulate online pseudo-labeling as an optimization problem to find pseudo-labels that maximize the mutual information between the label and data while being close to a given model probability. We derive a fixed-point iteration method and prove its convergence to the optimal solution. In contrast to baselines, MIRA combined with pseudo-label prediction enables a simple yet effective clustering-based representation learning without incorporating extra training techniques or artificial constraints such as sampling strategy, equipartition constraints, etc. With relatively small training epochs, representation learned by MIRA achieves state-of-the-art performance on various downstream tasks, including the linear/${\it k}$-NN evaluation and transfer learning. Especially, with only 400 epochs, our method applied to ImageNet dataset with ResNet-50 architecture achieves 75.6% linear evaluation accuracy.

Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment
Dong Hoon Lee KAIST donghoomlee@kiist.aag
Sungik Choi
LG Al Research sungik. choi choi@lgresearch...
Hyunwoo Kim
Sae-Young Chung KAIST
LG AI Research twkim@1greseerch...
sychungster
Abstract
This paper proposes Mutual Information Regularized Assignment (MIRA), pseudo-labeling algorithm for unsupervised representation learning inspired by information maximization. We formulate online pseudo-labeling tion problem to find pseudo-labels that maximize the mutual information between the label and data while being close to given model probability. We derive a fixed-point iteration method and prove its convergence to the optimal solution. In contrast to baselines, MIRA combined with pseudo-label prediction enables a simple yet effective clustering based representation learning without incorpo- rating extra training techniques artificial constraints such sampling strategy, equipartition constraints, etc. With relatively small training epochs, representation learned by MIRA achieves hate-of-theeaar performance tasks, including the linear/k-NN evaluation and transfer learning. Especially, with only 400 epochs, our method applied to ImageNet dataset with ResNet-50 architec- ture achieves 75.6% linear evaluation accuracy. Our implementation is available at https: //tp:://gttttttt www./ooovinoooo/omiiiiiiio.
optimiza-
various downstream
Introduction
There has been growing interest in using large-scale dataset to build powerful machine learning models [45]. Self-supervised learning (SSL), which aims to learn useful representation without labels, is suitable for this trend; it is actively studied in the fields of natural language processing 19 20 and computer vision [10 30). In the vision domain, recent SSL methods commonly use data augmentations and induce their visual representation to be
www.nnttttoooiiiiiiitt They have achieved state-of-ther oftthe-rrrr performance surpassing supervised representation a variety of visual tasks,
including semi-supervised learning g 53), transfer learning (22). and object detection [13]
Meanwhile, a line of work uses clustering for un-/self supeevisd representation learning. They explicitly assign pseudo-labels to embedded representation via clustering, and the model is thereby trained predict such labels. These clustering- representations are encouraged to encode the semantic structure of data. Prior works [51 49 4 32
methods can account for inter-data similarity:
have shown encouraging results in small-scaled settings; Caron et al. [6] show that it can also be applied to the large-scaled dataset 39 have adopted the philosophy of augmentation invariance and achieved strong empirical results.
to non-curated dataset [7]. Recently, several works 2 g
Work partially done during internship at I.G AT Research.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

They typically assign pseudo-labels using augmented views while predicting the labels by looking at other differently augmented views.
Despite its conceptual simplicity, a naive application of clustering to representation learning is hard to achieve, especially when training with large-scale datasets. This is because clustering- -aased methods are prone to collapse, i.c., all samples are assigned to single cluster; hence, recent methods heavily rely on extra training techniques or artificial constraints, such pre-training [50), sampling strategy [6), equipartition constraints [2] g to avoid collapsing. However, it is unclear if these additions are appropriate or how such components will affect the representation quality.
In this paper, we propose Mutual Information Regularized Assignment (MIRA), a pseudo-labeling algorithm that enables clustering- based SSL without any artificial constraints extra training techniques. MIRA is designed to follow the infomax principle [40] and the intuition that good labels are something that can reduce most of the uncertainty about the data. Our method assigns pseudo-label a principled way by constructing an optimization problem. For a given training model that predicts pseudo-labels, the optimization problem finds a solution that maximizes the mutual information (MI) between the pscudo-labels and data while considering the model probability, We convex optimization problem and derive the necessary and sufficient
formulate the problem condition of solution with the Karush-Kuu KurrnhhKKuhh-- Kuhn-Tucker (KKT) condition. This solution can be achieved by sred-pointt iteration that we prove the convergence. We remark that MIRA does not require any form of extra training techniques or artificial constraints, e.g., equipartition constraints.
We apply MIRA clustering- based representation learning and verify the representation quality on several standard self-supervised learning benchmarks. We demonstrate its state-of-the-art perfor- mance linear/k-NN evaluation, semi-supervised learning, and transfer learning benchmark. We further experiment with convergence speed, scalability, and different components of our method.
Our contributions are summarized follows:
We propose MIRA, a simple and principled pseudo-label assignment algorithm based on mutual information. Our method does not require extra training techniques artificial constraints.
We apply MIRA to clustering- baased representation learning, showing comparable performance against the state-of-the-art methods with half the training epochs. Specifically, MIRA achieves 75.6% top-1 accuracy ImageNet linear evaluation with only 400 epochs of training and the best performance in out 11 datasets in transfer learning.
Representation by MIRA also consistently improves over other information- bassad SSL methods. Especially our method without multi-crop crop augmentation achieves 74.1% 1% top-1 accuracy and outperforms BarlowTwins 53 a baseline information maximization-base self-supervised method.
2 Related works
Self-supervised learning SSL methods are designed to learn the representation by solving pretext tasks, and recent state-of-the-art methods encourage their learned representations be augmentation invariant. They are based on various pretext tasks: instance discrimination [T0
11 1 14] metric
learning [88 [1), self-training [54] 9. and clustering 2
6 8] only a few account for encoding the
35 consider the nearest neighbors in the latent space, our method belongs to the clustering-based SSL method that flexibly accounts for inter-data similarity. Meanwhile, many SSL methods are prone to collapsing into a trivial solution where every representation mapped into a constant vector. Various schemes and mechanisms are suggested to address this, e.g., the asymmetric structure, redundancy reduction, etc. We will review more relevant works in detail below.
semantic structure of data. While some works 47 21
Collapse preventing Many SSL approaches rely on extra training techniques and artificial as- sumptions to prevent collapsing. clustering- based methods, DeepCluster [6] adapts a sampling strategy to sample elements uniformly across pseudo-labels to deal with empty clusters; SeLa (2) and SWAV impose equipartition constraints to balance the cluster distribution. Similarly, SelfClassi- fier I uses uniform pseudo-label prior, and PCL. 39 employs concentration scaling. DINO 9 and ReSSL [54 address collapsing by specific combinations of implementation details, i.e., centering and scaling with an exponential moving average network; their mechanism for preventing collapse

---

grad
model learning
image 27
augmentation
model fe
qi prediction
swapped prediction
Σ Wij log 9
pseudo-labels U1:B
differently augmented
① pseudo-labeling
Pr
DkL(u, p)
MI(data;label)
model fo
D
fixed point iteration u (n+1) (m)
augmentation
P8
mini-batch
probabilities
Figure I: Overview representation learning via MIRA. our representation learning, MIRA pro- vides pseudo-labele with model probabilities, and the model is learned by predicting the pseudo-labed Our main contribution is in the ① pseudo-labeling process that accounts for mutual information between the pseudo-labes and data. In MIRA, optimal pseudo-labels are computed through the fixed-point iteration (Eq. 7 Given such pseudo-labels, @ model updates its parameters by gradient update swapped prediction loss.
unclear. In this work, we show our method can naturally avoid collapsing without any of these assumptions training techniques. We achieve results better than baselines with simple but novel information regularization algorithm. We take a more detailed comparison with SeLa and SwAV
after explaining our method in Sec. 3.3
Information maximization Information maximization a principal approach to learn representa- tion and to avoid collapse. DeepInfoMax [31] propose the MI maximization between the local and global views for representation learning; the existence of negative pairs prevents training toward the trivial solution. Barlow/Twnns (53) and W-MSE 2 address the collapsing with redundancy reduction that indirectly maximizes the content information embedding vectors. Among clustering- based 34 maximizes the MI between the embedding codes enable representation learn approaches, IIC
ing; similar to ours, TWIST (66] proposes combining the MI between the data and class prediction as negative loss the MI as a loss function and directly optimize their model parameters with gradient descent of the loss. However, the direct optimization of MI terms by updating model parameters often leads to sub-optimal solution [26]: TWIST copes with this issue by appending the normalization layer before softmax and introducing an additional self-labeling stage. In contrast, MIRA addresses the difficulty of MI maximization in principled way via explicit optimization.
with an augmentation invariance consistency loss. Both IIC and TWIST use
3 Method
In this section, we explain our pseudo-labeling algorithm-MIRA. When applying MIRA to rep- resentation learning, we follow the basic framework of clustering- based representation learning that alternates between labelin,,,,lin, i.c., cluster assignments, and model training to predict such labels. Figure iillustrates our representation training cycle. We will first explain our main contribution, MIRA (pseudo-labeling) and then explain how it applies to model training.
Our idea to employ the information maximization principle into pseudo-labeling. We formulate an optimization problem for online clustering that assigns soft pseudo-labels to mini-batch samples 3.1.. The problem minimizes the KI. divergence between the model prediction (probability) and pseudo-label while maximizing the mutual information between the data and pseudo-label. We propose an iterative fixed point method to solve the optimization problem (Sec. 3.2) For the model training, we use the swapped prediction loss
g
(Sec. 3.3

---

3.1 Mutual information regularized pseudo-labeling
We have a classification mode for an image æ, where K is a predefined number of clusters. The model probability p of an image then given by the temperature т scaled output of the model p = softmax(fo(z)))
fo parametrized by that outputs -ddimensional logit fo(x)) RRK
in Caron et al. 8 9.. For a mini-batch of input images X (æ. ==1 we denote the set of model probabilities P {p.} C R.K In our pseudo-labeling, for the given model probabilities P. we want to assign pseudo-labels W = {w,]] ! them.
that will be used for training the model by predicting
We argue that such pseudo-labels should maximize the mutual information (MI) between themselves and data while accounting for the model probabilities P. Let B E [1, B} and Vw E {1, K} be the random variables associated with the data index in mini-batch and labels by probability distributions W = ) respectively. Our online pseudo-label (cluster) assignment is determined by solving the following optimization problem:
W arg min WCAx B
E (w.. p:) 3i:::::::::
(1)
where Ak == fw E R wTlk 1}. indicates an empirical (Monte Carlo) estimates of MI, and / is a trade-off parameter. The problem consists of the (1) KL divergence term that makes pseudo-labels to be based on the model probability and (2) MI term between the pseudo-labels and data to induce more information about data into the pseudo-labels. By combining these two terms, we provide a refined pseudo-label that take account of both the model probability and MI.
To make the optimization problem tractable, we substitute the MI term í with the mini-batch estimates
of the entropy 11 (Q'wlB) and marginal entropy (1'w) in Eq. We get:
K
B
K
f(Y'w: B) = ()w)) - #( (3) =
1 log By
+
Wig log wig.
(2)
==1
1 B
B
K
1 B
K
DKL(Wi, p.)
Why Pig
Why log Wiis
(3)
B
j=1
1=1 1=1
1-3 3
K
K
W arg min W WCAR
Why Pig
Wig log my + B
> my log wj,
(4)
B
where Wy Ei=1 Why is the marginal probability of a cluster j with W. In practice, we find the optimal point W of the optimization problem Eq, 4 for pseudo-labeling.
3.2 Solving strategy
To solve efficiently, we propose a fixed point iteration that guarantees convergence to the unique optimal solution W of our optimization problem. The method based on the following proposition. Proposition 1. For 3 E [0, 1), the problem Eq. 4 is a strictly convex optimization problem with a unique optimal point W that satisfies the following necessary and sufficient condition:
P
vi.i) E (1,..., B} x (1, (1...,K), K), w is
(5)
K
W
K
P
The proposition T is driven by applying the Karush-Kuhn Kuhn-Tucker (KKT) conditions to the optimization By substituting the necessary and sufficient condition (Eq. (5) into the marginal
problem Eq. 4
probability By
B
Σ 1=
Waf.
we get the necessary and sufficient condition for w":
B
PG
P
n
w
(6)
B
B
W
Ps
W
Pik
In our setting, the model consists of an encoder, projection head, and classification (prototype) head as in Caron al. (8 [ the encoder output will be used representation.

---

Based on Eq. 6 we propose the update rule for (
1 using the fixed point iteration as Vj E (I,...,)) K), uj ((tl)) = B
C
1==1
follows:
1 B
1
1-8
Púj
1k= (uk
K
(3))
1-8 Pik
(n)
where Uj converges w" satisfied at the convergence. We can easily get i"i by Eq. 5 when the optimal marginal probability w" is given. The proof of the proposition and convergence is in the Appendix.
since the necessary and sufficient condition (Eq. 6 is
By using the iterative updates of Eq. we get desirable pseudo-labels W This requires few lines of code that are simple to implement. We observe that a few steps of iteration are enough for training. This is supported by the convergence analysis in Sec. 4.3 We use this fixed point iteration for pseudo-labeling and name the method-Mutual Information Regularized Assignment (MIRA) since it finds the pseudo-labels that regularized by the mutual information.
3.3 Representation learning with MIRA
We explain how our pseudo-labeling method is applied to representation learning. We integrate the computed pseudo-labels with swapped prediction loss [8] to train the model; the model is trained to predict the pseudo-labels from the augmented views using the other views. Specifically, given two mini-batches of differently augmented views X (m) = {
E {1,2), MIRA independently assigns pseudo-labels (mm) = {u. (m)) ),m for each mini-batch. In parallel, model fs provides the temperature тı scaled softmax prediction (()) {q (m) prediction loss is given as follows:
(m))
i==1 )
, of each mini-batch. The swapped
L(X x (2))
(2) + (2) Q()) ((U
K
1
(1)
(2) log 4ij
(2) (1) Wis log 9:
(8)
B
This loss function (Eq. 8 is minimized with respect to the parameters 0 of the model fe used to produce the predictions ((1) (2)) For more detailed information about swapped prediction loss,
please refer to Caron et al.
The pseudo-code of MIRA for representation learning with Eq, 8 is provided in the Appendix. In the following experiments, we verify the effectiveness MIRA for representation learning purpose. We note that MIRA can integrate recently suggested self-supervised learning components, such as exponential moving average (EMA) multi-crop (MC) augmentation strategy following the baselines [4] g [9. For convenience, in the rest of this paper, we call the representation learning with MIRA also as MIRA. We discuss some further details follows:
Preventing collapse The MI term in Eq. taakes a minimum value when collapsing happens. MIRA naturally avoids collapsed solution via penalizing assignment that exhibits low MI. Specifically, unless starting from the collapsed state, MIRA finds MI-maximizing points around the model prediction; it will not choose collapsed pseudo-labels. Hence, the iterative training to predict such labels will not collapse whenever the prediction of pseudo-labels is achievable. Our empirical results verify that MIRA does not require extra training techniques or artificial constraints to address collapsing.
Comparison to SwAV and SeLa SeLa 2 and SwAV g formulate their pheudo-labeling process into optimization problems, i.e., optimal transport (OT) problem, and solve iteratively with Sinkhorn- Knopp (SK) algorithm [16] To avoid collapse and apply the SK algorithm, they assume the equipartition of data into clusters. Mathematically, the difference to MIRA how to deal with the marginal entropy. SeLa and SwAV constrain the marginal entropy to maximum equipartuiaatiioonnn while MIRA decides by MI regularizationf] Asano et al. (2) argue that their pseudo-labels with the OT problem maximize the MI between labels and data indices under the equipartition constraint.
Thht q. (m) = wowtmao (m
"Adding the equipartition constraint into Eq.
problem converts the OT problem of SwAV B
5

---

Table 1: Linear evaluation with respect to training epochs. All models use a ResNet-50 encoder and trained on training set of ImageNet. Results style: best,
Table 2: Linear evaluation on ImageNet. Com- parison with other self-supervised methods on Im- ageNet. SL denotes for self-labeling by [26]. Re- sults style: best
i are results from 12
second best
Method
Arch.
Epochs
Top-1
Top-5
Epochs
Supervised
R50
Method
100
200
400
800
PCL R9
R50
200
67.6
without multi-crop augmentations
SimSiam
12
R.50
800
71.3
SimCLR T0
66.5
68.3
69.8
70.4
SimCLR-v2
R50
800
71.7
BYOL
28
66.5
70,6
73.2
74.3
InfoMin 46
R550
800
73
91.1
SimSiamf
30
68.1
70.0
70.8
71.3
BarlowTwing 53
R50
1000
73.2
91.0
MoCo-v3
68.9
73.8
VicReg
R50
1000
73.2
91.1
SelfClassifier
R50
800
74.1
DeepCluster-v2
70.2
TWIST w/o ST. 26
R50
800
74.1
SwAVf
66.5
69.1
70.7
71.8
BYOL 2X
R50
1000
74.3
91.6
TWIST
70.4
70.9
71.8
72.6
MoCo-v3 14
R50
1000
74.6
MIRA
69.4
72.3
73.3
74.1
DeepCluster-v2
R50
800
75.2
with multí-crop augmentations
SwAV
R50
800
75.3
DeepCluster-v2 g
75.2
DINO g
R50
800
75.3
SwAV S
72.1
73.9
74.6
75.3
TWIST w/ SL 26
R50
450
75.5
TWIST 26
72.9
73.7
74.4
74.1
MIRA
R550
400
75.6
92.5
MIRA
73.6
74.9
75.6
However, it more resembles assuming MI maximization and then finding the cluster assignments that are optimal transport to the model prediction. In contrast, MIRA directly maximizes the MI by regularization without artificial constraints. While SwAV performs better than SeLa in most self--upervised benchmarks, we verify that MIRA improves over SwAV in various downstream tasks.
Experiments
In this section, we evaluate the representation quality learned MIRA. We first provide the implementation details of our representation learning with MIRA (Sec. We present our main results on linear, k-NN, semi-supervised learning, and transfer learning benchmarks in comparison to other self-supervised baselines (Sec. 4.2). Finally, conduct analysis of MIRA (Sec. 4.3).
4.1 Implementation details
We mostly follow the implementation details of representation learning from our baselines 8 More training details about evaluation procedures and analysis are described in the Appendix.
Architecture The training model (network) consists of an encoder, projection head, and classi- fication head in Caron al. [8). We employ a widely used ResNet50 (2) our base encoder and use the output of average-pooled 2048d embedding for representation training and downstream evaluations. The projection head is a 3-layer fully connected MLP of sizes [2048, 2048, d]: each hidden layer is followed by batch normalization (33) and ReLU. The classification head is used to predict the pseudo-labels: is composed of an L2-normalization layer and a weight-normalizatin layer of the size d x K. We use d = 256 and K 3000.
Training details
train our model on the training set the ILSVRC-2012 ImageNet-Il Ik dataset [18] without using class labels. We use the same data augmentation scheme (color jit- tering, Gaussian blur, and solarization) and multi-crop strategy (two 224 224 and six
96)
used in Caron et al. [9]. We use batch size of 4096 and employ the LARS optimizer 52 with weight decay of 10 We use linearly scaled learning rate of x batch size/255 (27) with a base learning rate of 0.3 We adjust the learning rate with 10 epochs of a lincar warmup followed by cosine scheduling. We also use an exponential moving average (EMA) network by default. When
'Otherwise stated, also linearly scaled learning for evaluation training.

---

Table 3: K-NN classification results on Ima-
Table 4: Semi-supervised learning results on
geNet with respect subsets. For 1% and 10% results, we evaluate the baselines by models of
ImageNet. The baselines results are from [33 Results style: best, second best
official codes. Baseline results are from 9
1%
10%
ImageNet subset
Top-1
Top-S
Top-1
Top-5
Method
100%
10%
1%
Supervised
25.4
48.4
56.4
80,4
BYOL
28
64.8
57.4
45.2
SimCLR 10
48.3
75.55
65.6
87.8
SWAV
65.7
57.4
44.3
BYOL 28
53.2
78.4
68.8
89
Barlow Twins 5
66.0
59.0
47.7
SWAV B
53.9
78.5
70.2
89.9
DeepCluster-v2 8
67.1
59.2
46.5
BarlowTwing
[53]
55
79.2
69.7
89.3
DINO 9
67.5
59.3
47.2
MIRA
55.6
80.5
69.9
90.0
MIRA
68.7
60.7
47.8
using EMA, we set the momentum update parameter to start from 0.99 and increase it to by cosine scheduling. We use temperature scales of T. = 0.1, T; = 0.225 with a trade-off coefficient 3 = 2/3. We obtain soft pseudo-labels by 30 steps of the fixed point iteration. We further discuss this choice in Sec. Otherwise stated, we use the encoder model trained for 400 epochs with multi-crop augmentations for the downstream task evaluations in this section.
4.2 Main results
Linear evaluation Tables and report linear evaluation results. We follow the linear evaluation settings in [28 10. We train a linear classifier on the top of the frozen trained backbone with the labeled training set of ImageNet. We train for 100 epochs using an SGD optimizer with a batch size of 1024. We choose learning rate with local validation set in the ImageNet train dataset and adjust the learning rate by cosine annealing schedule. We apply random-resiae ree--rddeiiooooppooeeee and horizontal flip augmentations for training. We evaluate the representation quality by the linear classifier's performance on the validation set of ImageNet.
Table [ shows linear evaluation performance in top-1 accuracy for different training epochs. We train and report the performance MIRA two settings, with and without multi-crop augmentations. With multi-crop augmentations, MIRA consistently outperforms baselines while achieving 75.6% top- I accuracy with only 400 epochs of training. We also report that 200 epochs of training with MIRA can outperform the 800 epochs results of other baselines that do not multi-crops. Without multi- crop augmentations, MIRA performs slightly worse than B YOL (28) However, MIRA performs the best among the clustering-basee based 66, g] and information-driven [53 26 methods.
In Table we compare MIRA to other self- self-supervised methods with the final performance. MIRA achieves soute-of-the-aee performance on linear evaluation of ImageNet with only 400 epochs of training. While TWIST (26) can achieve similar performance to MIRA within 450 epochs, they require an extra training stage with self-labeling; without it, they achieve 74.1% 1% accuracy with 800 epochs of training. In contrast, MIRA does not require additional training.
Semi-supervised learning In Table 4 we evaluate the trained model on the semi-supervised learning benchmark of ImageNet. Following the evaluation protocol in 28 10], we add linear classifier on top of the trained backbone and fine-tune the model with ImageNet % and 10% subsets. We report top-1 and top-5 accuracies on the validation set of ImageNet. For the 1% subset, MIRA outperforms the baselines: both the top-1 and top-5 accuracies achieve the best. For the 10% subset, MIRA is comparable SWAV [8]
K-NN evaluation We evaluate the quality of learned representation via the nearest neighbor classifier. We follow the evaluation procedure of Caron et al. [9] We first store the representations of labeled training data; then, we predict the test data by the majority vote of the k-nearest stored representations. We use 1000/00% subsets of ImageNet training dataset to produce labeled representations. For ImageNet 1% and 10% subsets, we use the same subsets of semi sumi-sueervieed learning evaluation. We
We use the learning rate of 0.075 for multi-crop 400 epochs training.
7

---

Table 5: Linear evaluation results on the transfer learning datasets. Following Ericsson et al. we report top-1 accuracy on Food, CIFAR- SUN397, Cars, DTD; mean-per-class accuracy on Aircraft, Pets, Caltech- 101, Flowers; -oppint mAP metric on VOC2007. Results style: best
22
Aircraft
Caltech101
Cars
CIFAR1O
COFAR100
DTD
Flowers
Food
Pets
SUN397
VOC2007
wrg.
Supervised
90.18
73.75
InfoMin
46
38.58
87.3 47.84
41.04
91 49
73.43
74.73
87.18
69:53
86.24
61.00
83.24
72.21
McCo-v2
03
41.79
87.92
39.31
92.25
74.90
73.88
90.07
68.95
83,3
60.32
82.69
72.31
SimCLR-v2
92.9
BYOL.
28
53.87
91 46
56.4
93.26
77 86
76.91
94.5
73.01
89.1
59.99
81.14
77.05
DeepCluster-22
54.49
91.33
58.6
94,02
79,61
78.62
94.72
77.94
89.36
65,48
83.94
78.92
77.97
MIRA
$9.06
92.21
61.05
94,20
79.51
77.66
96.07
78.76
89.95
65.84
84.10
79.86
MIRA
MIRA
0.5
S innhorr-Knopp
Sinkham nopp
SSE (Log10)
-3.0
-5.5
-8.0
-10.5
-13.0
25
Iterations
lttrations
Figure 2: Convergence analysis of MIRA and Sinkhorn-Knopp We observe the converging behavior of MIRA (blue) and Sinkhorn- Knopp (yellow). We experiment with trained models of MIRA (left) and SwAV (right). Since both methods are proven to converge, we iterate each method 1000 steps and regard the results as ground truth. We report the aum-squared error (SSE) with respect to the converging point in the log scale.
use the same hyperparameter settings in Caron et with k 20 nearest neighbors, temperature scaling of 0.07, and cosine distance metric.
Table shows the classification accuracies the validation set of ImageNet. The results show that our method achieves the best evaluation performance. Specifically, our method outperforms the previous state-of-the-an DINO [9] 100% and 10% subset evaluation by 1.2 ~1.4%. We note that Barlow Twins [53], a method also motivated by information maximization, shows a strong performance of 47.7% in the 1 1% subset evaluation.
Transfer learning We evaluate the representation learned by MIRA on the transfer learning
benchmark with FGVC aircraft [T]. Caltech-101 (5]. Standford Cars 36
CIFAR-10//0 100 (37).
DTD 15), Oxford 102 Flowers 422 Food-101 [5). Oxford-IIIT Pets 43 SUN397 48 and Pascal
VOC2007 24 datasets. We follow the linear evaluation procedure from Eriesson et al. 22 that fits multinomial logistic regression classifier on the extracted representations of 2048d from the trained encoder. We perform a hyperparameter search on the L.2-nommaiization coefficient of the logistic regression model. The final performance is evaluated on the model that is retrained on all training and validation sets with the found coefficient.
Table T shows the performance of our algorithm compared to other basclines in 11 datasets. MIRA outperforms supervised representation on 10 out 11 datasets. Compared to other self-supervised methods, representations learned from MIRA achieved the best performance in 9 out of 11 datasets, with an average improvement of 0.9% over the second-best baseline method. The results confirm that the representation trained with MIRA has strong generalization ability for classification.
4.3 Analysis
Convergence pseudo-label assignment We study the convergence speed of the proposed fixed point iteration in MIRA. We also experiment with the Sinkhorn- Knopp (SK) algorithm [16] used in SwAV as baseline. Both methods have experimented with a batch size of 512. We observe the
TThe temperature scaling is used to calculate contributions exp(distance./r) and voting is weighted by the contributions of the nearest neighbors.

---

converging behavior with the pre-trained models from MIRA and SwAV. Results are averaged over 1000 randomly sampled batches.
Figure 2 shows the result of the converging behavior of our method (blue) and SK algorithm (yellow) on trained models of MIRA (left) and SWAV (right). Our fixed-point iteration converges faster than the SK algorithm in both pre-trained models, and our default setting of 30 steps of updates is sufficient for convergence. While we use 30 steps of updates follow the theoretical motivation in our main experiments, we observe that choosing a small number of iterations is possible in practice
Multi-crop and EMA reports an ablation study on how EMA and multi-crop augmentations affect our representation quality. We train model for 200 epochs in the settings with and without EMA or Multi-crop augmentations. EMA and Multi-crop augmentations greatly improve the linear evaluation performance as in Caron et al. 8 19. We take further comparison with baselines that are the same setups. While the only difference in the pseudo-labeling, our method outperforms SwAV 8 by 1.3% in top-1 accuracy. DINO [9] uses both multi-crop and EMA, our method outperforms DINO with fewer training epochs. The results validate the effectiveness of MIRA.
Table Ablation study about EMA and multi-crop augmentation, We report top-1 accuracy with linear evaluation on validation set of ImageNet. The results of SwAV is from [30].
Method
Multi-Crop
EMA
Epochs
Top-1
SWAV DINO
X
X
200
69.1
/
/
300
74.5
X
X
200
70.4
MIRA
X
/
200
72.3
V
V
200
74.9
Scalability We further validate MIRA's scalability on the small-and medium-scaled datasets. R8 is used base encoder throughout the experiments. While changing the base coder, other architectural details remain the same as in ImageNet- 1k. We do not apply multi-crop augmentations while using the EMA. We image sizes of 32x32 and 256x256 for small and medium datasets, respectively. Following the procedures in da Costa et al. [17]. we report the linear evaluation performance the validation set. More experimental details about the optimizer, batch size, augmentations, etc., are provided in the Appendix.
Table Linear evaluation performance in small-and medium sccaled datasets. We report top-1 and top-5 accuracies of linear evaluation validation dataset. The training results based 1000 and 400 epochs of training on CIFAR-10/100 10/100 and ImageNet-1000 ret.ee000.eeeeeeeeeeeeeeee respectively. Results style: best,
second best
CIFAR-10
CIFAR-100
ImageNct-100
Method
Arch.
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Barlow/wins
53
R18
92.10
99.73
70.90
91.91
80.16
95.14
BYOL
28
R18
92.58
99.79
70.46
91.96
80.32
94.94
DeepCluster-v2
R18
88.85
99.58
63.61
88.09
75.36
93.10
DINO g
R18
89.52
99.71
66.76
90.34
74.90
92.78
SWAV 8
R18
89.17
99.68
64.88
88.78
77.83
95.06
MIRA
R18
93.02
99.87
70.65
92.23
81.00
95.56
The results are in Table 7 In CIFAR-10 and ImageNet-100, method outperforms other self- supervised baselines by 0.4% and 0.7% in top-1 accuracy, respectively. For CIFAR-100, our method comparable to the best performing
Beeeiiin-BBllowwwiiii MIRA performs better in top-5 accuracy.
Training with small batch Throughout the experiments in Sec. 4.2
we use a batch size of 4096. While such batch size is commonly used in self-supervised learning, large amount of GPU memory required, limiting the accessibility. In Table g we test our method with smaller batch size of
'The result with small number of iterations is in the Appendix.

---

512 that can be used in an 8 GPU machine with 96GB memory. In this setting, we use the SGD optimizer with weight decay of 10 14-4_ We also test the robustness of pseudo-labeling with the Sinkhorn-Knopp algorithm in SwAV g and compare the results.
We
We report top-1 linear evaluation performance of both methods after 100 epochs of training. In the result, the performance gap between our method and SwAV is amplified from 2.9% to 6% in the reduced batch size of 512. One possible explanation is that since SwAV based on the equipartition constraint, the performance of SWAV harshly degrades when the batch size is not enough to match the number of clusters.
Table 8: Linear evaluation performance with smaller batch size. All results are based on ImageNet training. We also report the GPU memory usage and time spent for one epoch training. f result by us.
Method
Batch size
Epochs
GPU
GPU memory
Time per Epoch
Top-1
SwAVi MIRA w/o EMA MIRA
512
100
8 TITAN V
71 GB
23 min
62.3
512
100
TITAN V
71 GB
23 min
66.3
512
100
8 x TITAN V
73 GB
29 min 68.3
SWAV 12]
4096
100
66.5
MIRA w/o EMA
4096
100
16 x A100
486 GB
9 min
68.7
MIRA
4096
100
16 x A100
504 GB
9 min 69.4
5 Discussion
This paper proposes the mutual information maximization perspective pseudo-labeling algorithm MIRA. We formulate online labellig into a convex optimization problem with mutual information regularization and solve in a principled way. We apply MIRA for representation learning and demonstrate the effectiveness in various self-supervised learning benchmarks. We hope our simple yet theoretically guaranteed approach to mutual information maximization will guide many future applications.
Conclusion
Limitation and negative social impact Our mutual information regularization with optimization strategy seems applicable to various tasks and domains, e.g., semi-supervised training 38) However, we validate the effectiveness only sulf-supervised visual representation learning. We note that the performance of MIRA for non-classification downstream task e.g., detection, is not as dominating as in the classification tasks. In these tasks, methods that consider local or pixel-wise information achieve superior performance; incorporating such formulation into clustering- based methods seems to be an important future direction. Furthermore, despite our improved training efficiency, the self- supervised learning methods still require massive computation compared to supervised approaches. Such computational requirements may accelcrate the environmental problems of global warming.
Acknowledgments
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govemment(MSIT) (No.2022-0-00926), We thank Suyoung Lee, Minguk Jang, Dong Geun Shin, and anonymous reviewers for constructive feedbacks and discussions to improve our paper. Part of the experiments for this work was done at the Fundamental Research Lab of LG Al Research.
References
[1] E. Amrani, I.. Karlinsky, and Bronstein. Self-supervised classification network. arXiv preprint arXiv:2103. 10994, 2021.
[2] Y. M. Asano, C. Rupprecht, and A. Velaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020.
We list the experimental results in the Appendix.
10

---

[3] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: variance- iriiare-ivarrance-ovvaaaann regularization for self-supervised learning. ICLR, 2022.
[4] M. A. Bautista, A. Sanakoyeu, E. Sutter, and B. Ommer. Cliqueenn: Deep unsupervised
exemplar learning. In NIPS, 2016.
[5] L. Bossard, M. Guilaumin, and L. Gool. Food-100 101 mining discriminative components with random forests. In ECCV, 2014.
[6] M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[7] M. Caron, P. Bojanowski. J. Mairal, and A. Joulin. Unsupervised pre- pre-training of image features on non-curated data. In ICCV, 2019.
[8] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[9] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[11] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurlPS, 2020.
[12] X. Chen and K. He. Exploring simple siamese representation learning, In CVPR, 2021.
[13] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning, arXiv preprint 0rXh::2003.04297, 2020.
[14] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
[15] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
[16] M. Cuturi. Lightspeed computation of optimal transport. In NeurlPS, 2013.
[17] V. G. T. da Costa, E. Fini, M. Nabi, N. Sebe, and E. Ricci. solo-learn: library of self-supervised methods for visual representation learning. Journal of machine learning research, 2022.
[18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei. Imagenet: a large-scale hicrarchical image database. In CVPR, 2009.
[19] Devlin, M.-W. Chang. K. Lee, and K. Toutanova. Bert:pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[20] Du, E. Grave, B. Gunel, Chaudhary, O. Celebi, M. Auli, V. Stoyanov, and A. Conneau. Self-training improves pre-training for natural language understanding. In NAACL, 2021.
[21] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In ICCV, 2021.
[22] L. Ericsson, H. Gouk, and T. M. Hospedales. How well do self supervised models transfer? In CVPR, 2021.
[23] A. Ermolov, Siarohin, Sangineto, and N. Sebe. Whitening for self-supervised representa- tion learning. In ICML, 2021.
[24] M. Everingham, I.. V. Gool, C. K. Williams, Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 2010.
11

---

[25] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training ex- amples: an incremental bayesian approach tested on 101 object categories. In CVPR workshops. 2004.
[26] W. Feng, K. Tao, Z. Rufeng, L. Huaping, and L. Hang. Self-super learning by estimating twin class distributions. arXiv preprint arXiv:2110.074002 2021.
[27] Goyal, P. Dollár, Girshick, P. Noord-huis, L. Wesolowski, A. Kyrola, Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: training imagenet in I hour. arXiv preprint vrXiv:1006.02677 2017.
[28] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: a new approach self-supervised learning. In NeurIPS, 2020.
[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[31] R. D. Hjelm, A. Fedorov, S. L-Marchildron, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
[32] W. Hu, T. Miyato, S. Tokui, E. Matsumoto, and M. Sugiyama. Learning discrete representations via information maximizing self-augmented training. In ICML, 2017.
[33] loeffe and C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[34] X. Ji, J. Henriques, and A. Veldaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
[35] S. A. Koohpayegani, A. Tejankar, and H. Pirsiavash. Mean shift for self-supervised learning. In ICCV, 2021.
[36] J. Krause, J. Deng, M. Stark, and L.. Fei-Fei. Collecting a large-scale dataset of fine-grained cars. In Second workshop on fine-grained visual categorization, 2013.
[37] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[38] D.-H. Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on challenges in representation learning, 2013.
[39] J. Li, P. Zhou, C. Xiong, and S. Hoi. Prototypical contrastive learning of unsupervised represen- tations. In ICLR, 2021.
[40] R. Linsker. Self-organization in a perceptual network. IEEE Computer, 21:105-117 1988.
[41] S. Maji, E. Rahtu, Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 130..513,., 2013.
[42] M. E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics and image processing (IVCGIP), 2008.
[43] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[44] T. Piotrowski and R. L. G. Cavalcante. The fixed point iteration of positive concave mappings converges geometrically if a fixed point exists. arXiv preprint arXiv:2110.1 11055 2022.
[45] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Kruger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
12

---

[46] Y. Tian, Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? In NeurIPS, 2020.
[47] Wang, H. Liu, D. Guo, and F. Sun. Unsupervised representation learning by invariance propagation. In NeurIPS, 2020.
[48] Xiao, J. Hays, S. X. Yu, and D. Lin. Sun database: Large-scale scale scene recognition from abbey to zoo. In CVPR, 2010.
[49] J. Xie, R. Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
[50] X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan. Clusterfit: Improving generaliza- tion of visual representations. In CVPR, 2020.
[51] J. Yang, D. Parikh, and D. Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.
[52] Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 1708. 03888, 2017.
[53] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: self-supervised learning via redundancy reduction. In ICML, 2021.
[54] M. Zheng, S. You, F. Wang, C. Qian, C. Zhang, X. Wang, and C. Xu. Ressl: relational self-supervised learning with weak augmentation. In NeurlPS, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section Limitation and negative social impact
[|] Discussion
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section Discussion Limitation and negative social impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] in the Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] in the Appendix
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] supplemental material
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] in Appendix
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] in Appendix
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] in Appendix
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
code in supplemental material.
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yyOp9cT-ES", "reviewer": "Reviewer_mBbA", "summary": "This paper proposes Mutual Information Regularized Assignment (MIRA) which is a pseudo-labelling algorithm for unsupervised representation learning inspired by information maximization. The pseudo-labelling procedure is formulated as an optimization problem solved via a fixed-point iteration method that maximize the mutual information between the label and data while being close to a given model probability. The proposed methodology has been claimed to be converging relatively faster and to be achieving state-of-the-art performance on various downstream tasks such as linear evaluation and transfer learning. In general, the paper is well written and it is easily understandable, and I haven't found any typos and grammatical mistakes.", "questions": "(1) What will happen if the predicted pseudo labels become unbalanced in an iteration? What is the guarantee that the predicted pseudo labels be reasonably balanced in the update step?\n\n(2) Mutual information (MI) maximization has been widely used for representation learning, but what is the motivation of using MI for having pseudo labels and then using those pseudo labels for representation learning whereas one can directly use MI maximization for representation learning?\n", "limitations": "The authors have mentioned that the time complexity of the proposed MIRA model can be a limitation of the work, which I think is normal for self-supervised learning models. Another limitation could be the consideration of only the global information and not considering any kind of local information, which will make this method only applicable for global task, such as classification.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n(1) The paper is theoretically well grounded. The method can obtain an approximate solution of equation (3) by utilizing a fixed point iteration technique which has theoretical guarantee of convergence. In a nutshell, the proposed technique used optimal transport based pseudo labelling by maximizing the mutual information that avoids using hard artificial constraints. Altogether the idea looks very cool.\n\n(2) On different downstream tasks, the proposed model has consistently outperformed the existing methodologies. The method has been thoroughly ablated and different components of the method are well justified.\n\n**Weaknesses**\n\n(1) In this method the optimal transport based pseudo labelling by maximizing the mutual information is the main novel component, which I find is less ablated. Also justification behind a two step pipeline (1. pseudo label prediction, 2. classification using those pseudo labels) is not clear. Representation can be learnt by directly maximizing MI.\n\n(2) It is little difficult to follow the update rule in equation (5) from the unique optimal point found in equation (4). To this end, more explanation and references will be helpful to the reader.\n\n(3) The paper has shown experiments on linear evaluation, semi-supervised learning, k-nn evaluation, transfer learning etc,, which are basically image classification experiments. More diverse experiments (such as object detection, semantic segmentation etc) are missing, which is not completely validating the success of the proposed MIRA model. Note that most of the existing self-supervised learning models do consider diverse tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aavw41RbZTD", "reviewer": "Reviewer_y6Mg", "summary": "The paper proposes a method of pseudo-labeling such that the pseudo-labels have maximal mutual information with the samples. This is in contrast to SWaV where the pseudo-labels have maximum entropy. The paper proposes an iterative algorithm to recover the pseudo-labels, which is used inside a SSL method, as done in SWaV.", "questions": "* What are the MIRA numbers when using the online network for probing the representation for all epochs?\n* What are the test performances of MIRA, without re-training the linear probe, on ImageNet-v2? The author can compare with [2]\n* What is the performance of MIRA with only 3 iterations?\n* What are the practical reasons we should care about the convergence of the pseudo-labeling?\n* In Table 6, are the hyper-parameters of SwAV and MIRA *exactly* the same except for the pseudo-labeling? If not, what are the results if all of the hyper-parameters are exactly the same (including the number of iteration of the pseudo-labeling algorithm).", "limitations": "The authors have discussed generic limitations of SSL methods, but they have not discussed limitations related to their method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method proposes an interesting perspective on pseudo-labeling that was not previously discussed in the literature, as far as I know. Their methods is based on the KKT optimality conditions and the pseudo-labels are found\nby iterating such condition. I appreciated the connection, but I believe that this connection is important and should be presented in the main paper. There is a clear connection with SWaV and I believe that they can be connected through the framework of Optimal Transport. I would have appreciated a more in depth discussion comparing\nMIRA (the proposed method) with SWaV other than just saying that MIRA also minimizes the conditional entropy (while SWaV only maximizes the entropy). For example, could it be possible to define the updates of MIRA as a Sinkhorn algorithm\nas it is done in the lecture of [0]. Other than this, I don't have much issues with the method section other than I believe that the authors should explain how they obtain their necessary and sufficient condition of Equation 4 in the main paper.\n\nMy biggest issues with this paper are with the experiments. The authors claim state-of-the art for their method for several setups. But, their are several issues with this claim:\n* The authors select different hyper-parameters for the different training setups. E.g. they use different learning rates for 100/200 epochs in comparison to 400 and 800 epochs. This is not done for most of their baseline. E.g. BYOL [1] results are taken from the SimSiam paper and they used the same hyper-parameters for every setups.\n* The results are reported on the momentum encoder, while the results of the other methods are generally reported on the online network. The results should be reported on the same network or it should be clear where the results are probed for all the methods.\n* Looking at the code, it is not clear that the authors have not tuned their method on the validation/test set. This is not standard practice, see e.g. BYOL [1]\n* The above is true especially for the parameters chosen for semi-supervised learning for ImageNet-1k.\n* It is not clear what and how the linear classification hyper-parameters were chosen for ImageNet-1k.\n\nThe above concerns make it hard to be satisfied that the reported improvements are solely due to the proposed method, MIRA. Perhaps, if the authors made it clear that they used exactly the same parameters than SwAV and the the only intervention is the MIRA algorithm replacing the Sinkhorn, then it would be more satisfying.\n\nAnother concern that I have regards Figure 2. SwAV uses only 3 Sinkhorn iterations.\nHowever, from Figure 2, it seems that, at that point, SwAV is far from converging and the authors off SwAV claim that this amount of iterations is sufficient for obtaining their better numbers. Then, is convergence to the fixed point really important for downstream performance?\nIf so, it is not clear to me why we should care about such speed of convergence. And, I am also wondering how MIRA is comparing to SWaV with a much smaller number of iteration. Is 30 iterations necessary?\n\nFinally, I found the abstract and the introduction lacking clarity. For example, the phrase \"We derive a fixed-point iteration method and prove its convergence to the optimal solution.\" does not tell much\nabout the algorithm. I believe that the authors should be more explicit about the algorithm and the idea presented in the paper in the introduction so that the reader have a general idea of the paper after reading the introduction.\nIn particular, I believe that the fourth paragraph of the introduction should be clearer about the method.\n\nMinor: The learning rate is missing at L:427.\n\n[0] https://bicmr.pku.edu.cn/~wenzw/bigdata/lect-ot.pdf\n[1] https://arxiv.org/pdf/2006.07733.pdf\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O2icJxbjfBs", "reviewer": "Reviewer_E9n2", "summary": "This paper proposes a new self-supervised method for representation learning from unlabeled images. The core representation learning algorithm, called MIRA, relies on predicting pseudo labels from the data, which are chosen, actually learned, based on maximizing the mutual information between themselves and the data distribution. To make this proposed algorithm tractable, they derive a fixed-point iteration solution, for which they provide a proof for its ability to find the optimal solution. To achieve this, they prove that the proposed optimization problem for finding the optimal set of pseudo labels is a convex problem. \nThe proposed method is evaluated by assessing the quality of the learned data representations on multiple downstream tasks, obtained from several datasets. The experimental results highlight the richness of the learned semantic representations. ", "questions": "Please see weaknesses above.", "limitations": "The authors provide these limitations adequately. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1- The proposed method is novel, elegantly formulated, and encourages learning rich semantic representations with shorter training schedules compared to the baselines. \n2- The writing is clear, and the used language is understandable.\n3- The experimental evaluation is extensive, and is performed on several benchmarks and datasets. The results illustrate the merits of the method in improving the quality of the learned representations.\n4- Essential to me is that this work also provides experimental results using smaller batch sizes, showed in Table 8. This set of results, while it can be a little more comprehensive, is necessary, from my point of view. It encourages a wider adoption of self-supervised learning for budget resources.\n\nWeaknesses:\n- I believe the number of clusters K, which is the number of pseudo label assignments, is used as a fixed number (3000) across the experimental results section. It would be interesting to study the effect of this hyper-parameter on the performance on downstream tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MibC0T8e3I_", "reviewer": "Reviewer_UDR3", "summary": "The paper proposed a novel clustering-based self-supervised learning (SSL) method, MIRA. Motivated by the infomax principle, the paper constructs the SSL problem as an optimization problem that maximizes the mutual information between pseudo labels and data while taking the predictions of the model into consideration. To solve the online-clustering problem, the paper proposed a fixed-point iteration that takes only a few iterations to produce desirable pseudo-labels. Linear evaluation, KNN, and semi-supervised results on ImageNet verify the effectiveness of the method.", "questions": "1. As mentioned in the paper, MIRA converges faster than SwAV, and introduces MI regularization for better pseudo labeling. I guess MIRA will achieve better performance with 800 epochs of training. Or, will the method suffers a performance drop like TWIST from 400 epochs to 800 epochs?\n2.  In Tab.8, it seems the reported results are trained without multi-crops, will the phenomenon still be the same when multi-crops are applied?\n3. Will MIRA achieve desirable performance on downstream detection/segmentation tasks?", "limitations": "Yes. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-motivated, and the loss design is simple and neat. \n2. The paper is well-organized.\n3. The authors provide plenty of experiments to demonstrate the effectiveness of the method. The method achieves state-of-the-art on several evaluations. \n\nWeakness:\n1. The paper did not mention the performance of MIRA on downstream detection/segmentation tasks. It would be better if the method can also achieve better performance on detection benchmarks as detection is one of the main downstream tasks for computer vision. \n2. The paper did not provide the linear evaluation results with 800 epochs training to clarify the effectiveness of introducing Mutual information regularization. As Proposition 1 provides faster convergence, an 800-epochs result can further verify the improvement of pseudo-labels using MI regularized cluster assignment in Eq.3. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept