Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Mutual Information Regularized Assignment (MIRA) which is a pseudo-labelling algorithm for unsupervised representation learning inspired by information maximization. The pseudo-labelling procedure is formulated as an optimization problem solved via a fixed-point iteration method that maximize the mutual information between the label and data while being close to a given model probability. The proposed methodology has been claimed to be converging relatively faster and to be achieving state-of-the-art performance on various downstream tasks such as linear evaluation and transfer learning. In general, the paper is well written and it is easily understandable, and I haven't found any typos and grammatical mistakes.
strengths_and_weaknesses: **Strengths**

(1) The paper is theoretically well grounded. The method can obtain an approximate solution of equation (3) by utilizing a fixed point iteration technique which has theoretical guarantee of convergence. In a nutshell, the proposed technique used optimal transport based pseudo labelling by maximizing the mutual information that avoids using hard artificial constraints. Altogether the idea looks very cool.

(2) On different downstream tasks, the proposed model has consistently outperformed the existing methodologies. The method has been thoroughly ablated and different components of the method are well justified.

**Weaknesses**

(1) In this method the optimal transport based pseudo labelling by maximizing the mutual information is the main novel component, which I find is less ablated. Also justification behind a two step pipeline (1. pseudo label prediction, 2. classification using those pseudo labels) is not clear. Representation can be learnt by directly maximizing MI.

(2) It is little difficult to follow the update rule in equation (5) from the unique optimal point found in equation (4). To this end, more explanation and references will be helpful to the reader.

(3) The paper has shown experiments on linear evaluation, semi-supervised learning, k-nn evaluation, transfer learning etc,, which are basically image classification experiments. More diverse experiments (such as object detection, semantic segmentation etc) are missing, which is not completely validating the success of the proposed MIRA model. Note that most of the existing self-supervised learning models do consider diverse tasks.
questions: (1) What will happen if the predicted pseudo labels become unbalanced in an iteration? What is the guarantee that the predicted pseudo labels be reasonably balanced in the update step?

(2) Mutual information (MI) maximization has been widely used for representation learning, but what is the motivation of using MI for having pseudo labels and then using those pseudo labels for representation learning whereas one can directly use MI maximization for representation learning?

limitations: The authors have mentioned that the time complexity of the proposed MIRA model can be a limitation of the work, which I think is normal for self-supervised learning models. Another limitation could be the consideration of only the global information and not considering any kind of local information, which will make this method only applicable for global task, such as classification.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed a novel clustering-based self-supervised learning (SSL) method, MIRA. Motivated by the infomax principle, the paper constructs the SSL problem as an optimization problem that maximizes the mutual information between pseudo labels and data while taking the predictions of the model into consideration. To solve the online-clustering problem, the paper proposed a fixed-point iteration that takes only a few iterations to produce desirable pseudo-labels. Linear evaluation, KNN, and semi-supervised results on ImageNet verify the effectiveness of the method.
strengths_and_weaknesses: Strengths:
1. The paper is well-motivated, and the loss design is simple and neat. 
2. The paper is well-organized.
3. The authors provide plenty of experiments to demonstrate the effectiveness of the method. The method achieves state-of-the-art on several evaluations. 

Weakness:
1. The paper did not mention the performance of MIRA on downstream detection/segmentation tasks. It would be better if the method can also achieve better performance on detection benchmarks as detection is one of the main downstream tasks for computer vision. 
2. The paper did not provide the linear evaluation results with 800 epochs training to clarify the effectiveness of introducing Mutual information regularization. As Proposition 1 provides faster convergence, an 800-epochs result can further verify the improvement of pseudo-labels using MI regularized cluster assignment in Eq.3. 
 
questions: 1. As mentioned in the paper, MIRA converges faster than SwAV, and introduces MI regularization for better pseudo labeling. I guess MIRA will achieve better performance with 800 epochs of training. Or, will the method suffers a performance drop like TWIST from 400 epochs to 800 epochs?
2.  In Tab.8, it seems the reported results are trained without multi-crops, will the phenomenon still be the same when multi-crops are applied?
3. Will MIRA achieve desirable performance on downstream detection/segmentation tasks?
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new self-supervised method for representation learning from unlabeled images. The core representation learning algorithm, called MIRA, relies on predicting pseudo labels from the data, which are chosen, actually learned, based on maximizing the mutual information between themselves and the data distribution. To make this proposed algorithm tractable, they derive a fixed-point iteration solution, for which they provide a proof for its ability to find the optimal solution. To achieve this, they prove that the proposed optimization problem for finding the optimal set of pseudo labels is a convex problem. 
The proposed method is evaluated by assessing the quality of the learned data representations on multiple downstream tasks, obtained from several datasets. The experimental results highlight the richness of the learned semantic representations. 
strengths_and_weaknesses: Strengths:
1- The proposed method is novel, elegantly formulated, and encourages learning rich semantic representations with shorter training schedules compared to the baselines. 
2- The writing is clear, and the used language is understandable.
3- The experimental evaluation is extensive, and is performed on several benchmarks and datasets. The results illustrate the merits of the method in improving the quality of the learned representations.
4- Essential to me is that this work also provides experimental results using smaller batch sizes, showed in Table 8. This set of results, while it can be a little more comprehensive, is necessary, from my point of view. It encourages a wider adoption of self-supervised learning for budget resources.

Weaknesses:
- I believe the number of clusters K, which is the number of pseudo label assignments, is used as a fixed number (3000) across the experimental results section. It would be interesting to study the effect of this hyper-parameter on the performance on downstream tasks. 
questions: Please see weaknesses above.
limitations: The authors provide these limitations adequately.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method of pseudo-labeling such that the pseudo-labels have maximal mutual information with the samples. This is in contrast to SWaV where the pseudo-labels have maximum entropy. The paper proposes an iterative algorithm to recover the pseudo-labels, which is used inside a SSL method, as done in SWaV.
strengths_and_weaknesses: The method proposes an interesting perspective on pseudo-labeling that was not previously discussed in the literature, as far as I know. Their methods is based on the KKT optimality conditions and the pseudo-labels are found
by iterating such condition. I appreciated the connection, but I believe that this connection is important and should be presented in the main paper. There is a clear connection with SWaV and I believe that they can be connected through the framework of Optimal Transport. I would have appreciated a more in depth discussion comparing
MIRA (the proposed method) with SWaV other than just saying that MIRA also minimizes the conditional entropy (while SWaV only maximizes the entropy). For example, could it be possible to define the updates of MIRA as a Sinkhorn algorithm
as it is done in the lecture of [0]. Other than this, I don't have much issues with the method section other than I believe that the authors should explain how they obtain their necessary and sufficient condition of Equation 4 in the main paper.

My biggest issues with this paper are with the experiments. The authors claim state-of-the art for their method for several setups. But, their are several issues with this claim:
* The authors select different hyper-parameters for the different training setups. E.g. they use different learning rates for 100/200 epochs in comparison to 400 and 800 epochs. This is not done for most of their baseline. E.g. BYOL [1] results are taken from the SimSiam paper and they used the same hyper-parameters for every setups.
* The results are reported on the momentum encoder, while the results of the other methods are generally reported on the online network. The results should be reported on the same network or it should be clear where the results are probed for all the methods.
* Looking at the code, it is not clear that the authors have not tuned their method on the validation/test set. This is not standard practice, see e.g. BYOL [1]
* The above is true especially for the parameters chosen for semi-supervised learning for ImageNet-1k.
* It is not clear what and how the linear classification hyper-parameters were chosen for ImageNet-1k.

The above concerns make it hard to be satisfied that the reported improvements are solely due to the proposed method, MIRA. Perhaps, if the authors made it clear that they used exactly the same parameters than SwAV and the the only intervention is the MIRA algorithm replacing the Sinkhorn, then it would be more satisfying.

Another concern that I have regards Figure 2. SwAV uses only 3 Sinkhorn iterations.
However, from Figure 2, it seems that, at that point, SwAV is far from converging and the authors off SwAV claim that this amount of iterations is sufficient for obtaining their better numbers. Then, is convergence to the fixed point really important for downstream performance?
If so, it is not clear to me why we should care about such speed of convergence. And, I am also wondering how MIRA is comparing to SWaV with a much smaller number of iteration. Is 30 iterations necessary?

Finally, I found the abstract and the introduction lacking clarity. For example, the phrase "We derive a fixed-point iteration method and prove its convergence to the optimal solution." does not tell much
about the algorithm. I believe that the authors should be more explicit about the algorithm and the idea presented in the paper in the introduction so that the reader have a general idea of the paper after reading the introduction.
In particular, I believe that the fourth paragraph of the introduction should be clearer about the method.

Minor: The learning rate is missing at L:427.

[0] https://bicmr.pku.edu.cn/~wenzw/bigdata/lect-ot.pdf
[1] https://arxiv.org/pdf/2006.07733.pdf

questions: * What are the MIRA numbers when using the online network for probing the representation for all epochs?
* What are the test performances of MIRA, without re-training the linear probe, on ImageNet-v2? The author can compare with [2]
* What is the performance of MIRA with only 3 iterations?
* What are the practical reasons we should care about the convergence of the pseudo-labeling?
* In Table 6, are the hyper-parameters of SwAV and MIRA *exactly* the same except for the pseudo-labeling? If not, what are the results if all of the hyper-parameters are exactly the same (including the number of iteration of the pseudo-labeling algorithm).
limitations: The authors have discussed generic limitations of SSL methods, but they have not discussed limitations related to their method.
review_title: null

Meta-review: metareview: This paper proposes a pseudo-labelling algorithm for unsupervised representation learning inspired by information maximization. 

The reviewers found  that the proposed method is theoretically well grounded and that the authors provide extensive experimentations to demonstrate the validity of their approach. I agree with those conclusion after reading the paper. I therefore support acceptance.
confidence: Certain
Decision: Accept