Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of self-supervised feature learning for downstream tasks where the unsupervised dataset follows a long-tail distribution.
In particular, it seeks to address the issue of head classes covering much more of the feature space than the tail classes (expansion/collapse).
The proposed method is to obtain initial surrogate class likelihoods for each example, transform these by applying reverse post-hoc logit adjustment based on the surrogate class frequencies, and then obtain final (soft) surrogate labels by solving an optimal transport problem with the Sinkhorn-Knopp algorithm.
The initial surrogate class likelihoods are obtained using a Simplex ETF (linear, constant, orthogonal) classifier and the final confidence of the assigned surrogate label is used to weight an additional loss term that encourages the Simplex ETF to be more confident.
The marginal likelihoods of the surrogate labels are recomputed every epoch, and the logit adjustment and optimal transport are computed in every minibatch.
Empirical studies show that this technique generally improves the accuracy of several baseline algorithms (SimCLR, Focal, SDCLR, BCL) across the Many, Med and Few class subsets for several datasets with a wide range of imbalance (CIFAR100-LT-10, -50, -100, ImageNet-LT, Places-LT).
It is applied after an initial warmup phase that uses the baseline algorithm alone.
strength_and_weaknesses: **Strengths**

1. Dealing with long-tail data in unsupervised pre-training seems like an important and difficult problem which has not received much attention. Indeed, it's hard to even measure the tail since the dataset does not have labels.
1. Including focal loss as a baseline was good as it may help with class imbalance.
1. The proposed method is shown to significantly complement a wide range of algorithms for a wide range of datasets.
1. The method does not incur a high computational cost.

**Weaknesses**

1. The motivation and description of the Simplex ETF classifier, the transformation of its predictions to obtain surrogate labels and the additional loss term is extremely unclear. Similarly, the motivation for applying logit adjustment in reverse is unclear. If I understood correctly, the idea is to make the predictions of head classes even more confident, such that they are moved further from their (fixed) decision boundary. It might be possible to achieve this more elegantly using the logit-adjusted softmax from that same paper (rather than post-hoc logit adjustment)? Also, could the single $q \log p$ term in the loss be replaced with the more familiar cross-entropy over all labels $t$?
1. It wasn't clear what the "trivial unconstrained probabilities" and "trivial solution" to eq. 4 were. One-hot distributions? This was used to motivate the use of Optimal Transport. Can't we just use the (normalized) logit-adjusted likelihoods $p(\tilde{t})$ as targets? Does this cause the method to fail?
1. I found it surprising that the proposed approach achieved significant gains for the "Many" subset as well as the "Few" subset. This was not explained. Perhaps the method is achieving its improvement by some mechanism other than accounting for imbalance? Perhaps it's accounting for imbalance in some unlabelled attribute? This might be revealed by inspecting the clusters. Is it possible to run some experiments in the balanced setting and see if the method still yields gains?
1. No empirical comparison to the method of Asano et al. (SeLa), which does consider the imbalanced setting.
1. The values for $\tau$ seemed much lower than in the logit adjustment paper (0.02-0.07 vs. 1-4). How can this be explained?
1. While the method is claimed to be end-to-end, it involves non-differentiable computations in each epoch and in each mini-batch.
1. Does 300 iterations of Sinkhorn-Knopp really not have a significant impact on the training time? How long does this take compared to the typical time for a minibatch?
1. The number of training epochs seemed very high (e.g. for CIFAR-100-LT, 1000 epochs plus 500 warmup epochs). Is this typical for self-supervised training?
1. Figure 4(a), which shows a comparison to temperature scaling, would benefit from error-bars, since the values seem noisy and S2LA is not that far from the baseline.
1. Related work section does not include discussion of Supervised Long-tail Learning. For example, the paper "Decoupling Representation and Classifier for Long-Tailed Recognition" seems relevant.
1. Why was 100-shot evaluation used for ImageNet-LT and Places-LT? Wouldn't this impair the accuracy of the "Many" classes compared to a linear classifier trained with all examples?
1. While Algorithm 1 in the appendix makes the method much more clear, it shouldn't be necessary to refer to the appendix to understand the method. For example, it wasn't clear in the main text that the surrogate class prior was re-estimated once per epoch.

**Nitpicks**

1. Self-supervised learning includes some methods that have labels (i.e. automatically generated labels). The paper often uses "self-supervised learning" to refer to *contrastive* methods exclusively. It would be good to be more clear.
1. The acronyms BCL and SDCLR were not defined.
1. Needs to be proofread for grammar.
1. I was confused by the use of bold characters for the single labels $\boldsymbol{t}$ and $\tilde{\boldsymbol{t}}$. This is typically used to denote a vector.
clarity,_quality,_novelty_and_reproducibility: **Clarity and quality.** The description and motivation of the method is unclear, see concerns above. The empirical evaluation seems to be of a high standard. However, it would be good to perform multiple trials and include confidence intervals for the most important results.

**Novelty.** The method seems novel. Note that the use of Sinkhorn-Knopp for self-supervised learning via self-labelling was introduced in Asano et al. (2020). While this was not claimed as a contribution, I feel that this should be acknowledged in the paper.

**Reproducibility.** I believe there is sufficient information to reproduce the experiments.
summary_of_the_review: While the results are strong, I find that the description and motivation of the method to be very unclear. I am leaning negative but I'll consider upgrading my review if these concerns can be mitigated in the process of the rebuttal and discussions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses an important problem of self-supervised learning on data with imbalanced classes. SSL trained on sample-level uniformity will learn representation for common classes. However, less represented classes will suffer due to bias for "head" classes. The paper introduces a method to achieve category-level uniformity by using surrogate classes. A distinct property of the method is that it can be applied to any SSL method by modifying the loss function along with the training procedure. Authors demonstrate strong empirical results of the method.
strength_and_weaknesses: Strength:
1. An important problem of SSL on imbalanced classes.
2. An original idea of using surrogate classes to prevent bias due "head" classes.
3. Substantial empirical results.
4. Authors share code for reproducibility.

Weaknesses:
1. Mathematical exposition of the material is poor.
2. Ideas are not communicated clearly. 
3. There is no discussion on the technique's limitations.
clarity,_quality,_novelty_and_reproducibility: Communication of the ideas can be improved. I don't think I can reproduce results from the paper itself, however authors release the code which somewhat mitigates the issue.
summary_of_the_review: After addressing the feedback, communication of ideas somewhat improved. I am giving a conditional acceptance, if authors can add a discussion on the limitations of the approach. Such a discussion will bring forth implicit assumption where the method expected to work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Given the research progress on logit adjustment for supervised long-tailed learning, this paper presents a new method S^2LA to conduct self-supervised long-tailed learning  from the geometric perspective. In fact, without knowing the class distribution, S^2LA uses a constant simplex ETF to measure the geometric characteristics of the embedding space. Using a surrogate label allocation to represent the target distribution, this paper intends to compress the space of the head classes whilst avoiding the collapse of the tail classes. Treating the above transformation as a balancing and optimal transport problem, the proposed solution is intended to effectively handle the category-level uniformity problem. 
strength_and_weaknesses: Strengths - 

1. The presentation of the proposed method is generally understandable. The discussion on the theoretical analysis and the experimental is clear. 
2. The balance between the proposed approach and the evaluation section look reasonable. Many results have been provided to support the arguments in the theoretical analysis.
3. The mathematical principles and basics have been clearly defined and explained.  

Weaknesses - 

1. The major concern is that the proposed approach seems incremental, compared to the established works and similar technologies. 
2. The evaluations seem to support the arguments that the proposed technology outperforms the other state of the art methods, however, the justification is not comprehensive and requires much additional evidence to support the overall statements.
3. The experimental results show that the proposed algorithm can only achieve slightly better performance (often < 1%) than the other state of the art technologies.

Actions to be taken:

1. Since there is no ground-truth annotation of each sample, and using geometric labels tends to generate trivial unconstrained probabilities, a surrogate label allocation with logit adjustment is used in this paper. Eq. (4) is proposed and then needs to be solved using Eqs. (5)-(6). Finally, the overall objective function is expressed as Eq. (7). However, Eq. (7) has a from similar to that of Zhou et al. (2022). The only difference between them is that the current paper introduces Eq. (4) to its final objective function, which plays as a constraint term. It is not clear if or not this objective function is correctly derived in this paper. If it is, how much difference between the two papers indeed?
2. In the ablation experiments, Fig. 4(c) is not of much value. This is because presenting different distributions without showing the classification results is not convincing.
3. In Fig. 5, increasing skewness also leads to higher accuracy. However, the improvement of accuracy is fairly mild, normally less than 1%. If this is a successful mechanism, the increment should be significant, is not it? 
clarity,_quality,_novelty_and_reproducibility: Clarity: The value and importance of the work seems clear in the current form. The presentation of the proposed method seems understandable. The explanations to the experimental results are sensible. However, the derivation of the objective function in the proposed algorithm needs further interrogation so that the reader understands the technical details. 

Quality: The idea and concept of the proposed algorithm look quite plain. It seems that the proposed approach is a straightforward extension of similar approaches. The proposed research seems acceptable in terms of the motivation but lacks significance in theory and experiments. 

Originality: Logit adjustment is a reasonable choice and the current paper uses a number of mathematical means to deal with the objective functions. The proposed framework, based on several works with similar drives, has made incremental contribution on solving the SSL problems with the SK algorithm. The originality of the proposed method is very minor.

Reproducibility: There is no indication that the source code and other corresponding materials will be published. 
summary_of_the_review: In summary, this paper presents an interesting topic, worth of being investigated. The theoretical analysis sounds reasonable but lacks certain explanations to the proposed objective function and its derivation. The experimental results are quite considerate but some of the systematic parameters have not been properly justified in the examination.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to solve the self-supervised imbalanced learning by enforcing category-level uniformity instead of sample-level uniformity.
strength_and_weaknesses: Strength:

- The paper presents a strong motivation, which is to achieve class-level feature uniformity instead of instance-level feature uniformity. The overall method is closely built around this motivation. The toy example in Figure 1. is a good illustration of the paper's idea. The idea is new to the field.

- The empirical results are sufficiently strong. Improvements are observed on multiple datasets (CIFAR-LT, ImageNet-LT, and Places-LT) on top of several SSL methods.


Weakness:

- I find the method part convoluted:

  - I have difficulty understanding the importance of Simplex ETF. To my understanding, Simplex ETF is a fixed linear classifier that outperforms the standard learnable linear classifiers in end-to-end __supervised__ imbalanced learning. However, why is it necessary in the unsupervised setting? Should other alternatives to "measure the geometric characteristics", e.g., a fixed random linear classifier, be discussed and compared?

  - The motivation for logit adjustment is not very clear. Why is logit adjustment required on top of Simplex ETF? "All vectors in a Simplex ETF have the same pair-wise angle", is Simplex EFT alone sufficient to ensure the class-level uniformity already?

  -  Why is Eqn. (4) the objective? Is it to find some surrogate labels $q(t|x)$ to compute the cross-entropy loss with $p(t|x)$, so that the marginal $p(t)$ can be adjusted to $p(\tilde{t})$? This part may need more clarity.

  - The notions are not clear sometimes. Should the expression $p(\tilde{t}|x) \propto p(t|x)p(t)^{1+\tau}$ be $p(\tilde{t}|x) \propto p(\tilde{t}|x)p(\tilde{t})^{1+\tau}$? $q$ is also not defined in Eqn. (4) until later. 

  - __Overall__, I believe that the mixing of statistical interpretations (logit adjustment, Bayes) and geometric interpretations (Simplex ETF, geometric span) is what makes the method part hard to understand. Currently, I don't see how the two interpretations can be linked, i.e., the relation between "skewed geometric distribution" and the "skewed statistical distribution". If the authors are just trying to "shrink the feature span of head classes for the category-level uniformity", I would suggest the authors stick to the geometric interpretations and view "logit adjustment" as a margin loss.

- I may have missed the part about how focal loss is used for self-supervised learning. It would be better if more related works besides Lin et al. 2017  can be provided.





clarity,_quality,_novelty_and_reproducibility: - Clarity: can be improved in the method part.
- Quality: good.
- Originality: good.
- Reproducibility: good, code is promised to be provided.
summary_of_the_review: The idea of the paper is new and comes with a clear motivation, and the experiment results seem sufficiently strong. However, I find the method part of the paper very difficult to understand which heavily affects my assessment of its correctness. I believe that the method part can be clearer with more polishing efforts, but based on the current version, I would recommend a weak rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper tackles the problem of unbalanced classes in self-supervised learning. Authors posits that imbalanced classes (long-tailed data) can lead to the model producing higher quality embedding for the head samples (of high frequency classes) and lower quality embedding for tail samples (of low frequency classes). This problem has been studied recently in prior work. Authors propose a new training strategy which aims to balance out (equally) the spread of various underlying classes on the embedding space. It does this by creating surrogate labels for the unlabeled data, measuring the imbalance and then correcting it through an adjustment of the surrogate logits. 

The method uses some interesting tools and is quite flexible that it can work on top (in addition to) of prior method. Authors provide experiments which shows that their method improves the downstream-task performance on top of other baseline method. Reviewer requested for additional experiments and clarification and authors tried to address many of these concerns. Unfortunately, it is notable that none of numerical results in the table provide mean and/or standard errors over multiple runs, so there is still some uncertainty about the level of improvement. All reviewers had mentioned that the proposed method is hard to understand (Sec 4) due to jargon/imprecise wording, and lack of rigorous mathematical derivation. Currently it takes considerable effort to understand even the basic details of the method. This concern still remained even after the revisions. Finally, a reviewer also noted no limitations of the method was discussed which was concerning. This lead to a discussion between authors and reviewer which brought up an interesting failure case when there could be multiple underlying class taxonomies, and authors have promised add them to the main text.
justification_for_why_not_higher_score: Lack of clarity of method. Proper discussion and analysis of limitations and assumptions.
Decision: Reject