Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors consider the task of classifying data with long-tailed distributions of class samples under resource constraints. The resource constraints are addressed by using binary neural networks. The authors propose to use floating point networks pretrained on standard benchmarks and to retrain the classifier layer on the long-tailed data (here called _calibration_). The proposed objective is a linear combination of the loss of the binary network and the retrained floating point network, and the floating point and binary encoders. The weight of this linear combination is learned in an adversarial way, such that the part of the objective with a higher loss gets a higher weight. In addition, a mult-resolution module is added. The experiments indicate high accuracy of the proposed method in comparison to competitors on various datasets for which a long-tail class distribution has been created. 
strength_and_weaknesses: # Strengths
* I like that the weight of the linear combination of losses is optimized
* Figures and plots are used to illustrate various points
* Dataset statistics are given in the appendix
* The considered task of long-tail class distributions under resource constraints is interesting and possibly impactful for many applications
# Weaknesses
* The paper is not clear. Many terms are never properly defined, there is a lack of mathematical definitions, big chunks of the method are a black-box to me, I don't understand most Figures and the reasoning is also often unmotivated. 
* The writing is often repetitive in what the authors are going to do without actually explaining it, which makes for a frustrating read
* The related work just lists related papers without actually pointing out how the existing work builds the fundament for benchmarks.
clarity,_quality,_novelty_and_reproducibility: The clarity and reproducibility of the paper are very low and it's the main weakness.
I am missing proper definitions of major relevant terms, such as
* long-tailed data distribution
* classifier weight norms 
* imbalance ratio
* variance at classes
* multi-resolution input
* encoders
* $Atten_\phi$

Further, the issue of optimization is largely glossed over. The training of binary networks can be done in multiple ways and is non-trivial. Here, it's unclear how the binary weights are trained. Further, the authors propose a min-max optimization, which typically also requires some finesse. I read somewhere that Adam is used but I doubt that you can just through an optimizer on this objective and get decent results. How is the step size set? Some choices for the objective are unmotivated, for example, the choice for the cosine distance loss for the encoders. How are hyperparameters tuned? I didn't understand at all how the multi-resolution input is integrated and how this is handled in the training.

I am not familiar enough with the field that I could assess the novelty. Quality is also quite unclear due to the clarity issue.
summary_of_the_review: Possibly a good paper that is not acceptable at this point in my view, because of its unclarity.

# After Rebuttal Thoughts
I see that the authors revised the paper carefully and that they provided more clarity regarding the terminology and methodology. However, in order to assess the revised version, I would have to re-read the whole paper. Unfortunately, I don't have time to do so, hence, I'll decerease my confidence score and give other reviewers the floor.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a binary network for long tailed data. They transfer knowledge from a calibrated teacher network to the binary network, and utilize adversarial learning for learning the hyperparameters. Experimental results demonstrate the effectiveness of their method.
strength_and_weaknesses: Strength:
The authors proposed a bianry network for long tailed data, which is interesting. The presentation is good, and easy to follow. 

Weaknesses:
1. The authors claimed high variance in binary network can influence the performance of the model, and attempted to lower the high variance. Is there theoretical analysis on this? This seems to be claimed only based on some empirical observation. 

2. The proposed method utilized the traditional knowledge distillation for learning a binary network, leading to the limited novelty.

3. In Figure 2, the authors claimed "The classifier weight norms increase at the tail classes, implying high variance". why?

4. Why learn the balance bwtween two losses can be realized by an adversarial learning manner? It is expected to give more explanations.
clarity,_quality,_novelty_and_reproducibility: The authors are expected to give more clarity.
summary_of_the_review: I incline to reject this paper based on the current issues. However, I will be happy to change my score if the authors can solve my questions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents CANDLE, a binary-based network trained using a teacher-student framework in which a learned weight selects the largest loss between two loss terms in the optimization problem. In this way, CANDLE ensures the minimization of the maximum loss at every iteration. The motivation of using a binary network is to ensure a simple but efficient network that can be used in long-tail scenarios with efficiency in mind. Also, the paper adds a multi-resolution training schema in order to boost performance. The paper presents experiments on synthetically long-tailed and natural long-tail dataset demonstrating that the average accuracy improves considerably.
strength_and_weaknesses: Strengths:
1. The motivation of the paper has a practical impact. Tackling long-tail scenarios is an important problem in real applications, and especially, considering the computational efficiency aspect is an important problem.
2. The idea of introducing a learned weight (\lambda) that controls the “focus” between the two loss-terms in order to guarantee that the maximum loss at every iteration is minimized is interesting. I think the learning formulation in the teacher-student framework is a new idea.
3. Overall, the clarity of the intuition and high-level ideas are explained well. However, there are some concerns I have with details and motivation; see the section below.

Weaknesses:
1. Assumptions are not well justified, and the paper is not self-contained: 
- The main assumption from the paper is that a neural-network-based image classifier suffers from performance issues due to the weight-norm variances. This already assumes that the networks use a linear-based classifier. However, previous work has shown that cosine-based classifiers also improve the performance of image classifiers in the long-tail scenario; see references A and B. In theory, these cosine-based classifiers should be more robust to the weight-norm variance because only angles are the main discriminative statistic.
- The paper is not self-contained because it lacks a full discussion about the bad impact the weight-norm variance has when learning from long-tailed datasets. The reader has to dig in deeper and read extra to understand this claim. 
- Moreover, the narrative in the introduction projects to the reader that the main cause of the bad performance when learning from long-tailed datasets is the weight-norm variance. However, as shown in previous work (see references A and B for example), the lack of data from tail classes is the main cause of the problem. Thus, over sampling methods also alleviate the bad performance effect when learning from a long-tailed dataset. I think the introduction needs a revision in which it acknowledges previous work in a more structured and inclusive manner of previous work.

2. The paper lacks clarity:
- It is unclear from the introduction and Section 3.2 how the teacher network is pre-trained on non-LT data. What does this mean? Does it mean the teacher network used a balanced dataset (e.g., CIFAR 100), and then the teacher transfers its knowledge to the student network? I could not find a clear explanation about this.
- In section 3.1, the paper states that it is believed that the “model is likely to exhibit high [weight-norm] variance as it fits to the few training samples in the tail classes”. I think this statement needs to be backed up with an experiment.

3. Insufficient experiments: In the long-tail context, the mean average accuracy is not very informative. This is because it is unclear if the head classes are improving and thus moving the average accuracy metric up. I think a plot showing accuracy performance per class can be more informative (see Fig. 4 in A). It is unclear if the method favors more head or medium-shot classes. Note that this also applies to the ablation study.

4. While CANDLE can improve efficiency during the inference stage due to the use of binary-based networks, the training of these networks is quite elaborate. In my (humble) opinion, this decreases the practical benefit as it requires training a teacher network first, then training a student network using CANDLE again.

References:

A. Kozerawski, et al.. BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed Images. ACCV 2020.

B. Park, et al.. The Majority Can Help the Minority: Context-Rich Minority Oversampling for Long-Tailed Classification. CVPR 2022



================= Post-Discussion =================

After engaging in a discussion with the authors, I will downgrade my recommendation to 3 - Reject, not good enough. The reason for this is the following: the proposed approach is not practical. To me, the proposed approach has a fundamental problem. The proposed approach assumes that a teacher model (i.e., a non-LT model) exists and can be used to train a student model that will effectively deal w/ the long-tail dataset. In a realistic scenario and as an example, if I want to train a classifier using CANDLE and the Google Open Images dataset, which is a long-tailed one, it would be very hard to get a non-LT teacher model. Therefore, I cannot use CANDLE in large scale and real long-tailed dataset. Therefore, I question the approach as a viable solution to the long-tailed problem. Unfortunately, I could not get a satisfactory answer to this concern.
clarity,_quality,_novelty_and_reproducibility: 
I think the clarity can be improved, and consequently this can improve the reproducibility of the paper. However, I think the idea of using an adversarial learning framework is interesting. 
summary_of_the_review: Overall, I think the paper lacks clarity, better justifications of their algorithmic design choices, and more informative experiments showing the performance on head, medium-shot, and tail classes. However, I am intrigued about the adversarial learning framework applied to the teacher and student paradigm.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies long-tailed learning for binary networks. This is a novel setting. To address this, this paper proposes a new CANDLE method, which is based on three main strategies: (1) Calibrate and Distill, (2) adversarial learned balancing, (3) multi-resolution training. Promising empirical results demonstrate its effectiveness.
strength_and_weaknesses: Strengths:
1. Learning binary networks on long-tailed data is a novel and practical task. Overall, I like this idea.
2. The analysis is interesting, which shows the differences between learning binary networks and learning float-point networks on long-tailed data.
3. Empirical results are encouraging. 

Weaknesses:
1. Figure 2 finds that the classifier weight norms increase at the tail classes on both float-point networks and binary networks. However, this is contrary to several classic methods, like decoupling (Kang, ICLR 2020). They find classifier weight norms of float-point networks are positively related to long-tailed class distribution with standard training.  The authors explain that conventional FP methods use SGD and weight decay but binary networks use ADAM with zero weight decay. It makes sense, but it will be better to provide a baseline that binary networks use SGD and weight decay to demonstrate this.
2. Moreover, it is interesting to know the size of feature space per class. In conventional FP networks, head classes usually have large feature space than tail classes. How about binary networks? 
3. Furthermore, many long-tailed approaches use the cosine classifier since its classifier weights are normalized. It is good to show how the proposed method performs with the cosine classifier.
4. Why do you use MiSLAS to calibrate the classifier? Efficiency seems not a superiority of MiSLAS.

 
clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is easy to understand.
Quality: the quality is high but there are several questions that can be resolved.
Novelty: the novelty is high.
Reproducibility: authors are expected to release the source code to the public.

summary_of_the_review: Despite a few questions that can be resolved, I like this paper as it explores a new and practical task, i.e., learning binary networks on long-tailed datasets, and provides a few insights. I tend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ### Description

The goal of the paper is to train a binary neural network to the best accuracy for a given imbalanced (long-tailed) dataset. The test distribution is assumed to be the same. For the application of the method this is the only dataset considered, so I will refer to it as just the dataset.

The method of the paper is the following pipeline:
1) Start with a pretrained full precision network, throw away the classifier part.
2) Refit the classifier part of the network to the dataset.
3) Train the binary network with a variant of distillation to mimic the full precision teacher in 2) in logits and features at the last layer.

The novelty of the paper, besides the pipeline itself, is around several points:
* experimental diagnostics of long-tail performance issues through the perspective of weight norms
* training procedure (adversarially learned balance)
* efficient multi-resolution distillation

### Decision

Reviewers have positively evaluated that the problem addressed is important and practically relevant, that the method outperforms baselines by a large margin, that the training procedure is interesting. However there were several important points not fully clarified by the rebuttal and further issues (all detailed below). We carefully discussed these issues and recommend that the paper is revised and resubmitted.

### Clarity / Soundness issues

1) The paper argues a lot about the high variance (e.g. Section 3.1 HIGH VARIANCE FOR BINARY NETWORKS IN LONG -TAILED DISTRIBUTION). The notion of variance used in different places seems to be different and is never made clear (see meeting summary). 
It would be appropriate to define exactly variance of what and with respect to which source of randomness. The implications such as "large weight magnitude" implies "high variance" should be substantiated in the paper. The paper should elaborate (or review the prior work) on why large weights are indicative of overfitting and poor generalization for rare classes. Additionally, in contrast to the submission, plots in Alshammari et al., (2022) show a reverse dependence: common classes have large weight magnitude than rare ones, adding to the confusion.

2) The importance of having the pretrained teacher on a non-LT data is not clear. Authors argue that it is a practical setup and such networks are readily available. The reviewer ah6w argues that this is not practical since in the large-scale setup we rather have LT datasets collected. In the meeting we converged rather to the interpretation that the assumption of non-LT pretrained model is not important. It only provides useful embeddings and as the classifier part of the network is discarded, it does not retain specific information about the rare classes. In that case the paper should support this by evidence and not emphasize the pretraining on a **balanced** dataset. Any pretrained model, e.g. unsupervised, semi-supervised, self-supervised, should be good. It would be expected to bring gains to the training of binary neural networks on any data, not necessarily LT data.
See more details in the "Summary Of AC-reviewer Meeting". Please contact PCs if you don't get to see these comments.

3) Another major concern is the justification of the adversarially learned balance procedure. The optimally trained $\phi$ should be equivalent to 
$\max_{\lambda \in [0,1]} (\lambda L_{KL} + (1-\lambda) L_{FS}) = \max(L_{KL}, L_{FS})$
So from the viewpoint of the design of the objective, this is just equivalent to considering $\max(L_{KL}, L_{FS})$ as the learning objective. The latter is not sound, because $L_{KL}$ and $L_{FS}$ are different kinds of losses on different number of units, and need a weight coefficient to make the comparison meaningful. The procedure is therefore does not appear rational. Its effect is some indirect schedule of changing $\lambda$ due to learning dynamics. The fact that eventually $\lambda$ converges to $1$ means that towards the end only the encoder is improved but not the classifier.  This schedule is controlled not by a hyperparameter but by e.g. the design of the Attention network and the optimizer. This is very obscure from the point of view of a rational design. If, nevertheless, it is a good heuristic in practice for some reason, more experimental evidence is needed to evaluate, explain and justify it. 
justification_for_why_not_higher_score: lack of clarity and justification on several points
Decision: Reject