Learning Binary Networks on Long-Tailed Distributions
Authors: Jihun Kim
Dahyun Kim
Hyung Rok Jung
TaeIL Oh
Jonghyun Choi
Venue: Submitted to ICLR 2023 (2023)
Keywords: binary neural network
long-tailed recognition
distillation

Abstract: In deploying deep models to real world scenarios, there are a number of issues including computational resource constraints and long-tailed data distributions. For the first time in the literature, we address the combined challenge of learning long-tailed distributions under the extreme resource constraints of using binary networks as backbones. Specifically, we propose a framework of calibrating off-the-shelf pretrained full precision weights that are learned on $\textit{non-long-tailed}$ distributions when training binary networks on long-tailed datasets. In the framework, we additionally propose a novel adversarial balancing and a multi-resolution learning method for better generalization to diverse semantic domains and input resolutions. We conduct extensive empirical evaluations on 15 datasets including newly derived long-tailed datasets from existing balanced datasets, which is the largest benchmark in the literature. Our empirical studies show that our proposed method outperforms prior arts by large margins, $\textit{e.g.}$, at least $+14.33\%$ on average.

Under review as a conference paper at ICLR 2023
LEARNING BINARY NETWORKS ON LONG-TAILED DISTRIBUTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In deploying deep models real world scenarios, there are number of issues including computational resource constraints and long-tailed data distributions. For the first time in the literature, we address the combined challenge of learn- ing long-tailed distributions under the extreme resource constraints of using bi- nary networks backbones. Specifically, we propose a framework of calibrating aff-the-sheld pretrained full precision weights that are learned on non-long long--aaaall distributions when training binary networks on long-tailed datasets. In the frame- work, we additionally propose a novel adversarial balancing and multi-resolution learning method for better generalization to diverse semantic domains and input resolutions. We conduct extensive empirical evaluations on 15 datasets including newly derived long-tailed datasets from existing balanced datasets, which is the largest benchmark in the literature. Our empirical studies show that our proposed method outperforms prior arts by large margins, e.g., at least +14.33% on average.
INTRODUCTION
In recent years, there grows emphasis on resource constraints, especially for edge devices, in learn- ing deep models, resulting in breakthroughs such as MobileNet (Howard et al., 2017) and YOLO- V7 (Wang et al., 2022) that concern not only accuracy but also computational costs. This has at- tracted attention for deep learning both in the research and the industrial communities. Besides, long-tailed (LT) training data are frequently encountered in the wild (He Garcia, 2009). Thus, many deep learning methods have been developed to combat it (Cui et al., 2021; He et al., 2021; Zhong et al., 2021). Yet, the devices of daily usages, of many which aim to utilize these deep mod- els, suffer from lack of sufficient computing power. Thus, for real world deployment of such deep models, methodological advances in LT recognition should also consider the resource constraints.
Unfortunately, current LT recognition methods largely assume sufficient computing resources and are designed to work with a large number of full precision parameters, i.e., floating point (FP) weights. While recognition on LT distributions by FP models may have significantly improved (Cui et al., 2021; He et al., 2021; Zhong et al., 2021), it is not clear whether these improvements would immediately translate real world scenarios not where resource constraints limit the model selection to those with lower capacity. To this end, we argue that it is necessary to benchmark and improve the performance of long-tailed recognition with capacity-limice models.
As binary networks are at the extreme end of sapacity-limited models (Rastegari et al., 2016), the long-tailed recognition performance using the 1-bit networks would roughly correspond to the 'worst case scenario' for resource constrained LT. If we can show sufficient LT performance with binary networks, we can reasonably expect, at the very least, matching or better LT performance with --bit models, where N > 1. Here, we take an initiative to benchmark and develop long-tailed recognition methods using binary networks as a challenging reference.
In the LT scenario, the data scarcity in the tail classes is one of the major issues that may cause prob- lems such as the class weights having varying magnitudes for head to tail classes (Kang et al., 2019; Alshammari et al., 2022), leading to disappointing performance. Prior methods (Liu et al., 2019; Kozerawski et al., 2020; Park et al., 2022) use cosine classifiers which eliminate the effect of class weight norms discriminative statistic and improve accuracy. However, since binary networks lack learning capacity and exhibit worse generalization performance than FP networks (Rastegari al., 2016; Courbariaux et al., 2016), we want reduce the adverse effect of the uneven class

---

Under review as a conference paper at ICLR 2023
6.0
500
Binary FP
4.7
Samples
From Scratch
Co Calbrate and Distil
400-
3.4
300
2.1
200
0.8
100
-0.5
100 -0
0
20
40
60
80
Head
Class Index
Tail
Figure 1: Gap between the classifier weight norms of binary and floating point (FP) network (i.e., Binary-FP") trained using Adam on CIFAR-100 (imbalance ratio: 100) (Sec. 4), also used in Al- shammari et al. (2022)). *# Samples' indicate the number of training samples for each class and the classes are sorted by the number of training samples (large small), e.g., head to tail. When trained from scratch on the long-tailed (LT) distribution, the gap widens towards the tail classes due to data scarcity. The widening gap implies that the binary network has higher variance (i.e., larger weight magnitudes) compared to FP. Using our 'Calibrate and Distill' mitigates the gap substantially.
weight norms without sacrificing its usefulness as a discriminative statistic. Inspired by previous LT literature (Kang et al., 2019; Alshammari et al., 2022) that analyzed the classifier weight norms (of the last molle-coneecte layer) in FP networks with linear classifiers, we first investigate the high variance issue in binary networks (Zhu et al., 2019) by visualizing the Binary-FP gap in the mag- nitude of the weight norms of the classifier in Fig. 1. The widening gap towards the tail classes indicates that a binary network has higher variance at tail classes than FP models. We argue that we need to mitigate the above issue to improve LT recognition with binary networks.
One can reduce the high variance in binary networks by providing additional supervisory signals to the binary networks during training. The additional supervision would supplement the insufficient information due to the scarce data in the tail classes. In different line of work, binary network learning methods (Martinez et al., 2020) employ additional supervision by a pretrained FP network used teacher in knowledge distillation when training binary networks. Furthermore, works in LT recognition with FP backbones also use distillation (He et al., 2021) to improve the accuracy.
However, these methods are not straightforwardly applicable our scenario for the following rea- son. As most of pretrained FP networks are learned on non LT datasets, using them as teachers may not be helpful for training binary networks on LT datasets due the domain gap. One can train the FP teacher on the target LT datasets from scratch (He et al., 2021) before distillation but that is not scalable. Recent works in various domains such as prompt engineering (Radford al., 2021; Liu al., a0) and self-supervised learning (Bachman et al., 2019; Chen at, 2020) suggest utilizing pretrained weights desirable for scalability (He et al., 2020). Here, we propose a way to utilize pretrained teacher networks on non LT data (e.g. from open source projects) for LT distillation with- out training the teacher from scratch. Specifically, we propose a 'Calibrate and Distill' framework where we calibrate the pretrained teacher on non LT data by attaching and training the classifier on the target LT dataset. We investigate how the proposed method affects the variance and visualize the results in Fig. 1. As shown in the figure, the calibrate and distill method significantly decreases the 'BinaryFFP' gap, suggesting a reduction of the high variance in binary networks. Note that while the magnitudes are within similar magnitude ranges, they are not equal for all classes, which could be useful for discriminating different classes.
In addition, semantic domains and image resolutions vary across multiple real-world data. In ad- dressing the variety, we found that balancing the amount of knowledge transferred from the dis- tillation loss to the feature extractor and classifier part of the binary network largely affects the performance, depending on how varied the target data is to the pretraining data. However, manual tuning may result good performance only for a limited number of benchmark datasets and may not generalize to various scenarios. For better generalization to different data distributions, we propose a novel adversarially learned balancing scheme for long-tailed recognition that learns the balance dependent on the input data without requiring finicky hyper-parameter tuning.
Furthermore, we incorporate various input resolutions for better generalization to different data dis- tributions (Jacobs et al., 1995; Rosenfeid, 2013). This would ease the burden for the model to learn various optimal receptive fields that can help generalize to the differently sized data. But using multi-resolution inputs (Rosenfeld, 2013) may incur increasing computational costs depending on what and how many resolutions are used. For negligible training time increase in multi-scale learn- ing, propose to selectively use multi-resolution inputs mitigate the extra computation costs
2

---

Under review as a conference paper at ICLR 2023
within our calibrate and distill framework. Combining all the proposals, we name our method as Calibrate and Distill: Long-Taie Tailed Recognition on the Edge or CANDLE for short.
We empirically evaluate suder-sampling existing non LT datasets and some are from existing LT benchmarks, that far out- numbers the previously used datasets in the LT literature. On our empirical validations in the various datasets, our method outperforms prior arts by large margin of at least + 14.33% on average.
wide variety of datasets totalling to 15, where some derived by
We summarize contributions follows:
Addressing resource constrained long-tailed recognition with binary networks for the first time, Proposing a calibrate and distill framework for long-tailed recognition with binary networks uti- lizing the off-the-shelf pretrained teachers on non-LT distributions.
Proposing a adversarially learned balancing scheme and efficient usage of multi-resolution inputs for the calibrate and distill framework.
Evaluating the methods in widely varied set of 15 long-tailed datasets for a comprehensive empirical study (the largest empirical study in the LT recognition literature).
2 RELATED WORK
Long-tailed recognition. As training deep learning models on LT distributions is challenging,
numerous works have been proposed. We categorize them into class rebalancing (He & Garcia, 2009; Buda et al., 2018; Shen et al., 2016), logit adjustment (Cao et al., 2019; Cui et al., 2019; Tan al., 2020), two stage methods sequential learning of representation and classifier (Kang et al., 2019; Xiang et al., 2020; Ren et al., 2020: Tang et al., 2020; Li et al., 2020), and knowledge distillation (He et al., 2021: Wang et al., 2020a; Zhang al., 2021a; Xiang et al., 2020).
Prior arts usually train the model from the scratch on the target LT dataset, which may be less efficient than using pretrained weights. For instance, two stage methods (Kang et al., 2019; Xiang et al., 2020; Ren et al., 2020; Tang et al., 2020; Li et al., 2020) perform representation learning without utilizing off-the-ssheff pretrained weights. Granted, the pretrained weights are usually trained on nT dataset with a considerable domain gap. However, we aim to utilize those pretrained weights on non-LT distributions for more efficient and effective long-tailed recognition.
Furthermore, some works tend to increase the model capacity for the long-tailed problem (Wang et al., 2020a; Zhou et al., 2020; Cui et al., 2021; 2022) which incurs additional costs. For the real- world scenario, one has to consider the resource constraints, thus resource efficient methods should be preferred. Additionally, most works are benchmarked only on a few datasets with little regards to actual data variety in the wild. To more accurately benchmark LT methods that consider the realistic variety, we use a total of 15 different LT datasets whereas previous works mostly showed results on 3-9 (Cao et al., 2019; Kang al., 2019; Cui et al., 2021; 2022) datasets.
Binary networks. As one of the ways of bringing extreme efficiency to deep learning models, many prior arts have been proposed that either improves upon the underlying architecture of binary networks (Rastegari et al., 2016; Lin et al., 2017; Liu et al., 2018; 2020; Bulat et al., 2021), use neural architecture search to find better binary networks (Kim et al., 2020; Kim & Choi, 2021; Bulat et al., 2020), claborate training schemes for effective optimization of binary networks (Martinez al., 2020; Han et al., 2020; Meng et al., 2020; Liu et al., 202 Le al., 2022), or explore using binary networks as backbones for previously unused scenarios such as unsupervised learning (Kim & Choi, 2022; Shen et al., 2021).
Recent successful attempts in training binary networks utilize FP network to provide additional supervisory signals (Martinez et al., 2020; Liu al., 2020; Bulat et al., 2021; 2020; Kim et al., 2020; Kim & Choi, 2021; 2022). However, these methods use teacher networks that are trained on data with similar distributions to that of the target data which amounts to little domain gap in the teacher network. As this process is not scalable, we propose to utilize teacher networks trained on non-LT data for many different target LT data distributions, which is scalable but challenging.
Additionally, most existing works focus on the supervised classification problem and methods that consider other scenarios such as unsupervised representation learning (Kim & Choi, 2022; Shen et al., 2021) and object detection (Wang et al., 2020b) relatively scarce. As such, there is hardly any previous work regarding long-tailed recognition with binary networks which brings resource efficiency learning on long-tailed distributions. Here, aim train resource efficient binary
3

---

Under review as a conference paper at ICLR 2023
35
8.0
500
Weight Nom
# of Samples
Binary
FP
Weight Norm
6.4
of Samples
Binary
FP
or Samples
17
3.2
200
11
1.6
100
5-0
20
40
60
80
100
0.0
0
20
40
60
80
100 0
Head
Class Index
Tail
Head
Class Index
Tail
(a) From Scratch
(b) CANDLE (Ours)
Figure 2: Classifier weight norms of binary and FP models trained with Adam on CIFAR- 100 (im-
balance ratio: 100). (a) The classifier weight norms increase at the tail classes, implying high
variance. When trained from scratch, binary networks show larger weight norms than FP in tail classes, implying that the high variance issue is more severe in binary networks. (b) Using CAN-
DLE, the weight norms of binary network become smaller than the FP model with little increase for tail classes. This implies that our method reduces the high variance in LT for binary networks.
networks that can also perform accurate long-tailed recognition to better prepare for long-tailed recognition in real-world deployment.
APPROACH
To develop long-tailed recognition methods for real world deployment that considers realistic re- source constraints, we use binary networks as backbone networks, which are extremely efficient neural networks with single bit weights and activations (Rastegari et al., 2016) (see Sec. A.1 for a brief preliminary). We first explain how the class weight norms (Kang et al., 2019; Alshammari model and analyze its
et al., 2022) can be used as an approximate to visualize the variance trends in LT (Sec. 3.1). With the analysis, we describe the proposed 'Calibrate and Distill' frame- work (Sec. 3.2) and plot the classifier weight norms to investigate how the proposed framework reduces the high variance in binary networks (Fig. 4). Additionally, to make the model more gener- alizable to the various semantic domains and image resolutions present in the wild that may differ from benchmark to benchmark, further propose adversarially learned balancing (Sec. 3.3) and an efficient usage of multi-resolution inputs with negligible computational cost increase (Sec. 3.4).
3.1 HIGH VARIANCE FOR BINARY NETWORKS IN LONG-TAILED DISTRIBUTION
The scarcity of data for tail classes impacts the bias and variance of a model when it is trained on LT distributions (Wang et al., 2020a). Specifically, we believe the model with linear classifiers is likely to exhibit high variance as it fits to the few training samples in the tail classes. We hypothesize that the above will be more severe for binary networks as they have limited capacity, making the models more susceptible to overfitting to the scarce training data in the tail. Previous works (Kang et al., 2019; Alshammari et al., 2022) analyzed the imbalance in the classifier weight norms for classes ranging from head to tail. We carefully re-visit how the classifier weight norm analysis could be used in our scenario as there are number of differences that might affect the analysis as follows.
First, we found that previous works (Kang et al., 2019; Zhou et al., 2020; Alshammari et al., 2022) used the SGD optimizer with weight-decay when analyzing the classifier weight norm. However, using the Adam optimizer significantly outperforms using the SGD optimizer when training binary networks (Han et al., 2020). Thus, using SGD will lead to inaccurate observations as the analysis will be done for models that have not sufficiently fit to the training data yet.
Second, weight decay is a regularization technique that pushes the weights to be closer to zero, which is not desirable for binary networks (Han et al., 2020) which quantize the weights to +1/ 1. Furthermore, as weight decay regularizes the training, it may result in the binary network under- fitting to the training data, similar to using the SGD optimizer (Han et al., 2020; Kim & Choi, 2021). 1). Perhaps even more problematic, weight decay may artificially reduce the weight magnitudes, making accurate analysis of the variance challenging. Thus, weight decay in observing the classifier weight norms for our experiments Please see Sec. A. for more discussion the difference in weight norm trend with Adam and SGD.
the Adam optimizer with zero
4

---

Under review as a conference paper at ICLR 2023
Muli-Rees Feat. Map
Mu Res.
Non LT Pretrained
Trahing Teacher Network
Teachor
Size Interpolazion & Concat
Encoder
Ch
Cat
LTddaaa
Size Interpolation Concar
Network
FP Teacher
Encoder
LTdua
CEL
Callbration
Distilation
Attend
Load Weight
Enooder
Ch
Forward Path
LT-at
Badkward
Figure 3: Overview of the proposed CANDLE. The FP classifier is calibrated with multi-resolution inputs. In distillation, only the teacher network receives multi-resolution inputs which saves com- putational costs significantly (see Fig. 5). Adversarially learned balancing via 'Atteng" is used in training the binary encoder. During calibration, the FP teacher encoder is frozen. During distillation, the FP teacher encoder, size interpolation and concat module, and the FP classifier are frozen.
We visualize the classifier weight norms in Fig. 2-(a). When the model is trained from scratch, we observe that the classifier weight norms get bigger as the classes become more tail. A similar phenomenon can be observed in machine learning literature where the model is trained to have high variance (e.g., high weight magnitudes) when it fits excessively to the training data (Kohavi et al., 1996; Von Luxburg & Schölkopf, 2011). Thus, overfitting to the scarce training samples in the tail, leading to the weights for the tail classes having high magnitudes and large variance (see Sec. A.2 for more details). In line with hypothesis, this trend is intensified for binary networks. As shown in Fig. 2-(a), the binary networks exhibit consistently larger weights with widening gap between the binary and floating point models (see Fig. 1) towards the tail classes.
interpret the above observation as a signal of
In contrast, we apply the proposed CANDLE as depicted in Fig. 3 and conduct the same analysis using classifier weight norms in Fig. 2-(b). The magnitudes of the binary weights are largely reduced with the gap between the binary and FP being reduced significantly (as also shown in Fig I). We attribute the reduction in variance to the *Calibrate and Distill' component of CANDLE that supplies additional supervisory signals in the tail classes from the FP teacher.
3.2 CALIBRATE AND DISTILL.
To overcome the high variance issue, propose 'Calibrate and Distill' scheme that provides additional information to the binary network during training over the given scarce training data in the tail classes. Inspired by recent methods in training binary networks that use FP teacher for distillation (Martinez et al., 2020), we use pretrained FP networks as distillation teacher networks in training the binary network on LT data. Note that most existing pretrained weights are not neces- sarily trained on LT distributions and is also not scalable to manually acquire pretrained weights on each target LT datasets. Here, we propose the 'Calibrate and Distill' scheme utilize non-LT pretrained weights for use on LT datasets in an efficient manner.
Specifically, we propose first calibrate the teacher network with non-LT pretrained weights and then transfer the knowledge from the calibrated teacher. In particular, we attach a randomly ini- tialized classifier the non-LT pretrained FP network. During calibration, the classifier trained using the cross entropy loss LCE on the target LT datasets using Zhong et al. (2021), which is more efficient than training the entire teacher. We use the KL divergence and feature similarity losses be- tween the calibrated teacher and the binary networks for distillation following (Romero et al., 2014; Heo et al., 2019). The optimization objective is
min EewDer [(1 (FT (f) + XCrs(e" (x),
where DLT is the target LT dataset and (O, are the calibrated teacher and binary net- works, respectively. e"(-), ef(() (-) are the teacher and binary encoders, i.e., feature extractors, and CRI((,)) Crs(s)) are the KL divergence and cosine distance, inspired by Chen et al. (2020); He al. (2020), respectively.
5

---

Under review as a conference paper at ICLR 2023
Weight Norm
22.0
500
17.8
* of Samples
salline
Cal lbrate and Distil
400
13.6
300
9.4
200
100 O
5.2
1.0
40
60
80
100
0
Head
Class Index
Tail
Figure 4: 'Calibrate and Distill' vs. the baseline of using the pretrained weights without calibration in
terms of classifier weight norms on CIFAR- 100 (imbalance ratio: 100). 'Calibrate and Distill' main-
tains the magnitudes relatively small even for tail classes whereas the baseline does not, showing the effectiveness of 'Calibrate and Distill' reducing the high variance compared to the baseline.
When empirically compare 'Calibrate and Distill' to the baseline, i.e., using the pretrained weights without calibrating with feature distillation (Romero et al., 2014; Heo et al., 2019), we see an accuracy gain of 7.7%%% average (see Tab. 3). More interestingly, observing the classifier weight norms we found that the proposed 'Calibrate and Distill' method is highly effective mitigating the high variance compared the baseline as shown in Fig. 4-)a). Note that while the high variance in binary networks is reduced through 'Calibrate and Distill', varying semantic domains with diverse image resolutions in the real world data are not sufficiently addressed yet.
3.3
ADVERSARIALLY LEARNED BALANCING
The encoder of our teacher network is pretrained on non-LT data but the classifier is trained on the target LT datasets. Optimizing the binary encoder and the classifier altogether with Eq. will disregard this difference and may disrupt the training of the binary encoder by using the gradients from LKL in the same way that we use to train the binary classifier. Thus, depending on how much the LT target data differs from the non-LT pretraining data, we may employ different strategies for balancing LFs and LK via À to train the binary encoder and the classifier. However, manually tuning for LKI. and Lis for each different target data is not desirable for generalization.
To address the semantic diversity without manual tuning, we propose to parameterize and learn during training as well. To that end, we first split the binary network's parameters 0 into the binary encoder parameters (0.) and classifier parameters (0c), and optimize them separately as
min Bgr-arDD ur (I" (x), 8. min 0g
Eer (A (z), (Q, ACrs(e"(xx, (z))).
To learn X, we use LKz. and LFs as inputs and use an attention module with learnable parameters output single scalar value As such that A dependent on the two loss functions
A = Af (rs(x)),
where (-) is a MLP that takes in (x)(x) and (Fs(T), and outputs Ag. Note that minimizing with respect to & results in the learned À outputting zero as the balancing coefficient for whichever loss function is larger ie., 0 for LFs in Eq. if Lrs > LKL. While this does minimize the loss, it also prevents the gradient flow from Les when optimizing with respect A. Thus, we employ adversarial learning where N is learned to maximize the loss which is then minimized by optimizing 0. We can rewrite objective
(x), min En I (f (1) (x)))
6e
min max EewDir [[1 (11-111CLLL AoCosl (x))).
d
3.4 LEARNING WITH MULTI- MULTI-RESOLUTION INPUTS
To make our model more robust to various image resolutions frequent in the wild, we argue that using multi-resolution inputs during training would be plausible solution (Rosenfeld, 2013). Specifically, we propose to use 128 x 128, 224 x 224 and 480 x 480 resolution images as multi-resolution inputs, following Tan & Le (2019). However, directly using multi-resolution inputs dramatically increases

---

Under review as a conference paper at ICLR 2023
VRAM Peak (MiB)
10000
15000
20000
25000
30000
35000
Single-Res Our Multi-
VRAM Peak (MiB)
Training Time (sec.)
Direct Multi-Res
150
200
250
Training Time (sec.)
Figure 5: Computational (blue bars) and memory (orange bars) costs comparison between directly using multi-resolution imputs to our method of using them only for the teacher ("Our Multi-Res' in terms of VRAM peak (MiB) and I epoch training time (sec.). 'Our Multi-Res' shows negligible increase in training time compared using single resolution. While the VRAM peak does increase with "Our Multi-Res', it is still far more efficient than directly using multi-resolution inputs.
the training time shown in Fig. 5 (Single-Res Direct Multi-Res), Thus, we propose novel way to efficiently use multi-resolution inputs for our 'Calibrate and Distill' framework such that there is negligible increase in training time (Single-Res vs. Our Multi-Res in Fig. 5).
As we harness supervisory signals from both the input data and the teacher network, we can se- lect which supervisory signal source the multi-resolution inputs are used. Instead of using multi- resolution inputs both sources, we only use them when obtaining supervisory signals from the teacher. To do so, we first calibrate the teacher using multi-resolution inputs shown in Fig. 3. As the calibration only trains the classifier (see Sec. 3.2), the usage of multi-resolution inputs is rela- tively inexpensive. During distillation, use single-resolution inputs for the binary network. Because the teacher is not part of the backward computational graph, the increase training time is negligible (see Fig. 5). The 'VRAM peak' indicates the peak consumption of GPU memory during training. Our scheme does incur some VRAM peak increase but much more efficient than training schemes that directly use the multi- resolution inputs to both teacher and binary networks. Note that with the multi-resolution inputs, the feature similarity loss is calculated using only the channels that correspond to the input with the same resolution as that of the input given to the binary network as depicted in Fig. 3.
feed the multi-resolution inputs the teacher only and
We use calibrate and distill with the adversarially learned balancing and an efficient way of using multi-resolution inputs as our full method named as CANDLE.
EXPERIMENTS
Setup. For empirical evaluations, we use 12 small-scale and large-scale LT datasets as detailed in Sec. A.4. We use pretrained FP networks from public open source projects that are either trained on ImageNet- IK for the small-scale and on ImageNet-21K for the large-scale LT datasets. Fol- lowing (Liu et al., 2021b), we use the Adam optimizer with zero weight decay for training binary networks. Further implementation details are in Sec. A.5. Note that we use the authors' implemen- tation of prior arts when available or re-implement by reproducing the reported results. More details on the compared prior arts are in Sec. A.3. We will publicly release the code for CANDLE along with the newly derived LT datasets.
Extended benchmarks for long-tail recognition. In addition to developing a method for LT recognition using binary networks, we also want improve how the performance of LT methods are measured to better reflect the real-world scenario (Yang et al., 2022). To that end, we compile large number of derived LT datasets from commonly used computer vision datasets along with the datasets often used in the LT literature to gather 12 small-scale and large-scale datasets for evalua- tion. We denote the imbalance ratio in a parenthesis following the dataset names for the small-scale datasets and add a postfix of LT" to each of the large-scale dataset names. More details regarding on the newly added datasets along with dataset statistics for all 15 datasets are given in Sec. A.4.
4.1 LONG- TONG-TAILED RECOGNITION ACCURACY IN VARIOUS DATASETS
We present comparative results our method with various state of the arts LT methods tuned for binary networks. We show the average accuracy over multiple datasets. Please see Sec. A.11 for head, medium, and tail class accuracy and per-class accuracy plots for CANDLE. All results of CANDLE E are with linear classifiers (see Sec. A.12 for results with cosine classifiers small-scale
7

---

Under review as a conference paper at ICLR 2023
Table I: Average accuracy (%) of the proposed CANDLE and other methods on small-scale datasets. We compare with 7-norm (Kang et al., 2019), BALMS (Ren et al., 2020), PaCo (Cui et al., 2021), MiSLAS (Zhong et al., 2021), and DiVE (He al., 2021). DiVE" trained using the same teacher as ours. Our method shows superior performance by large margins across many different datasets
with a margin of at least + 14.90% in mean accuracy.
Dataset (Imb. Ratio)
7-norm
BALMS
PaCo
MiSLAS
DiVE
DiVE
CANDLE
Caltech101 101 (10)
20.68
17.00
45.85
36.38
20.91
39.66
55.31
CIFAR-10 (10)
24.28
17.40
58.59
82.25
13.73
79.46
91.76
110AR-10 (100)
21.72
16.50
61.57
69.50
16.26
58.41
85.01
CIFAR-I00 (10)
8.18
2.70
49.63
47.03
2.26
49.93
66.09
CIFAR-100 1-0 (100)
5.56
3,40
39.66
32.31
1.94
32.95
50.88
-UB-200-2011 (10)
8.52
1.30
15.41
14.74
4.09
12.82
42.94
Stanford Dogs (10)
13.93
9.80
34.04
29.73
6.72
29.21
58.79
Stanford Cars (10)
7.73
2.90
23.74
24.82
4.49
19.31
51.06
DTD (10)
16.86
14.10
36.65
36.54
12.71
30.37
38.56
FGVC-Aireraft (10)
7.71
4.30
15.12
22.08
1.95
14.19
39.78
Flowers- 102 (10)
45.29
43.60
52.16
60.20
23.82
58.53
64.61
Fruits-360 (100)
85.82
97.30
99.23
99.49
50.87
99.71
100.00
Mean Acc.
22.23
19.19
44.30
46.14
13.31
41.80
62.07
datasets). The best accuracy is in bold and the second best is underline. Surprisingly, DiVE (He ct al., 2021) does not perform well on the small-scale datasets as the teacher network used in DiVE's distillation process, e.g., BALMS (Ren al., 2020) exhibits disappointing accuracy (~ 26.65% mean acc.) in the first place. Hence, we also present results with DiVE" which uses the same teacher network as our method. More discussion regarding some of the other low performing methods on the small-scale datasets such as r-norm (Kang et al., 2019) and BALMS is in Sec. A.6.
In small-scale datasets. We summarize the comparative results on the 12 small-scale datasets in Tab. 1 along with the mean accuracy over all 12 datasets. The mean accuracy over the 12 datasets suggests that CANDLE performs better than the prior arts by noticeable margins of at least +15.93% More precisely, the proposed method outperforms all other existing works by large margins for the popular CIFAR-I0 (10,100) 100) and CIFAR- 1-0 (10, 100) datasets. In addition, our method exhibits superior performance in some of the newly added LT datasets with limited data such as 21B-200-2011 (10). Interestingly, comparing DiVE and DiVE", we can see that using calibrated teacher boosts the performance significantly. However, the performance difference between DiVE" and CANDLE (+-00.27%) also indicates that our adversarially learned balancing (Sec. 3.3) and multi-resolution teacher is effective (Sec. 3.4). Additional results changing the teacher network to the larger one used for large-scale experiments (Sec. 4) is in Sec. A.7, which show that the mean accuracy improves when using the larger teacher. Further comparison against the ensemble-based method, TADE (Zhang et al., 2021b). is also in Sec. A.8.
In large-scale datasets. Similar to Tab. 1, we compare our method to prior arts on the large-scale datasets in Tab. 2. While our multi-resolution teacher is far more efficient than directly using multi- resolution (Fig. 5), VRAM peak still increases. Thus, we could not run the multi-resolution experiments with the large-scale datasets with our resources and only use single-resolution, indicated using i Tab. 2. Still, even without using multi-resolution, the proposed method outperforms prior arts with binary networks on all three large-scale datasets. Looking at the mean accuracy, our method shows a margin of at least +7.93% when compared to the previous LT works.
4.2 LEARNED A IN DIFFERENT DATASETS
The learned values of Ao (Eq. 4) for each dataset vs. training iteration is visualized in Fig. Note that we do not schedule Às in any specific way. Interestingly, observe smooth transitions from the initial values different initial loss values for each dataset. In addition, while the learned X converges to I (using only the LFs) in Eq. 4, the way in which the A changes different depending on the dataset.
for all the shown datasets. The initial values are different as they depend on the
Combined with the empirical gain (+5.40% mean average accuracy in Tab. 3) of using adversarially learned balancing, the various different ways in which A varies in Fig. 6 imply that adversarially learned balancing (Eq. is effective in balancing LKL. and Lrs wide variety of LT datasets.
8

---

Under review as a conference paper at ICLR 2023
Table 2: Average accuracy (%) of CANDLE and other methods on large-scale datasets. indicates that we only use single-resolution due VRAM limits. DIVE" trained using the same teacher network as ours. The proposed method outperforms other existing works in all three large-scale
datasets with at least + 7.93% margin terms of mean accuracy.
Dataset
7-norm
BALMS
PaCo
MiSLAS
DIVE
DIVE
CANDLE
Places-LT
25.99
23.60
23.30
28.49
20.96
22.93
34.11
ImageNet-LT
30.59
33.00
34.58
34.71
31.21
30.86
49.10
iNat-LT
37.36
44.30
45.73
43.59
41.32
41.30
47.38
Mean Acc.
31.31
33.63
34.54
35.60
31.16
31.70
43.53
0.8
Caltech101 (10)
Stanford Dogs (10)
0.7
CIFAR-10 (10)
Stanford (10)
0.6
CIFAR-10 (100)
DTD (10)
CIFAR-1 00 (10)
FGVC-Aircrat (10)
0.5
CIFAR-1 00 (100)
Flowers-100 102 (10)
CUB 1199-200-000 2011 (10)
Fruits-360 (100)
20
40
60
80
100
Training Iteration
Figure 6: Learned value of A on different datasets during training by Eq. Note that the conver- gence behavior of A to 1.0 differ for each dataset.
Table 3: Ablation studies of our method, CANDLE. We present the mean accuracy over the 12 small-scale datasets (Tab. 4). Using our 'Calibration & Distill" framework drastically improves the performance from the baseline of using the pretrained weights without calibration. Adversarially learned balancing and multi-resolution inputs also improve the mean accuracy substantially.
Accuracy (%)
Method
Calibrate Distill
Leamed Bal. Multi-Res.
Avg.
Head
Med.
Tail
Baseline
45.27
53.55
44.40
32.47
55.59
62.99
55.90
42.57
1+2
60.99
66.64
61.72
49.61
U+2+3 (ECANDLE)
62.07
71.22
63.33
50.09
4.3
ABLATION STUDIES
We finally conduct ablation studies of our I 'Calibrate & Distill', 2 'Adv. Learned Bal.', and 3 'Multi-Res.' the 12 small-scale datasets and report the average, head, medium, and tail class accuracy in Tab. The baseline is set to using the uncalibrated teacher with feature distil- lation (Romero al., 2014; Heo al., 2019) (refer Sec. 3.2 for detail). Note that the baseline achieves 45. 15.27% mean average accuracy, only behind MiSLAS. All proposed components of CAN- DLE improve the mean average accuracy, e.g., +10.32% for 'Calibrate & Distill', +5.40% for 'Adv. Learnd Bal.', and +1.08% for 'Multi-Res... Furthermore, the mean tail class accuracy also im- proves consistently with each component, indicating the effectiveness of respective components for LT. The average accuracy for each dataset for the ablated models can be found in Sec. A.9.
CONCLUSION
To develop efficient deep learning models for LT, we use binary networks as backbones for learning long-tailed recognition. In doing so, we analyze high variance issue and propose to mitigate using our 'Calibrate and Distill' framework where pretrained weights on non-LT data are utilized as distillation teachers for learning binary networks on LT target data distributions. We further propose adversarially learned balancing and efficient usage of multi-resolution inputs to better prepare the model for the wide variety of semantic domains and input resolutions present the wild. We empirically evaluate the proposed CANDLE and other existing works on a total of 15 datasets, which the largest benchmark the literature. The proposed method achieves superior performance in all empirical validations with ablations showing clear benefits of each component of CANDLE.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This work aims to equip edge-compatible binary networks with long-tailed recognition capabilities. Thus, though there is no intent on the authors, application of deep models to real-world scenarios with long-tail tail data distributions might become prevalent. This may expose the public to discrimina- tion by the deployed deep models due to unsolved issues in deep learning such as model bias. We will take all available measures to prevent such outcomes as that is not the intention of this work.
REPRODUCIBILITY STATEMENT
We take the reproducibility in deep learning very seriously and highlight some of the contents in the manuscript that might help in reproducing our work. First, we will release the code and newly derived datasets used in our experiments as mentioned in Sec. Second, we include additional in- formation regarding the extended long-tailed benchmark in Sec. A.4 that may help with future work looking use the datasets that were used in this work. Third, we include relevant implementation details in both Sec. and Sec. A.5. Last, present our final optimization objective (Eq. 4) and the overall view our method (Fig. 3) with necessary details to reproduce the methodology.
REFERENCES
Shaden Alshammari, Yu-Xiong Wang. Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In CVPR, pp. 6897-6907, 2022. 2. 4,
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. NeurlPS, 32, 2019.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance
problem convolutional neural networks. Neural networks, 106:249-259, 2018. 3
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks. In BMVC, 2019. 14
Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos. Bats: Binary architecture search. In ECCV, 2020.
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. High-capacity expert binary networks. In ICLR, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-disrrbuution-aware margin loss. NeurlPS, 32, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2,
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014. 15
Matthieu Courbariaux, Yoshua Bengio, and Sean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015. 16
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1, arXiv preprint 0002.00022220000 2016. 1, 14
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In ICCV, pp. 715-724, 2021. 1, 8, 14
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tail tailed recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
10

---

Under review as a conference paper at ICLR 2023
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, pp.
2019.
Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In ICML, 2020. 3,
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263-1284, 2009. 1,3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2015. 15
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 2,
Yin-Yin Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recognition. In ICCV, pp. 235-244, 2021. 3, 14
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, October 2019. 5, 6,
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: FrxXiv:/044.044 2017. 1
Charles E Jacobs, Adam Finkelstein, and David H Salesin. Fast multiresolution image querying. In Proceedings of the 22nd annual conference Computer graphics and interactive techniques, pp. 277-286, 1995. 2
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint 1rXiv:1910.09217, 2019. 1, 2, 3, 4, 8, 14, 18
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, CVPR, Colorado Springs, CO, June 2011. 15
Dahyun Kim and Jonghyun Choi. Bnas v2: Learning architectures for binary networks with empir- ical improvements. arXiv preprint http:://urri.or////s21100...... 2021. 3,
Dahyun Kim and Jonghyun Choi. Unsupervised represnnttinnn learning for binary networks by joint classifier learning. In CVPR, pp. 9747-9756, June 2022. 3
Dahyun Kim, Kunal Pratap Singh, and Jonghyun Choi. Learning architectures for binary networks. In ECCV, 2020. 3
Ron Kohavi, David Wolpert, et al. Bias plus variance decomposition for zero-one loss functions. In ICML, volume 96, pp. 275-83, 1996. 5
Jedrzej Kozerawski, Victor Fragoso, Nikolaos Karianakis, Gaurav Mittal, Matthew Turk, and Mei Chen. Blt: Balancing long-tailed datasets with adverssarially-pertubeel images. In Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020. 1, 18
Krause, M. Stark, J. Deng, and L. Fei-Fei. 3D Object Representations for Fine-Grained Catego- rization. In ICCV Workshop, 2013. 15
Huu Le, Rasmus Kjær Høier, Che-Tsung Lin, and Christopher Zach. Adaste: An adaptive straight- through estimator to train binary neural networks. In CVPR, pp. 460-469, June 2022. 3
Li, Andreeto, Ranzato, and Perona. Caltech 101, 4 2022. 15
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcom- ing classifier imbalance for long-tail object detection with balanced group softmax. In CVPR, pp. 10991-11000 1000, 2020. 3
11

---

Under review as a conference paper at ICLR 2023
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In NeurIPS, 2017.
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmenta- tion. In CVPR, 2019. 1
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and prediet: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107. 13586, 2021a. 2
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capabilitty and advanced training algorithm. In ECCV, 2018. 3
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Tingg Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020. 3, 15
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang Ting Cheng. How do adam and training strategies help bans optimization? In ICML. PMLR, 2021b. 3,7
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515// 2013. 15
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. In ICLR, 2020. 3,5
Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan. Training binary neural net- works using the bayesian learning rule. In ICML, 2020. 3
Horea Mureşan and Mihai Oltean. Fruit recognition from images using deep learning. arXiv preprint arXiv:1712.0058) 2017. 15
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number
of classes. In ICVGIP, Dec 2008. 15
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In CVPR, pp. 6887-6896, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pp. 7748887633. PMLR, 2021. 2
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016. 1, 4, 14
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et Balanced meta-softmax for long- tailed visual recognition. NeurIPS, 33:4175 4186, 2020. 3, 8, 14
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.6550, 2014. 5, 6,
Azriel Rosenfeld. Multiresolation image processing and analysis, volume 12. Springer Science & Business Media, 2013. 2.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In ECCV, pp. 467-482. Springer, 2016.
Zhiqiang Shen, Zechun Liu, Jie Qin, Lei Huang, Kwang-Ting Cheng, and Marios Savvides. S2-bnn: Bridging the gap between self-supervised real and 1-bit neural networks via guided distribution calibration. In CVPR, pp. 2165-2174, 2021.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjic
Yan. Equalization loss for long-tailed object recognition. In CVPR, pp. 11662-11671, 2020.
12

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019. 6, 15
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang, Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurIPS, 33:1513-1524, 2020. 3
Ulrike Von Luxburg and Bernhard Schölkopf. Statistical learning theory: Models, concepts, and results. In Handbook of the History of Logic, volume 10, pp. 651-706. Elsevier, 2011.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd 2ods-2200-0011 dataset. Technical Report CNS-TR-2011 1-00,, California Institute Technology, 2011. 15
Chien- Yao Wang, Alexey Bochkovskiy, and Hong- Hlong-Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the for real-time object detectors. arXiv preprini urXiv:2207.02696. 2022. 1
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by routing diverse distribution-aware experts. ICLR, 2020a. 3,4
Ziwei Wang, Ziyi Wu, Jiwen Lu, and Jie Zhou. Bidet: An efficient binarized object detector. In CVPR, 2020b. 3
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self- saff-paced knowl- edge distillation for long-tailed classification. In ECCV, pp. 247-263. Springer, 2020. 3
Yuzhe Yang. Hao Wang, and Dina Katabi. On mulli--omali long-tailed recognition, generalization and beyond. arXiv preprint :xXi::2233.09113 2022. 7
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019. 14
Shaoyu Zhang, Chen Chen, Xiyuan Hu, and Silong Peng. Balanced knowledge distillation for long- tailed learning. arXiv preprint arXiv:2104. 10, 2021a. 3
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint 1rXiv:2107.09249, 2021b. 8, 16,
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recog- nition. In CVPR, pp. (6489-16498, June 2021. 5, 8, 14, 15
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with
cumulative learning for long-tailed visual recognition. In CVPR, 9719-9728, 2020. 3,4
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In CVPR, 2019. 2
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PRELIMINARY ON LEARNING BINARY NETWORKS (SEC. 3)
Let WF E RCC.C,xxxk be the 32-bit floating point weights of a convolution layer where C, Co and l represent the number of input channels, the number of output channels and the kernel size, respectively. The corresponding binarized weights are given by WB sign (WF), where the sign(-) outputs +1/ depending on the sign of the input. The floating point input activation AF E RCixwxh, where w, are the width and height of the input activation, can be similarly binarized to the binary input activation by AB AAgn A floating point scaling factor is also used, which can either be the L1 Lorm of WF (Rastegari et al., 2016) or learned by back- propagation (Bulat & Tzimiropoulos, 2019).
The forward pass using convolutions approximated as following
(5) where @ + denote the floating point convolution operation, Hadamard product, and the binary XNOR convolution with popcount, respectively. STE (Courbariaux et al., 2016) is used for the weight update in the backward pass.
WF AF ~ a (Wn AB),
A.2 DETAILED ANALYSIS ON THE HIGH VARIANCE (SEC. 3.1)
We give a more detailed explanation on the high variance analysis in Sec. 3.1. As we are more interested in the high weight magnitudes which signals high variance, focused on showing the classifier weight norms in the main part of the paper. Here, we also show the bias portion of the classifier and see how it is affected for binary networks in LT conditions.
As seen in Fig. 7-(a), we see the classifier weight norm (e.g. variance) become larger and larger for the tail classes when the binary network is trained from scratch. Looking the bias in Fig. 7-(b), we see the classifier bias getting smaller and smaller for the tail classes accordingly when the binary network is trained from scratch. contrast, when CANDLE is applied instead, the large increase in the classifier weight norm for the tail classes is substantially mitigated, resulting in the weight norm barely changing from head to tail classes. Similarly, the bias also relatively unchanging compared to training from scratch and shows that CANDLE stabilizes the binary network in terms of both variance and bias.
35
4.0
500
Weight Norm
Samples
From scratch
CANDLE
Samples
From scratch
CANDLE
29
2.4
400
samppl
23
0.8
300
17
-0.8
200
11
-2.4
100
4.0
0
20
40
60
80
100
0
20
40
60
80
100
Head
Class Index
Tail
Head
Class Index
Tail
(a) Cls. Weight Norm (Variance)
(b) Cls. Bias
Figure 7: Classifier weight norm (variance) and bias for binary networks trained from scratch or us- ing CANDLE on CIFAR- 100 (100). We can see that training from scratch results in binary networks with large variance and small bias whereas using CANDLE, binary networks have relatively stable variance and bias.
A.3 DETAILS ON COMPARED METHODS (SEC. 4)
We give brief summary of the compared methods. 7-norm (Kang et al., 2019) is a two-stage de- coupling method, which consists of classifier re- re-training(cKT) and learnable weight scaling(LWS), BALMS(Ren et al., 2020) is also a two-stage method, where the balanced softmax loss and meta sampler is used in classifier training stage. PaCo(Cui et al., 2021) uses supervised contrastive learn- ing to learn the class centers in long-tailed data distributions. MiSLAS(Zhong et al., 2021) focuses on calibrating the model properly on LT data with two-stage method that uses CutMix (Ym al., 2019) to enhance the representation learning stage. The classifier learning stage consists of label- aware smoothing and shift learning on batch normalization layers. DiVE(He et al., 2021) is distil-
14

---

Under review as a conference paper at ICLR 2023
lation method where the teacher network is trained by existing well perfroming LT-methods such as BALMS and then trains student model with KL divergence loss and balanced softmax loss.
A.4 DETAILS ON THE EXTENDED LT BENCHMARK (SEC. 4)
Table Dataset statistics all the LT datasets used our experiments. The dataset source denotes the original non-LT dataset from which the LT datasets were derived. The imbalance ratio for the large scale datasets are not explicitly calculated.
Dataset Source
Imbalance Ratio
# Train Samples
Test Samples
Caltech101
10
1151
6084
CIFAR-10
10
20431
10000
CIFAR-10
100
12406
10000
CIFAR-100
10
19573
10000
CIFAR-I00
100
10847
10000
CUB-200-2011
10
2253
5794
Small Scale
Stanford Dogs
10
4646
8580
Stanford Cars
10
3666
8041
DTD
10
1457
1880
FGVC-Aircraft
10
2564
3333
Flowers- 102
10
753
1020
Fruits-360
100
1770
3110
Places-LT
N/A
62500
7300
Large Scale
ImageNet-LT
N/A
115846
50000
iNat-LT
N/A
437513
24426
newly added LT datasets from existing computer vision datasets e.g., Caltech101 (Li et 2022),
CUB-200-200 (Wah et al., 2011), Stanford Dogs (Khosla et al., 201 Stanford Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), FGVC-Aircraft (Maji et al., 2013), Flowers- 102 (Nilsback & Zisserman, 2008), and Fruits-360 (Mureşan & Oltean, 2017). We first sort the class indices with the number samples per class to create a class order. We then set the imbalance ratio and sub-sample the data in each of the classes. In sub-sampling the data, use an exponential decaying curve to determine the class sample frequency from head to tail classes. As the number of data in each of the classes is limited, we often can only use certain imbalance ratio such 10 or 100.
We then detail the dataset statistics for all the LT datasets used in our experiments in Tab, 4, including the newly added ones and the previously used datasets in the LT literature. The added datasets have a wide variety of semantic domains which may have not been covered by the existing LT datasets in the literature.
A.5 FURTHER IMPLEMENTATION DETAILS (SEC. 4)
In the calibration stage, we used the class-balanced sampling and label-aware smoothing following MiSLAS (Zhong al., 2021). We
pretrained ResNet-152 (He et al., 2015) backbone for the ImageNet1K teacher and a pretrained EfficientNet (Tan & Le, 2019) backbone for the ImageNet2 K teacher. In distillation stage, we used ReActNet (Liu et al., 2020) our binary backbone network. Atteng is a MLP with 3 hidden layers with the number of channels expanding by 16 per layer starting
channels in the first layer. The MLP takes turned into a 2-dimensional vector and outputs single scalar value which is passed through sigmoid function be used as Ag. To process the multi resolution inputs to the teacher network, we use the size interpolation & concat module. The size interpolation & concat module is comprised of IxI convolutions that match the number of channels to that of the binary network and a nearest interpolation step where the different sized feature maps from the multi-resolution inputs are spatialy resized. Once the feature maps from the different resolution inputs have the same spatial size and number of channels, they channel-wise concatenated. To optimize the min-max objective of Eq. 4, we perform maximization step per every 1 minimization step. The binary networks are trained for 400 epochs with batch size of 256. We use the cosine annealing scheduling for
scalar values L and CFs(x)
15

---

Under review as a conference paper at ICLR 2023
Table 5: Accuracy (%) of the proposed CANDLE with different teacher networks. CANDLE uses the ImageNetIk teacher detailed in Sec. and CANDLE uses the ImageNe121K teacher which was originally used for large-scale experiments in Tab. 2.
Datascts (Imb. Ratio)
CANDLE (ImageNet Teacher)
CANDLE (ImageNet2IK Teacher)
Caltech101 (10)
55,31
52.27
CIFAR-I0 (10)
91.76
91.27
CIFAR-10 (100)
85.01
85,59
CIFAR-I00 (10)
66.09
64.93
CIFAR- 100 (100)
50.88
53.15
CUB-200-2011 (10)
42.94
49.45
Stanford Dogs (10)
58.79
63.51
Stanford Cars (10)
51.06
49.73
DTD (10)
38.56
40.64
FGVC-Aircraft (10)
39.78
41.6
Flowers- 102 (10)
64.61
66.27
Fruits-360 (100)
100.00
100.00
Mean Acc.
62.07
63.20
Table 6: Accuracy (%) of the proposed CANDLE with TADE (Zhang et al., 2021b) which is an ensemble-based method. Our method shows superior performance against even ensemble-based TADE by large margins across many different datasets with margin of 24.94% in mean accuracy.
Dataset (Imb. Ratio)
TADE
CANDLE
Caltech101 (10)
43.86
55.31
CIFAR-10 (10)
53.54
91.76
CIFAR-10 (100)
47.63
85.01
CIFAR- 100RR-00 (10)
29.75
66.09
CIFAR-IOS (100)
28.68
50.88
LUB-200-2011 (10)
13.00
42.94
Stanford Dogs (10)
17.20
58.79
Stanford Cars (10)
8.73
51.06
DTD (10)
27.98
38.56
FGVC-Aircraft (10)
12.69
39.78
Flowers- 102 (10)
62.46
64.61
Fruits-360 (100)
100.00
100.00
Mean Acc.
37.13
62.07
learning rate scheduler, where the initial learning rate is set as 0.01 for most of the datasets. Although did try different learning rates, found the difference to be negligible. We used one GPU for training on the 12 small-scale datasets and four GPUs for training on the large-scale datasets.
A.6 MORE DISCUSSION REGARDING 7-NORM AND BALMS (SEC. 4.1)
On comparison with existing LT methods on the small-scale datasets in Tab. 7-norm and BALMS seems to show relatively low accuracy. We hypothesize the following reasons for the low accuracy. For r-norm, it is based on how the weight norms distributed when the model trained with the SGD optimizer. Since we use the Adam optimizer to train the models due to having binary backbones (Sec. 3.1), the core assumption for 7-norm may not hold and hence the resulting low accuracy. For BALMS, it may have lacked the sufficient number of epochs to properly converge as we use binary backbone networks which usually take longer to train than floating point models (Courbariaux al., 2015).
A.7 EFFECT OF USING DIFFERENT TEACHER NETWORKS FOR CANDLE (SEC. 4.1)
We used the ImageNetlk pretrained teacher network for small-scale experiments in Tab. as we felt that the teacher network could still provide sufficient supervisory signals for the small-scale datasets.
16

---

Under review as a conference paper at ICLR 2023
Nonetheless, also conduct experiments the small-scale datasets with the larger teacher net- work pretrained on ImageNet21k which was originally used only for the large-scale datasets.
As shown in Tab. 5, not all datasets benefit from the larger teacher network such Caltech101 (10), CIFAR-10 (10), CIFAR- 100 (10), and Stanford Cars (10), However, there are noticeable gains of +6.51% % CUB-200-201 1-10 and +4.72% on Stanford Dogs-10 and the resulting mean accuracy is improved by +1.13%. For the large-scale datasets in Tab. 2, we could not use the ImageNet teacher that a direct superset of ImageNet-LT. The above reason why we used the ImageNet21K K teacher for the large-scale experiments. However, the gains in Tab. suggest that one could use the ImageNet21K teacher for the small-scale datasets as well.
A.8 ADDITIONAL. COMPARISON TO TADE (ZHANG AT.., 20218) (SEC. 4.1)
For more comprehensive comparison, also present results for TADE (Zhang et al., 20216), which an ensemble-based method. As shown Tab. 6, we show results for TADE on the 12 small-scale datasets along with CANDLE. Our method still outperforms TADE by large margins of +24.94% in terms of mean accuracy. Not only that, the proposed method seems to show higher accuracy both on the frequently used CIFAR-10 (10, 100) and CIFAR- (10, 100) datasets but also on the newly added datasets such as CUB-200-2011 (10), Stanford Dogs (10), and Stanford Cars (10).
A.9 MORE DETAILED ABLATION STUDY RESULTS (SEC. 4.3)
We presented the mean accuracy over the 12 small-scale datasets for our ablation studies in Tab. to more clearly present the performance of the ablated models. We also report the accuracy on each of the 12 datasets Tab. provide more details regarding our ablation studies. However, we also emphasize that looking at the accuracy of particular dataset only may not give the most accurate description.
Going from ① to 1 ② C (=CANDLE), 9 out of 12 datasets show an accuracy increase. On the 3 datasets where an accuracy decrease observed, the accuracy drops by 5.17% at most, whereas on datasets where the accuracy increases, it increases by up to 16%. Ultimately, the mean accuracy consistently goes up when add more and more parts of CANDLE, as also shown Tab. 3.
Table 7: A detailed ablation studies of method CANDLE. The accuracy 12 small-scale datasets as well as the accuracy for each dataset are shown. The circled numbers denote the following D: 'Calibrate & Distill', 2: Data-Driven Bal., 3: Multi-Res., respectively. Using our 'Calibration & Distill' framework drastically improves the performance across all the tested datasets when compared to the baseline of using the pretrained weights without calibration. 'Data- Driven Bal.' and 'Multi-Res Res.." also improve the accuracy substantially over all the tested datasets.
Ablated Models
Datasets (Imb. Ratio)
Baseline
1 + ②
+ (=CANDLE)
Caltech101 (10)
36.75
53.44
53.37
55.31
CIFAR-I0 (10)
88.90
82.08
92.34
91.76
CIFAR- 10 (100)
70.00
43.85
87.24
85.01
CIFAR- 100 (10)
48.96
69.07
68.58
66.09
CIFAR- 100 (100)
33.62
54.92
53.70
50.88
CUB-200-2011 (10)
12.69
37.66
37.61
42.94
Stanford Dogs (10)
25.66
63.96
64.42
58.79
Stanford Cars (10)
25.87
44.00
45.19
51.06
DTD (10)
21.33
40.21
36.38
38.56
FGVC-Aircraft (10)
28.02
29.91
29.22
39.78
Flowers- 102 (10)
51.86
62.16
63.82
64.61
Fruits-360 (100)
99.61
85.85
100.00
100.00
Mean Acc.
45.27
55.59
60.99
62.07
17

---

Under review as a conference paper at ICLR 2023
4.5
500
of Samples
SGD with WD (Acc: 13.24%)
SGD without WD (Ace: 15.46%)
3.7
400
Samples
2.9
300
Weight
2.1
200
1.3
100
0.5
0
20
40
60
80
100
Head
Class Index
Tail
Figure 8: Classifier weight norm of binary networks trained from scratch on CIFAR-100 (100) using SGD with or without weight decay. Using the SGD, we get similar trends for the class weight norms as in Kang et al. (2019); Alshammari al. (2022) where the class weight norms get smaller for the tail classes. However, as mentioned in Sec, 4.2, using the SGD when training binary networks results in very low accuracy of 13.24% or 15. 46% depending on whether weight decay used or not. Thus, we use the Adam optimizer, in which the classifier weight trends are different (see Sec. 3.1).
A. MORE DISCUSSION ON DIFFERENCE IN WEIGHT NORM TREND WITH ADAM OR SGD
As discussed in Sec. 3.1, we use the Adam optimizer for training the binary networks. This results in different trend in the classifier weight norms than some prior works (Kang et al., 2019; Alshammari et al., 2022). To give a more information regarding this matter, we also plot the classifier weight norm trend of training binary networks from scratch CIFARI00 (100) using the SGD optimizer in Fig. 8. We see that using the SGD shows similar trends in terms of the classifier weight norms as the prior arts (Kang al., 2019; Alshammari et al., 2022). However, the accuracy of the learned binary networks with SGD is very low for both with and without weight decay. We believe that further developing our analysis on such an under-fitted model will not be meaningful and use Adam optimizer instead.
Note that combined with results in Fig. 2, we see opposite trends in weight norms depending on whether we use Adam or SGD for both binary and FP networks. With Adam, we have weight norms increasing as the classes become tail, and with SGD, we have weight norms decreasing as the classes become tail instead. We conjecture that Adam adjusts the learning rates such that parameters corre- sponding to frequently occurring head class data use relatively small learning rates and parameters corresponding to infrequent tail class data use relatively large learning rates. As result, the opti- mization may result in the tail class weights having large weight norms due to the relatively larger learning rate used in the update. Note that more thorough studies the above phenomenon is ranted to gain deeper understanding and we leave that as an interesting future direction, respecting the original scope of this work.
A.11 ADDITIONAL INFORMATION REGARDING HEAD, MEDIAN, AND TAIL CLASS ACCURACY
To provide more information regarding the experimental results of CANDLE, we provide average, head, medium, and tail class accuracy of CANDLE for the 15 datasets in Tab. 8. For most of the datasets, CANDLE shows high accuracy at the tail classes compared the average, head, or the medium classes. This is especially true for large-scale datasets such as Places-LT, ImageNet-LT, and LN where the tail class accuracy is either on par or higher than the average accuracy.
Additionally, in Fig. 9, 10, 11, plot per-class accuracy improvement of CANDLE over the baseline used in Tab. following Kozerawski et al. (2020) on Places-LT, ImageNet-Li and iNat-LT, respectively. The figures show that CANDLE improves the accuracy at the tail and medium classes by large margins for all three datasets. Additionally, in the case of ImagetNet- -T, even the head accuracy shows noticeable improvements. The figures empirically show that CANDLE shows good
18

---

Under review as a conference paper at ICLR 2023
Table Average, head, medium, and tail class accuracy of CANDLE. On most datasets, CANDLE shows high tail class accuracy compared to the average, head, and medium class accuracy. Interest- ingly, this trend is stronger for large scale datasets where the tail class accuracy higher or on par with the average accuracy.
Dataset
Average
Head
Medium
Tail
Caltech101 (10)
55.31
49.25
46.74
35,54
CIFAR-10 (10)
91.76
95.50
90.50
91.03
CIFAR-10 (100)
85.01
93.90
81.90
82.67
CIFAR-I00 (10)
66.09
75.64
67.20
54.00
CIFAR-100 (100)
50.88
73.94
54.11
19.07
CUB-200-2011 (10)
42.96
50.94
40.13
26.30
Small Scale
Stanford Dogs (10)
58.79
67.06
57.09
45.35
Stanford Cars (10)
51.06
67.65
48.84
37.09
DTD (10)
38.56
59.82
41.32
26.61
FGVC-Aircraft (10)
39.78
29.55
51.05
42.04
Plowers-102 (10)
64.61
91.39
81.11
41.33
Fruits-360 (100)
100.00
100.00
100.00
100.00
Places-LT ImageNet-LT iNat-LT
34.11
34.62
33.83
37.41
Large Scale
49.10
53.89
42.21
45.66
43.53
48.94
41.96
46.35
Accuracy Gain of CANDLE Over Baseline
100
Head
Hedium
of Samples
5000
of Samples per Class
4000
Accuracy Gain
3000
2000
Avg Gain: -2.37
Avg Gain: 30.62
Avg Gain: 25.43
1000
-50
150
230
300
200
Class Index
Figure 9: Per-class accuracy gain of CANDLE over the baseline on Places-LT. Our method improves the accuracy at the tail classes by + 43%, showing its effectiveness for LT. Additionally, the proposed method also improves the accuracy at the medium classes by +30.62% and only drops by -..77% for the head classes
Accuracy Gain CANDLE Over Baseline
1250
1000
of Samples per
750
Accuracy
500
-25
Avg
Gain: 26.17
Avg Gain: 22.65
250
-50
Class Index
Figure 10: Per-class accuracy gain of CANDLE over the baseline on ImageNet-L.T Out method improves the accuracy at the tail classes by 22.65%, showing its effectiveness for LT. Furthermore, the proposed method also shows gains in the accuracy at the head and medium classes by +10.57% and 126.17% respectively.
average accuracy not because it is only good head classes but mostly because it improves the accuracy of tail and medium classes by large margins.
19

---

Under review as a conference paper at ICLR 2023
Number of Samples per Class
Accuracy Gain of CANDLE Over Baseline
Head
Hedium
Samplees
1000
Accuracy Gain (%)
100
800
50
600
400
-50
Avg Gain: -7.76
Avg Gain: 15.46
Avg Gain: 17.77
200
-100
Class Index
Figure 11: Per-class accuracy gain of CANDLE over the baseline on iNat-LT. Our method improves the accuracy the tail classes by -1,,77%, showing its effectiveness for LT. The medium class ac- curacy is also improved by +15.46%. There is a slight reduction in head class accuracy -7.76%
CANDLE w/ Cosine Cls.
CANDLE w/ Linear Cls.
Dataset
Average
Head
Medium
Tail
Average
Head
Medium
Tail
Caltech101 (10)
55.52
50.99
46.28
22.92
55.31
49.25
46.74
35.54
CIFAR-10 (10)
91.27
94.97
89.00
90.60
91.76
95.50
90.50
91.03
CIFAR-10 (100)
81.57
95.17
80.32
69.63
85.01
93.90
81.90
82.67
CIFAR- (10)
53.92
71.28
55.20
30.83
66.09
75.64
67.20
54.00
CIFAR- 100 (100)
40.11
71.56
36.34
5.59
50.88
73.94
54.11
19.07
CUB- CUB-200-2011 200-2001 (10)
39.13
51.54
39,49
24.76
42.96
50.94
40.13
26.30
Stanford Dogs (10)
56.74
66.53
55.82
40.03
58.79
67.06
57.09
45.35
Stanford Cars (10)
38.88
55.84
32.41
21.11
51.06
67.65
48.84
37.09
DTD (10)
40.85
61.25
38.82
23.21
38.56
59.82
41.32
26.61
FGVC-Aireraft (10)
30.03
25.03
41.36
22.58
39.78
29.55
51.05
42.04
Flowers- 102 (10)
65.98
90.83
73.33
27.33
64.61
91.39
81.11
41.33
Fruits-360 (100)
99.87
100.00
99.63
100.00
100.00
100.00
100.00
100.00
Mean Acc. (%)
57.82
69.58
57.33
39.88
62.07
71.22
63.33
50.09
Table Average, head, medium, and tail class accuracy of CANDLE with either cosine classifiers or linear classifiers on 12 small-scale datasets. Using linear classifiers almost always outperforms using cosine classifiers all average, head, medium, and tail class accuracy except for results on Caltech101 (10).
A.12 RESULTS WITH COSINE CLASSIFIESS FOR CANDLE ON SMALL-SCALE DATASETS
We present results of the proposed method with the cosine classifiers instead of the linear classifier in Tab. 9. As shown in the table, using linear classifiers results in better average accuracy than cosine classifiers for all datasets except Caltech101 (10). Not only is the average accuracy higher for linear classifiers, but all head, medium, and tail class accuracy show clear advantages of linear classifiers over cosine classifiers on all 12 datasets. Taking the mean over the 12 datasets shows +4.25% gain in average accuracy as well as a +10.21% gain tail class accuracy. This is in-line with our hypothesis in Sec. that cosine classifiers may not work as well for binary networks due to the reduced usefulness of weight norms magnitudes in supplementing the limited capacity of binary networks.
1. LIMITATIONS.
We mainly focus on the algorithmic aspects learning binary networks with accurate LT recognition capabilities. Actual deployment of LT capable binary networks to the real-world requires more application-oricn orientted development on top of the proposed algorithmic improvements and is out of scope of this work. Additionally, while usage of multi-resolution inputs incurs little increase
20

---

Under review as a conference paper at ICLR 2023
in training time, an increase in the VRAM peak is observed. Reducing the VRAM peak increase would be an interesting future direction for making the algorithm more efficient.
21


Reviews:
[{"review_id": "z7BmFnLgXN", "reviewer": "Reviewer_xZC3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a binary network for long tailed data. They transfer knowledge from a calibrated teacher network to the binary network, and utilize adversarial learning for learning the hyperparameters. Experimental results demonstrate the effectiveness of their method.", "summary_of_the_review": "I incline to reject this paper based on the current issues. However, I will be happy to change my score if the authors can solve my questions.", "strength_and_weaknesses": "Strength:\nThe authors proposed a bianry network for long tailed data, which is interesting. The presentation is good, and easy to follow. \n\nWeaknesses:\n1. The authors claimed high variance in binary network can influence the performance of the model, and attempted to lower the high variance. Is there theoretical analysis on this? This seems to be claimed only based on some empirical observation. \n\n2. The proposed method utilized the traditional knowledge distillation for learning a binary network, leading to the limited novelty.\n\n3. In Figure 2, the authors claimed \"The classifier weight norms increase at the tail classes, implying high variance\". why?\n\n4. Why learn the balance bwtween two losses can be realized by an adversarial learning manner? It is expected to give more explanations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GmWt2_0nHJ", "reviewer": "Reviewer_F3k8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the task of classifying data with long-tailed distributions of class samples under resource constraints. The resource constraints are addressed by using binary neural networks. The authors propose to use floating point networks pretrained on standard benchmarks and to retrain the classifier layer on the long-tailed data (here called _calibration_). The proposed objective is a linear combination of the loss of the binary network and the retrained floating point network, and the floating point and binary encoders. The weight of this linear combination is learned in an adversarial way, such that the part of the objective with a higher loss gets a higher weight. In addition, a mult-resolution module is added. The experiments indicate high accuracy of the proposed method in comparison to competitors on various datasets for which a long-tail class distribution has been created. ", "summary_of_the_review": "Possibly a good paper that is not acceptable at this point in my view, because of its unclarity.\n\n# After Rebuttal Thoughts\nI see that the authors revised the paper carefully and that they provided more clarity regarding the terminology and methodology. However, in order to assess the revised version, I would have to re-read the whole paper. Unfortunately, I don't have time to do so, hence, I'll decerease my confidence score and give other reviewers the floor.", "strength_and_weaknesses": "# Strengths\n* I like that the weight of the linear combination of losses is optimized\n* Figures and plots are used to illustrate various points\n* Dataset statistics are given in the appendix\n* The considered task of long-tail class distributions under resource constraints is interesting and possibly impactful for many applications\n# Weaknesses\n* The paper is not clear. Many terms are never properly defined, there is a lack of mathematical definitions, big chunks of the method are a black-box to me, I don't understand most Figures and the reasoning is also often unmotivated. \n* The writing is often repetitive in what the authors are going to do without actually explaining it, which makes for a frustrating read\n* The related work just lists related papers without actually pointing out how the existing work builds the fundament for benchmarks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7bk5Ad86ctS", "reviewer": "Reviewer_2EyL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies long-tailed learning for binary networks. This is a novel setting. To address this, this paper proposes a new CANDLE method, which is based on three main strategies: (1) Calibrate and Distill, (2) adversarial learned balancing, (3) multi-resolution training. Promising empirical results demonstrate its effectiveness.", "summary_of_the_review": "Despite a few questions that can be resolved, I like this paper as it explores a new and practical task, i.e., learning binary networks on long-tailed datasets, and provides a few insights. I tend to accept this paper.", "strength_and_weaknesses": "Strengths:\n1. Learning binary networks on long-tailed data is a novel and practical task. Overall, I like this idea.\n2. The analysis is interesting, which shows the differences between learning binary networks and learning float-point networks on long-tailed data.\n3. Empirical results are encouraging. \n\nWeaknesses:\n1. Figure 2 finds that the classifier weight norms increase at the tail classes on both float-point networks and binary networks. However, this is contrary to several classic methods, like decoupling (Kang, ICLR 2020). They find classifier weight norms of float-point networks are positively related to long-tailed class distribution with standard training.  The authors explain that conventional FP methods use SGD and weight decay but binary networks use ADAM with zero weight decay. It makes sense, but it will be better to provide a baseline that binary networks use SGD and weight decay to demonstrate this.\n2. Moreover, it is interesting to know the size of feature space per class. In conventional FP networks, head classes usually have large feature space than tail classes. How about binary networks? \n3. Furthermore, many long-tailed approaches use the cosine classifier since its classifier weights are normalized. It is good to show how the proposed method performs with the cosine classifier.\n4. Why do you use MiSLAS to calibrate the classifier? Efficiency seems not a superiority of MiSLAS.\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7L3XmIZCBNc", "reviewer": "Reviewer_ah6w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents CANDLE, a binary-based network trained using a teacher-student framework in which a learned weight selects the largest loss between two loss terms in the optimization problem. In this way, CANDLE ensures the minimization of the maximum loss at every iteration. The motivation of using a binary network is to ensure a simple but efficient network that can be used in long-tail scenarios with efficiency in mind. Also, the paper adds a multi-resolution training schema in order to boost performance. The paper presents experiments on synthetically long-tailed and natural long-tail dataset demonstrating that the average accuracy improves considerably.", "summary_of_the_review": "Overall, I think the paper lacks clarity, better justifications of their algorithmic design choices, and more informative experiments showing the performance on head, medium-shot, and tail classes. However, I am intrigued about the adversarial learning framework applied to the teacher and student paradigm.", "strength_and_weaknesses": "Strengths:\n1. The motivation of the paper has a practical impact. Tackling long-tail scenarios is an important problem in real applications, and especially, considering the computational efficiency aspect is an important problem.\n2. The idea of introducing a learned weight (\\lambda) that controls the “focus” between the two loss-terms in order to guarantee that the maximum loss at every iteration is minimized is interesting. I think the learning formulation in the teacher-student framework is a new idea.\n3. Overall, the clarity of the intuition and high-level ideas are explained well. However, there are some concerns I have with details and motivation; see the section below.\n\nWeaknesses:\n1. Assumptions are not well justified, and the paper is not self-contained: \n- The main assumption from the paper is that a neural-network-based image classifier suffers from performance issues due to the weight-norm variances. This already assumes that the networks use a linear-based classifier. However, previous work has shown that cosine-based classifiers also improve the performance of image classifiers in the long-tail scenario; see references A and B. In theory, these cosine-based classifiers should be more robust to the weight-norm variance because only angles are the main discriminative statistic.\n- The paper is not self-contained because it lacks a full discussion about the bad impact the weight-norm variance has when learning from long-tailed datasets. The reader has to dig in deeper and read extra to understand this claim. \n- Moreover, the narrative in the introduction projects to the reader that the main cause of the bad performance when learning from long-tailed datasets is the weight-norm variance. However, as shown in previous work (see references A and B for example), the lack of data from tail classes is the main cause of the problem. Thus, over sampling methods also alleviate the bad performance effect when learning from a long-tailed dataset. I think the introduction needs a revision in which it acknowledges previous work in a more structured and inclusive manner of previous work.\n\n2. The paper lacks clarity:\n- It is unclear from the introduction and Section 3.2 how the teacher network is pre-trained on non-LT data. What does this mean? Does it mean the teacher network used a balanced dataset (e.g., CIFAR 100), and then the teacher transfers its knowledge to the student network? I could not find a clear explanation about this.\n- In section 3.1, the paper states that it is believed that the “model is likely to exhibit high [weight-norm] variance as it fits to the few training samples in the tail classes”. I think this statement needs to be backed up with an experiment.\n\n3. Insufficient experiments: In the long-tail context, the mean average accuracy is not very informative. This is because it is unclear if the head classes are improving and thus moving the average accuracy metric up. I think a plot showing accuracy performance per class can be more informative (see Fig. 4 in A). It is unclear if the method favors more head or medium-shot classes. Note that this also applies to the ablation study.\n\n4. While CANDLE can improve efficiency during the inference stage due to the use of binary-based networks, the training of these networks is quite elaborate. In my (humble) opinion, this decreases the practical benefit as it requires training a teacher network first, then training a student network using CANDLE again.\n\nReferences:\n\nA. Kozerawski, et al.. BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed Images. ACCV 2020.\n\nB. Park, et al.. The Majority Can Help the Minority: Context-Rich Minority Oversampling for Long-Tailed Classification. CVPR 2022\n\n\n\n================= Post-Discussion =================\n\nAfter engaging in a discussion with the authors, I will downgrade my recommendation to 3 - Reject, not good enough. The reason for this is the following: the proposed approach is not practical. To me, the proposed approach has a fundamental problem. The proposed approach assumes that a teacher model (i.e., a non-LT model) exists and can be used to train a student model that will effectively deal w/ the long-tail dataset. In a realistic scenario and as an example, if I want to train a classifier using CANDLE and the Google Open Images dataset, which is a long-tailed one, it would be very hard to get a non-LT teacher model. Therefore, I cannot use CANDLE in large scale and real long-tailed dataset. Therefore, I question the approach as a viable solution to the long-tailed problem. Unfortunately, I could not get a satisfactory answer to this concern.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject