Alpha Net: Adaptation with Composition in Classifier Space
Authors: Nadine Chang
Jayanth Koushik
Michael Tarr
Martial Hebert
Yu-Xiong Wang
Venue:  (2021)
Keywords: long-tail recognition
classifier composition

Abstract: Deep learning classification models typically train poorly on classes with small numbers of examples. Motivated by the human ability to solve this task, models have been developed that transfer knowledge from classes with many examples to learn classes with few examples. Critically, the majority of these models transfer knowledge within model feature space. In this work, we demonstrate that transferring knowledge within classifier space is more effective and efficient. Specifically, by linearly combining strong nearest neighbor classifiers along with a weak classifier, we are able to compose a stronger classifier. Uniquely, our model can be implemented on top of any existing classification model that includes a classifier layer. We showcase the success of our approach in the task of long-tailed recognition, whereby the classes with few examples, otherwise known as the tail classes, suffer the most in performance and are the most challenging classes to learn. Using classifier-level knowledge transfer, we are able to drastically improve - by a margin as high as 10.5% - the state-of-the-art performance on the tail categories.

Under review as a conference paper at ICLR 2021
ALPHA NET: ADAPTATION WITH COMPOSITION IN CLASSIFIER SPACE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning classification models typically train poorly on classes with small numbers of examples. Motivated by the human ability to solve this task, models have been developed that transfer knowledge from classes with many examples learn classes with few examples. Critically, the majority of these models transfer knowledge within model feature space. In this work, we demonstrate that transfer- ring knowledge within classifier space is more effective and efficient. Specifically, by linearly combining strong nearest neighbor classifiers along with weak clas- sifier, we are able to compose stronger classifier. Uniquely, our model can be implemented on top of any existing classification model that includes a classifier layer. We showcase the success of our approach in the task of long-tailes recog- nition, whereby the classes with few examples, otherwise known as the "tail" classes, suffer the most in performance and are the most challenging classes learn. Using classifier-level knowledge transfer, we are able to drastically im- prove by margin as high as 10.5% the state-of-the-arr performance on the "tail" categories.
INTRODUCTION
The computer vision field has made rapid progress in the area of object recognition due to several factors: complex architectures, larger compute power, more data, and better learning strategies. However, the standard method to train recognition models on new classes still relies on training using large sets of examples. This dependence on large scale data has made learning from few samples natural challenge. Highlighting this point, new tasks such as low-shot learning and long- tailed learning, have recently become common within computer vision.
Many approaches to learning from small numbers of examples are inspired by human learning. In particular, humans are able to learn new concepts quickly and efficiently over only a few samples. The overarching theory is that humans are able to transfer their knowledge from previous experi- ences to bootstrap their new learning task (Lake et al., 2017; 2015; Gopnik & Sobel, 2000).
Inherent in these remarkable capabilities are two related questions: what knowledge is being trans- ferred and how is this knowledge being transferred? Within computer vision, recent low-shot learn- ing and long-tailed recognition models answer these questions by treating visual "representations" as the knowledge structures that are being transferred. As such, the knowledge transfer methods implemented in these models transfer learned features from known classes learned from large data to the learning of new classes with low data (Liu et al., 2019; Yin et al., 2019). These models exem- plify the broader assumption that, in both human and computer vision, knowledge transfer occurs within model representation and feature space (Lake et al., 2015). In contrast, we claim that previ- ously learned information is more concisely captured in classifier space. This inference is based on the fact that sample representation is unique to that sample, but classifiers are fitted for all the sam- ples in a given class. The success of working within classifier space to improve certain classifiers has been established in several papers (Elhoseiny et al., 2013; Qi et al., 2018), where the models are able to directly predict classifiers from features or create new models entirely by learning other models. Other non-deep learning models use classifiers learnt with abundant data to generate novel classifiers (Aytar & Zisserman, 2011; 2012).

---

Under review as a conference paper at ICLR 2021
Despite these successes, the con- cept of learning within classi- fier space is not common in deep learning models. We sug- gest that transfer learning can, likewise, be implemented in the classifier space of deep learn- ing models. Specifically, we combine known, strong classi- fiers (i.e., learned with large datasets) with weak classifiers (i.e., learned with small datasets) to improve our weak classifiers.
O
Our classifier space method is il- lustrated Figure 1. In this toy example demonstrating method, we are given n classi- fiers C trained with large data and a weak classifier a, which was trained for a class with very few samples. Our goal is to combine the most relevant strong classifiers to adaptively adjust and improve a. Our method implements this approach in the simplest way possible. To select the most effective strong classifiers to combine, we take the target's a nearest neighbor classifiers. Given the appropriate nearest neighbor classifiers, the challenge becomes how to combine these strong classifiers with the target weak classifier so as improve its performance given no further samples.
Figure A classifier space depicting how Alpha Net adaptively adjusts weak classifiers through nearest neighbor compositions.
We address this challenge using what we view as the most natural approach creating new classifier by linearly combining nearest neighbor classifiers and the original weak classifier. Embedded in this solution is the further challenge of choosing strategy for computing the combination co- efficients. We propose learning the coefficients the linear combination through another neural network, which we refer to as "Alpha Net". As compared to many other approaches to learning from small numbers of examples, our methodology has three characteristics:
1.
Our approach can be implemented on top any architecture. This is because the Alpha Net does not need to re-learn representations and only operates within classifier space to im- prove weak classifiers. As a result, our approach is agnostic to the type architecture used to learn the classifiers; it merely provides systematic method for combining classifiers.
2. Our approach demonstrates the importance of combining not only the most relevant classi- fiers but also the original classifier. the absence of the original classifier, any combination of classifiers becomes possible solution, without being constrained by the initial classifier.
3. Our approach creates for every target class a completely different set of linear coefficients for our new classifier composition. In this manner, we are learning our coefficients more adaptive way, which is extremely difficult to achieve through classical methods.
To illustrate the efficacy of our method, we apply it to the task of long-tailed recognition. "Long- tailed" refers a data distribution which there realistic distribution of classes with many examples (head classes) and classes with few examples (tail classes). We compare our Alpha Net method recent state-of-the-art models Kang al. (2020) on long-tailed datasets: ImageNet- LT and P.. Critically, we are able to improve the tail classifiers accuracy by as much as 10.5%.
RELATED WORK
Creating, modifying, and learning model weights are concepts that are seen in many earlier models. In particular, these concepts appear frequently in transfer learning, meta-learning low-shot learning, and long-tailed learning.
Classifier Creation The process creating new classifiers is captured within meta-learning con- cepts such learning-to-leam, transfer learning, and multi-task learning (Thrun & Pratt, 2012;
2

---

Under review as a conference paper at ICLR 2021
Schmidhuber et al., 1997; Pan & Yang. 2009; Caruana, 1997; Santoro et al., 2016). These ap- proaches generalize to novel tasks by learning shared information from a set of related tasks. Many studies find that shared information is embedded within model weights, and, thus, aim to learn struc- ture within learned models to directly modify another network's weights (Schmidhuher, 1987; 1992; 1993; Bertinetto et al., 2016; Ha et al., 2016; Finn et. 2017; Rebufhi et al., 2017; Sinha et al., 2017; Munkhdalai & Yu, 2017). Other studies go even further and instead of modifying networks, they create entirely new networks from training samples only (Socher al., 2013; Lei Ba et al., 2015; Noh et al., 2016). However, our method only combines existing classifiers, without having to create new classifiers or networks from scratch.
Classifier or Feature Composition In various classical approaches, there has been work that learns better embedding spaces for image annotation (Weston et al., 2010) uses classification scores as useful features (Wang et al., 2009). However, they do not attempt to compose classifiers nor do they address the long-tail problem. Within non-deep methods in classic transfer learning, there have been attempts of using and combining SVMs. In Tsochantaridis et al. (2005), SVMs are trained per ob- ject instance and hierarchical structure is required for combination in the datasets of interest. Such a structure is typically not guaranteed nor provided in our long-tail datasets. Additional SVM work uses regularized minimization learn the coefficients necessary to combine patches from other classifiers (Aytar & Zisserman, 2012), While conceptually similar to Alpha Net, our method has an additional advantage of learning the compositional coefficients in more adaptive way. Specifically, different novel classes will have their own sets of alphas, and similar novel classes will naturally have similar sets of alphas. Learning such varying sets of alphas is difficult in previous classical ap- proaches. They either learn a fixed set of alphas for all of the novel classes are forced to introduce more complex group sparsity-like constraints. Finally, in zero-shot learning there exist methods which compose classifiers of known visual concepts to learn completely new classifier (Elhoseiny ct al., 2013; Misra et al., 2017: Lei Ba et al., 2015; Changpinyo et al., 2016). However, such com- position is often guided by additional attribute supervision textual description, which Alpha Net does not depend on.
Learning Transformations Between Models and Classes Other studies have demonstrated differ- ent ways of learning transformations to modify model weights in an attempt learn these trans- formations during stochastic gradient descent (SGD) optimization (Andrychowicz et al., 2016; Ravi & Larochelle, 2017). Additionally, Wang & Hebert (2016) empirically show the existence of a generic nonlinear transformation from small-sample to large-sample models for different types of feature spaces and classifier models. Finally, Du et al. (2017) provide theoretical guarantees on performance when one learns the transformation going from the source function to a related target function. In our work, we are also inferring that our target classifier is a transformation from a set of source classifiers.
Low-shot Learning Meta-learning, transfer learning, and learning-to-learn are frequently applied to the domain of low-shot learning (Fei-Feei et al., 2006; Koch et al., 2015; Lake et al., 2015; Santoro et al. 2016; Wang & Hebert, 2016; Li & Hoiem, 2017; Hariharan & Girshick, 2017; Bromley al., 1994; Koch et al., 2015; Snell et al., 2017; George et al., 2017; Wang et al., 2017; Akata et al., 2015). Many prior studies have attempted to transfer knowledge from tasks with abundant data to completely novel tasks (Vinyals et al., 2016; Ravi & Larochelle, 2017; Bertinetto et al., 2016). However, the explicit nature of low-shot learning consisting of tasks with small fixed samples means that these approaches do not generalize well beyond the arbitrary few tasks. This is indeed a signifi- cant problem as the visual world very clearly involves a wide set of tasks with continuously varying amounts of information.
Long-tailed Learning The restrictions on low-shot learning have directly led to new paradigm referred to as "long-tailed" learning, where data samples are continuously decreasing and the data distribution closely models that of the visual world. Recent work has approached long-tailed recog- nition by ralbanann the samples in different stages of model training (Cao et al., 2019). One other study attempts to transfer features from common classes to rare classes (Liu et al., 2019) or trans- fer intra-class variance (Yin et al., 2019). However, both these approaches to knowledge transfer require complex architectures, such as a specialized attention mechanism and memory models as in Liu et al. (2019). While most studies have largely focused representation space transferability, recent work explores the potential of operating in classifier space (Kang et al., 2020). Results sug- gest that decoupling model representation learning and classifier learning is actually a more efficient
3

---

Under review as a conference paper at ICLR 2021
Alpha Net
EnfF
Fx
Class scores
Alpha Net
En,, - U
Conv Net
Figure 2: An illustration of Alpha Net. "Conv Net" is a stand-in for any fixed pre-trained model with a classification layer. For figure simplicity, the bias is omitted; Vž and oj are denoted as Va and a;
way to approach long-tailed learning. Specifically. normalizing classifiers and adjusting classifiers only with samsampingg strategies are able to achieve good performance. Their success in working only with classifiers supports general concept that combining strong classifiers is natural and direct way to improve upon weak classifiers.
METHODS
Long-tailed recognition is uniquely characterized by its unbalanced, long-tailed distribution of data. Due to the lack of sufficient data in parts of the distribution, otherwise known the "tail", training on the tail classes significantly suffers compared to that of "base" classes that have sufficient data. As result, tail classes typically low accuracy performance, and this remains the dominating challenge of long-tailed recognition. Our method directly addresses this issue, and successfully improves the tail performance by combining classifiers in order to adaptively adjust weak classifiers.
3.1 NOTATION
We first split up our long tail dataset into two broad splits: our "base" classes with number of classes and our "few" F classes. The "base" classes contain classes with many examples, and conversely the "few" classes contain classes with few examples. We denote by N B F the total number of classes.
Additionally, we denote generic sample by and its corresponding label by y. We split the training set into two subsets of pairs (Ji, w:): the subset X Xbose contains samples from the B classes, and the subset X few contains samples from the classes.
Finally, we work with fixed pre-trained model with classifiers W, and biases by. j E (1.....NV N); here, classifiers are the last layers of networks, which map input representations to class scores. Given target class j. and its classifier W, and bias bj. let class j's top k nearest neighbor classifiers and biases be defined as V and p/. (I...., K). Furthermore, we also define the trivial case where the immediate nearest neighbor itself, V W, and p/ by.
3.2
ALPHA NET
Our combination model, Alpha Net, takes in trained classifiers from the B "base" classes and weak classifiers from the "few" classes, in order to learn a new classifier composition for each "few" class. The following section will detail the model and training depicted in Figure 2.
4

---

Under review as a conference paper at ICLR 2021
Architecture In order to create effective individual classifiers, Alpha Net is split up into several sub-modie modules A,, i E (I...., ......,F) ,F). Each A, is responsible for the composition of a new classifier for class i, and consists of 2 fully connected layers with a ReLU in between. Its imput is flattened concatenated vector of VE k E (O...., ....,K), K), where Vi = W, the original classifier for class i. Its (O....,K, K) corresponds to input V
output an a vector, where ag.
Training Alpha Net is trained with all few and an equal number of random samples from Xbase acting as negative samples. For each A,, its input, Ve h E (o,...,K) K) is fixed for the entirety of training. After the inputs are forward through As, we clamp ag (which is responsible for Vg), such that 0 < ag S % a hyper-parameter.
After clamping, given our of and input Vi, k E (O...., K), we define our new classifier and bias for class as
K
U,
Σ a
Vj.
j=0
K
4 = Σ
0z p).
(2)
For each A; we sequentially perform clamping, Eq. 1, and Eq. 2 only once to learn U2 and 4, (I,..., F) for each target class.
It is important note that while working on a long-tailed recognition problem. Thus, when training we must also consider our "base" classifiers that have been sufficient trained. Now, given training image sample and label pair {Ti, ys } and its final feature representation I;, we compute our sample score as
attempting to compose new "few" classifiers,
still
$ = I2 - Uy tty-
f F) b (l...., ......B) B)
(3)
Sib = x; W + tb.
(4)
Finally, we compute softmax cross entropy loss on s and backpropagate throughout Alpha Net and its modules.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets To properly compare our approach to state-of-the-art models as discussed in Kang et al. (2020), we perform all of our experiments using the same two long-tailed benchmark datasets: ImageNet-LT and Places-LT (Liu et al., 2019). Both datasets are sampled from their respective original datasets, ImageNet (Russakovsky et al., 2015) and Places365 (Zhou et al., 2017), such that the new distributions follow a standard long-tailed distribution. ImageNet-LT contains 1000 classes with the number of samples per class ranging from 5 to 4980 images. Places-LT contains 365 classes with the number of samples per class ranging from to 1280 images.
ImageNet-LT and Places-LT originally broken down into three broad splits that indicate the num- ber of samples per class: 1) "many" contains classes with greater than 100 samples; 2) "medium" contains classes with greater than equal to 20 samples but less than equal to 100 samples; 3) "few" contains classes with less than 20 samples, In our experiments we typically combine the "many" and "medium" splits together and refer this set of classes as the "base" split.
Network and Data Setup Alpha Net is designed such that there is an individual alpha sub-module for every target class for which we will recompose its weak classifier. In our experiments, we only adjust the weak classifiers belonging to the "few" split of each dataset. Therefore, we have exactly number of sub-modules for all of our Alpha Nets. Due to the number of sub-modules, as well as to facilitate more efficient computation, we reduce the dimensionality of our input classifiers through PCA. For ImageNet-LT, all classifiers are reduced from 2048 dimensions to 500. Because there a total of 365 classes in Places-LT, all classifiers are reduced from 2048 dimensions to 300. Finally, for alpha sub-modules, we set the dimension of our 2nd fc layer to be equal to our reduced classifier
5

---

Under review as a conference paper at ICLR 2021
Methods
Few
Medium
Many
All
topl
top5
top1
top5
topl
top5
top1
top5
Joint
7.7
37.5
65.9
44.4
NCMF
28.1
45.3
56.6
47.3
t-normalizedi
30.7
46.9
59.1
49.4
cRT*
27.4
57.3
46.2
73.4
61.8
81.8
49.6
74.4
LWS*
30.3
61.5
47.2
73.7
60.2
81.6
49.9
75.1
Alpha cRT
34.5
63.1
44.3
72.5
60.5
81.1
49.2
74.6
Alpha - LWS
40.8
67.6
43.2
72.2
57.6
80.6
48.4
74.8
Table I: ImageNet-LT topl and top5 accuracy for Alpha Net. Methods with f indicate that the
results were directly taken from Kang et al. (2020). Methods with indicate that the results were reproduced and verified to be the same as that reported in Kang et al. (2020).
Tap. inperneeenii
(b) A depiction of the relationship between classwise improvement and the 12 distance to its closest neighbor for Alpha cRT (best few).
(a) Histogram classwise improvement for Alpha cRT (best few).
Figure 3: Class improvements in average top 1 accuracy for classes in "few" test set of ImageNet-LT:
dimension size. That is, for ImageNet-LT Alpha Net, the 2nd fc layer is of dimension 500; for Places-LT Alpha Net, the 2nd fe layer is dimension 300.
Implementation All experiments were run using the PyTorch framework. To be comparable to Kang al. (2020), perform most of experiments using the ResNeXt-50 backbone. Further- more, as our method is designed to run on top of pre-trained models, we use models from Kang al. (2020) in order to be directly comparable their results. All models used for training were downloaded directly from the GitHub repository of Kang et al. (2020). Unless specified otherwise, all experiments are trained with SGD optimizer of 0.9 momentum, learning rate 0.1, batch size 64, weight decay by order of 0 every 20 epochs, and image size of 224 224. All Alpha Nets were trained for 80 epochs. All models reported are selected by cross-validation.
4.2 RESULTS AND ANALYSIS
State-of-the-Art Comparison
We ran Alpha Net using state-of-the-as art models on ImageNet-LT and showcase our improvement. According Kang al. (2020), the best models are: Model with re-trained classifier

---

Under review as a conference paper at ICLR 2021
on
samples, denoted classifier re-training (cRT); 2) Model trained with learnt scalar for weight normalization, denoted as learnable weight scaling (LWS). We directly run Alpha Net on the classifiers taken from their publicly available ResNeXt-50 models cRT and LWS (which we refer to as Alpha cRT and Alpha LWS) and evaluate with the corresponding fixed models' convolutional weights. Alpha cRT and Alpha LWS have alpha sub-module for every class in the "few" split. Our inputs to each alpha sub-module are the top 5 nearest neighbor classifiers with the addition of the original weak classifier for the target class leading For each target class, we take its mean feature from the last layer before "fc" and select nearest neighbor classes from "base" classes that have the most similar class mean features in euclidean space.
total of classifiers as input.
In Table 1, we present our ImageNet models with the best gamma selected (y = 0.7) through cross-validation. As seen in Table 1, our models maintain overall state-of-the-art cRT and LWS performance, while outperforming cRT' and LWS by 7.1 and 10.5 points respectively in the "few" split, the hardest challenge in long-tail recognition. Concisely, we are able to significantly improve on "few" with comparable overall topl accuracy.
We now look at per-class accu- racy; Figure 3a shows our class- wise top 1 improvement com- pared to state-of-the-aar, where each bar indicates separate class in "few". As shown, we observe overall large improvement for each class. Simultancously. in Figure we show a KDE es- timator that was fitted on the improvements in "few", "many", and "medium". We note that the average class decline in "many" and "medium" classes are con- strained to low numbers while a large number of strong classifiers retain original top class perfor- mance.
Molien
Figure 4: Improvements in average Alpha cRT (best few) top 1 accuracy for each class in the test set, grouped as "few", "medium", or "many" classes. KDE estimator was fit for each group shown.
Furthermore, we examine the re- lationship between the improved "few" classes and their nearest neighbor classifiers. In Figure 3b we show each "few" class im- provement and the corresponding euclidean distance of the class" closest neighbor. We observe that the closer the neighbor, the better the average class improvement, and vice versa. We believe this demonstrates the power of composing new classifiers with neighbors that are highly correlated with the target class.
Finally, we present results on Places-LT in Table 2. As only Resnet- models were available for cRT and LWS trained on Places-LT, we present our results on these models. We outperform cRT and LWS by a respective 2.1 and 1.4 in the "few" split.
Exploration of ? We explore the hyperparameter y and its effects on Alpha Net. As Alpha Net learss the coefficients for linearly combining classifiers, the limit of y forcefully restricts the contri- bution of the original weak classifier to the final composition. We set the range of possible from [0.1,..., 0.9), where the higher the 7. the more the original weak classifier weighted. We show our results on all 7 values for ResNeXt-50 cRT using ImageNet-LT in Figure 5. We note that while y is a hyperparameter that affects the results, validaaioo consistently selects ? 0.7 for both ImageNet-L.T and PLaces-LT, with the exception of 7 = 0.6 for Alpha-LWS on Places. More impor- tantly, we consider ry to our 'control' for performance trade-off. observed in Figure 5 if one wants to preserve and improve upon overall results while still improving the weak classifiers, higher y is appropriate. If a higher improvement on weak classifiers is prioritized, a lower y is able to achieve that while obtaining comparable overall performance. Our results reaffirm our assertion
7

---

Under review as a conference paper at ICLR 2021
Methods
Few
Medium
Many
All
topl
top5
top1
top5
topl
top5
top1
top5
Joint
8.2
27.3
45.7
30.2
NCMF
27.3
37.1
40.4
36.4
t-normalizedi
31.8
40.7
37.8
37.9
CRTY
24.9
37.6
42.0
36.7
LWSt
28.6
39.1
40.6
37.6
cRT=
24.3
55.0
36.7
69.5
41.3
72.1
35.9
67.6
LWST
27.7
58.5
38.3
70.1
39.1
72.0
36.5
68.5
Alpha cRT
26.4
59.6
35.1
67.7
40.6
71.5
35.4
67.5
Alpha LWS
29.1
60.7
37.0
68.8
38.6
71.3
36.0
68.1
Table 2: Places-LT topl and top5 accuracy for Alpha Net. Methods with indicate that the sults were directly taken from Kang et al. (2020). Methods with F indicate that the results were reproduced with the models from Kang et al. (2020) but did not achieve the reported results.
dip
Figure 5: Alpha cRT top accuracies on Imagenet-LT with varying max clip, ?, Dotted lines show baseline cRT accuracies.
that while a combination of relevant strong classifiers can improve a weak classifier, a composition without including the original weak classifier leads poor results. Indeed, qualitatively, what Alpha Net is doing is adjusting the original position of the weak classifier towards the appropriate strong classifiers. However, without the weak classifier starting point, any combination of the strong classifiers becomes possible and less effective solution.
5 CONCLUSION
We present a novel model, Alpha Net, which showcases the ability to compose better performing classifiers model space. Specifically, Alpha Net able to linearly combine strong classifiers to adaptively adjust weak classifier. Our method demonstrates the power of working within classifier space and affrrms that least for some vision problems classifier space is more efficient space to work in compared to model feature space. With Alpha Net, are able establish new bench- mark for state-of-the-art performance on the "few" classes within standard long-tailed recognition any classification model suggests
datasets. Finally, we believe that Alpha Net's capability its future potential to improve upon general classification tasks beyond long-tailed recognition.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classifi- cation. IEEE transactions on pattern analysis and machine intelligence, 38(7):1425-1438 2015.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp- 3981-3989, 2016.
Yusuf Aytar and Andrew Zisserman. Tabula rasa: Model transfer for object category detection. In 2011 international conference on computer vision, pp. 2252-2256 2259.. IEEE, 2011.
Yusuf Aytar and Andrew Zisserman. Enhancing exemplar svms using part level transfer regularization. In BMVC, pp. 1-11, 2012.
Luca Bertinetto, João Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning one-shot learners. In Advances in neural information processing systems, pp. 523-531, 2016.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744,
1994.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arcchiga, and Tengyu Ma. Learning imbalanced datasets with label-distribttion-aaaa margin loss. In Advances Neural Information Prucessing Systems, pp. 1565- 1576, 2019.
Rich Caruana. Multitask learming. Machine learning, 28(1):41-75, 1997.
Soravit Changpinyo, Wei-Lum Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recugnition, pp. 5327-5336, 2016.
Simon S Du, Jayanth Koushik, Aarti Singh, and Barnabas Poczos. Hypothesis transfer learning via transfor- mation functions. In Advances in Neural Information Processing Systems 30, pp. 574-584. 2017.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write classifier: Zero-shot learning using purely textual descriptions. Proceedings of the IEEE International Conference on Computer Vision, pp. 2584- 2591, 2013.
Li Fei-FFii Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594 611, 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 2017.
Dileep George, Wolfgang Lehrach, Ken Kansky, Miguel Grearl-Geeeilla.. Christopher Laan, Bhaskara Marthi, generative vision model that trains with high
Xinghua Lou, Zhaoshi Meng. Liu, Huayan Wang, data efficiency and breaks text- based captchas. Science, 358(6368), 2017.
Alison Gopnik and David M Sobel. Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child development, (2:5):12051111 2000.
David Ha, Andrew Dai, and Quoc V Lc. Hypemettorks.. arXiv preprint arXiv: 160.009099000... 2016.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEFE International Conference Computer Vision, pp. 3018-3022 2017.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recog- nition. In ICML. deep learning workshop, volume 2. Lille, 2015.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-leva concept learning through probabilistic program induction. Science, 350(6266): (1166666)13332 1332-1338, 2015.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenhaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.

---

Under review as a conference paper at ICLR 2021
Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4247-4255. 4255, 2015.
Zhizhong Li Derek Hoiem. Learning without forgetting. IEFE transactions on pattern analysis and machine intelligence, 20112)29935-2447, 2017.
Ziwei Liu, Zhonggi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- tailed recognition in an open world. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine red tomato: Composition with context. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1792-1801, 2017.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. Proceedings of machine learning research, 70:2554, 2017.
Hyeonwoo Noh, Paul Hongsuck Seo, and Bohyung Han. Image question answering using convolutional neural network with dynamic parameter prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 30-38, 2016.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 12(10):1345-1359 2009.
Hang Qi. Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of 2018
IEEE conference on computer vision and pattern recognition, pp.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In ICLR, 2017.
Sylvestre-Alvis Alviss Rebufti, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjecv Satheesh, Sean Ma, Zhiheng Huang, An- drej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 1-2112 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memore-dugmented neural networks. arXiv preprint 1605 06065, 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
Jürgen Schmidhuber. A neural network that embeds its own meta-levels. IEEE International Conference on Neural Networks, 407 407-412. IEEE, 1993.
Jürgen Schmidhuher, Jieyu Zhao, and Marco Wiering. Shifting inductive hias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28(1):105-130, 1997.
Abhishek Sinha, Mausoom Sarkar, Aahitagni Mukherjce, and Balaji Krishnamurthy, Introspection: Accelerat- ing neural network training by learning weight evolution. arXiv preprint 0rXiv:1004.04959, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances neural information processing systems. pp. 4077-4087, 2017.
Richard Socher, Milind Ganjoo, Christopher Manning, Andrew Ng. Zero-shot learning through cross-
modal transfer. In Advances in neural information processing systems, pp. 935-943, 2013.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science Business Media, 2012.
Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, and Yasemin Altun. Large margin methods for structured and interdependent output variables. Journal of machine learning research, 6(Scp): 1484, 2005.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing processiin systems, pp. 3630-3638, 2016.
10

---

Under review as a conference paper at ICLR 2021
Gang Wang, Derek Hoiem, and David Forsyth. Learning image similarity from flickr groups using stochastic intersection kernel machines. 2009 IEEE 12th International Conference on Computer Vision, pp. 428- 435. IEEE, 2009.
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for casy small sample learning. In European Conference Computer Vision, 2016.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural
Information Processing Systems, 2017.
Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 81(1):11-35, 2010.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1704504-577 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.
A ALPHA NET WEIGHT SHARING
Alpha Net consists of a sub-module for every new class for which one wishes to compose a new classifier. With every additional new class, Alpha Net's memory and computation time linearly increases. We explore the possibility of weight sharing in Alpha Net address this issue. Note that each alpha block consists of fo layers; as such, we report results Alpha Figure 6, we observe a slight drop in best accuracy as compared to Alpha Net with multiple alpha blocks. This showcases Alpha Net's ability to compose new classifiers without large computation and memory overhead. However, this increase in efficiency comes with cost: by weight sharing, Alpha Net loses the fiexibility to adjust each sub-module
where both layer weights are shared across all alpha blocks. In
maximally benefit that class. As example, some classes may benefit from dynamically changing the inputs or the number of nearest neighhor classifiers that are combined. Thus, the advantage of multiple alpha blocks is class-wise flexibility and versatility in training dynamics.
Myda
Set
03
19
(a) Top accuracy with varying
(b) Top 5 accuracy with varying Y
Figure 6: Evaluation of weight sharing for Alpha Net with varying 7. Alpha cRT accuracies both figures come from the ImageNet-LT "few" test set. "Not shared" denotes Alpha Net trained with separate alpha blocks' weights as used in the main paper. "Shared" denotes Alpha Net with weights shared across all alpha blocks. See the text for discussion.
B VISUALIZATION OF LEARNED ALPHA VALUES
To better understand classifier composition, we show the learned alpha coefficient values for our best Alpha cRT model trained on ImageNet-LT. As seen in Table 3, each set of alphas for each class different. While we observe consistent high alpha leamned for the original classifier, the distribution of the remaining alphas varies across different classes. Of note, we that the immediate nearest neighbor (denoted as "NN1") does always have the highest alpha. In fact, the highest alpha changes within the top k depending the class.
11

---

Under review as a conference paper at ICLR 2021
NN1
NN2
NN3
NN4
NN5
Original
0.2941
0.0600
0,0600
0.0600
0.0600
0.7000
0.1672
0.1326
0.0600
0.0600
0.0600
0.6996
0.0600
0.2995
0.0600
0.0600
0.0600
0.7000
0.0600
0.0600
0.1534
0.1458
0.0600
0,7000
0.1081
0.0600
0.2358
0.0600
0.0600
0.6560
0.0600
0.0600
0.0600
0.1760
0.1238
0.7000
0.0600
0.0600
0.0600
0.2997
0.0600
0.7000
Table 3: Sets of learned alphas from our Alpha cRT model for 10 classes ImageNet-LT. Each row represents a set of alphas for a single class, where nearest neighbor ("NNI") is the top 1 NN's classifier and "original" the original classifier of that class, respectively. Alphas are taken from the model reported in Table trained with y = 0.7 and the same hyperparameters listed Section 4 of the main paper.
C TOP K NEAREST NEIGHBOR STUDY
We explore the effect of the number of top K nearest neighbor classifiers used in Alpha Net. We varied our top K from 12,5,00) and show our results on ImageNet-LLT with Alpha cRT in Figure 7. Interestingly, there is no significant accuracy difference when increase our & from 5 to 10. However, we note large decline in performance when we limit only This decline demonstrates the importance of combining a sufficient number of classifiers: using an insufficient number of classifiers limits the capability to adjust our classifier.
Dai
(a) Top accuracy with varying top
(b) Top 5 accuracy with varying top
Figure 7: Effects of top K. Alpha cRT accuracies in both figures come from the ImageNet-LT test set, grouped as "few", "medium", or "many" classes.
D WHERE TO SAMPLE TOP K CLASSIFIERS?
all experiments shown in Section 4 of the main paper, we sampled our top K nearest neighbors from the "base" classes, which we define as the combination of "many" and "medium" splits. Here, in addition to "base", we explore the effect of sampling top K from alternative splits. Specifically, we show results of using splits "many" and "all". where "all" is the combination of all splits - "many". "medium", and "few". In Figure we observe noticeable accuracy drop when using only "many" split, as compared using "base" split. We attribute this result better nearest neighbor possibilities. Finally, we do not see significant difference between "all" split versus "base" split. "All" split contains the additional classes from the "few" classes, which have weak classifiers. Thus, potential freedom to choose weak classifiers does not provide additional benefits to achieve our main goal of composing new classifier from strong classifiers.
fact that "base" split has twice the number classes select from, thus enabling
12

---

Under review as a conference paper at ICLR 2021
(a) Top 1 accuracy with varying 7
(b) Top accuracy with varying 7--
Figure 8: Evaluation of class splits from which we sample top K classifiers over all gammas. Alpha cRT accuracies in both figures come from the ImageNet-LT "few" test set. The top K classifiers are sampled from "many", "base", or "all" classes.
E NEGATIVE TRAINING SAMPLES
experiments Section the main paper used both equal amount of positive and negative "few" training samples for Alpha Net. Here we remove all negative training samples to explore its effects on accuracy. We showcase our results in Table 4. With only positive training samples, we observe higher "few" compared training with equal amount positive and negative samples. However, we simultancously larger decrease of 3.2% in overall top accuracy. This result illustrates that training with only "few" positive samples will cause Alpha Net to compose a better classifier for "few" classes at higher expense of the overall top 1 performance. In order to ensure good overall top performance, suggest training with negative samples well.
accuracy
Negative Samples
Few
Medium
Many
All
topl
top5
topl
top5
topl
top5
topl
top5
No Yes
40.1
66.9
39.6
70.9
57.1
80.0
46.4
73.8
37.1
65.4
42.0
71.4
59.3
80.5
48.0
74.1
Table Impact of negative training samples for Alpha Net on ImageNet-LT. Models displayed are trained with 0.5 and the same hyperparameters listed in Section 4 of the main paper.
F ADDITIONAL EXPLORATIONS OF Y
Figure 6 of the main paper, we showed the top accuracy for ImageNet-I.T with varying y and top K. Here, we first display the accompanying top accuracy results Figure 9. In addition to varying ? for Alpha cRT, also show the best "ys" effects Alpha LWS in Figure 10. We observe that "'s effects on Alpha cRT and Alpha LWS are similar. We observe that 7 effects Alpha cRT and Alpha LWS similar. The few accuracy consistently peaks at 7 = 0.6 and higher lower y, higher lower accuracy. Lastly, Figure ll and Figure 12 we show the ys' effects for Alpha cRT and LWS respectively on Places-LT. While the improvement in "few" is not as high as that in ImageNet-La LT, we again observe that ys' effects are consistent.
13

---

Under review as a conference paper at ICLR 2021
Tork closve
All
Mane
Mod.
Flw
Max
Figure 9: Top accuracy with varying max clip y. Alpha cRT accuracies come from the ImageNet- LT test set, grouped as "few", "medium", "many". or "all" classes.
cluss
May
Maay
(a) Top accuracy with varying max clip y
(b) Top accuracy with varying max clip 7
Figure 10: Effects of max clip y. Alpha LWS accuracies in both figures come from the ImageNet-LT test set, grouped as "few", "medium", "many", or "all" classes.
Toi Aowe
Eia deoae
(a) Top accuracy with varying max clip 7-
(b) Top 5 accuracy with varying max clip
Figure 11:
Effects of max clip y. Alpha cRT accuracies both figures come from the Places-LT test set, grouped as "few", "medium", "many", or "all" classes.
14

---

Under review as a conference paper at ICLR 2021
(a) Top accuracy with varying 7.
(b) Top 5 accuracy with varying ?
Figure 12: Effects of max clip y. Alpha LWS accuracies in both figures come from the Places-LT test set, grouped as "few", "medium", "many", or "all" classes.
15


Reviews:
[{"review_id": "uFjZaM84PSt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the well-known long-tail classification problem. The argument made here is that most of the existing methods attempt to transfer knowledge in the feature space, which is true. Based on this motivation, the paper proposes a method to do the knowledge transfer in the model space instead. The idea is to apply K-NN method in the model space to pick up a group of strong classifiers trained on the head classes with sufficient training samples available that are closest to a weak classifier in the model space and then a linear combination of the group of strong classifiers and the weak classifier to form a stronger classifier to the tail classes where only few samples are available for training; the linear combination weights are learned from a simple neural network, called Alpha Net. Two datasets artificially truncated from ImageNet and Places, respectively, that were also used in the peer work in the literature, were used to report the evaluations.\n\nThe paper reads very well, except for a few grammatical errors. The presentation is clear and easy to follow.\n\nMy major comments follow. The long-tail learning problem is not new, and the idea of knowledge transfer in the model space is not new either (e.g., King et al 2019 referenced in the paper and in fact that reference was updated in 2020 with better results beating what this paper reported). Consequently, the novelty of this work is rather limited.\n\nFurther, I have a strong reservation in considering the proposed method as a technically sound approach. Conceptually, the idea of combining a group of closest strong classifiers with a weak classifier to form a stronger classifier in the model space is based on the proximity presumption. Regretfully, unlike in the feature space where the proximity presumption is valid in general (unless the feature points are located close to the class boundaries), I am not convinced that the same proximity presumption is valid in the model space, as it is easy to give many counter examples.\n\nRegarding to the experiments, I would like to mention that the authors of the closest competitor, Kang et al 2019 referenced in the paper, have updated their work in arXiv this year with results beating what was reported in the paper. Also they used more datasets to evaluate their method than the two datasets used in the paper. So it is difficult to argue that the proposed method represents the state-of-the-art.\n\nOverall, I am not convinced that the proposed method is technically sound and advances the state-of-the-art literature.\n\n---\n\nI appreciate the authors' effort in responding to my comments. But the arguments in their response appear in conflict. Overall, I am still not convinced by their arguments. So I stay with my original review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MCv6XE-j_T", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "============================================\n\nFinal recommendation after rebuttal\n\nThe authors gave a good rebuttal, and the current version of Fig 5 and the new Fig 6 are making the paper stronger in my opinion.  However, I will stick to my previous rating as:\na) the main weakness, the tradeoff in performance for few vs overall makes the contributions weak, especially since the cRT baseline is as simple as finetuning only the classifers with balanceed sampling. I would expect gains over that\n\nb) Fig 6 further shows the marginal gains over a weight-sharing baseline and makes the basic approach questionable\n\nc) there are no experiments on real-life long tailed datasets - a note here about iNaturalist: the argument the authors make about iNat makes some sense to me, and I want to thank them for replying. But this is exactly why we should test on real long-tail datasets, ie they dont behave as the artificially created ones.\n\n============================================\n\n\n\nThe paper presents an interesting idea, transferring of knowledge between head and tail classes at the classifier level, ie create stronger classifiers for the tail classes by linear combinations of  a tail classes' nearest neighbour classifiers with the current \"weak\" one.  \n\nIn general, although interesting conceptually, the approach doesn't seem to work better than the baselines overall, and the paper doesnt offer any further interesting analysis or insights for long-tailed recognition that would make the performance part be negligible.\n\nStrengths:\n\n*Long-tail recognition and learning from imbalanced data is an interesting, realistic and important problem\n* The authors propose an approach that helps learn better few shot classifiers increasing the performance on the tail classes, trading it off with slightly reducing performance overall\n\nWeaknesses:\n* The authors compare to strong baselines, and do indeed increase performance for tail classes, but in the end they harm overall performance. To get the \"10%\" margin mentioned in the abstract, they also reduce overall performance by 1.5% and head-class performance by around 3%. \n* It seems there is a tradeoff here, where to learn better tail classifiers you hurt head class performance. Would another hyper-parameter setup give the same med/many performance (not harm) and increase few shot performance? Maybe baseline performance (as horizontal lines for few/med/many/all) could be included in the hyperparameter exploration plots.\n* The authors do not present results on any real long-tail dataset, eg iNaturalist, but only on two smaller and artificially created LT datasets (which are the standard, but also many times accompanied with results on a real long-tailed dataset like iNat or faces)\n* The clipping hyperparameter $\\gamma$ seems to control performance a lot, more than the number of classifiers combined (K). Controlling this with a clamping parameter seems heuristic without further discussion.  Although the parameters are ablated,  and seem relatively stable, it is not discussed why clamping is so important and why  \"we consider γ to be our ‘control’ for performance trade-off\". \n* Given that the input to the added 2-layer network Alpha-net is the ordered list of the K closest classifiers and the weak classifier, it is unclear why the authors choose to have one $A_k$ model per class.  What would performance be if there was a single network for all classes? This is a missing baseline that is kind of needed to justify the added computational complexity.\n\nNotes:\n* From the \"three advantages\" enumerated in Sec1, I dont understand how the second is an advantage over other approaches; to me it is more of the way to make this approach work. Same with the third advantage; the coefficients are learned for this method - I dont understand how it is an advantage that they are learned more adaptively when related works dont have those coefficients in the first place.\n* It is unclear what Figure 4 offers and it is hard to comprehend. Some more analysis (or a citation) on what the kernel density estimator is is needed\n* Do all $\\alpha_i$'s for a weak classifier sum to 1 after clamping?", "extra_scores": {}, "extra_text": {}}, {"review_id": "A7nTZc9o3jf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Significance:\nThis article is a useful contribution to transfer learning for tasks where there is not enough data available, showing a modest improvement over the other methods that employ transfer learning in the classifier space.\n\nNovelty:\nThe main contribution of this paper is the improvement of weak classifiers when there is not enough data for a class by combining the weak classifiers with the most relevant strong classifiers. This method finds k closest strong classifiers to the weak classifier and then combines the weak classifier with existing classifiers without creating new classifiers or networks from scratch.\n\nPotential Impact:\nThe approach presented in this paper is well-evaluated in computer vision, but potentially useful in many other settings.\n\nTechnical Quality:\nThe technical content of the paper appears to be correct.\n\nPresentation/Clarity :\nThe paper is generally well-written and structured clearly. While this method is a clear winner on Few classes, it is not performing as well in Medium classes, as shown in Table 1. An explanation about this issue could strengthen the paper.\n\nReproducibility:\nThe paper describes all the algorithms in full detail and provides enough information for an expert reader to reproduce its results. I would suggest the authors release their code on GitHub or other sites to help other researchers reproduce their results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3321g2o3EoD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper focuses on how to transfer knowledge between classes. The authors proposed to transfer classifiers instead of features. The proposed to linearly combine the classifiers from rich classes to construct more robust classifiers or rare classes. The combination weights are predicted from a learned neural network for each rare class. The experimental results on two benchmark datasets outperform some existing methods. \n\n# Strengths\n- While the idea of constructing classifiers by a linear combination of other classifiers has been proposed for several different problems (e.g., zero-shot learning), its application to long-tailed classification seems to be novel.\n\n- The idea is clean and clear. The approach part is well-written.\n\n- The proposed method improves the performance of tail classes.\n\n# Weaknesses\n1. There is no comparison to existing methods (proposed in other problems) that use linearly combine classifiers. Note that, those methods can be applied to long-tailed classification with minimal modifications. For example, for ZSL, due to the lack of visual information of unseen classes, the combination weights can only be estimated from the semantic descriptions. Here, with the W_j for each class, one baseline is thus to replace the semantic descriptions by W_j or even visual features for estimating the combination weights. The authors should compare to those methods.\n\n2. While the approach is clearly written mostly, the need for independent alpha-net modules for each tail class is unclear. Note that, an alpha-net is a two-layer network with lots of parameters, and for tail classes, there are only a few labeled data instances. Learning for each class an alpha-net may be vulnerable to over-fitting. \n\n- There is no equation of the training loss for the alpha-nets. It will be great to provide it. If I understand correctly, the loss is still a softmax loss across all the classes (head and tail), but only the alpha-net parameters (for the tail classes) are being learned.\n\n- The alpha-nets seem to be learned from the data that have been used to train the classifiers in the first stage. As neural networks can usually achieve very low training error, alpha-nets with a one-hot output (i.e., only use V^j_0) may already lead to very high training accuracy. I’m not sure if alpha-nets learn anything meaningful. \n\n- An ablation study on the algorithm design, for example, using a shared alpha-net for all the classes, should be included.\n\n3. The related work and experimental comparison are insufficient. Only “one” paper has been compared in Table 1 and Table 2. There have been many papers published in CVPR 2020 and ECCV 2020. The authors, however, cited NO papers published in 2020.\n\n4. Can the authors provide more discussions on why the performance at medium, many, and all classes drop in comparison to the baselines? For now, it seems that the performance improvement comes simply by trading the prediction/adjusting the classifier strengths among classes: for example, increasing the classifier “norms” of tail classes.\n \n# Minor\n- The figures and captions can be improved. Specifically, Figures 1 and 2 are not self-contended: it is hard to understand the figures without looking at the main text.\n\n# Justifications\nWhile the proposed idea seems novel for long-tailed classification, the paper lacks comparisons to existing methods and comparisons to similar algorithms proposed in other problems (with minimal changes). There is no ablation study on why we need an alpha-net module for each class. There is no overall performance gain, making it hard to tell if alpha-nets really improve classifiers or simply trading predictions/adjusting the classifier strengths among classes. I thus give a score of 3.\n\n----------------------------- Post rebuttal -----------------------------\n\nI read the author's rebuttal and I greatly appreciate their efforts. The authors have done many more experiments and I would like the authors to incorporate them into their manuscript and modify their manuscript, even methods, accordingly. I think these new materials can greatly strengthen the paper.\n\n1) It seems that ZSL with the original classifier involved is quite strong (this could not happen in ZSL as there is no original classifier for the unseen classes). I would suggest that the authors further investigate this for a detailed comparison. These methods may even simplify the authors' methods, and a connection to ZSL can strengthen the paper. For instance, Changpinyo et al., 2016) showed that their method can outperform [1] and it will be interesting to have some further comparison.\n\n2) It's nice that the authors compare the shared and non-shared alpha net. I still have doubt that why non-shared alpha net won't over-fit given that there are only a few labeled data instances. A shared alpha net might be more suitable for robustness.\n\n3) There is one difference to Kang's method. Kang's first stage stopped earlier so tailed classifiers have not covered. Did the author do the same thing?\n\n4) One method that can simply trade-off the accuracy is Kang's method. I think you can tune their hyperparameter to get a higher tail accuracy. Now the question will be, what will their head accuracy be? Without having a more ground comparison among methods, my question still remains unsolved.\n\n5) Besides ImageNet-LT and PlaceNet-LT, there are several CVPR/ECCV papers that outperform Kang's paper on CIFAR, iNaturalist but do not report on these two datasets.\n\nI have increased the score to 4, but I think the paper needs significant work to incorporate my comments as well as other reviewers' comments to be ready for being published.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to create models that address tail classes by computing a linear combination over models (concatenated weight vectors). Reviewers had grave concerns about the technical contribution, including justification of linear averaging of non-linear models, and about the experimental results, which improve on tail classes but hurt overall performance. As a result, the paper cannot be accepted to ICLR. 
Decision: Reject