Rethinking Curriculum Learning With Incremental Labels And Adaptive Compensation
Authors: Madan Ravi Ganesh
Jason J. Corso
Venue:  (2020)
Keywords: Curriculum Learning
Incremental Label Learning
Label Smoothing
Deep Learning

Abstract: Like humans, deep networks learn better when samples are organized and introduced in a meaningful order or curriculum. While conventional approaches to curriculum learning emphasize the difficulty of samples as the core incremental strategy, it forces networks to learn from small subsets of data while introducing pre-computation overheads. In this work, we propose Learning with Incremental Labels and Adaptive Compensation (LILAC), which introduces a novel approach to curriculum learning. LILAC emphasizes incrementally learning labels instead of incrementally learning difficult samples. It works in two distinct phases: first, in the incremental label introduction phase, we unmask ground-truth labels in fixed increments during training, to improve the starting point from which networks learn. In the adaptive compensation phase, we compensate for failed predictions by adaptively altering the target vector to a smoother distribution. We evaluate LILAC against the closest comparable methods in batch and curriculum learning and label smoothing, across three standard image benchmarks, CIFAR-10, CIFAR-100, and STL-10. We show that our method outperforms batch learning with higher mean recognition accuracy as well as lower standard deviation in performance consistently across all benchmarks. We further extend LILAC to state-of-the-art performance across CIFAR-10 using simple data augmentation while exhibiting label order invariance among other important properties.

Under review as a conference paper at ICLR 2020
RETHINKING CURRICULUM LEARNING WITH INCRE- MENTAL LABELS AND ADAPTIVE COMPENSATION
Anonymous authors Paper under double-blind review
ABSTRACT
Like humans, deep networks learn better when samples are organized and intro- duced in a meaningful order or curriculum (Weinshall et al. (2018). While con- ventional approaches to curriculum learning emphasize the difficulty of samples as the core incremental strategy, it forces networks to learn from small subsets of data while introducing pre-computation overheads. In this work, we propose Learning with Incremental Labels and Adaptive Compensation (LILAC), which takes a novel approach to curriculum learning. LILAC emphasizes incrementally learning labels instead of incrementally learning difficult samples. It works in two distinct phases: first, in the incremental label introduction phase, we recursively reveal ground-truth labels in small installments while using fake label for the remaining data. In the adaptive compensation phase, we compensate for failed predictions by adaptively altering the target vector to smoother distribution. We evaluate LILAC against the closest comparable methods in batch and curriculum learning and label smoothing, across three standard image benchmarks, CIFAR- 10, CIFAR- 100, and STL-10. We show that our method outperforms batch learn- ing with higher mean recognition accuracy as well as lower standard deviation in performance consistently across all benchmarks. We further extend LILAC to show the highest performance on CIFAR-10 for methods using simple data aug- mentation while exhibiting label-order invariance among other properties.
INTRODUCTION
Deep networks have seen rich applications high-dimensional problems characterized by a large number of labels and a high volume of samples. However, successfully training deep networks to solve problems under such conditions is mystifyingly hard (Erhan et al.] (2009); [Larchhlle et al.) (2007). The go-to solution in most cases is Stochastic Gradient Descent with mini-batches (simple batch learning) and its derivatives. While offering a standardized solution, simple batch learning often fails to find solutions that are simultaneously stable, highly generalizable and scalable to large systems (Das et a (2016); Keskar et al.) (016);; Goyal et al. (2017): You et al. (2017)). This is a constructed. For example, the uniform prior assumption over
by-product of how mini-batches datasets emphasizes equal contributions from each data point regardless of the underlying distribu- tion; small batch sizes help achieve more generalizable solutions, but do not scale as well to vast computational resources as large mini-batches. It is hard to construct a solution that is a perfect compromise between all cases.
Two lines of work, curriculum learning and label smoothing, offer alternative strategies to improve learning in deep networks. Curriculum learning, inspired by strategies used for humans (Skinner (1958); Avrahami et al. (1997)). works by gradually increasing the conceptual difficulty of samples used train deep networks (Bengio et al.] (2009); Florensa et al. (2017): Graves et al. (2017). This has been shown to improve performance on corrupted (Jiang et al. (217)) and small datasets (Fan ct al. (2018). More recently, deep networks have been used to categorize samples (Weinshall et al. (2018) ) and variations on the pace with which these samples were shown to deep networks were analyzed in-depth (Hacohen & Weinshall (019)). To the best of our knowledge, previous works assumed that samples cover broad spectrum of difficulty and hence need to be categorized and presented in a specific order. This introduces computational overheads e.g. pre-computing the relative difficulty of samples, and also reduces the effective amount of data from which model can

---

Under review as a conference paper at ICLR 2020
learn in early epochs. Further, curriculum learning approaches have not been shown to compete with simple training strategies at the top end of performance in image benchmarks.
A complementary approach to obtaining generalizable solutions is to avoid over-fitting or getting stuck in local minima. In this regard, label smoothing offers an important solution that is invariant to the underlying architecture. Early works like et al. (016) replace ground-truth labels with noise while ReeleRReeeeee et al. (2014) uses other models' outputs to prevent over-fitting. This idea was extended in Bagherinezhad al. (2018) an iterative method which uses logits obtained from previously trained versions of the same deep network. While MMyato et al. (2015) use local distributional smoothness, based on the robustness of a model's ' distribution around a data point, to regularize outcomes, Pereyra et al. (2017) penalized highly confident outputs directly. Closest in spirit to our work is the label smoothing method defined in SZzeedy et al. (2016), which offers an alternative target distribution for all training samples with no extra data augmentation. In general, label smoothing is applied to all examples regardless of how affects the network's understanding of them. Further, in methods which use other models to provide logits/labels, often the parent network used to provide those labels is trained using an alternate objective function or needs to be fully re-trained on the current dataset, both of which introduce additional computation.
In this work, we propose LILAC, Learning with Incremental Labels and Adaptive Compensation, which emphasizes a label-based curriculum and adaptive compensation, to improve upon previous methods and obtain highly accurate and stable solutions. LILAC is conceived as a method to learn strong embeddings by using the recursive training strategy of incremental learning alongside the data as hard negative examples. It works in two key phases, 1)
use of gulabelled/wrongly-abbee incremental label introduction and 2) adaptive compensation.
In the first phase, we incrementally introduce groups of labels in the training process. Data, corre- sponding to labels not yet introduced the model, use a single fake label selected from within the dataset. Once network has been trained for a fixed number of epochs with this setup, an additional set of ground-truth labels is introduced to the network and the training process continues. recur- sively revealing labels, LILAC allows the model sufficient time to develop strong understanding of each class by contrasting against large and diverse set of negative examples.
Once all ground- truth labels are revealed the adaptive compensation phase of training is initiated. This phase mirrors conventional batch learning, except we adaptively replace the target one-hot vec- tor of incorrectly classified samples with softer distribution. Thus, we avoid adjusting labels across the entire dataset, like previous methods, while elevating the stability and average performance of the model. Further, instead of being pre-computed by an alternative model, these softer distributions are generated on-the-fly from the outputs of the model being trained. We apply LILAC to three standard image benchmarks and compare its performance to the strongest known baselines.
While incremental and continual learning work on evolving data distributions with the addition of memory constraints ((Rebuffi et al. 2017 Castro et al. 2018) and derivative works), knowledge distillation ((chwwarz et al. [2018] Rolnick et al. 2018) and similar works) or other requirements, this work is departure into using negative mining and focused training improve learning on fully available dataset. In incremental/continual learning works, often the amount of data used to retrain the network is small compared to the original dataset while LILAC we fully use the entire dataset, distinguished by Seen and Unseen labels. Thus, it avoids data deficient learning. Further, works like Bucher et al. (2016): [i et al. (2013): Wang & Gupta (2015) emphasize the importance of hard negative mining, both in size and diversity, in improving learning. Although the original formulation of negative mining was based on imbalanced data, recent object detection works have highlighted its importance in contrasting and improving learning in neural networks.
To summarize, our main contributions LILAC are as follows,
we introduce a new take on curriculum learning by incrementally learning labels as op- posed to samples.
our method adaptively compensates incorrectly labelled samples by softening their target distribution which improves performance and removes external computational overheads, we improve average recognition accuracy and decrease the standard deviation of perfor- mance across several image classification benchmarks compared to batch learning, a prop- erty not shared by other curriculum learning and label smoothing methods.
2

---

Under review as a conference paper at ICLR 2020
2 LILAC
In LILAC, our main focus is to induce better learning in deep networks. Instead of the conventional curriculum learning approach of ranking samples, we consider all samples equally beneficial. Early on, we foeus on learning labels fixed increments (Section 2.T). Once the network has had a chance to learn all the labels, we shift to regularizing the network to prevent over-fitting by providing softer distribution as the target vector for previously misclassified samples (Section 2.2). An overview of the entire algorithm discussed is available in the appendix Algorithm
2.1 INCREMENTAL LABEL INTRODUCTION PHASE
In the incremental phase, we initially replace the ground-trute sound-truth labels of several class using a constant held-out label. Gradually, over the course of several fixed intervals of training reveal the true label. Within a fixed interval of training, we keep constant two sets of data, "Seen", whose ground- truth labels are known and "Unseen", whose labels are replaced by fake value. When training,
Full Dataset [ 44)
Virtual Data Partition in LILAC ( (b=1)
Seen (b) Uses Ground-truth labels
Unseen ( ( b)
Uses a labol
(a) Data setup in LILAC
Incremental Step
Incremental Step 2
Incremental Step
(Final Incremental Step)
(b) Evolution of data partition over each incremental step, m 1
Legend
(Data, Label 1)
(Data, Label 2)
(Data, Label 3) (Data, Label 4) (Data, Forced Label 4)
Figure 1: (Top) In LILAC, data is virtually partitioned into either S Seen or U Unseen. Data under use their ground-truth labels while data in U use a designated fixed unseen label. (Bottom) Illustration of the evolution of data partitions in the incremental label introduction phase for four label dataset. In the first incremental step, only one label used for training while the remaining data use label 4. A short period of training is performed with this fixed setup, where data from is uniformly sampled to match the number of samples from S, every mini-batch. The final incremental step depicted is equivalent batch learning since all the labels are available to the network. Once all the ground labels are revealed we begin the adaptive compensation phase described in Sec. 2.2

---

Under review as a conference paper at ICLR 2020
mini-batches are uniformly sampled from the entire training set, but the instances from "Unseen" classes use the held-out label. By the end of the final interval, we reveal all ground-truth labels.
We now describe the incremental phase in more detail. At the beginning of the incremental label introduction phase, we virtually partition data into two mutually exclusive sets, Seen and V : Unseen, as shown in Fig. [ Data samples in S use their ground-truth labels target values while those in V use a designated unseen label, which is held constant throughout the entire training process. LILAC assumes a random ordering of labels, Or(M), where M denotes the total number of labels in the dataset. Within this ordering, the number of labels and corresponding data initially placed in S is defined by the variable The remaining labels. M b, are initially placed in U and incrementally revealed in intervals of m labels, hyper-parametee defined by the user.
Training the incremental phase happens at fixed intervals of E epochs each. Within a fixed inter- val, the virtual data partition is held constant. Every mini-batch of data is sampled uniformly from the entire original dataset and within each mini-batch, labels are obtained based on their placement in S or V. Then the number of samples from U is reduced augmented, using a uniform prior, to match the number samples from S. This is done to ensure no unfair skew in predictions towards V since all data points use the same designated label. Finally, the curated mini-batches of data are used to train the neural network. At the end of each fixed interval, we reveal another set of m ground- truth labels and move samples of those classes from V to S after which the entire data curation and training process is repeated for the next interval.
2.2 ADAPTIVE COMPENSATION
Once all the ground-truth labels are available to the deep network, we begin the adaptive compen- sation phase of training. The main idea behind adaptive compensation is, if the network is unable to correctly predict a sample's ' label even after allowing sufficient training time, then alter the target vector to a less peaked distribution. Compared to learning one-hot vectors, this softer distribution can more easily be learned by the network. Unlike prior methods we adaptively modify the target vector only for incorrectly classified samples on-the-fly.
In this phase, the network is trained for small number of epochs using standard batch learning. Let T be the total number of training epochs in the incremental phase and batch learning. During the adaptive compensation phase, we start at epoch e, where epoch e, predictions from the model at - 1 are used to determine the final target vector used in the objective function; specifically, we soften the target vector for an instance iff was misclassified by the model at the end of epoch e 1. The final target vector for the ith instance at epoch e, le,i, is computed based on the model Φ using Equation]] [
T. For a mini- of samples in
M-T yo, + ( arg max max((a- 10.- # y² 8
te
otherwise
Here, (#,, ys) denote a training sample and its corresponding ground-trute -ruth label for sample index while 0g. represents the corresponding one-hot vector. 1l is vector of M dimensions with all entries as and is a scaling
3 EXPERIMENTS
Datasets We use three datasets, CIFAR-IO, CIFAR- (Krizhevsky al. (209g) and STL- 10 (Coates et al. (211)), to evaluate our method and validate our claims. CIFAR- 10 and CIFAR-100 are 10 and 100 class variants of the popular image benchmark CIFAR. Each of these contains 50,000 images in the training set and 10,000 images in the testing set. STL-10 is a 10 class subset of Ima- geNet with 500 and 800 samples per class for training and testing subsets, respectively.
The common metric used to evaluate the performance of all the learning algorithms is average recognition accuracy(%) and standard deviation across 5 trials. We also report consis- tency, which is a binary metric that indicates whether the training strategy results in higher average performance and lower standard deviation compared to standard batch learning across all datasets.
Metrics
4

---

Under review as a conference paper at ICLR 2020
Table 1: Performance comparison across all comparable methods and datasets. Under similar tups, without any extra overheads in computation, LILAC consistently achieves higher accuracy and lower std. dev. than batch learning across all compared datasets, which is not the case for any other baseline. The highlighted performances represent the best case scenario which indicate higher average performance combined with lower standard deviation compared batch learning.
Performance (%)
Training
Consistency
CIFAR 10
CIFAR 100
STL 10
Batch Learning
95.28 + 0.141
78.51 + 0.145
73.68 + 0.764
N/A
Fixed Curriculum
95.32 + 0.099
77.36 + 0.412
72.13 + 0.953
X
Label Smoothing
95.39 + 0.050
79.05 0.216
72.26 + 0.638
X
Dynamic Batch Size (DBS)
95.32 + 0.022
78.98 + 0.206
73,20 + 0.656
X
Random Augmentation (RA)
95.22 + 0.060
72.49 = 0.467
73.63 = 0.387
X
LILAC w/o AC (ours)
95.26 + 0.110
78.44 0.227
73.53 + 0.331
X
LS LILAC (ours)
95.39 + 0.088
79.07 + 0.171
73.80 + 0.714
X
LILAC (ours)
95.35 + 0.098
78.73 + 0.127
74.35 + 0.568
Experimental Setup For CIFAR-10/100, we use ResNet18 (He al. 2016)) as the architectural backbone for all methods; for STL-10, we use ResNet34. In each interval of LILAC's incremental phase, we train the model for 10 epochs each for CIFAR-10/100, and 5 epochs each for STL-10. During these incremental steps, we use a learning rate of 0.1, 0.01 and 0.1 for CIFAR-10, CIFAR- 100, and STL-10 respectively. The standard batch learning settings used across all datasets are listed in the appendix. These settings reflect the setup used in LILAC once the incremental portion of training is complete and the algorithm moves into adaptive compensation phase. Within this phase epochs 175, 525 and 120 used as thresholds (epoch 7)) for CIFAR-10, 100 and STL-10 respectively.
Baselines
Stochastic gradient descent with mini-batches is the baseline against which all methods are com- pared.
Curriculum learning (Bengio et al. 2009) forms a family of related works which aim help models learn faster and optimize to better minimum. Following the methodology proposed in this work we artificially create a subset of the dataset called "Simple",, by selecting data that is within value of 1.1 predicted by a linear one-vs-all SVR model trained to regress to the ground-truth label. The deep network is trained on the "Simple" dataset for fixed period time that mirrors the total number of epochs of the incremental phase of LILAC after which the entire dataset used to train the network.
Label Smoothing (Szegedy et al., 2016) the closest relevant work to use label smoothing as form of regularization without extra data augmentation. This non-invasive baseline is used measure the importance of regularization and for its ability to boost performance.
Dynamic Batch Size (DBS) is custom baseline used highlight the importance of variable batch size in training neural networks. DBS randomly copies data available within a mini-batch to mimic variable batch size, Further, all ground- -oruun labels are available this model throughout the training process.
Random Augmentation (RA) is a custom baseline used to highlight the importance of virtual data partitions LILAC. Its implementation closely follows LILAC but excludes the adaptive compensation phase. The main difference between LILAC and RA is that RA uses data from one randomly chosen class, U, within mini-batch while data from all classes in V is used in LIIAA to equalize the number of samples from S and U.
3.1 STANDARDIZED COMPARISON RESULTS
Table I clearly illustrates improvements in average recognition accuracy, decrease in standard de- viation and consistency when compared to batch learning. While certain setups have the highest

---

Under review as a conference paper at ICLR 2020
Table 2:
Comparison of LILAC'ss performance against top performing algorithms on CIFAR-10 with standard pre-processing (random crop ffip). Our method easily outperforms both the base shake-drop network (Yamada al. (2018) as well as previous methods.
Method
CIFAR-10
Wide Residual Networks (aaoruykoo & Komodakis
2016
96.11
Multilevel Residual Networks (Zhang al. 2017)
96.23
Fractional Max-pooling (Graham 2014)
96.53
Densely Connected Convolutional Networks (Huang al. 2017
96.54
Drop-Activation (Liang et al.
2018
96.55
Shake-Drop (Yaaada al. 2018)
96.59
Shake-Drop + LILAC (ours)
96.79
Table 3: Impact of different phases on the final performance of LILAC. Rows 1 and 2 compare batch learning to LILAC without adaptive compensation while rows 3 and 4 highlight the impact of adding adaptive compensation. While incremental label introduction on its own is not a stand- out performer, adding adaptive compensation to it improves performance beyond standard batch learning. The highlighted values indicate higher average performance combined with lower standard deviation compared to batch learning.
Performance (%)
Training
CIFAR 10
CIFAR 100
STL 10
Batch
95.28 + 0.141
78.51 + 0.145
73.68 + 0.764
LILAC w/o AC (ours)
95.26 0.110
78.44 + 0.227
73.53 0.331
Batch + AC
95.38 0.136
78.54 + 0.175
73.32 0.581
LILAC (ours)
95.35 + 0.098
78.73 + 0.127
74.35 + 0.568
performance on specific datasets (e.g.. Label Smoothing on CIFAR-10/100), they are not consistent across all datasets and do not find more stable solutions than LILAC (std. of 0.216 compared to 0. from LILAC) LILAC is able to achieve superior performance without unnecessary overheads such computing sample difficulty irreversibly altering the ground-truth distribution across all samples.
key takeaway from DBS the relative drop standard deviation combined with higher average performance when compared to baselines like fixed curriculum and label smoothing. RA serves to highlight the importance of harvesting data from all classes U simultaneously. for "negative" samples. The variety of data to learn from provides a boost in performance and standard deviation across the board in LILAC w/o AC as opposed to RA. DBS and RA underline the importance of variable batch size and data partitioning in the makeup of LILAC.
We further extend LILAC to train the base pyramidnet with shake-drop regularization (p = 1.0) Ya- mada et al. (2018)). From Table we clearly see that LII.AC can be extended to provide the highest performance on CIFAR-10 given a standard preprocessing setup. To provide a fair comparison we highlight top performing methods with standard preprocessing setups that avoid partial inputs (at the node or image level) since LILAC was developed with fully available inputs in mind. Across all these learning schemes, LILAC is the only to consistently increase classification accuracy and decrease the standard deviation across all datasets compared to batch learning.
3.2 ABLATION: BREAKDOWN OF LILAC'S PHASES
Fig. 2 illustrates the evolution of the embedding across the span the incremental phase. This space has more degrees of separation when compared to equivalent epoch of training with batch learning where all the labels are available. 3 provides a breakdown of the contribution each phase of LILAC and how they combine elevate the final performance. Here, in LILAC w/o AC we replace the entire AC phase with simple batch learning while in Batch + AC we include adaptive compensation with adjusted thresholds. The first half of this table compares the impact of incre-

---

Under review as a conference paper at ICLR 2020
CI
CIFAR-100
STL-10
(a) Evolution of representation space after 1/4, 1/2, 3/4 and full span of incremental phase
(b) An epoch of batch learning
Figure 2: Side-by-side comparison between the representation spaces learned by LILAC and batch learning. Through the entire incremental label introduction phase, the representation space evolves to being more well spaced. Images in column and 5 show comparable points in training space when all labels are available to the deep network being trained. These images support our claim that the deep network starts at better initialization than standard batch learning, whose effect is carried throughout the training process.
mentally introducing labels to a deep network against standard batch learning. We clearly observe that performances across Rows 1 and fall within the indicated standard deviation of each other. However, from Fig. 2 we know that LILAC start from a qualitatively better solution. Combining lengthy secondary batch learning phase erodes
these results, conclude that the emphasis overall performance.
The second half of Table shows the impact of adding adaptive compensation batch learning and LILAC. When added to standard batch learning there isn't a clear and conclusive indicator of improvement
all benchmarks in both average performance and standard deviation. However, in combination with the incremental label introduction phase of LILAC, adaptive compensation im- proves average performance as well as decreases standard deviation, indicating an improved stability and consistency. Given the similarity in learning between the batch setup and LILAC, when all la- bels have been introduced, we show that the embedding space learned by incrementally introducing labels (Fig. 2 is distinct from standard batch learning and is more amenable to AC
3.3 PROPERTIES OF LILAC
Through previous experiments we have established the general applicability of LILAC while con- trasting its contributions to that of standard batch learning. In this section we dive deeper to reveal some characteristics of LILAC that further supplement the claim general applicability. Specifi- cally, we characterize the impact of label ordering, smoothness of alternate target vector distribution and injection of larger groups of labels in the incremental phase.
Ordering of Labels Throughout the standard experiments, we assume labels are used the as- cending order of value. When this is modified to random ordering in ascending order of diff-

---

Under review as a conference paper at ICLR 2020
Table 4: (Top) This table illustrates the impact of using random label order vs. ascending vs. cending difficulty on LILAC w/o AC). LILAC doesn't t provide any explicit pattern in the outcomes. (Middle) Varying the smoothness of the alternate target vector causes slight variations in perfor- mance. (Bottom) Introducing multiple labels per incremental learning interval hurts performance. The learned embedding space is different, which leads diverging performances. The introduction of lower number of label allows for thorough learning and improved final performance in LILAC. Highlighted numbers are peak average performance values.
Performance (%)
Property
Training
CIFAR-10
CIFAR-100
STL-10
Label Order: Rnd.
95.26 + 0.065
78.27 + 0.122
73.74 + 0.846
LILAC w/o AC
Label Order: Asc.
95.26 + 0.110
78.44 = 0.227
73.53 0.331
Label Order: Asc. Difficulty
95.42 + 0.089
78.23 + 0.084
73.31 + 1.117
1.0
95.25 + 0.111
78.33 + 0.178
73.68 + 0.815
c 0.9
95.31 + 0.073
78.46 0.211
73.26 = 1.285
e 0.8
95.25 + 0.088
78.58 0.364
73.47 0.734
e 0.7
LILAC
95.25 0.129
78.71 + 0.256
73.27 + 0.179
0.6
95.33 + 0.073
78.38 0.257
= 73.71 0.421
E = 0,5
95.35 0.098
78.73 + 0.127
74.35 = 0.568
0.4
95.30 + 0.127
78.62 + 0.208
73.38 + 1.296
Label Group (m)):
95.35 + 0.098
78.73 = 0.127
74.35 = 0.568
Label Group (m) 2 (5)
LILAC
95.22 + 0.175
78.73 = 0.215
72.95 = 0.372
Label Group (m) 3 (10)
95.29 + 0.189
78.47 + 0.275
73.61 + 0.771
culty, results from Table suggest that there is no explicit benefit or pattern. Other than the extra impact of continually fluctuating label orders across trials, there isn't a large gap in performance. Thus, we claim LILAC is relatively invariant to the order of label introduction.
Smoothness of Target Vector in Adaptive Compensation During adaptive compensation, e 0.5 is used in the alternate target vector for samples with failed predictions throughout all exper- iments Sections 3.T and 3.2 When extended to a variety of values, we observe that most variations of the peak performance still fall within the standard deviation range for each dataset. However, the peak average performance values usually occur between 0.7 to 0.5.
Injection of Label Groups While LILAC was designed to allow the introduction of multiple la- bels in a single incremental step, throughout the experiments in Sections 3.1 andddd333. 3.2 1 label was introduced per step allow thorough learning while eliminating the chance of conflicting decision boundaries from multiple labels. Revealing multiple labels instead of I label per incremental step has negative impact on the overall performance of the model. Table 4 shows that adding large groups of labels force lower performance, which is in line with our hypothesis that revealing fewer labels per incremental step makes the embedding space more amenable to adaptive compensation.
4 CONCLUSION
In this work, we proposed LILAC which rethinks curriculum learning based on incrementally learn- ing labels instead of samples. This approach helps kick-start the learning process from a substan- tially better starting point while making the learned embedding space amenable to adaptive negative logit compensation. Both these techniques combine well in LILAC to show the highest performance on CIFAR- 10 for simple data augmentations while casily outperforming batch and curriculum learn- ing and label smoothing potential of this setup is to extend this setup to include a confidence measure on the predictions of network so that it can handle the effects of dropout partial inputs. In further expanding LILAC's ability to handle partial inputs, we aim to explore its effect on standard incremental learning (mem- ory constrained) while also extending applicability to more complex neural network architectures.
comparable network architectures. The next step in unlocking the full
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Judith Avrahami, Yaakov Kareev, Yonatan Bogot, Ruth Caspi, Salomka Dunaevsky, and Sharon Lerner. Teaching by examples: Implications for the process of category acquisition. The Quarterly Journal of Experimental Psychology Section A, 50(3):586-6066 1997.
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint arXiv: 000..800500666 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 41-48. ACM, 2009.
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Hard negative mining for metric learning based zero-shot classification. In Gang Hua and Hervé Jégou (eds.), Computer Vision ECCV 2016 Workshops, pp. 524-531, Cham, 2016. Springer International Publishing. ISBN 978-3-319- 49409-8.
Francisco M Castro, Manuel Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 233- 2018.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223, 223, 2011.
Dipankar Das. Sasikanth Avancha, Dheevatsa Mudigere, Karthikeyan Vaidynathan, Srinivas Srid- haran, Dhiraj Kalamkar, Bharat Kaul, and Pradeep Dubey. Distributed deep learning using syn- chronous stochastic gradient descent. arXiv preprint arXiv: 16X.v:660.0670000 0600006709, 2016.
Dumitru Erhan, Pitrre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect unsupervised pre-training. In Artificial Intelligence and Statistics, pp. 153-160, 2009.
Yang Fan, Fei Tian, Tao Qin, Xiang- Yang Li, and Yin Liu. Learning to teach. arXiv preprint arXiv:1805.03644. 2018.
Xiang-Yang
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. arXiv preprint arXiv: 2017.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint 1706:0777 2017.
Ben Graham. Fractional max-pooling (2014). arXiv preprint arXiv: 14226674, 2014.
Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. Proceedings of the 34th International Conference on Machine Learning-Voliee Volume 70, pp. 1311-1320. JMLR. org, 2017.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. arXiv preprint arXiv:1904.0mm 03620. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4700 2017.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:/712. 055555 2017.

---

Under review as a conference paper at ICLR 2020
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batcl training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.048: 04836, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empir- ical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pp. 473-480. ACM, 2007.
Xirong Li, CeesG M Snoek, Marcel Worring, Dennis Koelma, and Aroold WM Smeulders. Boot- strapping visual categorization with relevant negatives. IEEE Transactions on Multimedia, 15(4): 933-945, 2013.
Senwei Liang, Yuehaw Kwoo, and Haizhao Yang. Drop-activation: Implicit parameter reduction and harmonic regularization. arXiv preprint wrXiv:/8/1.088550 2018.
Takeru Miyato, Shin-ichi Macda, Masanori Koyama, Ken Nakac, and Shin Ishii. Distributional smoothing with virtual adversarial training. arXiv preprint arXiv: 150.000677. 2015.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXlv:1701.06548, 2017.
Sylvestre-Alvise Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596. 2014.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning. arXiv preprini arXiv: 2018.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International Conference on Machine Learning, pp. 4535-4544, 2018
Burrhus Skinner. Reinforcement today. American Psychologist, 13(3)99, 1958.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2015.
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. arXiv preprint arXiv:1802.0377 2018.
Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. Disturblabel: Regularizing enn on the loss layer In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7753-4762, 2016.
Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regularization for deep residual learning, arXiv preprint arXiv:1802.02375. 2018.
Yang You, lgor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708. 070880388888888 6, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
10

---

Under review as a conference paper at ICLR 2020
Ke Zhang, Miao Sun, Tony X Han, Xingfang Yuan, Liru Guo, and Tao Liu. Residual networks of residual networks: Multilevel residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 28(6):1303-111 2017.

---

Under review as a conference paper at ICLR 2020
A LILAC: ALGORITHM
Algorithm 1:
Training strategy inspired by incremental learning
initialization;
Input: (X, Y) where Y E C1, C2, .., CMr
M Total number of labels in the dataset;
772 Number of labels to introduce in incremental step;
Total number of samples;
b Starting incremental batch;
C {c1, Cas
for inc batch= b to Ad do m
for fixed epochs do
C = {c1, (isne af +m)
n number of samples with labels in C;
S = (illi E Cy:::
((1g,,;)1ys E ,
F" S U :" vi)lus C/O}² selected random;
s.t. S = F
inc batch = and 2 then F' update target vectors using Eqn.
end
Train Model with data (S)
end
end
B HYPER-PARAMETER SETUPS
Table 5: List of hyper-parameters used in batch learning. Note: All experiments used the SGD optimizer.
Parameters
CIFAR10/100
STL10
Epochs
300
450
Batch Size
128
128
Learning Rate
0.1
0.1
Lr Milestones
[90 180 260]
[300 400]
Weight Decay
0.0005
0.0005
Nesterov Momentum
Yes
Yes
Gamma
0.2
0.1
12

---

Under review as a conference paper at ICLR 2020
Table 6: List of hyper-parameteess used in ISNE.
Parameters
Value
# Components
Perplexity
30.0
Early Exaggeration
12.0
Lr
200.0
Iterations
1000
Iteaations w/o Progress
300
Min. Grad. Norm
1e-07
Metric
Euclidean
Method
Barnes-Hut
Angle
0.5
C APPLICABILITY TO VIDEOS
Table 7: We extend LILAC to a 3D convolutional architecture and the HMDB51 (video) dataset.
We clearly observe an improvement in over performance when compared to batch learning.
Model
Training
HMDB51 Performance (%)
Batch
45.52
C3D
LILAC w/o N.L
45.29
Batch N.L
46.14
C3D
LILAC
46.17
D PROPERTY: VARIATION OF FIXED INTERVAL SIZE IN INCREMENTAL LABEL INTRODUCTION
Table 8: The table captures the effect of varying the number of epochs used for the fixed training intervals in the incremental label introduction phase. Across CIFAR-10 there is obvious peak after which the mean value decreases. However, in STL-10 there seems to be a consistent increase, with the assumption of minor noise. Finally, in CIFAR-100 there isn't a clear pattern.
Performance (%)
Property
Training
CIFAR-10
CIFAR-100
STL-10
E=1
95.20 0.042
78.39 + 0.221
73.02
0.949
E =
95.19 + 0.138
78.59 + 0.147
73.26 + 1.180
E =
LILAC w/o AC
95.33 0.195
78.30 + 0.274
73.53 + 0.331
E 7
95.37 + 0.167
78.51 + 0.209
73.23 + 0.864
E = 10
95.26 + 0.110
78.44 + 0.227
73.89 + 0.603
From the results in Table 8 an explicit pattern that can be used to select the value of E without trial runs. Further, the available run-time is an important constraint when select E from range of values since both and E affect it.
observe that the choice of E is dependent the dataset. There isn 't
13


Reviews:
[{"review_id": "rkeSQLs0KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\nThis paper makes the observation that a curriculum need not depend on the difficulty of examples, as most (maybe all) prior works do. They suggest instead a curriculum based on learning one class at a time, starting with one and masking the label of all others as 'unknown' (i.e. treating them as negative examples), and unmasking classes as learning progresses. This is the \"incremental labels\" part. They make another observation, that label smoothing is applied to all examples regardless of difficulty, and propose an alternative \"adaptive\" version where labels are smoothed only for difficult examples. This is the \"adaptive compensation\" part.\n\nPaper contributions: \n- the two observations described above are both interesting, and methods addressing these seem like good ideas in light of the observations\n - the explanation of the method and experiments are very clear.\n - ablation and exploration studies are well-done to further investigate the proposed methods\n\nReview summary & decision: \nI like the core of this paper a lot, and recommend acceptance. It makes some insightful observations, reasonable suggestions, explains things clearly, and does reasonable experiments. The observations and proposed methods are both valuable contributions to the field. If the related work and clarity of abstract/intro are improved, along with addressing false claims and some other relatively minor things, I think this could be a very good paper, and I would be happy to increase my score.\n\nReasons for decision:\n - Interesting observations are made and the approaches taken are interesting and well-motivated.\n - The paper overall is well structured, easy to understand, and thorough in the experiments.\n - Some related work (detailed below) in other fields and just about curriculum learning seems to be missing. Very strange claim (which is false, as far as I know) that curriculum learning has only been used in shallow networks emphasizes that the related work is lacking. \n - The abstract and intro summary of contributions didn't do a very good job of conveying what the methods do, although they are clearly explained elsewhere (see suggestions below)\n - A lot of time is spent on detail of experiments and long, clear explanations (this is great), but makes it read a bit like a lab report. Figure 2 and the \"properties of LILAC\" section are nice, but could be improved with more discussion and reference to other works/fields to provide insight about _why_ LILAC works, not just _that_ it works.\n - Results seem very incremental to me (improvements over other methods are in the decimal places), and I think many people will criticize or dismiss the methods on that basis. I hesitated about this, and in the end decided that the interesting observations are the most valuable part of the paper, not pushing SOTA (while of course it's valuable to report numbers, I think our field should focus less on SOTA numbers in general). \n - Misleading results are presented for CIFAR-10; ShakeDrop is not SOTA, and no reason is provided for why the particular citations in that table are there (there are many others that could be...). I don't mind that you don't get SOTA, but I mind being misled. If I've misunderstood something here, please clarify! :)\n\nFeedback/suggestions/nits (not necessarily part of decision assessment): \n1. Briefly review some continual learning work; incremental labels are very similar to open set learning; this is worth mentioning. Would also be nice to mention anomaly detection here. \n2. Briefly review negative mining (cases where this improves learning e.g. hard negative mining in text). \n3. More discussion of motivation and why you think that LILAC works would be nice; connections to the above-mentioned fields could help. \n4. Space could be made for the things I suggest here by decreasing spacing in the \"main contributions\" bullets and reducing the size of Figure 1 (the coloured tiles are very large and there's a lot of unnecessary white space. The size of text is mostly good, although the legend could also be decreased in size) \n5. Incremental labels are well-explained in the dedicated section, but not until then. In the abstract, intro, and first section on \"LILAC\", I didn't really understand what was going on. I would suggest not using the words mask/unmask; this term is overloaded and gives the wrong connotation to me (that you don't use the predictions for the masked out classes). If you can't find a different term, I'd suggest explaining more clearly what it means to \"mask\" in this context in the abstract. It would also be good in the abstract to have a sentence motivating your approach (after \"... learning difficult samples\"). The next sentence sort of tries to do this \"... starting point from which\" but I think this is really unclear and makes it sound like you're learning an init. \n6. Adaptive compensation is well explained in the abstract, but very confusingly in the intro summary of contributions. \n7. I think it would be clearer to call it \"adaptive smoothing\". I know LILAC is a nice name, but so is LILAS. \n8. Cite first sentence in abstract or reword\n 9. \"in this work, we propose.... which introduces\" propose/introduce is repetitive\n10. \"to the best of our knowledge,\" [insert \"all\" here]\n11. \"curriculum learning approaches have only been tested using shallow networks\" This is just false as far as I know; apart from the misleading presentation of CIFAR-10 results, addressing this is probably the most important suggestion I have. e.g. here's a whole thesis on doing CL with deep convnets that I found from a few minutes googling: http://www.diva-portal.org/smash/get/diva2:878140/FULLTEXT01.pdf  \n12. LILACto -> LILAC to\n13. inconsistent use of italics vs. bold for emphasis in the text. Typically use italics; bold is for keywords and subtitles. \n14. Mention tsne hyperparameters in an appendix \n15. Conclusion is more of a summary; the first couple sentences are good but the repeating of the explanation of both methods could be moved to the abstract and/or intro (these sentences are much clearer than the ones used there), making more room to discuss future work and connections to other fields. \n16. CIFAR-10 table should list method names, not just citations (they're referred to by name in the caption, which makes it hard to read the table)\n\n Questions:\n1. What other explorative experiments could you do to investigate properties of IL and AC?\n2. Did I misunderstand something about the CIFAR-10 SOTA presentation (is there some reason for the choice of numbers cited here?)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgN1K_0FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---- Paper summary ----\nThis paper proposes a curriculum learning approach for classification. The proposed curriculum consists of two phases:\n(1) a “label introduction phase”, in which the model is able to see and learn to classify only subset of labels (the model still trains on the samples belonging to the “unseen” classes, but their label is now set to a default class). The subset of seen labels is expanded incrementally, until the entire label set is observed. \n(2) an “adaptive compensation phase”, where the model trains on all labels, but the targets for each class are replaced from 1-hot vectors to smooth version. This only applies to the classes on which the model has made mistakes in a previous training round.\nThis method is tested on 3 image classification datasets, and a single neural network architecture is tested per dataset (either ResNet18 or ResNet34).\n\n---- Overall opinion ----\nWhile the ideas introduced in this paper may have merit, I believe the experimental evidence is quite limited. Based on the results shown in the paper, I am not convinced that this approach is better than the baselines it compares to. Moreover, since the claim is that this approach is a general curriculum learning method, I find the setting it was tested on very limited (3 datasets, 1 model per dataset), especially since there are no theoretical results to complement the empirical evidence. Finally, the method introduces several parameters (b, m, E, T, eta) that are treated is a somewhat hand-wavy manner, without a proper analysis on the effect of such parameters and how one should set them. Details on this, and other major issues, can be found below. For these reasons, I believe the paper in its current form is not yet ready for publication.\n\n---- Major issues ----\n1. The paper simply mentions that the unseen labels are set to a default label, which Figure 1 implies (and is not otherwise clarified) is one of the labels in the dataset. I am not sure intuitively why it makes sense to force the model in the beginning to map a sample’s inputs to another label from the dataset, which in the end it has to learn is wrong. Doesn’t it make more sense to map the unseen labels to a new, fictional label? If not, then how do you decide which of the M labels to choose as the fake label?\n\n2. The method introduces several hyperparameters, such as b (number of visible labels in the beginning), m (number of labels to reveal at each step), E (number of epochs in each incremental phase), T, epsilon. The specific numbers used in the experiments are reported, but it is not clear how these were chosen, and how one would choose them for a new dataset or model.\n\n3. In Table 1, the LILAC results are bolded with a caption saying that bold they means “the best case scenario”, and the main text also claims that “, LILAC is the only one to consistently increase classification accuracy and decrease the standard deviation across all datasets compared to batch learning”. However, from Table 1 it seems LILAC has neither the highest accuracy (label smoothing has overall higher accuracy), neither the lowest std. It may seem that the authors arbitrarily decided what is the best accuracy/std tradeoff which makes their method seem the best. Please define clearly the criteria for establishing the best method, and explain in what setting this criteria is a valid choice.\n\n4. Aside from the issue mentioned above, the differences in accuracy or std in Table 1 seem minor (e.g., within 0.10% on Cifar10, and within 1% in the others). Please provide more evidence that these differences are significant.\n\n5. How was the consistency in table 1 decided? Please provide the specific metric.\n\n6. Regarding the choice of models and datasets, the method was only tested on image datasets. This is could be enough as a contribution, but in this case the introduction and abstract should not claim a generality that has not been tested. Similarly, the paper only considers 1 model per dataset. Does this work for other models too? \n\n7. From Table 3, it looks like the LILAC w/o AC is actually worse than the baseline. In this case, what is the benefit of having the label introduction phase? Why not just have the AC component alone? If there is a reason, please include the results for AC alone in Table 3.\n\n8. I find the comparison in Figure 2 potentially misleading, because I believe 1 epoch of batch training is not equal in terms of amount of training as 1 full span of the incremental phase. I believe these embeddings should be shown at convergence time.\n\n9. In Table 4 and accompanying text, the authors conclude that “LILAC is relatively invariant to the order of the label introduction”. However, to me both random and ascending order seem actually random with respect to how this order is used. I advise the authors to try other orderings too, such as sorting them by the error of an initial training of a classifier, or other more difficulty-based orders.\n\n---- Minor issues ----\n1. In Algorithm 1, there is a missing equation (“Eqn. ??”), also I’m not sure why the first line says “Write here the result”.\n\n2. “small batch sizes help achieve more generalizable solutions, but do not scale as well to vast computational resources as large mini-batches” → this is a bit confusing. How do large mini-batches scale better, and what is the difference between “small batch sizes” and “large mini-batches”? \n\n---- Questions ----\nPlease see the major issues questions above.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeWvQqcFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary:\n\nThis paper proposes a novel direction for curriculum learning.  Previous works in the area of curriculum learning focused on choosing easier samples first and harder samples later when learning the neural network models.  This is problematic since we need to first compute how difficult each samples are, which introduces computational overheads.  In this work, the paper propose to gradually learn with a class-wise perspective instead.  The neural network has only access to the labels of certain classes (chosen randomly) in the beginning, and the samples that belong to the rest of the classes are treated as unseen samples but with a label forced into the last class.  Then, the true labels of unseen classes are gradually revealed, and this is repeated until in the final incremental step, all labels are revealed.  The method further has an adaptive compensation step, which use a less peaked distribution label for supervision only for the incorrectly predicted samples.  The experiments show that with only the first step, the proposed method is worse than the original batch learning, but by adding the second label smoothing step, there is improvement over the original batch learning setup.\n\n\n2. Decision:\n\nWeak Reject -- The class-wise idea for curriculum learning is interesting but the motivation and intuition behind the design of the proposed method is weak.\n\n\n3. Reasons for decision:\n\nPros:\n\nThe class-wise idea used in this paper seems to add a new direction to the area of curriculum learning.  The experiments are well designed, with ablation study to understand the behavior of the proposed method in depth.\n\nCons:\n\nThe motivation behind the ideas of the paper and the design of the procedure was not so clear.  Why would it be beneficial to start learning from a few classes in the beginning and then gradually expanding the class labels?  I was also curious if this will be more beneficial with a dataset with more classes (maybe not so different by looking at the comparison of CIFAR10 and 100 according to Table 1).\n\nIf my understanding was correct, the masked labels from unseen samples are all forced into the last class.  Does this design cause new issues, since the last class will have a lot of label noise until the final incremental step?  If yes, is it possible to consider other ways to add a fake label?   A naive way might be: for unseen samples, attach a fake soft label that has uniform probability over masked classes (M-b) and zero probability for the seen classes (in b).\n\nIn the experiments, it seems that the proposed method has consistency but no other baselines have consistency.  However, if you just look at the best mean accuracy, a naive label smoothing is often better than the proposed method.\n\n4. Additional comments not related to the decision of the paper:\n\nIn Algorithm 1 in appendix A, the equation link is lost: “Eqn. ??”\n\nIn appendix B, what is the decay rate for LR milestones?\n\n\n\nAfter response:\n\nThank you for answering my questions and for the comments.  Even though towards the end of the learning stage, the true labels are given for all samples, I feel these noisy steps go against the spirit of curriculum learning.  Starting from small classes seems to be a good curriculum and is intuitive, but the noisy part seems to be adding a bad curriculum.  This paper can be much stronger if this part is solved.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers appreciated the ideas presented in the paper and their novelty, there were major concerns raised about the experimental evaluation. Due to the serious doubts that the reviewers raised about the effectiveness of the proposed approach, I do not think that the paper is quite ready for publication at this time, though I would encourage the authors to revise and resubmit the work at the next opportunity.
Decision: Reject