A Simple Approach To Define Curricula For Training Neural Networks
Authors: Vinu Sankar Sadasivan
Anirban Dasgupta
Venue:  (2021)
Keywords: Curriculum learning
neural networks

Abstract: In practice, sequence of mini-batches generated by uniform sampling of examples from the entire data is used for training neural networks. Curriculum learning is a training strategy that sorts the training examples by their difficulty and gradually exposes them to the learner. In this work, we propose two novel curriculum learning algorithms and empirically show their improvements in performance with convolutional and fully-connected neural networks on multiple real image datasets. Our dynamic curriculum learning algorithm tries to reduce the distance between the network weight and an optimal weight at any training step by greedily sampling examples with gradients that are directed towards the optimal weight. The curriculum ordering determined by our dynamic algorithm achieves a training speedup of $\sim 45\%$ in our experiments. We also introduce a new task-specific curriculum learning strategy that uses statistical measures such as standard deviation and entropy values to score the difficulty of data points in natural image datasets. We show that this new approach yields a mean training speedup of $\sim 43\%$ in the experiments we perform. Further, we also use our algorithms to learn why curriculum learning works. Based on our study, we argue that curriculum learning removes noisy examples from the initial phases of training, and gradually exposes them to the learner acting like a regularizer that helps in improving the generalization ability of the learner.

Under review as a conference paper at ICLR 2021
A SIMPLE APPROACH TO DEFINE CURRICULA FOR TRAINING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In practice, sequence of mini-batches generated by uniform sampling of examples from the entire data is used for training neural networks. Curriculum learning is training strategy that sorts the training examples by their difficulty and grad- ually exposes them to the learner. In this work, we propose two novel curricu- lum learning algorithms and empirically show their improvements in performance with convolutional and fully-connected neural networks on multiple real image datasets. Our dynamic curriculum learning algorithm tries to reduce the distance between the network weight and an optimal weight at any training step by greed- ily sampling examples with gradients that are directed towards the optimal weight. The curriculum ordering determined by our dynamic algorithm achieves training speedup of ~ 45% in our experiments. We also introduce new task-specific cur- riculum learning strategy that uses statistical measures such as standard deviation and entropy values to score the difficulty of data points in natural image datasets. We show that this new approach yields a mean training speedup of 43% in the experiments we perform. Further, we also use our algorithms to learn why cur- riculum learning works. Based removes noisy examples from the initial phases of training, and gradually exposes them to the learner acting like regularizer that helps in improving the general- ization ability of the learner.
study, we argue that curriculum learning
INTRODUCTION
Stochastic Gradient Descent (SGD) (Robbins Monro 1951 is simple yet widely used algo- rithm for machine learning optimization. There have been many efforts to improve its performance. number of such directions, such AdaGrad (Duchi et al. 2011 RMSProp (Tieleman & Hinton 2012). and Adam (Kingma & Ba, 2014). improve upon SGD by fine-tuning its learning rate, often adaptively. However, Wilson et al. (2017) has shown that the solutions found by adaptive meth- ods generalize worse even for simple overparameterized problems. Reddi et al. (2019) introduced AMSGrad hoping to solve this issue. Yet there is performance gap between AMSGrad and SGD in terms of the ability to generalize (Keskar & Socher 2017). Further, Choi et al. (2019) shows that more general optimizers such as Adam and RMSProp can never underperform SGD when all their hyperparameterss carefully tuned. Hence, SGD still remains one of the main workhorses of the ML optimization tooliit.
SGD proceeds by stochastically making unbiased estimates of the gradient on the full data (Zhao & Zhang 2015). However, this approach does not match the way humans typically learn various tasks. We learn concept faster if exposed to examples with more complexity, based on curriculum. An orthogonal extension to SGD (Weinshall & Cohen 2018), that has some promise in improving its performance is choose examples according to specific strategy, driven by cognitive science this is curriculum learning (CL) (Bengio et al. 2009). wherein the examples are shown the learner based on a curriculum.
presented the easy examples first and then gradually
1.1 RELATED WORKS
Bengio et al. (2009) formalizes the idea of CL in machine learning framework where the examples fed to the learner in an order based on its difficulty. The notation of difficulty of examples

---

Under review as a conference paper at ICLR 2021
has not really been formalized and various heuristics have been tried out: uses manually crafted scores, self-paced learning (SPL) (Kumar et al. 2010 uses the loss values with respect to the learner's current parameters, and CL by transfer learning uses the loss values with respect to a pre-trained learner to rate the difficulty of examples in data. Among these works, what makes SPI. particular is that they use dynamic CL strategy, i.e. the preferred ordering is determined dynamically over the course of the optimization. However, SPL does not really improve the performance of deep learning models, as noted in (Fan et al. 2018). Similarly, Loshchilov & Hutter (2015) uses function of rank based on latest loss values for online batch selection for faster training of neural networks. |Katharopoulos & Fleuret (2018) and Chang et (2017) perform importance sampling to reduce the variance of stochastic gradients during training- Grrves et al.) (2117) and Matiisen et al. (2020) propose teacher-guided automatic CL. algorithms that employ various supervised measures to define dynamic curricula. The most recent works in CL show its advantages in reinforcement learning (Portelas et al. 2020: Zhang et al. 2020).
Beegio et al. (2009)
The recent work by Weinshall & Cohen (2018) introduces the notion of ideal difficult score to rate the difficulty of examples based on the loss values with respect to the set of optimal hypotheses. They theoretically show that for linear regression, the expected rate of convergence a training step t for an example monotonically decreases with its ideal difficulty score. This is practically validated by Hacohen & Weinshall (2019) by sorting the training examples based on the performance of a network trained through transfer learning. However, there is a lack of theory to show that CL improves the performance of a completely trained network. Thus, while CL indicates that it is possible to improve the performance of SGD by a judicious ordering, both the theoretical insights as well concrete empirical guidelines to create this ordering remain unclear.
While the previous CL works employ tedious methods to score the difficulty level of the examples, Hu al. 2020) uses the number of audio sources to determine the difficulty for audiovisual learning. Liu et al. 2220)) uses the norm of word embeddings as difficulty measure for CL for neural machine translation. light of these recent works, we discuss the idea of using task-specific statistical (unsupervised) measures to score examples making it easy to perform CL. on real image datasets without the aid of any pre-trained network.
1.2 OUR CONTRIBUTIONS
Our work proposes two novel algorithms for CL. We do a thorough empirical study of our algorithms and provide some more insights into why CL works. Our contributions are follows:
We propose novel dynamic curriculum learning (DCL) algorithm to study the behaviour of CL. DCL not a practical CL. algorithm since requires the knowledge of a reasonable local optima as needs to compute the gradients of full data after ever training epoch. DCL uses the gradient information to define a curriculum that minimizes the distance between the current weight and desired local minima. However, this simplicity in the definition of DCL makes it easier to analyze its performance formally.
Our DCL algorithm generates a natural ordering for training the examples. Previous CL works have demonstrated that exposing a part of the data initially and then gradually ex- posing the rest is a standard way to setup curriculum. We use two variants of our DCL framework to show that is not just the subset of data which is exposed to the model that matters, but also the ordering within the data partition that is exposed. We also analyze how DCL is able to serve as a regularizer and improve the generalization of networks.
We contribute simple, novel and practical CL approach for image classification tasks that does the ordering of examples in completely unsupervised manner using statistical measures. Our insight is that statistical measures could have association with the dif- ficulty of examples in real data. We empirically analyze our argument of using statistical scoring measures (especially standard deviation) over permutations of multiple datasets and networks. Additionally, we study why CL based on standard deviation scoring works using our DCL framework.
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Approximate greedy dy- namic curriculum learning (DCL+).
Input: Data X, local minima vú, weight Wt, batch size b, and pacing function pace.
W0'
Output: Sequence of mini-batches
W1 = Wo -1 (wm())
B, for the next training epoch.
cos
Ro n(ll/fowaall cose8 [IFf( cos [P/2(W))),, cns
1: āt v
W2 = W1 - (w2
2: Pt -
ag
3: B, t
||oal]
Ro
62
4: for (i = 0; N: 1) do
5: append a (ww))
Wa = W2 Vf2(W2)
to Pt
Rs
6: end for
Rg sin /[IF//(me]l] sin [IF//(,))I, sin -11P%6(w)11, sin
7: X' +- X' sorted according to Pt. in
ascending order
8: size t (ace(t)
Figure 1: A geometrical interpretation of gradient steps for the understanding of equation [
9: for (i 0; size; b) do
append (ii, + b 1] to B.
11: end for return B,
2
PRELIMINARIES
At any training step t, SGD updates the weight w, using Vf((w:) which is the gradient of loss of example æ; with respect to the current weight. The learning rate and the data are denoted by and X' = {(a;; yi)) N-1 ) sponding label for a dataset with K classes. We denote the learner as hgg IR + [K]. Generally, SGD is used train he by giving the model sequence of mini-batches (Bo, B ..., BT where B, S vi E [T]. Each B, is generated by uniformly sampling examples from the data. We denote this approach as vanilla.
respectively, where x; E Rd denotes a data point and y; E [K] its corre-
In CL. the curriculum is defined by two functions, namely the scoring function and the pacing function. The scoring function, score Mi) Rd x [K] + R, scores each example in the dataset. Scoring function is used sort X in ascending order of difficulty. A data point (æ;, Ms) is said to be easier than (æj.3y) ) if score, (æ,, y4) < score, (xy, 3j), yz). where both the examples belong to X. Unsupervised scoring measures do not use the data labels to determine the difficulty of data points. The pacing function, pacea(() [T] + [N], determines how much of the data is to be exposed at training step (T).
We define speedup for CL model as its improvement over vanilla model (in terms of the number of training steps) to achieve given test accuracy. For example, CL has 2x speedup vanilla model achieves 90% test accuracy in 100 training steps while CL achieves the same 90% test accuracy in 50 training steps.
3 DYNAMIC CURRICULUM LEARNING
For DCL algorithms (Kumar et al. 2010: Graves et al. 2017; Matüsen et al.) 2020). examples are scored and sorted after every few training steps since the parameters the scoring function change dynamically with the learner as training proceeds. Hacohen & Weinshall (2019) and Bengio ct al. (2009) use a fixed scoring function and pace function for the entire training process. They empirically show that a curriculum helps to learn fast in the initial phase of the training process. In this section, we propose and analyze our novel DCL algorithm that updates the difficulty scores of all the examples in the training data every epoch using their gradient information. We hypothesize the following: Given a weight initialization and a local minima obtained by full training of vanilla SGD, the curriculum ordering determined by DCL variant leads speedup in training. We
3

---

Under review as a conference paper at ICLR 2021
0.46
0.014
0.44
Top-1 test accuracy
0.012
Test logs
0.010
0.40
vanilla
DCL+
0.38
DCL+
0.008
DCL-
DCL-
0.006
0.34
0.004
0.32
0.002
650
700
750
800
850
900
950
1000
1000
2000
3000
4000
5000
Training step
Training step
(a) Experiment
(b) Experiment 2
Figure 2: Learning curves of experiments 1 and comparing DCL+, DCL., and vanilla SGDs. Error bars signify the standard error of the mean (STE) after 30 independent trials.
first describe the algorithm, then the underlying intuition, and finally validate the hypothesis using experiments.
Our DCL algorithm iteratively works on reducing the L2 distance, Rg between the weight parameter wP and given optimal weight at any training step Suppose, for any é < t, S is the ordered set containing the (1-i+) 1) indices of training examples that are to be shown to the learner from the training steps through Let us define a₂ = (w - w;), R2 || and 0f as the angle between ((we) and az. Then, using a geometrical argument, (see Figure
j=t-1 Σ
j=t-1
R²
Rí "
(IVA((2;))222 cos of) 1
Σ
(IV sin ot)
j=t, i
jzi, iESt.-1
==-11
j=t-1
R
Σ
(IVB)( cos 0f) +12
Σ
(IVSi(0;;)l2 cos of)
f=t, iESi.-1
j=l, iESt.-1
j=--1
T1²
Σ
(IVf(()))l2 sin sin of)
(I)
j=H, 1E
For vanilla model, So, is generated by uniformly sampling indices from [N] with replacement. Since, finding a set So,r to minimize R? and an optimal v are intractable for nonconvex opti- mization problems, we approximate the DCL algorithm (DCL+, see Algorithm We approximate v with w. which is a local minima obtained from training the vanilla SGD model. Also, to re- duce computational expense while sampling examples, neglect the terms with coefficient n² in equation I while designing our algorithm. Algorithm 1 uses a greedy approach to minimize R2 by sampling examples at every epoch using the scoring function
score, [[())) cos 0f a.- V.((w.)
Pt.i.
|
Let us denote the models that use the natural ordering of mini-batcher greedily generated by Algo- rithm [ for training networks as DCL+. DCL- uses the same sequence
that DCL+ exposes to the network any given epoch, but the order is reversed. We empirically show that DCL+ achieves a faster and better convergence with various initializations of wn- We use learning rates with exponential step-decay rate for the optimizers in all our experiments traditionally
4

---

Under review as a conference paper at ICLR 2021
0.50
Figure Learning curves for experi- ment 2 with varying paac(() [KN] for DCL+. The parameter needs to be finely tuned for improving the gen- cralization of the network. A low value exposes only examples with less noise to the network at every epoch whereas a high & value exposes most of the dataset including highly noisy examples the network. A moder- ate k value shows less noisy examples along with some examples with moder- ate level of noise to the learner. Here, moderate k 0.6 generalizes the best.
0.45
Top-1 test accuracy
0.40
k 0.2
0.35
0.4
k 0.6
0.30
0.8
1.0
0.25
0.20
0.15
1000
2000
3000
4000
5000
Training step
done (Simonyan & Zisserman 2014 Szegedy et al. 2016). For fair comparison, we tune the learning rates and decay rates of the models.
Experimental setup: In our experiments, we pace(l) [N VI, where k E b/N, 1] is tunable hyper-parameter. We use a 2-layer fully-connected network (FCN) with 10 hidden neurons and Exponential Linear Unit (ELU) nonlincarities to empirically validate our algorithms (k 0.9) on a subset of the MNIST dataset with class labels O and (Experiment 1), Since, this is a very easy 99.9%), we compare the test loss values across
task (as the vanilla model accuracy is high training steps in Figure 2a to see the behaviour of DCL on an easy task. DCL+ shows the fastest convergence, although all the networks achieve the same test accuracy. DCL+ achieves vanilla's final test loss score at training step 682 (~ 30% speedup). In Experiment 2, we use 2-layered FCN with 128 hidden neurons and ELU nonlinearities to evaluate our DCL algorithms relatively difficult mammals dataset (Krizhevsky et al. 2009), super-class of CIFAR-100 Figure 2b shows that DCL+ achieves a faster and better convergence than vanilla with respect to the test accuracy in experiment 2. DCL+ achieves vanilla's 's convergence test accuracy score training step 1896 (~ 60% speedup). Further experimental details are deferred to Appendix B.1
0.6) on
Since, DCL is computationally expensive, perform DCL experiments only small datasets. Fine-tuning of & is crucial for improving the performance DCL+ on the test set (see Figure 3 We fine-tune by trial-and-error over the test accuracy score.
WHY IS CURRICULUM USEFUL?
At an intuitive level, we say that DCL+ converges faster than the vanilla SGD greedily sample those examples whose gradient steps are the most aligned towards an approximate optimal weight vector. In previous CL works, mini-batches from a partition of the dataset which is made by putting a threshold on the difficulty scores of the examples. Notice that our DCL algorithms generate mini-batches with natural ordering at every epoch. We design DCL+ and DCL- to investigate an important question: can CL benefit from having of mini-batches with a specific order or is it just the subset of data that exposed to the learner that matters? Figure 2 shows that the ordering of mini-batche matters while comparing DCL+ and DCL-, which expose the same set of examples to the learner in any training epoch. Once the mini-batch sequence for epoch is computed, DCL- provides mini-batches to the learner in the decreasing order of noise. This the reason for DCI- to have high discontinuities in the test loss curve after every epoch in Figure 2a With empirical results, argue that the ordering of mini-batches within an epoch does matter.
generated by uniformly sampling examples
Bengio et al. (2009) illustrates that removing examples that are misclassified by Bayes classifier ("noisy" examples) provides a good curriculum for training networks. SPL tries to remove examples that might be misclassified during training step by avoiding examples with high loss. CL by transfer learning avoids examples that are noisy to an approximate optimal hypotheses in the initial phases of training. DCL+ and DCL- try avoid examples with noisy gradients that might slow
5

---

Under review as a conference paper at ICLR 2021
Figure 4: Top 10 images with the highest standard deviation values (top row) and top 10 images with the lowest standard deviation values (bottom row) in CIFAR- --0 dataset.
Algorithm Curriculum learning method.
Input: Data X. batch size b, scoring function score, and pacing function pace. Output: Sequence of mini-batches [Bo, B1,..., Br-
sort X according to score, in ascending order
2: B [-
3: for (i 1; 1) do
4:
size <- (i)
5: X x 1, size 1]
6:
uniformly sample B, of size b from X.
7: end for
8: return B
down the convergence towards the desired optimal minima. Guo al. (018) empirically shows that avoiding noisy examples improves the initial learning of convolutional neural networks (CNNs). According their work, adding noisy examples to later phases of training serves regularizer and improves the generalization capability of CNNs. DCL+ uses its pace function to avoid highly chosen such
noisy examples (in terms of gradients). In our DCL experiments, the parameter that few moderately noisy examples (examples present in the last few mini-batches within an epoch) are included training along with lesser noisy examples to improve the network's generalization. We show the importance of tuning the pace function for DCL+ in Figure Hence, the parameter k serves as a regularizer and helps in improving the generalization of networks.
5 STATISTICAL MEASURES FOR DEFINING CURRICULA
In this section, discuss simple approach of using task-specific statistical measures define curricula for real image classification tasks. We perform experiments and validate our proposal over various image classification datatsets with different network architectures.
Based on the classification task, one could find statistical measure that could serve the purpose of a scoring function for defining curriculum. For instance, standard deviation and entropy informative statistical measures for images and used widely in digital image processing (DIP) tasks (Kumar& & Gupta 2012 Arora 198I). Mastriani & Giraldez (2016) uses standard deviation filters for effective edge-preserving preserving smoothing of radar images. Natural images might have a higher standard deviation if they have of edges and/or vibrant range of colors. Edges and colours are among the most important features that help in image classification at higher level. Figure 10 images that have the lowest and highest standard deviations in the CIFAR-100 dataset. Entropy measure gives measure of image information content and is used for various DIP tasks such automatic image annotation [Joon& & Manmatha 2004). We experiment using the standard deviation measure (stddev), the Shanon's entropy measure (entropy) (Shannon 1951), and different norm measures as scoring function for CL (see Algorithm 2). The performance improvement with norm measures not consistent and significant over the experiments perform (see Appendix A for details). For

---

Under review as a conference paper at ICLR 2021
vanilla
0.475
stddev+
entropy
Top-1 accuracy
0.425
0.400
0.350
0.325
100000
Training step
0.80
vanilla
stddev-
entropy+
Top-1
0.76
0.75
10000
40000
50000
60000
70000
Training step
Figure 5: Learning curves for experiments 3 (top) and 4 (bottom). Error bars represent the STE after 25 independent trials.
a flattened image example represented as æ
[2)(
()) E Rd, we define
m(x) Σiz Z(t)
and
d
(3)
Σ d- Dizo (I(1) ((x))²
stddev(x)) =
We use fixed exponential pace function that exponentially increases the amount of data ex- posed to the network after every fixed steplength number of training steps. For training step i, it is formally given as: pace(i) starting fraction the fraction the data that exposed to the model initially, inc the expo- nential factor by which the the pace function value increases after step, and N is the total number of examples in the data.
min[1,
inclamppinge NJ, where
We denote CL models with scoring function stddew as stddev+, stddew as stddev-, entropy as entropy+, and -entropy as entropy- Even though vanilla is competitive benchmark, we also use the CL by transfer learning framework (Hacohen & Weinshall 2019) (denoted as TL) as a baseline. We employ two network architectures for our experiments: a) FCN-512 A 2-layer FCN with 512 hidden neurons and ELU nonlincarities, and b) CNN-8 A moderately deep CNN with 8 convolution layers and fully-connected layers. We perform the fol- lowing experiments: CNN-8 with a) CIFAR-100 (Experiment 3), b) CIFAR-10 (Experiment 4), c)
Baseline and experimental setup:

---

Under review as a conference paper at ICLR 2021
vanilla
stddev+
stddev-
entropy+
entropy-
TL
80.0
95.00
88.00
94.75
87.75
49
79.5
87.50
48
79.0
57.5
94.25
87.25
47
78.5
57.0
94.00
87.00
56.5
93.75
86.75
56.0
93.50
86,50
55.5
93.25
86.25
77.0
55.0
93.00
86.00
(a) Experiment
(b) Experiment
(c) Experiment
(d) Experiment 6
(c) Experiment 7
Figure 6: Bars represent the final mean top-1 test accuracy (in %) achieved by models in experiments
3 - 7. Error bars represent the STE after 25 independent trials for experiments 3, 4 and 7, and 10 independent trials for experiments 5 and 6.
Standiadd deviation data points Standard daviation data points
Epoch = 001
Epoch = 005
Epoch 100
1.5
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
5000
10000
Data points ranked based on p value
Epoch 001
Epoch 005
Epoch 100
1.5
1.5
1.5
1.0
1,0
1.0
0.5
0.5
0.5
1000
2000
1000
2000
1000
2000
Data points ranked based on value
Figure 7: Relation of p and siddev values of examples over training epochs 1, 5, and 100 for exper- iments (top row) and (bottom row).
small mammals (Experiment 5) (these are the benchmark experiments used in Hacohen Wein- shall )2019)), and FCN-512 with d) MNIST (Experiment 6), e) Fashion-MNIS' (Experiment 7). For experiments 3 5, we use the same experimental setup as used in HaHachen Weinshall (2019). B In all our experiments, the models use fine- tuned hyper-parameter for the purpose of an unbiased comparison. Our experiments (3 and 4) show that both stddev and entropy measures scoring function provide superior results. Since stddew performs the best, we further investigate its benefits on multiple tasks (experiments 5, 6, and 7). Figures 5 and show the results from experiments. The best CL models achieve speedups of 56.6%. 55.0%. 46.9%, 8.1%. and 50.1% on experiments 3 - 7, respectively. With these empirical evidences argue that stddey is good measure to define curricula for image classification tasks.
More experimental details are deferred to Appendix
Analyzing siddev with our DCL framework: We use our DCL framework to understand why siddev works scoring function. We try to analyze the relation between the standard deviation
8

---

Under review as a conference paper at ICLR 2021
and Pt.: values of examples over training epochs. Figure shows the plots of standard deviations the Y-axis against examples plotted on the X-axis ranked based on their pt.; values an ascending order at various stages of training. It shows the dynamics of Pt.: over initial, intermediate and final stages of training. Relation between Pt.i and sidder is evident from these plots. In the initial stage of training, examples with high standard deviations tend to have high p values. In the final stage of training (the trend changes to the exact opposite after the intermediate stage), examples with high p values tend have low standard deviation values. This shows that stddev can also be useful in removing noisy examples from the initial phases of training and hence help in defining a good curriculum.
CONCLUSION
In this paper, we propose two novel CL algorithms that show improvement in performance over multiple image classification tasks with CNNs and FCNs. Our DCL algorithm greedily samples data to move towards an optimal weight in faster manner. It tries to avoid noisy gradients from slowing down the convergence. Its two variants, DCL+ and DCL-, provide insights on how impor- tant ordering of mini-batches is for CL. The requirement to finely tune the pace function of DCL+ shows that adding moderate amount of noisy examples to training helps in improving the network's generalization capability. In this work, a fresh approach to define curricula for image classification tasks based on statistical measures is introduced. This technique makes it easy to score examples in completely unsupervised manner without the aid of any teacher network. We thoroughly eval- uate our new CL method and find it benefits from faster (mean speedup of ~ 43%) and better convergence (test accuracy improvement of ~ 0.2% 2.2%). We use our DCL framework to un- derstand stddev. With our results, we argue that CL algorithms help in faster initial learning by removing noisy examples that slow down the convergence towards a minima. Gradually, they add noisy examples training order improve the performance of the network on unseen data.
REFERENCES
PN Arora. On the shannon measure of entropy. Information Sciences, 1981.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, 2009.
Haw-Shiuan Chang, Erik Learned-Mill Miller, and Andrew McCallum. Active bias: Training more accu- rate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, 2017.
Dami Choi, Christopher J Shallue, Zachary Nado, Jachoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint 1rX1v:1910.05446, 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Mach. Learn. Res., 2011.
Yang Fan, Fei Tian, Tao Qin, Kiang-Yang Li, and Tie- Yan Liu. Learning to teach. In International Conference on Learning Representutions, 2018.
Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. Proceedings of the 34th International Conference on Machine Learning. 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. CoRR, abs/1904.036266, 2019.
Di Hu, Zheng Wang, Haoyi Xiong, Dong Wang, Feiping Nie, and Dejing Dou. Curriculum audio- visual learning. arXiv preprint arXiv:2001.094 0..00/ 2020.

---

Under review as a conference paper at ICLR 2021
Jiwoon Jeon and R Manmatha. Using maximum entropy for automatic image annotation. In Inter- national Conference on Image and Video Retrieval, 2004.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. arXiv preprint arXiv: 2018.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to SGD, CoRR, abs/1712.00628 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
M. P. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems. 2010.
V. Kumar and Priyanka Gupta. Importance of statistical measures in digital image processing. 2012.
Xuebo Liu, Houtim Lai, Derek Wong, and Lidia Chao. Norm-based curriculum learning for neural machine translation. arXiv preprint arXiv:2006.0200 020014 2020.
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv: 1et:::5111.03444 2015.
Mario Mastriani and Alberto E. Giraldez. Enhanced directional smoothing algorithm for edge- preserving smoothing of synthetic-aperture radar images. CoRR, abs/1608.01110 00980.999,.... 2016.
T. Matiisen, A. Oliver, T. Cohen, and Schulman. Teacher-student curriculum learning. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference on Robot Learning, 2020.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. CoRR, abs/1904.09237, 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. Ann. Math. Statist., 1951.
Claude Shannon. Prediction and entropy of printed english. Bell system technical journnal, 1951.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.1533 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
Daphna Weinshall and Gad Cohen. Curriculum learning by transfer learning: Theory and experi- ments with deep networks. CoRR. abs/1802.03796, 2018.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems. 2017.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. arXiv preprint arXiv:2006.0966 2020.
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In International Conference on Machine Learning, 2015.
10

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL EMPIRICAL RESULTS
In Section 5] we study the performance of CL using stddew and entropy as scoring measures. Other important statistical measures are mode, median, and norm (Kumar & Gupta 2012). A high standard deviation for real image could mean that the image having a lot edges and wide range of colors. A low entropy could mean that an image is less noisy. Norm of an image gives information about its brightness. Intuitively, norm not a good measure for determining difficulty of images as low norm valued images really dark and high norm valued images are really bright. We experiment with different norm measures and find that they do not serve as a good CL scoring measure since they have lesser improvement with higher accuracy variance over multiple trials when compared to stddev- on the CIFAR datasets. We use two norm measures
norm(x) || [|2, and class norm(x) = || M
(4)
where an image in the dataset represented a vector, and Pe is the mean of all the images belonging to the class of æ. All the orderings are performed based on the scoring function and the examples are then arranged avoid class imbalance within mini-batch in our experiments. Let us denote the models that use the scoring functions norm norm+, -norm norm- class_orm as norm++, and -class_ -classsoorm jorm as -class_norm
Figure X shows the results of our experiments CIFAR-100 and CIFAR-10 10 datasets with CNN-8 using norm and dlass_norn scoring functions. We find that the improvements reported for norm-, the best model among the models that norm measures, have lower improvement than stddev- Also, norm- has higher STE when compared to both vanilla and stddev- Hence, based on our results, suggest that standard deviation is more useful statistical measure than norm measures for defining curricula for image classification tasks.
B EXPERIMENTAL DETAILS
B. NETWORK ARCHITECTURES
All FCNs (denoted FCN-M) we use are 2-layered with a hidden layer consisting of M neurons with ELU nonlinearities. Experiment 1 employs FCN-10 while experiment 2 employs FCN-128 with no bias parameters. The outputs from the last layer fed into softmax layer. Experiments 6 and 7 employ FCN-512 with bias terms. The batch-size is 50.
For experiments 3 5. we use the CNN architecture that is used in Hacohen & Weinshall (2019) The codes are available in their GitHub repository. The network (CNN-8) contains 8 convolution layers with 32, 32, 64, 64, 128, 128, 256, and 256 filters respectively and ELU nonlincarities. Except for the last two layers with filter size 2, all other layers have fliter size of 3 x 3. Batch normalization is performed after every convolution layer, and 2 max-poou max-pooling and 0.25 dropout layers after every two convolution layers. The output from the CNN is flattened and fed into fully-connected layer with 512 neurons followed by a 0.5 dropout layer. A softmax layer follows the fully-connected output layer that has number of neurons same as the number of classes in the dataset. The batch-size is 100. All the CNNs and FCNs are trained using SGD with cross-entropy loss. SGD uses an exponential step-decay learning rate. Our codes will be published on acceptance.
B.2
TUNING
For fair comparison of models, the typer-parameters should be finely tuned as rightly mentioned in Hacohen & Weinshal] 2019). We exploit hyper-parameter grid-search to tune the hyper-parameters of the models in our experiments. For vanilla models, grid-search is easier since they do not have pace function. For CL models, follow coarse two-step tuning process as they have lot of hyper-parameters. parameters. First we tune the optimizer hyper parameers fixing other pareee-arsmettrs. Then we fix the obtained optimizer parameters and tune other hyper-parameters.
11

---

Under review as a conference paper at ICLR 2021
vanilla
stddev
norm+
nom-
dlass_nomi
class_norm-
81
50
80
79
48
78
77
46
76
44
75
(a) CIFAR-100 with CNN-8
(b) CIFAR-10 with CNN-8
Figure 8: Bars represent the final mean top-1 test accuracy (in %) achieved by models. Error bars
represent the STE after 10 independent trials.
B.3 DATASET DETAILS
We use CIFAR-100, CIFAR-10. maammlls MNIST, and Fashion-MNIST datasets. CIFAR- 100 and CIFAR-10 contain 50, 000 training and 10, 000 test images of shape 32 32 3 belonging to 100 and 10 classes, respectively. small mammals is super-class of CIFAR-100 containing classes. It has 2, 500 training and 500 test images. MNIST and Fashion-MNND training and 10, 000 test gcay-scale images of shape 28 x 28 belonging to 10 different classes. All the datasets are pre-processed before training to have zero mean and unit standard deviation across each channel.
contain 60, 000
12


Reviews:
[{"review_id": "em87MxTYhBs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies curriculum learning and proposes two methods to order the examples by (1) gradient information and (2) statistical measures like standard derivation and entropy. The experiment results show that the proposed curriculum learning strategies can speed up the convergence by a large margin and the authors provide some insights about why curriculum learning works.\n\nStrengths:\n1. The proposed \"dynamic curriculum algorithm\" can speed up the convergence by ~45% and the proposed task-specific curriculum strategy based on standard deviation and entropy can yield an average speedup of ~43%.\n2. The code and data are shared and helpful for reproducing the experiments conducted in the paper.\n\nWeaknesses:\n1. The paper is poorly written. There are even no sections to discuss related works and experimental settings in the main paper. Although some related works are discussed scatteredly in the paper, it might be helpful to have a specific section to compare related works with the proposed methods, which makes it much easier to identify the contributions and novelty of this work. Besides, although I found the experimental settings in the supplementary, the main paper at least should have discussed the basic experimental setup to understand how the experiments are conducted.\n\n2. The proposed DCL algorithm requires an optimal weight or a local optimal weight to calculate the difficulty scores. This requirement is unreasonable and renders the proposed methods useless.\n\n3. The proposed scoring function (the equation at the end of page 3) requires to compute the gradients on each sample. Perform back-propagation and computing gradient are highly prohibitive. The learning curves against the time cost should also be reported, in complement to the learning curves against the training steps in Fig.1.\n\n4. The pace function is just a constant, dependent on a tunable hyperparameter k. From Fig.2, it seems that the value of k has a large impact on the testing accuracy. It is not mentioned in the paper how the value of k is selected.\n\nSuggestions for improvement:\n1. It might be better to have a specific section to discuss related works and compare them with the proposed method.\n\n2. Index all equations. \n\nQuestions:\nThe questions to be addressed in the rebuttal are listed below:\n1. Where does the optimal weight in DCL comes from? Can the authors justify why a given optimal weight can be used during the training?\n\n2. What is the time cost of the scoring function?\n\n3. How is the value of k in the pace function tuned?\n\n-------------------------------Post-rebuttal-------------------------------\n\nThank you for revising the submission and the clarification in the rebuttal. After reading the rebuttal and other reviews, my main concerns about the novelty and computation cost are still unsolved. Therefore, I will keep my original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cRbsXenFBcL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work studies a number of curriculums for faster training of neural networks. They first propose a curriculum named DCL+ that is designed to order data points based on their alignment of gradient with the direction of optimization. This curriculum depends on the evaluation of individual gradients of datapoints as well as an approximation to a local optima. Next, they study a number of easy-to-compute statistical measures for ordering data points.\n\nPros:\n- The idea of ordering points based on using alignment of their gradients with the direction to the local optima is interesting.\n- The idea of using easy-to-compute statistical measures of data is also interesting.\n\nCons:\n- The empirical setup needs improvement. Important baselines are missing as described below. Standard datasets are not used and the ones used are relatively easy. The combination of these makes it hard to make any conclusions.\n- The ideas are not formalized well enough. Specifically, important details in the definition of optimal weights in DCL is missing. Also, it is not clear why standard deviation of an image or its entropy could be a proxy for how useful a data point is for training.\n\nDetailed notes:\n\nIntro:\n- The discussion about SGD and the generalization of other optimization methods is missing recent works such as [1].\n- It is said that “...SGD samples examples from data uniformly at random.” This is an inexact description of common training setups. We can use SGD with any stochastic estimate of gradients as long as it is unbiased. It doesn’t have to be from uniformly sampling a subset. See for example [2].\n- Related works on example reweighting and ordering are missing. For example [3], [4], and [5].\n\nSection 3:\n- Eq 1: 1) theta is not defined, 2) why isn’t the first equality an inequality?\n- Figure 3: 1) what is theta? 2) Why is R3 marked as the distance between w3 and w0? Shouldn’t it be the distance between w3 and bar{w}?\n- “We approximate w ̄ with w ̃, which is a local minima obtained from training the vanilla SGD model”, isn’t the goal of the entire training to find w^bar? How many steps do you train for to get w~?\n- “score(x)” change of notation, score was defined as a function of both x and y in Section 2.\n- “We use learning rates with an exponential decay rate for the optimizers” what is an exponential decay rate? How do you tune? Tune both base and the exponent? Why is it fair? There are numerous other learning rate schedules that are more common like the step-decay.\n- Figure 2: what is the dataset? What is the task? Why is a data point noisy if it has a low score? Does this figure mean the hyperparameters are tuned on the test set rather than a validation set?\n- Experiment 1 is mnist with labels 0 and 1. As noted in the text this task is very easy. Both DCL- and DCL+ eventually seem to classify the test set correctly.\n- Experiment 2 is small-mammals dataset. No citation is given. It is said to be a super-class of CIFAR-100. No comparison is done with optimization methods other than SGD. No references are given for prior baselines on this dataset. No reason is given for not trying out the proposed method on common benchmarks such as full MNIST, CIFAR-10, and CIFAR-100.\n- Figure 1: the placement of this figure is before Figures 2, 3 and algorithm 1 but it is referred to after them. The caption is not descriptive enough as it refers to experiments 1 and 2 that doesn’t make it easy to understand the figure without going back and forth between the figure and the text.\n\nSection 5:\n- There is no empirical comparison with prior works.\n- There is no empirical comparison between the DCL method proposed in prior sections.\n- Figure 6: same problem in captions as Figure 1.\n\n\nOther:\n- I could not uncompress the supplementary material. It needed PK compatibility V4.6 which does not come in the standard zip package.\n\n[1] Choi, Dami, et al. \"On empirical comparisons of optimizers for deep learning.\" arXiv preprint arXiv:1910.05446 (2019).\n[2] Zhao, Peilin, and Tong Zhang. \"Stochastic optimization with importance sampling for regularized loss minimization.\" international conference on machine learning. 2015.\n[3] Loshchilov, Ilya, and Frank Hutter. \"Online batch selection for faster training of neural networks.\" arXiv preprint arXiv:1511.06343 (2015).\n[4] Katharopoulos, Angelos, and François Fleuret. \"Not all samples are created equal: Deep learning with importance sampling.\" arXiv preprint arXiv:1803.00942 (2018).\n[5] Chang, H. S., Learned-Miller, E., & McCallum, A. (2017). Active bias: Training more accurate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems (pp. 1002-1012).\n\n\n=============\nAfter rebuttal:\nThank you for improving the clarity. Unfortunately, the following issues are still unresolved to me:\n- Figure 1: this geometrical argument seems to be at the core of Eq. 1 but I still have a hard time understanding it. You might want to formalize the argument in text.\n- Experiments are not convincing. Even the original MNIST is not a representative dataset for optimization methods. The theoretical contributions of this work is not enough to justify having limited experiments.\n- The rebuttal says: \"The hyperparameter k in DCL is tuned by trial and error on the test set.\". Does that mean mistakenly using the test set as the validation set?", "extra_scores": {}, "extra_text": {}}, {"review_id": "SC2yW2Mfgwt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper contains two curriculum learning algorithms of which one assume knowledge of the parameters found by the baseline, uniform-sampling, model to push updates in that direction, and the second orders images according to an increasing stddev/entropy of pixels. While the first approach is impractical because of the strong assumption, the second approach demonstrates small gains that lie within random variance (Fig. 5, Fig. 6) and would be not straight-forward to apply to non-image data e.g. text. These reasons make the paper hard to accept.\n\nThe main problem is knowing the parameters of the baseline, SGD, optimization. It's not clear why would one even need optimization again, if (a good enough) result is already known and gains from this re-optimization do not significantly improve over this baseline. The speedups mentioned in the abstract (45% and 43%) could not be located in the results in main body of the paper. How were they measured? Even if aligning updates with the SGD-trained parameters does speed up convergence, re-training from scratch will cost 143% of baseline time instead of 43%, as the standard training needs to be counted too.\n\nIssues include:\n- How to sample using \\rho_{t,i}? It's not a distribution and can be negative.\n- Figure 1: Judging from the plot, the vanilla curve converges faster than the curriculum. How can one see the >40% curriculum speed up?\n- Abstract's claim of removing noise is only supported in Section 4 through citing related works. Also, more evidence would be needed to call k a regularizer.\n- lines 9-10 in Algorithm 1 would interfere with bucketing in seq2seq applications and adversely affect performance.\n\nRegarding related work in Sec. 3: I couldn't confirm in (Graves et al, 2017) that they also sort examples by difficulty.\n\nThe last approach to define curriculum through statistical quantities makes sense, in principle, although the difference between curves in Fig. 5 is very small and could be caused by random variance as the error bars on Fig. 5 and Fig. 6 show. Another problem is that it's straight-forwardly applicable only to images and not categorical data, like text. \n\nOne suggestion of possible paper improvement: consider swapping and reworking sections 5 and 3, so that the content of sec. 5 becomes the main proposal and a reworked sec. 3 - its analysis. There one could analyze if the example ordering according to stddev does bias updates towards some \"good\" point of convergence, with one possible definition of \"good\" according to (now, unknown during optimization) SGD results.\n\nOther minor remarks:\n- \"greedy approach\" is mentioned multiple times before being explained on page 4. Consider deferring the use of term to that place.\n- Contributions: useful is a vacuous word, consider dropping it.\n- notation: square brackets used to denote several objects - sequences [B1, B2, ..] , ranges [T] and vectors [x1, x2, .. ]. Using different brackets could be better.\n- well-known concepts:\n  * no need to define stddev and mean in (2)\n  * (Arora, 1981): if entropy requires a citation at all then citing Shannon directly would be more appropriate.\n- Sec. 2: curriculum is defined by two functions -> we define curriculum by two functions\n- conclusion: display -> show\n- while CL indicate that -> while CL indicates that\n- judicial ordering -> judicious ordering\n\n=== After rebuttal ===\n\nThank you for your answers. I'm keeping the rating at 3.\n\n1)+2). I'm still not convinced that it's fair to claim an improvement of X% for a curriculum that, relying on final weights of a \"vanilla\" SGD-trainedmodel, converges in _additional_ X% to the 100% of \"vanilla\" time.\n\n3). Fair enough, but the revised draft still reads like examples are sampled from it.\n\nI double checked the context of citing (Graves et al, 2017) and believe it's still imprecise as in the original draft.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MOFiq9KUA1h", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n- This paper considers curriculum learning for neural networks in the context of supervised learning (specifically image classification). \n- First, the authors propose and assess a DCL+ algorithm. DCL+ uses a scoring function based on the alignment of an example's gradient with the vector from the current weight to a local minimum weight (obtained via a previous \"vanilla\" run with standard SGD). The pacing function is a constant fraction of the dataset size. The effect is that for a given epoch, only the subset of data that induces gradients which most point towards the local minimum is used for training. DCL+ is empirically shown to result in marginal improvement over the vanilla run in terms of test performance at convergence, and a significant speedup to reach vanilla performance. To investigate the effect of minibatch ordering within an epoch, the authors propose DCL-, an ablation that reverses the DCL+ minibatch ordering. Since DCL- performs worse than DCL+, the authors argue that minibatch ordering matters. \n- Second, the authors propose and assess a few curricula based on scoring functions that only use per-example statistics, e.g. standard deviation (stddev) or entropy of pixel values. Interestingly, it seems that for CNNs on CIFAR tasks, using stddev- (descending order of standard deviation) as the scoring function is best, while for MLPs on MNIST tasks, using stddev+ (ascending order of standard deviation) is best.\n- The authors connect their DCL+ algorithm to Bengio et al. (2009), which argues that a successful strategy for CL methods is to remove \"noisy\" examples. They also provide scatterplots of the data under DCL+ and stddev scoring functions.\n\n#### Strengths\n- The algorithmic innovations considered are simple.\n\n- The comparison between DCL+ and DCL- shows that minibatch ordering can matter in CL.\n\n- If one is willing to treat the unsupervised curriculum as a tunable hyperparameter (e.g. sweep over stddev+, stddev-, and vanilla), then the experiments suggest that we obtain some gain in test performance. A caveat is that this only is supported for the few relatively toy image classification datasets and architectures considered in the paper.\n\n#### Weaknesses\n- For a purely empirical paper, I would not say that the experiments are very comprehensive. For example, a clear discrepancy exists between Exps. 3-5 and Exps. 6-7 (in the former, stddev- is best, while in the latter, stddev+ is best, and most of the time the opposite curriculum is worse than vanilla). No attempt is made to discuss this or tease out the underlying reason: is it the dataset (CIFAR vs. MNIST) or the model (CNN vs MLP), or something else?\n\n- Conceptually, the curricula defined by image statistics have a critical weakness: they are completely agnostic to the label. This is in contrast with both DCL+/- and Bengio et al. (2009), which both consider \"example noise\" grounded in the full supervised task. The same image (and therefore statistics) with different labels (say, correct vs. incorrect) would play drastically different roles in a curriculum.\n\n- The connection between the notion of a \"noisy example\" as considered by Bengio et al. (2009) (misclassified by a Bayes classifier) and that of DCL+ (the example's gradient has strong anti-alignment with the direction to the local optimum) is not made technically clear or explicit. \n\n- Regarding Figs. 1, 5, and 6: I appreciate that the authors included a measure of uncertainty based on 25 or 30 trials, but I would strongly suggest adding significance tests for a more compelling analysis. This is sorely needed for, e.g., Experiment 4.\n\n- Fig. 7: The caption and the text imply that you'd provide correlations, but this is missing. Also, I don't see significant correlations for the bottom row.\n\n#### Recommendation\n- I currently recommend rejection (4). While the ideas are simple and interesting, the weaknesses in this submission preclude it from being very informative or useful to the community.\n\n#### Questions\n- Why does this work not consider comparisons with prior CL methods other than the vanilla baseline? If none are applicable, please explain why.\n- Can you unify the notion of a \"noisy\" example as considered by Bengio et al. (2009) and by DCL+? \n- Fig. 1: Why does subplot (a) only show a truncated x-axis?\n\n#### Minor suggestions\n- p. 2: What is \"theoretical evidence\"? Do you mean that there is a lack of theory, or a lack of empirical evidence, or both?\n- p. 2: Attribute the concepts of scoring and pacing functions to Hacohen and Weinshall (2019).\n- In general, the clarity of the writing (e.g. sentence style, diction) can be improved by a few careful passes.\n\n-------------------------------Post-rebuttal comments-------------------------------\n\nThank you for taking the time to revise your submission. I will maintain my original score of 4. The main justification for this is that the two main weaknesses I see in this paper (the first two I list in the original review) remain unresolved, and it indeed is unclear whether the second one could feasibly be addressed without significant changes to the core methodology currently proposed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed two algorithms for curriculum learning, one based on the the knowledge of a good solution (e.g. a local minima or a solution found by SGD) and another one proposed for natural image datasets based on entropy and standard deviation over pixels. 

Reviewers seem to like the ideas behind the proposed algorithms and their simplicity. However, there are several major concerns that are shared among reviewers:
1- One of the algorithms needs knowledge of a good solution (e.g. a local minima or a solution found by SGD) which makes it impractical and the other one doesn't use any information about the mapping between input and the label.
2- Discussing previous work on curriculum learning, explaining how proposed algorithms are different than previous work and empirical comparison to other curriculum learning methods are lacking or need a significant improvement.
3- The experiment section needs improvement both in terms of experimental methodology and having more tasks/datasets.

Reviewers have done a great job at pointing to specific areas that need improvement. I hope authors would use reviewers' comments to improve their work.

Given the above major concerns, I recommend rejecting this paper.


Decision: Reject