LeRaC: Learning Rate Curriculum
Authors: Florinel-Alin Croitoru
Nicolae-Catalin Ristea
Radu Tudor Ionescu
Nicu Sebe
Venue: NeurIPS 2022 Submitted (2022)
Keywords: curriculum learning
deep neural networks
model-level curriculum
data-free curriculum

Abstract: Most curriculum learning methods require an approach to sort the data samples by difficulty, which is often cumbersome to perform. In this work, we propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC), which leverages the use of a different learning rate for each layer of a neural network to create a data-free curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained as usual. This creates a model-level curriculum learning strategy that does not require sorting the examples by difficulty and is compatible with any neural network, generating higher performance levels regardless of the architecture. We conduct comprehensive experiments on eight datasets from the computer vision (CIFAR-10, CIFAR-100, Tiny ImageNet), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering various convolutional (ResNet-18, Wide-ResNet-50, DenseNet-121), recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures, comparing our approach with the conventional training regime. Moreover, we also compare with Curriculum by Smoothing (CBS), a state-of-the-art data-free curriculum learning approach. Unlike CBS, our performance improvements over the standard training regime are consistent across all datasets and models. Furthermore, we significantly surpass CBS in terms of training time (there is no additional cost over the standard training regime for LeRaC). Our code is freely available at: http//github.com/link.hidden.for.review.

LeRaC: Learning Rate Curriculum
Anonymous Author(s)
Affiliation
Address email
Abstract
Most curriculum learning methods require approach to sort the data samples by difficulty, which is often cumbersome to perform. In this work, we propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC), which leverages the use of a different learning rate for each layer of a neural network to create a data-free curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained usual. This creates model-level curriculum learning strategy that does not require sorting the examples by difficulty and compatible with any neural network, generating higher performance levels regardless of the architecture. We conduct comprehensive experiments on eight datasets from the computer vision (CIFAR-10. CIFAR-100, Tiny ImageNet), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering various convolutional (ResNet- 18, Wede-ResNet-50, DenseNet-121] recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures, comparing our approach with the conventional training regime. Moreover, we also compare with Curriculum by Smoothing (CBS), a state- of-thc-art data-free curriculum learning approach. Unlike CBS, our performance improvements over the standard training regime are consistent across all datasets and models. Furthermore, we significantly surpass CBS terms of training time (there is additional cost over the standard training regime for LeRaC). Our code is freely available at: http//git giggtu..b..... com/ hidden. for review
2
11
13
15
16
17
19
21
23
Introduction
25 Machine learning researchers relentlessly strive improve the performance of Al models. Much of this effort has been directed to the development of novel neural architectures 119. which have grown size and complexity M 7 [0] to leverage the availability of increasingly larger datasets. However, we believe the dominant trend to develop deeper and deeper neural networks is not sustainable on
26
27
29 the long term. To this end, we turn our attention to an alternative approach to increase performance of deep neural models without growing the size of the respective models. More specifically,
31 focus on curriculum learning, an approach initially proposed by Bengio et al. [I1] to train better neural networks by mimicking how humans learn, from easy to hard. As originally introduced by Bengio et al. [1]. curriculum learning is a training procedure that first organizes the examples in their increasing order of difficulty, then starts the training of the neural network on the easiest examples, gradually adding increasingly more difficult examples along the way, until all training examples fed the network. The success of the approach relies in avoiding force the learning of very difficult examples right from the beginning, instead guiding the model on the right path through the imposed curriculum. This type of curriculum is later referred to data-level curriculum learning Indeed, Soviany et al. 12 identified several types of curriculum learning approaches in the
39
35
36
37
38
39
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

literature, dividing them into four categories based on the components involved in the definition of machine learning given by Mitchell [3]. The four categories are: data-level curriculum (examples are presented from easy to hard), model-level curriculum (the modeling capacity of the network is gradually increased), task-level curriculum (the complexity of the learning task is increased during training), objective-level curriculum (the model optimizes towards an increasingly more complex objective). While data-level curriculum is the most natural and direct way to employ curriculum learning, its main disadvantage is that it requires a way to determine the difficulty of the data samples. The task of estimating the difficulty of the data samples has been addressed in different domain- specific ways, e.g. the length of text has been used in natural language processing [14 15]. while the number or size of objects were shown to work well in computer vision [6 17]. Despite having many successful applications [ [8) there is no universal way to determine the difficulty of the data samples, making the data-level curriculum less applicable to scenarios where the difficulty is hard estimate, e.g. classification of radar signals. The task-level and objective-level curriculum learning strategies suffer from similar issues, e.g. it is hard to create a curriculum when the model has 54 to learn easy task (binary classification) the objective function is already convex.
40
41
43
44
45
46
47
49
51
53
55
Considering the above observations, we recognize the potential of model-level curriculum learning strategies of being applicable across wider range of domains and tasks. To date, there are only few works [1 in the category of pure model-level curriculum learning methods. Furthermore, the existing methods have some drawbacks caused by their domain- deeenddndd or scchttctuur-speciffee design. For instance, Karras et al. [20] gradually increase the resolution of input images as new layers are appended to generative network, but the notion of imput resolution does not exist in other domains, e.g. text. Burduja et al. [99 blur the input images with Gaussian kernels, but this method is not applicable an input format for which there is convolution operation, e.g. tabular data. Sinha et al. 21 apply Gaussian kernel smoothing on convolutional activation maps, but this operation makes less sense for a feed-forward neural network formed only of dense layers.
56
57
59
61
69
64
65
To benefit from the full potential of the model-level curriculum learning category, we propose LeRaC (Learnng Rate Curriculum), novel and simple curriculum learning approach which leverages the use of a different learning rate for each layer of a neural network to create a data-free curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. This prevents the propagation of noise caused by the random initialization of the network's s weights. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained as usual. This creates a model-level curriculum learning strategy that is applicable to any domain and compatible with any neural network, generating higher performance levels regardless of the architecture, without adding any extra training time. To the best of our knowledge, we are the first to employ a different learning rate per layer to achieve the same effect as conventional (data-level) curriculum learning.
66
67
68
69
71
73
74
75
78
77
We conduct comprehensive experiments on eight datasets from the computer vision (CIFAR- 10 (22), CIFAR-100 (2). Tiny ImageNet [23]), language (BoolQ 24]. QNLI 25), RTE 25 and audio (ESC-
78
79
50 26
CREMA-D 27 domains, considering various convolutional (ResNet- 18 (4, Wide-ResNet-
80
50
28
DenseNet-121 121
29 ), recurrent (LSTM (0)) and transformer (CvT R. BERT 22, Sep'rr
81
31) architectures, comparing our approach with the conventional training regime and Curriculum by
82
Smoothing (CBS) (21). our closest competitor. Unlike CBS, our performance improvements over the standard training regime are consistent across all datasets and models. Furthermore, we significantly
83
84 surpass CBS in terms of training time, since there is no additional cost over the conventional training regime for LeRaC, whereas CBS adds Gaussian kernel smoothing layers.
85
In summary, contributions are twofold:
We propose novel and simple model-level curriculum learning strategy that creates curriculum by updating the weights of each neural layer with a different learning rate, considering higher learning rates for the low-level feature layers and lower learning rates for the high-level feature layers.
88
88
90
91
We empirically demonstrate the applicability to multiple domains (image, audio and text), the compatibility to several neural network architectures (convolutional neural networks, recurrent neural networks and transformers), and the time efficiency (no extra training time added) of LeRaC through comprehensive set of experiments.
92
93
94

---

95
2
Related Work
96
Curriculum learning was initially introduced by Bengio et al. [T as a training strategy that helps machine learning models to generalize better when the training examples presented in the ascending order of their difficulty. Extensive surveys on curriculum learning methods, including the most recent advancements on the topic, were conducted by Soviany et al. [ and Wang et
97
98
99
100
al. [8]] In the former survey, Soviany et al. [12 emphasized that curriculum learning is not only
101
applied the data level, but also with respect to the other components involved in machine learning approach, namely at the model level, the task level and the objective (performance measure) level. Regardless of the component on which curriculum learning is applied, the technique has demonstrated its effectiveness on a broad range of machine learning tasks, from computer vision
102
103
104
105
16 17
21
34] to natural language processing [5 and audio processing [39 40).
106
The main challenge for the methods that build the curriculum at the data level is measuring the difficulty of the data samples, which is required to order the samples from easy to hard. Most studies have addressed the problem with human input 41 [41]]] or metrics based on domain-specific heuristics.
107
108
109
For instance, the length of the sentence [36 44
and the word frequency [1 38 have been employed
110
in natural language processing. In computer vision, the samples containing fewer and larger objects have been considered
be easier in some works [16 17 Other solutions employed difficulty
111
112
estimators 45] or even the confidence level of the predictions made by the neural network 46 47 to
113
approximate the complexity of the data samples.
114
The solutions listed above have shown their utility in specific application domains. Nonetheless, measuring the difficulty remains problematic when implementing standard (data-level) curriculum learning strategies, at least in some application domains. Therefore, several alternatives have emerged over time, handling the drawback and improving the conventional curriculum learning approach. In [ the authors introduced self- self-paced learning to evaluate the learning progress when selecting the easy samples. The method was successfully employed in multiple settings [48) Furthermore,
116
118
119
120
some studies combined self-paced learning with the traditional pre-computed difficulty metrics
121
55 An additional advancement related to self- iife-pacdd learning the approach called self-paced learning with diversity [56). The authors demonstrated that enforeing certain level of variety among the selected examples can improve the final performance. Another set of methods that bypass the need for predefined difficulty metrics is known as teacher-student curriculum learning [57 58]. In this setting, a teacher network learns a curriculum to supervise student neural network.
53
122
123
124
125
126
Closer to our work, a few methods 21 proposed to apply curriculum learning at the model level, by gradually increasing the learning capacity (complexity) of the neural architecture. Such curriculum learning strategies do not need to know the difficulty of the data samples, thus having a great potential to be useful in a broad range of tasks. For example, Karras et al. 20 proposed to gradually add layers to generative adversarial networks during training, while increasing the resolution of the input images at the same time. They are thus able to generate realistic righrresouution images. However, their approach is not applicable to every domain, since there is no notion of resolution for some input data types, c.g. text. Sinha et al. [21] presented a strategy that blurs the activation maps of the convolutional layers using Gaussian kernel layers, reducing the noisy information caused by the network initialization. The blur level is progressively reduced to zero by decreasing the standard deviation of the Gaussian kernels. With this mechanism, they obtain a training procedure that allows the neural network to see simple information at the start of the process and more intricate details towards the end. Curriculum by Smoothing (CBS) (21] was only shown to be useful for convolutional architectures applied in the image domain. Although we found that CBS is applicable to transformers by blurring the tokens, it is not necessarily applicable to any neural architecture, e.g. standard feed- forward neural networks. As an alternative to CBS, Burduja et al. [99 proposed to apply the same smoothing process on the input image instead of the activation maps. The method was applied with success in medical image alignment. However, this approach is not applicable to natural language input, as it it not clear how to apply the blurring operation the input text.
127
128
129
130
131
132
133
134
135
136
137
139
140
141
142
143
145
Different from Burduja et al. [9 and Karras et al. 20,, our approach is applicable to various domains, including but not limited to natural language processing, as demonstrated throughout our experiments. To the best of our knowledge, the only competing model-level curriculum method which is applicable to various domains is CBS 21] Unlike CBS, LeRaC does not introduce new operations, such as smoothing with Gaussian kernels, during training. As such, our approach does not increase the training time with respect to the conventional training regime, as later shown in
146
147
149
3

---

151
the experiments. In summary, we consider that the simplicity of our approach comes with many important advantages: applicability any domain and task, compatibility with any neural network
152
153
architecture, time efficiency (adds no extra training time). We support all these claims through the
154
comprehensive experiments presented in Section
Method
155
Deep neural networks are commonly trained on set of labeled data samples denoted as:
S = { Ma E X, y E Y, E 21, m.})
(I)
157
where m is the number of examples, Ty is a data sample and y, is the associated label. The training process of neural network f with parameters 0 consists of minimizing some objective (loss) function C that quantifies the differences between the ground-trut -ruth labels and the predictions of the model f:
159
1 m
min
(Vi. f(,, 0))
180
The optimization is generally performed by some variant of Stochastic Gradient Descent (SGD), where the gradients are back-propagated from the neural layers closer to the output towards the neural layers closer to input through the chain rule. Let /1, /2. ..... fn and 01, 02. .... On, denote the neural layers and the corresponding weights of the model f. such that the weights 0, belong to the layer
161
162
163
184
fy. Vj E {1.2...., ...,, nJ. The output of the neural network for some training data sample E X is
165
formally computed as follows:
U 0) fn (-. (f1 (z,, 02) ...... 0n).
(3)
166
To optimize the model via SGD, the weights are updated as follows:
0g++1) - 0 1 (
Oc
0050 Vj (1,,,...,)
(4)
where is the index of the current training iteration, () 0 is the learning rate at iteration t, and the gradient of L with respect to o5²) is computed via the chain rule. Before starting the training process.
167
188
the weights 0 are commonly initialized with random values.
189
170
Due to the random initialization of the weights, the information propagated through the neural model during the early training iterations
contain large amount of noise (21) which can negatively impact the learning process. Due to the feed-forward processing, we conjecture that the noise level tends grow with each neural layer, from f, fJ+1- The same issue can occur if the weights are pre-trained on a distinct task, where the misalignment of the weights with a new task is likely higher
172
173
174
175
for the highllevel feature layers. To alleviate this problem, we propose to introduce curriculum learning strategy that assigns a different learning rate N7 to each layer fy, as follows:
178
0f(+1) = 8(())
OC
nj σg(i)
Vj E
(5)
177 such that:
() 2 1/ 1/2
(0) (0)
> ... 2 1/11
(6)
(A) m 172 = nn
(R)
(k)
where (0) are the initial learning rates and n/ (k) are the updated learning rates iteration k. The
179
180
condition formulated in Eq. (6) indicates that the initial learning rate n7 of neural layer fy gets lower as the level of the respective neural layer becomes higher (farther away from the input). With each training iteration / S k. the learning rates are gradually increased, until they become equal, according to Eq. 77) Thus, our curriculum learning strategy is only applied during the early training iterations, where the noise caused by the random weight initialization is most prevalent. Hence, is a hyperparameter of LeRaC that is usually adjusted such that k < T, where T' is the total number of
181
182
184

---

186
training iterations. In practice, we obtain optimal results by running LeRaC up to any epoch between
187
2 and
188 We increase each learning rate fij from iteration O to iteration k using an exponential scheduler that is based on the following rule:
189
(0) (1) nij"" c -110,, 1. "1" llgo 1 n VI ko.
(8)
(8) across all our experiments. In practice, we obtain optimal results by initializing the lowest learning rate yn²" with a value that is around five or six orders magnitude
We set 10 in Eq.
191
(0) is usually equal to 17(") Apart from these general lower than 17(0), while the highest learning rate V7j
192
193
practical notes, the exact LeRaC configuration for each neural architecture is established by tuning
194
the hyperparameters on the available validation sets.
Experiments
195
4.1
Datasets
197
In general, we adopt the official data splits for the eight benchmarks considered in our experiments.
198
When a validation set is not available, we keep 10% of the training data for validation.
199
CIFAR-10. CIFAR-I0 [22 is popular dataset for object recognition in images. It consists of 60,000
200
color images with resolution of 32 32 pixels. An images depicts one of 10 object classes, each class having 6,000 examples. We use the official data split with training set of 50,000 images and
202
test set of 10,000 images.
1-00 The CIFAR-100 [22] dataset is similar to CIFAR-10 10, except that it has 100 classes with 600 images per class. There are 50,000 training images and 10,000 test images. 204
203
205
Tiny ImageNet. Tiny ImageNet is a subset of ImageNet [23] which provides 100,000 training images, 25, 000 validation images and 25, ,0000 test images representing objects from 200 different classes. The size of each image is 64 x 64 pixels.
207
208
BoolQ. BoolQ [24] is question answering dataset for yes/no questions containing 15,942 examples. The questions are naturally occurring, being generated in unprompted and unconstrained settings. Each example is triplet of the form: [question, passage, answer]. We use the data split provided in
209
210
211
the SuperGLUT benchmark [59], containing 9,427 examples for training, 3,270 for validation and
3.245 for testing.
QNLI The QNLI (Question-answertt) NLI) dataset 25 is natural language inference benchmark automatically derived from SQuAD [60]. The dataset contains [question, sentence] pairs and the task to determine whether the context sentence contains the answer the question. The dataset is constructed on top of Wikipedia documents, each document being accompanied, on average, by
214
215
216
217
4 questions. We consider the data split provided the GLUE benchmark 2.5 which comprises
218
(04,743 examples for training, 5,463 for validation and 5, 5,463 for testing.
natural language inference dataset containing pairs of sentences with the target label indicating if the meaning of one sentence can be inferred from the other. The training subset includes 2,490 samples, the validation set 277, and the test set 3,000 examples.
RTE. Recognizing Textual Entailment (RTE) 251
220
222
CREMA-D. The CREMA-D multi-moda database (27 is formed of 7,442 videos of 91 actors (48 male and 43 female) of different ethnic groups The actors perform various emotions while uttering 12 particular sentences that evoke one of the 6 emotion categories: anger, disgust, fear, happy, neutral, and sad. Following [54] we conduct experiments only on the audio modality, dividing the set of audio samples into 70% for training, 15% for validation and 15% for testing.
224
226
228
seconds each, comprising 50 classes of various common sound events. Samples are recorded at a 44.1 kHz sampling frequency. with single channel. our evaluation, we employ the 5-fold cross-validatis validation procedure, as described
ESC-50. The ESC-50 (26) dataset is a collection of 2,000 samples
230
in related works (26] 31].
5

---

Image preprocessing. For the image classification experiments, we apply the same data preprocessing approach as Sinha et al. 21 Hence, we normalize the images and maintain their original resolution, 32 32 pixels for 10 and CIFAR-100. and 64 64 pixels for Tiny ImageNet. Similar to Table 1: Optimal hyperparameter settings for the various neural architectures used in our experiments.
Architecture
Optimizer Mini-batch #Epochs
17!0))
CBS
LeRaC
0
d
u
k
7 T)
(0)
[0]
R8
SGD
64
100-200
10-1
0.9
2-5
5-7
10 10-1-10- 10-8
Rede-ResNet-50
SGD
64
100-200
10-1
0.9
2-5
5-7
10-1 10-
CvT-13
Adamax
64-128
150-200
2-10-3
0.9
2-5
2-5
2-10-3
CvT-13 pre trained
Adamax
64-128
25
5-10-1
0.9
2-5
3-6
5-10-1-5-10-10
BERT large-uneased
Adamax
10
7-25
5-10-
0.9
1
3
5-10-
LSTM
AdamW
256-512
25-70
10
0.9
2
3-4
10-3.
SepTR
Adam
2
50
10-4
0.8
0.9
1-3
2-5
10-4 10-8
DenseNet-121
Adam
64
50
10-1
0.8
0.9
1-3
2-5
232
4.2
Experimental Setup
233
Architectures. To demonstrate the compatibility of LeRaC with multiple neural architectures, we
234
select several convolutional, recurrent and transformer models. As representative convolutional neural networks (CNNs), we opt for ResNet-18 18 [4). Wide-ResNet--3 (28) and DenseNet-121 [29].
236
As representative transformers, we consider CvT-13 [8) BERT uncased-large (2] and SepTr [31. For
CvT, consider both pre-trae pre-trained and randomly initialized versions. We use uncased large pre-
238
trained version of BERT. As Ristea et al. B1,, we train SepTr from scratch. In addition, we employ long short-term memory (LSTM) network 30 represent recurrent neural networks (RNNs).
240
The recurrent neural network contains two LSTM layers, each having a hidden dimension of 256 components. These layers are preceded by one embedding layer with the embedding size set 128 elements. The output of the recurrent layers is passed to a classifier comprised of two fully connected layers. The LSTM is activated by rectified linear units (ReLU). We apply the aforementioned models on distinct input data types, considering the intended application domain of each mode[" Hence, R 18, Resee-0 and CvT applied on images, BERT and LSTM are applied on text, and Sep'rr and DenseNet-121 are applied on audio.
242
244
245
246
Baselines. We compare LeRaC with two baselines: the conventional training regime (which uses carly stopping and reduces the learning rate on plateau) and the state-of-the-as Curriculum by Smoothing (21). For CBS, to ensure the replicability of their method in our experimental settings, which include more diverse selection of input data types and neural architectures.
248
the official code released by Sinha et al. (2I) at htttts::/giiiee
250
com/pairlabe
252
Hyperparameter tuning. We tune all hyperparameters on the validation set of each benchmark. In T we present the optimal hyperparameters chosen for each architecture. In addition to the standard parameters of the training process, we report the parameters that are specific for the CBS and LeRaC strategies. In the case of CBS, denotes the standard deviation of the Gaussian kernel, d
254
256 is the decay rate for o,, and u is the decay step. Regarding the parameters of LeRaC, l represents the number of iterations used in Eq. (8) and 1/1
(0) and (0) are the initial learning rates for the first
257
and last layers of the architecture, respectively. We underline that 1/1 = ()) and c = 10, in all experiments. Moreover, 71 (k) ())), i.c. the initial learning rates of LeRaC converge the original learning rate set for the conventional training regime. All models are trained with early stopping and the learning rate is reduced by factor of 10 when the loss reaches plateau.
258
280
262
Evaluation. We evaluate all models in terms of the classification accuracy. We repeat the training process of each model for 5 times and report the average accuracy and the standard deviation.
263
264
285
286
287
Sinha et al. 21, we do not employ data augmentation.
TThe only exception is DenseNet- 1211, which is applied audio instead of image data.

---

Table 2: Average accuracy rates (in %) over runs on CIFAR-10, 10, CIFAR- 100 and Tiny ImageNet for various neural models based on different training regimes: conventional, CBS (2I] and LeRaC. The accuracy of the best training regime each experiment highlighted in bold.
Architecture
Training Regime
CIFAR- 10
CIFAR- 100
Tiny ImageNet
ResNet-18
conventional
89.20 = 0.43
65.28 = 0.16
57.41 + 0.05
ResNet-18
CBS [2T
89.53 = 0.22
66.41 = 0.21
55.49 + 0.20
ResNet-18
LeRaC (ours)
89.56 + 0.16
66.02 + 0.17
57.86 + 0.20
Wide-ResNeet-51
conventional
91.22 = 0.24
68.14 + 0,16
55.97 = 0.30
Wide-ResNet-50l
CBS [2]
89.05 + 1.00
65.73 0.36
48.30 + 1.53
Wide-ResNet-50
LeRaC (ours)
91.58 + 0.16
69.38 + 0.26
56.48 + 0.60
CvT-13
conventional
71.84 0.37
41.87 0.16
33.38 + 0.27
CvT-13
CBS 21
72.64 1 0.29
44.48 0.40
33.56 + 0.36
CvT-13
LeRaC (ours)
72.90 + 0.28
43.46 0.18
33.95 + 0.28
CvT-13 pre-trained
CvT-13pre-urined
conventional
93.56 0.05
0.16
70.71 + 0.35
pre-treeedd CvT-13pre pre trained
CBS 22
85.85 = 0.15
12.35=0.45
68.41 + 0.13
CvT-13 117-133re.traicee trained
LeRaC (ours)
94.15 + 0.03
78.93 + 0.05
71.34 + 0.08
Table Average accuracy rates (in %) over runs on BoolQ, RTE and QNLI for BERT and LSTM based on different training regimes: conventional, CBS [21] and LeRaC. The accuracy of the best training regime in each experiment is highlighted in bold.
Architecture
Training Regime
BoolQ
RTE
QNLI
BERT large-uncused
conventional CBS [2]
74.12 0.32
74.48 = 1.36
92.13 = 0.08
BERT large-uneused
74.37 + 1.11
74.97 + 1.96
91.47 = 0.22
BERT largo-uneased
LeRaC (ours)
75.55 + 0.66
75.81 = 0.29
92.45 0.13
LSTM
conventional
64.40 1.37
54.12 + 1.60
59.42 - 0.36
LSTM
CBS 22
64.75 + 1.54
54.03 0.45
59.89 0.38
LSTM
LeRaC (ours)
65.80 + 0.33
55.71 1.04
59.98 + 0.34
268
Text preprocessing. For the text classification experiments with BERT, we lowercase all words and add the classification token ([CLS]) at the start of the input sequence. We add the separator token ([SEP]) to delimit sentences. For the LSTM network, we lowercase all words and replace them with indexes from vocabularies constructed from the training set. The input sequence length is limited to 512 tokens for BERT and 200 tokens for LSTM.
289
270
271
272
273
Speech preprocessing. We transform each audio sample into a time-frequency matrix by computing the discrete Short Time Fourier Transform (STFT) with N₂ FFT points, using a Hamming window of length L and hop size Rt For CREMA-D, we first standardize all audio clips fixed dimension of 4 seconds by padding or clipping the samples. Then, we apply the STFT with N, = 1024,
274
275
276
277
R- 64 and a window size of L 512. For ESC-50, we keep the same values for N₂ and L, but we increase the hop size to R = 128. Next, for each STFT. we compute the square root of the magnitude and map the values 128 Mel bins. The result is converted to a logarithmic scale and normalized to the interval [o, 1]. Furthermore, in all our speech classification experiments, we use the following data augmentation methods: noise perturbation, time shifting, speed perturbation, mix-up and Spec Augment [6]. The speech preprocessing steps are carried out following Ristea et al. [31.
278
279
281
282
283
4.3
Results
284
Image classification. In Table we present the image classification results on CIFAR-10, CIFAR- 100 and Tiny ImageNet. On the one hand, there are two scenarios (ResNet-18 on CIFAR-100 and CvT-13 on CIFAR-100)
285
286
which CBS provides the largest improvements over the conventional regime, surpassing LeRaC in the respective cases. On the other hand, there are seven scenarios
288
where CBS degrades the accuracy with respect to the standard training regime. This shows that the improvements attained by CBS are inconsistent across models and datasets. Unlike CBS, our strategy
7

---

Table 4: Average accuracy rates (in %) over runs on CREMA-D and ESC-50 for SepTr and DenseNet-121 based on different training regimes: conventional, CBS 21 and LeRaC. The accuracy of the best training regime each experiment highlighted in bold.
Architecture
Training Regime
CREMA-D
ESC-50
SepTr
conventional
70.47 0.67
91.13 = 0.33
SepTr SepTr
CBS [2]]
69.98 + 0.71
91.15 = 0.41
LeRaC (ours)
70.95 + 0.56
91.58 0.28
DenseNet- 121
conventional
67.21 + 0.12
88.91
DenseNet- 121
CBS [2]
68.16 + 0.19
88.76 0.17
DenseNet-122 121
LeRaC (ours)
68.99 + 0.08
90.02
Table 5: Average accuracy rates (in %) over 5 runs on CIFAR-10, CIFAR-100 and Tiny ImageNet for CvT-13 based on different training regimes: conventional, CBS 21], LeRaC with linear update, LeRaC with exponential update (proposed), and combination of CBS and LeRaC.
Architecture
Training Regime
CIFAR-10
C1FAR-100
Tiny ImageNet
CvT-13
conventional
71.84 0.37
41.87 + 0.16
33.38 + 0.27
CvT-13
CBS [2]]
72.64 0.29
44.48 + 0.40
33.56 + 0.36
CvT-13
LeRac (linear update)
72.49 0.27
43.39
33.86 + 0.07
CvT-13
LeRaC (exponential update)
72.90 + 0.28
43.46+
33.95 + 0.28
CvT-13
CBS 21 LeRaC
73.25 + 0.19
+
34.20 + 0.61
290
surpasses the baseline regime in all twelve cases, thus being more consistent. In four of these cases, the accuracy gains of LeRaC are higher than 1%. Moreover, LeRaC outperforms CBS in ten out of twelve cases. We thus consider that LeRaC can be regarded as a better choice than CBS, bringing consistent performance gains.
292
293
294
Text classification. In Table we report the text classification results on BoolQ, RTE and QNLI Here, there are only two cases (BERT QNLI and LSTM on RTE) where CBS leads performance drops compared to the conventional training regime. In all other cases, the improvements of CBS are below 0.6%. Just as in the image classification experiments, LeRaC brings accuracy gains for each and every model and dataset. In four out of six scenarios, the accuracy gains yielded by LeRaC are higher than 1.3%. Once again, LeRaC proves to be the best and most consistent regime, generally outperforming CBS by significant margins.
296
298
289
300
Speech classification. In Table 4 present the results obtained on the audio data sets, namely CREMA-D and ESC-50. We observe that the CBS strategy obtains lower results compared with the baseline in two cases (SepTr on CREMA-D and DenseNet-121 on ESC-50), while method provides superior results for each and every case. By applying LeRaC on SepTr,, we set a new oftte-of-theaane art accuracy level (70.95%) the CREMA-D audio modality, surpassing the previous state-of-the the-art value attained by Ristea et al. [31] with SepTr alone. When applied on DenseNet- 121, LeRaC brings performance improvements higher than 1%, the highest improvement (1.78%) over the baseline being attained on CREMA-D.
302
304
306
308
Additional results. An interesting aspect worth studying is to determine if putting the CBS and LeRaC regimes together could bring further performance gains. Across all our experiments, we identified single model (CvT-13) for which both CBS and LeRaC bring accuracy gains all datasets (see Table 2). We thus consider this model to try out the combination of CBS and LeRaC. The corresponding results shown in Table The reported results show that the combination brings accuracy gains across all three datasets (CIFAR-10, CIFAR-100, Tiny ImageNet). We thus conclude that the combination of curriculum learning regimes is worth try, whenever the two independent regimes boost performance.
310
312
314
316
Another important aspect is to establish if the exponential learning rate update proposed in Eq. 8 is
318 a good choice. To test this out, we keep the CvT-13 model and change the LeRaC regime to use linear update of the learning rate. We observe performance gains with both types of update rules,
8

---

60
Validation accuracy (9)
50
Validation accuracy (%)
40
40
30
30
Mue
20
20
10
+ CBS 21
CBS 21
LeRaC (ours)
LeltaC (ours)
0
0.5
1.5
2
2.5
3
6
8
10
Training time (h)
Training time (h)
ResNet-18 Tiny ImageNett
Wide-ResNet-Bee Tiny ImageNet.
80
80
Validation accuracy (%
Validation accuracy (%)
75
70
50
40
60
[21]
30
[2]
LeRaC (ours)
LeRaC (ows)
55
20
0.8
1.2
10
Training time (h)
Training time (h)
(c) BERT on BoolQ.
(d) SepTr CREMA-D.
Figure 1: Validation accuracy (on the y-axis) versus training time (on the x-axis) for four distinct architectures. The number of training epochs is the same for both LeRaC and CBS, the observable time difference being caused by the overhead of CBS due to the Gaussian kernel layers.
but exponential learning rate update conclude that the update rule defined in Eq. 8 is sound option.
bring higher gains all three datasets. We thus
321
Training time comparison. For particular model and dataset, all training regimes are executed for the same number of epochs, for a fair comparison. However, the CBS strategy adds the smoothing operation at multiple levels inside the architecture, which increases the training time. To this end, we compare the training time (in hours) versus the validation error of CBS and LeRaC. For this selected four neural models and illustrate the evolution of the validation accuracy over time in Figure [] We underline that I LeRaC introduces faster convergence times, being around 7-12% faster than CBS. It is trivial to note that LeRaC requires the same time the conventional regime.
323
325
experiment,
327
329
Conclusion
330
In this paper, we introduced a novel model-level curriculum learning approach that is based on starting the training process with increasingly lower learning rates per layer, as the layers get closer the output. We conducted comprehensive experiments on eight datasets from three domains (image, text and audio), considering multiple neural architectures (CNNs, RNNs and transformers),
332
333
334
335
to compare our novel training regime (LeRaC) with a ofate-of-the-ar regime (CBS (2T) as well as
336
the conventional training regime (based on early stopping and reduce on plateau). The empirical results demonstrate that LeRaC significantly more consistent than CBS, perhaps being the most versatile curriculum learning strategy to date, due to its compatibility with multiple neural models and usefulness across different domains. Remarkably, these benefits come for free, LeRaC
337
339
does not add any extra time over the conventional approach.

---

References
341
312
[1]
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, "Language Models are Few-Shot Learners," " in Proceedings of NeurlPS, 2020, vol. 33, pp. 1877-1901.
344
346
347
348
349
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." in Proceedings of NAACL, 2019, pp. 4171-4188
350
351
352
[3]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," in Proceedings of ICLR, 2021.
353
355
356
[4] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun, "Deep Residual Learning for Image Recognition," in Proceedings of CVPR, 2016, pp. 770-778.
358
[5] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah, "Transformers in Vision: A Survey," ACM Computing Surveys, 2021.
359
380
66] Zhuang Liu, Hanzi Mao, Chao-Yual Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. "A ConvNet for the 2020s," in Proceedings of CVPR, 2022.
381
362
[7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, "Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer," Journal of Machine Learning Research, vol. 21, no. 140, pp. 1-67, 2020.
363
364
[8] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, "CvT: Introducing Convolutions to Vision Transformers," " in Proceedings of ICCV, 2021, pp. 22-31.
367
99] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai, "Deformable DETR: Deformable Transformers for End-to-End Object Detection," in Proceedings of ICLR, 2020.
370
371
[10]
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals. Mahyar Bordbar, and Nando de Freitas, "A Generalist Agent," " arXiv preprint 2XXi::2005006177 2022.
372
373
374
375
376
[11]
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, "Curriculum Learning." in Proceedings of ICMI., 2009, pp. 41-48.
377
378
[12] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe, "Curriculum learning: A survey," International Journal of Computer Vision 2022.
[13] Tom M. Mitchell, Machine Learning, McGraw-Hin New York, 1997.
[14] Yi Tay, Shuohang Wang, Anh Tuan Luu, Jie Fu, Minh C. Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang, "Simple and Effective Curriculum GennterrGenerator Networks for Reading Comprehension over Long Narratives," in Proceedings of ACL, 2019, pp. 4922-4931.
382
383
384
385
[15] Wei Zhang, Wei Wei, Wen Wang, Lingling Jin, and Zheng Cao. "Reducing BERT Computation by Padding Removal and Curriculum Learning." " in Proceedings of ISPASS, 2021, pp. 90-92.
386
387
[16] Miaojing Shi and Vittorio Ferrari, "Weakly Supervised Object Localization Using Size Esti- mates." " in Proceedings of ECCV, 2016. pp. 105-121.
10

---

389
[17] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. "Curriculum self- self-pacee paced learning for cross-domain object detection," Computer Vision and Image Understanding. vol. 204, pp. 103-166, 2021.
390
[18] Xin Wang, Yudong Chen, and Wenwu Zhu, "A Survey on Curriculum Learning." IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
393
394
[19] Mihail Burduja and Radu Tudor Ionescu, "Unsupervised Medical Image Alignment with Curriculum Learning." in Proceedings of ICIP, 2021, pp. 3787-3791.
395
396
[00] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen, "Progressive Growing of GANs for Improved Quality, Stability, and Variation," in Proceedings of ICLR, 2018.
[21] Samarth Sinha, Animesh Garg, and Hugo Larochelle, "Currieulum by smoothing," in Proceed- ings of NeurlPS, 2020, pp- 21 21653-21664.
389
400
[22] Alex Krizhevsky, "Learning multiple layers of features from tiny images," Tech. Rep., University of Toronto, 2009.
401
402
[23]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al., "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision, vol. 115, no. 3, pp. 211-252, 2015.
404
406
[24] Christopher Clark, Kenton Lee, Ming-We Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions," " in Proceedings of NAACL, 2019. pp. 2924-2936.
407
408
409
[25] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding." in Proceedings of ICLR, 2019.
411
412
[26] Karol J. Piczak, "ESC: Dataset for Environmental Sound Classification," " in Proceedings of ACMMM, 2015, pp. 1015-1018.
413
414
[27] Houwei Cao, David G. Cooper, Michael Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma, "CREMA-D: Crowd-sourced emotional multimodal actors dataset," IEEE Transactions on Affective Computing, vol. 5, no. 4, pp. 377-390, 2014.
415
416
417
[28] Sergey Zagoruyko and Nikos Komodakis, 1605 146, 2016.
"Wide Residual Networks," T arXiv preprint
419
[29] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger, "Densely Connected Convolutional Networks." in Proceedings of CVPR, 2017, pp. 2261-2269.
420
[30] Sepp Hochreiter and Jürgen Schmidhuber, "Long Short-Term Memory," Neural Computing, vol. 9, no. 8, pp. 1735-1780, 1997.
422
423
[1] Nicolae-Catee Catalin Ristea, Radu Tudor Ionescu, and Fahad Shahbaz Khan, "SepTr: Separable Transformer for Audio Spectrogram Processing." arXiv preprint arXiv:2203009581 2022.
424
425
[32] Liangke Gui, Tadas Baltrušaitis, and Louis-Philippe Morency, "Curriculum Learning for Facial Expression Recognition," in Proceedings of FG, 2017, pp. 505-511.
426
[33] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei, "MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels," in Proceedings of ICML,, 2018, pp.
428
430
[34] Xinlei Chen and Abhinav Gupta, "Webly Supervised Learning of Convolutional Networks." in Proceedings of ICCV, 2015, pp. 1431-1439.
431
[35] Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell, "Competence-basee curriculum learning for neural machine translation," " in Proceed- ings of NAACL, 2019, pp. 1162-1172.
433

---

435
[36]
Tom Koemi and Ondrej Bojar, "Curriculum Learning and Minibatch Bucketing in Neural Machine Translation," in Proceedings of RANLP, 2017. pp. 379-386.
436
437
[37]
Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky, "Baby steps: How "less is more" in unsupervised dependency parsing," in Proceedings of NIPS, 2009.
439
[38] Cao Liu, Shizhu He, Kang Liu, and Jun Zhao, "Curriculum Learning for Natural Answer Generation," in Proceedings of IJCAI, 2018, pp. 4223-4229.
440
[39] Shivesh Ranjan and John H. I.. Hansen, "Curriculum Learning Based Approaches for Noise Robust Speaker Recognition," " IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, pp. 197-210, 2018.
442
[40]
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg. Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, Jie Chen, Jingdong Chen, Zhijie Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Ke Ding. Niandong Du, Erich Elsen, Jesse Engel. Weiwei Fang, Linxi Fan, Christopher Fougner, Liang Gao, Caixia Gong, Awni Hannun, Tony Han, Lappi Vaino Johannes, Bing Jiang. Cai Ju, Billy Jun, Patrick LeGresley, Libby Lin. Junjie Liu, Yang Liu, Weigao Li, Xiangang Li, Dongpeng Ma, Sharan Narang, Andrew Ng, Sherjil Ozair. Yiping Peng, Ryan Prenger, Sheng Qian, Zongfeng Quan, Jonathan Raiman, Vinay Rao, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Kavya Srinet. Anuroop Sriram, Haiyuan Tang, Liliang Tang, Chong Wang, Jidong Wang. Kaifu Wang. Yi Wang. Zhijian Wang. Zhiqian Wang, Shuang Wu, Likai Wei, Bo Xiao, Wen Xie, Yan Xie, Dani Yogatama, Bin Yuan, Jun Zhan, and Zhenyao Zhu, "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin," in Proceedings of ICML. 2016, pp. 173-182.
445
447
448
449
450
451
452
453
454
455
[41] Anastasia Pentina, Viktoriaa Sharmanska, and Christoph H. Lampert, "Curriculum learning of multiple tasks," in Proceedings of CVPR, June 2015, pp. 5492-5500.
457
458
[42] Amelia Jiménez-Sánchez, Diana Mateus, Sonja Kirchhoff, Chlodwig Kirchhoff, Peter Bib- erthaler, Nassir Navab, Miguel A. González Ballester, and Gemma Piella, "Medical-based Deep Curriculum Learning for Improved Fracture Classificationn" in Proceedings of MICCAI, 2019,
459
480
461
pp. 694-702.
462
[43]
Jerry Wei, Arief Suriawinata, Bing Ren, Xiaoying Liu, Mikhail Lisovsky, Louis Vackkss, Charles Brown, Michael Baker, Mustafa Nasir-Moin, Naofumi Tomita, Lorenzo Torresani, Jason Wei, and Saeed Hassanpour, "Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification," in Proceedings of WACV, 2021, pp. 2472-2482.
464
486
[44] Volkan Cirik, Eduard Hovy, and Louis-Philippe Morency, "Visualizing and Understanding Cur- riculum Learning for Long Short-Term Memory Networks," arXiv preprint arXiv: 10111611200000 1.06204, 2016.
467
488
469
[45] Radu Tudor lonescu, Bogdan Alexe, Marius Leordeanu, Marius Popescu, Dim Papadopoulos, and Vittorio Ferrari, "How Hard Can It Be? Estimating the Difficulty of Visual Search in an Image," in Proceedings of CVPR, 2016, pp. 2157-2166.
471
472
[46] Chen Gong, Dacheng Tao, Stephen J. Maybank, Wei Liu, Guoliang Kang, and Jie Yang, "Multi- Modal Curriculum Learning for Semi-Supervised Image Classification," IEEE Transactions on Image Processing, vol. 25, no. 7. pp. 3249-3260, 2016.
473
474
475
[47] Guy Hacohen and Daphna Weinshall, "On The Power of Curriculum Learning in Training Deep Networks," in Proceedings of ICML, 2019, pp. 2535-2544
476
477
[48] M. Kumar, Benjamin Packer, and Daphne Koller, "Self-Paced Learning for Latent Variable Models," in Proceedings of NIPS, 2010, vol. 23, pp. 1189-1197.
479
[49] Maoguo Gong, Hao Li, Deyu Meng, Qiguang Miao, and Jia Liu. "Decomposition- based evolu- tionary multiobjective optimization to self-paced learning," IEEE Transactions on Evolutionary Computation, vol. 23, no. 2. pp. 288-302, 2019.
480
481
482
[50] Yanbo Fan, Ran He, Jian Liang, and Bao-Gang Hu, "Self-Paced Learning: An Implicit Regularization Perspective," in Proceedings of AAAI, 2017, pp. 1877-1883.
12

---

484
[51]
Hao Li, Maoguo Gong, Deyu Meng, and Qiguang Miao, "Multi-Obeective Self-Paced Learning." in Proceedings of AAAI, 2016, pp. 1802-1808.
485
486
[52] Sanping Zhou, Jinjun Wang, Deyu Meng, Xiaomeng Xin, Yubing Li, Yihong Gong, and Nanning Zheng. "Deep self-paced learning for person re-identification." Pattern Recognition, vol. 76, pp. 739-754 100--5,,220 2018.
488
489
[53] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann, "Self-Paced Curriculum Learning." in Proceedings of AAAI, 2015, pp. 2694-2700.
491
[54] Nicolac-Catalin Ristea and Radu Tudor Ionescu, "Self-paced ensemble learning for speech and audio classification," in Proceedings of INTERSPEECH, 2021, pp. 2836-2840.
492
[55] Fan Ma, Deyu Meng, Qi Xie, Zina Li, and Xuanyi Dong, "Self-paced co-training," in Proceedings of ICMI., 2017, vol. 70, pp. 2275-2284.
494
495
[56] Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander G. Haupt- mann, "Self-Pacer Pace- Learning with Diversity," in Proceedings of NIPS, 2014, pp. 2078-2086
[57]
Min Zhang, Zhongwei Yu, Hai Wang, Hongbo Qin, Wei Zhao, and Yan Liu, "Automatic Digital Modulation Classification Based on Curriculum Learning," " Applied Sciences, vol. 9, no. 10,
488
489
2019.
500
[58]
Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie- Yan Liu, "Learning to Teach with Dynamic Loss Functions," in Proceedings of NeurlPS, 2018, vol. 31, pp. 6467-
501
502
6478.
503
[59] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, "SuperGLUE:: A Stickier Benchmark for Perpral-Purpose Language Understanding Systems," in Proceedings of NeurIPS, 2019, vol. 32, pp. 3266-3280.
505
506
[60] Pranay Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, "SQuAD: 100,000+ Questions for Machine Comprehension of Text," in Proceedings of EMNLP, 2016, pp. 2383- 2392.
508
509
[61] Daniel S Park, William Chan, Yu Zhang, Chung-Chene Chiu, Barret Zoph, Ekin D Cubuk, and Quoe V Le. "Specaugment: A simple data augmentation method for automatic speech recognition," Proceedings of INTERSPEECH, pp. 2613-2617, 2019.
510
Checklist
512
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
514
516
517
Did you include the license the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] The code and the data proprietary.
519
520
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
522
1. For all authors...
525
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
528
527
(b) Did you describe the limitations of your work? [No] We did not identify any significant limitations of LeRaC.
13

---

529
(c) Did you discuss any potential negative societal impacts of your work? [No] We did not identify any negative societal impacts.
530
531
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
538
2. If you are including theoretical results...
534
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
535
(b) Did you include complete proofs of all theoretical results? [N/A]
536
3. If you ran experiments..
537
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] In the supplementary.
539
540
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Table 1.
542
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report average accuracy rates + standard deviations over runs.
544
545
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] For some cases, please see Figure 1.
547
548
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
549
(a) If your work uses existing assets, did you cite the creators? [Yes]
550
(b) Did you mention the license of the assets? [No] But we provided the download link of the used code.
(c) Did you include any assets either in the supplemental material or URL? [No] We will release our code after acceptance.
553
554
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
555
556
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
558
5. If you used crowdsourcing or conducted research with human subjects...
559
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
560
561
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
562
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
564
14


Reviews:
[{"review_id": "czGPqu3OEg6", "reviewer": "Reviewer_Z9wJ", "summary": "This paper proposes to modify neural net training by applying a different learning rate to each layer: the deeper layers are assigned a lower learning rate that progressively grows to match the first layer learning rate. ", "questions": "== Optimization vs curriculum learning ==\n\nThe proposed method uses a different learning rate per layer. This endeavor has been widely explored in the literature on optimization of neural nets while the name curriculum learning is usually employed for methods modifying the ordering of the examples during training. I agree that the name curriculum could make sense in your context but the paper needs to spend more time on previous work on optimization. This means expanding the related work section and the empirical analysis.\n\nThe Adadelta paper (Zeiler 2012) observes the benefit to use different learning rates for different layers and Adadelta results in \"effective learning rates [...] are larger for the lower layers of the network and much\nsmaller for the top layer at the beginning of training.\" Similar observations can be made for other adaptive learning rate algorithms like Adagrad, Adam, Adafactor, Adamax and AdamW or even later algorithms like e.g. SM3 (Memory-Efficient Adaptive Optimization, NeurIPS'19, see Figure 1.).\n\nTo justify your approach it seems necessary to compare your method at least with Adam and report the effective average learning rate (e.g. see Zeiler 2012's definition) per layer for both methods and how their parameters control the differentiation of learning rates across layers during training. The fact that your method is used in conjunction with Adam (or variants) could mean that either (i) the same type of learning rate schedule could be attained by changing Adam's parameters or (ii) your method's benefit comes from schedules unreachable by Adam. It seems very important to explore both methods' parameters to distinguish these two cases.\n\n== Benefit is faster training or better generalization? ==\n\nModifying the training algorithm requires reporting training curves with *both* validation and training loss over time. The paper does not report any training loss which left us wonder if the generalization benefit comes from better optimization (lower training and validation loss) versus a better generalization trajectory (reaching a lower validation loss at the same training loss). \n\n== Schedule Tuning ==\n\nCould you explain how the exponential scheduler of Eq. 8 has been selected? Is it the only option that you tried? How is the training speed affected by the choice of the additional hyperparameters (c, \\nu^{(0)}_n, k)? In particular, compared to a robust baseline like Adam, would most hyperparameter choices yield faster or slower training? I do not understand how c can be set to 10 without validation.\n\n== Number of training steps ==\n\nThe experimental section should spend more time describing how the number of training steps was selected. In which cases the baselines would reach the same or better validation loss with more steps? E.g. Figure 1.d. seems to suggest that better validation loss with CBS can be reached with more training. This improvement does not seem to agree with the sentence \"all models are trained with early stopping\", does it?\n\n== Justification of the noise argument ==\n\nAdaptive methods (Adam etc) justify higher learning rates for first layers with a curvature argument: in laymans term, to get the same impact on the output the first layers' updates require a larger learning rates than the last ones. Adadelta (Zeiler 2012) also mentions the larger learning rate in light of the vanishing gradient effect (Bengio 2014). Your motivation (from [21]) is to reduce the effect of \"noise due to untrained parameters\". This argument is unclear to the reader and it would help if you could find a way to measure/characterize that \"noise\" and show how your method reduces its effect.\n", "limitations": "This work does not raises concerns on societal impact. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear, the approach is simple and can be implemented easily. The experiments shows better generalization when adopting the proposed learning rate schedule. The paper could be improved on two main axes:\n(i) presenting the approach as an optimization method seem more appropriate than the curriculum learning angle, (ii) more analysis and justifications of why the method works seem necessary.\nOverall, I feel that the paper does not explain how/if the proposed differentiated learning rate schedule is different from the effective learning rates reached by different settings of adaptive optimizers (momentum, Adadelta, Adam, SM3...).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TMmDnXOtthQ", "reviewer": "Reviewer_oPup", "summary": "The manuscript proposes a model-level curriculum learning framework that uses variable learning rates across a DNN to jump-start the learning process. The core idea revolves around compensating for the difference in learning across the depth of a DNN by using larger learning rates closer to the input and gradually decreasing their value towards the output layers. As intended, this curriculum is data-free and removes the need to assess the difficulty of samples, similar to how data-level curriculum operates, and instead can be applied across a range of architectures and tasks. The proposed idea improves on the performance of standard training regimes across vision, language and audio tasks, using residual, recurrent and transformer-based architectures.", "questions": "- Specific to the vision datasets and results, could the authors provide insights into (a) how the use of common preprocessing techniques affects the behavior of the proposed approach, and (b) the balance in peak performance achievable in standard training vs. the proposed method when preprocessing is included. While preprocessing techniques add some element of stochasticity, it is important to understand an algorithm in that context.\n- The addition of significance tests/metrics would be beneficial to further emphasize the benefits of the proposed algorithm as well as other areas for potential improvement. Could the authors reassess their results using significance metrics?\n- Moving a small portion of the experimental setup to the appendices/supplementary material and including a study of how k (iterations/epochs), from small to absurdly large, affects the improvement in performance offered by the proposed algorithm would add another dimension to the evaluations presented in this work. The results themselves should be a by-product of the suite of experiments already executed so hopefully there is minimal overhead.\n- Given the variable levels of improvement, even within the scope of the same task, could the authors discuss plausible schemes that take into account other factors (like data, gradient flow, etc.) that could provide more consistent gains?\n- At a more abstract level could the authors address the potential negative implications/limitations of their approach.", "limitations": "While there aren't glaring limitations or issues with the proposed method, the authors can consider potential negative effects in alternative properties of the DNN, as a by-product.\nSince there isn't an evaluation of the calibration quality or adversarial robustness of the final solutions, these could be potential issues or limitations to the proposed algorithm. \nKeeping robust/adversarial training or alternative objective functions to classification in mind could lead to unexpected outcomes.\n", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The writing offers a clear and understandable representation of the curriculum learning approach.  More importantly, the categorization of various levels of curriculum learning provides important context to the ideas being discussed.\n- The depth of experimental evaluation, stretching across datasets, tasks and architectures is commendable.\n\nWeaknesses\n- Data augmentation has proven valuable to the training process, be it commonly used approaches like random cropping/flipping or more intricate mixed approaches. They offer substantial improvement in performance and as well as alternative properties of a DNN. The proposed method is not evaluated alongside any common preprocessing techniques (vision datasets), thus offering a gap in the exploration of the behavior of the proposed algorithm alongside them.\n- The set of results posted across various tables in the manuscript is commendable. However, in certain cases the the average performance and standard deviation across multiple methods are extremely close. While this does not take away from the final conclusions, the use of significance tests/metrics could help further distinguish improvements.\n- The descriptions of the preprocessing and training regimes offer extremely important context to the results presented throughout the paper.\nHowever, an exploration of the two key parameters that make up the proposed method, various update setting for the learning rate and the choice of iteration/epoch at which to converge to the standard learning rate would offer more insight into the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SDetO11sdpE", "reviewer": "Reviewer_YyP2", "summary": "This paper proposes setting different learning rates for each layer of a deep neural network such that the first layers (closer to the input) have a much higher learning rate than the last layers. Learning rates are all gradually increased until they reach the same value $\\eta_0$ (the optimal learning rate of no-curriculum models). ", "questions": "- What is the method for optimizing $\\eta_j$, j = {1 ... n}?\n- How sensitive is the model to different settings of $\\eta_j$?\n- How do the training dynamics compare to standard training where all layers are trained at the same rate? Would the earlier layers generate qualitatively different features maps than under-trained later layers?", "limitations": "- The interaction of the model with adaptive optimizers that scale the learning rate of each parameter is not well justified. \n- Learning rate schedulers may disturb the proposed model. If the base learning rate is reduced before the learning rate of the last layer $\\eta_n$ is sufficiently increased, the last layer may never be trained enough, training may take much longer or may become unstable.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simple, novel, and effective method. \n- Good review of related work and reasoning for using model-level curriculum learning.\n- Consistent performance improvements across eight audio, image, and text datasets and seven convolutional, recurrent, and transformer architectures. \n\n\nWeaknesses:\n- Insufficient analysis on why the method works. \n- The motivation for setting different learning rates per layer is not sound. The authors state that random initialization of parameters causes the propagation of noise during the forward pass, and that assigning higher learning rate to first layers and lower learning rates to last layers prevents this propagation of noise. But there is no proof for this argument.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Ts1fDU6UZ", "reviewer": "Reviewer_mKJ3", "summary": "The paper proposes a kind of model-level curriculum learning strategy, which assigns higher initial learning rates to shallow layers than deep ones and continues increasing all learning rates until they reach the same value during the training process. It is a model- and task-agnostic approach. To verify its effectiveness, the authors apply it to multiple domains and different neural networks.", "questions": "1. There are many hyperparameters of this strategy, so I’d like to know how to choose proper initial learning rates for all layers and how does the sensitivity of hyperparameters affect the experimental results?\n2. For pre-trained models, this paper states that “The same issue can occur if the weights are pre-trained on a distinct task, where the misalignment of the weights with a new task is likely higher for the high-level feature layers. ” However, it is generally believed that the learning rates of the layers close to the outputs are relatively high and the learning rates of the layers close to the inputs are low or even equals to zero in the process of fine-tuning, which seems to be the opposite of the strategy proposed in this paper. How to explain the difference between them?\n3. It may be easier to understand Eq. (8), if the authors can plot a figure for μ(j).\n4. Whether assigning higher learning rate to the deeper layers can also obtain better performance through hyper-parameter tuning ? The learning landscape of the models is not clear.", "limitations": "The authors do not discuss the limitation. However, there is a lack of both an intuitive explanation or theoretical analysis for the proposed method, making it hard to be a convincing work. A large line of related works is missed, making the contributions vague.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.The proposed strategy concerning layer-wise learning rate is important.\n\n2.The empirical experiments are sufficient and comprehensive to prove the generality and effectiveness of the strategy.\n\n3.This paper is well written and easy to read.\n\nWeaknesses:\n\n1.The relation between this work and curriculum learning is vague.  Curriculum learning is a training strategy that learns from easy to hard, or more generally, in a certain kind of meaningful order. For example, CBS, which is mentioned in the paper, anneals the standard deviation of Gaussian kernels to pass more high-frequency information. However, the relation between this work and CL is not so clear.\n\n2.The discussed related work is not sufficient. Since this work focuses on the layer-wise learning rate, it should be discussed more clearly about how this work is different from the works about layer-wise(or adaptive) learning rate[1][2][5], which is a common and widely adopted strategy. Additionally, training with increasing learning rate in the early stages is a kind of warmup[3][4], the relations between this work and warmup strategies are required to discussed.  The vague position of this work makes its contribution vague.\n\n3.This paper lacks theoretical analysis and does not explain why increasing the learning rates until reaching the same value is reasonable. It is not convincing only with the empirical results. The designed mechanism is neither intuitive nor well theoretically supported, making its justification questionable. The only way for justification is through experiments, which may be tricky, because the proposed method needs more hyper-parameters than that of the conventional methods.\n\n[1] Singh, B., De, S., Zhang, Y., Goldstein, T., & Taylor, G. (2015, December). Layer-specific adaptive learning rates for deep networks. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA) (pp. 364-368). IEEE.\n\n[2] Ginsburg, B., Gitman, I., & You, Y. (2018). Large batch training of convolutional networks with layer-wise adaptive rate scaling.\n\n[3] Gotmare, A., Keskar, N. S., Xiong, C., & Socher, R. (2018). A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. \n\n[4] You, Y., Gitman, I., & Ginsburg, B. (2017). Large batch training of convolutional networks. \n\n[5] Adam:A Method for Stochastic Optimization. ICLR 2015\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject