Curriculum Discovery through an Encompassing Curriculum Learning Framework
Authors: Mohamed Elgaar
Hadi Amiri
Venue: ICLR 2022 Submitted (2022)
Keywords: curriculum learning
natural language processing

Abstract: We describe a curriculum learning framework capable of discovering optimal curricula in addition to performing standard curriculum learning. We show that this framework encompasses existing curriculum learning approaches such as difficulty-based data sub-sampling, data pruning, and loss re-weighting. We employ the proposed framework to address the following key questions in curriculum learning research: (a) What is the best curriculum to train a given model on a given dataset? (b) What are the characteristics of optimal curricula for different datasets and different difficulty scoring functions? We show that our framework outperforms competing state-of-the-art curriculum learning approaches in natural language inference and other text classification tasks. In addition, exhaustive experiments illustrate the generalizability of the discovered curricula across the three datasets and two difficulty scoring functions.

Under review as a conference paper at ICLR 2022
CURRICULUM DISCOVERY THROUGH AN ENCOMPASS- ING CURRICULUM LEARNING FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
We describe curriculum learning framework capable of discovering optimal curricula in addition to performing standard curriculum learning. We show that this framework encompasses existing curriculum learning approaches such difficulty-based data sub-sampling, data pruning, and loss re-weighting. We em- ploy the proposed framework address the following key questions in curriculum learning: (a) what is the best curriculum to train given model on given dataset? and (b) what are the characteristics of optimal curricula for different datasets and difficulty metrics? We show that our framework outperforms competing state- of-the-art curriculum learning approaches in natural language inference and two other text classification tasks. Exhaustive experiments illustrate the generalizabil- ity of the discovered curricula across the datasets and difficulty metrics.
INTRODUCTION
Curriculum Learning (CL) is a technique in Machine Learning that mimics human education sys- tems. To learn complex subject, students must first learn the foundational and basic materials before learning more complex ones. Without a curriculum, learning may be intractable, inefficient and learners may never reach full understanding of topic due to lack of the required background knowledge. Machine learning optimization through stochastic gradient descent trains models by ob- serving example data instances. Some data instances are harder than others and require background knowledge, which could be acquired by observing and being adept at casier examples before harder ones. CL techniques seek to order examples according to their difficulty for training to generate bet- ter models. It has been shown that CL improves performance in solving harder tasks, or in cases of limited or noisy data (Wu et al.] (02T). CL research has made significant progress in the last decade through the work of Bengio et al. (2009), although the principle of ordering training samples from easier to harder was introduced in the 1990s (Elman 1993] Sanger
1994 Rohde & Plaut 1999).
A curriculum can be defined by ordering training samples based on their difficulty to learn. Given measure of difficulty, there are different types of CL approaches for ordering the data: sub-sampling techniques, which sample the easiest or hardest data points at every iteration for training (Zhou et al. 2020 Bengio et al., 2009; Xu et al. 2020 Guo et al., 2018 Platanios et al., 2019). sample weighting techniques, which use the complete data at every iteration but weight data points differ- ently according to their difficulty (Castells et al., 2020: Kumar et al. 2010 Jiang et al. 2015 2018
Zhou et al. 2020; Yang et al.) 2019), and pruning techniques, which prune the hard or noisy samples from the dataset prior to training (Northcutt et al. 2021 Guo er al., 2018). Sub-sampling methods can be cumulative, exclusive or combination of both. Cumulative approaches add new data points to the ones that have previously been used for training. On the other hand, exclusive approaches create a new subset of data at every training stage. Such methods can introduce samples from easy to hard (Bengio et al. [209 Kumar et al.] (010) or hard to easy, which can be an effective learning strategy in some specific tasks (Koemi & Bojar 2017 7
Zhang et al. 2018 2019). addition, CL methods may impose curriculum by adjusting model's ss capacity according to the difficulty of their inputs (Karras et al.] 2018, Sinha et al.] 2020, Morerio et al. 2017) or schedule the order of tasks
in the context of multi-task learning (Caubrière et 2019| Sarafianos et al. 2017 Florensa et al.)
2017). Other CL approaches such as (Zhou et al. 2020 Saxena et al. 2019 use O (n), where n is the number of training samples, extra parameters for learning curricula.

---

Under review as a conference paper at ICLR 2022
1.0
1.0
0.8
0.8
Confidence
#15. 51
0.6
+10.55
0.4
0.4
1100 755
easy (50, 0)
0.2
med (50, .3)
hard (50, 6)
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Training Progress
Effect of varying the parameters.
(b) Easy Hard Curriculum.
The effect of the rate of growth and shift parameters (r, s). (a) illustrates different weight- ing strategies that can be obtained by varying the rate and shift parameters. (b) specifie parameter configuration for a curriculum that first introduces easier training samples to a model, and then gradually introduces medium and hard samples 30% and 60% the training epochs.
Figure 1:
Current CL approaches calculate difficulty scores for training samples based on the loss of a trained Wu et al., 202T), trainable parameters that weight samples (Kumar al, Castells et al. 2020), loss value during training (Wu et al. (2021) moving average loss during training (Zhou et al. 2020), transformations of the loss during training (Jiang et al. 2018 Castells et al. 2020), and consistency in the correct classification of samples (Amiri ct al.
model Xu al.,
2020;
2010
Jiang et al. 2015;
2017 Xu et al. 2020). The difficulty scores will then be used order samples for training.
alternative approach, where examples can be ordered based on prior knowledge about their difficulty, e.g., object shape or orientation image classification (Bengio al.) 2009). We will demonstrate CL framework that encompasses existing CL approaches through an effective and flexible data partitioning and weighting scheme. Our framework provides a new paradigm for selecting ordering strategy. Instead of petermeerdd strategy, the framework allows searching over the curriculum space to identify the best curriculum for a particular dataset and model. It partitions training data into several groups of training samples, e.g. {easy, medium, hard} samples, according to a difficulty scoring function. Parameterized weighting functions will then be defined for each data group to specify the weight its samples during training. Each weight function is controlled by two parameters, which can be set empirically or adjusted using Bayesian optimization. In addition, the framework discovers optimal curricula by optimizing the parame- ters of the weight functions (only parameters per function) using hyper-parameter optimization algorithm. Furthermore, the curricula identified through this search provide useful insight about the dataset, such as the relative importance of different samples knowledge dependency between samples, e.g. which samples should be learned first
In this work, we focus
We begin by explaining framework and showing how it is capable of approximating existing CL approaches. Then, in the context the proposed framework, we investigate curriculum discovery, characteristics of discovered curricula, and generalizability of curricula with respect to their datasets.
CURRICULUM LEARNING FRAMEWORK
2.1
WEIGHTING FUNCTIONS
We define the curriculum using generalized logistic functions (Richards 1959) of the form:
1
w(1r,,)) 1 expppppppp (t- s))
where r E R is the rate-of-change parameter, which specifies how fast the weight can increase (r 0) or decrease (r < 0); 10 is the training progress (iteration number divided by max iterations); and E R shifts the pivot weight of the logistics function (w(.) = .5) to the left or right such that t the weight is 0.5. Figure
illustrates the effect of varying these parameters.
Greater absolute values for the rate parameter enforce greater slope and faster rate of changes in weights, while greater values of the shift parameter enforce longer delays in reaching the pivot
2

---

Under review as a conference paper at ICLR 2022
2.0
Density
Density
Density
1.0
0.5
0.5 Entropy
1.0
2
0.0
0.5
1.0
1.5
Entropy
Entropy
(a) SNLI Entropy
(b) Alcohol Entropy
(c) Cancer Entropy
0.8
0.8
Density
0.8
Density
0.6
Density
0.6
0.2
0.2
0.2
0.0
0.0
0.0
10
0
10
10
Loss
Loss
Loss
(d) SNLI Loss
(c) Alcohol Loss
(l) Cancer Loss
Figure 2: Distributions of entropy and loss of the three datasets used in the experiments. Figures
(a) (c) show entropy, and (d) - (f) show loss. Samples of the easy class are to the left of the first vertical line, those the medium class are between the two vertical lines, and samples of the hard class are to the right of the second line.
weight of 0.5. These parameters provide flexibility in controlling sample weights during training, which critical for deriving effective curricula. Given the weighting function, we encode prior knowledge about sample difficulty into the training paradigm of our CL framework by splitting the data into several partitions of increasing difficulty according to the prior knowledge. Then, we define a weight function by pair of parameters (r, s) for each partition,
The proposed sample weighting framework provides flexibility in sample-ordering according to difficulty. It can be learned to approximate existing predetermind curricula. For example, it is possible begin with only easy instances or only difficult instances (as an anti-curriculum) or combination of both. Figure Tb shows a specific configuration for the logistic functions based on standard CL (Bengio et al., 2009 Kumar et al. 2010), where training starts with easier samples and gradually proceeds with harder ones. In addition, approach enables discovering data-driven curricula from data.
Prior to training, the difficulty scores of samples computed and each sample is assigned to a difficulty class e E {easy, medium, hard), see $2.2 addition, the hyper-parameters of three weight functions are optimized prior to training and kept fixed throughout training {(res Se), (rm, sm); (Th, 8p)}. During training, the weighted loss is computed as follows.
i, = w(t 8c)
Where 4 is the unweighted and instantaneous loss of instance i, I, the weighted loss, is the current training iteration divided by the maximum number of iterations, is the difficulty class of instance i, and (ray se) are the corresponding rate and shift parameters for the difficulty class c.
2.2 SCORING FUNCTIONS
Ground-truth labels for many datasets are often obtained through human annotation and crowd- sourcing. This is achieved by collecting multiple annotations per data sample and aggregating the results, typically by majority voting. We use sample-level annotator disagreement to define a dif- ficulty score for each sample using Shannon entropy (Shannon 2001). where higher disagreement among annotators corresponds to higher sample difficulty. Entropy is natural measure diffi- culty (for human population) and may serve reliable prior knowledge for partitioning data by
3

---

Under review as a conference paper at ICLR 2022
difficulty. Entropy of each sample 2; is calculated as (x,;) = - Pe log Pe where c is class category and Pe is the fraction of annotators who chose label c for the sample. The use of entropy is supported by [Nic al. 2020), who studied the correlation between human agreement and model performance and reported a consistent positive correlation between model accuracy and level of human agreement, showing that model performs better on samples with high level of agreement.
Furthermore, training loss contains valuable information about difficulty with respect to the model (learner), which may be different among architectures and tasks, and indicative of the model's spe- cific needs. However, loss at a particular step (e.g.. final loss) is dictated by the stochastic gradient descent and batihngtching dynamics and therefore is not good indicator of difficulty (Zhou al.) 2020 Wu et al. (02I). Using baseline model, trained with no curriculum and with default hyper- parameters, we collect the loss values of all training instances at intervals of 0.5 epochs and use the average loss to estimate sample difficulty. In our experiments, we obtain twenty observations of the loss and compute the average for each instance. Such an estimation is supported by /Zhou et al.] (020)) who showed that the moving average of a sample's instantaneous loss is good metric for difficulty. Partitioning the data into three groups of increasing difficulty can be done using difficulty score percentiles, 1-dimensional k-means clustering of the scores. Examples of data partitions using entropy and loss are shown in Figure 2
2.3 ENCOMPASSING FRAMEWORK
Curriculum learning approaches can be divided into three categories depending on how they process their input data: approaches that identify and prune noisy data that may hurt performance (Northcutt ct aal.|[20. [202]] Guo et al., 2018 Rooyen et al. 2015 Patrini et al.) 2016 Chen et al.) 2019), approaches that use different sub- sub-samples of data during training (Bengio et al. 2009; Zhou et al. 2020 Xu jet al. 2020 Platanios et al.] 2019 Zhou Bilmes| (018). and approaches that re-weight loss ac- cording to sample difficulty, choosing to emphasize either easy or hard samples (Castells et al.) [2020]
Jiang et al. 2015 2018 Yang et al., 2019; Saxena et al. 2019). The framework presented in this paper capable of representing all of the three approaches.
First, data pruning can be done by assigning negative values to the rate change and shift parameters in our framework, r and in Eq. [2] A negative r causes the weight to approach zero, and negative s shifts the curve to the left, so the curve reaches zero before training begins. The framework also allows flexibility in pruning: by setting a small positive the noisy data can be seen by the model for a short amount of time before reaching zero weight, or by setting a positive r and a large positive the noisy data will only be seen at the end of training (after stabilizes).
Second, data sub-sampling can be represented by the weight going to zero or increasing from zero at different stages of training. For instance, Figure Ib illustrates curriculum where the easy samples are sub-sampled in the beginning, and harder samples are introduced at later stages.
Third, we display in Figure the confidence scores assigned to our data by three loss re-weighting approaches. The results are generated by our implementations of the three approaches, evaluated on the three datasets introduced $3.I where each model runs with five random seeds. The parti- tioning of easy, medium, and hard is according to the entropy- difficulty classes, as described in $2.2 We record the average weight (confidence) assigned to each class. The result is averaged over all runs, and the shaded area is the 95% confidence interval. The approaches that estimate sample confidence based on loss (Castells et al., 2020 Zhou et al.] 2020; Kumar et al. 2010 Diang et al. [215] Felzenszwalb al.) 2009) tend to generate monotonic curves over the course of training because training loss tends to be non-increasing at every step. Therefore, the confidence scores as- signed by these weigtigg approaches follow monotonic curve that can be approximated by our weighting functions 1. Figure (). We note that although the weight scale of SuperLoss (Castells) et al., 2020) in Figure Bajis is larger than one, this model can still be represented by our CL framework because the increased scale corresponds to a scaling of the learning rate, as shown below:
Σ od = 0z-1 - (11 Tmaz 0, = 00= - - n
Os
bis
n
T Omar
where I, and o; are the loss and confidence of sample i, respectively. Therefore, our framework can also represent CL. approaches with confidence scale larger then one.
4

---

Under review as a conference paper at ICLR 2022
11
0.5
16-
0m
0.8
12-
0.2
12-
06-
0.5
05
06-
66-
Supe
(c)
Hard
Negative
Min-
(a)
The LuperLoss
Castells
(b) Self-paced Learning (umaa
ing
Felzenszwalb
al.
al.
2020
al. 2010)
2009
Figure 3: Confidence assignment to samples in our datasets by three curriculum learning approaches that re-weight their loss functions by computing confidence scores for samples. The x-axis is the epoch number, and y-axis is the average weight assigned instances of different difficulty. Blue (solid) easy, orange (dashed) is medium, and green (dash-dot) is hard. The shaded area is the 95% confidence interval (CI) over three datasets with five random seeds each. The curves monotonic for most parts, and can be approximated by the monotonic curves generated by our framework.
2.4 CURRICULUM DISCOVERY
We employ hyperparameter optimization framework (Akiba et al. 2019) to find the optimal value of the curriculum parameters (r, s) for each difficulty class. Using this method, we can learn data- driven curriculum beyond what we could manually design through empirical settings or a choice among the limited ordering strategies. To optimize the three pairs of parameters, we use the Tree- structured Parzen Estimator (TPE) sampling algorithm (Bergstra et 2011). Unlike grid or random search [Bergstra Bengio 2012). TPE traverses the parameter space by estimating the parameters that are most probable to perform better
trial, based on the previous trials. TPE defines two Gaussian Mixture Models, 1(x) and g(x) which are formed using the best and remaining observed parameters, respectively. TPE selects the parameter with high probability under [(x) and low probability under g i.e. arg max, ((x) This choice of sampling algorithm greatly speeds up the search of our eurriculum parameters.
We note that the discovered curricula are optimal within this framework, constrained by the method of data partitioning and the class of weight functions. We argue that the proposed framework is able to approximate curricula defined by existing CL approaches, and outperform existing CL approaches across several datasets.
3 EXPERIMENTS
3.1 DATASETS
We evaluate our approach on three datasets that contain multiple annotations for each sample. First, the Stanford Natural Language Inference (SNLI) benchmark (Bowman et al. 2015), which contains 550k training samples, 10k development samples, and 10k samples. Within the training samples, there are 36.7k samples annotated by 5 workers and 2.6k annotated by 4 workers, which we use for our experiments and refer as SNLI "full." Furthermore, in order to control for variance due to imbalanced difficulty classes, create a balanced subset of the data. As shown in Figure 2a (notice the y-axis scale), SNLI highly imbalanced in entropy classes. The data is downsampled by selecting equal number of samples from each entropy-class. The balanced subset contains total of 2.3k samples, i.e. 774 samples in each entropy class. On the other hand, loss classes are fairly distributed, see Figure 2d The downsampled subset contains both entropy and loss classes which will be used in experiments.
The Alcohol dataset (Amiri et al. 2018 Weizzman et al. 2020 has been developed to obtain population-level statistics alcohol use reports through social media. The dataset consists more than 9k tweets. Given an alcohol relevant tweet, annotators asked determine if it reports first-

---

Under review as a conference paper at ICLR 2022
person alcohol use, and if yes, the intensity of the drinking (light vs. heavy), the context of drinking (social vs, individual), and the time of drinking (past, present, or future). All samples in the dataset are labeled by at least three workers, including over 1.3k samples labeled by five more workers. We define a multi-class classification task for this dataset based on alcohol relevance, intensity and context of drinking. The categories and their data distributions domly split the data into 5.4k training samples, 1.8k development samples, and 1.8k test samples. The balanced version of the training set contains total of 2.5k training samples, i.e. 863 samples in each entropy class.
reported in Appendix [ We ran-
The Cancer dataset has been developed to obtain population-Ievel statistics of cancer patients: contains 3.8k Reddit posts. Annotators are asked to determine if the post describes the experience of a cancer patient, the type of cancer, and the relation of the author of the post to the patient. We define a multi-class classification task based on post relevance and caner type. The categories and their data distribution are reported in Appendix[A] All samples are labeled by least three workers, including about Ik labeled by at least five. We randomly split the data into around 2.2k training samples, 765 development samples, and 765 test samples. The balanced version of the training set contains total number of 1.7k sample, i.e. 578 samples in each entropy class.
We note that the datasets are significantly different in average document length, ranging from 10 (SNLI), to 15 (Alcohol) to 174 (Cancer) words. This variation can induce significant variance in the created models.
3.2 BASELINES
We compare the performance of our CL approach with the following state-of-the-art approaches. SuperLoss (SL) (Castells et al.) 2020) is a CL. approach that defines task-agnostic confidence- aware loss function. infers the confidences of instances from the instance loss with minimal cost, providing a closed-form solution function for estimating confidence. It up-weights samples with smaller loss values (casier instances), while down-weights those with larger loss values (harder ones) (Figure Bap. MentorNet (Jiang et al.) 2018) uses an auxiliary network to generate weight for training samples at every training iteration. It incorporates additional signals such as epoch number and instance loss history to learn data-driven curricula and is particularly strong against noisy data. Difficulty Prediction (DP) (Yang et al.) 2019) defines a difficulty score based on multi-annotator data, and weight samples according the following formula where d, is the sample difficulty and 7 is pre-defined threshold.
a(d, - TDP)/(1 TDP),
As discussed before, our approach employs two scoring functions ($ 2.2 and two curriculum con- figurations for each dataset. A curriculum configuration refers to a particular setting of the six parameters controlling the three weight curves (8 2.I The two scoring functions are labeled as Loss and Ent (entropy). In addition, the first curriculum configuration is gradually increasing ap- proach in Figure [bb named inc., this configuration is applied identically to models. The second configuration is the specialized configuration (sp.) that is obtained through hyper-parameter search as discussed in $ 2.4
3.3 SETTINGS
We tune the parameters À of SL and a and TDP of DP using development data. The optimal values found are À = 1.2, a = 0.9 and TDp is set dynamically upon loading the dataset to the 50 -0-percee difficulty value. Following Castells FollowingCasteelss hard) to the moving average of the loss in all experiments.
al. 22020,, we set TsL (instances with 1, > TsL are considered
We use the transformers python package (Wolf et al., 2020), using the bert-bbaseunncased model for SNLI and Cancer, and Switter-roberte ceroobeetta base for Alcohol. We set learning to le - 5, batch to 16, epochs to 10 (we confirm that this number of iterations is sufficient for all models to converge), and the optimizer to Adam (Kingma & Ba 2017). For each experiment, we train five models using five random seeds applied to both pytor ch and numpy. Additionally, during all data pre- processing, splitting, and sub-sampling, the random seed is set to 0, and single NVIDIA A100 40GB GPU is used for training. The development set is used to determine the best training step which is used for the final evaluation.

---

Under review as a conference paper at ICLR 2022
0.79
0.78
0.77
0.76
0,74
Loss (sp)
Less (Ime)
DP
Ent (imc)
Mentortiet
St.
No CL
Ent
Figure 4: Overall accuracy averaged over three datasets with five random seeds. Loss and Ent indicate curricula that partition the data based on difficulty classes determined by loss and entropy respectively ($2.2). inc is the easy to hard curriculum shown in Figure [b] sp is the specialized curriculum obtained by curriculum discovery 2.4), which different for each dataset.
In addition, conduct hyper-parameter search over the (r s) for each weight function difficulty classes (easy, medium, and hard). We set the search space of to be from -10 to 10 with a step of and of to be -0.5 1.5 with step 0.25. We observe that changes smaller than this step size have little effect on performance. This search space consists of 11 possible values for r and 9 for s, for total of 970k combinations. The search is run for least 100 trials, as compared over 1000 trials by random search, using the method described in $2.4 Each trial is run with 5 random seeds and the result averaged. The search objective is to maximize accuracy over development data.
3.4 PERFORMANCE GAIN FROM CURRICULA DISCOVERY
Results shown in Figure Accuracy is averaged over the six datasets (full and balanced version of each dataset). The gradually increasing curriculum (inc) achieves a significant improvement over No-CL using either of the scoring functions while being static, off-the-shelf curriculum configura- tion. This improvement shows the effectiveness of our approach of partitioning the data using gen- eralized logistic functions (22.)). Moreover, both (inc) and the specialized (sp) curricula obtained through curriculum discovery perform significantly better than the state-of-the-are CL approaches. In our three datasets, loss as a scoring function performs better than entropy on average. This ex- pected as loss values can capture sample difficulty with respect to the downstream learner (model), as apposed to entropy values which do not take into account the model.
Appendix B includes further breakdown of the results by dataset and accuracy across samples of different difficulty.
3.5 CHARACTERISTICS OF DISCOVERED CURRICULA
Figure shows the mean and 95% CI of the top 25 performing configurations on datasets and scoring functions. We observe several insightful patterns: the resulting curricula are non-trivial and greatly differ from the known strategies reported in current literature, such as gradually increasing difficulty or unti-curiculum. In addition, the weights of hard samples tend to approach zero, sup- porting the hypothesis that either these instances are too difficult for the models to learn or they are noisy. The results support the principle of pruning techniques because noisy samples induce more noise than useful signal; in several plots, the weight hard samples increases only at the end of training after the model stabilizes. In addition, in SNLI and Alcohol easy samples carry the most significant weight, unlike Cancer, where easy samples are woin-weighted early during the training. These weighting patterns reveal the relative importance of samples in each dataset. Finally, the full SNLI dataset with entropy-class partitions provides useful information. Figure 2a shows that en- tropy classes are highly imbalanced, with hard samples being much fewer than easy ones. In the
7

---

Under review as a conference paper at ICLR 2022
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4 0.6 0.8
0.0
0.2
0.4
0.6 0.8
0.0
0.2
0.4
0.6 0.8
0.0
0.2
0.4
0.6
0.8
(a) S-B-F
(b) S-B-I.
(c) S-F-E
(d) S-F-I
1.0
0.6
1.0
0.8
0.8
0.6
0.6
0.4
0.6
0.4
0.2
0.2
0.1
0.2
0.0
0.2
0.4
0.6 0.8
0.0 0.2
0.6 0.8
0.2
0.8
0.0
0.2
0.4
0.6 0.8
(c) A-B-E
(f) A-B-I.
(g) A-F-E
(h) A-F-L
1.0
1.0
0.8
0.9
0.8
0.6
0.6
0.8
0.4
0.4
0.6
0.4
0.5
0.0
0.0
0.0
0.6
0.4
(i) C-B-E
(j) C-B-L
(k) C-F-E
(I) C-F-L
Figure 5: Each caption is composed of the first character of the name of dataset: {SNLI, Alcohol, Cancer), the type of the dataset [Balanced or Full), and the difficulty score used (Enrropy. Loss). The x-axis is the training progress and y-axis is the confidence assigned to samples of a difficulty- class. The blue line (circle marker) is easy, orange line (x marker) is medium, and green line (dia- mond marker) is hard. The solid line is the mean of the top performing configurations for each dataset and scoring function pair, and the shaded area represents the 95% CI.
optimal curriculum shown in Figure 5c hard samples assigned weights around 0.5, unlike the three other cases of SNLL. We attribute this result to the reduced presence/effect hard samples.
3.6 GENERALIZABLE CURRICULA
Figure shows the accuracy obtained when training using different curriculum configurations from Figure 5 Each cell in the figure is the average result of 5 seeds. We observe common characteristics among datasets that cause the curriculum be transferable between them. First, the top three configurations are all products of down-sampled, balanced datasets. Second, the curricula obtained using the small balanced datasets generalize and achieve high performance on the large datasets. This is useful it allows performing the hyper-parameter search much faster and cheaper smaller datasets, providing evidence that framework can applied to large datasets by searching for curriculum on small subset of the data. Third, the inc curriculum is available off-the-shur and performs consistently well across datasets and scoring functions. Fourth, as noted previously, instances of the Cancer dataset consist of long paragraphs, causing high variance in models trained using the dataset. Consequently, the curricula obtained using the Cancer and loss as measure of difficulty are of lower quality and perform the worst.
An extended version of Figure 6 is included in Appendix C with results models trained with balanced versions of datasets.
8

---

Under review as a conference paper at ICLR 2022
5-F-E
99.6
99.7
100.0
99.5
99.0
99.8
99.8
100.0
99.0
99.8
99.5
99.9
98.9
99.1
S-F-L
99.5
99.8
99.5
100.0
99.8
99.2
99.8
99.2
99.8
99.6
99.4
99.6
99.3
98.6
C-F-E
99.2
99.1
98.3
99.2
99.5
100.0
98.6
98.7
99.1
98.5
98.9
Models
CFL
99.9
100.0
99.5
99.4
99.3
99.0
98.1
98.6
99.5
98.4
98.7
97.0 0
99.2
99.1
A-F-E
99.0
99.0
99.6
99.1
98.6
99.3
98.8
100.0
99.0
98.9
98.6
99.3
98.0
98.1
A-F-L
100.0
99.1
99.4
99.2
100.0
99.2
99.3
97.6
98.8
98.9
98.6
98.6
98.7
97.6
Avg.
99.5
99.5
99.4
99.4
99.3
99.3
99.2
99.2
99.1
99.0
99.0
99.0
98.8
98.6
A-B-L
5-B-L
S-B-E
A-L-
CBBEE
S-F-E
inc.
A-F-E
S-F-L
CFFEE
No-CL
A-B-E
C-B-L
C---L
Configurations
Figure 6: Using the notation Figure the x-axis lists different curriculum configurations from the increasing curriculum inc (Figure 1b). to No-CL, to curricula discovered using particular dataset and scoring function. The y-axis lists models that are trained using each configuration. For example, the cell at the intersection of row "S-F-L" and column "A-F-E" represents a model trained on SNLI full dataset that partitioned by loss as measure of difficulty, using the curriculum discovered for the full Alcohol dataset partitioned by entropy (Figure 5g). Each row of the table is normalized to match the scales different models.
4 CONCLUSION AND FUTURE WORK
We introduce an effective curriculum learning (CL) framework that employs prior knowledge about sample-level difficulty in its training paradigm, and effectively creates and explores curricula space for curricula discovery and model generalizability. The proposed framework partitions its input data into three groups of increasing difficulty, defines three parameterized logistic weight functions to weight the loss of samples each of the three groups, and provides the capability of tuning the pa- rameters of the weight function to discover new curricula performing better than existing baselines. We demonstrate that this framework capable of representing major CL approaches. In addition, important advantage of our approach is that the curricula that discovers for smaller and balanced datasets work well on larger datasets, across the three datasets that we experimented with. The pro- posed research opens a new paradigm in CL by removing the limitations imposed by selecting single CL strategy, and instead, using a flexible framework discover optimal curricula for given datasets and models, and extract valuable insights about the data. Our work has the limitation of using monotonic logistic functions because the weight curves can take arbitrary forms. This challenge can be addressed in future work by adding extra set of logistic functions and having every weight function be a linear combination of two more logistic functions, which makes possible to represent non-moneee non-monotonic functions. Nevertheless, approaches that estimate difficulty as a function of training loss result in monotonic curves because of gradient descent that causes training loss be non-increasing.
have demonstrated that major CL
There are several promising areas for future work. These include approaches for learning new dif- ficulty indicators that centered data (e.g., text difficulty), prioritizing medium level instances and those with greatest progress in training, and developing challenge datasets that contain diverse data samples with different level of difficulty.

---

Under review as a conference paper at ICLR 2022
5 ETHICS STATEMENT
This investigation included publicly available yet sensitive data from social media, developed for an important task: obtaining population-level statistics about different public health issues. Our work does not access, use, or release any personal data, and only textual data is used for experiments.
REPRODUCIBILITY STATEMENT
The source code is included supplementary material. It provides the required details for data processing, hyper-parameter setting, random seed setting, model evaluation, etc.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
Hadi Amiri, Timothy Miller, and Guergana Savova. Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks. In Proceedings of the 2017 Conference on Em- pirical Methods Natural Language Processing (EMNLP), pp. 2401-2410, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/1/D17-1222
URL
Hadi Amiri, Kara M Magane, Lauren E Wisk, Guergana Savova, and Elissa R Weitzman. Toward large-scale and multi-facet analysis first person alcohol drinking. In American Medical Infor- matics Association (AMIA), 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
ACM International Conference Proceeding Series, volume 382, pp. 1-8, New York, New York, USA, jul 2009. ACM Press. ISBN 978 160558516.. doi: 10. 55314444/533..11555
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research (JMLR), 13(2), 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances Neural Information Processing Systems (NIPS), 24, 2011.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large an- notated corpus for learning natural language inference. In Conference Empirical Methods in Natural Language Processing. EMNLP 2015, pp. 332-642. Association for Computational Lin- guistics (ACL), 2015.
Thibault Castells, Philippe Weinzaepfel, and Jerome Revaud. Superloss: A generic loss for robust curriculum learning. Advances Neural Information Processing Systems (NeurIPS), 33, 2020.
Antoine Caubrière, Natalia Tomashenko, Antoine Laurent, Emmanuel Morin, Nathalie Camelin, and Yannick Estève. Curriculun-based transfer learning for effective end-to-end spoken language understanding and domain portability. In 20th Annual Conference of the International Speech Communication Association (InterSpeech), pp. (198-1202. 2019.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learn- ing. pp. 1062-1070, PMLR, 2019.
Jeffrey I. Elman. Learning and development in neural networks: The importance of starting small. Cognition, 4811)199999 1993.
Pedro F Felzenszwalb, Ross B Girshick. David McAllester, and Deva Ramanan. Object detection with discriminatively trained bast- models. IEEE transactions on pattern analysis and ma- chine intelligence, 3279):1627-1645 2009.
10

---

Under review as a conference paper at ICLR 2022
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robot learning, pp. 482-499 PMLR, 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135-150, 2018.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL HLT 2018, pp. 107-112, Association for Computational Linguistics (ACL), 2018.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced cur- riculum learning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning (ICML), pp. 2304-2313 PMLR, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Tom Koemi and Ondřej Bojar. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 379-386, 2017.
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in Neural Information Processing Systems (NIPS), 13:1189-1197, 2010.
Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, Rene Vidal, and Vittorio Murino. Curriculum dropout. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3564 3572. IEEE Computer Society, 2017.
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9131-9143, 2020.
Curtis Northeutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly su- pervised learning and label noise robustness. In Proceedings of the 33rd International Conference on International Conference Machine Learning- Volume 48, pp. 708-717 2016.
Emmanouil Antonios Platanios, Otilia Streteu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. Competence-basee curriculum learning for neural machine translation. In Proceed- ings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HITT, pp. 1162-1172. 2019.
Adam Poliak, Jason Naradowsky. Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. NAACI. HLT 2018, pp. 180, 2018.
FJ Richards. A flexible growth function for empirical use. Journal of experimental Botany (JXB), 10(2):290-301, 1959.
Douglas LT Rohde and David C Plaut. Language acquisition in the absence of explicit negative evidence: How important is starting small? Cognition, 72(1):67-109, 1999.
11

---

Under review as a conference paper at ICLR 2022
Brendan van Rooyen, Aditya Krishna Menon, and Robert C Williamson. Learning with symmetric label noise: the importance of being unhinged. In Proceedings of the 28th International Confer- on Neural Information Processing Systems- Volnss-- 1, pp. 10-18, 2015.
Terence Sanger. Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE transactions on Robolics and Automation, 10(3):323-333, 1994.
Nikolaos Sarañianos, Theodore Giannakopoulos, Christophoros Nikou, and Ioannis A Kakadiaris. Curriculum learning for multi-task classification of visual attributes. In Proceedings the IEEE International Conference Computer Vision Workshops, pp. 2608-2615, 2017.
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum, Advances in Neural Information Processing Systems, 32: 11095- 11005, 2019.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Samarth Sinha, Animesh Garg, and Hugo Larochelle. Curriculum by smoothing. Advances in Neural Information Processing Systems, 33, 2020.
Elissa R Weitzman, Kara M Magane, Po-Hua Chen, Hadi Amiri, Timothy S Naimi, and Lauren Wisk. Online searching and social media detect alcohol use risk at population scale. American journal of preventive medicine (ACPM), 58(1):79-8, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. BtLhttps: www. aclweb. org/anthologyy/
2020. www.eeennnpeeneee demos.em.... 6
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In International Conference on Learning Representations (ICLR), 2021. URL [tLL:ttttpss//eeeeeeeeee
Forum?id
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Cur- riculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 6095-6104, 2020.
Yinfei Yang, Oshin Agarwal, Chris Tar, Byron C Wallace, and Ani Nenkova. Predicting annotation difficulty to improve task routing and model performance for biomedical information extraction. In Proceedings of the Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, NAACL-HLT, pp. 1471-1480, 2019.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup. Marianna J Mar- tindale. Paul McNamee. Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint arXiv: /81 I. 007339, 2018
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. Curriculum learning for domain adaptation in neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- ties: Human Language Technologies (NAACL-HI..), pp. 1903-1915, 2019.
Tianyi Zhou and Jeff Bilmes. Minimax curriculum learning: Machine teaching with desirable diffi- culties and scheduled diversity. In International Conference on Learning Representations, 2018.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by dynamic instance hardness. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
12

---

Under review as a conference paper at ICLR 2022
A
DATA CATEGORIES DISTRIBUTION
Class
Count
(no)
5,325
(yes, light use, individual)
1,464
(yes, heavy use, individual)
964
(yes, not sure, individual)
457
(yes, heavy use, other)
423
(yes, heavy group)
284
(yes, light use, group)
161
Total
9,078
(a) Alcohol
Class
Count
(irrelevant, no patient experience)
1.996
(relevant, breast cancer)
617
(relevant, colon cancer)
444
(relevant, brain cancer) (irrelevant, none of the above)
284
251
(irrelevant, other cancer types)
162
(irrelevant, related to cancer)
70
Total
3,824
(b) Cancer
Table 1: Statistics of the Alcohol and Cancer datasets.
13

---

Under review as a conference paper at ICLR 2022
B ACCURACY BREAKDOWN
The accuracy achieved on each dataset using each approach is shown in Figure 7
Models that achieve high accuracy may be generating correct predictions for high percentage of easy samples while failing to correctly predict the output of the medium and hard samples. Standard evaluation benchmarks often contains artifacts that make it east correctly predict the label of some samples (Gururangan et al. 2018 Poliak et al. 2018). Therefore, it is important to closely analyze the model's performance on harder instances. We break down the accuracy of each difficulty class (with entropy-class partitioning) for balanced, full, and all datasets in Figures X 9 and T0)
respectively. Our approach exceptionally powerful in predicting samples of medium difficulty. Furthermore, the accuracy achieved on hard samples is almost equal by all approaches, including No-CL, supporting the hypothesis that those samples tend to noisy or inaccurately labeled.
Accuracy
Accuracy
snil speciia
snli balanced
0.865
0.76
3.860
0.74
0.855
0.72
alcohol_ 7
alcohol 7. bai balanced
0.77
0.76
0.78
0.74
cancer
Gencer balanced
0.74
0.73
0.72
0.71
0.71
0.70
Loss (inc)
Loos (op)
Ent linc)
Ent (sb)
St.
Montoniet
N5-CL
Lass (sp)
Loss (inc)
Ent tsp)
Ent imme)
MentorHot
D2
No
(a) Full datasets.
(b) Balanced datasets.
Figure 7: Accuracy of different CL approaches each dataset.
14

---

Under review as a conference paper at ICLR 2022
0.925
0.700
0.900
0.675
0.650
0.850
0.825
0.600
0.800
0.575
0.550
0.750
Loss (ap)
Ent tsp)
Loss (Inc)
Op
Mentortet
Ent (inc)
Loss (op)
Ent (ip)
Lass lnee]
Up
MentorWes
Ent (inc)
CL.
(a) Easy
(b) Medium
0.72
0.60
0.55
0.68
0.50
0.64
0.45
0.62
0.60
Menterfet bp
Ert [ime)
Less (sp)
st.
No CL
Lase (ine)
Ent (sp)
Los (sp)
Ent (sp)
Op
MentorNet
Loss linc)
St.
Ent (ine)
No
Hard
(d) Balanced Accuracy
Figure 8: Average over balanced datasets.
0.94
0.72
0.92
0.70
0.90
0.50
0.88
0.86
0.56
0.84
Lass (sp)
Ent (sp)
Leas (ine)
Ent (inc)
Op
st.
No-CL MentorNet
Ent (sp)
Lods (inc)
DP
Lass (op)
Ent (inc)
st.
MenterNet
No
Easy
(b) Medium
0.600
0.575
0.72
0.550
0.71
0.525
0,500
0.70
0.475
0.450
No-CL DP
Mentor Net Ent (inc)
Lass (sp)
Loss (inc)
Ent (sp)
D#
Lass (gg)
Ent (mn)
Loss tinc)
No ea
Ent tep)
St
Mentorhit
(c) Hard
(d) Balanced Accuracy
Figure 9: Average over full datasets.
15

---

Under review as a conference paper at ICLR 2022
0.725
0.90
0.675
0.650
0.85
0.800
0.80
0.575
Loss (sp)
Ent (sp)
DP
Loss (inc)
Mentorkit
CL
Enn-
Loss (sp)
Ent (sp)
Loss (trc)
Up
Mentortter
(irc)
No-CL
(a) Easy
(b) Medium
0.60
0.72
0.70
0.55
0.68
0.50
0.66
0.64
0.45
0.62
0.60
Mentortee Op
Ent (mn)
St.
Loss (sp)
No.CI
Loss (tnc)
Ent (sp)
Loss (sp)
(sp)
Mentor St.
Loss (inc)
Ent (inc)
No Ct
(c) Hard
(d) Balanced Accuracy
Figure 10: Average over all datasets.
16

---

Under review as a conference paper at ICLR 2022
C
EXTENDED CONFIGURATION GENERALIZABLITY EXPERIMENTS
S-F-E
100.0
99.6
99.7
100.0
99.5
99.0
99.8
99.0
99.9
99.8
99.5
98.9
99.
S-F-L
99.5
99.5
99.8
99.2
100.0
99.8
99.8
99.8
99.6
99.6
99.2
99.4
99.3
98.6
A-B-E
100.0
99.3
99.9
100.0
99.5
99.8
99.9
99.6
99.5
99.7
97.9
99.8
98.4
98.3
C-F-E
98.3
99.2
99.1
100.0
99.0
99.4
99.5
98.6
99.3
98.7
99.2 99.2
99.1
98.5
98.9
C-F-L
99.5
99.9
100.0
98.6
99.4
99.3
98.1
99.5
97.0
98.4
99.0
98.
99.2
99.1
AFF.
Models
99.6
99.0
99.0
100.0
99.1
98.6
98.8
99.0
99.3
98.9
99.3
98.
98.0
98.1
A-F-
99.4
100.0
99.1
100.0
99.3
98.8
98.6
98.9
97.6
A-8-L
99.7
100.0
98.4
96.9
98.6
100.0
98.6
98.0
98.5
99.1
99.2
99.6
98.0
97.4
C-B-E
98.9
99.3
98.8
100.0
98.0
99.4
97.5
98.9
98.2
97.6
98.7
97.1
98.1
98.2
C-B-L
99.0
98.9
98.4 .
98.7 7
98.6
100.0
97
98.9
96.0
98.6
96.1
95.9
98.9
98.8
S-B-E
100.0
98.5
99.1 1
99.7
98.
96.5
99.3
96.7 7
99.1 1
98.3
97.3
96
92.2
91.5
S-B-L
99.7
100.0
99.0
99.9
96.3
99.3
97.7
98.3
95.0
92.3
93.3
87.7
83.8
Avg.
99.5
99.4
99.3
99.0
98.9
98.7
98.6
98.5
97.2
96.6
5-8-E
A-B-L
5-B-L
A-F-E
A-F-L
C-B.E
Inc.
S-F-
A-B-E
C-F-E
S-F
No-CL
CBL
CFL
Configurations
Figure 11: An extended version of Figure including experiments on balanced versions of the datasets.
Figure shows the full results including evaluation on the balanced datasets.
17

---

Under review as a conference paper at ICLR 2022
D FULL CURRICULUM SEARCH RESULTS
This section shows the top 25 performing configurations on each dataset scoring function pair. Blue lines (circle marker) is easy, orange marker) is medium, and green (diamond marker) hard. Above each plot is the development set accuracy. The plot legend contains the parameters (r, 8) for each class.
(.74.955%
(74.92%)
3.74.018)
MAY/SL. WEIN?
resd 14:1
0.0
0.0
0.00
0.25
0.50 0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50 0.75
0.00 0.25 0.50 0.75
(74.77%)
(74.74%)
(74.74%)
(74.74%%
(74.74%)
1.0
-51
eay 63 751
eay 13, 51
eary 16, 75
mad
Red
0.25
0.25
hand
0.0
0.00
0.00
0.0
0.0
0.50
0.00
0.00
0.25
0.50
10
(74.633)
11 (74.62%)
12 (74.59%)
13 (74.59%)
14 (74.58%)
1.0
1.0
0.6
0.0
0.5
med
0.5
24.1
0.4
Rord 14
0.2
750
0.0
0.0
0.0
0.50
0.00
0.50
15 (74.58%)
(74.58%)
17 (74.46%)
18 (74.45%)
19 (74.4416)
1- 220
0.75
say 14,
0.6
0.5
0.50
tardi
0.6
rend P))
0.25
14.1
0.00
0.00
0.25
0.50
0.75
0.00
0.25 0.50 0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25 0.50 0.75
(74.43%)
(74.42%5)
(74.37%)
23
(74.37%)
24 (74.30%)
1.0
0.4
18,31
0.5
hart
28.1
hand 04.1
ren
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00 0.25
0.50
0.75
0.00 0.25
0.50 0.75
0.00
0.25
0.50 0.75
Figure 12: S-B-E
18

---

Under review as a conference paper at ICLR 2022
(78.19%)
1(78.1995)
(78.139))
(78.1255)
(78.10%%
1.0
1.0
40g 1-0)
(2.
eepp 14 2.))
0.5
0.5
Te 1N 122
1o0 1.M1
Fasd 122
med .11
med
0.0
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0,50
0.75
0.00
0.25
0.50
0.75
0.50
0.25 0,50 0.25
(78.09%)
(78.08%)
(78.07%)
(78.04%)
(78.04%))
1.0
1.0
1.20
3.1
B41 14, 3:25
0.5
mal 18 2+
0.5
0.5
FRO 0.23
0.0
0.0
0.0
0.0
0.0
1.21
0.00
0.25
0.50
0.00
0.50
0.75
0.00
0.75
0.25
0.50
0.00
0.50 0.75
10 (78.01%)
11
(78.015))
12 (78.00%)
13
(78.00%)
24 (78.00%)
1.0
1.0
1.0
1.0
1.2
14.
790
eey 1.2
0.5
0.5
0.5
ned 0.1
0.5
red
000 12 L.)
0.0
0.0
0.0
0.0
0.0
war 15, 1.2
0.00
0.25
0.50
0.00
0.50
0.75
0.0
0.5
0.00
0.25
0.50
0.00
0.25
0.50 0.75
15
(77.99%)
16
(0(777995)
17
(77.9777%
18
(77.97))
19 (77.97%)
1.0
1.0
1.0
0.5
0.5
0.5
rend
0.5
hard
eary 13, 3.51
hard
hard 32
0.0
hard 13 1.2
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50
0.75
0.0
0.5
0.0
0.5
20
(77.96%)
(77.95%)
22
(77.94%)
(27.9933%
(77.92%)
1.0
1.0
0.5
0.5
hand 24, 3.5
hand (6, 3.22
<me 1.55
1A 3:5
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Figure 13: S-B-L
0 (86.50%%)
(86.500)
(86.48%)
(86.4755)
(86.45%%
1.0
1.0
2.0
-sayy 16,1
say 6.
08ay (5,
0.5
4.25
wer KS 70
hard 14, 20
haddi4 44.0
real 770
0.0
0.0
0.0
0.5
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25 0.50
0.75
0.00
0.25
0.50 0.75
(186.555)
(86.42%)
(86.42%%)
(86.415))
(86.41%)
1.0
3.0
eety (2E. 6:50
eme 18 -8251
say M. 2:55
eepp 4.25
eaty 14
0.5
14,
0.5
0.5
0.5
hard
hadd 1..d
hard 14,
hadd:
0.0
0.0
0.0
0.0
0.0
0.00
0.25
0,50
0.00
0.25
0.75
0.25
0.50
0.75
0.00
0.25
0.50
0.00
0.50 0.75
10
(86.40%5)
11 (86 40%)
12 (86 40%)
13
(86.400))
14 (B6 (0%))
1.0
3.0
1.0
1.0
1.00
twid
hadd 75
radik 151
0.75
cmyy 20,1
0.5
0.5
0.5
0.50
0 1)
0.25
0.0
0.0
0.25
0.50
0.00
0.25
0.50
0.75
0.25
0.50
0.75
0.00
0.25
0.75
0.50 0.75
15
(0%.4
16 (86 39%)
17
(86.39%)
18
(8%63
19 (86. 38%)
1.0
1.0
1.0
1.0
3.0
Fand 75
0.5
mad P4 it
0.5
0.5
0.5
read (06,
0.5
red L.1
herd
herd
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.0
0.5
0. 0.00
0.25 0.50 0.75
20
(86.37%)
21
(66.35%)
22 (86.35%)
23 (86 (86.35%)
24 (06 34%)
1.0
1.0
1.0
1.0
sery 25 .0
medi8. 21
nedit, 751
0.5
mad H4
0.5
medid 44 2.51
0.5
1a212. 35
0.5
Iood 16, 2.22
hart 14.1
250
0.0
0.0
0.30
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
Figure 14: S-F-E
19

---

Under review as a conference paper at ICLR 2022
[86.48%%
(86.475))
(86.465))
43%6
(86.435)
1.0
rrad 10,
rec (3, 50
0.4
mood 10
0.5
say
0.5
0.5
Fand 36,
0.2
Faxd:t6, 31
0.0
0.0
0.0
0.00
0.25
0.50 0.75
0.0
0.5
0.0
0.5
0.00
0.25 0.50 0.75
0.0
0.5
1.0
(86.42%%]
(86.41%)
(86.415%)
(86.)11%)
(B6 (86%41%)
0.75
0.4
0.5
1add 11
0.5
0.2
rxdit 231
*eay 13
0.25
0.50
red 15,
-m0012 20
0.0
0.0
0.0
0.0
0.00
0.00
0.25
0.75
0.00
0.25 0.50 0.75
0.00
0.25
0.50
0.75
0.75
20 (86.40%)
11 (86.38%)
(86.38%)
13 (86.36%)
14 (86.36%%
1.0
smy
0.75
eary
hoad 18
0.4
13. 20
12. 51
0.5
Fand 00 11
Fand
0.50
hand 34
0.0
0.0
0.0
0.0
0.00
0.5
0.00
0.25 0.50 0.75
0.50
0.0
0.5
0.50
15
186.30%)
16 (86.36%61
17 (86.36%)
18
(86.35%)
19 (86.3516)
cuo 14, 251
0.75
city 14 31
1.00
1.0
say 13 31
0.5
18
0.50
hard
0.50
0.75
rwilds D:0
0.5
4.31
0.25
hand
0.50
771
0.0
0.00
0.25
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50 0.75
0.00
0.25 0.50 0.75
(86.
34%)
21 186.34%)
(86.34%)
23 (86.33%)
(86.33%))
0.6
0.6
0.6
<ay 11
nedr2
0.5
0.4
0.4
0.5
0.2
hard
0.2
hand 06, 31
0.2
hard 00
0.0
0.0
0.0
0.5
0.00
0.25
0.50 0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50
0.75
0.0
0.5
Figure 15: S-F-L
(78.529%)
1 (78.40%)
178.400%)
(78.37%)
(78.377%
1.0
0.75
0.75
0.50
KRR 12
sale red 11
0.50
40Y +2
14,2
0.25
herd HL 700
Aec 9.10
0.25
MM 12, D
had 2.
0.00
0.0
0.0
0.00
0.0
0.00
0.25 0.50
0.75
0.00
0,25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50
0.75
(78.36%5)
(78.30%)
(78.28%%
(78.27%)
9 (78.25%))
1.00
1.0
0.50
hard 12, 1
00 31
0.5
eaty GA
0.75
0.50
(l JMd
hadd
0.5
#M/G4 13
0.50
0.75
<my 14, 751
hard 750
0.25
0.25
0.00
0.0
0.25
0.50 0.75
0.00
0.25
0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50 0.75
10
23%)
11 (78.22%6)
12 (78.20%)
(78.17%)
14
178.17%)
1.0
0.75
0.75
0.75
0.5
0.50
F0ATE 251
12 E
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.25
0.50 0.75
0.00
0.25
0.00
0.25
0.50 0.75
0.25
0.00
0.25
0.50
0.75
15
(70.17%)
16 (78.16%))
17 (78.16%)
18 (78.15%)
19
(78.15%)
1.0
1.0
1.00
0.75
751
0.75
0.5
cedick 2x1
0.5
0.5
rwc Ca. 220
0.50
0.50
31
0.25
0.25
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
20
(78. 14%)
21
(78.14%)
22 (78.12%)
23
(76.10%)
24 (78.10%)
1.0
1
0.75
0.75
0.75
1.0
0.5
0.50
0.50
.2
0.25
Nam02.11 11
0.25
heel 31
0.25
hani 22
0.0
0.00
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50
0.75
0.00
0.25 0.50 0.75
Figure 16: A-B-E
20

---

Under review as a conference paper at ICLR 2022
(80.12%%
(880.06%)
(80.049))
(.99.9355
(79.93%%
1.0
0.5
0.5
0.3
AMy
1o 10
Saad 14 20
Faxti 21)
251
med 14, 1.21
0.0
0.0
0.00
0.25
0,50 0.75
0.00
0.25
0.50 0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50
0.75
0.00
0.25 0.50 0.75
5 (79.933%)
(79.93%)
(79.92%%
(79.905))
(79.90%)
1.0
0.75
0.5
#4y 50
0.5
440 51
0.5
awy 2, 1.2
ned 12:32
0.50
reed 751
0.25
0.0
0.0
0.0
0.0
Faxd 15, 2M
0.00
FFad 2M1
0.00
0.25
0.50
0.00
0.25
0.75
0.25 0.50
0.00
0.25
0.75
0.00
0.50
0.75
10
(79.90%)
11 (79 88%)
12 (79.88%)
13 (79.86%))
14 (79.84%)
1.0
eay 2.
0.4
hard 140 201
0.5
13:333 21
0.5
0.5
400
0.2
0.0
0.0
harr (16 201
0.0
0.0
Laad (5A 200
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.50 0.75
15
(79.84%5)
16 (79 82%)
17
(79.81%)
18
(79.79961
19 (79. 79%))
0 1:20
1.0
1
0.75
0.5
0.5
0.5
231
0.5
hwrd 14
0.50
kmy 11
0.25
KRy 10, 31
0.0
hadd 1-2 35
0.0
0.00
hand-4
0.00
0.25
0.50
0.75
0.00
0.25 0.50 0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25 0.50 0.75
20
(79.79%)
(79.799) (79.79%)
22 (79.79%)
23
(79.775)
24.
1.0
onyy 21
1.21
red 103.22
12,
0.5
0.5
0.5
hord (10 251
smy
14, 1.23
rnd 1.0
meer12 29)
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0,25
0.50
0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50 0.75
Figure 17: A-B-L
(82.15%)
(82.04%)
(81.98%)
(81.93%))
(61 93%)
0.4
1
3:20
0.6
-35
0.4
151
02
runic RXN
L.
0.4
0.4
0.7
0.2
0.1
0.2
0.0
0.0
0.0
0.00
0.25
0.50
0.75
0.30
0.25 0.50
0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50
0.75
0.00
0.25
0.50
0.75
(B1.83%)
(91.83%)
(81.82%)
(81.80%)
(81.78%)
read
0.2
rend-32 12 eund
rodice Euhi
cod, .n
136 J
hand (00
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.25
0.50
0.75
0.30
0.50
0.00
0.25
0.75
0.00
0.50 0.75
0.25
0.50 0.75
10
(81.78%)
11
(8%)
12
(81.78%)
13
(81)70%)
14
(81.74%5
0.4
0.4
0.6
casy
0.4
(4.1
0.3
0.2
cay 3.22
inc LA)
0.2
3.50
0.4
sar 1t L21
0.2
hand 110 2:0
0.2
haid 1.5
0.1
0.0
0.0
0.0
0.0
0.0
0.00
0.25
0.50 0.75
0.30
0.25
0.50
0.00 0.25
0.75
0.50 0.75
15
(0%)
16 (01. (1%%)
17
(01.71%) 71%)
38 (81.71%)
19 (81.7155]
crse 18 (..)
Cor 1 1.30
Coy 1.0
0.2
eay 3. 3.M
16, 3.20
hard 221
Fand do.
12 1.1
0.5
rwc
hart
0.1
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
20
(91)
21 (81. 69%)
22
(81.69%)
23
(81.65%)
24
(81.60%)
0.2
1.51
1.29
0.6
sere 15 270
44-4231
0.075
ned (20 233
ewyy 15, 1.00
0.1
med 1-3.)
herd (20 1.20
0.5
02
0.4
0.050
0.025
0.30
0.25
0.50
0.75
0.30
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Figure 18: A-F-E
21

---

Under review as a conference paper at ICLR 2022
(81 (81.43%)
(82.435)
(81.419%)
(81.40%)
(81 38%)
1.0
0.4
0.5
Aay
Faxis 3-21
0.5
eaa 18
12 1.21
0.5
eay 4.31
1-21
0.2
say -2351
F1006631 31
<eag 16 -650
104 1.20
0.0
0.0
0.0
0.0
00.5
0.00
0.25 0.50 0.75
0.00
0.25 0.50
0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50 0.75
(81.36%)
(82.345))
(81.29%%
(81.29%)
9 (81.27%)
1.0
0.75
say 12 -2.1
440 06 4.25
16. 351
eay 15 -25
0.50
0.5
hard 1:31
I0A
0.5
0.5
0.5
real
4.25
0.25
0.0
0.0
0.0
0.25
0.50 0.75
0.50 0.75
0.00
0.0
0.0
10 (81.2555)
12 (81.25))
12 (81.23%)
13 (81. (81.21%)
14 (B1 20%)
1.00
1.0
1.0
0.75
0.75
<ee .0200
12, 0.5)
0.50
rel Ii, 232
06 655-
0.5
re 0.00
X 4.51
eaay ..5)
rred IVA
0.50
122
13. 0.25
0.25
0.25
0.00
0.0
0.00
0.25
0.50 0.75
0.0
0.00
0.25 0.50
0.00
0.25
0.50
0.75
15 (81.20%5)
16
(81.18%)
17 (81.18%)
18 (81. 18%)
19
16%61
1.0
1.0
3.0
1.0
0.4
0.5
rend 18 a.1
0.5
rend 17 2.39
wed 1:21
0.5
rwerds 6.74
0.5
hard
2.33)
Fard 3.21
0.0
0.00
0.25
0.50 0.75
0.0
0.5
0.00
0.25
0.50 0.75
0.0
0.5
0.00
0.25
0.50 0.75
20 00.(811110)
181.14%)
22 (81.14%%
23 (81.12%)
24 (81.125)
1.0
1.0
0.75
0.5
herd 12 12:
0.5
may
kedl 3:1
4..00
red
0.50
0.25
Cas 12.1
hard 3)
hard 1.2
0.0
9.00
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Figure 19: A-F-L
(75.57%)
(75.333%
(75.33%)
(75.29%
(75.27%)
1.0
1.0
say
000 15, 221
0.50
hard H4, 304
NWS 14, .0
0.25
4M0/00 03
me.1 13, (50
0.0
0.0
0.0
0.00
0.00
0.25
0.50
0.75
0.30
0.25 0.50 0.75
0.00
0,25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
5
(75.20%)
(75.13%)
(75.08%)
(75.08%)
(75.07%%
1.0
1.0
OETH 21
e019 12. 351
oaly 16, 50
ear 18, 75
enry (4, 751
0.5
0.5
0.5
end 11
0.
0.5
ved 31
hard
hadd 251
0.0
0.0
0.0
0.0
0.0
0.25
0.50
0.75
0.30
0.25
0.50
0.00
0.25
0.75
0.00
0.50
0.75
0.25
0.50 0.75
10
(0550003))
11
(75.02%)
12 (75.01%)
13
(75.00%)
14
(75.00%5
1.0
1.0
1.0
3.0
1.0
cnrit 750
0.5
Wen (4, 114
0.5
rad
0.5
and 11
0.5
red 14, 11
0.0
0.0
0.0
0.0
0.0
0.30
0.25
0.50
0.0
0.5
0.00
0.50
0.25
0.50
15 (74.95%) 95%)
16
(74.95%)
174.999%%
18
(74.99%)
19 (74.9993%
1.0
1.0
1.00
cne 15,
1.0
0.75
0.75
hand 75
0.50
0.5
rad 21
0.5
0.50
0.5
loed .50
amy 19, 30
0.25
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.0
0.5
20 (74, 90%)
21
(74.88%)
22
(74.86%)
23 174, 84%))
24
(74.83%)
1.0
1.0
1.0
1.0
0.5
mad
21
0.5
14,
hund 331
red
0.0
0.0
0.0
0.30
0.25
0.50
0.75
0.30
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Figure 20: C-B-E
22

---

Under review as a conference paper at ICLR 2022
(72.49%)
2.122.40%)
(722.40%
(.22.355)
(72.33%%
1.0
1.0
1.0
0.5
0.5
med .70
medd 31
med 10, (5)
reod 10, 770
0.0
0.0
0.0
0.0
0.0
0.0
0.5
0.0
0.5
0.0
0.5
0.0
0.5
0.00
0.25
0.50
0.75
(72.27%)
6.22.23%%
(72.20%)
(72.09%%
(72.07%)
1.0
1.0
1.0
1.0
rest 31
hard 24 220
hard (1)
0.5
0.5
0.5
PX 1)
0.0
11 mg
0.0
0.0
0.0
0.0
0.5
0.00
0.25
0.50
0.00
0.75
10 (72.06%)
11 (72.03%)
12 (72.00%)
13 (72. 91%)
14 (71.706)
1.0
say
1.0
cat (33
1.0
hhid 16, 754
251
0.5
0.5
0.5
0.5
eagy
0.0
Foard 16 .70
0.0
0.0
hord 64,
0.0
0.5
0.0
1.0
0.0
0.5
0.0
0.5
0.25
0.50 0.75
15
(71.67%)
16
(71.00%)
17
(71.58%)
18 (71.62%) 62%)
19 (71.62285
1.0
1.0
1.0
hord
hard
0.5
0.5
0.5
0.5
0.5
hard 751
601 (3. 5
enp
0.0
hard
hard 12)
0.00
0.25
0.50
0.75
0.0
0.5
0.0
0.5
0.0
0.5
1.0
0.00
0.25
0.50 0.75
20:
(71.59%)
21
(71.58%)
22
(71.50%)
(71.566)
24
(71.50%)
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
13.
(1o,
06
-51
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25 0.50
0.75
0.00
0.25
0.50
0.75
0.0
0.5
Figure 21: C-B-L
(73.75%)
1.73.75%)
173.755%)
(73.75%)
(73.75%%
1.0
1.0
0.8
eaay 110,
0.8
eaay (28,
0.8
aay 530
say F36
0.8
40ap 130 3:21
hard 11, 2521
hwrd 12 4.35
aed 15
hard 11
0.6
0.6
0.6
0.6
0.6
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
0.00
0.25 0.50 0.75
0.00
0.25 0.50
0.75
0.00
0.25 0.50 0.75
(73.75%)
(73.755
(73.75%%
(73.75%%
(73.555%)
1.0
1.0
0.8
eany 130 3:30
eaay (20. 1.20
say 538 1.21
Kmy 138, 3:1
4npp 31
red LM1
Fe0 1:11
hard 12,
hard 1.31
hard 251
0.6
0.6
0.25 0.50 0.75
0.50
0.75
0.00
0.00
0.25 0.50
0.00
0.25 0.50 0.75
(73.418%
173.30%)
(73.30%%
(73.311%
(73.3111
1.0
1.0
3.0
1.0
1.0
0.8
redd 1.M
+10, 3.20
0.9
0.8
eed 1.
238.D
070 1.2)
136: 2.11
0.8
rad 1:2
1330 3:2
0.8
oup
0.5
0,7
hard 12, 3.20
0.25 0.50 0.75
0.25
0.50
0.75
0.00 0.25 0.50
0.00
0.25
0.50
0.00
0.25 0.50 0.75
15
(73.315)
16 173.28%)
27 (73.19%)
18 (73.18%)
(73.1665
1.0
1.00
18, 770
1.0
1.00
0.8
0.75
20
0.8
0.75
read
14 11
Fee 200
red a1
0.50
0.0
77
0.5
0.6
0.6
0.25
0.25
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0:75
0.00
0.25
0.50
0.75
0.00
0.25
0.50 0.75
(73.1115%
(73.15%)
22 (73.15%)
23
(73.14%%
24 (73.12%)
1.00
1.0
1.00
1.00
1.9
0.75
0.50
1h. 6:30
0.8
4.2
025
0.50
0.75
Mey
12
0.75
0.50
hurd 22 3.21
med
0.6
med
0.25
redd
0.6
0.25
0:25
0.4
0.00
6.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Figure 22: C-F-E
23

---

Under review as a conference paper at ICLR 2022
(74.01%)
(73.92%)
(73.88%)
((73.89%
(73.877%)
0.75
cap
herd 75 751
0.25
G0A 200
0.3
Padd 30
0.0
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.0
0.5
(73.80%)
(73.79%))
(73.79%)
SE
000
0.9
0.5
0.5
0.5
hord 130
rend
hard
0.0
0.0
0.75
0.00
0.00
0.50
10 (73.5776)
(73.57%)
12 (73.55%)
(33.5554%
14 (73.5353)
1.00
1.0
cew herd 14. 50
hart
kurtuct 31
red 22
0.5
0.5
05
0.5
say 10,
0.25
0.0
hard +10, 5
0.0
0.0
0.50
15 (73.49%)
16 (73.48%)
17
(73.499)%
18 (73 (73.44%) 44%)
19
(73.445)
1.0
1.0
1.0
1.00
0.75
0.5
mad 24.251 251
0.5
mad 16, 201
med
0.5
hard-33
0.25
herd 130.
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.00
0.25
0.50 0.75
20
(73.445)
(73.40%)
22 (73.36%)
23
(73.331)
24
1.0
1.0
1.00
1.0
0.75
0.5
0.5
16,
medic
kani 50
0.50
kasi 1.21
0.25
0.0
0.5
0.00
0.25
0.50
0.75
0.00 0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Figure 23: C-F-L
24


Reviews:
[{"review_id": "s4pED1kpR9w", "reviewer": "Reviewer_gK7s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the approach appears to be rather constrained (only 3 difficulty levels; limited to sigmoids; reliance in part on multi-annotator labels), may be onerous to implement (pre-training a baseline; fitting the sigmoids; making manual decisions on entropy vs loss) and was validated on rather small data (balanced datasets with <10k samples) to be useful in practice.\n\nStrengths:\n1. Addresses a hard unsolved problem.\n2. Modeling simplicity of the curriculum functional class (simple sigmoids that would cover most of the human-hypothesized heuristics; however, it may not be what we need - see below).\n\nWeaknesses:\n1. I find the approach conceptually different to classical curriculum learning, since it reweighs instances in the loss rather than schedules them in a particular order (except for the zero-weight case, equivalent to example skipping, where both ways converge); as far as could tell, SGD still samples data uniformly at any time. Besides, since the loss function changes, the baseline and the weighted tasks optimize different objectives and it's not straight-forward to compare them.\n2. Regarding sigmoids, I'm not convinced that practical cases of ML curricula can be covered by such simple monotonic schedules; rather, the functional class should be capable of dynamically changing the scheduling weight depending on the learning stage to mitigate forgetting and revisiting learned difficulty levels (and this encompasses monotonic functions too). In fact, such bandit-based curricula increasing/decreasing importance dynamically have been already proposed (Graves et al, 2017, https://arxiv.org/abs/1704.03003) and even evaluated for NLP tasks beyond classification (Kreutzer et al, 2021, https://arxiv.org/abs/2110.06997). The draft could be completed with a discussion of why the approach of Graves et al. is not considered or, better, a comparison experiment could be added. Note that the bandit approach is free from limitations such as number of difficulty levels.\n3. The requirement to pre-train the baseline prior to curriculum-enabled training defeats one of the main purposes of curriculum learning -- saving resources in the large data regime and relieving developer from manual training design. Again, dynamic curricula that are trained on the fly would be a more practical approach, as they don't require pre-training.\n4. The entropy-based curriculum relies heavy on assumed human-perceived difficulty, which may not be the same for the network's point of view. The fact, that Zhang et al. (2018) and Kocmi & Bojar 2017 both found that reverse curriculum works similarly well, witnesses that human intuitions of what is difficult for the network may be wrong.\n5. It looks like data balancing increases the gap to the no-curriculum approach. This again changes the task (in addition to different objectives) and raises the question of practical relevance for tasks where such balancing is not desired or not possible (seq2seq).\n\nMinor:\n- i found Fig. 1 redundant, it is just illustrating what is sigmoid function\n- fuzzy wording in a few places (\"good metric\", \"fairly distributed\", \"significant variance\")\n- change \\citet to \\citep in some places, Richards (1959), Shannon (2001)\n- sec 3.2: \"configurations is\" -> \"configuration is\"\n- \"in Nie et al (2020), which\" -> \"by Nie et al (2020), who\"", "summary_of_the_paper": "The paper proposes to learn training curricula by considering 3 sigmoids (representing \"easy\", \"mid\" and \"hard\" difficulty levels), that would define instances' weight as a function of time. The sigmoids' parameters are fitted to 3-bucket values of entropy of multi-annotator labels (that need to pre-exist), or of the instances' loss value dynamics (as provided by a baseline, no-curriculum, model that also needs to be trained). The paper emphasizes that the approach can replicate some of existing curriculum heuristics (e.g. easy-to-hard, hard-to-easy etc.), which however so far failed to produce a general recipe of training deep models, -- this diminishes the value of this flexibility. Finally, the approach is illustrated on 3 small to mid-size datasets with largest gaps to no-curriculum achieved on reduced datasets to re-balance classes.\n\n", "summary_of_the_review": "In my opinion, the proposed approach is too limited to be of practical usefulness, and its weaknesses outweigh the strengths so that the current draft is below the ICLR bar.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y4k2H7mVw0", "reviewer": "Reviewer_mHjR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The authors conducted a braod set of experiments.\n+ The paper is easy to follow.\n\nWeaknesses:\n- The main contribution of this work is to add changing weights to easy, medium, and hard training subsets, which an incremental development in my opinion. \n\n- At the beginning of section 2.3, the authors should acknowledge curriculum methods that do not maipulate data samples, such the \"model-level\" curriculum approaches surveyed in [A].\n\n- The approach divides the training data into 3 parts {easy, medium, hard}, based on a difficulty score. Ablation results with a different number of parts and other difficulty scores should be presented. I think it is particulary interesting to see what happens if th examples are randomly partitioned, i.e. does the difficulty play an important role?\n\n- The introduction leaves the impression that weights for data shards are somehow predicted by the model at each iteration. However, in section 2.4, the reader learns that \"curriculum discovery\" is based on hyperparameter tuning. The claims from the introduction should be toned down.\n\n- The performance gains shown in Figure 4 are rather small (wintin the standard deviations) compared to competing curriculum methods. The same applies to the supplementary.\n\nFormatting and language mistakes:\n- \"Without a curriculum learning will be an intractable\" => \"Without a curriculum, learning will be an intractable\";\n- \"Nie et al. (2020), which also studies\" => \" Nie et al. (2020), who also study\";\n- many citations are wrongly formatted, e.g. \"logistic functions Richards (1959)\" should be \"logistic functions (Richards, 1959)\".\n\nReferences:\n[A] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. arXiv preprint arXiv:2101.10382, 2021.", "summary_of_the_paper": "The paper presents a curriculum learning approach applied to NLP models. Texts are separated into easy, medium and hard subsets based on a difficulty score that is given a standard NLP model. Hyperparameter tuning is used to determining evolving weights for each batch. The approach is evaluated on three benchmark datasets (SNLI, Alcohol, Cancer).", "summary_of_the_review": "In my opinion, the weaknesses outweigh the strengths of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XdCNRJ7VKoL", "reviewer": "Reviewer_XfAD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\n- Proposed curriculum method encompasses other well known curricula by parameterizing the data partition and weighing schemes.\n- Data partitioning is effective, on the three datasets the authors evaluate upon, as is evident from improvements over no curriculum approach.\n- Specialized curricula obtained using curriculum discovery improves over no curriculum and other state of the art approaches.\n- Sample weight patterns shows the relative importance of samples on different datasets, which is interesting.\n- One quite interesting advantage of this approach is that curriculum discovered for smaller, balanced datasets work well on larger datasets.\n\n#### Weaknesses\n- The proposed method partitions a dataset into three classes $\\textit{easy}$, $\\textit{medium}$ and $\\textit{hard}$. Will this partitioning scheme work for different types of datasets, where there might be multiple levels of difficulty, multiple sub tasks? How will this method scale? \n- Using human agreement as a measure of entropy might not be applicable to datasets where there is less ambiguity between samples. That will cause majority of samples to be in a particular partition. For those datasets using entropy as a measure to partition samples might not be optimal. \n-  In Section 2.3, the authors describe how the proposed framework encompasses other well known CL approaches. It will be interesting to add more analysis on how often the TPE algorithm(Section 2.4) discovers curriculums where the parameters fall into pruning or sub-sampling strategy space. \n- How does this approach scale to larger datasets in vision like Imagenet, CIFAR etc, that other SOTA methods like SuperLoss evaluate on? Authors evaluate on three datasets, not all of which are well known to the community. I will encourage the authors to also evaluate this approach on more popular datasets.\n- How significant are the improvements over other baselines? Showing average over full and balanced datasets(Fig 4) hides some important issues. It seems from Figure 10, that DP method outperforms the proposed CL method on full datasets. We should pay more interest to this number as we would want any approach to work well on full dataset, rather than a subset(balanced) dataset. \n- Hard examples are down weighted more aggressively in this approach. It can be seen that the proposed approach often lags behind other approaches like No-CL, MentorNet, DP when it comes to hard examples. This issue is more prominent when it comes to full datasets. Authors should provide some additional analysis on these hard samples from these datasets for more clarity. What % of these datasets are partitioned into $\\textit{hard}$ class?", "summary_of_the_paper": "This paper proposes a new parameterized data partitioning and weighing scheme, that partitions data into three groups {easy, medium, hard} and determines a curriculum based on relative importance of different samples. They evaluate on three datasets (full and balanced versions) and show improvements over other CL approaches. The curriculum also provides interesting insights about the datasets and scale from smaller datasets to larger datasets. ", "summary_of_the_review": "I like the idea of encompassing different curriculum approaches inside a parameterized function. The authors show improvements over other approaches on three datasets with curriculum discovered by this approach. My reservations are mostly around how well this approach will scale to larger datasets and on unbalanced datasets. Some results show that other methods are better when it comes to full datasets. Due to these reasons, I will am recommending the current score. I encourage authors to address these concerns and I will be happy to bump up my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new curriculum learning framework by parameterizing data partitioning and weighting schemes. Extensive experiments are performed on three different datasets to demonstrate the effectiveness of the proposed framework. The reviewers acknowledged that the proposed framework is interesting as it encompasses several existing curriculum learning methods. However, the reviewers pointed out several weaknesses in the paper and shared concerns, including the scalability of the framework to larger datasets and the significance of the improvements over baselines. I want to thank the authors for their detailed responses. Based on the reviewers’ concerns and follow-up discussions, there was a consensus that the work is not ready for publication. The reviewers have provided detailed feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject