Ling-CL: Understanding NLP Models through Linguistic Curricula
Authors: Mohamed Elgaar
Hadi Amiri
Venue: EMNLP 2023 Main (2023)
Keywords: linguistic index
curriculum learning

Abstract: We employ a characterization of linguistic complexity from psycholinguistic and language acquisition research to develop data-driven curricula to understand the underlying linguistic knowledge that models learn to address NLP tasks. The novelty of our approach is in the development of linguistic curricula derived from data, existing knowledge about linguistic complexity, and model behavior during training. Through the evaluation of several benchmark NLP datasets, our curriculum learning approaches identify sets of linguistic metrics (indices) that inform the challenges and reasoning required to address each task. Our work will inform future research in all NLP areas, allowing linguistic complexity to be considered early in the research and development process. In addition, our work prompts an examination of gold standards and fair evaluation in NLP.

Ling-CL: Understanding NLP Models through Linguistic Curricula
Mohamed Elgaar and Hadi Amiri University of Massachusetts Lowell (melgaar hadi yecs. umlledd
Abstract
Sample Count
500
Accuracy
We employ characterization of linguistic plexity from psycholinguistic and language ac- quisition research to develop data-driven cur- ricula understand the underlying linguistic knowledge that models learn to address NLP tasks. The novelty of our approach velopment of linguistic curricula derived from data, existing knowledge about linguistic com- plexity, and model behavior during training, By analyzing several benchmark NLP datasets, our curriculum learning approaches identify sets of linguistic metrics (indices) that inform the challenges and reasoning required to address each task. Our work will inform future research in all NLP areas, allowing linguistic complex- ity to be considered early in the research and development process. addition, our work prompts examination of gold standards and fair evaluation NLP.
250
40%
the de-
Verb Variation
Figure I: Data distribution and trend of model accuracy against the linguistic index verb variation computed on ANLI (Nie et al., 2020) validation data. Samples with greater verb variation are more complex and also harder the model classify. Such linguistic indices can inform difficulty estimation and linguistic curriculum development for NLP tasks.
for individual bins separately. The accuracy trend indicates that verb variation can describe the diffi- culty of ANLI samples to the model. In addition, the data distribution illustrates potential linguistic disparity in ANLI; see $3.4
1 Introduction
Linguists devised effective approaches deter- mine the linguistic complexity text data (Wolfe- Quintero et al., 1998; Bulté and Housen, 2012; Housen et al., 2019). There is a spectrum of lin- guistic complexity indices for English, ranging from lexical diversity (Malvern et al., 2004; Yu, 2010) to word sophistication (O'Dell et al., 2000; Harley and King, 1989) to higher-level metrics such as readability, coherence, and information entropy (van der Sluis and van den Broek, 2010). These indices have not been fully leveraged NLP.
To reveal the linguistic knowledge NLP models learn during their training, we will employ known linguistic complexity indices to build multiview lin- guistic curricula for NLP tasks. A curriculum a training paradigm that schedules data samples in a meaningful order for iterative training, e.g., by start- ing with easier samples and gradually introducing more difficult ones (Bengio et al., 2009). Effective curricula improve learning in humans (Tabibian et al., 2019; Nishimura, 2018) and machines (Ben- gio et al., 2009; Kumar et al., 2010; Zhou et al.. 2020; Castells et al., 2020). Curriculum learning has been found effective in many NLP tasks (Set- tles and Meeder, 2016; Amiri et al., 2017; Platanios et al., 2019; Zhang et al., 2019; Amiri, 2019; Xu et al., 2020; Lalor and Yu, 2020; Jafarpour et al., 2021; Kreutzer al., 2021; Agrawal and Carpuat, 2022: Maharana and Bansal, 2022). A multiview curriculum is a curriculum able to integrate multi- ple difficulty scores simultancously and leverage their collective value (Vakil and Amiri, 2023).
We investigate the explicit incorporation of lin- guistic complexity of text data into the training process of NLP models, aiming to uncover the linguistic knowledge that models learn to address NLP tasks. Figure 1 shows data distribution and accuracy trend of Roberta-large (Liu et al., 2019) against the linguistic complexity index "verb vari- ation" (ratio of distinct verbs). This analysis conducted on ANLI (Nie et al., 2020) validation data, where balanced accuracy scores are computed

---

We assume there exists a subset of linguistic complexity indices that are most influential learn- ing an NLP task by a particular model. To identify these indices for each model and NLP task, we de- rive a weight factor P: E [-1,1] for each linguistic index that quantifies how well the index estimates
Algorithm 1 Correlation Method
Require: Derains Dyat, Model e, Optimizer g, Loss func-
tion f. Curriculum
1: step < --
2: / random initialization
3: while step < total_ steps do
4: training_ batch + SamppcBatch(stee. Dirain)
5:
loss <- ComputeLosss (training batch, θ)
the true diffieulty of data samples to the model,
6:
ling + Getl anutuunniiiiiccacurrtrtraaaa
batch)
determined by model instantancous loss against validation data. By learning these weight factors,
7:
difficultyy + SasallulateDiffuullyyytttto p)
8:
confidence <- DetermineeConfideme
difficulty)
9:
weighted_ioo leosseedd_osdeeeee + loss @ confidence e < UpdateModel (weighted Joss. θ)
we obtain precise estimations that shed light on the core linguistic complexity indices that each model
10:
11:
if step % validation step = 0 then
12:
I ComputeLoss (D..1, ling <- GeetLinguissticeFettusss...... for pi E p
needs at different stages of its training to learn an NLP task. In addition, these estimates can be read-
13:
14:
ily used for linguistic curriculum development, e.g..
15:
--epposorr( ling(:.
16:
end for end if
by training models with linguistically easy samples (with respect to the model) and gradually introduc- ing linguistically challenging samples.
17:
18:
step f- step
19:
end while
To achieve the above goals, we should address First, existing curricula are often limited to a single criterion of difficulty and are not applicable to multiview set- tings. This is while difficulty is a condition that can be realized from multiple perspectives, can vary across a continuum for different models, and can dynamically change the model improves. Sec- ond, existing approaches quantify the difficulty of data based on instantaneous training loss. However, training loss provides noisy estimates of sample difficulty due to data memorization (Zhang et al., 2017; Arpit et al., 2017) in neural models. We will address both issues as part of this research. The contributions of this paper are:
two gaps in the existing literature:
2 Multiview Linguistic Curricula
We present a framework for multiview curriculum learning using linguistic complexity indices. Our framework estimates the importance of various lin- guistic complexity indices, aggregates the result- ing importance scores to determine the difficulty of samples for learning NL.P tasks, and develops novel curricula for training models using complexity in- dices. The list of all indices used is in Appendix A.
2.1
Linguistic Index Importance Estimation
2.1.1 Correlation Approach
Given linguistic indices (X,) وززز of n data sam- ples, where k is the number of linguistic indices and X, R start by standardizing the in- dices {Z, = X weight factors for indices {p,) {o,} which are ran- domly initialized at the start of training. At every validation step, the weights are estimated using the validation dataset by computing the Pearson's 's correlation coefficient between loss and linguis- tic indices of the validation samples Pj- = r(1, Z,) where r is the correlation function and 1 E R. is the loss of validation samples. The correlation factors are updated periodically. It is important to use vali- dation loss as opposed to training loss because the instantaneous loss of seen data might be affected by memorization in neural networks (Zhang et al., 2017; Arpit al., 2017: Wang et al., 2020). This is while unseen data points more accurately represent the difficulty of samples for a model. Algorithm presents the correlation approach.
incorporating human-verified linguistic com- plexity information to establish an effective scoring function for assessing the difficulty of text data with respect to NLP models,
;; 1==1- We consider importance
deriving linguistic curricula for NLP models based on linguistic complexity of data and model behavior during training, and identifying the core sets of linguistic complex- ity indices that most contribute to learning NLP tasks by models.
We evaluate our approach on several NLP tasks that require significant linguistic knowledge and reasoning to be addressed. Experimental results show that our approach guistic knowledge that is most important for ad- dressing NLP tasks. In addition, our approach obtains consistent performance gain over compet- ing models. Source code and data is available at https:/ / //gi thib. CWL-LLLLUUMLLLEL
uncover latent lin-

---

2.1.2 Optimization Approach
Algorithm/ 2 Optimization Method
Let E Raxk be the matrix linguistic indices computed for n validation samples and с IR" indicate the corresponding loss vector of validation samples. We find the optimal weights for linguistic indices to best approximate validation loss:
Require: Divuin. Dout. Model @ Optimizer Optimizer h,
Loss function f. [Optional] Curriculum C
1: step <-
2: P random initialization
3: while step < fotal__st do
4: training_batte batch + SampleBatch(stepp Dirain)
5: loss + -oomputel (training b_aach, b)
6: ling + GeeLinngissicceeeuueee (tatiuesttrrainnngg batch)
p = arg min P
|| |1- | + Apllel1,
(1)
7:
difficulty + Calculate
p)
8: confidence <-
difficulty)
9: weighted loss <- loss @ confidence
10: e < UpdateModel (weighted llsss 9)
where Ap E R and pt E & is jointly optimized
11:
if step % validation step = 0 then
over all indices. The index that best correlates with loss can be obtained as follows:
12:
I Computelloss (Deat, Q)
13:
ling <- GetLLinguistceFeattuess (Don)) p. mrgmin, |ing / Al] + Aplpp
14:
15:
end if
i" arg min || Zi.ppill.
(2)
16: step f- step +
17:
end while
where Zui denotes the 5th column of Z. Algo-
rithm 2 presents this approach.
2.2 Linguistic Curriculum
We note that the main distinction between the two correlation and optimization approaches lies in their scope: the correlation approach operates at the index level, whereas the optimization approach uses the entire set of indices.
We evaluate the quality of weighted linguistic in- dices difficulty score and introduce three new curricula based on moving logistic (Richards, 1959) and Gaussian functions, Figure 2.
2.2.1 Time-varying Sigmoid
2.1.3 Scoring Linguistic Complexity
We develop a time- vareing sigmoid function produce weights (Eq. 3). The sigmoid function as- signs a low weight to samples with small difficulty scores and a high weight to larger difficulty scores. Weights are used to emphasize or de-emphasize the loss of different samples. For this purpose, we use the training progress t E [0, 1] as shift pa- rameter, to move the sigmoid function to the left throughout training, so that samples with a small difficulty score are assigned a higher weight in the later stages of training. By the end of the training, all samples are assigned a weight close to 1. Addi- tionally, we add a scale parameter 3 E [L, oo) that controls the growth rate of weight (upper bounded by 1) for all samples.
We propose methods for aggregating linguistic indices (X,)* 3 and their corresponding importance factors { k into a linguistic complexity score. The first method selects the linguistic index with the maximum importance score at each timestep:
S = Ziz arg max Pj.
(3)
which provides insights into the specific indices that determine the complexity to the model.
The second method computes weighted aver- age of linguistic indices, which serves as a difficulty score. This is achieved as follows:
St EzpsZin
V EEg
t(S,,:;3) 1 exp(-SS - -- B)
(5)
where S; C RR (HS,, os.) = (0, 1), is an aggregate of linguistic complexity indices for the input text. If an index Zj is negatively correlated with loss, Pj will be negative, and Z₁ will be positively correlated with loss. Therefore. S; is an aggregate complexity that is positively correlated with loss. And using weighted average results in the indices that are most highly correlated with loss to have the highest contribution to S,-
The sigmoid function saturates as the absolute value of its input increases. To account for this issue, our input aggregated linguistic index follows the standard scale, mean of zero, and variance of one, (4) and (3).
2.2.2 Moving Negative-sigmoid
The positive-sigmoid function assigns greater weights to samples with large value for S that

---

Confidence
Confidence
Confidence
0.8
0.6
0.6
0.6
6.
0.
0.2
-2 0 2 Linguistic Complexity
-2
Linguistic Complexity
Linguistic Complexity
(a) Sigmoid
(b) Negative Sigmoid
(c) Gaussian
Figure the beginning training. the sigmoid function with the lowest opacity is used. the right-most curve in (a), the left-most curve in (b), and the middle curve in (c). Then, training progresses, the function is shifted using the parameter L in (5) and (7), causing samples with a higher complexity to be assigned higher confidence if (a) is used, samples with lower complexity be assigned higher confidence (b) used, and samples with medium complexity to be assigned higher confidence if (c) is used.
are linguistically more complex. In order es- tablish a curriculum that starts with easy samples and gradually proceeds to harder ones, we use a negative sigmoid function:
2.2.4 Weighting-based Curriculum
We define a curriculum by weighting sample losses according their confidence. Samples that are most useful for training receive higher weights, and those that are redundant or noisy receive smaller weights. Weighting the losses effectively causes the gradient update direction to be dominated by the samples that the curriculum thinks are most useful. Weights w are computed using either Equa- tion 5, 6 or 7:
wn( u(S.,t;)) + exp (S: - 8)
(6)
Figure 2 illustrates the process of time-varying positive and negative sigmoid functions. Over the course of training. larger intervals of linguistic com- plexity are assigned full confidence, until the end of training when almost all samples have a confidence one and are fully used in training.
Σ w ((.,,;;)) 4,
- w(
(8)
E.
where l is the loss of sample i, t the current train- ing progress, and L is the average weighted loss.
2.2.3 varying Gaussian Function
We hypothesize that training samples that are not too hard and not too easy are the most useful in training, and should receive the most focus. In fact, samples that are too easy hard may contain arti- facts that are harmful to training, may contain noise, and may not be generalizable to the target task. Therefore, we use the Gaussian function to prior- itize learning from medium-leve samples. The function starts with a variance of 1, and scales up during the course of training so that the easier and harder samples, having lower and higher linguis- tic complexity values, respectively, are assigned increasing weights and are learned by the end of training. We propose the following function:
2.3 Reducing Redundancy in Indices
We have curated list of 241 linguistic complex- ity indices. In the case of text pair input (e.g. NLI), concatenate the indices of the text inputs, for a total of 482. Our initial data analysis reveals significant correlation among these indices in their estimation of linguistic complexity. To op- timize computation, avoid redundancy, and ensure no single correlated index skews the complexity ag- gregation approach 2. 1.33, we propose two methods to select diverse and distinct set of indices for our study. We consider the choice of using full indices or filtering them as hyper-pane paramee.
In the first approach, for each linguistic index, we split the dataset into m partitions based on the index values (similar to Figure l). Next, using a trained No-CL ($3.3) model, we compute the accu- racy for each partition. Then, we find the first-order
S?
w(Sn,ti)) = exp( 2(1 (1l+t-7)
(7)
where y is the rate of growth of variance and is the training progress, see Figure 2.
We use numpy. hi sestttrammbii_edeee.

---

3 Experiments
3.1 Datasets
04
233
We evaluate NLP models in learning the tasks of the following datasets:
100
SNLI:
Stanford Natural Language Infer- ence (Bowman et al., 2015). The task is classify a pair of sentences by the relation be- tween them as one of entailment, neutral, or contradiction.
240
246
(a) Pair-wise correlation between indices
84
CoLA: Corpus of Linguistic Acceptabil- ity (Warstadt et al., 2019). It is task of classifying sentences as grammatical vs. un- grammatical.
ANLI: Adverserial Natural Language Infer- ence (Nie et al., 2020). This NLI dataset was created with model in the loop, by only adding samples to the dataset that fool the model. We train only the ANLI training set of 162k samples.
24
240
(b) Clustered correlation matrix.
Figure 3: Eliminating redundancy in linguistic indices.
(a) shows the Pearson's correlation coefficient between each pair of linguistic indices. (b) is created by re-
SST-2: Stanford Sentiment Treebank (Socher et al., 2013). The task is to predict the senti- ment of a given sentence as positive or nega- tive.
ordering the rows and columns of (a), such that mutually correlated indices clustered into blocks using hier-
archical clustering (Kumar et al., 2000). Best seen in color; lighter areas indicate greater correlations among index pairs groups.
RTE: Recognizing Textual Entailment (Wang al., 2018). The task is to determine if given sentence entailed by another given sentence.
accuracy trend across these partitions. Linguistic indices with pronounced slope describe great vari- ance in the data and are considered for our study; we select the top 30% of indices, reducing their count from 482 144 for text pair inputs.
AN-Pairs: Adjective-noun pairs from the Cam- bridge ESOL First Certificate in English (FCE) exams (Kochmar and Briscoe, 2014). The task is detect if an adjective-noun pair, including pairs that are typically confusing language learners, is used correctly the context of a sentence.
In the second approach, we compute pair-wise wise correlations between all indices. Then, we group highly correlated indices, as shown in Figure 3. From each cluster, we select representative index, aiming to prevent correlated indices from domi- nating the aggregation approach and to eliminate redundancy. This method narrows our focus to the following 16 key indices: 1) type-token ratio (TTR), 2) semantic richness. 3) ratio of verbs to tokens, 4) mean TTR of all word segments, 5) To- tal number of verbs, 6) number of unique words, 7) adverbs per sentence, 8) number of unique words in the first k tokens, 9) ratio of nouns to verbs, 10) semantic noise, 11) lexical sophistication, 12)
GED: Grammatical Error Detection (Yan- nakoudakis et al., 2011) 1). The task is to iden- tify grammar errors at word level in given sentences.
3.2 Difficulty Scoring Functions
The curriculum learning approaches in $2.2 use difficulty scores or compute confidence to quantify sample difficulty in order to rank sentences. We use as difficulty scores: aggregate linguistic complexity Ling, see Section 2.1.3, and Loss (Xu et al.. 2020: Wu al., 2021; Zhou et al., 2020). We take the loss from a proxy model (No-CL in $3.3) by record- ing all samples losses two times per epoch during training and computing the sample-wise average.
verb sophistication, 13) clauses per sentence, 14) average UU CDlow value per token, 15) ad- jective variation, 16) ratio of unique verbs. Please refer to Appendix for definitions and references to indices.

---

3.3 Baselines
sigmoid, negative-sigmoid, and Gaussian weight- ing, as well as sampling an competence-basee ap- proaches, see$3.3. We test variants of our approach with the correlation method, optimization method, and indices filtering. We report results of the max aggregation ($2.1.3) approach as it performs better than the weighted average and is computationally cheaper. Loss-CL computes loss score by recording the losses of samples during training of No-CL. The loss during the early stages of training generated by an
We consider a no-curriculum baseline as well as several recent curriculum learning approaches.
No-CL; no-curriculum uses standard random mini-batch sampling from the whole dataset without sample weighting.
difficulty
Sampling (Bengio al., 2009) uses the easiest subset of the dataset each stage of training. Instead of randomly sampling a mini-batch from the whole dataset, a custom data sam- pler is created that provides the subset consist- ing of the easiest a% of data when training progress is at 0%.
model is good mcasure of the relative difficulty both training and validation samples.
3.4 Evaluation Metrics
SL-CL & WR-CL (Platanios et al., 2019) is a curriculum learning approach that defines a time-varying function of the model's compe- tence (defined as the fraction of training data that the model uses at every step), and dif- ficulty score of the data. At each iteration, minibatch sampled from the subset of data with difficulty smaller than the model's competence- pre-defined non-linear func- tion. The model employs sentence length (SL- CL) and word rarity (WR-CL) as difficulty measures. Sampling is the same as Compe- basce-based curriculum learning with linear competence function.
Linguistic disparity can be quantified by the extent of asymmetry in the probability distribution of the linguistic complexity of samples in a dataset, e.g.. Figure in $1. A natural solution evaluate models is group samples based on their linguis- tic complexity. Such grouping is crucial because if easy samples are overrepresented in a dataset, then models can result in unrcalistically high per- formance on that dataset. Therefore, we propose to partition datasets based on a difficulty metric (linguistic index or loss) and compute balanced ac- curacy of different models on the resulting groups. This evaluation approach reveals great weaknesses in models, and benchmark datasets or tasks that seemed almost "solved" such as as the complex tasks NLL.
SuperLoss (Castells et al., 2020) uses instan- taneous loss to compute task-agnostic sam- ple confidence. It emphasizes easy samples and de-emphasizes hard samples based on the global average loss as the difficulty threshold.
3.5 Experimental Settings
We use the transformer model roberta-base (Liu et al., 2019) from (Wolf et al., 2020), and run each experiment with at least two random seeds and report the average performance. We use AdamW (Loshchilov and Hutter, 2018) optimizer with a learning rate of 1 10-5, batch size of 16, and weight decay of 1 10-2 for all models. The model checkpoint with the best validation accuracy is used for final evaluation. In NLJ tasks with pair of text inputs, the indices of both texts are used. For Ling-CL, we optimize the choice of index impor- tance estimation method and aggregation method. For the baselines. optimize the parameters of SuperLoss (A and moving average method), and the two parameters of SL-CL and WR-CL mod- els for each dataset. For the data selection, we use a warm-up period of 20% the total training iterations.
Concat (Lee et al., 2021) concatenates linguistic indices to language embeddings before clas- sification. Lee et al. (2021) and Meng al. (2020) reported low performance appending features to embeddings. However, our approach succeeds utilizing concate- nated features.
result of
Data Selection (Mohiuddin et al., 2022) is an online curriculum learning approach. It evalu- ates the training data at every epoch and uses loss as the difficulty score. It selects the mid- dle 40% of samples according to difficulty.
We compare the above models against our ap- proaches, Ling-CL, which aggregates linguistic indices using weighted average or max-inder aggre- gation, and applies different curriculum strategies:

---

ANLI
COLA
RTE
SNLI
SST2
AN-Pairs
GED
Average
Ling-CL [NegSig]
1 255
72.4 1: 0:000
79.1 K
82.8 $38
92.2 0.222
79.1 1 1.5
75.3 1.87
77.2 1: 3.17
Ling-CI. [Gauss]
60.9 1
73.0 1 00000
77.2 808
83.5 1
92.4 0
82.9 1
75.5 0.4
77.9 1:
Ling-CL [Sig]
58.1 0.37
64.6 4: 891
78.7 887
83.0 1: 8.08
92.3 0.00
82.3 0.90
75.9 0.10
76.4 1: 3,92
Loss-CI. [NegSigl
59.0 1.3
55.6 0.64
68.1 1.59
75.1 0.05
91.6 0.36
76.4 1: 3.70
75. + 1:44
71.6 1:
Loss-CL. [Sig]
49.7 + 9.58
56.6 + 0:37
66.8 0.20
83.6 = 8.37
90.9 + 0.42
81.4 + 0.61
73.3 0.20
71.8 2.85
Loss-CI. [Gauss]
49.4 1107
57.0 4: 1:39
67.2 = 1:41
75.1 0.5
91.8 0.12
80.5 1: 2.08
74.5
0.00
70.8 +: 2,37
Sampling
49.9 + 10.00
64.6 + 8.89
67.9 0.03
83.2 = 9.72
91.5 0.07
82,6 + 3.93
73.8 1.23
73.4 #:47
Competence
50.1 + 11.27
63.4 + 9.08
68.8 0.64
74.7 + 0.06
91,6 Q.
84.0 + 1.14
74.1 0.39
72.4 3.223
SL-CI
50.3 + 10:5
55.8 + 0.06
67.7 1.27
82.6 = 8.35
93.1 0.00
81.6 0.72
75.2 0.35
72.3 + 2,06
WR-CL
50.9 1: 9.50
56.1 1 0.53
68.4 0.3
74.5 0.16
91.5 0.16
80.1 0.81
2:117
71.0 1:
SuperLoss
39.5 +: 0.14
56.9 + 0.69
69.6 0.50
75.2 0.14
91.7 0.26
77.8 + 1.89
74.2
0:01
69.3 + 0.54
Concat
51.3 1: 9.83
64.3 1: 6.8
71.4
75.2 0.2
91.9 0.14
81.8 1: 1.66
73.8
0.91
72.8 1: 3.6
Data Selection
46.8 +: 6.32
55.1
4 1.11
66.6
74.4
0.99
91.5 0.30
79.6 1.0
75.5 0.52
69.9 4: 1.67
No-CL
51.7 1 8.21
57.0
0:22
70.0 2 0.45
83.3
1: 8.02
83.7 8.22
82.1
1 0.51
74.0
0.14
71.7 3,34
Table I: Balanced accuracy by linguistic index (Word rarity). Accuracy the metrie for all datasets except CoLAA
and GED, CoLA uses Matthew's correlation and GED uses Fo=o.5 score. Ling-CL uses aggregate linguistic complexity as a difficulty score we create, and Loss-CL. uses the average loss of a sample throughout a full training.
3.6 Enhanced Linguistic Performance
3.7 Learning Dynamics for NLP Tasks
Identification of Key Linguistic Indices We an- alyze the linguistic indices that most contribute to learning NI.P tasks. For this purpose, we use the evaluation approach described in $3.4 for com- puting balanced accuracy according to linguistic indices. Table 2 shows the top three important linguistic indices for each dataset as identified by our optimization algorithm using the Gaussian cur- riculum. Importance is measured by the average p value. Early, middle, and late divide the training progress into three equal thirds. The top index in the early stage is the index with the highest average during the first 33.3% of training. The top indices are those that most accurately estimate the true dif- ficulty of samples, they should highly correlate with validation loss.
Tables 1 show the performance of different models when test samples are grouped based on word rar- ity. The results show that the performance of the baseline models severely drops compared to stan- dard training (No-CL). This is while our Ling-CL approach results in 4.5 absolute points improve- ment in accuracy over the averaged across tasks, owing to its effective use of linguistic indices. Appendix D shows the over- all results the entire test sets, and results when test samples are grouped based their loss; use loss because it is widely-used measure of difficulty in curriculum learning. These groupings allow for a detailed examination of the model's 's per- formance across samples with varying difficulty, providing insights into the strengths and weak- nesses of models. For example, the performance on SNLI varies from 89.8 to 90.6. However, when word rarity is used to group data based diffi- culty, the performance range significantly drops from 74.4 to 83.6, indicating the importance of the proposed measure of evaluation. We observe that such grouping does not considerably change the performance on ANLI, which indicates the high quality of the dataset. In addition, it increases model performance on AN-Pair and GED, which indicates greater prevalence of harder examples in these datasets.
baseline
Table 2 shows that different indices are impor- tant for different tasks. This means that not possible to use a single set of linguistic indices as a general text difficulty score, important indices can be identified for each task, which can be achieved by our index importance estimation approach ($2. and evaluation metric ($3.4).
Analysis of Linguistic Indices for Grammar Tasks We consider the grammar tasks for anal- ysis. For AN-Pairs (adjective-noun pair), during the early stage, the top indices are the number of tokens per sentence, age of acquisition (AoA) of words, and mean length of sentences. This is mean- ingful because longer sentences might introduce modifiers or sub-clauses that can create ambiguity or make more challenging to discern the intended adjective-nove noun relationship accurately. Regarding AoA, words that are acquired later in life or belong
On average, the optimization approach outper- forms correlation by 1.6% 1.9% accuracy in our experiments. Also notably, on average, the argmax index aggregation outperforms the weighted aver- age by 1.9% +1.9%, and the filtered indices out- perform the full list of indices by 1.4% +1.1%. 1%.

---

Early
Middle
Late Adverbs per sentence
Tokens per sentence
Lemmas age acquisition
AN-Pairs
Lemmas age of acquisition
Tokens per sentence
Corrected TTR
Mean sentence length
Adverbs per sentence
Nouns adjective ratio
Corrected noun variation
GED
Tokens per sentence Nouns per sentence
Nouns per sentence
Tokens per sentence
Tokeess per sentence
Nouns per sentence
Ratio of Adverbs to Verbs (P)
RTE
Ratio of Subordinating Conjunctions to Verbs (P)
Adverb Variation (P)
Verb sophistication (P)
Adverbs per sentence (P)
Lexical verb variation (P)
Function words
sentence (HI
ANLI
Unique Entities (P)
Log Tokens per log sentences
Unique Entities per token (P)
Complex nominals
SST-2
Noun variation
Ratio of nouns to verbs Function words
Verb variation
Coordinating Conjunctions
CoLA
Number of T-units
T-units per
Lemmas age acquisition (P)
SNLI
Linsear Write Formula Score (P)
Gunning Fog Count Score (P)
Table 2: Top three important linguistic indices at each stage of learning. For datasets with premise (P) and hypothesis (H), they are indicated in parentheses.
0.02
Balaneez ptma
66.0
Balanced Accuracy
0.03
boar
69.5
Balanced Accuracy
0.01
Beanceed ascurecy
64.8
Balaced Accuracy
0.02
65.8
0.01
64.5
69.0
0.02
0.01
64.2
65,2
68.5
0.02
0.01
64.0
65.0
63.8
0
20
40
20
40
20
40
Training Iterations
Training Iterations
Training Iterations
(a) # Unique words
(b) Unique sophisticated words
(c) # Verb sophistication
Figure 4: The progression the estimated importance factors p. and balanced accuracy for groups linguistic indices.
to more specialized domains might pose challenges in accurately judging the correct usage of adjective- noun pairs because of their varying degrees of fa- miliarity and potential difficulty associated with specific vocabulary choices.
nouns due to increasing syntactic complexity.
In the third stage, the number of adverbs per sentence becomes the top important index, while AoA and the number of tokens per sentence drop out of the top three. In the early stage, AoA and the number of tokens has p values of 0.168 and 0.164, respectively. In the late stage, they drop to 0.1 and 0.13, while the number of adverbs per sentence is 0. 138 early, and increases to 0.181 in the late stage. We that indices may become dominant not only by increasing their p value but also by waiting for other indices drop down when they have been learned by the model. Therefore, Ling-CI. can determine the order to learn linguistic indices, and then learn them sequentially.
During the middle stage the AoA increases in importance and remains challenging the model, the number of adverbs per sentence increases in rank and joins the top three indices. In the context adjective- noun pairs, the presence of multiple ad- verbs in a sentence can potentially affect the inter- pretation and intensity of the adjective's meaning. This is because adverbs often modify verbs, adjec- tives, or other adverbs in sentences. In addition, depending the specific adverbs used, they may enhance, weaken, alter the intended relationship between the adjective and the noun. Moreover, the presence of several adverbs can simply introduce potential challenges in identifying and correctly in- terpreting the relationship between adjectives and
Regarding GED, noun variation is the dominant index throughout the training process. Such varia- tion important because affects syntactic agree- ment, subject-vert agrecment, modifier placement, and determiner selection. These factors affect gram-

---

matical consistency and coherence within the sen- tence structure, leading to the importance of noun variation throughout the training process.
most constant across samples-75%% of the samples consist of a single sentence in the datasets.
Conclusion and Future Work
Dominant Indices for CoLA Task Regarding CoLA, the number of function words and coordinat- ing conjunctions indices are the dominant indices at the early stage, and middle and late stages of training respectively. These words are crucial in establishing the syntactic structure of a sentence. They directly contribute to agreement and refer- ences, coherence, and adherence to grammar rules. We note that T-units (independent/main clauses clauses with their associated subordinate clauses) are higher-order linguistic constituents that pro- vide information about the dependency relations between sub-constituents, and the overall coher- ence of sentences. Indices related to T-units are among the top three crucial indices.
We propose new approach to linguistic curricu- lum learning, Our approach estimates the im- portance of multiple linguistic indices and aggre- gales them, provides effective difficulty estimates through correlation and optimization methods, and introduces novel curricula for using difficulty esti- mates, to uncover the underlying linguistic knowl- edge that NLP models learn during training. Fur- thermore, we present method for a more accurate and fair evaluation of computational models for NLP tasks according to linguistic indices. Further- more, the estimated importance factors present in- sights about each dataset and NLP task, the linguis- tic challenges contained within each task, and the factors that most contribute to model performance on the task. Further analysis of such learning dy- namics for each NLP task will shed light on the linguistic capabilities of computational models at different stages of their training.
Trends and Relationships between and Bal- anced Accuracy We use the GED dataset (63.1)
to analyze the trends of p throughout training, and the relation between and balanced accuracy. Fig- ure 4 shows the progression of with the progres- sion of balanced accuracy for selected linguistic indices. This figure is produced using No-CL, We observe across several indices that p high when balanced accuracy is low, indicating that the index is challenging to the model and therefore used for learning with high p. and decreases as the in- dex is learned. However, Figure 4a shows that it is not necessary that when balanced accuracy in- creases p would decrease. In this case, it means that the model is performing relatively well on the index, but the index remains predictive of loss. So, although the average performance increased, the variation in performance among different values of the index remains high. We find that numerous in- dices follow the same of trend of In Appendix B, we propose method for clustering p to effectively uncover patterns and similarities in the learning of different indices. However, further analysis of the dynamics of p is the subject of our future work.
Our framework and the corresponding tools serve as a guide for assessing linguistic complex- ity for various NLP tasks and uncover the learn- ing dynamics of the corresponding NLP models during training. While we conducted our analysis seven tasks and extracted insights the key indices for each task, NLP researchers have the flexibility to either build on our results or apply our approach to other NLP tasks to extract rele- vant insights. Promising areas for future work in- clude investigations on deriving optimal linguistic curriculum tailored for each NLP task; examining and enhancing linguistic capabilities of different computational models, particularly with respect to linguistically complex inputs; and developing challenge datasets that carry fair distribution of linguistically complex examples for various NLP tasks. In addition, future work could study why spe- cific indices are important, how they connect to the linguistic challenges of each task, and how differ- ent linguistic indices jointly contribute to learning target task. We expect other aggregation func- tions, such log-average, exponential-averagee and probabilistic selection of maximum, be ef- fective approaches for difficulty estimation based on validation loss. Finally, other variations of the proposed Gaussian curriculum could be investi- gated for model improvement.
In addition, we find that the rank of top indices is almost constant throughout the training. This quality may be useful in creating an approach that gathers the indices rankings early on and utilizes them for training. Appendix E lists influential in- dices by their change in p across stages of training. We note that the "number of input sentences" index is the least important metric because the index is al-

---

5 Limitations
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 5e0.0008800
Our work requires the availability of linguistic in- dices, which in turn requires expert knowledge. Such availability requirements may not be fulfilled in many languages. Nevertheless, some linguis- tic complexity indices are language independent, such as the commonly-used "word rarity" measure, which facilitates extending our approach to other languages. Moreover, our approach relies the effectiveness of specific linguistic complexity in- dices for target tasks and datasets employed for evaluation; different linguistic complexity indices may not capture all aspects of linguistic complex- ity and may yield different results for the same task or dataset. In addition, the incorporation of linguistic complexity indices and the generation of data-driven curricula can introduce additional computational overhead during the training process. Finally, our approach does not provide insights into the the interactions between linguistic indices dur- ing training.
Bram Bulté and Alex Housen. 2012. Defining and op- erationalising 12 complexity. Dimensions of L2 per- formance and proficiency: Complexity, accuracy and fluency in SLA, pages 23-46.
Thibault Castells, Philippe Weinzaepfel, and Jerome Revaud. 2020. Superloss: A generic loss for robust curriculum learning. Advances in Neural Information Processing Systems, 33:4308 4319.
Birgit Harley and Mary Lou King. 1989. Verb lexis in the written compositions of young 12 learners. Stud- ies in Second Language Acquisition, 1 144):415-433.
Alex Housen, Bastien De Clercq, Folkert Kuiken, and Ineke Vedder. 2019. Multiple approaches complex- ity in second language research. Second language research, 35(1):3-21.
Kenneth Hyltenstam. 1988. Lexical characteristics of near-native lecond-anguuaee learners of swedish. Journal of Multilingual & Mullicultural Develop- ment, (1-2))67-84
Borna Jafarpour, Dawn Sepehr, and Nick Pogrebnyakov. 2021. Active curriculum learning. Proceedings of the First Workshop on Interactive Learning for Natural Language Processing, pages 40-45.
References
Sweta Agrawal and Marine Carpuat. 2022. An imita- tion learning curriculum for text editing with non- autoregressive models. Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 7550- 7563, Dublin, Ireland. Association for Computational Linguistics.
Proceedings of the 60th
Ekaterina Kochmar and Ted Briscoe. 2014. Detecting learner errors in the choice of content words using compositional distributional semantics. In Proceed- ings of COLING 2014, the 25th International Con- ference on Computational Linguistics: Technical Pa- pers, pages 1740-1751.
Hadi Amiri 2019. Neural self-training through spaced repetition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 21-31, Minneapolis, Minnesota. Association for Computational Linguistics.
Julia Kreutzer, David Vilar, and Artem Sokolov. 2021. Bandits don't follow rules: Balancing multi-face machine translation with multi-armes bandits. Findings of the Association for Computaaioonal Lin- guistics: EMNLP 2021, pages 3190-3206 Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Hadi Amiri, Timothy Miller, and Guergana Savova. 2017. Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2401-2410.
M Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. Ad- vances in neural information processing systems. 23: 13:1189-1197.
Devansh Arpit, Stanislaw Jastrzębski. Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017. A closer look at mem- orization in deep networks. In International confer- ence on machine learning, pages 233-242. PMLR.
Michael Steinbach George Karypis Vipin Kumar, M Steinbach, and G Karypis. 2000. A comparison of document clustering techniques. Department of Computer Science and Engineering, University of Minnesota.
John P Lalor and Hong Yu. 2020. Dynamic data selec- tion for curriculum learning via ability estimation. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing. Conference Empirical Methods in Natural Language Processing. volume 2020, page 545. NIH Public Access.
Yoshua Bengio. Jérome Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In ACM International Conference Proceeding Series, volume 382, pages 1-8, New York, New York, USA. ACM Press.

---

Bruce W Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A trans- former meets handcrafted linguistic features. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10669- 10686.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 4885
Moira Linnarud. 1986. Lexis in composition: A perfor- mance analysis of Swedish learners' written English. 74. CWK Gleerup.
Joel Nishimura. 2018. Critically slow learning in flash- card learning models. Chaos, 28(8):0831) 15.
Felicity O'Dell, John Read, Michael McCarthy, et al. 2000. Assessing vocabulary. Cambridge university press.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 1692.
Lourdes Ortega. 2003. Syntactic complexity measures and their relationship to 12 proficiency: research synthesis of college-level 12 writing. Applied linguis- tics, 24(4))492-518.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Emmanouil Antonios Platanios, Otilia Streteu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019, Competencee-asse curriculum learning for neural machine translation. In Proceedings of the Confer- ence of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HILT, pages 1162-1172.
Xiaofei Lu. 2010. Automatic analysis of syntactic com- plexity in second language writing. International Journal of Corpus Linguistics, 15:474-496.
Xiaofei Lu. 2012. The Relationship of Lexical Rich- ness the Quality of ESL Leamnrr' Oral Narratives, Source: The Modern Language Journal, 96(2):190- 208.
FJ Richards. 1959. A flexible growth function for empir- ical use. Journal of experimental Botany, 10(2):20- 301.
Burr Settles and Brendan Meeder. 2016. A trainable spaced repetition model for language learning. In Proceedings of the 54th annual meeting of the associ- ation for computational linguistics (volume 1: Long papers), pages 1848-1858.
Adyasha Maharana and Mohit Bansal. 2022. On cur- riculum learning for commonsense reasoning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 983-992, Seattle, United States. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
David Malvern, Brian Richards, Ngoni Chipere, and Pilar Durán. 2004. Lexical diversity and language development. Springer.
Erica McClure. 1991. A comparison of lexical strategies in 11 and written english narratives. Pragmatics and language learning, 2: 2:141-154.
Behzad Tabibian, Utkarsh Upadhyay, Abir De, Ali Zarezade, Berhhard Schölkopf, and Manuel Gomez- Rodriguez. 2019. Enhancing human learning via spaced repetition optimization. Proc. Natl. Acad. Sci. U. A., 116(10))3988-39)) -3933.
Gerard McKee, David Malvern, and Brian Richards. 2000. Measuring vocabulary diversity using dedi- cated software. Literary and linguistic computing, 15(3):323-338.
Mildred Templin. 1957. Certain language skills in children: Their development and intervelationships, volume 10. JSTOR.
Changping Meng, Muhao Chen, Jie Mao, and Jennifer Neville. 2020. Readnet: A hierarchical transformer framework for web article readability analysis. Eu- ropean Conference on Information Retrieval, pages 33-49. Springer.
Nidhi Vakil and Hadi Amiri. 2023. Multiview competence-basee curriculum learning for graph neu- ral networks. In Proceedings of the 61th Annual Meeting of the Association for Computational Lin- guistics (ACL 2023). Association for Computational Linguistics.
Tasnim Mohiuddin. Philipp Koehn, Vishrav Chaudbary, James Cross, Shruti Bhosale, and Shafiq Joty. 2022. Data selection curriculum for neural machine trans- lation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, Online. Associa- tion for Computational Linguistics.
Frans van der Sluis and Egon I. van den Broek. 2010. Using complexity measures in information retrieval. In Proceedings of the third symposium on information interaction in context, pages 383-388.

---

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:/804.07466
Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes, 2020. Curriculum learning by dynamic instance hardness. Advances in Neural Information Processing Systems, 33:8602-8613.
Xinyi Wang. Hieu Pham, Paul Michel, Antonios Anas- tasopoulos, Jaime Carbonell, and Graham Neubig. 2020. Optimizing data usage via differentiable re- wards. In International Conference Machine Learning, pages 9983-9995, PMLR.
Alex Warstadt, Amanpreet Singh, and Samuel Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Compitaaionnl Linguistics, 7:625-641.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of oftte-of-the-att natural language processing. In Proceedings of the 2020 con- ference on empirical methods natural language processing: system demonstrations, pages 38-45.
Kate Wolfe- Quintero, Shunji Inagaki, and Hae- Young Kim. 1998. Second language development writing: Measures of fluency, accuracy, and complexity. Hon- olulu, HI: University of Hawai'i, Second Language Teaching & Carriculum Center.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. 2021. When do curricula work? In International Confer- ence Learning Representations (ICLR).
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understand- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguis- human language technologies, pages 180-189.
tics:
Guoxing Yu. 2010. Lexical diversity in writing and speaking task performances. Applied linguistics, (1)):236-225.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. Understanding deep learning requires rethinking generalization. In 5th International Conference Learning Representa- tions. ICLR 2017, Toulon, France, April 24-26, 2017. Conference Track Proceedings. OvenReview.net.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul Mc- Namee, Marine Carpuat, and Kevin Duh. 2019. Cur- riculum learning for domain adaptation in neural ma- chine translation. In Proceedings of NATC pages 1903-1915.

---

A List of indices
Unique words
Unique sophisticated words
Unique lexical words
# Unique sophisticated lexical words
Total words
# Total sophisticated words
Total lexical words
# Total sophisticated lexical words
Lexical density
Lexical sophistication (total)
Lexical sophistication (unique)
Verb sophistication
Verb sophistication (squared numerator)
Verb sophistication (sqri denominator)
Type-token ratio (TTR)
Mean TTR of all k word segments
Corrected TTR (sqrt(2N) denominator)
Root TTR (sqrt(N) denominator)
Log TTR
Uber index
Noun variation
Adjective variation
Adverb variation
(Ajd + Adv) variation
D Mcasure
Ratio of unique verbs
Verb variation with squared numerator
Verb variation with (sqr((N)) denominator
Verb variation over all lexical words
Unique words in first k tokens
Unique words in random tokens
Unique words in random sequence of tokens
Table 3: Lexical indices
Words
Sentences
Verb pluases
Clauses
T-units
Dependent clauses
Complex T-units
Coordinate phrases
Complex nominals
Mean length of sentence
Mean length of TTunit
Mcan unit of clause
Clauses per sentence
Verb phrases per T-unit Dependent clause ratio
Clauses per T-unit
Dependent
per T-unit Complex T-unit ratio
T-units per sentence
Coordinate phrases per T-unit Complex nominals per T-unit
Coordinate phrases per clause
Complex nominals per clause
Table 4: Syntactic indices
In our work we make use indices from Lu (2010), Lu (2012), and Lee et al. (2021). Table 3 lists the lexical indices (33 indices) and table 4 lists the syntactic indices (23 indices) that we use. For their full descriptions please refer to Lu (2010) and Lu (2012). In this section, we provide descriptions of few relevant indices. Please refer to Lee al. (2021) for the comprehensive list of lingfeat (185 indices) indices.
TTR is the ratio of unique words in the text. D-measure is modification TTR that is biased by sample size. Lexical words are nouns, verbs, adjectives, and adverbs. Sophisticated words are the unconventional words. consider words beyond the 2000 most frequent words the American National Corpus as sophisticated. Uber index a transformation of TTR. SubtlexUS CDlow is a word frequency measure, specifically, "document frequency" of words starting with a lower case letter.

---

B Clustering P
Mean absolute difference
Mean absolute difference (clustered)
0
72
72
108
100
220
120
144
0.93
144
0.03
156
156
180
-0.2
180
-002
197
192
216
0.01
216
0.0
228
220
0.00
NO
160
170
100
200
210
100
110
130
150
170
180
210
220
230
(a) Pair-wise differences between
(b) Clustered differences map.
Figure 5: Figure (a) shows mean absolute difference between of each pair of linguistic indices, averaged over the whole training. Figure (b) created by re-ordering the rows and columns of (a), such that groups of indices have minimal difference between them.
p Progression
p Progression
p Progression
0.0
Enlly Dersky Teat's Fedtiaacicall Feeds Strrtu Peatsepos Rests Strature PratsePhaaatl Feats Enthy Festor Lesical POS Syntortt
Deratty Fedls
-857
Entty Crid Feats Lox Fasts
-064
Pheasal Fwets -inoo Twets
Thee Strutture Feats-Enniy
Feets
Syntactie Fests POS Tests
POS Feots
Training iterations
Training iterations
Training iterations
(a) CoLA
(b) SNLI
(c) SST-2
Balanced Accuracy Progression
Balanced Accuracy Progression
Balanced Accuracy Progression
Balanced accuracy
Balanced accuracy
Balanced accuracy
0.0
Training iterations
Training iterations
Training iterations
(d) CoLA Balanced Accuracy
(e) SNLI Balanced Accuracy
(f) SST-2 Balanced Accuracy
Figure 6: The progression of the estimated importance factors p. and balanced accuracy for groups of linguistic indices. Each pair of figures (a) and (d), (b) and (e), (c) and (f) share the legend. The solid line is the mean value of the group of lines, and the shaded area the 95% confidence interval.
We observe that several indices follow the same patterns. Therefore, we devise a method to group indices that follow the same pattern of p. We compute the mean absolute difference between of each pair of linguistic indices. Then, we cluster the groups of indices that all have minimum distance between them. Appendix B displays the effect of clustering. Note that the common trends among lines in Figures 6a, 6b, and 66 are because they are all governed by the trend of the validation loss (using both optimization and correlation approaches). Figures 6d, 6e, and 6f show the trend of balanced accuracy for the same groups. The grouped balanced accuracy has very high variance. It shows that indices with similar / do have similar values balanced accuracy. Moreover, it shows that indices with the highest p do not necessarily have the highest mean balanced accuracy. Furthermore, indices that have = 0 perform comparably to other indices, indicating that the model performs well according to such linguistic

---

indices, despite them not being correlated with loss.
Figure 5 illustrates the process of clustering together linguistic indices based on their matching P curves. We cluster the indices using hicrarchical clustering with complete linkage using the flat clustering method?s.
C Linguistic Complexity Indices
We consider linguistic complexity in terms of variability and sophistication in productive vocabulary and grammatical structures in textual content. We employ characterization of such complexity based on existing findings in language acquisition research WVolfe-Quinter et al., 1998: Lu, 2010, 2012). Specifically, we obtain 56 complexity measures from Lu (2010) and Lu (2012), including lexical and syntactic measures. Additionally, we use 185 linguistic features from the lingfeat library (Lee et al., 2021), including semantic, lexical, syntactic, discourse, and traditional features. In total, we use 241 indices. For inputs that consist of a pair of sentences, we concatenate the indices for a total of 482 indices,
C.1 Lexical Complexity
terms of lexical complexity, we consider three dimensions: lexical density, sophistication, and variation described below:
Lexical density: is quantified by the ratio of the number of open-class words to the total number of words in a given text. Texts with higher lexical density are expected to be more complex as they contain larger amounts of information-carrying words.
Lexical sophistication: measures the proportion of sophisticuued unusual or advanced- words in the input text (O'Dell et al., 2000), e.g., words not in the top K (K= 5000) frequent words the target dataset or language. Example indices include the ratio of the number of sophisticated lexical words (Linnarud, 1986; Hyltenstam, 1988), sophisticated word types Siolfe-Quintero et al., 1998) and sophisticated verb types (Harley and King, 1989) in texts, which include several variations reported in Appendix A, Table 3. We use the top K most frequent words of each dataset and consider different inflections of the same lemma as one type for computing lexical sophistication.
Lexical Variation: refers to the diversity of vocabulary in given text. Examples of such variations include type-token ratio (Templin, 1957) which is the ratio of the number of word types the number of words in the text and several different variations of this metric (Malvern et al., 2004; McKee et al., 2000; McClure, 1991) including D-measure (Malvern et al., 2004), which determines lexical variation of an input text by finding the curve that best matches the actual curve of type-token ratio against tokens of the input.
C.2 Syntactic Complexity
Syntactic complexity determines variability and sophistication with respect to grammatical structures. Simple sentences such as "the mouse ate the cheese" can be converted to their linguistically-conpi counterparts, e.g., "the mouse the cal the dog bit chased ate the cheese," which are still well-formed sentences but force readers to suspend their partial understanding of the entire sentence by encountering subordinate clauses that substantially increase the cognitive load of the sentences. We employ syntactic complexity measures that quantify the length of production units at the clausal, sentential, T-unit levels; indices that reflect the amount of subordination, e.g., T-unit complexity ratio (clauses per T-unit) dependent clause ratio (dependent clauses per clause); indices that quantify the amount of coordination. e.g., number of coordinate phrases per clause, T-unit or complex T-umit; as well those that quantify the range of surface and particular syntactic and morphological structures (e.g., frequency and variety of tensed forms or extent affixation) WWolfe-Quinters et al., 1998; Ortega, 2003). See Appendix A, Table 3.
chttps://docs http://cocsssc.......///.. scipy. wrgp:////cciyy//eeeeee/////eeeee cluster. hllstieeeretteeeeeerrrhrt linkage. httka http:///ws...pppoopoooo//css/pps
chttps://docs.eet wtwp:///cccccppp///eeeeeeeeeee/e http:///eeeeeeeeeeeeeeeeee/////e wwt.ggeee/ecciiieeeeeeee cluster hierarchy. Fcluster.html

---

D Full results
Tables 5 and show the overall performance and performance balanced by loss. Our Ling- CL approach results in 1.3 absolute points improvement in accuracy over the best-performing baseline averaged across tasks, balanced by loss.
ANLI
COLA
RTE
SNLI
SST2
AN-Pairs
Average
Ling-CL. [NegSig]
49.6 0.44
62.8 1 0.19
1.010
90.0 1 0.00
95.0 006
74.0 =
71.6 2.3
74.9 0.33
Ling-CL [Gauss]
51.5 0.39
64.5 1 0.35
79.8 1
90.4 4: 001
95.2 0m0
75.0 4: 208
71.9 0.0
75.5 :: 0.66
Ling-CI. ISigl
51.6 1.017
65.1 0.71
81.9 1:021
90.2 1 0.7
95.1 2011
74.5 1: 2:00
71.8 200
75.7 0.56
Loss-CL. [NegSig]
52.7 + 0.03
63.5 + 0.66
80.5
0.3.6
90.5 + 0.12
95.0 000
69.3
= 1.54
72.2
0.0
74.8
= 0.39
Loss-CI. [Gauss]
51.9 0.06
65.7 1: 1.66
1.08 82.3
90.3 1 0.38
94.7 2011
72.9 =
72.1
0.08
75.7 1 0.63
Loss-CL. [Sig]
51.1 4
65.0 0.57
78.9
90.4 1 0.26
94.9
74.3 188
71.7
2.15
75.2
Sampling
49.3 0.33
64.0 + 1,22
78.4
90.3 0.14
94.6
72.9 4.17
71.1
=007
74.4 0.8
Competence
50.7 0.03
63.2 0.63
77.8
90.3
95.0 a011
74.5 2.52
71.2 =010
74.7
SL-CL.
51.1 0.95
63.6
80.3
90.1 1
94.9
69.3
71.9
74.5
WR-CL
51.9
64.5 1,00
80.0
90.2 0.88
94.4
64.6
71.7 =032
73,9 094
Concat
52.2
65.2 1
81.8
90.6
94.6
72.4 2.60
71.7 2.21
75.5
SuperLoss
51.7
64.3
80.5
90.5
94.8
67.7
71.7
74.5
Data Selection
48.5
59.4 1
79.2
89.8
94.2 2017
72.4 1
71.1
73.5
No-CL
51.4
64.1
0.42
81.4
90.4
94.9
71.9
72.0
75.2
Table Overall performance each approach. Unlike Tables and 6, this table presents the standard un un-balanced accuracy.
ANLI
COLA
RTE
SNLI
SST2
AN-Pairs
GED
Average
Ling-CL. [NegSig]
23,2 4.61
27.1 + 0,66
45,4 6:23
26.3 a 0.4
25.7 #31
36.7
4.8
73.9 083
36.9 : 2:72
Ling-CL. [Gauss]
21.0 1.89
26.7 1/2
37.1 +.552
42,9 012
22.7 =291
34.7
692
73.9 a36
37.0 205
Ling-CI MnngCLLFe [Sig]
22,2 +
26.4 + 0.34
38.5 0.27
35.2 839
25.0 058
38.4
3.66
74.6 a1
37.2 1.8
Loss-CL. [NegSigl
20.9 1.89
26.7
1.0
33.8
23.8
29.8
73.4
35.2 1:73
Loss-CL [Sig]
19.5
27.6 0.51
36.3
34.6 758
24.2
31.6
72.6
35.2
Loss-CL [Gauss]
20.6
26.3
40.8 3.39
26.2
23.1
31.7
73.0
34.5
Sampling
19.0 2.12
26.2 + 0.36
32,9 0.75
42.8 :: 0:50
24.1
33.6 : 1.09
72.8
35.9 =
Competence
21.5 0.55
26.8 1 1.84
36,9 2.57
27.0 1 0.34
35.5 ::
73.4
35.5 1: 1.29
SL-CL
21.3 0.2.
26.2 + 0.00
31.6 0.33
34.6 + 8:5
26.7 000
32.3 +: 239
077
1.8
WR-CL
19.3 1.5.2
26.4 1 0.88
1.51
26.1 1 003
29.0 0:03
33.5 1:
SuperLoss
20.7 +
29.0 + 2.1
35.6
0.99
26.0 + 0:8
0m8
25.9 + 228
+ 008
33.4 +
Concat
21.7 0.17
28.1 + 0.85
37.8
+ 3.00
26.6 + 0:3
27.4 0.9
33.4 + 1.20
72.5 030
35.4 =: 0.89
Selection
19.5 0,76
25.7 + 0.28
33.8
4:2
25.9 + 065
22.6
30.6 +
025
33.2 + 1.06
No-CI.
20.1 +
28.1 + 0.85
37.8 3.00
26.6 + 033
27.4 0.29
31.8 028
72.2 0.46
34.9 =: 0.96
Table 6: Balanced accuracy by loss.

---

E
Index Importance Changes
Index
Change Stages
Magnitude
Corrected TTR
Medium to late
8.70%
AN-Pairs
Ratio of entity grid transitions
Medium to late
7.89%
Semantic Richness
Medium late
7.86%
Ratio of entity grid transitions
Medium late
3.18%
GED
Lexical sophistication
Medium late
2.26%
Ratio of Coordinating Conjunction to Adjectives
Medium to late
1.84%
Ratio of Subordinating Conjunction to Adverbs Noun-subject transitions
Early to late
1.0%%
SNLI
Early to late
0.79%
Number of topics Weebbt-based)
Early to late
0.77%
Verb sophistication
Medium to late
0.98%
ANLI
T-unit length
Medium to late
0.96%
Log tokens over log sentences
Farly to medium
0.94%
Object-noun transitions
Medium to late
0.78%
CoLA
TTR
Early to late
0.75%
# Clauses
Medium to late
0.75%
Adverb to adjective ratio
Early to late
3.30%
RTE
# TTunits
Early late
3.27%
Mean sentence length
Early late
3.15%
Noun-subject transitions
Early
1.76%
SST-2
Coordinating conjunction per sentence
Early to
1.75%
Verbs per token
Medium late
1.72%
Table 7: Top three moving linguistic indices.
Table 7 shows indices with maximum change in their between any two stages. Only the relative differences and ranking of values are important. Therefore, the table displays relative changes in the magnitude of the importance factors. The indices with a large change magnitude indicate that they either influential at early stage of training and drop in importance the later stage, or vise versa. See our analysis these indices in $3.7.


Reviews:
[{"review_id": "Xa1B7lMkCm", "reviewer": "Reviewer_7FZF", "summary": "", "questions": "Question A: There are trends in Second Language Acquisition and Psycholinguistics research to cast doubt on the robustness of some linguistic complexity measures such as age of acquisition of forms, e.g., acquisition of morphology or syntactic complexity, e.g., surprisal in complex syntax isn't shared. Would removing the number of indices used in the method to more objective ones offer the same insights or change the outcome of the experiments? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time, comments, and positive feedback on the merit of our work. In what follows, we try to address the reviewer’s questions and comments. \n\n**Q. Would the exclusion of subjective linguistic indices in experiments, in favor of the objective ones, have a significant impact on the experimental outcomes?**\n\n******A:****** While including more objective complexity signals is certainly preferable, we expect removing subjective indices to have a relatively minimal impact on our results. This is because our method is designed to automatically de-emphasize underperforming indices through the index importance factor \\rho, which measures the correlation between validation loss (as a true measure of difficulty to model) and the index. So, indices that do not correlate well with the validation loss are assigned a weight of almost 0. It would be informative (and interesting) to determine which subjective indices are assigned low weights and which low-weight indices have the potential to be subjective indices.       \n\n**C. Availability of complexity indices for extension to other languages beyond English.**\n\n******A:****** We will update the limitations Section in the final version of this submission to highlight the reviewer’s point about extension to other languages.  \n\n**C. Should the described experiments be carried out by others or if this is a more self-contained experiment?**\n\n******A:****** We hope our framework and the corresponding tools to serve as a guide for assessing linguistic complexity for various NLP tasks and uncover the learning dynamics of the corresponding NLP models during training. While we conducted our analysis on seven tasks and extracted insights on the key indices for each task, NLP researchers have the flexibility to either build on our results or apply our approach to other NLP tasks to extract relevant insights. We will clarify this point in the final version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper nicely delineates how individual complexity measures contribute to performance on each task differently, and at what stage of the model training process; this may help others identify what impacts their model performance given what task they're working on. \n- Well motivated approach of classifying data samples, then sequencing and weighting them during training - the insights how to use the data could be generalized to other methods \n- Appendix provides helpful context of what indices were used and follow up experiments \n", "reasons_to_reject": "- As the authors point out, linguistic complexity measures require expert knowledge to define and are not language agnostic. This method then may not generate over to other languages until similar measures of complexity are defined. \n- It was not all together clear if the described experiments should be carried out by others in their own work such as identifying complexity of data to better understand their model performance, or if this is a more self contained experiment \n\n", "questions_to_authors": "Question A: There are trends in Second Language Acquisition and Psycholinguistics research to cast doubt on the robustness of some linguistic complexity measures such as age of acquisition of forms, e.g., acquisition of morphology or syntactic complexity, e.g., surprisal in complex syntax isn't shared. Would removing the number of indices used in the method to more objective ones offer the same insights or change the outcome of the experiments? ", "paper_topic_and_main_contributions": "This paper proposes a method of characterizing data samples into linguistically informed complexity measures (such as verb complexity or word rarity) then weighting these data samples during model training to progressively emphasize data samples with increasingly higher complexity scores. This method is evaluated on seven NLP tasks including identifying entailment, inference making, grammar detection and others. The use of the linguistic complexity measures then offers an insight into what the NLP models use in terms of linguistic indices at different stages of training to perform the NLP tasks. The paper shows indices impact tasks differently, and we can learn from how the linguistic complexity of the data can impact model performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L282: Adjective-nous -> Adjective-noun (here, and elsewhere in the paper) \nTable1: matthew's correlation -> uppercase "}, "extra_text": {}}, {"review_id": "eoee8dpdnS", "reviewer": "Reviewer_5jzZ", "summary": "", "questions": "1. There are two variables, the number of samples and verb variation, so why can you say that \"the accuracy trend indicates the linguistic index---verb variation---can describe the difficulty of ANLI samples to the model.\"\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time, comments, and positive feedback on the merit of our work. In what follows, we try to address the reviewer’s questions and comments. \n\n**Q. There are two variables, the number of samples and verb variation, so why can you say that \"the accuracy trend indicates the linguistic index---verb variation---can describe the difficulty of ANLI samples to the model.”**\n\n******A:****** We eliminate the number of samples as a variable by categorizing samples into evenly distributed groups based on their linguistic complexity and reporting the average performance of samples in each group. Then, we find the accuracy trend across these groups (determined by verb variation). Therefore, the accuracy trend is a direct consequence of verb variation as a linguistic complexity measure.\n\n**C. What is the distinction between the correlation and optimization methods and their respective impacts on system performance?**\n\n******A:****** The main distinction between the two approaches lies in their scope: the correlation approach operates on the index level, whereas the optimization approach uses the entire set of indices. On average, across all datasets, the optimization approach outperforms correlation in our experiments. Also notably, on average, the argmax index aggregation outperforms the weighted average, and the filtered indices outperform the full list of indices. We will add the corresponding detailed results in the appendix.  \n\n**C. The experimental focus narrows primarily to word rarity. Yet, word rarity is not the top three most important linguistic indices in each dataset**\n\n******A:****** We use “word rarity” as a case study index to illustrate how a linguistic complexity index can facilitate identifying vulnerabilities in existing evaluation approaches for NLP tasks and how can such vulnerabilities be potentially addressed. In Figure 3 and line 509, we illustrate and discuss the relation between balanced accuracy and index importance, showing that it is not necessary for an index with low balanced accuracy to have a high \\rho and vice versa.\n\nWe address all presentation suggestions provided by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Several innovative methodologies are introduced in this work:\n(1) The incorporation of correlation and optimization methods to gauge the importance of diverse linguistic indices.\n(2) Utilization of either the maximum or weighted average for aggregation.\n(3) Introduction of curricula encompassing time-varying sigmoid, moving negative-sigmoid, time-varying Gaussian function, etc.\n\nThe paper demonstrates adept composition and a well-structured layout. Readers can readily discern the research inquiries pursued and the corresponding actions taken by the authors.\n\nThe authors undertake a substantial array of experiments, bolstering the paper's foundation and substantiating its claims convincingly.", "reasons_to_reject": "There are a series of methods proposed as mentioned before. However, it's worth noting that not all of these methods have been examined within the conducted experiments. For instance, the distinction between the correlation and optimization methods, as well as their respective impacts on system performance, is notably absent. This paper could potentially benefit from additional pages to accommodate a more extensive array of experiments, thereby offering a more comprehensive and in-depth understanding.\n\nThis concern extends to the treatment of linguistic indices. While multiple indices collaboratively contribute, the experimental focus narrows primarily to word rarity when evaluating different models. Yet, word rarity is not the top three most important linguistic indices in each dataset, which casts a certain degree of uncertainty upon the conclusions drawn.\n\nFurthermore, certain expressions within the paper lack clarity. For instance, the term \"competence-based\" in line 350 and \"indices filtering\" in line 353 could benefit from a more explicit explanation. While the intended meanings can be inferred to some extent, providing a clear and concise elucidation would enhance the overall understanding of the content.", "questions_to_authors": "1. There are two variables, the number of samples and verb variation, so why can you say that \"the accuracy trend indicates the linguistic index---verb variation---can describe the difficulty of ANLI samples to the model.\"\n\n", "paper_topic_and_main_contributions": "1. This paper proposes a series of approaches to enrich the process of linguistic curriculum learning. These methods encompass a spectrum of linguistic indices, involving the estimation of their significance, subsequent aggregation, and the introduction of innovative curricula.\n2. This work also encompasses a method that takes the influence of linguistic indices into account and finally results in a balanced accuracy, which is a more precise and equitable evaluation of computational models employed in NLP tasks.\n3. By delving into the importance factors of a series of linguistic indices, this paper gains deeper insights into individual datasets and NLP tasks, discerning the linguistic hurdles inherent in each undertaking.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Usage of hyphens and dashes should be reviewed, e.g. line 043 and 324.\n\n2. In line 065, consider changing \"integrates\" to \"integrate.\"\n\n3. The sentence in lines 069 and 070 is challenging to grasp. Is the intended meaning \"there are subsets of linguistic complexity indices  that have the greatest impact on the learning of distinct models for various NLP tasks\"?\n\n4. Is equation 4 referenced correctly in line 191?\n\n5. Two colons can be found in a sentence in lines 348-349."}}, {"review_id": "ao6ORw67Dj", "reviewer": "Reviewer_jVRL", "summary": "", "questions": "Languages: the paper only studies English, given that many resources are needed to estimate the linguistic indices. However, I wonder whether the results also hold for languages that are different from English.\nWould it be possible to reduce the set of linguistic features to a subset that is available to more languages and then repeat the experiment at a smaller scale with more languages? (While this is certainly beyond the scope of this paper, a brief discussion about the applicability to other languages might be interesting...)\n\nHow do your results for the different evaluation sets compare to the results of related work? ", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time, comments, and positive feedback on the merit of our work. In what follows, we try to address the reviewer’s questions and comments. \n\n**Q.  Would the results also hold for other languages beyond English? Could the experiment be repeated using a subset of linguistic features available to more languages?**\n\n******A:****** In fact, some linguistic complexity indices are language independent (e.g., the commonly-used “word rarity” measure), which facilitates extending our approach to other languages, as suggested by the reviewer. We appreciate the reviewer’s insightful input, which formed the subject of our future research.   \n\n**Q. How do your results for the different evaluation sets compare to the results of related work?**\n\n******A:****** Our baseline models show good performance when linguistic complexity is not considered for evaluation, but their effectiveness drops when test samples are categorized according to linguistic complexity. Our approach improves the best-performing baseline performance by 4.5 absolute points in average F1 score, without compromising performance when linguistic complexity is not considered for evaluation. \n\nWe follow the other comments provided by the reviewer on adding additional information on the model and the experimental settings (we will move some information from appendix to the the main part) and address all presentation suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel approach to curriculum learned based on linguistic difficulty that goes beyond previous approaches with regard to the features used, and the way of modelling the curricula.\n\nFurthermore, the paper provides an insightful analysis on the importance of different linguistic indices during different training stages. \n\nOverall, the paper presents a very interesting idea, in combination with a meaningful evaluation. ", "reasons_to_reject": "There is not much information on the model and the experimental settings, only in the appendix.\nI think it would be better to have some more information also in the main part of the paper.\n\nA general downside of the approach is the dependency on linguistic knowledge generated from human experts, which will limit the amount of languages that can be modelled with this approach.", "questions_to_authors": "Languages: the paper only studies English, given that many resources are needed to estimate the linguistic indices. However, I wonder whether the results also hold for languages that are different from English.\nWould it be possible to reduce the set of linguistic features to a subset that is available to more languages and then repeat the experiment at a smaller scale with more languages? (While this is certainly beyond the scope of this paper, a brief discussion about the applicability to other languages might be interesting...)\n\nHow do your results for the different evaluation sets compare to the results of related work? ", "paper_topic_and_main_contributions": "The paper investigates a new approach to curriculum learning, in which the importance of several linguistic indices are estimated and aggregated to provide estimates for the overall linguistic difficulty, in relation to the model's linguistic competence.\nThe authors find that their approach leads to improved performance compared to standard training (without curriculum) over several standard evaluation sets that require a high level of linguistic knowledge.\n\nThe paper also present an analysis of the key linguistic indices for grammar tasks that provide interesting insights into linguistic challenges and the factors relevant to model the respective tasks, finding that different factors take indeed different prominence for different tasks.\n\nThe main contributions are the incorporation of linguistic complexity information and the development of curricula based on linguistic\ncomplexity, as well as the identification of the linguistic complexity indices that contribute most to different NLP tasks.\n\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I find section 3.4 remains a bit vague and could provide more information."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a multi-view curriculum learning framework, which include characterizing data samples using linguistic complexity measures and weighting these samples during model training based on their complexity scores. The paper also explores the impact of linguistic complexity measures on various NLP tasks and provides insights into how different linguistic indices affect model performance at different training stages. Reviewers have found the proposed approach well motivated, interesting and innovative; therefore, I recommend acceptance of this paper. Please also address the comments by reviewers on limitations.
Decision: Accept-Main