Complexity-Guided Curriculum Learning for Text Graphs
Authors: Nidhi Vakil
Hadi Amiri
Venue: EMNLP 2023 Findings (2023)
Keywords: Curriculum Learning
Graph Neural Network
Text Graph

Abstract: Curriculum learning provides a systematic approach to training. It refines training progressively, tailors training to task requirements, and improves generalization through exposure to diverse examples. We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is a novel data scheduler, which employs ``spaced repetition'' and complexity formalisms to guide the training process. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures. The proposed model gains more and uses less data; consistently prefers text over graph complexity indices throughout training, while the best curricula derived from text and graph complexity indices are equally effective; and it learns transferable curricula across GNN models and datasets. In addition, we find that both node-level (local) and graph-level (global) graph complexity indices, as well as shallow and traditional text complexity indices play a crucial role in effective curriculum learning.

Complexity-Guided Curriculum Learning for Text Graphs
Nidhi Vakil
Hadi Amiri
Department of Computer Science University of Massachusetts Lowell nvakil@cs. uml edu
Department of Computer Science
University of Massachusetts Lowell hadi@cs uml. edu
Abstract
In Curriculum learning (CL) (Bengio al., 2009) data samples are scheduled in a meaningful difficulty order, typically from easy to hard, for iter- ative training. CL approaches have been successful in various areas (Graves et al., 2017; Jiang et al.. 2018; 8; Castells et al., 2020), including NLP (Settles and Meeder, 2016; Amiri et al.. 2017: Zhang et al.. 2019; Lalor and Yu, 2020; Xu al., 2020; Kreutzer et al., 2021: Agrawal and Carpuat, 2022; Maharana and Bansal, 2022). Existing approaches use data properties such as sentence length, word rarity or syntactic features (Platanios et al., 2019; Liu et al.. 2021); and model properties such as training loss and its variations (Graves et al., 2017; Zhou et al., 2020) to order data samples for training. However, other types of complexity formalisms such as those developed for graph data largely underexplored. Recently, Wang al. (2021) proposed estimate graph difficulty based on intra- and inter-class dis- tributions of embeddings, realized through neural density estimators. Wei et al. (2023) employed a selective training strategy that targets nodes with diverse label distributions among their neighbors as difficult to learn nodes. Vakil and Amiri (2022) used loss trajectories to estimate the emerging diffi- cuity of subgraphs and weighted sample losses for data scheduling, We encourage readers to see (Li et al., 2023; Yang et al., 2023) for recent surveys on graph CL approaches.
Curriculum learning provides a systematic ap- proach to training. It refines training progres- sively, tailors training to task requirements, and improves generalization through exposure to di- verse examples. We present curriculum learn- ing approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is novel data scheduler, which employs "spaced repetition" and com- plexity formalisms to guide the training pro- cess. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures. The proposed model gains more and uses less data; consistently prefers text graph complex- ity indices throughout training, while the best curricula derived from text and graph complex- ity indices are equally effective; and it learns transferable curricula across GNN models and datasets. addition, we find that both node- level (local) and graph-level (global) graph complexity indices, as well as shallow and tra- ditional text complexity indices play crucial role in effective curriculum learning.
1 Introduction
Message passing (Gilmer et al., 2017) is a widely used framework for developing graph neural net- works (GNNs), where node representations are iter- atively updated by aggregating the representations of neighbors (a subgraph) and applying neural net- work layers to perform non-linear transformation the aggregated representations. We hypothesize that topological complexity of subgraphs or linguis- tic complexity of text data can affect the efficacy of message passing techniques in text graph data, and propose to employ such complexity formalisms in novel curriculum learning framework for effec- tive training of GNNs. Examples of graph and text complexity formalisms are node centrality and con- nectivity (Kriege et al., 2020); and word rarity and type token ratio (Lee et al., 2021a) a)) respectively.
We propose that existing knowledge about text and graph complexity can inform better curriculum development for text graph data. For example, a training node pair that shares many common neigh- bors is expected to be easier for link prediction than local bridge that lacks common neighbors. Mo- tivated by this rationale, we present a complexity- guided CL approach for text graphs (TGCL), which employs multiview complexity formalisms to space training samples over time for iterative training. It advances existing research as follows:
AAn edge that is not part of a triangle in the graph.

---

Gane (u): ARMCS
Dompliions ore Fer drondrer
Silay Saling voaling
rspert saazercing that
Subgrapha
GNN Trainer
undaiispraan irametton
irpartan for uandaas
muren satwert comples Hell cells, and atheracation cemper cade looa
Detenttnn
Spaced Repetition
degree assertativity
confficient
degree
eigenvector
Enoch
centrality
Disease (V) PMAH
Input Graph
i an aikourmal cort hamer suacept by with soroirerii: per the gann regline to dewnlop rracronokion toperiiic...
local bridges
$10'0
20
$20
connectivity
average degree
Index Rankings
c(e)
degree mixing matrix
closeness
Subgraph :
centrality
node connectivity
treewidth min
Subgraph
degree
Figure I: The architecture of the proposed model, TGCL. It takes subgraphs and text(s) of their target node(s) as input. The radar chart shows graph complexity indices which quantify the difficulty of each subgraphs from different perspectives (text complexity indices are not shown for simplicity). Subgraphs are ranked according to each complexity index and these rankings are provided to TGCL scheduler to space samples over time for training.
new curriculum learning framework that em- ploys graph and text complexity formalisms for training GNNs text graph data, and insights into the learning dynamics of GNNs, i.e., which complexity formalisms are learned by GNNs during training.
based on a flexible "competence" function ($2.2), and employing different data schedulers that der training samples for learning with respect model behavior ($2.3). By integrating these com- ponents, TGCL, establishes curricula that are both data-driven and model -eeeedddeeenent.. what follows, we present approaches for addressing these tasks.
We conduct extensive experiments on real-world datasets and across GNN models, focusing on link prediction and node classification tasks text graphs. The proposed model gains 5.1 absolute points improvement in average score over the state- of-the-art model, across datasets and GNN models, while using 39.2% less data for node classification than high-performing baselines. The results show that both node-level (local) and graph-level (global) complexity indices play a crucial role in training. More interestingly, although the best curricula de- rived from text and graph complexity indices are equally effective, the model consistently prefers text over graph complexity indices throughout all stages of training. Finally, the curricula learned by the model are transferable across GNN models and 2
2.1 Complexity Formalisms
Graph complexity (Kashima et al., 2003; Vish- wanathan et al., 2010; Kriege et al., 2020) indices are derived from informative metrics from graph theory, such node degree, centrality, neighbor- hood and graph connectivity, motif and graphlet features, and other structural features from random walk kernels or shortest-path kernlls (Borgwardt and Kriegel, 2005). We use 26 graph complexity indices to compute the complexity score of data instances in graph datasets, see Table 1, and details in Appendix A. Since data instances for GNNs are subgraphs, we compute complexity indices for each input subgraph. For tasks involving more than one subgraph (e.g.. link prediction), aggre- gate complexity scores of the subgraphs through an order-invariant operation such as sum().
2 Method
A curriculum learning approach should estimate the complexity of imput data, determine the pace of introducing samples based on difficulty, and sched- ule data samples for training. As Figure I shows, TGCL tackles these tasks by quantifying sample difficulty through complexity formalisms ($2.1), gradually introducing training samples to GNNs
Linguistic Complexity Lee et al. (2021a) im- plemented various linguistics complexity features for readability assessment. We use 14 traditional and shallow linguistics indices such as Smog in- dex, Coleman Liau Readability Score, and sentence length-related indices as text complexity indices in our study. See Appendix A.2 for details. We nor- malize complexity scores for each text and graph index using the L.2 norm.
*Code and data are available at https://clu.cs.um edu/tools. tools. html

---

Degree based
Computing based
degree
ramsey R2 x average clustering resource allocation index Connectivity
Competence
treewidth min degree
degree mixing matrix
0.4
average neighhor degree
d= 0.1
0.2
average degree connectivity
subgraph connectivity
dogree assortativity cocf.
local node connectivity
0.0
Centrality
Basic properties large clique size * common neighbors
250
1000
katz centrality +
Figure 2: Three competence functions, each imposing a different type of curriculum on GNNs.
degree centrality
closeness centrality
number of edges number of nodes density
eigenvector centrality *
group degree centrality
larger/smaller fraction of training data than what the linear competence allows different stages of training. This consideration results in a more general form of competence function:
Flow property
local bridges
min weighted dominating set min weighted vertex cover min edge dominating set min maximal matching
ctt) = min (1, -1 (1-
(1 c0)
t))+). (4)
Table I: Graph complexity indices. These indices manually divided into six categories to case the presenta- tion and analysis of our results. Indices that are used in our experiments are labeled by the symbol. Appendix A provides details on the selection process.
where a specifies the rate of change for compe- tence during training. As Figure shows, a larger a quickly increases competence, allowing the model to use more data after a short initial training with easier samples. We expect such curricula to be more suitable for datasets with lower prevalence of easier samples than harder ones, so that the learner do not spend excessive time the small set of easy samples earlier stages of training. On the other hand, a smaller a results in curriculum that allows more time for learning from easier samples. We expect such curricula to be more suitable for datasets with greater prevalence of easier samples, as it provides sufficient time for the learner to as- similate the information content in easier samples before gradually moving to harder ones.
2.2 Competence for Gradual Inclusion
Following the core principle of curriculum learn- ing (Bengio et al., 2009), we propose to gradually increase the contribution of harder samples as train- ing progresses. Specifically, we derive the compe- tence function c(t) that determines the top fraction of training samples that the model is allowed to use for training time step f. We derive general form of c(t) by assuming that the rate of competence- the rate by which new samples are added to the current training data-is equally distributed across the remaining training time:
do(t) 1 1-((() 1-t
2.3 Spaced Repetition for Ordering Samples
dt
Spaced repetition a learning technique that in- volves reviewing and revisiting information at inter- vals over time. We propose to use spaced repetition to schedule training data for learning. Specifically, develop schedulers that determine (data from) which complexity indices should be used for train- ing at each time. For this purpose, we learn delay parameter for each index, which signifies the num- ber of epochs by which the usage of data from the index should be delayed before re-introducing the index into the training process. The schedulers dy- namically (during training) increase or decrease the delay for indices based on the difficulty of learning their top c(t) samples by the GNN model.
where t E 1] is the normalized value of the current training time step, with = 1 indicating the time after which the learner is fully competent. Solving this differential equation, obtain:
1 I 1 de(l) 1
which results in c(t) - 1- -(x))=]e - t) for some constant b. Assuming the initial competence c(t- 0) is co and final competence c(t = 1) is 1, we obtain the following linear competence function:
c(t) min 1, (1 co) (1) 1)).
As Algorithm shows, the model first computes complexity indices for training and validation sam- ples, and sorts the samples for each index according to a pre-defined order. All indices are initialized
We modify the above function by allowing flexibility competence so that models can use

---

Algorithm 1: TGCL Scheduler
Algorithm 2: Compute Optimized Delay
input
input
L: Complexity indices
v: Index
M: GNN Model
d. Loss vector
D: Training data size ı
a,: Probability score vector
V: Validation data of size m
1): Recall threshold
S: Index sort order(s)
2: Current model performance on val. data
output: : Trained model M
output :ō;: Delay for index
L. <- Complexity of training data based index
7; <- calculate optimal using (12)
2 I. - Complexity validation data based on index i LP sontL." P. S), Vi
a E; <- calculate optimal delay using (5)
6; +- Σ, tu using (6)
I. + sort(l., V ..)iI S). vi
[e,]
return 6;
6, 1, Vi E L #initialize delay for indices for t + 0 to E do
porrent_batchl - (i: & <=
t; (Amiri et al., 2017) for the top c(t) samples of each index i in the current batch. We learn the delays such that model performance the samples is maintained improved after the delay:
delayed_ batch (t: & > 1] e(l) <- competence from Eq (4) for i E current batch do
10
Train M with top c(t) samples L end for delayed butch do
13
(1( xtu 7i
2
c end
E arg max maxt,y
(5)
15
16
for i E current batch
where Q is the top c(t) fraction of samples of index at iteration L, d, is the instantaneous losses of these samples, is the delay for these samples (a vector to be learned), y is the performance of the current model on validation data, and 17 E (0, 1) is the expected model performance on samples in O after the delay. f() is a non-increasing function of X; = dixt, and is responsible for assigning greater delays (tz) to easier samples (smaller di) in stronger networks (greater 7) (Amiri et al., 2017). Intuitively, (5) estimates the maximum delay &; for the samples in Q such that, with a probability of 1), the performance of the model is maintained improved for these samples at iteration e E. The hyperparameter T controls the rate of decay for f. which is optimized using the achieved model performance in hindsight, see $2.3.2. The delay for each index is obtained by averaging the optimal delays of its top c(t) samples (0)) as follows:
+ top m c(f) samples in L.
17
18
d: f loss(s)
19
a, + prediction_scare(s)
20
+ iilidation_pefformmnn
21
6: < compute_delay(i, v, d, a,
22
end
23
with delay of one, 6, - vi At every iteration, the model divides the indices into two batches: the current batch, those with estimated delay 8, < 1 iteration; and the delayed batch, those with & > 1. Indices in the current batch are those that the sched- uler is less confident about their learning by the GNN and those in the delayed batch are indices that are better learned by the GNN. At each iter- ation, the scheduler prioritizes indices in the cur- rent batch by training the GNN using their top c(t)
?
fraction of samples, see (4), while samples of the delayed indices not used for training. After each iteration, all delay values are updated.
2.3.1 Delay Estimation
5,
by. jCQ
We develop schedulers that assign greater delays to indices that contain relatively easier samples within their top c(t) samples. Our intuition is that these samples are already learned to a satisfactory de- gree, thus requires less frequent exposure during training. Delaying such indices can result in better allocation of training resources by preventing un- necessary repetition of already learned samples and potentially directing training toward areas where model's generalization can be further improved. For this purpose, we first develop a scheduler f() to estimate the optimized sample-level delay
In addition, indices in the delayed batch are not used for training at current iteration and thus their delays are reduced by one. Line 14, Algorithm 1. We note that, given the improvement gain by the GNN model training progresses, the above ap- proach conservative and provides a lower bound of the optimal delays for indices.
2.3.2 Scheduling Functions
A good scheduler should assign greater delays to easier samples in stronger models. Therefore, we

---

can use any non-increasing function of x; We consider the following functions:
d:xt:
ence articles. Each paper is provided with an em- bedding vector of size 128, obtained from average word embeddings of the title and abstract of the paper and categorized into one of the 40 categories.
Stap (xi, Ta) = exp( X,Ti)
2
Ssec (xi,
expl Tix² + cop (T))
Cora (McCallum et al., 2000) is relatively small citation network, in which papers are cat- egorized into one of the seven subject categories and is provided with feature word vector obtained from the content of the paper.
2 cos (Tx
Xi
foos (xs; Ti)
=
0
otherwise
x < 1 -
Squa (xi; Ti)
(10)
0
otherwisc
Citeseer (Kipf and Welling, 2017b) a citation network of scientific articles, in which nodes are classified six classes. We use the same data split as reported in (Zhang et al., 2022).
1 TiX
X
<
fiin (xs; Ti)
(11)
0
otherwise
For each index i, we estimate the optimal value of the hyperparameter using information from previous iteration. Specifically, given the sample loss and validation performance from the previous iteration for the top c(t - 1) samples of index i (o and the current accuracy of the GNN model on these samples (p:), we estimate T₃ as:
Gene, Disease, Phenotype Relation (GDPR) (Vakil and Amiri, 2022) large scale dataset for predicting causal relations between genes and diseases from their text descriptions. Each node the graph a gene or disease and an edge repre- sents (causal) relation between genes and diseases.
A
arg min, (xi, 73) p.)
(12)
Gene Phenotype Relation (PGR) (Sousa et al., 2019) is a dataset for extracting relations between gene and phenotypes (symptoms) from short sen- tences. Fach node in the graph is a gene, pheno- type or disease and an edge represents a relation between its end points. Since the original dataset does not contain validation split, we generate a validation set from training data through random sampling, while leaving the test data intact. The data splits will be released with our method.
V, C O Pij >= 17.
See the steps for delay estimation in Algorithm 2.
2.4
Base GNN Models
Our approach can be used to train any GNN model. We consider four models for experiments: Graph- SAGE (Hamilton al., 2017), graph convolutional network (GCN) (Kipf and Welling, 2017a), graph attention networks (GAT) (Veličković et al., 2018), and graph text neural network (GTNN) (Vakil and Amiri, 2022). GraphSAGE is a commonly-use model that learns node embeddings by aggregating the representation of neighboring nodes through order-invariant operation. GCN is efficient and scalable approach based on convolution neural networks which directly operates on graphs. GAT extends GCN by employing self-attention layers to identify informative neighbors while aggregating their information. GTNN extends GraphSAGE for NLP tasks by directly using node representations at the prediction layer as auxiliary information, al- leviating information loss in the iterative process of learning node embeddings in GNNs.
3.2 Baselines
In addition to the GNN models described in $2.4, we use the following curriculum learning baselines for evaluation:
Competence CL (CCL) (Platanios et al., 2019) competence-based CL approach that gradually introduces the data in increasing order of difficulty to the model according to a competence function. The model only works with one difficulty score, which we provide by summing the complexity in- dices for each training sample.
SuperLoss (SL) (Castells et al., 2020) is a CL framework that determines the difficulty of samples by comparing them against the loss value of their corresponding batches. It assigns greater weights to easy samples and gradually introduces harder examples as training progresses.
3 Experiments
3.1 Datasets
Ogbn-arxiv from Open Graph Benchmark (Hu et al., 2020) is a citation network of computer sci-

---

Node Classification
Link Prediction
GNN Model
Curriculum
Ogbn-Arxáv
Cora Acc
Citeseer Acc
GDPR
PGR
Acc
F1
Average
No-CL
71.620.1
90.4=1.0
76.8+0.1
84.9+0.3
93.9=2.0
83.5+0.7
CurGraph
68.620.1
86.920.8
59.9+1.1
81.5+1.4
73.9
74.2+0.7
GTNN
SL
16.1
91.0=0.3
77.9+0.8
85.020.3
94.920.6
85.0+0.4
Trend-SL.
71.7=0.3
90.0=0.5
1
84.9+0.0
95.3+10.0
84.0+0.2
CCL
76.420.2
07.640.3
76.640.7
83.640.0
92.5+0.7
85,340.4
CI.Node
69.7=0.5
75.0=0.1
55.745.9
66.8+2.2
TGCL
76.3+0.0
96.1=0.8
76.740.8
84.940.3
93.1=0.7
85.4+0.5
No-CL
71.4+0.1
90.0=0.5
75.6+0.5
25.4+0.1
91.6*1.0
70.8+0.4
GraphSAGE
CurGraph
69.0=0.2
86.7=1.0
62.8+1.0
65.620.5
71.3=0.0
71.
SL
71.8=0.2
89.7=0.5
75.5+1.3
25.2+0.1
91.2=006
70.7+0.5
Trend-SL
71.5=0.4
88.7+1.3
74.641.
25.2+0.3
91.2200.
70.3+0.8
CCL
75.9=0.0
96.1=0.3
74.4+0.2
54.8#0.8
88.7=1.6
78.0+0.6
CL.Node
60.2+2.4
68.9=2.2
61.6+4.7
63.5+3.1
TGCL
75.8=0.3
95.8=0.3
75.4+0.5
56.820.1
92.420.1
79.2+0.2
No-CL
71.8=0.1
90.8=1.0
76.3+0.6
25.2+1.7
85.9=1.1
70.0+0.9
CurGraph
70.3+0.3
88.2+0.0
61.34117
70.0+1.4
67.2+0.7
71.420.8
SL.
71.7=0..
89.9=0.8
75.5+1.3
24.5+1.1
84.9:0.4
69.3+0.8
GCN
Trend-SL
71.8=0.3
90.0=0.5
76.341.5
24.940.7
84.5+1.0
69.540.8
CCL
74.420.2
91.9=1.0
72.5+0.6
52.320.5
84.6#1.5
8
CLNode
60.7+2.0
75.5=1.4
65.540.7
67.2+1.4
TGCL
74.620.2
92.3=0.0
73.6+0.3
53.2+0.4
85.2+0.1
75.8+0.2
No-CL
71.0%0.1
89.120.3
76.5+0.7
18.8+0.3
85.0=1.9
18.5
CurGraph
69.8=0.2
85.620.0
61.3+1.7
92.9+0.2
57.5=1.6
73.4+0.7
SI.
71.7=0.4
88.2+0.0
75.4+0.6
18.8+0.3
84.8+1.1
67.8+0.5
GAT
Trend-SL
71.5=0.1
89,9=0.3
76.4+1.1
18.8+0.3
85.1
68.3+0.5
CCL
74.7=0.3
92.6+0.0
73.3+0..4
34.0#0.4
84.3+009
71,8+0.4
CLNode
64.7=0.3
67.814.3
63.940.0
65.5
TGCL
74.8=0.0
91.3=0.8
73.8+0.1
34.520.6
85.9#1.1
72.1+0.5
Table Fl and Accuracy performance of different curriculum learning models on node classification (Ogbn-Arxiv, Cora, and Citeseer datasets), and link prediction (GDPR and PGR datasets) using GTNN, GraphSAGE. GCN and GAT as base GNN models across three different seeds. All GNN models are initialized with corresponding text embeddings for nodes of each dataset. For the proposed model, TGCL, the best performing kernels for Ogbn-Arxiv, Cora Citeseer, GDPR and PGR are lap, qua, sec, and qua respectively; we report the top-performing kernel function with average performance and standard deviation over two runs the Table, bold indicates best performing model, see $3.3 for details.
CurGraph (Wang et al., 2021) a CL approach for GNNs that computes difficulty scores based on the intra- and inter-class distributions of embed- dings, realized through neural density estimator, and develops smooth-step function to gradually harder samples in training. We implemented this approach by closely following the paper.
3.3 Settings
In the competence function (4), we set the value of from [0.2, In (5) and (12), we set from [0.6, 1) with step size of 0.1 for link prediction and from [0.7, 1) with the step size of 0.5 for node clas- sification. The best kernel for the datasets in the reported results are cos, qua, lap, qua, sec, and the best value of is 0.7, 0.9, 0.8, 0.75, 0.9 for GDPR, PGR, Ogbn-Arxiv, Cora and Citeseer re- spectively. We consider a maximum number of 100 and 500 training iterations for link prediction and node classification respectively. In addition, we order samples for each index in four different ways, ascending/descendingg (low/high to high/low com- plexity), and medium aasending/desceedding (where instances are ordered based on their absolute dis- tance to the standard Normal distribution mean the complexity scores ascending/descending or-
Trend SL (Vakil and Amiri, 2022) extends Su- perLoss by discriminating easy and hard samples based on their recent loss trajectories. Similar to SL, Trend-SL can be used with any loss function.
CLNode (Wei et al., 2023) employs selective training strategy that estimates sample difficulty based on the diversity in the label of neighboring nodes and identifies mislabeled difficult nodes by analyzing their node features. CLNode implements an easy to hard transition curriculum.

---

der). We evaluate models based on the Fl score for link prediction and accuracy for node prediction task using (Buitinck et al., 2013). Finally, we run all experiments on a single A100 40GB GPU.
Phase-1
Phase-2 Phase-3
Frequeet
150
Frequency
Phaxe-1
Phase-2
Phase-3
100-
2000
1000-
bast
Aspeques
Fusnduuo
fuunduro
suthep
3.4 Main Results
(a) POR
(b) Ogbu-Arxiv
Table 2 shows the performance of our approach for link prediction and node classification tasks using four GNN models. The performance of all models on link prediction and node classification signifi- cantly decreases when switch from GTNN to any other GNN as encoder, which indicates ad- ditional text features as auxiliary information in GTNN are useful for effective learning.Ses Ap- pendix A.6 for the performance of kernel functions. For both link prediction and node classification tasks, most curricula improve the performance com- pared to standard training (No-CL. in Table 2). On average, TGCL performs better than other CL baselines with most GNN models. CurGraph show lower performance than other curricula and No- CL in almost all settings, except when used with GraphSAGE, GCN and GAT on GDPR. The lower overall performance of CurGraph and CLNode may be due to the static order of samples or monotonic curricula imposed by the model, or our implemen- tation of CurGraph. The large performance gains of TGCL node classification and link prediction datasets against No-CL indicates the importance of the information from difficulty indices, their systematic selection, timely delays, and revisiting indices progressively during training, which help the model generalize better.
Figure 3: The number of times each index appeared in the current batch at different phases of training for (a): PGR (link prediction) and (b): Ogbn-Arxiv (node classification) tasks. Phases 1-3 indicate early, mid, and late training respectively quantified by the first, second, and last 33% of training iterations. Degree (local) and centrality (global) based indices are frequently used for link prediction, while degree and basic (local) based indices are frequently used for node classification.
Figures 3a and 3b show the results. The fre- quency of use for indices follows a decreasing trend. This is expected as in the initial phase the model tends to have lower accuracy in its predictions, re- sulting in higher loss values. Consequently, the scheduler assigns smaller delays to most indices, ensuring that they appear in the current batch at the early stage of training. However, the model improves its prediction accuracy, the scheduler be- comes more flexible with delays during the latter stages of training. This increased flexibility allows the scheduler to adjust the delay values dynamically and adapt to the learning progress of the model. In addition, the results show that the model focuses on both local and global indices (degree and centrality respectively) for link prediction, while it prioritizes local indices (degree and basic) over global indices for node classification throughout the training. See Appendix A.4 for detailed results.
4 Curricula Introspection
We conduct several analysis on TGCL scheduler to study its behavior and shed light on the reasons for its improved performance. Due space limitations, we conduct these experiments on one representative dataset from each task, PGR for link prediction and Ogbn-Arxiv for node classification.
4.2 TGCL Gains More and Uses Less Data
In standard training, model uses all its n training examples per epoch, resulting in a total number of E' updates. TGCL uses on an average 39.2% less training data for node classification for GTNN model, by strategically delaying indices throughout the training. Figure shows the average number of examples used by different CL models for training across training iteration, computed across all node classification datasets. Our model TGCL uses less data as the training progresses, the standard training (No-CL) and some other curricula such as SL and SL-Trend uses all training data at each iteration. CCL, apart from TGCL, uses less data compared to other CL models. An intriguing observation is that despite both CCL and TGCL are allowed to use
4.1 Learning Dynamics
For these experiments, we divide training iterations into three phases: Phase-1 (early training, the first 33% of iterations), Phase-2 (mid training, the sec- ond 33% of iterations), and Phase-3 (late training, the last 33% of iterations). We report the number of times graph complexity indices appeared in the cur- rent batch each phase. We group indices based on their types and definitions as reported in Table I to ease readability.

---

more data as training progresses, TGCL uses less data by strategically delaying indices and avoiding unnecessary repetition of samples that have already been learned, resulting in better training resources and reduced redundancy.
unchanged compared using only graph indices. However, we observe that the model consistently prefers linguistic indices (Coleman Liau Readabil- ity and sentence length related indices), followed by the degree based indices, throughout all phases of the training. Figure 6 shows the contribution of linguistic and graph indices in different phases of training. While linguistic indices do not lead to accuracy beyond 76.4, they are consistently priori- tized by TGCL over graph indices. Incorporating additional linguistic indices have the potential to further enhance performance.
15400
30000
15000
20000
CC2
Spach
Figure 4: Average number of samples used by different CL frameworks across all node classification datasets. The number remains constant for No-CL, increases for CCL as the training progresses. TGCL uses less data than others by spacing samples over time.
Phase-1
Phase-2
Phase-3
Frequency
5000
4000
3000
2000
1000
Sha
basic
centrality itality
degree
sha
Trai
sha
200
Figure 6: The number of times each index appeared in the current batch at different phases of training for Ogbn-Arxiv when linguistic indices are included. Shaf and TraF are shallow and traditional formulas features described in Appendix A.2.
% teatres
1.00
300
Figure 5: Percentage of indices used for training every epoch and the average validation accuracy on samples used for training at each iteration on Ogbn-Arxiv. In initial epochs, most indices are frequently used for train- ing until the performance reaches the recall threshold " = -80, after which the scheduler starts delaying some of the indices. Model prevents repetition of already learned samples, while directing the training towards areas where generalization can further improved.
4.4 TGCL Learns Transferable Curricula
We study the transfcrability of curricula learned by TGCL across datasets and models. For these experiments, we track the curriculum (competence values and indices used for training at every itera- tion) of a source dataset and apply the curriculum to a target dataset using GTNN as the base model. Table 3 shows learned curricula are largely transfer- able across dataset, considering the performance of No-CL as the reference. We note that the slight re- duction in performance across datasets (compared to the source curricula), can be negligible consid- ering the significant efficiency that can be gained through the adoption of free curricula (39.2% less training data, see $4.2). Table 4 shows the curric- ula learned by TGCL can be transferred across GNN models, and in some cases improves the per- formance, e.g., GAT to GCN, Further analysis on these results is the subject of our future works.
In spaced repetition, spacing effect is observed when the difference between subsequent reviews of learning materials increases as learning progresses. As a result, the percent of the indices used by model for training should decrease as the model gradually becomes stronger. Figure 5 illustrates this behavior exhibited by TGCL. This results demonstrates that the delays assigned by the scheduler effectively space out the data samples over time, leading to an effective training process.
4.3 TGCL Prioritizes Linguistics Features
For this analysis, we calculate linguistic indices (detailed in $2.1 and Appendix A.2) from the paper titles in Ogbn-Arxiv. We augment the graph indices with linguistics indices and retrain our top perform- ing model, GTNN, on Ogbn-Arxiv to assess the importance of linguistics indices in the training pro- cess. The resulting accuracy is 76.4, which remains
5 Related Work
In Curriculum lcarning (CL) (Bengio et al., 2009) data samples are scheduled in a meaningful diffi- culty order, typically from easy hard, for iterative training. In graph machine learning, Wang et al. (2021) introduced CurGraph, a curriculum learning method designed for sub-gra bub-graph classification. This

---

target
learning dynamics.
Oghn-Arxiv
Cora
Citeseer
source
Current curriculum learning methodologies in NLP rely on data properties, e.g.. sentence length, word rarity, or syntactic features (Platanios et al., 2019; Liu et al., 2021), or annotation disagree- ment (Elgaar and Amiri, 2023); as well as model properties such as training loss and its variations (Graves et al., 2017; Amiri et al., 2017) to sequence data samples for training. Elgaar and Amiri (2023) developed curriculum discovery framework based on prior knowledge of sample difficulty, utilized an- notation entropy and loss values. They concluded that curricula based on easy-to- so-hard or hard-to-esdd -to-easy transition are often at the risk of under perforner..
Ogbn-Arxiv
76.6
94.5
75.3
Cora
76.1
96.7
76.8
Citeseer
71.9
93.7
77.8
No-CL (GTNN)
71.1
91.5
75.3
Table 3: Performance of curricula transfer across node classification datasets using GTNN. Underline indicates the curricula learned on source dataset.
target
GTNN
GraphSAGE
GCN
GAT
source
GTNN
76.4
75.9
74.7
74.3
GraphSAGE
76.4
75.9
74.7
74.6
GCN
75.8
75.2
74.4
74.0
GAT
76.2
76.0
75.2
74.8
effective curricula are often non-monotonic, and
No-CL
71.1
71.5
71.8
71.8
curricula learned from smaller datasets perform well on larger datasets.
Table Performance of curricula transfer across GNN models on Ogbn-Arxiv. Underline indicates the curric- ula learned on source GNN model.
Other instances of curriculum learning for tex- tual data have primarily centered on machine trans- lation and language comprehension. For instance. Agrawal and Carpuat (2022) introduced frame- work for training non-autoorerressive sequence-to- sequence models for text editing. Additionally, Ma- harana and Bansal (2022) designed various curricu- lum learning approaches where the teacher model assesses the difficulty each training example by considering factors such as question-answering probability, variability, and out-of-distribution mea- sures. Other notable work in various domain in- cludes (Graves et al., 2017; Jiang al., 2018; Castells et al., 2020; Settles and Meeder, 2016; Amiri et al., 2017: Zhang al., 2019; Lalor and Yu, 2020; Xu et al., 2020; Kreutzer et al., 2021) which have contributed to its broader adoption.
model assesses the difficulty of samples by analyz- ing both intra-class and inter-class distributions of sub-graph embeddings. It then organizes the train- ing instances, by first exposing easier sub-graphs and gradually introducing more challenging ones. Wei et al. (2023) adopted a selective training strat- egy, targeting nodes with diverse label distributions among their neighbors as particularly challenging to learn. Liu et al. (2023) proposed HSAN, which clusters graphs using curriculum and contrastive learning and measures the difficulty of training pairs using attribute and structural similarity and use weights to select hard negative samples. Wang al. (2023) proposed approach called CHEST improve recommendation using heterogeneous graph data and combine local and global context information to guide curriculum development.
Conclusion and Future Work
In contrast to static curriculum approaches, Sax- ena al. (2019) proposed dynamic curriculum ap- proach that automatically assigns confidence scores to samples based on their estimated difficulty. How- ever this model requires additional trainable pa- rameters. To address this limitation, Castells et al. (2020) introduced the SuperL framework to cal- culate optimal confidence scores for each instance using a closed-form solution. In (Vakil and Amiri, 2022), we extended SuperLoss to incorporate trend information the sample level. We utilized loss trajectories estimate the emerging difficulty of subgraphs and employed weighted sample losses for data scheduling in order to create effective cur- ricula for training GNNs and understanding their
We introduce novel curriculum learning approach for text graph data and graph neural networks, in- spired by spaced repetition. By leveraging text and graph complexity formalisms, our approach deter- mines the optimized timing and order of training samples. The model establishes curricula that are both data-driven and model- or learner-dependent. Experimental results demonstrate significant per- formance improvements in node classification and link prediction tasks when compared to strong base- line methods. Furthermore, our approach offers po- tential for further enhancements by incorporating additional complexity indices, exploring different scheduling functions and model transferability. and extending its applicability to other domains.

---

Broader Impacts
ternational conference on data mining (ICDM '05), pages 8- pp. LEEE.
The advancements in curriculum learning signal a promising direction for the optimization of training processes within NLP and graph data. Based on the principles of "spaced repetition" and text and graph complexity measures, the proposed work enhances the efficiency of training and improves model gen- eralization capabilities. This particularly crucial for applications reliant on graph representations of text, such as social network analysis, recommen- dation systems, and semantic web. Furthermore, the method's 's ability to derive transferable curric- ula across different models and datasets suggests a more applicable strategy, potentially enabling seamless integration and deployment across varied NLP applications and domains.
Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer. Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Ar- naud Joly, Brian Holt, and Gaël Varoquaux. 2013. API design for machine learning software: experi- ences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning.
Thibault Castells, Philippe Weinzaepfel, and Jerome Revaud. 2020. Superloss: A generic loss for robust curriculum learning, Advances in Neural Information Processing Systems.
Mohamed Elgaar and Hadi Amiri. 2023. HuCurl: Human-induced curriculum discovery. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1862-1877, Toronto, Canada. Association for Computational Linguistics.
Limitation
The proposed approach relies on the availability of appropriate complexity formalisms. If the selected indices do not capture the desired complexity, the curricula may not be optimally designed. The ap- proach primarily focuses on text graph data and graph neural networks, and the results may not directly apply to other types of data or architec- tures. The estimation of optimized time and order for training samples introduces additional compu- tational overhead. This can be limitation in sce- narios where real-time training is required, e.g., in processing streaming data of microposts.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural mes- sage passing for quantum chemistry. In International conference on machine learning, pages 1263-1272 PMLR.
Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. 2017, Automated curriculum learning for neural networks. In interna- tional conference on machine learning, pages 1311- 1320. PMLR
Aric Hagherg, Pieter Swart, and Daniel Chult. 2008 Exploring network structure, dynamics, and func- tion using networkx. Technical report, Los Alamos National Lab,(LNNL) Los Alamos, NM (United States).
References
Sweta Agrawal and Marine Carpuat. 2022. An imita- tion learning curriculum for text editing with non- autoregressive models. In Proceedings of the 60th
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs In Advances in neural information processing systems.
Annual Meeting of the Association for Computational Linguistics VVolume 1: Long Papers), pages 7550- 7563, Dublin, Ireland. Association for Computational Linguistics.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18-22133.
Hadi Amiri, Timothy Miller, and Guergana Savova. 2017. Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2401-2410, Copenhagen, Denmark. Association for Computational Linguistics.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. Mentomet: Learning data- driven curriculum for very deep neural networks corrupted labels. International Conference on Machine Learning.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international confer- ence on machine learning.
Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. 2003. Marginalized kemels between labeled graphs. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 321 221-328.
Karsten M Borgwardt and Hans-Peter Kriegel. 2005. Shortest-path kernels on graphs. In Fifth IEEE in-

---

Thomas N Kipf and Max Welling. 2017a. Semi- supervised classification with graph convolutional networks. International Conference on Learning Representatioss.
Adyasha Maharana and Mohit Bansal, 2022. On cur- riculum learning for commonsense reasoning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 983-992, Seattle, United States. Association for Computational Linguistics.
Thomas N. Kipf and Max Welling. 2017b. Semi- supervised classification with graph convolutional networks. In 5th International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Andrew Kachites McCallum, Kamal Nigam, Jason Ren- nie, and Kristie Seymore. 2000. Automating the con- struction of internet portals with machine learning. Information Retrieval, 3(2):177-633
Julia Kreutzer, David Vilar, and Artem Sokolov. 2021. Bandits don't 't follow rules: Balancing multi-facet machine translation with multi-armed bandits. In Findings of the Association for Computational Lin- guistics: EMNIP 2021, pages 3190-3204, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Emmanouil Antonios Platanios, Otilia Streteu, Graham Neubig, Barnbbas Poczos, and Tom Mitchell. 2019. Competence-baser curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1162-1172 1172. Minneapolis, Minnesota. Association for Computational Linguisties.
Nils M Kriege, Fredrik D Johansson, and Christopher Morris. 2020. A survey graph kernels. Applied Network Science, 5(1):1-42.
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. 2019. Data parameters: A new family of parameters for learning a differentiable curriculum. Advances in Neural Information Processing Systems.
John P. Lalor and Hong Yu. 2020. Dynamic data se- lection for curriculum learning via ability estimation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 545-555, Online. Association for Computational Linguisties.
Burr Settles and Brendan Meeder. 2016. A trainable spaced repetition model for language learning. In Proceedings of the 54th annual meeting of the associ- ation for computational linguistics (volume 1: Long papers). pages 1848-1858.
Bruce W Lee, Yoo Sung Jang, and Jason Lee. la. Pushing on text readability assessment: trans- former meets handcrafted linguistic features. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10669- 10686.
Diana Sousa, André Lamúrias, and Francisco M Couto. 2019. A silver standard corpus of human phenotype- gene relations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers).
Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021b. Pushing on text readability assessment: A trans- former meets handcrafied linguistic features. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10669- 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online, Association for Computa- tional I Linguistics.
Haoyang Li. Xin Wang, and Wenwu Zhu. 2023. Cur- riculum Graph Machine Learning: A Survey, arXiv e-prints, page arXiv:2302.02926.
Fenglin Liu, Shen Ge, and Xian Wu. 2021. Competence- based multimodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 3001-3012, Online. Association for Computational Linguistics.
Nidhi Vakil and Hadi Amiri. 2022. Generic and trend- aware curriculum learning for relation extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2202-222 2213, Seattle, United States. Association for Computational Linguistics.
Yue Liu, Xihong Yang. Sihang Zhou, Xinwang Liu, Zhen Wang, Ke Liang, Wenxuan Tu, Liang Li, Jing- can Duan, and Cancan Chen. 2023. Hard sample aware network for contrastive deep graph clustering. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 8912-8922.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.

---

S Vichy N Vishwanathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. 2010. Graph kernels. Journal of Machine Learning Research, 11:1201-1242.
Hui Wang. Kun Zhou, Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. 2023. Curriculum pre-training hetero- geneous subgraph transformer for top-n recommen- dation. ACM Transactions on Information Systems. 41(1):1-28.
Yiwei Wang. Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. 2021. Curgraph: Curriculum learning for graph classification. In Proceedings of the Web Conference 2021.
Xiaowen Wei, Xiuwen Gong, Yibing Zhan, Bo Du, Yong Luo, and Wenbin Hu. 2023. Clnode: Curricu- lum learning for node classification. Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 670-678.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understand- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticx, pages 6095-6104, Online. Association for Computational Linguistics.
Cheng Yang. Deyu Bo, Jixi Liu, Yufei Peng, Boyu Chen, Haoran Dai, Ao Sun, Yue Yu, Yixin Xiao, Qi Zhang, et al. 2023. Data-centric graph learning: A survey. arXiv preprimi 2rX11:2310.04987.
Wentao Zhang, Zheyu Lin, Yu Shen, Yang Li, Zhi Yang. and Bin Cui. 2022. Deep and flexible graph neural architecture search. In Proceedings of the 39th Inter- national Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages
PMLR.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019. Curriculum learning for domain adaptation in neu- ral machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1903-1915, Minncapolis, Minnesota. Association for Computational Linguistics.
Tianyi Zhou, Shengjie Wang, and Jeff A Bilmes. 2020. Curriculum learning by dynamic instance hardness. Advances in Neural Information Processing Systems.

---

A Appendix
where æ; is the Katz centrality of node A is the adjacency matrix of Graph G with eigen- values X. The parameter 3 controls the initial centrality and a < Xmaz-
A.1 Graph Indices Definition
Below are the list of 26 indices which we consider for TGCL. All these indices are computed on the subgraph of the node or an edge. These definition and code to caleulate the indices, used Net- workx package (Hagberg al., 2008).
Degree centrality: Given the graph, the de- gree centrality for a node is the fraction of nodes connected it.
Closeness centrality: is the distance to all other nodes in the graph or in the case that the graph is not connected all other nodes in the connected component containing that node. Given the subgraph and the nodes, added the values of the nodes find the complexity index value.
The closeness of a node
Degree: The number of immediate neighbors of a node in a graph.
Treewidth min degree: The treewidth of an graph is an integer number which quantifies, how far the given graph is from being tree.
Average neighbor degree: Average degree of the neighbors of node is computed as:
Eigenvector central- ity computes the centrality for a node based on the centrality its neighbors. The eigen- vector centrality for node Ar Ax. where the adjacency matrix of the graph G with eigenvalue
Eigenvector centrality:
Kj
W
jCA;
where NG is the set neighbors of node and ky is the degree node j.
Group Degree centrality: Group degree cen- trality of a group of nodes S is the fraction of non-group members connected group members.
Degree mixing matrix: Given the graph, it calculates joint probability, of occurrence of node degree pairs. Taking the mean, gives the degree mixing value representing the given graph.
Ramsey R2: This computes the largest clique and largest independent set in the graph G. We calculate the index value by multiplying number of largest cliques to number of largest independent set.
Average degree connectivity: Given the graph, calculates the average of the near- est neighbor degree of nodes with degree We choose the highest value of & obtained from the calculation and used its connectivity value as the complexity index score.
The local clustering of each node in the graph G is the fraction of triangles that exist over all possible triangles in its neighborhood. The average clustering coefficient of graph G the mean of local clusterings.
Average clustering:
Degree assortativity coefficient: Given the graph, assortativity measures the similarity of connections in the graph with respect to the node degree.
Resource allocation index: For nodes and j in subgraph, the resource allocation index is defined as follows:
Katz centrality: The centrality of a node, i, computed based on the centrality its neigh- bors j. Katz centrality computes the relative influence of a node within a network by mea- suring taking into account the number of im- mediate neighbors and number of walks be- tween node pairs. It is computed follows:
kE(M
which quantifies the closeness of target nodes based on their shared neighbors.
Subgraph density: The density of an undi- rected subgraph is computed follows:
0j = a Σ
A +
u(u 1)

---

where the number of edges and is the number of nodes in the subgraph.
Minimum edge dominating set: Minimum edge dominating set approximate solution the edge dominating set.
Local bridge: A local bridge is edge that is not part of triangle in the subgraph. We take the number of local bridges in a subgraph as a complexity score.
Minimum maximal matching: Given a graph G (V,E), a matching M G is a set of pairwise non-adjacent edges; that is, no two edges share a common vertex. That is, out of all maximal matchings of the graph G, the smallest is returned. We took the length of the set as the complexity index.
Number of nodes: Given the graph G, num- ber of nodes in the graph is chosen the complexity score.
Number of Edges: Given the graph G, num- ber of edges in the graph is chosen as the complexity score.
A.2 Linguisties Indices
Below the list of linguistic (Lee al., 2021b) indices used in our experiments. We follow (Lee et al., 2021b) to measure all scores.
Large clique size: Given the graph G, the size of a large clique the graph is chosen as the complexity score.
Traditional Formulas (TraF) These features computes the readability score of the given text based the content, complexity of its vocabu- lary and syntactic information. Readability can be defined the ease with which a reader can under- stand a written text.
Common neighbors: Given the graph and the nodes, it finds the number of common neigh- bors between the pair nodes. We chose number of common neighbors as the complex- ity score.
Gunning Fog Count score: The Gun ning fog index is readability test for En- glish writing, It commonly used to con- firm that text can be read easily by the in- tended audicnce. It is computed as follows: 0.4 words + 100 complezwords
Subgraph connectivity: is measured by the minimum number of nodes that must be re- moved to disconnect the subgraph.
Local node connectivity: Local node connec- tivity for two non adjacent nodes and is the minimum number of nodes that must be removed (along with their incident edges) to disconnect them. Given the subgraph and the nodes, gives the single value which used as complexity score.
sentenoes
words
where words is the number of words, tences is the number of sentences, and com- plexwords the number of complex words
New Automated readability index: The au- tomated readability index is a readability test for texts, which determines the understand- ability of a text. It is computed as follows:
Minimum Weighted Dominating Set: For graph G (V, E), the weighted dominating set problem is to find vertex set C V such that when each vertex is associated with a pos- itive number, the goal to find a dominating set with the minimum weight.
4.71 [[ choractere + 0.5
words sentenees
where characters is the number of letters and numbers, words is the number of spaces, and sentences is the number of sentences.
Weighted vertex cover index: The weighted vertex cover problem to find a vertex cover S-a set of vertices that include at least one endpoint of every edge of the subgraph-that has the minimum weight. This index and the weight of the cover S defined by Exes (s), where w(s) indicates the weight of s. Since wss) = 1, in our unweighted subgraphs, the problem will reduce to finding a vertex cover with minimum cardinality.
This readability test design to determine how difficult is the given text to understand. It can be computed as follows:
Flesch Kincaid Grade level:
0.39
words senilences
syllablee + 11.8
wordss }
where words is the total number of words, sentences the total number of sentences, and syllables is the total number of syllables

---

Linsear Write Formula score: is read- ability metrie for text originally developed to calculate the readability technical manuals. can be computed as follows:
Token sentence multiply: computed by the total count of tokens multiply by the total count of sentences, and square root.
A.3
TGCL Exploits All Ranking Orders
Algorithm Compute Linscar Write score
Initialize r=0
Phase-1
Phase-2
Phase-3
For each easy word, defined as word with 2 syllables
Frequency
75
or less = ++1
For each hard word, defined as word with 3 syllables
25
or more r = +3
Asc.
Des.
Med. Asc.
Med. Des.
Asc.
Des.
Med. Asc.
Med. Des.
Asc.
Des..
Med. Asc..
Med. Des.
sentences
where sentences = number sentences word sample
r 20, LinsearW Writesooeee = 1
(a) PGR
r =< 20, LinseerW W rilescore
Phase-1
Phase-2
Phase-3
Frequency
1500
1000
Coleman Liau Readability Score: The Cole-
500
man-Liau index is calculated as follows:
0.0588 L 0.296 S 15.8
Asc.
Des.
Med. Asc.
Med. Des.
Asc.
Des.
Med. Asc.
Med. Des.
Asc.
Des.
Med. Asc.
Med. Des.
where L is the average number of letters per 100 words, S is the average number of sen- tences per 100 words
(b) Ogbn-Arxiv
SMOG Index: SMOG index can be calcu-
Figure distribution of indices used for training based their ordering strategy. Overall, scheduler (a): relies more on ascending order in the case of PGR and (b): relies almost equally on all order types in the case of Ogbn-Arxin order is in line with recent studies showing the impor- tance of using medium-level samples for effective train- ing (Swayamdipta et al., 2020).
lated as follows:
1.0430 (polysyllables sentences 1/2 3.1291
Shallow Features (ShaF) These features cap- tures the surface level difficulties. Features used are as follows:
The of medium ascending
Average count of characters per token: The average count of characters per token is taken as the complexity score.
Figures 7a and 7b show the distribution of in- dices used for training based on to their ordering strategy (see $2.1 and Algorithm 1) during different phases of training. As mentioned before, complex- ity scores can be sorted in four ways: ascending. descending, medium ascending, and medium de- scending orders, which represent easy-to-hard hard-to sard-to-easy order types. The results on the PGR dataset show that in the initial phase of training the scheduler uses all order types, while emphasizing most on ascending followed by descending orders. And, in the mid and late training phases, the model prioritizes the ascending difficulty order over the other orders with fairly larger difference in usage. The results Ogbn-Arxiv show that TGCL relies equally on all order types during its training with slightly greater emphasis on descending order at the latter stages of training. The relatively signif- icant use of medium ascending order, especially
Average count of characters per sentence: The average count of characters per sentence is taken as the complexity score.
Average count of syllables per token: The average count of syllables per token is taken as the complexity score.
Average count of syllables per sentence: The average count of characters per syllables is taken as the complexity score.
Sentence length: computed by count of token per sentence
Token sentence ratio: computed by the log of total count of tokens divided by the log of total count of sentences.

---

the early stage of training, is in line with recent studies showing the importance of using medium- level samples for effective training (Swayamdipta et al., 2020).
were used in one of the dataset used in the experi- ments.
A.6 Detailed results
Table shows the performance of TGCL model with different kerne functions for all the datasets on GTNN as the base model.
A.4 Fine-grained Index Analysis
Figure 8 shows fine-grained analysis for Oghn- Arxiv when linguistic indices are included along with the graph indices. the Phase-1, scheduler focuses on all the indices with more frequency of SraF based indices from linguistics, and Ramsey and degree based indices from graph features. In the Phase-2, the overall use of all the indices is reduced and it focuses more on readability indices (TraF) from linguisties features, and uses Ramsey R2 more from the graph indices. Phase-3, sched- uler uses very less indices at the end the training and focuses on the average count of characters per sentence from linguistic features and degree assor- taivity coefficient from the graph indices.
As shown in the Figure for PGR dataset, cen- trality and degree based indices are used more fre- quently. Closeness centrality, density, and degree assortativity coefficient indices are used more fre- quently in Phase-1 and Phase-2, initial part and middle part of training. In the final phase of the training Phase-3, scheduler focuses closeness centrality and degree connectivity based indices more frequently.
As shown in the Figure 10, for Ogbn-Arxiv dataset, basic and degree based indices are used more frequently. In the initial phase of training, as the threshold is high (1)) scheduler uses all avail- able indices. In the Phase-2, scheduler uses degree centrality, Ramsey R2 and degree assortativity coef- ficient more frequently. Phase-3 scheduler uses local bridge, degree and degree centrality more frequently compare to the other indices.
A.5 Selection Process for Complexity Indices
To avoid the over representation similar indices. we group indices based on their similarity. For this purpose, we compute the Pearson Co-relation co- efficient between complexity scores of each pair of indices and create an n X n correlation matrix (where, n is the total number of indices). We use this co-relation matrix an input to K-means and empirically group the indices into 10 clusters. We randomly select one metric from each cluster for use our curriculum framework. See indices with * labels in Table I. Here, * indicates the indices

---

Phace-1
Phase-2
Phace-2
100
Frequency
400-
204
KARgA AND SOEE Po CF TIRTI
OPEIE PHO PO
un enter
Aajeznn outep
un anee sken
der RET
Co Now Sone
AND Nerte FA Satmanouus Eoous
Figure 8: Fine grained indices priority for Ogxi-Arxiv with linguistic indices
GNN Model
TGCL
Ogbn-Arxiv
Cora Ace
Citeseer Ace
GDPR
PGR
Kernel
Acc
F1
F1
cos
76.3
95.9
76.1
85.4
94.5
gau
76.2
95.6
75.8
85.1
94.5
CINN
lap
76.4
95.2
76.1
85.3
89.2
lin
75.7
95.6
76.1
84.4
87.9
see
76.2
95.6
77.8
84.3
94.5
qua
75.9
96.7
77.7
85.0
93.2
Table 5: Fl and Accuracy performance of TGCL model for different kernel functions on node classification (Ogbn-Are Arxiv, Cora, and Citeseer datasets), and link prediction (GDPR and PGR datasets) using GTNN as base GNN model.
Phase-1
Phase-2
Phate-3
Phane-1
Phate-2
Phate-3
Frequuet
209
0.
Axy Kyreaino cquurap
Sur Aanaeuoo sutep
sedile unusnu oppuneen
Aqyaain erunraa shuane
soutap
sueg Aunqeuioo wutep
nuiie fumu fumuu eouiiee
sa.lip unusnu sapuwen
Aqyatuma cnunrap
sadfap
seeeeeo Aunqeu wudap
Ppame fumu Somuusadin soufiee
Asting
Noarn duptoo suntunske sealep
Aamung
Autuasses
walheg meperrs
ui
DK fone
Laubap aoperee abcane
sjanu funu eaaboy
Kamap
Nattppo imponcally
Intpugjeng 391 smpp somm
Durruu eouuae
impaststios
saulapp
abeane
Figure 9: Fine grained indices priority for edge predic- tion task.
Figure 10: Fine grained indices priority for node predic- tion task.


Reviews:
[{"review_id": "YH7VOwucIg", "reviewer": "Reviewer_8ekE", "summary": "", "questions": "None.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. \n\n***Q :*** It seems that the method needs to select appropriate complexity formalisms based on human experience.\n\n***A :*** The process of establishing complexity formalisms for graph and text is an active area of research. In the current literature, complexity measures are derived through both manual development (by experts) and data-driven automated extraction, which reflects the ongoing exploration in this area. Our model can work with any number of current and future complexity indices and can dynamically emphasize/deemphasize them for effective training. In addition, we note that our model utilizes both the complexity of samples and the learning dynamics of a downstream model to guide training. \n\n***Q :*** Can the proposed complexity-guided CL approach be well applied to recent techniques that jointly [1] or iteratively [2] train GNN and LM models?\n\n***A :*** The proposed curricula (and the ones developed by our baselines) strategically arrange examples for iterative training and, as training paradigms, can be applied to a wide range of neural models including the joint or iterative frameworks suggested by the reviewer. Therefore the key evaluation aspect in our paper centers on comparative assessment among different curricula. While we incorporated four distinct base GNN architectures in our experiments, it is important to acknowledge that including other architectures would greatly benefit from in-depth exploratory research, which falls beyond the primary scope of our current paper. We will highlight this point in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing is clear and easy to follow.\n2. This paper proposes a new curriculum learning framework for training GNNs on text graph data and provides insights into what complexity formalisms are learned by GNNs during their training.", "reasons_to_reject": "I am not an expert in the field of curriculum learning, and I have the following concerns:\n1. It seems that the proposed method needs to select appropriate complexity formalisms based on human experience.\n2. Recent research on learning from text graphs has achieved impressive performance by jointly [1] or iteratively [2] training GNN and LM. Can the proposed complexity-guided CL approach be well applied to such joint or iterative framework?\n\n[1] Chien E, Chang W C, Hsieh C J, et al. Node feature extraction by self-supervised multi-scale neighborhood prediction. ICLR 2022.\n\n[2] Zhao J, Qu M, Li C, et al. Learning on large-scale text-attributed graphs via variational inference. ICLR 2023.", "questions_to_authors": "None.", "paper_topic_and_main_contributions": "This paper presents a curriculum learning approach for effective training with text graph data. The authors hypothesize that existing knowledge about text and graph complexity can inform better curriculum development for text graph data. Motivated by this rationale, a complexity-guided CL approach is proposed for text graphs (TGCL). Specifically, TGCL employs multiview complexity formalisms to space training samples over time for iterative training. The experimental results demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "gM0YQ8geV7", "reviewer": "Reviewer_xYVm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments.\n \n***Q :*** Some parameters in formulas in Section 2 are not explained in detail.\n\n***A :*** We thank the reviewer for the comment. However, we cross checked all the equations and found that all parameters are explained in the paper. In order to assist the reviewer, we reiterate these parameters and provide concise definitions below:\n\nSection 2.2 Competence for Gradual Inclusion\n\n* c(t) indicates the fraction of data that can be used for training (i.e., competence) at time t \n* $c_0$ is the initial competence value \n* $\\alpha$ > 0 controls the shape of the competence function\n\nSection 2.3 Spaced Repetition for Ordering Samples\n\n* $t_i$ is the current delay of the complexity index i \n* $\\hat{t}_i$ is the predicted/optimized delay for complexity index i \n* $\\gamma$ is the performance of the current model on the validation data\n* $d_i$ is the vector of instantaneous losses of the top examples ranked by complexity index i\n* $\\eta$ is the expected performance of the model during training, which is used to calculate optimal delay \n* $\\hat{\\tau}_i$ is the optimal value for the hyperparameter $\\tau$ used in the kernel functions.\n\n***Q :*** Section 2.3 and Algorithm 1 are difficult to follow.\n\n***A :*** The key idea of Section 2.3 and algorithm 1: given that different complexity indices present different orders of training samples, the idea is to strategically deemphasize redundant and already-learned (i.e., easy) complexity indices by increasing intervals between their subsequent use in training. The model dynamically and iteratively learns the duration of these intervals for indices (see the delay parameter) according to the difficulty of their top k sample and learning dynamics of the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They take full account of serval difficulty features and design a data scheduler, see the above part.\n\n2. They conduct sufficient experiments on node classification datasets and link prediction datasets.\n\n3. They also analyze the effect of different difficulty features.\n", "reasons_to_reject": "\n1. There are many parameters in the formulas in Section 2, but there are some that are not explained in detail. \n\n2.  Section 2.3 and Algorithm 1 are difficult to follow.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this submission, they propose a data scheduler for text graph tasks. \n\nFirst, they evaluate the difficulty of data samples based on 26 graph complexity indices (e.g., Degree, Treewidth min degree, Average neighbor degree ) and 14 traditional and shallow linguistics indices (e.g. Readability, Average count of characters per token, Average count of characters per sentence).  \n\nSecond, they design a  competence function to introduce training samples to GNNs gradually. Specifically, they use Spaced Repetition to determine which complexity indices should be used for training at each time.\n\n\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZQLNPyXtD5", "reviewer": "Reviewer_G4id", "summary": "", "questions": "1.\tWhy the existing methods can’t be used for text graphs? What are the research gaps?\n2.\tWhat is the intuition of Equation 5?\n3.\tHow many independent runs have been conducted to get the accuracy and F1 in Table 2?\n4.\tHow the proposed method enables transferability of the learned curricula.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. \n\n***Q :*** Why the existing methods can’t be used for text graphs? What are the research gaps? Moreover, formal formulations of text graphs and the research question are missing in the paper. The need for designing a new curriculum learning method for text graphs is not justified. \n\n***A :*** Existing methods are applicable to text graphs, please see the range of baseline models in our paper. The existing gap in current research, however, is that existing models are often limited to a single criterion for assessing sample difficulty, typically instantaneous training loss and its variations. This approach overlooks the nature of difficulty: difficulty is a condition that can be realized from multiple perspectives, can vary across a continuum for different models, and can dynamically change as the model improves. To bridge this gap, we propose to utilize graph and text complexity formalisms, which provide a more natural and holistic framework for understanding sample complexity. To our knowledge, no existing model utilizes these formalisms for curriculum learning on text-graphs. Our model addresses these gaps in existing research. In addition, we note that text graphs in existing literature are defined as structured representations that interconnect textual elements.\n\n***Q :*** The paper lacks a clear motivation for considering text graphs. Except for the choice of complexity indices, which can easily be changed according to the domain, the proposed method is general and can be applied to other graphs or even other types of data. \n\n***A :*** As mentioned by the reviewer our approach is applicable to other graphs or genres of data, which we consider as a key strength indicating the potential of the model beyond NLP. However, our research is rooted in NLP, which motivates our use of text and graph complexity indices and text-graph data.\n\n***Q :*** What is the intuition of Equation 5?\n\n***A :*** Equation 5 gradually deemphasizes redundant and already-learned complexity indices by increasing intervals between their subsequent use in training. It dynamically and iteratively learns the duration of these intervals for indices (see the delay parameter, $t_i$) according to the difficulty of their sample ($d_i$) and learning dynamics of the model ($\\gamma$). \n\n***Q :*** How many independent runs have been conducted to get the accuracy and F1 in Table 2? \n\n***A :*** In our experiments, we fixed the random seed and ran all the models once to report the accuracy and F1 in Table 2. However, to answer the reviewer's question, we ran our model with GTNN as a base model on Arxiv dataset with 3 different seeds and the average performance is 76.04 ± 0.06 (mean, SD). We will report results with 3 seeds for all models and datasets in Appendix. \n\n***Q :*** In the overall performance comparison (Table 2), other curriculum learning methods do not improve performance compared to No-CL. \n\n***A :*** Our baseline curriculum learning approaches including CurGraph, CLNode, and CCL occasionally perform worse than no curriculum learning setting (No-CL). This could be attributed to the monotonic nature (i.e., easy-to-hard transition) of these curricula. Recent research shows that monotonic curricula “are often at the risk of underperforming” (Elgaar, et al., ACL’23). On the other hand, non-monotonic methods such as SL and Trend-SL baselines and our model (TGCL), often outperform No-CL across all based GNN models and datasets, showing their effectiveness.   \n\n***Q :*** How the proposed method enables transferability of the learned curricula? In line 118, the authors said the learned curricula are model-dependent, but they also said the curricula are transferrable across models. These two statements seem to be contradictory.\n\n***A :*** The primary objective of our work is to develop curricula for text-graph data by leveraging graph and linguistic complexity indices. Transferability experiments show that these learned curricula can be applied to other datasets or GNN models, resulting in significant efficiency gain and comparable performance. Although investigating the reasons falls beyond the primary scope of our current paper, we conjecture that the transferability of the learned curricula derived from complexity formalisms can be attributed to two key factors: (a): shared topological properties among graphs within the same or different domains, such as prevalent features and characteristics in citation networks or power law distribution of node degrees in most graphs, and (b): the fundamental message-passing architecture inherent to most GNN models, which could associates their learning dynamics. However, further analysis on these ideas is the subject of the future work as mentioned in Line# 522. \n\n***Q :*** Equations 7-11 provide several choices of function f. However, there is no theoretical analysis or empirical experiments to advise on the choice of function f.\n\n***A :*** As mentioned in Line 266, our model is applicable to any non-increasing function f (sample scheduler), and Equations 7—11 exemplify such functions. We reported the best performing function for each dataset, considering the space limit. However, we note that model performance varies across these functions as they differ in their flexibility in delaying samples for training. Overall, quadratic, laplacian and secant functions are more effective (and more flexible in delaying indices) while linear and cosine functions are less effective (and also less flexible in delaying indices). We will report detailed results of these functions in Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe proposed training scheduler can be generalized to other curriculum learning scenarios. The novelty of the work lies in the utilization of various complexity indices and the algorithms for selecting/delaying the use of some indices during the training process.\n2.\tThe paper analyses the importance of different indices during different periods of the training process.", "reasons_to_reject": "1.\tThe paper lacks a clear motivation for considering text graphs. Except for the choice of complexity indices, which can easily be changed according to the domain, the proposed method is general and can be applied to other graphs or even other types of data. Moreover, formal formulations of text graphs and the research question are missing in the paper.\n2.\tSeveral curriculum learning methods have been discussed in Section 1. However, the need for designing a new curriculum learning method for text graphs is not justified. The research gap, e.g., why existing methods can’t be applied, is not discussed.\n3.\tEquations 7-11 provide several choices of function f. However, there is no theoretical analysis or empirical experiments to advise on the choice of function f. \n4.\tIn the overall performance comparison (Table 2), other curriculum learning methods do not improve performance compared to No-CL. These results are not consistent with the results reported in the papers of the competitors. At least some discussions about the reason should be included. In addition, it is unclear how many independent runs have been conducted to get the accuracy and F1. What are the standard deviations?\n5.\tAlthough experimental results in Table 3 and Table 4 show that the performance remains unchanged, it is unclear how the transfer of knowledge is done in the proposed method. An in-depth discussion of this property should strengthen the soundness of the paper.\n6.\tIn line 118, the authors said the learned curricula are model-dependent, but they also said the curricula are transferrable across models. These two statements seem to be contradictory.", "questions_to_authors": "1.\tWhy the existing methods can’t be used for text graphs? What are the research gaps?\n2.\tWhat is the intuition of Equation 5?\n3.\tHow many independent runs have been conducted to get the accuracy and F1 in Table 2?\n4.\tHow the proposed method enables transferability of the learned curricula.", "paper_topic_and_main_contributions": "This paper proposed a curriculum learning approach for text graphs. The main hypothesis of the paper is that the graph and text complexity have impact on the learning process. Following the existing curriculum learning paradigm, the authors employed various graph and text complexity indices to quantify the difficulty of a training instance and used a competence function to include examples from easy to hard. In my understanding, the main contributions of the paper are two-fold. First, a training schedular is proposed to select the graph/text complexity indices that may improve the learning process in each epoch. The training schedular is general and is applicable to general curriculum learning problems. Second, the paper finds that both text and graph complexity indices are useful in curriculum learning for text graphs and text complexity indices are more frequently used.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\tGrammar: Line 110, “TGCL tackle” --> “TGCL tackles”, Line 353, “gradually introduce” --> “gradually introduces”\n2.\tSection 2.2 spent a large space to discuss a linear/sub-linear competence function. In my understanding, these competence functions are not this paper’s contribution. I recommend the authors shrink this section and allow more space to discuss the motivation and explain the proposed training scheduler more in-depth.\n3.\tEquation 5 is essential for the proposed training scheduler. However, the notations are not clearly explained. For instance, the dimension of d_i and t_i are unknown. The design of the equation lacks an intuitive explanation. For example, it is unclear why x_i is defined as it is in the paper."}}]

Author rebuttal:

Meta-review: This paper proposes a curriculum learning (CL) approach (especially a data scheduler) for effective training with text graph data.

Pros:
1) The proposed curriculum learning method is overall novel and can generalize to other scenarios besides text graphs.
2) The experiment on several node classification and link prediction datasets is sufficient.

Cons:
1) In the main results (Table 2), some recent baselines cannot outperform the No-CL method, which is also mentioned by one reviewer. The authors' rebuttal cannot fully convince me because the most recent baseline CLNode performs worse than No-CL in almost all the datasets. I wonder whether the experimental setting is fair for the proposed method and all the basellines.
2) The connection between the proposed CL method and text graph tasks should be described more clearly. Since this paper targets at a specific data type, the design of the proposed method should include some features that can deal with the core problems in text graph data.
Decision: Accept-Findings