SlowGuess/NAACL_2022_OCR huggingface dataset:
Nidhi Vakil
Department of Computer Science  
University of Massachusetts Lowell  
nvakil@cs.uml.edu
Hadi Amiri
Department of Computer Science  
University of Massachusetts Lowell  
hadi@cs.uml.edu
We present a generic and trend-aware curriculum learning approach for graph neural networks. It extends existing approaches by incorporating sample-level loss trends to better discriminate easier from harder samples and schedule them for training. The model effectively integrates textual and structural information for relation extraction in text graphs. Experimental results show that the model provides robust estimations of sample difficulty and shows sizable improvement over the state-of-the-art approaches across several datasets.
Relation extraction is the task of detecting (often pre-defined) relations between entity pairs. It has been investigated in both natural language processing (Mintz et al., 2009; Lin et al., 2016; Peng et al., 2017; Zhang et al., 2018) and network science (Zhang and Chen, 2018; Fout et al., 2017). Relation extraction is a challenging task, especially when data is scarce. Nonetheless, the ability to automatically link entity pairs is a crucial task as it can reveal relations that have not been previously identified, e.g., informing clinicians about a causal relation between a gene and a phenotype or disease. Figure 1 shows an example sentence from a PubMed article in the Gene Phenotype Relation (PGR) dataset (Sousa et al., 2019), which describes the application domain of the present work as well.
Previous research has extensively investigated relation extraction at both sentence (Zeng et al., 2015; dos Santos et al., 2015; Sousa et al., 2019) and document (Yao et al., 2019b; Quirk and Poon, 2017) levels. Furthermore, effective graph-based neural network approaches have been developed for various prediction tasks on graphs, including link prediction between given node pairs (Kipf and Welling, 2017; Hamilton et al., 2017; Xu et al., 2018; Velicković et al., 2018). Several recent approaches (Li et al., 2020; Zhang and Chen, 2018;
Figure 1: An example showing the report of a causal relation between a gene and a phenotype (symptom) from the PGR dataset (Sousa et al., 2019).
Alsentzer et al., 2020) illustrated the importance of enhancing graph neural networks using structurally-informed features such as shortest paths, random walks and node position features.
In this work, we develop a graph neural network titled Graph Text Neural Network (GTNN) that employs structurally-informed node embeddings as well as textual descriptions of nodes at prediction layer to avoid information loss for relation extraction. GTNN can be trained using a standard approach where data samples are fed to the network in a random order (Hamilton et al., 2017). However, nodes, edges or sub-graphs can significantly vary in their difficulty to learn, owing to frequent substructures, complicated topology and indistinct patterns in graph data. We tackle these challenges by presenting a generic and trend-aware curriculum learning approach that incorporates sample-level loss trajectories (trends) to better discriminate easier from harder samples and schedule them for training graph neural networks.
The contributions of this paper are: (a): a graph neural network that effectively integrates textual data and graph structure for relation extraction, illustrating the importance of direct use of text embeddings at prediction layer to avoid information loss in the iterative process of learning node embeddings for graph data; and (b): a novel curriculum learning approach that incorporates loss trends at sample-level to discover effective curricula for training graph neural networks.
Human Language Technologies, pages 2202 - 2213
We conduct extensive experiments on real world datasets in both general and specific domains, and compare our model against a range of existing approaches including the state-of-the-art models for relation extraction. Experimental results demonstrate the effectiveness of the proposed approach; the model achieves an average of 8.6 points improvement in F1 score against the best-performing graph neural network baseline that does not directly use text embeddings at its prediction layer. The proposed curriculum learning approach further improves this performance by 0.7 points, resulting in an average F1 score of 89.9 on our three datasets. We conduct extensive experiments to shed light on the improved performance of the model. Code and data are available at https://clu.cs.uml.edu/tools.html.
Consider an undirected graph  $G = (\mathcal{V},\mathcal{E})$  where  $\mathcal{V}$  and  $\mathcal{E}$  are nodes and edges respectively, and nodes carry text summaries as their descriptions. Edges in the graph indicate "relations" between their end points, e.g., causal relations between genes and diseases, or links between concepts in an encyclopedia. Our goal is to predict relations/links between given node pairs in  $G$ .
2.1 Graph Text Neural Network
We present the Graph Text Neural Network (GTNN) model which directly operates on  $G$  and textual descriptions of its nodes. Figure 2 shows the architecture of GTNN, which we describe below.
2.1.1 Graph Encoder
Given  $G$  and its initial text embeddings,  $\mathbf{x}_i$  for each node  $i$ , we apply a graph encoder (Hamilton et al., 2017) to generate a  $d$ -dimensional embedding for each node by iteratively aggregating the current embeddings of the node and its  $t$ -hop neighbours through the sigmoid function denoted by  $g$ :
$$
\mathbf {h} _ {i} ^ {(t + 1)} = g \left(\mathbf {W} _ {1} \mathbf {h} _ {i} ^ {(t)} + \mathbf {W} _ {2} \left(\frac {1}{| \mathcal {N} _ {i} |} \sum_ {j \in \mathcal {N} _ {i}} \mathbf {h} _ {j} ^ {(t)}\right)\right), \tag {1}
$$
where  $\mathbf{h}_i^{(t)}$  is the embedding of node  $i$  at the  $t^{th}$  layer of the encoder and is initialized by  $\mathbf{x}_i$ , i.e.,  $\mathbf{h}_i^{(0)} = \mathbf{x}_i, \forall i$ , and  $\mathcal{N}_i$  is the set of neighbors of node  $i$  aggregated through a mean operation.  $\mathbf{W}_1$  and  $\mathbf{W}_2$  are parameter matrices to learn during training. Equation (1), applied iteratively, generates node embeddings  $\mathbf{z}_i = \mathbf{h}_i^{(t + 1)} \in \mathbb{R}^d$ .
2.1.2 Additional Text Features
In addition to the representations obtained from the graph encoder, we use additional features from text data to better learn the relations between entities. Here, we consider three types of features: (a) relevance score between the descriptions of node pairs obtained from information retrieval (IR) algorithms; we use BM-25 (Robertson et al., 1995), classic TF/IDF (Jones, 1972), as well as DFR-H and DFR-Z (Amati and Van Rijsbergen, 2002) models. These IR models capture lexical similarities and relevance between node pairs through different approaches; (b): we also use the initial text embeddings of nodes  $(\mathbf{x}_i,\forall i)$  as additional features because the direct uses of these embeddings at prediction layer can avoid information loss in the iterative process of learning node embeddings for graph data; and (c): if there exist other text information for a given node pair, e.g., a sentence mentioning the node pair as in Figure 1, we use the embeddings of such information as additional features.
2.1.3 Graph Text Decoder
For a given node pair  $(u,v)$ , we combined representation of their additional features using a single hidden layer neural network as follows:
$$
\mathbf {h} _ {u v} = \operatorname {R e L U} \left(\mathbf {W} ^ {e} \mathbf {a} _ {u v} + \mathbf {b} ^ {e}\right), \tag {2}
$$
where  $\mathbf{a}$  is obtained by concatenating the additional feature vectors of  $u$  and  $v$ . We combine  $\mathbf{h}_{uv}$  with node representations,  $\mathbf{z}_u$  and  $\mathbf{z}_v$ , and pass them to a two layer decoder to predict their relations:
$$
\mathbf {h} = \operatorname {R e L U} \left(\mathbf {W} ^ {\text {l a s t}} f \left(\mathbf {h} _ {u v}, \mathbf {z} _ {u}, \mathbf {z} _ {v}\right) + \mathbf {b} ^ {\text {l a s t}}\right), \tag {3}
$$
$$
p (u, v) = g \left(\mathbf {W} ^ {o u t p u t} \mathbf {h} + \mathbf {b} ^ {o u t p u t}\right),
$$
where  $f$  is a fusion operator,  $g$  is the sigmoid function, and  $p(u,v)$  indicates the probability of an edge between nodes  $u$  and  $v$ . Flattened outer product, inner product, concatenation and 1-D convolution can be used as the fusion operator (Amiri et al., 2021). In our experiments, we obtained better performance using outer product, perhaps due to its better encoding of feature interactions:
$$
f \left(\mathbf {h} _ {u v}, \mathbf {z} _ {u}, \mathbf {z} _ {v}\right) = \mathbf {h} _ {u v} \otimes \left[ \mathbf {z} _ {u}; \mathbf {z} _ {v} \right]. \tag {4}
$$
2.2 Generic Trend-aware Curricula
Graph neural networks are often trained using the standard or "rote" approach where samples are fed to the network in a random order for training (Hamilton et al., 2017). However, edges (and
Figure 2: The architecture of the proposed graph text neural network (GTNN) model with Trend-SL curriculum learning approach. The proposed model consists of an encoder-decoder component that determines relations between given node pairs. The graph neural encoder takes as input features from textual descriptions of nodes and sub-graph extracted for a given node pair to create node embeddings. The resulting embeddings in conjunction with additional text features are directly used by the decoder to predict links between given entity pairs. The resulting loss is given as an input to our Trend-SL approach to dynamically learn a curriculum during training.
other entities in graphs such as nodes and subgraphs) can vary significantly in their classification difficulty, and therefore we argue that graph neural networks can benefit from a curriculum for training. Recent work by Castells et al. (2020) described a generic loss function called SuperLoss (SL) which can be added on top of any target-task loss function to dynamically weight training samples according to their difficulty for the model. Specifically, it uses a global difficulty threshold  $(\tau)$ , determined by the exponential moving average of all sample losses, and considers samples with an instantaneous loss smaller than  $\tau$  as easy and the rest as hard. Similar to the commonly-used easy-to-hard transition curricula, such as those in (Bengio et al., 2009) and (Kumar et al., 2010), the model initially assigns higher weights to easier samples, thereby allowing back-propagation to initially focus more on easier samples than harder ones.
However, SL does not take into account the trend of instantaneous losses at sample-level, which can (a): improve the difficulty estimations of the model by making them local, sample dependent and potentially more precise, and (b): enable the model to distinguish samples with similar losses based on their known loss trajectories. For example, consider an easy sample with a rising loss trend which
is about to become a hard sample versus another easy sample with the same instantaneous loss but a falling loss trend which is about to become further easier for the model. Trend information allows distinguishing such examples.
The above observations inspire our work to utilize trend information in our curriculum learning framework, called Trend-SL. The model uses loss information from the local time window before each iteration to capture a form of momentum of loss in terms of rising or falling trends and determine individual sample weights as follows:
$$
\begin{array}{l} T r e n d S L _ {\lambda , \alpha} \left(l _ {u v}\right) = \arg \min  _ {\sigma_ {u v}} \left(l _ {u v} - (\tau - \alpha \Delta_ {u v})\right) \tag {5} \\ \times \sigma_ {u v} + \lambda (\log \sigma_ {u v}) ^ {2}, \\ \end{array}
$$
where  $\sigma_{uv}$  is the latent weight for the training sample  $(u,v)$ ,  $l_{uv}$  is the target-task loss (binary cross-entropy in our experiments) for  $(u,v)$  at current iteration,  $\tau$  is the batch-level global difficulty threshold determined by the exponential moving average of sample losses (Castells et al., 2020), and  $\Delta \in [-1,1]$  is the trend indicator quantified by the normalized sample-level loss trend weighted by  $\alpha \in [0,1]$ ; our approach reduces to SL with  $\alpha = 0$ .  $\Delta$  captures the trend in the instantaneous losses of samples over recent  $k$  iterations, effec
Figure 3: Difficulty dynamics in Trend-SL.  $\tau$  is the fixed difficulty threshold of SL, which can be thought of as a global difficulty metric to separate easy and hard samples. Dotted (red) and dashed (green) trend lines indicate four samples with rising and falling loss trends respectively. Trend-SL uses trend dynamics to shift the difficulty boundaries and adjust global difficulty using local sample-level loss trends. The vertical dashed and dotted lines show updated sample-specific difficulty thresholds for easy and hard samples respectively.
tively utilizing local sample-level information to determine difficulty. There are various techniques for fitting trends to time series data (Bianchi et al., 1999). We use differences between consecutive losses to determine the trend for each sample:
$$
\Delta_ {u v} = \sum_ {j = i - k + 2} ^ {i} \left(l _ {u v} ^ {j} - l _ {u v} ^ {j - 1}\right) / \sum_ {j = i - k + 2} ^ {i} \mid l _ {u v} ^ {j} - l _ {u v} ^ {j - 1} \mid , \tag {6}
$$
where  $i$  is the current iteration,  $l^j$  indicates loss at iteration  $j$  and  $k$  controls the number of previous losses to consider. As Figure 3 illustrates, Trend-SL increases the difficulty threshold for samples with falling loss trends (negative  $\Delta \mathrm{s}$ ), becoming more flexible in increasing the weights of such samples by allowing greater instantaneous losses. On the other hand, it becomes more conservative in weighting samples with rising trends (positive  $\Delta \mathrm{s}$ ) by reducing the difficulty threshold.
Finally, we note that the weight  $\sigma_{uv}$  in (5) can be computed as follows, where  $W$  is the Lambert W function (Euler, 1783); see details in the supplementary materials in (Castells et al., 2020):
$$
\sigma_ {u v} ^ {*} = \exp \left(- W \left(\frac {1}{2} \max  \left(- \frac {2}{e}, \beta\right)\right)\right), \tag {7}
$$
$$
\beta = \frac {l _ {u v} - (\tau - \alpha \Delta_ {u v})}{\lambda}. \tag {8}
$$
Gene, Disease, Phenotype Relation (GDPR) dataset contains textual descriptions for genes, diseases and phenotypes (symptoms) as well as their relations, and is obtained by combining two freely available datasets: Online Mendelian Inheritance in Man (OMIM) (Amberger et al., 2019) and Human Phenotype Ontology (HPO) (Köhler et al., 2021). OMIM is the primary repository of curated information on the causal relations between genes and rare diseases, and HPO provides mappings of phenotypes to genes/diseases in the OMIM. We introduce a challenging experimental setup based on the task of differential diagnosis (Raftery et al., 2014) using GDPR, where competing models should distinguish relevant diseases to a gene from irrelevant ones that present similar clinical features, making the task more difficult because of high textual and structural similarity between relevant and irrelevant diseases. For example, diseases 3-methylglutaconic type I, Barth syndrome and 3-methylglutaconic type III are of the same disease type and have high lexical similarity in their descriptions, but they are not related to the same genes. We include such harder negative gene-disease pairs by sampling genes from those that are linked to diseases that share the same disease type with a target disease, but are not linked to the target disease. We also include an equal number of randomly sampled negative pairs to this set.
Gene Phenotype Relation (PGR) (Sousa et al., 2019) is created from PubMed articles and contains sentences describing relations between given genes and phenotypes (Figure 1). We only include data points with available text descriptions for their genes and phenotypes. For fair comparison, we apply the best model from (Sousa et al., 2019) to this dataset.
Wikipedia (Rozemberczki et al., 2021) is on the topic of the old world lizards Chameleons with 202 species. In this dataset, nodes represent pages and edges indicate mutual links between them. Each page has an informative set of nouns, which we use as additional features. We note that this dataset contains only these noun features but not the original
${}^{1}$  A gene can cause one or more diseases and a disease can have several disease types. As a pre-processing step, we remove isolated nodes from the dataset and explicit mentions of relations between entities from summaries.
Metric	GDPR	PGR	Wikipedia
# Nodes	18.3K	20.4K	2.2K
# Edges	365.0K	605.4K	31.4K
# Sampled Edges	37.6K	3.0K	188.5K
→# pos. Edges	6.2K	1.4K	31.4K
→# neg. Edges	31.4K	1.6K	157.1K
Table 1: Statistics of the three datasets. Sampled edges are used to create training, validation and test sets. All models take the entire graph as input.
text, which is required by our text only models.
Table 1 shows statistics of these datasets. In case of GDPR and WIKIPEDIA, we create five negative examples for every positive pair. We divide these pairs into  $80\%$ ,  $10\%$  and  $10\%$  as training, validation and test splits respectively. The data splits for PGR is the same as the original dataset, except that we discard data points (node pairs) that do not have text descriptions.
We use the following baselines:
- Co-occurrence labels a test pair as positive if both entities occur together in the input text.
- Relevance Score uses scores from IR models (Section 2.1.2) as features of a logistic classifier.
- Doc2Vec (Le and Mikolov, 2014) uses domain-specific text embeddings obtained from Doc2Vec as features of a logistic classifier.
- BioBERT (Lee et al., 2020; Devlin et al., 2019) is a BERT model pre-trained on PubMed articles. BioBERT is most appropriate for relation extraction on both GDPR and PGR datasets as they are also developed based on PubMed articles. It is the current state-of-the-art model on PGR (Sousa et al., 2019). We also include a version of BioBERT that uses graph information by concatenating the representation of each given pair with the average embedding of its neighbors.
- Graph Convolutional Network (GCN) (Kipf and Welling, 2017) is an efficient and scalable approach based on convolution neural networks which directly operates on graphs.
- Graph Attention Network (GAT) (Velicković et al., 2018) extends GCN by employing self-attention layers to identify informative neighbors while aggregating their information, effectively prioritizing important neighbors for target tasks.
- GraphSAGE (Hamilton et al., 2017) is an inductive framework which aggregates node features
and network structure to generate node embeddings, see (1). It uses both text and graph information. We use Doc2Vec (Le and Mikolov, 2014) embeddings to initialize node features of GraphSAGE, as they led to better performance than other embeddings in our experiments.
- Graph Isomorphism Network (GIN) (Xu et al., 2018) identifies the graph structures that are not distinguishable by the variants of graph neural networks like GCN and GraphSAGE. Compared to GraphSAGE and GCN, GIN uses extra learnable parameters during sum aggregation and uses MLP encoding.
- CurGraph (Wang et al., 2021) is a curriculum learning framework for graphs that computes difficulty scores based on the intra- and inter-class distributions of embeddings and develops a smooth-step function to gradually include harder samples in training. We report the results of our implementation of this approach.
- SuperLoss (SL) (Castells et al., 2020) is a generic curriculum learning approach that dynamically learns a curriculum from model behavior. It uses a fixed difficulty threshold at batch level, determined by the exponential moving average of all sample losses, to assign higher weights to easier samples than harder ones.
We compare these baselines against GTNN and Trend-SL, described in Section 2.
We reproduce the results reported in (Sousa et al., 2019) using BioBERT and therefore follow the same settings on the PGR dataset. Initial domain-specific node embeddings are obtained using Doc2Vec (Le and Mikolov, 2014) or BioBERT (Lee et al., 2020). In case of Bio-BERT, since nodes carry long descriptions, we first generate sentence level embeddings and use their average to represent each node, following (Zhang et al., 2020a). More recent techniques can be used as well (Beltagy et al., 2020). We consider 1-hop neighbors and set  $t = 1$  in (1). To optimize our model, we use the Adam optimizer (Kingma and Ba, 2015) and apply hyper-parameter search and tuning for all competing models based on performance on validation data. In (5), we set  $\alpha$  from [0,1] with a step size of 0.1,  $\lambda$  from {0.1, 0.5, 1.0, 5, 10, 100}, and loss window  $k$  from [1, 10] with a step size of 1. We consider a maxi
Modality	Model	GDPR	PGR	Wikipedia	avg F1
P	R	F1	P	R	F1	P	R	F1
-	Co-occurrence	16.7	100	28.6	47.5	100	64.4	16.7	100	28.6	40.5
T	Relevance Score	59.2	83.4	69.2	75.6	64	69.1	-	-	-	69.2
T	BioBERT (node pairs)	20.3	55.6	29.7	84.9	74.7	79.4	-	-	-	54.6
T	BioBERT (neighbors)	21.1	57.4	30.9	74.0	76.0	75.0	-	-	-	53.0
T	Doc2vec (node pairs)	19.8	45.0	27.5	80.5	82.7	81.6	-	-	-	54.6
T	Doc2vec (neighbors)	20.6	51.9	29.5	83.1	78.7	80.8	-	-	-	55.2
G	GCN	34.2	44.5	38.6	61.1	79.5	68.6	72.8	89.7	80.3	62.5
G	GAT	23.7	50.3	31.7	75.8	91.1	82.5	78.2	86.7	82.2	65.5
G	GIN	21.8	48.1	29.8	54.2	88.1	67.0	76.4	77.2	76.1	57.6
G	GraphSAGE (random)	17.2	90.4	28.5	84.8	79.2	81.8	57.9	82.28	67.9	59.4
G,T	GraphSAGE (Doc2Vec)	54.0	79.2	64.1	91.8	90.2	91.0	81.5	93.0	86.6	80.6
G,T	GTNN	78.0	87.9	82.6	93.6	93.2	93.4	87.9	95.4	91.5	89.2
Table 2: Performance of different models on GDPR, PGR, and WIKIPEDIA datasets. Here, (T) indicates "Text only", (G) indicates "Graph only", (G,T) indicates combination of both. Note that the WIKIPEDIA dataset contains only noun features but not the original text, which is required by the text only models.
Model	GDPR	PGR	Wikipedia	avg F1
GTNN	82.6	93.4	91.5	89.2
CurGraph	75.9	85.1	80.3	80.3
SL	83.5	94.0	92.0	89.8
Trend-SL	84.3	94.2	91.3	89.9
Table 3: Performance of curriculum models on GDPR, PGR, and WIKIPEDIA datasets. The base model for all curriculum learning approaches is GTNN, see the last row in Table 2.
mum number of 100 training iterations with early stopping based on validation data for all models. In addition, we evaluate models based on the standard Recall, Precision and F1 score for classification tasks (Buitinck et al., 2013). We experiment with five random seeds and report the average results. For all experiments, we use Ubuntu 18.04 with one 40GB A100 Nvidia GPU, 1 TB RAM and 16 TB hard disk space. GPU hours to train our model have been linear to the size of the datasets ranging from 30 min to 5 hours. We use Precision (P), Recall (R) and F1 score (F1) as evaluation metrics.
Table 2 shows the results. We start with text only and graph only baselines followed by baselines that incorporate both data modalities.
Text models (T): Comparing all text based model, Relevance Score and Doc2Vec outperform other models. In case of GDPR, high performance of Relevance Score indicates the ability of unsupervised IR models in finding relevant information in long text descriptions. However, Relevance Score shows poor performance on PGR compared to Doc2Vec, which is better at semantic representation of input data. BioBERT (node pair) obtains
higher precision on both datasets and good performance on PGR. In addition, the F1 score of the BioBERT model developed in (Sousa et al., 2019) for PGR is 76.6. We note that Doc2Vec obtains better performance than BioBERT, perhaps due to its in-domain pre-training.
Graph models (G): The results show that GCN and GAT perform better than other competing graph models. We attribute their performance to the use of convolution and attention networks, which effectively prioritize important neighboring nodes with respect to the target tasks.