SlowGuess/NAACL_2022_OCR huggingface dataset:
Bornali Phukon $^{1}$ , Anasua Mitra $^{2}$ , Sanasam Ranbir Singh $^{1,2}$ , Priyankoo Sarmah $^{1}$
<sup>1</sup>Centre for Linguistic Science and Technology
$^{2}$ Department of Computer Science and Engineering
Indian Institute of Technology Guwahati, Assam, India
{mamunp,anasua.mitra, ranbir, priyankoo}@iitg.ac.in
Taxonomy expansion is a crucial task. Most of the taxonomy expansion approaches are of two types, attach and merge. In a taxonomy like WordNet, both merge and attach are integral parts of the expansion operations, but the majority of studies consider them separately. This paper proposes a novel multi-task learning-based deep learning method known as Taxonomy Expansion with Attach and Merge (TEAM) that performs both the merge and attach operations. This is the first study that integrates both the merge and attach operations in a single model to the best of our knowledge. The proposed models have been evaluated on three separate WordNet taxonomies, viz., Assamese, Bangla, and Hindi. From the various experimental setups, it is shown that TEAM outperforms its state-of-the-art counterparts for attach operation and also provides highly encouraging performance for the merge operation.
Taxonomy, such as the WordNet, is a crucial resource for developing NLP related technologies, as it plays a vital role in various text processing tasks such as information retrieval, information extraction, text classification, summarization, etc. (Pang et al., 2008; Allan et al., 1998; Singhal et al., 2001) (Miller, 1998). As most of the WordNets are manually curated, it often suffers from the problem of limited coverage. Therefore, an automatic taxonomy expansion is a crucial problem to handle the above issue. For taxonomy expansion, WordNet in particular, may need two types of operations; (i) merge, where a new concept  ${}^{1}$  is merged to an existing node, and (ii) attach, where a new concept is inserted as a new node. Figure 1 illustrates these two operations where the word Mango is inserted as a new concept with the attach operation, and the
Figure 1: Example of WordNet taxonomy expansion with attach and merge operations to include new terms "Mango" and "Nutrient".
"Mango" is a specific concept of Fruit not present in the existing WordNet. Hence, a new concept node is created in the taxonomy by attaching it to its generic concept Fruit. As "Nutrient" refers to the same concept as "Food", no new concept is created. "Nutrient" is merged with the existing concept "Food".
word Nutrient is inserted as a new synonym in an existing concept with the merge operation.
Though both of these operations are integral parts of a WordNet taxonomy expansion, all of the existing studies on taxonomy expansion have considered expansion with either attach operation (Schlichtkrull and Alonso, 2016; Vedula et al., 2018; Shen et al., 2020; Yu et al., 2020b; Zhang et al., 2021; Takeoka et al., 2021; Liu et al., 2021) or merge operation (Nakashole et al., 2012; Nguyen et al., 2017; Nakashole et al., 2012; Qu et al., 2017; Boteanu et al., 2018; Wang et al., 2019; Fei et al., 2019), but not together. Realizing the need to apply both the operation, SemEval-2016:task 14 (Semantic taxonomy enrichment) Jurgens and Pilehvar (2016) includes a call for expansion with both attach and merge operations. However, none of the submissions incorporate both operations in a single model.
1 Concept is a basic building block of WordNet, which refers a definition with associated synonym words
Findings of the Association for Computational Linguistics: NAACL 2022, pages 366 - 378
Motivated by the above observations, in this study, we propose an integrated deep learning-based method, namely, Taxonomy Expansion with Attach and Merge (TEAM), which performs both the attach and merge operations in a multitask-learning framework. Though most of the existing studies consider the expansion a regression problem (Shen et al., 2020; Yu et al., 2020b; Zhang et al., 2021), considering that our method performs both the attach and merge operation in a single model, it can also be considered a classification task. As a result, we propose two versions of TEAM, namely, TEAM-RG: Regression, and TEAM-CL: Classification to perform with explicit and implicit rankings. The proposed models have been evaluated on three different WordNet taxonomies, viz., Assamese, Bangla, and Hindi. From the various experimental setups, it is observed that the proposed TEAM-RG and TEAM-CL outperform their baselines counterparts for attach operation, and also obtained encouraging performance for merge operation as well. The major contributions of the paper are summarized as follows:
- A multi-task learning based taxonomy expansion framework TEAM is jointly trained to perform both the Attach and Merge operations. To the best of our knowledge, it is the first integrated model to perform both the Attach and Merge operations in a single model.
- Two variants of TEAM, namely TEAM-Regression (RG) and TEAM-Classification (CL) are proposed.
2 Taxonomy Expansion - Attach and Merge
In this study, we have considered WordNets as our target taxonomies. A WordNet may be defined by a collection of concepts connected by various semantic relationships such as hypernymy, hyponymy, troponymy, etc., where each concept is further defined by a set of attributes such as definition, synonyms, examples, etc (Bhattacharyya, 2010). In this study, we have considered only the hypernymy relation and the definition and synonymy attributes.
In order to be able to apply the proposed model, we first transform the original WordNet taxonomy into an experimental intermediate taxonomy (directed unweighted acyclic graph)  $\mathcal{T} = (V,E)$  where  $V$  represents the set of concepts and  $E$  represents the set of hypernymy relations between the
concepts. A concept  $v \in V$  is further defined by a tuple  $v = (d_v, s_v)$  where  $d_v$  represents the definition of the concept, and  $s_v$  represents the set of associated synonyms. An edge  $e \in E$  represents a hypernymy relation from a parent concept  $v_p$  to its child concept  $v_{ch}$  and is denoted as  $e : (v_p \xrightarrow{\text{hyper}} v_{ch})$ . The taxonomy  $\mathcal{T}$  is arranged in a hierarchical manner with directed edges in  $E$ , as shown in Figure 1. Given the taxonomy  $\mathcal{T}$  and a query concept  $q = (d_q, s_q)$ , the attach and the merge expansion operations are defined below.
Attach (A) — An attach operation is performed when the concept  $q$  is not present in  $\mathfrak{T}$ . The objective of the attach operation is to identify the best matching parent node in taxonomy network known as anchor concept  $a = (d_a, s_a)$ , and insert a new concept  $q$  with an edge  $e : (a \xrightarrow{\text{hyper}} q)$ . In a taxonomy network, a parent node represents a more generic concept of its children. After an attach operation i.e., insertion of  $q$  in  $\mathfrak{T}$  under the anchor  $a$ , the expanded taxonomy is updated as follows.
$$
\mathcal {T} = (V \cup \{q \}, E \cup \{e \}) \tag {1}
$$
Merge (M) — A merge operation is performed when an equivalent concept  $a = (d_a, s_a)$  of the query  $q$  (i.e.,  $d_a \equiv d_q$ ) is already present in  $\mathcal{T}$ , but the synset  $s_q$  is not present in  $a$  (i.e.,  $s_q \cap s_a = \emptyset$ ). The objective of the merge operation is to identify the best matching concept  $a = (d_a, s_a)$ , known as the anchor concept, in the taxonomy network  $\mathcal{T}$  and add the synset  $s_q$  to  $s_a$ . It neither creates a new node nor adds a new edge. It only updates the synset of the anchor concept. After the merge operation, the updated anchor concept in the expanded taxonomy can be expressed as follows.
$$
a = \left(d _ {a}, s _ {a} \cup s _ {q}\right): a \in V \tag {2}
$$
Our objective is to develop an integrated model that performs both attach and merge operations for taxonomy expansion. Since we have two tasks to unify in a single model, we resort to a multi-task learning framework known as Taxonomy Expansion Framework with Attach and Merge (TEAM). This joint learning objective facilitates information flow so that the two tasks can aid each other. Also, we are interested in deciding which expansion operation is to perform given a triplet (expansion task classification) and retrieving the ranked list of candidates
Figure 2: Example of training dataset generation.
The table shows positive and negative training instances corresponding to the query concept "Rock" for both operations Attach and Merge.
(ranking) as prospective anchors to associate the query with. For this first-of-its-kind novel taxonomy expansion task, we propose two versions of TEAM, namely TEAM-Regression (TEAM-RG) and TEAM-Classification (TEAM-CL) — where we show that using either regression or classification learning objectives, this task can be accomplished.
3.1 Training dataset generation
Given a transformed taxonomy  $\mathcal{T}$  (as described in Section 2), we generate a training dataset for building the model as follows. The training samples are defined by a 3-tuple  $< q, a, label>$ , where  $q$  is the query,  $a$  is the potential anchor, and  $label$  is associated class, i.e., true/false  $(1/0)$ . We randomly select a set of nodes in  $\mathcal{T}$  as a set of queries $^2$ , and generate the training samples for the attach and the merge operations separately as follows.
Attach (A) - We first remove the query nodes from the  $\mathfrak{T}$ . For each query  $q = (d_q, s_q)$ , we consider its parent as anchor node  $a = (d_a, s_a)$  and generate positive sample  $< q, a, TRUE>$ . We then randomly pick up  $\mathcal{N}$  number other nodes  $a' = (d_{a'}, s_{a'})$ , and generate  $N$  negative samples  $< q, a', FALSE>$ . Thus, for a given query node  $q$ , we extract one positive and  $\mathcal{N}$  negative samples.
Merge (M) For each of the randomly selected query node  $x = (d_x,s_x >)$  in  $\mathfrak{T}$ , we generate the following positive training sample  $< q,x, True>$  where  $q = (d_x,s_q), s_q \subset s_x$  is the query and  $x = (d_x,s_x - s_q)$  is the anchor. The  $s_q$  is a randomly selected synonym in  $s_x$ . Unlike attach, for generating the training sample for the query  $q$ , we only remove the query synset  $s_q$  from the anchor synset  $s_x$  i.e.,  $s_x = s_x - s_q$ , and, not the node. Like attach, we randomly pick up  $\mathcal{N}$  number other nodes  $a'$ , and generate  $N$  negative samples  $< q,a',FALSE >$ .
Figure 3: Ego tree of the anchor node "Food". 1-hop ego-tree is extracted around the anchor "Food". The color-codes distinguish various roles w.r.t the anchor node "Food", eg., Deep Purple: Grand-parent, Red: Anchor/Parent, Orange: Childrens
Figure 2 illustrates the generation of the training samples from a taxonomy.
3.2 TEAM-Regression (TEAM-RG)
The proposed TEAM-RG works in two tiers process. Given a training input sample  $< q, a, c >$ , it first generates encoding of the query  $q$  and the anchor  $a$ . It then merges to a shared layer to produce two different multi-tasking dense networks; one for merge and another for attach, as shown in figure 4.D.
For learning embedding of the anchor concept from the taxonomy network and the query concept from the associated attributes, we consider the publicly available Fasttext pre-trained embedding available at https://fasttext.cc/docs/en/crawl-vectors.html.
Processing of the query concept: As mentioned in Section 2, a query concept consists of its definition and the associated synset i.e.,  $q = (d_q, s_q)$ . The definition is a piece of text describing the concept, and the synset is a synonym associated with the query concept. . The two embeddings are then concatenated to represent the query.
Processing of the anchor concept: For generating the encoding of the anchor concept, we exploit the proximity structure of the nodes in the taxonomy  $\mathcal{T}$ . For a given anchor node  $a \in \mathcal{T}$ , we first extract its ego-tree from the taxonomy. An ego tree  $\mathcal{T}_a : (V_a, E_a)$  of a node  $a$  in the taxonomy  $\mathcal{T}$  is a sub-tree that comprises the node  $a$  and its  $k$ -hop neighborhood nodes. In this study, we considered  $k = 1$ , i.e., the anchor node, its parent node, and all its children nodes. Figure 3 illustrates an example of an ego tree. A similar approach has also been used in (Wang et al., 2021; Yu et al., 2020b; Zhang et al., 2021; Shen et al., 2020) studies. To obtain the embedding of the anchor concept, we further apply graph embedding as described below.
As we consider the same query set for both attach and merge experiments, nodes with at least two synonyms are considered.
3.2.1 Embedding Ego-tree
Ideally, we should be able to use any graph embedding method to obtain the embedding of the anchor node. As the objective is to incorporate the positional information of the parent and children node in the ego tree, we use the Graph Attention Network (GAT) proposed in Taxo-Expan (Shen et al., 2020). This GAT is a special type of graph neural network (GNN) (Kipf and Welling, 2016) with a neighborhood-based attention mechanism. The details of GAT and its difference from GNN are given in Section B of Appendix. Thus we used position enhanced GAT to obtain the node embeddings of an anchor's ego tree.
We summarize the tree by applying an activation function over the average of the embedding vectors of all nodes in the ego-tree as given in equation 3 to define the encoding of the anchor node.
$$
\bar {\mathcal {T}} _ {a} = \sigma \left(\frac {1}{| V _ {a} |} \sum_ {x \in V _ {a}} \bar {x}\right) \tag {3}
$$
where  $\sigma (.)$  is an activation function. We have considered Sigmoid function in this study.
3.2.2 Multi-task Learning
Once we obtain the embeddings of the anchor and query concepts, the concatenated vector is subjected to a shared dense layer and then build two multi-task layers to perform the merge and attach operations as shown in Figure 4.D. Given a query concept and its true anchor concept with  $\mathcal{N}$  false anchor concepts, the task is to design a regression-based ranking model such that the true concept is ranked higher than the  $\mathcal{N}$  false concepts. This objective should be realized for all the queries in the training dataset.
Given the embedding vectors of anchor  $\bar{a}$  and query  $\bar{q}$  as learned above, we first estimate similarity between the two using a bi-linear model proposed in (Gutmann and Hyvarinen, 2010). It learns the discrimination between  $q$  and  $a$  through a learnable bi-linear scoring matrix  $B\in \mathbb{R}^{|\bar{q} |\times |\bar{a}|}$  via a function  $\mathcal{D}:\mathbb{R}^{|\bar{q} |\times |\bar{a}|}\mapsto \mathbb{R}$  as follows.
$$
\mathcal {D} (q, a) = \sigma (\bar {q} ^ {T} B \bar {a}) \tag {4}
$$
Here  $\sigma$  is sigmoid non-linearity. The output of this matching module is a probability estimate indicating the strength of association between the query and anchor. Now, considering the query concept  $q$  and its associated  $\mathcal{N} + 1$  anchor concepts, we estimate the probability of being the correct anchor
using InfoNCE loss proposed in (Oord et al., 2018). Let  $\mathcal{X}$  be a set of query concepts and their respective  $\mathcal{N} + 1$  anchor nodes (one positive and  $\mathcal{N}$  negative). An element of  $x_{q} \in \mathcal{X}$  for a given query  $q$  consists of  $\{(q, a, 1), (q, a_{1}', a_{2}', \dots, a_{\mathcal{N}}', 0)\}$ , where  $a$  is the positive anchor, and  $a'$  are the negative anchors of  $q$ . InfoNCE estimates loss function using an average probability of being true anchor node across the dataset  $\mathcal{X}$  as follows.
$$
\mathcal {L} _ {A / M} = - \frac {1}{| \mathcal {X} |} \sum_ {x _ {q} \in \mathcal {X}} \log \frac {\mathcal {D} (\bar {q} , \bar {a})}{\sum_ {v \in \mathcal {M} (q)} \mathcal {D} (\bar {q} , \bar {v})} \tag {5}
$$
where  $\mathcal{M}(q)$  denotes the set of both positive and negative anchors of  $q$ . As mentioned earlier, the loss defined in Equation 5 is estimated separately for attach and merge operations. Therefore, we generate two different training datasets for attach and merge, and estimate  $\mathcal{L}_A$  and  $\mathcal{L}_M$  separately using respective datasets. The final model loss is defined as  $\mathcal{L} = \mathcal{L}_A + \mathcal{L}_M$  — considering both the operations attach and merge.
3.3 TEAM-Classification (TEAM-CL)
Figure 4E shows the schematic diagram of the TEAM-CL. We use the identical representations for query  $q$  and candidate anchors  $a_A, a_M$  as described for TEAM-RG. We also adopt the same position-enhanced graph propagation and read-out modules as described in Section 3.2.1 for learning anchor  $a = (d_a, s_a)$  concept representation. Once we obtain the query and anchor representations, we model the strength of association of an input query and the candidate anchors based on their features to predict the expansion task i.e., merge M or attach A. The matching module, a multilayer perceptron (MLP) based classifier, takes the features of query  $\bar{q} \in \mathbb{R}^{|\bar{q}|}$  and anchor  $\bar{a} \in \mathbb{R}^{|\bar{a}|}$ , and generates a contextualized pair representation  $\bar{k} = [\bar{q} \oplus (\bar{q} - \bar{a}) \oplus (\bar{q} \times \bar{a}) \oplus \bar{a}]$  (assuming  $|\bar{q}| = |\bar{a}|$ ). Here,  $\oplus$  denotes concatenation. The anchor  $a$  can be any of the attach or merge candidates  $(a_A / a_M)$ . A three-way classifier is learned to produce the categorical probability distribution over the training samples for Merge (M), Attach (A) and No-operation (N) — three classes  $(|\mathcal{Z}| = 3)$  of operations. If  $\theta \in \mathbb{R}^{|\bar{k}| \times |\bar{\mathcal{Z}}|}$  be a learnable projection matrix that projects the contextualized pair embedding  $\bar{k}$  to the label space  $\mathcal{Z} \in \mathbb{R}^3$ . The predictions are obtained as below,
$$
\hat {Y} = \operatorname {s o f t m a x} (\mathrm {M L P} (\bar {k}; \theta)) \tag {6}
$$
Figure 4: Taxonomy Expansion framework with Attach and Merge (TEAM) D: TEAM-Regression-RG —  $\cdot$ 1. (Query Q, Anchor,  $(\mathrm{N} + 1)$  Negative Anchors) are fed to the model,  $\cdot$ 2. Representation learning via shared graph propagation and readout modules,  $\cdot$ 3. Projection to shared hidden layers,  $\cdot$ 4. Task-specific matching modules with non-shareable weights,  $\cdot$ 5. Task-specific regression outputs. E: TEAM-Classification (CL) —  $\cdot$ 1. (Query Q, Anchor-A, Anchor-M,  $(\mathrm{N} + 1)$  Negative Anchors) are fed to the model,  $\cdot$ 2. Representation learning via shared graph propagation and readout modules,  $\cdot$ 3. Projection to shared hidden layers,  $\cdot$ 4. Simultaneous optimization of classification and ranking losses.  $\cdot$ 5.Three-way merge, attach, no-operation (M, A, N) prediction. Explanation of used color-codes. Light-Purple: Anchor (A/M) nodes, Orange: Children of the anchor, Purple: Parent of the anchor. Green: Definition representation of anchors, Blue: Synset representation of anchors, Yellow: Query representation.
For two versions of TEAM, we chose two different kinds of matching models based on empirical performances to capture different kinds of embedding interaction in the latent space.
Classification. Unlike in TEAM-RG, where we posit taxonomy expansion as a regression task with implicit ranking viz. discriminating true and false examples via InfoNCE loss, in TEAM-CL, we simultaneously optimize for classification and explicit ranking objectives. We obtain classification predictions from the matching module as described before. Given a training set  $\mathcal{X}$ , and a set of classes  $\mathcal{Z}$  (M: Merge, A: Attach, N: No-operation), we optimize for the self-supervised cross-entropy loss over the task predictions  $\hat{Y}$  given the ground-truth task-classes  $Y$  for an input query-anchor pair.
$$
\mathcal {L} _ {C} = - \frac {1}{| \mathcal {X} |} \sum_ {i \in \mathcal {X}} \sum_ {z \in \mathcal {Z}} Y _ {i z} \ln \hat {Y} _ {i z} \tag {7}
$$
Ranking. The classification objective can only learn and infer the confidence score of an operation (M/ A/ N) for a training sample. It fails to give us a reliable ranked list of prospective anchors-(A/ M) given a query — since it does not learn the relative ranks of positive and negative anchors for a query. As illustrated in Figure 4, for a query q, (i) the ego-tree of anchor-A comprises of that query's parent's hierarchical neighborhood, and (ii) the ego-tree of anchor-M comprises of that query's
replica's (same/similar definition with a missing portion of synset) hierarchical neighborhood. Since a query  $q$  is very similar to both of its anchor-A and anchor-M's ego-trees – these operations are hardly distinguishable. Thus, a model must accommodate a provision for directly comparing the prediction scores of M and A operations and learning a margin of separation between the scores. Here, we introduce two ranking objectives in the framework — (i) a contrastive objective to compare and contrast among a positive anchor and  $\mathcal{N}$  negative anchors, (ii) a pair-wise hinge loss to learn a maximum margin between the M and A prediction scores.
Let,  $dist(.)$  be a function to measure the distance between a query  $\bar{q}$  and its true/ false anchor-(A/M) representations  $(a_{\bar{A}},a_{\bar{A}}^{\prime}),(a_{\bar{M}},a_{\bar{M}}^{\prime})$  .We use "slash" (/) to denote either. We intend to rank a positive query-anchor pair  $(q,a_{A} / a_{M})$  higher than  $\mathcal{N}$  no of negative pairs  $(q,a_A^\prime /a_M^\prime)$  by enforcing a group-wise contrastive loss using a margin  $\lambda$  as,
$$
\begin{array}{l} \mathcal {L} _ {R 1 _ {A}} = \\ \frac{1}{\mathcal{X}}\sum_{i\in \mathcal{X}}\frac{1}{|\mathcal{N}(\bar{q}_{i})|}\sum_{a_{A_{i}^{\prime}}^{\prime}\in \mathcal{N}(\bar{q}_{i})}\max (0,\lambda -m + m^{\prime}) \\ m = - \operatorname {d i s t} \left(\bar {q} _ {i} - a _ {A _ {i}}\right), m ^ {\prime} = - \operatorname {d i s t} \left(\bar {q} _ {i} - a _ {A _ {i} ^ {\prime}}\right) \tag {8} \\ \end{array}
$$
We can similarly compute the margin-based groupwise contrastive loss  $\mathcal{L}_{R1_M}$  for the Merge (M).
Now, to distinguish between  $M$  and  $A$  operations, let,  $f(\bar{k})$  be a function that projects the contextualized  $(q,a)$  embedding  $\bar{k}$  in Equation 6, to
a hidden space  $\mathbb{R}^h$ . Here we introduce a margin-based hinge-loss on sample anchor pairs attach-merge  $\langle a_A, a_M \rangle$  for a given query  $q$  via their contextualized vectors  $\langle k_A, k_M \rangle$ . If class labels of merge and attach are  $M = 2$ ,  $A = 1$ , we ensure the prediction scores  $\hat{Y}_{A / M} = f(\bar{k}_{A / M})$  for M and A are separated by a margin of  $\lambda$ .
$$
\mathcal {L} _ {R 2} = \sum_ {Y (k _ {A} ^ {-}) > Y (k _ {M} ^ {-})} \max  \left(0, \lambda - f (k _ {M} ^ {-}) + f (k _ {A} ^ {-})\right)
$$
Therefore, the final loss is,  $\mathcal{L} = \mathcal{L}_C + \mathcal{L}_{R1_A} + \mathcal{L}_{R1_M} + \mathcal{L}_{R2}$  — considering both margin-based group-wise contrastive loss and pairwise hinge loss comprising the overall ranking loss.
3.4 Model Inference
We follow Taxo-Expan's (Shen et al., 2020) evaluation strategy for inferring the best candidate anchor  $a$  given a query  $q$ . We use our classification objective to decide which operation among merge, attach, or no-operation (M, A, N) to perform when  $q$  is given. i) For TEAM-RG, we augment a classification layer on top of the task-specific regression layers. Given a query  $q$  and a set of candidate anchors  $a$ , we obtain the merge and attach regression scores and choose the best value along with the corresponding operation as the apt operation to perform. ii) For TEAM-CL choosing which operation to perform is obtained based on the three-way prediction scores, given  $< q, a, (0/1) >$  as input. Since both of our proposed frameworks optimize for ranking loss, i.e., discriminates true candidate pairs from the negative ones — we get a ranked list of candidate anchors  $a$  while matching each of them with  $q$  via respective matching modules.
Here we give you an overview of our experiment settings and provide the detailed reproducibility information in the Sections C, D, E of the Appendix.