SlowGuess/ACL_2020_OCR huggingface dataset:
Chao Shang<sup>1</sup>, Sarthak Dash<sup>2</sup>, Md Faisal Mahbub Chowdhury<sup>2</sup>, Nandana Mihindukulasooriya<sup>2</sup>, Alfio Gliozzo<sup>2</sup>
<sup>1</sup>University of Connecticut, Storrs, CT, USA
$^{2}$ IBM Research AI, Yorktown Heights, NY, USA
chao.shang@uconn.edu, sdash@us.ibm.com
mchowdh@us.ibm.com, nandana.m@ibm.com, gliozzo@us.ibm.com
Extracting lexico-semantic relations as graph-structured taxonomies, also known as taxonomy construction, has been beneficial in a variety of NLP applications. Recently Graph Neural Network (GNN) has shown to be powerful in successfully tackling many tasks. However, there has been no attempt to exploit GNN to create taxonomies. In this paper, we propose Graph2Taxo, a GNN-based cross-domain transfer framework for the taxonomy construction task. Our main contribution is to learn the latent features of taxonomy construction from existing domains to guide the structure learning of an unseen domain. We also propose a novel method of directed acyclic graph (DAG) generation for taxonomy construction. Specifically, our proposed Graph2Taxo uses a noisy graph constructed from automatically extracted noisy hyponym-hypernym candidate pairs, and a set of taxonomies for some known domains for training. The learned model is then used to generate taxonomy for a new unknown domain given a set of terms for that domain. Experiments on benchmark datasets from science and environment domains show that our approach attains significant improvements correspondingly over the state of the art.
Taxonomy has been exploited in many Natural Language Processing (NLP) applications, such as question answering (Harabagiu et al., 2003), query understanding (Hua et al., 2017), recommendation systems (Friedrich and Zanker, 2011), etc. Automatic taxonomy construction is highly challenging as it involves the ability to recognize – (i) a set of types (i.e. hyPennyms) from a text corpus, (ii) instances (i.e. hyponyms) of each type, and (iii) is-a (i.e. hybernymy) hierarchy between types.
Existing taxonomies (e.g., WordNet (Miller et al., 1990)) are far from being complete. Tax-
onomies specific to many domains are either entirely absent or missing. In this paper, we focus on construction of taxonomies for such unseen domains<sup>1</sup>. Since taxonomies are expressed as directed acyclic graphs (DAGs) (Suchanek et al., 2008), taxonomy construction can be formulated as a DAG generation problem.
There has been considerable research on Graph Neural Networks (GNN) (Sperduti and Starita, 1997; Gori et al., 2005) over the years; particularly inspired by the convolutional GNN (Bruna et al., 2014) where graph convolution operations were defined in the Fourier domain. In a similar spirit to convolutional neural networks (CNNs), GNN methods aggregate neighboring information based on the connectivity of the graph to create node embeddings. GNN has been applied successfully in many tasks such as matrix completion (van den Berg et al., 2017), manifold analysis (Monti et al., 2017), predictions of community (Bruna et al., 2014), knowledge graph completion (Shang et al., 2019), and representations of network nodes (Hamilton et al., 2017; Kipf and Welling, 2017).
To the best of our knowledge, there has been no attempt to exploit GNN for taxonomy construction. Our proposed framework, Graph2Taxo, is the first to show that a GNN-based model using a cross-domain noisy graph can substantially improve the taxonomy construction of unseen domains (e.g., Environment) by exploiting taxonomy of one or more seen domains (e.g., Food). (The task is described in detail in Section 3.1.)
Another novelty of our approach is we are the first to apply the acyclicity constraint-based DAG structure learning model (Zheng et al., 2018; Yu et al., 2019) for taxonomy generation task.
The input of Graph2Taxo is a cross-domain
<sup>1</sup>By unseen domain, we refer to a domain for which taxonomy is not available to the system.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2198-2208 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
noisy graph constructed by connecting noisy candidate is-a pairs, which are extracted from a large corpus using standard linguistic pattern-based approaches (Hearst, 1992). It is noisy because pattern-based approaches are prone to poor coverage as well as wrong extractions. In addition, it is cross-domain because the noisy is-a pairs are extracted from a large-scale corpus which contains a collection of text from multiple domains.
Our proposed neural model directly encodes the structural information from a noisy graph into the embedding space. Since the links between domains are also used in our model, it has not only structural information of multiple domains but also cross-domain information.
We demonstrate effectiveness of our proposed method on science and environment datasets (Bordea et al., 2016), and show significant improvements on F-score over the state of the art.
Taxonomy construction (also known as taxonomy induction) is a well-studied problem. Most of the existing works follow two sequential steps to construct taxonomies from text corpora (Wang et al., 2017). First, is-a pairs are extracted using pattern-based or distributional methods. Then, a taxonomy is constructed from these is-a pairs.
The pattern-based methods, pioneered by Hearst (1992), detect is-a relation of a term pair  $(\mathrm{x},\mathrm{y})$  using the appearance of  $\mathbf{x}$  and  $\mathbf{y}$  in the same sentence through some lexical patterns or linguistic rules (Ritter et al., 2009; Luu et al., 2014). Snow et al. (2004) represented each  $(\mathrm{x},\mathrm{y})$  term-pair as the multiset of dependency paths connecting their co-occurrences in a corpus, which is also regarded as a path-based method.
An alternative approach for detecting is-a relation is the distributional methods (Baroni et al., 2012; Roller et al., 2014), using the distributional representation of terms to directly predict relations.
As for the step of taxonomy construction using the extracted is-a pairs, most of the approaches do it by incrementally attaching new terms (Snow et al., 2006; Shen et al., 2012; Alfarone and Davis, 2015; Wu et al., 2012). Mao et al. (2018) is the first to present a reinforcement learning based approach, named TaxoRL, for this task. For each term pair, its representation in TaxoRL is obtained by the path LSTM encoder, the word embeddings of both terms, and the embeddings of features.
Recently, Dash et al. (2020) argued that strict partial orders² correspond more directly to DAGs. They proposed a neural network architecture, called Strict Partial Order Network (SPON), that enforces asymmetry and transitive properties as soft constraints. Empirically, they showed that such a network produces better results for detecting hyponym-hypernym pairs on a number of datasets for different languages and domains in both supervised and unsupervised settings.
Many graph-based methods such as Kozareva and Hovy (2010) and Luu et al. (2014) regard the task of hypernymy organization as a hypernymy detection problem followed by a graph pruning problem. For the graph pruning task, various graph theoretic approaches such as optimal branching algorithm (Velardi et al., 2013), Edmond's algorithm (Karp, 1971) and Tarjan's algorithm (Tarjan, 1972) have been used over the years. In addition to these, Wang et al. (2017) mentions several other graph-based taxonomy induction approaches. In contrast, our approach formulates the taxonomy construction task as a DAG generation problem instead of an incremental taxonomy learning (Mao et al., 2018), which differentiates it when compared with the existing methods. In addition, our approach uses the knowledge from existing domains (Bansal et al., 2014; Gan et al., 2016) to build the taxonomies of missing domains.
3 The Graph2Taxo Framework
In this section, we first formulate the problem statement and then introduce our proposed Graph2Taxo framework as a solution. We describe the individual components of this framework in detail, along with justifications of how and why these components come together as a solution.
Figure 1: An illustration of our GNN-based cross-domain transfer framework for taxonomy construction.
${}^{2}$  A binary relation that is transitive,irreflexive and asymmetric.
The problem addressed in this paper is, given a list of domain-specific terms from a target unseen (aka missing) domain as input, how to construct a taxonomy for that target unseen domain. In other words, the problem addressed in this paper is how to organize these terms into a taxonomy.
This problem can be further abstracted out as follows: Given a large input corpus and a set of gold taxonomies  $G_{gold}$  from some known domains (different from the target domain), our task is to learn a model (trained using the corpus and taxonomies of known domains) to construct multiple taxonomies for the target unseen domains.
As a solution to the aforementioned problem, we propose a GNN-based cross-domain transfer framework for taxonomy construction (see Figure 1), called Graph2Taxo which consists of a cross-domain graph encoder and a DAG generator.
The first step in our proposed approach is to build a cross-domain noisy graph as an input to our Graph2Taxo model. In this step, we extract candidate  $is-a$  pairs from a large collection of input corpora that spans multiple domains. To do so, we used the output of Panchenko et al. (2016), which is a combination of standard substring matching and pattern-based approaches. Since such pattern-based approaches are too rigid, the corresponding output not only suffers from recall (i.e., missing  $is-a$  pairs) but also contains incorrect (i.e., noisy) pairs due to the ambiguity of language and richness in syntactic expression and structure in the input corpora. For example, consider the phrase "... animals other than dogs such as cats ..." As (Wu et al., 2012) noted, pattern-based approaches will extract (cat is-a dog) rather than (cat is-a animal).
Based on the noisy  $is-a$  pairs, we construct a directed graph  $G_{input} = (V_{input}, E_{input})$ , which is a cross-domain noisy graph. Here,  $V_{input}$  denotes a set of terms, and  $(v_i, v_j) \in E_{input}$  if and only if  $(v_i, v_j)$  belongs to the list of extracted noisy  $is-a$  pairs. The input document collection spans multiple domains, therefore  $E_{input}$  not only has intradomain edges, but also has cross-domain edges (see Figure 1).
Graph2Taxo is a subgraph generation model which uses the large cross-domain noisy graph as the input. Given a list of terms for a target unseen domain, it aims to learn a taxonomy structure for the corresponding domain as a DAG. Graph2Taxo takes advantage of additional knowl
edge in the form of previously known gold taxonomies  $\{G_{gold,i},1\leq i\leq N_{known}\}$  to train a learning model. During inference phase, the model receives a list of terms from the target unseen domain and aims to build a taxonomy by using the input terms. Here,  $N_{known}$  denotes the number of previously known taxonomies used during the training phase.
This problem of distilling directed acyclic substructures (taxonomies of many domains) using a large cross-domain noisy graph is challenging, because of relatively lower overlap between noisy edges in  $E_{input}$  and true edges in the available taxonomies in hand.
The following sections describe our proposed Cross-domain Graph Encoder and the DAG Generator in further detail.
3.2 Cross-domain Graph Encoder
This subsection describes the Cross-domain Graph Encoder in Figure 1 for embedding generation. This embedding generation algorithm uses two strategies, namely Neighborhood aggregation and Semantic clustering aggregation.
3.2.1 Neighborhood Aggregation
This is the first of the two strategies used for embedding generation. Let  $A \in \mathbb{R}^{n \times n}$  be the adjacency matrix of the noisy graph  $G_{input}$ , where  $n$  is the size of  $V_{input}$ . Let  $h_i^l$  represent the feature representation for the node  $v_i$  in the  $l$ -th layer and thus  $H^l \in \mathbb{R}^{n \times d_l}$  denotes the intermediate representation matrix. The initial matrix  $H^0$  is randomly initialized from a standard normal distribution.
We use the adjacency matrix  $A$  and the node representation matrix  $H^l$  to iteratively update the representation of a particular node by aggregating representations of its neighbors. This is done by using a GNN. Formally, a GNN layer (Gilmer et al., 2017; Hamilton et al., 2017; Xu et al., 2019) employs the general message-passing architecture which consists of a message propagation function  $M$  to get messages from neighbors and a vertex update function  $U$ . The message passing works via the following equations,
$$
m _ {v} ^ {l + 1} = M \left(h _ {u} ^ {l}\right) \quad \forall u \in \mathcal {N} (v)
$$
$$
h _ {v} ^ {l + 1} = U (h _ {v} ^ {l}, m _ {v} ^ {l + 1})
$$
where  $\mathcal{N}(v)$  denotes the neighbors of node  $v$  and  $m$  is the message. In addition, we use the following
definitions for  $M$  and  $U$  functions,
$$
M \left(h _ {u} ^ {l}\right) = \sum_ {u \in \mathcal {N} (v)} A _ {v u} h _ {u} ^ {l}, \forall u \in \mathcal {N} (v)
$$
$$
U (h _ {v} ^ {l}, m _ {v} ^ {l + 1}) = \sigma (m _ {v} ^ {l + 1} \Theta^ {l} + h _ {v} ^ {l} \Theta^ {l})
$$
where  $\Theta^l\in \mathbb{R}^{d_l\times d_{l + 1}}$  denotes trainable parameters for layer  $l$  and  $\sigma$  represents an activation function.
Let  $\tilde{A} = A + I$ , here  $I$  is the identity matrix, the information aggregation strategy described above can be abstracted out as,
$$
H ^ {l + 1} = G N N _ {l} (A, H ^ {l}) = \sigma (\tilde {A} H ^ {l} \Theta^ {l})
$$
3.2.2 Semantic Clustering Aggregation
This is the second of the two strategies used for embedding generation, which operates on the output of the previous step. The learned representations from the previous step are highly likely not to be uniformly distributed in the Euclidean Space, but rather form a bunch of clusters. In this regard, we propose a soft clustering-based pooling-unpooling step, that uses semantic clustering aggregating for learning better model representations. In essence, this step shares the similarity information for any pair of terms in the vocabulary.
Analogous to an auto-encoder, the pooling layer adaptively creates a smaller cluster graph comprising of a set of cluster nodes, whose representations are learned based on a trainable cluster assignment matrix. This idea of using an assignment matrix was first proposed by the DiffPool (Ying et al., 2018) approach. On the other hand, the un-pooling layer decodes the cluster graph into the original graph using the same cluster assignment matrix learned in the pooling layer. The learned semantic cluster nodes can be thought of as "bridges" between nodes from the same or different clusters to pass messages.
Mathematically speaking, we learn a soft cluster assignment matrix  $S^l \in \mathbb{R}^{n \times n_c}$  at layer  $l$  using the GNN model, where  $n_c$  is the number of clusters. Each row in  $S^l$  corresponds to one of  $n$  nodes in layer  $l$  and each column corresponds to one of the  $n_c$  clusters. As a first step, the pooling layer uses the adjacency matrix  $A$  and the node feature matrix  $H^l$  to generate a soft cluster assignment matrix as,
$$
S ^ {l} = \operatorname {s o f t m a x} \left(G N N _ {l, c l u s t e r} \left(A, H ^ {l}\right)\right) \quad (1)
$$
where the softmax is a row-wise softmax function,  $\Theta_{cluster}^{l}\in \mathbb{R}^{d_{l}\times n_{c}}$  denotes all trainable parameters in  $GNN_{l,cluster}$ .
Since the matrix  $S^l$  is calculated based on node embeddings, nodes with similar features and local structure will have similar cluster assignment.
As the final step, the pooling layer generates an adjacency matrix  $A_{c}$  for the cluster graph and a new embedding matrix containing cluster node representations  $H_{c}^{l}$  as follows,
$$
H _ {c} ^ {l} = \left(S ^ {l}\right) ^ {T} H ^ {l} \in \mathbb {R} ^ {n _ {c} \times d _ {l}}
$$
$$
A _ {c} = \left(S ^ {l}\right) ^ {T} A S ^ {l} \in \mathbb {R} ^ {n _ {c} \times n _ {c}}
$$
A GNN operation is used within the small cluster graph,
$$
H _ {c} ^ {l + 1} = G N N _ {l} \left(A _ {c}, H _ {c} ^ {l}\right) \in \mathbb {R} ^ {n _ {c} \times d _ {l + 1}}
$$
to further propagate messages from the neighboring clusters. The trainable parameters in  $GNN_{l}$  are  $\Theta^{l}\in \mathbb{R}^{d_{l}\times d_{l + 1}}$ .
For passing clustering information to the original graph, the unpooling layer restores the original graph using cluster assignment matrix, as follows,
$$
\tilde {H} ^ {l + 1} = S ^ {l} H _ {c} ^ {l + 1} \in \mathbb {R} ^ {n \times d _ {l + 1}}
$$
The output of the pooling-unpooling layer results in the node representations possessing latent cluster information. Finally, we combine the neighborhood aggregation and semantic clustering aggregation strategies via a residual connection, as,
$$
H ^ {l + 1} = \operatorname {c o n c a t e} \left(\tilde {H} ^ {l + 1}, H ^ {l}\right)
$$
where  $\text{concat}$  means concatenate the two matrices.  $H^{l+1}$  is the output of this pooling-unpooling step.
Figure 2: An illustration of DAG generator.
3.3 DAG Generator
The DAG generator takes in the noisy graph  $G_{input}$  and representations of all the vocabulary terms (output of Section 3.2) as input, encodes acyclicity as
a soft-constraint (as described below), and outputs a distribution of edges within  $G_{input}$  that encodes the likelihood of true is-a relationships. This output distribution is finally used to induce taxonomies, i.e., DAGs of is-a relationships.
In each training step, DAG generator is applied to one domain (see Figure 2), using a noisy graph  $G$ , which is a subgraph from  $G_{input}$ , as a training sample and a DAG is generated for that domain. Here let  $N_{t}$  denote the number of (hypo, hyper) pairs belonging to the edge set of  $G$ . During the training, we also know label vector  $label \in \{0,1\}^{N_t}$  for these  $N_{t}$  pairs, based on whether they belong to the gold known taxonomy.
3.3.1 Edge Prediction
For each edge within the noisy graph  $G$ , our DAG generator estimates the probability that the edge represents a valid hypernymy relationship. Our model estimates this probability through the use of a convolution operation illustrated in Figure 2.
For each edge (hypo, hyper), in the first step the term embeddings and edge features are concatenated as follows,
$$
v _ {p a i r} = \text {c o n c a t e} (v _ {\text {h y p o}}, v _ {\text {h y p e r}}, v _ {\text {f e a s}})
$$
where  $v_{hypo}$  and  $v_{hyper}$  are the embeddings for hypo and hyper nodes (from Section 3.2) and  $v_{feas}$  denotes a feature vector for the edge (hypo, hyper), which includes edge frequency and substring features. The substring features include ends with, contains, prefix match, suffix match, length of longest common substring (LCS), length difference and a boolean feature denoting whether LCS in  $V_{input}$  (the set of terms) or not.
Inspired by ConvE model (Dettmers et al., 2018), a well known convolution based algorithm for link prediction, we apply a 1D convolution operation on  $v_{pair}$ . We use a convolution operation since it increases the expressiveness of the DAG Generator through additional interaction between participating embeddings.
For the convolution operation, we make use of C different kernels parameterized by  $\{w_{c},1\leq c\leq$ $C\}$  . The 1D convolution operation is then calculated as follows,
$$
v _ {c} = \left[ U _ {c} \left(v _ {\text {p a i r}}, 0\right), \dots , U _ {c} \left(v _ {\text {p a i r}}, d _ {v} - 1\right) \right] \tag {2}
$$
$$
U _ {c} \left(v _ {\text {p a i r}}, p\right) = \sum_ {\tau = 0} ^ {K - 1} \omega_ {c} (\tau) \hat {v} _ {\text {p a i r}} (p + \tau)) \tag {3}
$$
where  $K$  denotes the kernel width,  $d_v$  denotes the size of  $v_{pair}$ ,  $p$  denotes the position to start the kernel operation and the kernel parameters  $\omega_c$  are trainable. In addition,  $\hat{v}_{pair}$  denotes the padded version of  $v_{pair}$ , wherein the padding strategy is as follows. If  $|K|$  is odd, we pad  $v_{pair}$  with  $\lfloor K / 2\rfloor$  zeros on both the sides. On the other hand, if  $|K|$  is even, we pad  $\lfloor K / 2\rfloor -1$  zeros at the beginning, and  $\lfloor K / 2\rfloor$  zeros at the end of  $v_{pair}$ . Here,  $\lfloor value\rfloor$  returns the floor of value.
Each kernel  $c$  generates a vector  $v_{c}$ , according to Equation 2. As there are  $C$  different kernels, this results in the generation of  $C$  different vectors which are then concatenated together to form one vector  $V_{C}$ , i.e.  $V_{C} = \text{concatenate}(v_{0}, v_{1}, \ldots, v_{C})$ .
The probability  $p_{(hypo, hyper)}$  of a given edge (hypo, hyper) expressing a hypernymy relationship can then be estimated using the following scoring function,
$$
p _ {(h y p o, h y p e r)} = \operatorname {s i g m o i d} \left(V _ {C} ^ {T} W\right) \tag {4}
$$
where  $W$  denotes the parameter matrix of a fully connected layer, as illustrated in Figure 2.
Finally, for the loss calculations, we make use of differentiable F1 loss (Huang et al., 2015),
$$
\text {P r e c i s i o n} = \frac {\sum_ {t = 0} ^ {N _ {t} - 1} p _ {t} \times \text {l a b e l} _ {t}}{\sum_ {t = 0} ^ {N _ {t} - 1} p _ {t}}
$$
$$
\text {R e c a l l} = \frac {\sum_ {t = 0} ^ {N _ {t} - 1} p _ {t} \times \text {l a b e l} _ {t}}{\sum_ {t = 0} ^ {N _ {t} - 1} \text {l a b e l} _ {t}}
$$
$$
L _ {F 1} = \frac {2 \times \text {P r e c i s i o n} \times \text {R e c a l l}}{\text {P r e c i s i o n} + \text {R e c a l l}}
$$
3.3.2 DAG Constraint
The edge prediction step alone does not guarantee that the generated graph is acyclic. Learning DAG from data is an NP-hard problem (Chickering, 1995; Chickering et al., 2004). To this effect, one of the first works that formulate the acyclic structure learning task as a continuous optimization problem was introduced by Zheng et al. (2018).
In that paper, the authors note that the trace of  $B^{k}$  denoted by  $tr(B^{k})$ , for a non-negative adjacency matrix  $B \in \mathbb{R}^{n \times n}$  counts the number of length- $k$  cycles in a directed graph. Hence, positive entries within the diagonal of  $B^{k}$  suggests the existence of cycles. Or, in other words, B has no cycle if and only if  $\sum_{k=1}^{\infty} \sum_{i=1}^{n} (B^{k})_{ii} = 0$ .
However, calculating  $B^{k}$  for every value of  $k$ , i.e. repeated matrix exponentiation, is impractical and can easily exceed machine precision. To solve this
problem, Zheng et al. (2018) makes use of Taylor Series expansion as  $e^{B} = \sum_{k=0}^{\infty} \frac{B^{k}}{k!}$ , and show that a non-negative matrix  $B$  is a DAG iff,
$$
\sum_ {k = 1} ^ {\infty} \sum_ {i = 1} ^ {n} \frac {(B ^ {k}) _ {i i}}{k !} = t r (e ^ {B}) - n = 0
$$
To make sure this constraint is useful for an arbitrary weighted matrix with both positive and negative values, a Hadamard product  $B = A \circ A$  is used, which leads us to the following theorem.
Theorem 1 (Zheng et al., 2018) A matrix  $A \in \mathbb{R}^{n \times n}$  is a DAG if and only if:
$$
t r (e ^ {A \circ A}) - n = 0
$$
where tr represents the trace of a matrix,  $\circ$  represents the Hadamard product and  $e^B$  equals matrix exponential of  $B$ .
Since the matrix exponential may not be available in all deep learning frameworks, (Yu et al., 2019) propose an alternative constraint that is practically convenient as follows.
Lemma 2 (Yu et al., 2019) Let  $\alpha = c / m > 0$  for some  $c$ . For any complex  $\lambda$ , since  $(1 + \alpha |\lambda|)^m \leq e^{c|\lambda|}$ , the DAG constraint from Theorem 1 can be relaxed and stated as follows,
$$
h (A) = t r \big [ (I + \alpha A \circ A) ^ {n} \big ] - n = 0
$$
where  $\alpha$  is a hyper-parameter.
Finally, using an augmented Lagrangian approach, we propose the combined loss function,
$$
L = L _ {F 1} + \lambda h (A) + \frac {\rho}{2} h (A) ^ {2}
$$
where  $\lambda$  and  $\rho$  are the hyper-parameters. During the backpropagation, the gradients will be passed back to all domains through the intra-domain and cross-domain edges from  $G_{input}$  to update all parameters.