Venue: ACL 2020

# Taxonomy Construction of Unseen Domains via Graph-based

Cross-Domain Knowledge Transfer

 Chao Shang\({}^{1}\), Sarthak Dash\({}^{2}\), Md Faisal Mahbub Chowdhury\({}^{2}\),

Nandana Mihindukulasooriya\({}^{2}\), Alfio Gliozzo\({}^{2}\)

\({}^{1}\)University of Connecticut, Storrs, CT, USA

\({}^{2}\)IBM Research AI, Yorktown Heights, NY, USA

chao.shang@uconn.edu, sdash@us.ibm.com mchowdh@us.ibm.com, nandana.m@ibm.com, gliozzo@us.ibm.com

###### Abstract

Extracting lexico-semantic relations as graph-structured taxonomies, also known as taxonomy construction, has been beneficial in a variety of NLP applications. Recently Graph Neural Network (GNN) has shown to be powerful in successfully tackling many tasks. However, there has been no attempt to exploit GNN to create taxonomies. In this paper, we propose _Graph2Taxo_, a GNN-based cross-domain transfer framework for the taxonomy construction task. Our main contribution is to learn the latent features of taxonomy construction from existing domains to guide the structure learning of an unseen domain. We also propose a novel method of directed acyclic graph (DAG) generation for taxonomy construction. Specifically, our proposed _Graph2Taxo_ uses a noisy graph constructed from automatically extracted noisy hyponym-hypernym candidate pairs, and a set of taxonomies for some known domains for training. The learned model is then used to generate taxonomy for a new unknown domain given a set of terms for that domain. Experiments on benchmark datasets from science and environment domains show that our approach attains significant improvements correspondingly over the state of the art.

+
Footnote †: July 5 - 10, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

Taxonomy has been exploited in many Natural Language Processing (NLP) applications, such as question answering Harabagiu et al. (2003), query understanding Hua et al. (2017), recommendation systems Friedrich and Zanker (2011), etc. Automatic taxonomy construction is highly challenging as it involves the ability to recognize - (i) a set of types (i.e. hypernyms) from a text corpus, (ii) instances (i.e. hyponyms) of each type, and (iii) _is-a_ (i.e. hypernymy) hierarchy between types.

Existing taxonomies (e.g., WordNet Miller et al. (1990)) are far from being complete. Taxonomies specific to many domains are either entirely absent or missing. In this paper, we focus on construction of taxonomies for such _unseen domains_1. Since taxonomies are expressed as _directed acyclic graphs_ (DAGs) Suchanek et al. (2008), taxonomy construction can be formulated as a DAG generation problem.

Footnote 1: By **unseen domain**, we refer to a domain for which taxonomy is not available to the system.

There has been considerable research on Graph Neural Networks (GNN) Sperduti and Starita (1997); Gori et al. (2005) over the years; particularly inspired by the convolutional GNN Bruna et al. (2014) where graph convolution operations were defined in the Fourier domain. In a similar spirit to convolutional neural networks (CNNs), GNN methods aggregate neighboring information based on the connectivity of the graph to create node embeddings. GNN has been applied successfully in many tasks such as matrix completion van den Berg et al. (2017), manifold analysis Monti et al. (2017), predictions of community Bruna et al. (2014), knowledge graph completion Shang et al. (2019), and representations of network nodes Hamilton et al. (2017); Kipf and Welling (2017).

To the best of our knowledge, there has been no attempt to exploit GNN for taxonomy construction. Our proposed framework, _Graph2Taxo_, is the first to show that a GNN-based model using a cross-domain noisy graph can substantially improve the taxonomy construction of unseen domains (e.g., Environment) by exploiting taxonomy of one or more seen domains (e.g., Food). (The task is described in detail in Section 3.1.)

Another novelty of our approach is we are the first to apply the acyclicity constraint-based DAG structure learning model Zheng et al. (2018); Yu et al. (2019) for taxonomy generation task.

The input of _Graph2Taxo_ is a _cross-domain

noisy graph_ constructed by connecting noisy candidate _is-a_ pairs, which are extracted from a large corpus using standard linguistic pattern-based approaches Hearst (1992). It is noisy because pattern-based approaches are prone to poor coverage as well as wrong extractions. In addition, it is cross-domain because the noisy _is-a_ pairs are extracted from a large-scale corpus which contains a collection of text from multiple domains.

Our proposed neural model directly encodes the structural information from a noisy graph into the embedding space. Since the links between domains are also used in our model, it has not only structural information of multiple domains but also cross-domain information.

We demonstrate effectiveness of our proposed method on _science_ and _environment_ datasets Bordea et al. (2016), and show significant improvements on F-score over the state of the art.

## 2 Related Work

Taxonomy construction (also known as taxonomy induction) is a well-studied problem. Most of the existing works follow two sequential steps to construct taxonomies from text corpora Wang et al. (2017). First, _is-a_ pairs are extracted using pattern-based or distributional methods. Then, a taxonomy is constructed from these _is-a_ pairs.

The pattern-based methods, pioneered by Hearst (1992), detect _is-a_ relation of a term pair (x, y) using the appearance of x and y in the same sentence through some lexical patterns or linguistic rules Ritter et al. (2009); Luu et al. (2014). Snow et al. (2004) represented each (x, y) term-pair as the multiset of dependency paths connecting their co-occurrences in a corpus, which is also regarded as a path-based method.

An alternative approach for detecting _is-a_ relation is the distributional methods Baroni et al. (2012); Roller et al. (2014), using the distributional representation of terms to directly predict relations.

As for the step of taxonomy construction using the extracted _is-a_ pairs, most of the approaches do it by incrementally attaching new terms Snow et al. (2006); Shen et al. (2012); Alfarone and Davis (2015); Wu et al. (2012). Mao et al. (2018) is the first to present a reinforcement learning based approach, named _TaxoRL_, for this task. For each term pair, its representation in TaxoRL is obtained by the path LSTM encoder, the word embeddings of both terms, and the embeddings of features.

Recently, Dash et al. (2020) argued that _strict partial orders2_ correspond more directly to _DAGs_. They proposed a neural network architecture, called Strict Partial Order Network (SPON), that enforces asymmetry and transitive properties as soft constraints. Empirically, they showed that such a network produces better results for detecting hyponym-hypernym pairs on a number of datasets for different languages and domains in both supervised and unsupervised settings.

Footnote 2: A binary relation that is transitive, irreflexive and asymmetric.

Many graph-based methods such as Kozareva and Hovy (2010) and Luu et al. (2014) regard the task of hypernymy organization as a hypernymy detection problem followed by a graph pruning problem. For the graph pruning task, various graph-theoretic approaches such as _optimal branching algorithm_ Velardi et al. (2013), _Edmond's algorithm_ Karp (1971) and _Tarjan's algorithm_ Tarjan (1972) have been used over the years. In addition to these, Wang et al. (2017) mentions several other graph-based taxonomy induction approaches. In contrast, our approach formulates the taxonomy construction task as a DAG generation problem instead of an incremental taxonomy learning Mao et al. (2018), which differentiates it when compared with the existing methods. In addition, our approach uses the knowledge from existing domains Bansal et al. (2014); Gan et al. (2016) to build the taxonomies of missing domains.

## 3 The Graph2Taxo Framework

In this section, we first formulate the problem statement and then introduce our proposed _Graph2Taxo_ framework as a solution. We describe the individual components of this framework in detail, along with justifications of _how_ and _why_ these components come together as a solution.

Figure 1: An illustration of our GNN-based cross-domain transfer framework for taxonomy construction.

 



### Problem Definition

The problem addressed in this paper is, given a list of domain-specific terms from a target _unseen_ (aka missing) domain as input, how to construct a taxonomy for that target _unseen_ domain. In other words, the problem addressed in this paper is how to organize these terms into a taxonomy.

This problem can be further abstracted out as follows: Given a large _input corpus_ and a set of gold taxonomies \(G_{gold}\) from some known domains (different from the _target_ domain), our task is to learn a model (trained using the corpus and taxonomies of known domains) to construct multiple taxonomies for the _target_ unseen domains.

As a solution to the aforementioned problem, we propose a GNN-based cross-domain transfer framework for taxonomy construction (see Figure 1), called _Graph2Taxo_ which consists of a cross-domain graph encoder and a DAG generator.

The first step in our proposed approach is to build a _cross-domain noisy graph_ as an input to our _Graph2Taxo_ model. In this step, we extract candidate _is-a_ pairs from a large collection of input corpora that spans multiple domains. To do so, we used the output of Panchenko et al. (2016), which is a combination of standard substring matching and pattern-based approaches. Since such pattern-based approaches are too rigid, the corresponding output not only suffers from recall (i.e., missing _is-a_ pairs) but also contains incorrect (i.e., noisy) pairs due to the ambiguity of language and richness in syntactic expression and structure in the input corpora. For example, consider the phrase _"... animals other than dogs such as cats ..."_. As (Wu et al., 2012) noted, pattern-based approaches will extract _(cat is-a dog)_ rather than _(cat is-a animal)_.

Based on the noisy _is-a_ pairs, we construct a directed graph \(G_{input}=(V_{input},E_{input})\), which is a _cross-domain noisy graph_. Here, \(V_{input}\) denotes a set of terms, and \((v_{i},v_{j})\in E_{input}\) if and only if \((v_{i},v_{j})\) belongs to the list of extracted noisy _is-a_ pairs. The input document collection spans multiple domains, therefore \(E_{input}\) not only has intra-domain edges, but also has cross-domain edges (see Figure 1).

_Graph2Taxo_ is a subgraph generation model which uses the large cross-domain noisy graph as the input. Given a list of terms for a target _unseen_ domain, it aims to learn a taxonomy structure for the corresponding domain as a DAG. _Graph2Taxo_ takes advantage of additional knowledge in the form of previously known gold taxonomies \(\{G_{gold,i},1\leq i\leq N_{known}\}\) to train a learning model. During _inference_ phase, the model receives a list of terms from the target _unseen_ domain and aims to build a taxonomy by using the input terms. Here, \(N_{known}\) denotes the number of previously known taxonomies used during the training phase.

This problem of distilling directed acyclic substructures (taxonomies of many domains) using a large cross-domain noisy graph is challenging, because of relatively lower overlap between noisy edges in \(E_{input}\) and true edges in the available taxonomies in hand.

The following sections describe our proposed Cross-domain Graph Encoder and the DAG Generator in further detail.

### Cross-domain Graph Encoder

This subsection describes the _Cross-domain Graph Encoder_ in Figure 1 for embedding generation. This embedding generation algorithm uses two strategies, namely _Neighborhood aggregation_ and _Semantic clustering aggregation_.

#### 3.2.1 Neighborhood Aggregation

This is the first of the two strategies used for embedding generation. Let \(A\in\mathbb{R}^{n\times n}\) be the adjacency matrix of the noisy graph \(G_{input}\), where \(n\) is the size of \(V_{input}\). Let \(h_{i}^{l}\) represent the feature representation for the node \(v_{i}\) in the \(l\)-th layer and thus \(H^{l}\in\mathbb{R}^{n\times d_{l}}\) denotes the intermediate representation matrix. The initial matrix \(H^{0}\) is randomly initialized from a standard normal distribution.

We use the adjacency matrix \(A\) and the node representation matrix \(H^{l}\) to iteratively update the representation of a particular node by aggregating representations of its neighbors. This is done by using a GNN. Formally, a GNN layer Gilmer et al. (2017); Hamilton et al. (2017); Xu et al. (2019) employs the general message-passing architecture which consists of a message propagation function \(M\) to get messages from neighbors and a vertex update function \(U\). The message passing works via the following equations,

\[m_{v}^{l+1} =M(h_{u}^{l})~{}~{}~{}\forall u\in\mathcal{N}(v)\] \[h_{v}^{l+1} =U(h_{v}^{l},m_{v}^{l+1})\]

where \(\mathcal{N}(v)\) denotes the neighbors of node \(v\) and \(m\) is the message. In addition, we use the following

definitions for \(M\) and \(U\) functions,

\[M(h_{u}^{l})=\sum_{u\in\mathcal{N}(v)}A_{cu}h_{u}^{l},\forall u\in \mathcal{N}(v)\] \[U(h_{v}^{l},m_{v}^{l+1})=\sigma(m_{v}^{l+1}\Theta^{l}+h_{v}^{l} \Theta^{l})\]

where \(\Theta^{l}\in\mathbb{R}^{d_{l}\times d_{l+1}}\) denotes trainable parameters for layer \(l\) and \(\sigma\) represents an activation function.

Let \(\tilde{A}=A+I\), here \(I\) is the identity matrix, the information aggregation strategy described above can be abstracted out as,

\[H^{l+1}=GNN_{l}(A,H^{l})=\sigma(\tilde{A}H^{l}\Theta^{l})\]

#### 3.2.2 Semantic Clustering Aggregation

This is the second of the two strategies used for embedding generation, which _operates_ on the output of the previous step. The learned representations from the previous step are highly likely not to be uniformly distributed in the Euclidean Space, but rather form a bunch of clusters. In this regard, we propose a soft clustering-based pooling-unpooling step, that uses semantic clustering aggregating for learning better model representations. In essence, this step shares the similarity information for any pair of terms in the vocabulary.

Analogous to an auto-encoder, the pooling layer adaptively creates a smaller _cluster_ graph comprising of a set of cluster nodes, whose representations are learned based on a trainable cluster assignment matrix. This idea of using an assignment matrix was first proposed by the _DiffPool_(Ying et al., 2018) approach. On the other hand, the unpooling layer decodes the _cluster_ graph into the original graph using the same cluster assignment matrix learned in the pooling layer. The learned semantic cluster nodes can be thought of as "bridges" between nodes from the same or different clusters to pass messages.

Mathematically speaking, we learn a soft cluster assignment matrix \(S^{l}\in\mathbb{R}^{n\times n_{c}}\) at layer \(l\) using the GNN model, where \(n_{c}\) is the number of clusters. Each row in \(S^{l}\) corresponds to one of \(n\) nodes in layer \(l\) and each column corresponds to one of the \(n_{c}\) clusters. As a first step, the pooling layer uses the adjacency matrix \(A\) and the node feature matrix \(H^{l}\) to generate a soft cluster assignment matrix as,

\[S^{l}=softmax(GNN_{l,cluster}(A,H^{l})) \tag{1}\]

where the \(softmax\) is a row-wise softmax function, \(\Theta^{l}_{cluster}\in\mathbb{R}^{d_{l}\times n_{c}}\) denotes all trainable parameters in \(GNN_{l,cluster}\).

Since the matrix \(S^{l}\) is calculated based on node embeddings, nodes with similar features and local structure will have similar cluster assignment.

As the final step, the pooling layer generates an adjacency matrix \(A_{c}\) for the _cluster_ graph and a new embedding matrix containing cluster node representations \(H_{c}^{l}\) as follows,

\[H_{c}^{l}=(S^{l})^{T}H^{l}\in\mathbb{R}^{n_{c}\times d_{l}}\] \[A_{c}=(S^{l})^{T}AS^{l}\in\mathbb{R}^{n_{c}\times n_{c}}\]

A GNN operation is used within the small cluster graph,

\[H_{c}^{l+1}=GNN_{l}(A_{c},H_{c}^{l})\in\mathbb{R}^{n_{c}\times d_{l+1}}\]

to further propagate messages from the neighboring clusters. The trainable parameters in \(GNN_{l}\) are \(\Theta^{l}\in\mathbb{R}^{d_{l}\times d_{l+1}}\).

For passing clustering information to the original graph, the unpooling layer restores the original graph using cluster assignment matrix, as follows,

\[\tilde{H}^{l+1}=S^{l}H_{c}^{l+1}\in\mathbb{R}^{n\times d_{l+1}}\]

The output of the pooling-unpooling layer results in the node representations possessing latent cluster information. Finally, we combine the neighborhood aggregation and semantic clustering aggregation strategies via a residual connection, as,

\[H^{l+1}=concate(\tilde{H}^{l+1},H^{l})\]

where \(concate\) means concatenate the two matrices. \(H^{l+1}\) is the output of this pooling-unpooling step.

### DAG Generator

The _DAG generator_ takes in the noisy graph \(G_{input}\) and representations of all the vocabulary terms (output of Section 3.2) as input, encodes _acyclicity_ as

Figure 2: An illustration of DAG generator.

 

a soft-constraint (as described below), and outputs a distribution of edges within that encodes the likelihood of true _is-a_ relationships. This output distribution is finally used to induce taxonomies, i.e., DAGs of _is-a_ relationships.

In each training step, _DAG generator_ is applied to one domain (see Figure 2), using a noisy graph \(G\), which is a subgraph from \(G_{input}\), as a training sample and a DAG is generated for that domain. Here let \(N_{t}\) denote the number of \((hypo,hyper)\) pairs belonging to the edge set of \(G\). During the training, we also know label vector \(label\in\{0,1\}^{N_{t}}\) for these \(N_{t}\) pairs, based on whether they belong to the gold known taxonomy.

#### 3.3.1 Edge Prediction

For each edge within the noisy graph \(G\), our DAG generator estimates the probability that the edge represents a valid _hypernymy_ relationship. Our model estimates this probability through the use of a convolution operation illustrated in Figure 2.

For each edge \((hypo,hyper)\), in the first step the term embeddings and edge features are concatenated as follows,

\[v_{pair}=concate(v_{hypo},v_{hyper},v_{feas})\]

where \(v_{hypo}\) and \(v_{hyper}\) are the embeddings for \(hypo\) and \(hyper\) nodes (from Section 3.2) and \(v_{feas}\) denotes a feature vector for the edge \((hypo,hyper)\), which includes edge frequency and substring features. The substring features includes _ends with_, _contains_, _prefix match_, _suffix match_, _length of longest common substring (LCS)_, _length difference_ and a boolean feature denoting whether _LCS in \(V_{input}\) (the set of terms) or not_.

Inspired by ConvE model Dettmers et al. (2018), a well known convolution based algorithm for link prediction, we apply a 1D convolution operation on \(v_{pair}\). We use a convolution operation since it increases the expressiveness of the DAG Generator through additional interaction between participating embeddings.

For the convolution operation, we make use of C different kernels parameterized by \(\{w_{c},1\leq c\leq C\}\). The 1D convolution operation is then calculated as follows,

\[v_{c}=[U_{c}(v_{pair},0),...,U_{c}(v_{pair},d_{v}-1)] \tag{2}\] \[U_{c}(v_{pair},p)=\sum_{\tau=0}^{K-1}\omega_{c}(\tau)\hat{v}_{ pair}(p+\tau)) \tag{3}\]

where \(K\) denotes the kernel width, \(d_{v}\) denotes the size of \(v_{pair}\), \(p\) denotes the position to start the kernel operation and the kernel parameters \(\omega_{c}\) are trainable. In addition, \(\hat{v}_{pair}\) denotes the padded version of \(v_{pair}\), wherein the padding strategy is as follows. If \(|K|\) is odd, we pad \(v_{pair}\) with \(\lfloor K/2\rfloor\) zeros on both the sides. On the other hand, if \(|K|\) is even, we pad \(\lfloor K/2\rfloor-1\) zeros at the beginning, and \(\lfloor K/2\rfloor\) zeros at the end of \(v_{pair}\). Here, \(\lfloor value\rfloor\) returns the floor of \(value\).

Each kernel \(c\) generates a vector \(v_{c}\), according to Equation 2. As there are \(C\) different kernels, this results in the generation of \(C\) different vectors which are then concatenated together to form one vector \(V_{C}\), i.e. \(V_{C}=concatenate(v_{0},v_{1},\ldots,v_{C})\).

The probability \(p_{(hypo,hyper)}\) of a given edge \((hypo,hyper)\) expressing a _hypernymy_ relationship can then be estimated using the following scoring function,

\[p_{(hypo,hyper)}=sigmod(V_{C}^{T}W) \tag{4}\]

where \(W\) denotes the parameter matrix of a fully connected layer, as illustrated in Figure 2.

Finally, for the loss calculations, we make use of differentiable F1 loss Huang et al. (2015),

Precision \[=\frac{\sum_{t=0}^{N_{t}-1}p_{t}\times label_{t}}{\sum_{t=0}^{N_{ t}-1}p_{t}}\] Recall \[=\frac{\sum_{t=0}^{N_{t}-1}p_{t}\times label_{t}}{\sum_{t=0}^{N_{ t}-1}label_{t}}\] \[L_{F1}=\frac{2\times\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}}\]

#### 3.3.2 DAG Constraint

The edge prediction step alone does not guarantee that the generated graph is acyclic. Learning DAG from data is an NP-hard problem Chickering (1995); Chickering et al. (2004). To this effect, one of the first works that formulate the acyclic structure learning task as a continuous optimization problem was introduced by Zheng et al. (2018).

In that paper, the authors note that the trace of \(B^{k}\) denoted by \(tr(B^{k})\), for a non-negative adjacency matrix \(B\in\mathbb{R}^{n\times n}\) counts the number of length-k cycles in a directed graph. Hence, positive entries within the diagonal of \(B^{k}\) suggests the existence of cycles. Or, in other words, B has no cycle if and only if \(\sum_{k=1}^{\infty}\sum_{i=1}^{n}(B^{k})_{ii}=0\).

However, calculating \(B^{k}\) for every value of \(k\), i.e. repeated matrix exponentiation, is impractical and can easily exceed machine precision. To solve this 

problem, Zheng et al. (2018) makes use of Taylor Series expansion as \(e^{B}=\sum_{k=0}^{\infty}\frac{B^{k}}{k!}\), and show that a non-negative matrix B is a DAG ifl,

\[\sum_{k=1}^{\infty}\sum_{i=1}^{n}\frac{(B^{k})_{ii}}{k!}=tr(e^{B})-n=0\]

To make sure this constraint is useful for an arbitrary weighted matrix with both positive and negative values, a Hadamard product \(B=A\circ A\) is used, which leads us to the following theorem.

**Theorem 1**: _(Zheng et al., 2018) A matrix \(A\in\mathbb{R}^{n\times n}\) is a DAG if and only if:_

\[tr(e^{A\circ A})-n=0\]

_where \(tr\) represents the trace of a matrix, \(\circ\) represents the Hadamard product and \(e^{B}\) equals matrix exponential of B._

Since the matrix exponential may not be available in all deep learning frameworks, Yu et al. (2019) propose an alternative constraint that is practically convenient as follows.

**Lemma 2**: _(Yu et al., 2019) Let \(\alpha=c/m>0\) for some \(c\). For any complex \(\lambda\), since \((1+\alpha|\lambda|)^{m}\leq e^{c|\lambda|}\), the DAG constraint from Theorem 1 can be relaxed and stated as follows,_

\[h(A)=tr\big{[}(I+\alpha A\circ A)^{n}\big{]}-n=0\]

_where \(\alpha\) is a hyper-parameter._

Finally, using an augmented Lagrangian approach, we propose the combined loss function,

\[L=L_{F1}+\lambda h(A)+\frac{\rho}{2}h(A)^{2}\]

where \(\lambda\) and \(\rho\) are the hyper-parameters. During the backpropagation, the gradients will be passed back to all domains through the intra-domain and cross-domain edges from \(G_{input}\) to update all parameters.

## 4 Experiments

We evaluate _Graph2Taxo_ on _Semeval-2016 Task 13: Taxonomy Extraction Evaluation3_, otherwise known as _TEXEval-2 task_(Bordea et al., 2016). All experiments are implemented in PyTorch. Code is publicly available at [https://github.com/IBM/gnn-taxo-construction](https://github.com/IBM/gnn-taxo-construction).

Footnote 3: Semeval-2016 Task 13: [http://alt.qcri.org/semeval2016/task13](http://alt.qcri.org/semeval2016/task13)

### Benchmark Datasets

For experiments, we used the English _environment_ and the _science_ taxonomies within the _TExEval-2_ benchmark datasets. These datasets do not come with any training data, but a list of terms and the task is to build a meaningful taxonomy using these terms. The _science_ domain terms come from _Wordnet_, _Eurovoc_ and a manually constructed taxonomy (henceforth referred to as _combined_), whereas the terms for _environment_ domain comes from _Eurovoc_ taxonomy only. Table 1 shows the dataset statistics.

We chose to evaluate our proposed approach on _environment_ and _science_ taxonomies only, because we wanted to compare our approach with the existing state-of-the-art system named _TaxoRL_(Mao et al., 2018) as well as with _TAXI_, the winning system in the _TEXEval-2_ task. Note that we use the same datasets with _TaxoRL_(Mao et al., 2018) for _TEXEval-2_ task.

In addition, we used the dataset from Bansal et al. (2014) as gold taxonomies (i.e. sources of additional knowledge), \(G_{gold}=\{G_{gold,i},1\leq i\leq N_{known}\}\) that are known apriori. This dataset is a set of medium-sized full-domain taxonomies consisting of bottom-out full subtrees sampled from Wordnet, and contains 761 taxonomies in total.

To test our model for taxonomy prediction (and to remove overlap), we removed any taxonomy from \(G_{gold}\) which had term overlap with the set of provided terms for _science_ and _environment_ domains within TExEval-2 task. Because of this, we get 621 non-overlapping taxonomies in total, partitioned by 80-20 ratio to create _training_ and _validation_ datasets respectively.

### Experimental Settings

We ran our experiments in two different settings. In each of them, we train on a different noisy input graph (and the same gold taxonomies as mentioned before), and evaluate on the _science_ and _environ

\begin{table}
\begin{tabular}{c|c|c|c} \hline Domain & Source & \(V\) & \(E\) \\ \hline \hline Science & WordNet & 429 & 452 \\  & Eurovoc & 125 & 124 \\  & Combined & 453 & 465 \\ \hline Environment & Eurovoc & 261 & 261 \\ \hline \end{tabular}
\end{table}
Table 1: Dataset statistics for TExEval-2 task obtained from Bordea et al. (2016). The _Vertices_(\(V\)) and _Edges_(\(E\)) columns represent structural measures of taxonomies for _English_ language only.

 

ment_ domains, within TExEval-2 task. In the first setting, we used the same input as _TaxoRL_[14] for a fair comparison. This input of _TaxoRL_ consists of term pairs and associated dependency path information between them, which has been extracted from three public web-based corpora. For _Graph2Taxo_, we only make use of the term pairs to create a noisy input graph.

In the second setting, we used data5 provided by _TAXI_[30], which comprises of a list of candidate _is-a_ pairs extracted based on substrings and lexico-syntactic patterns. We used these noisy candidate pairs to create a noisy graph.

Footnote 5: Data is available at [http://panchenko.me/data/joint/taxi/res/resources.tgz](http://panchenko.me/data/joint/taxi/res/resources.tgz)

A _Graph2Taxo_ model is then trained on the noisy graph obtained in each of the two settings. In the _test_ phase, all candidate term-pairs for which both terms are present in the _test_ vocabulary are scored (between 0 and 1) by the trained _Graph2Taxo_ model. A threshold of 0.5 is applied, and the candidate pairs scoring beyond this threshold are accumulated together as the predicted taxonomy \(G_{pred}\). Notice that there are different optimal thresholds for different tasks. We get better performance if we tune the thresholds. However, we chose a harder task and proved our model has better performance than others even we simply use 0.5 as the threshold. In addition, We specify the hyper-parameter ranges for our experiments: learning rate \(\{0.01,0.005,0.001\}\), number of kernels \(\{5,10,20\}\) and number of clusters \(\{10,30,50,100\}\). Finally, Adam optimizer [8] is used for all experiments.

**Evaluation Metrics.** Given a gold taxonomy \(G_{gold}\) (as part of the _TExEval-2_ benchmark dataset) and a predicted taxonomy \(G_{pred}\) (by our proposed _Graph2Taxo_ approach), we evaluate \(G_{pred}\) using Edge Precision, Edge Recall and F-score measures as defined in Bordea et al. (2016).

### Hyper-parameters

We use the following hyper-parameter configuration for training the model. We set _dropout_ to 0.3, _number of kernels_\(C\) to 10, _kernel size_\(K\) to 5, _learning rate_ to 0.001 and initial _embedding size_ to 300. For the loss function, we use the \(\lambda=1.0\) and \(\rho=0.5\). In addition, _number of clusters_\(n_{c}\) is set to 50 for all our experiments. In the scenario wherein the input resource comes from TAXI, only hyponym-hyperrm candidate pairs observed more than 10 times are used to create a noisy graph. Also, we use one pooling and one unpooling layer for our experiments.

We use dropouts in two places, one at the end of the cross-domain encoder module, and the other after the Conv1D operation. Our models are trained using NVIDIA Tesla P100 GPUs.

### Results and Discussions

Table 2 shows the results on the _TExEval-2 task_ Evaluation on _science_ and _environment_ domains. The first row represents a string-based baseline method [7], that exploits term compositionality to hierarchically relate terms. For example, it extracts pairs such as (_Statistics Department_, _Department_) from the provided Wikipedia corpus, and utilizes aforementioned technique to construct taxonomy.

The next three rows in Table 2, namely, TAXI, JUNLP and USAAR are some of the top perform

\begin{table}
\begin{tabular}{r|c||c|c|c|c||c|c|c||c|c|c||c|c|c} \hline  & \multicolumn{3}{c||}{Science} & \multicolumn{3}{c||}{Science} & \multicolumn{3}{c||}{Science} & \multicolumn{3}{c||}{Environment} \\  & \multicolumn{3}{c||}{(Combined)} & \multicolumn{3}{c||}{(Eurovoc)} & \multicolumn{3}{c||}{(WordNet)} & \multicolumn{3}{c||}{(Average)} & \multicolumn{3}{c}{(Eurovoc)} \\ \hline Model & \(P_{e}\) & \(R_{e}\) & \(F_{e}\) & \(P_{s}\) & \(R_{e}\) & \(F_{e}\) & \(P_{s}\) & \(R_{e}\) & \(F_{e}\) & \(P_{s}\) & \(R_{e}\) & \(F_{e}\) & \(P_{e}\) & \(R_{e}\) & \(F_{e}\) \\ \hline \hline Baseline & 0.63 & 0.29 & 0.39 & 0.62 & 0.21 & 0.31 & 0.69 & 0.27 & 0.38 & 0.65 & 0.26 & 0.36 & 0.50 & 0.21 & 0.30 \\ JUNLP & 0.14 & 0.31 & 0.19 & 0.13 & **0.36** & 0.19 & 0.21 & 0.31 & 0.25 & 0.16 & 0.33 & 0.21 & 0.13 & 0.23 & 0.17 \\ USAAR & 0.38 & 0.26 & 0.31 & 0.63 & 0.15 & 0.25 & **0.82** & 0.19 & 0.31 & 0.61 & 0.20 & 0.29 & 0.81 & 0.15 & 0.25 \\ TAXI & 0.39 & **0.35** & 0.37 & 0.30 & 0.33 & 0.31 & 0.37 & **0.38** & 0.35 & 0.35 & 0.35 & 0.34 & 0.27 & 0.30 \\ TaxoRL\({}^{A}\) & – & – & – & – & – & – & – & – & – & 0.57 & 0.33 & 0.42 & 0.38 & 0.24 & 0.29 \\ TaxoRL\({}^{B}\) & – & – & – & — & – & – & – & – & – & 0.38 & **0.38** & 0.38 & 0.32 & 0.32 & 0.32 \\ \hline \hline Graph2Taxo1 & **0.91** & 0.31 & 0.46 & 0.78 & 0.26 & 0.39 & **0.82** & 0.32 & **0.46** & **0.84** & 0.30 & 0.44 & **0.89** & 0.24 & 0.37 \\ Graph2Taxo2 & 0.90 & 0.33 & **0.48** & **0.79** & 0.33 & **0.46** & 0.77 & 0.32 & **0.46** & 0.82 & 0.33 & **0.47** & 0.67 & **0.28** & **0.39** \\ \hline \end{tabular}
\end{table}
Table 2: Results on TExEval-2 task: Taxonomy Extraction Evaluation (a.k.a TExEval-2). First four rows represent participating systems in the TExEval-2 task, whose performances are taken from Bordea et al. (2016). TaxoRL\({}^{A/B}\) illustrate the performance of a Reinforcement Learning system by Mao et al. (2018) under the _Partial_ and _Full_ setting respectively. Graph2Taxo\({}^{1/2}\) represent our proposed algorithm under both the settings as described in Section 4.2. All results reported above are rounded to 2 decimal places.

 

ing systems that participated in the TExEval-2 task. Furthermore, TaxoRL\({}^{A,B}\) illustrates the performance of a Reinforcement Learning system by under the _Partial induction_ and _Full induction_ settings respectively Mao et al. (2018). Since Mao et al. (2018) has shown that it outperforms other methods such as Gupta et al. (2017); Bansal et al. (2014), we only compare the results of our proposed _Graph2Taxo_ approach against the state-of-the-art system TaxoRL.

Finally, Graph2Taxo\({}^{1}\) and Graph2Taxo\({}^{2}\) depict the results of our proposed algorithm under both aforementioned settings, i.e. using the input resources of TaxoRL in the first scenario, and using the resources of TAXI in the second scenario. In each of these settings, we find that the overall precision of our proposed _Graph2Taxo_ approach is far better than all the other existing approaches, demonstrating the strong ability of _Graph2Taxo_ to find true relations. Meanwhile, the recall of our proposed _Graph2Taxo_ approach is comparable to that of the existing state-of-the-art approaches. Combining the precision and recall metrics, we observe that _Graph2Taxo_ outperforms existing state-of-the-art approaches on the F-score, by a significant margin. For example, for the Science (Average) domain, Graph2Taxo\({}^{2}\) improves over TaxoRL's F-score by 5%. For the Environment (Eurovoc) domain, our model improves TaxoRL's F-score by 7% on the TExEval-2 task.

Besides, our proposed model has high scalability. For example, the GNN method has been trained for a large graph, including about 1 million nodes Kipf and Welling (2017). Besides, the GNN part can be replaced by any improved GNN methods Hamilton et al. (2017); Gao et al. (2018) designed for large-scale graphs.

#### Ablation Tests.

Table 3 shows the results of proposed _Graph2Taxo_ in the second setting for the ablation experiments (divided into _four_ blocks), which indicates the contribution of each layer used in our _Graph2Taxo_ model. In Table 3, all the experiments are run three times, and the average values of the three runs are reported. Furthermore, in Figure 3, we randomly choose Science (Eurovoc) domain as the one to report the _error-bars_ (corresponding to the standard-deviation values) for our experiments.

The first block of values in Table 3 illustrates results by _ablating_ layers from within our _Graph2Taxo_ model. Comparing the first two rows, it's evident that adding a _Semantic Cluster_ (SC) layer improves recall at the cost of precision, however improving the overall F-score. This improve

Figure 3: Results on Science (Eurovoc) domain: The average _Precision_, _Recall_ and _F-score_ values and their standard error values. It is clear that addition of Residual Layer and SC Layer _lowers_ the variance of the results.

 

ment is clearly seen for the Science (Eurovoc) domain, wherein we have an increase of 3%.

In the second block, we show that the addition of constraints improves performance. Row 4 represents a _Graph2Taxo_ i.e. 2GNN+SC+Res setup, but without any constraint. Adding the DAG Constraint (Row 1) to this yields can get a better F-score. Specifically, we observe a major increase of +5% F1 for the Science (Eurovoc) domain.

In the third block, we remove the features \(v_{feas}\) as mentioned in section 3.3.1. The results, i.e. row 5 in Table 3 shows that these features are critical in improving the performance of our proposed system on both Science (Eurovoc) and Environment (Eurovoc) domains. Note that these features denoted as \(v_{feas}\) are not a novelty of our proposed method, but rather have been used by existing state-of-the-art approaches.

Finally, we study the effect of initializing our model using pre-trained embeddings, rather than initializing at random. Specifically, we initialize the input matrix \(H_{0}\) of our _Graph2Taxo_ model with pre-trained fastText5 embeddings. Our model using fastText embeddings improves upon Row 1 by a margin of 4% in precision values for the Environment (Eurovoc) domain, but unfortunately has no significant effect on the F-score. Hence, we have not used pre-trained embeddings in reporting the results in Table 2.

Footnote 5: [https://fasttext.cc](https://fasttext.cc)

We provide an illustration of the output of the _Graph2Taxo_ model in Figure 4, for the Environment domain.The generated taxonomy in this example contains multiple trees, which serve the purpose of generating taxonomical classifications. As future work, we plan to figure out different strategies to connect the subtrees into a large graph for better DAG generation.

## 5 Conclusion

We have introduced a GNN-based cross-domain knowledge transfer framework _Graph2Taxo_, which makes use of a cross-domain graph structure, in conjunction with an acyclicity constraint-based DAG learning for taxonomy construction. Furthermore, our proposed model encodes acyclicity as a _soft_ constraint and shows that the overall model outperforms state of the art.

In the future, we would like to figure out different strategies to merge individual gains, obtained by separate application of the DAG constraint, into a setup that can take the best of both precision and recall improvements, and put forth a better performing system. We also plan on looking into strategies to improve recall of the constructed taxonomy.

## Acknowledgments

The authors would like to thank Dr. Jie Chen from MIT-IBM Watson AI Lab and Prof. Jinbo Bi from the University of Connecticut for in-depth discussions on model construction.

## References

* Alfarone and Davis (2015) Daniele Alfarone and Jesse Davis. 2015. Unsupervised learning of an IS-A taxonomy from a limited domain-specific corpus. In _Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015_, pages 1434-1441. AAAI Press.
* Bansal et al. (2014) Mohit Bansal, David Burkett, Gerard de Melo, and Dan Klein. 2014. Structured learning for taxonomy induction with belief propagation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1041-1051, Baltimore, Maryland. Association for Computational Linguistics.
* Baroni et al. (2012) Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, pages 23-32, Avignon, France. Association for Computational Linguistics.
* van den Berg et al. (2017) Rainne van den Berg, Thomas N. Kipf, and Max Welling. 2017. Graph convolutional matrix completion. _CoRR_, abs/1706.02263.
* Bordea et al. (2016) Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. SemEval-2016 task 13: Taxonomy extraction evaluation (TExEval-2). In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 1081-1091, San Diego,

Figure 4: A simple example of the taxonomy generated by _Graph2Taxo_ in the environment domain.

 

California. Association for Computational Linguistics.
* Bruna et al. (2014) Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.
* Fifth International Workshop on Artificial Intelligence and Statistics, AISTATS 1995, Key West, Florida, USA, January, 1995. Proceedings_, pages 121-130. Springer.
* Chickering et al. (2004) David Maxwell Chickering, David Heckerman, and Christopher Meek. 2004. Large-sample learning of bayesian networks is np-hard. _J. Mach. Learn. Res._, 5:1287-1330.
* February 12, 2020_. AAAI Press.
* Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, USA, February 2-7, 2018_, pages 1811-1818. AAAI Press.
* Friedrich and Zenker (2011) Gerhard Friedrich and Markus Zenker. 2011. A taxonomy for generating explanations in recommender systems. _AI Magazine_, 32(3):90-98.
* Gan et al. (2016) Chuang Gan, Yi Yang, Linchao Zhu, Deli Zhao, and Yueting Zhuang. 2016. Recognizing an action using its name: A knowledge-based approach. _Int. J. Comput. Vis._, 120(1):61-77.
* Gao et al. (2018) Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. 2018. Large-scale learnable graph convolutional networks. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, pages 1416-1424. ACM.
* Gilmer et al. (2017) Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, volume 70, pages 1263-1272. PMLR.
* Gori et al. (2005) Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._, volume 2, pages 729-734. IEEE.
* 10, 2017_.
* Hamilton et al. (2017) William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In _Advances in Neural Information Processing Systems, NeurIPS 2017, 4-9 December 2017, Long Beach, CA, USA_, pages 1024-1034.
* Harabagiu et al. (2003) Sanda M. Harabagiu, Steven J. Maiorano, and Marius Pasca. 2003. Open-domain textual question answering techniques. _Nat. Lang. Eng._, 9(3):231-267.
* Hearst (1992) Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In _14th International Conference on Computational Linguistics, COLING 1992, Nantes, France, August 23-28, 1992_, pages 539-545.
* Hua et al. (2017) Wen Hua, Zhongyuan Wang, Haixun Wang, Kai Zheng, and Xiaofang Zhou. 2017. Understand short texts by harvesting and analyzing semantic knowledge. _IEEE Trans. Knowl. Data Eng._, 29(3):499-512.
* Huang et al. (2015) Hao Huang, Haihua Xu, Xianhui Wang, and Wushour Silamu. 2015. Maximum f1-score discriminative training criterion for automatic mispronunciation detection. _IEEE ACM Trans. Audio Speech Lang. Process._, 23(4):787-797.
* Karp (1971) Richard M. Karp. 1971. A simple derivation of edmonds' algorithm for optimum branchings. _Networks_, 1(3):265-272.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Kozareva and Hovy (2010) Zornitsa Kozareva and Eduard Hovy. 2010. A semi-supervised method to learn and construct taxonomies using the web. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 1110-1118, Cambridge, MA. Association for Computational Linguistics.
* Luu et al. (2014) Anh Tuan Luu, Jung-jae Kim, and See Xiong Ng. 2014. Taxonomy construction using syntactic contextual evidence. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 810-819, Doha, Qatar. Association for Computational Linguistics 

Yuning Mao, Xiang Ren, Jiaming Shen, Xiaotao Gu, and Jiawei Han. 2018. End-to-end reinforcement learning for automatic taxonomy induction. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2462-2472, Melbourne, Australia. Association for Computational Linguistics.
* Miller et al. [1990] George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. _International journal of lexicography_.
* Monti et al. [2017] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M. Bronstein. 2017. Geometric deep learning on graphs and manifolds using mixture model cnns. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society.
* Panchenko et al. [2016] Alexander Panchenko, Stefano Faralli, Eugen Ruppert, Steffen Remus, Hubert Naets, Cedrick Fairon, Simone Paolo Ponzetto, and Chris Biemann. 2016. TAXI at SemEval-2016 task 13: a taxonomy induction method based on lexicro-syntactic patterns, substrings and focused crawling. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, San Diego, California. Association for Computational Linguistics.
* Ritter et al. [2009] Alan Ritter, Stephen Soderland, and Oren Etzioni. 2009. What is this, anyway: Automatic hypernym discovery. In _Learning by Reading and Learning to Read, Papers from the 2009 AAAI Spring Symposium, Technical Report SS-09-07, Stanford, California, USA, March 23-25, 2009_, pages 88-93. AAAI.
* Roller et al. [2014] Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_, pages 1025-1036, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
* February 1, 2019_, pages 3060-3067. AAAI Press.
* November 02, 2012_, pages 345-354. ACM.
* Snow et al. [2004]Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In _Advances in Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada_.
* Snow et al. [2006] Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_, pages 801-808, Sydney, Australia. Association for Computational Linguistics.
* Sperduti and Starita [1997] Alessandro Sperduti and Antonina Starita. 1997. Supervised neural networks for the classification of structures. _IEEE Trans. Neural Networks_, 8(3):714-735.
* Suchanek et al. [2008] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2008. YAGO: A large ontology from wikipedia and wordnet. _J. Web Semant_.
* Tarjan [1972] Robert Endre Tarjan. 1972. Depth-first search and linear graph algorithms. _SIAM J. Comput._, 1(2):146-160.
* Velardi et al. [2013] Paola Velardi, Stefano Faralli, and Roberto Navigli. 2013. OntoLearn reloaded: A graph-based algorithm for taxonomy induction. _Computational Linguistics_, 39(3):665-707.
* Wang et al. [2017] Chengyu Wang, Xiaofeng He, and Aoying Zhou. 2017. A short survey on taxonomy learning from text corpora: Issues, resources and recent advances. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, Copenhagen, Denmark. Association for Computational Linguistics.
* Wu et al. [2012] Wentao Wu, Hongsong Li, Haixun Wang, and Konny Qili Zhu. 2012. Probase: a probabilistic taxonomy for text understanding. In _Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2012, Scottsdale, AZ, USA, May 20-24, 2012_, pages 481-492. ACM.
* Xu et al. [2019] Keyulu Xu, Weinhua Hu, Jure Leskovec, and Stefanie Jeselka. 2019. Two powerful car group neural networks? In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Ying et al. [2018] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. 2018. Hierarchical graph representation learning with differentiable pooling. In _Advances in Neural Information Processing Systems, NeurIPS 2018, 3-8 December 2018, Montreal, Canada_, pages 4805-4815.
* Yu et al. [2019] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. 2019. DAGGNN: DAG structure learning with graph neural networks. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97, pages 7154-7163. PMLR.
* Zheng et al. [2018] Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. 2018. Dags with NO TEARS: continuous optimization for structure learning. In _Advances in Neural Information Processing Systems, NeurIPS 2018, 3-8 December 2018, Montreal, Canada_, pages 9492-9503