Venue: NAACL 2018

# Robust Cross-lingual Hypernymy Detection using Dependency Context

Shyam Upadhyay\({}^{1*}\), Yogarshi Vyas\({}^{2*}\), Marine Carpuat\({}^{2}\), Dan Roth\({}^{1}\)

\({}^{1}\) Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA

\({}^{2}\) Department of Computer Science, University of Maryland, College Park, MD

shyamupa@seas.upenn.edu, yogarshi@cs.umd.edu, marine@cs.umd.edu, danroth@seas.upenn.edu

These authors contributed equally.

###### Abstract

_Cross-lingual Hypernymy Detection_ involves determining if a word in one language ("fruit") is a hypermym of a word in another language ("pomme" i.e. apple in French). The ability to detect hyperpmmy cross-lingually can aid in solving cross-lingual versions of tasks such as textual entailment and event coreference. We propose BiSparse-Dep, a family of unsupervised approaches for cross-lingual hypernymy detection, which learns sparse, bilingual word embeddings based on dependency contexts. We show that BiSparse-Dep can significantly improve performance on this task, compared to approaches based only on lexical context. Our approach is also robust, showing promise for low-resource settings: our dependency-based embeddings can be learned using a parser trained on related languages, with negligible loss in performance. We also crowd-source a challenging dataset for this task on four languages - Russian, French, Arabic, and Chinese. Our embeddings and datasets are publicly available.1

Footnote 1: [https://github.com/yogarshi/bisparse-dep/](https://github.com/yogarshi/bisparse-dep/)

+
Footnote †: This work was supported by the National Natural Science Foundation under Grant No. 6074958.

## 1 Introduction

Translation helps identify correspondences in bilingual texts, but other asymmetric semantic relationships can improve language understanding when translations are not exactly equivalent. One such relationship is _cross-lingual hypernymy_ - identifying that _cureuil_ ("squirrel" in French) is a kind of _rodent_, or nopona ("crow" in Russian) is a kind of _bird_. The ability to detect hypernyms across languages serves as a building block in a range of cross-lingual tasks, including Recognizing Textual Entailment (RTE) Negri et al. (2012, 2013), constructing multilingual taxonomies Fu et al. (2014), event coreference across multilingual news sources Vossen et al. (2015), and evaluating Machine Translation output Pado et al. (2009).

Building models that can robustly identify hypernymy across the spectrum of human languages is a challenging problem, that is further compounded in low resource settings. At first glance, translating words to English and then identifying hypernyms in a monolingual setting may appear to be a sufficient solution. However, this approach cannot capture many phenomena. For instance, the English words _cook_, _leader_ and _supervisor_ can all be hypernyms of the French word _chelef_, as the French word does not have a exact translation in English covering its possible usages. However, translating _chef_ to _cook_ and then determining hypernymy monolingally precludes identifying _leader_ or _supervisor_ as a hypernyms of _chef_. Similarly, language-specific usage patterns can also influence hypernymy decisions. For instance, the French word _chroniqueur_ translates to _chronicler_ in English, but is more frequently used in French to refer to journalists (making _journalist_ its hypernym).2

Footnote 2: All examples are from our dataset.

This motivates approaches that _directly_ detect hypernymy in the cross-lingual setting by extending distributional methods for detecting monolingual hypernymy, as in our prior work Vyas and Carpuat (2016). State-of-the-art distributional approaches Roller and Erk (2016); Shwartz et al. (2017) for detecting monolingual hypernymy require syntactic analysis (eg. dependency parsing), which may not available for many languages. Additionally, limited training resources make unsupervised methods more desirable than supervised hypernymy detection approaches Roller and Erk (

2016). Furthermore, monolingual distributional approaches cannot be applied directly to the cross-lingual task, because the vector spaces of two languages need to be aligned using a cross-lingual resource (a bilingual dictionary, for instance).

We tackle these challenges by proposing BiSparse-Dep - a family of robust, unsupervised approaches for identifying cross-lingual hypernymy. BiSparse-Dep uses a cross-lingual word embedding model learned from a small bilingual dictionary and a variety of monolingual syntactic context extracted from a dependency parsed corpus. BiSparse-Dep exhibits robust behavior along multiple dimensions. In the absence of a dependency treebank for a language, it can learn embeddings using a parser trained on related languages. When exposed to less monolingual data, or a lower quality bilingual dictionary, BiSparse-Dep degrades only marginally. In all these cases, it compares favorably with models that have been supplied with all necessary resources, showing promise for low-resource settings. We extensively evaluate BiSparse-Dep on a new crowd-sourced cross-lingual dataset, with over 2900 hypernym pairs, spanning four languages from distinct families - French, Russian, Arabic and Chinese - and release the datasets for future evaluations.

## 2 Related Work

Cross-lingual Distributional Semantics Cross-lingual word embeddings have been shown to encode semantics across languages in tasks such as word similarity Faruqui and Dyer (2014) and lexicon induction Vulic and Moens (2015). Our works stands apart in two aspects (1) In contrast to tasks involving similarity and synonymy (symmetric relations), the focus of our work is on detecting _asymmetric_ relations across languages, using cross-lingual embeddings. (2) Unlike most previous work, we use dependency context instead of lexical context to induce cross-lingual embeddings, which allows us to abstract away from language specific word order, and (as we show) improves hypernymy detection.

More closely related is our prior work Vyas and Carpuat (2016) where we used lexical context based embeddings to detect cross-lingual lexical entailment. In contrast, the focus of this work is on hypernymy, a more well-defined relation than entailment. Also, we improve upon our previous approach by using dependency based embeddings (SS6.1), and show that the improvements hold even when exposed to data scarce settings (SS6.3). We also do a more comprehensive evaluation on four languages paired with English, instead of just French.

Dependency Based EmbeddingsIn monolingual settings, dependency based embeddings have been shown to outperform window based embeddings on many tasks Bansal et al. (2014); Hill et al. (2014); Melamud et al. (2016). Roller and Erk (2016) showed that dependency embeddings can help in recovering Hearst patterns Hearst (1992) like "animals such as cats", which are known to be indicative of hypernymy. Shwartz et al. (2017) demonstrated that dependency based embeddings are almost always superior to window based embeddings for identifying hypernyms in English. Our work uses dependency based embeddings in a cross-lingual setting, a less explored research direction. A key novelty of our work also lies in its use of syntactic transfer to derive dependency contexts. This scenario is more relevant in a cross-lingual setting, where treebanks might not be available for many languages.

## 3 Our Approach - BiSparse-Dep

We propose BiSparse-Dep, a family of approaches that uses sparse, bilingual, dependency based word embeddings to identify cross-lingual hypernymy.

Figure 1 shows an overview of the end-to-end pipeline of BiSparse-Dep. The two key components of this pipeline are: (1) _Dependency based contexts_ (SS3.1), which help us generalize across languages with minimal customization by abstracting away language-specific word order. We also discuss how to extract such contexts in the absence of a treebank in the language (SS3.2) using a (weak) dependency parser trained on related languages. (2) _Bilingual sparse coding_ (SS3.3), which allows us to align dependency based word embeddings in a shared semantic space using a small bilingual dictionary. The resulting sparse bilingual embeddings can then be used with a unsupervised entailment scorer (SS3.4) to predict hypernymy for cross-lingual word pairs.

Figure 2: Example Dependency Tree.

 



### Dependency Based Context Extraction

The context of a word can be described in multiple ways using its syntactic neighborhood in a dependency graph. For instance, in Figure 2, we describe the context for a target word (_traveler_) in the following two ways:

* Full context Pado and Lapata (2007); Baroni and Lenci (2010); Levy and Goldberg (2014): Children and parent words, concatenated with the label and direction of the relation (eg. _roamed#nsubj\({}^{-1}\)_ and _tired#amod_ are contexts for _traveler_).
* Joint context Chersoni et al. (2016): Parent concatenated with each of its siblings (eg. _roamed#desert_ and _roomed#seeking_ are contexts for _traveler_).

These two contexts exploit different amounts of syntactic information - Joint does not require labeled parses, unlike Full. The Joint context combines parent and sibling information, while Full keeps them as distinct contexts. Both encode directionality into the context, either through label direction or through sibling-parent relations.

We use word-context co-occurrences generated using these contexts in a distributional semantic model (DSM) in lieu of window based contexts to generate dependency based embeddings.

### Dependency Contexts without a Treebank

Using dependency contexts in multilingual settings may not always be possible, as dependency treebanks are not available for many languages. To circumvent this issue, we use related languages to train a weak dependency parser.

We train a _delexicalized_ parser using treebanks of related languages, where the word form based features are turned off, so that the parser is trained on purely non-lexical features (e.g. POS tags). The rationale behind this is that related languages show common syntactic structure that can be transferred to the original language, with delexicalized parsing Zeman and Resnik (2008); McDonald et al. (2011); _inter alia_) being one popular approach.3

Footnote 3: More sophisticated techniques for transferring syntactic knowledge have been proposed Ammar et al. (2016); Rasooli and Collins (2017), but we prioritize simplicity and show that a simple delexicalized parser is effective.

### Bilingual Sparse Coding

Given a dependency based co-occurrence matrix described in the previous section(s), we generate BiSparse-Dep embeddings using the framework from our prior work Vyas and Carpuat (2016), which we henceforth call BiSparse. BiSparse generates sparse, bilingual word embeddings using a dictionary learning objective with a sparsity inducing \(l_{1}\) penalty. We give a brief overview of this approach, the full details of which can be found in our prior work.

For two languages with vocabularies \(v_{e}\) and \(v_{f}\), and monolingual dependency embeddings \(\mathbf{X_{e}}\) and \(\mathbf{X_{f}}\), BiSparse solves the following objective:

\[\underset{\mathbf{A_{e}},\mathbf{D_{e}},\mathbf{A_{f}},\mathbf{ D_{f}}}{\text{argmin}} \sum_{i=1}^{v_{e}}~{}\frac{1}{2}||\mathbf{A_{ei}}\mathbf{D_{e}}^{\mathsf{T}}- \mathbf{X_{ei}}||_{2}^{2}+\lambda_{e}||\mathbf{A_{e}}_{i}||_{1}\] \[+\sum_{j=1}^{v_{f}}~{}\frac{1}{2}||\mathbf{A_{fj}}\mathbf{D_{f}}^ {\mathsf{T}}-\mathbf{X_{fj}}||_{2}^{2}+\lambda_{f}||\mathbf{A_{fj}}||_{1}\] \[+\sum_{i,j}\frac{1}{2}\lambda_{x}\mathbf{S}_{ij}||\mathbf{A_{e}}_ {i}-\mathbf{A_{fj}}||_{2}^{2}\] (1) s.t. \[\mathbf{A_{k}}>\mathbf{0}\qquad\|\mathbf{D_{ki}}\|_{2}^{2}\leq 1 \qquad\mathbf{k}\in\{\mathbf{e},\mathbf{f}\}\]

where \(\mathbf{S}\) is a translation matrix, and \(\mathbf{A_{e}}\) and \(\mathbf{A_{f}}\)

Figure 1: The BiSparse-Dep approach, which learns sparse bilingual embeddings using dependency based contexts. The resulting parse embeddings, together with an unsupervised entailment scorer, can detect hypernyms across languages (e.g., _pomme_ is a _fnuit_).



are sparse matrices which are bilingual representations in a shared semantic space. The translation matrix **S** (of size \(v_{e}\times v_{f}\)) captures correspondences between the vocabularies (of size \(v_{e}\) and \(v_{f}\)) of two languages. For instance, each row of **S** can be a one-hot vector that identifies the word in \(\boldsymbol{f}\) that is most frequently aligned with the \(\boldsymbol{e}\) word for that row in a large parallel corpus, thus building a one-to-many mapping between the two languages.

### Unsupervised Entailment Scorer

A variety of scorers can be used to quantify the directional relationship between two words, given feature representations of these words Lin (1998); Weeds and Weir (2003); Lenci and Benotto (2012). Once the BiSparse-Dep embeddings are constructed, we use _BalAPinc_Kolterman et al. (2009) to score word pairs for hypernymy. _BalAPinc_ is based on the distributional inclusion hypothesis Geffet and Dagan (2005) and computes the geometric mean of 1) _LIN_Lin (1998), a symmetric score that captures similarity, and 2) _APinc_, an asymmetric score based on average precision.

## 4 Crowd-Sourcing Annotations

There is no publicly available dataset to evaluate models of hypernymy detection across multiple languages. While ontologies like Open Multilingual WordNet (OMW) Bond and Foster (2013) and BabelNet Navigli and Ponzetto (2012) contain cross-lingual links, these resources are semi-automatically generated and hence contain noisy edges. Thus, to get reliable and high-quality test beds, we collect evaluation datasets using CrowdFlower4. Our datasets span four languages from distinct families - French (Fr), Russian (Ru), Arabic (Ar) and Chinese (Zh) - paired with English.

Footnote 4: [http://crowdflower.com](http://crowdflower.com)

To begin the annotation process, we first pool candidate pairs using hypernymy edges across languages from OMW and BabelNet, along with translations from monolingual hypermymy datasets Baroni and Lenci (2011); Baroni et al. (2012); Koterman et al. (2010).

### Annotation Setup

The annotation task requires annotators to be fluent in both English and the non-English language. To ensure only fluent speakers perform the task, for each language, we provide task instructions in the non-English language itself. Also, we restrict the task to annotators verified by CrowdFlower to have those language skills. Finally, annotators also need to pass a quiz based on a small amount of gold standard data to gain access to the task.

Annotators choose between three options for each word pair (\(p_{f},q_{e}\)), where \(p_{f}\) is a non-English word and \(q_{e}\) is a English word : "\(p_{f}\) is a kind of \(q_{e}\)", "\(q_{e}\) is a part of \(p_{f}\)" and "none of the above". Word pairs labeled with the first option are considered as positive examples while those labeled as "none of the above" are considered as negative.5 The second option was included to filter out meronymy examples that were part of the noisy pool. We leave it to the annotator to infer whether the relation holds between any senses of \(p_{f}\) or \(q_{e}\), if either of them are polysemous.

Footnote 5: We collected more negative pairs than positive, but sampled so as to keep a balanced dataset for ease of evaluation. We will release all annotated pairs along with the dataset.

For every candidate hypernym pair \((p_{f},q_{e})\), we also ask annotators to judge its reversed and translated _hyponym_ pair \((q_{f},p_{e})\). For instance, if \((citron,food)\) is a hypernym candidate, we also show annotators \((aliments,lemon)\) which is a potential hyponym candidate (_potential_, because as mentioned in SS1, translation need not preserve semantic relationships). The purpose of presenting the hyponym pair, \((q_{f},p_{e})\), is two-fold. First, it emphasizes the directional nature of the task. Second, it identifies hyponym pairs, which we use as negative examples. The hyponym pairs are challenging since differentiating them from hypernyms truly requires detecting asymmetry.

Each pair was judged by at least 5 annotators, and judgments with 80% agreement (at least 4 annotators agree) are considered for the final dataset. This is a stricter condition than certain monolingual hypernymy datasets - for instance, EVALUATION Santus et al. (2015) - where agreement by 3 annotators is deemed sufficient. Inter-annotator agreement measured using Fleiss' Kappa Fleiss (1971) was 58.1 (French), 53.7 (Russian), 53.2 (Arabic) and 55.8 (Chinese). This indicates moderate agreement, on par with agreement obtained on related fine-grained semantic tasks Pavlick et al. (2015). We cannot compare with monolithic

\begin{table}
\begin{tabular}{l c r} \hline \hline pair & \#crowdsourced & \#pos (= \#neg) \\ \hline French-English & 2115 & 763 \\ Russian-English & 2264 & 706 \\ Arabic-English & 2144 & 691 \\ Chinese-English & 2165 & 806 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Crowd-sourced dataset statistics. #pos (#neg) denote positives (negatives) in the evaluation set. We deliberately under-sample negatives to have a balanced evaluation set.

 

gual hypernymy annotator agreement as, to the best of our knowledge, such numbers are not available for existing test sets. Dataset statistics are shown in Table 1.

We observed that annotators were able to agree on pairs containing polysemous words where hypernymy holds for some sense. For instance, for the French-English pair (_avocat_, _professional_), the French word _avocat_ can either mean _lawyer_ or _avocado_, but the pair was annotated as a positive example. Hence, we leave it to the annotators to handle polysemy by choosing the most appropriate sense.

### Two Evaluation Test Sets

To verify if the crowdsourced hyponyms are challenging negative examples we create two evaluation sets. Both share the (crowdsourced) positive examples, but differ in their negatives:

* negative examples are the crowdsourced hyponyms.
* negative examples are _cohyponyms_ drawn from OMW.

Cohyponyms are words sharing a common hypernym. For instance, _biere_ ("beer" in French) and _vodka_ are cohyponyms since they share a common hypernym in _alcoholalcohol_. We choose cohyponyms for the second test set because: (a) They require differentiating between similarity (a symmetric relation) and hypernymy (an asymmetric relation). For instance, _biere_ and _vodka_ are highly similar yet, they do not have a hypernymy relationship. (b) Cohyponyms are a popular choice of negative examples in many entailment datasets Baroni and Lenci (2011).

## 5 Experimental Setup

### Data and Evaluation Setup

Training BiSparse-Dep requires a dependency parsed monolingual corpus, and a translation matrix for jointly aligning the monolingual vectors. We compute the translation matrix using word alignments derived from parallel corpora (see corpus statistics in Table 3). While we use parallel corpora to generate the translation matrix to be comparable to baselines (SS5.2), we can obtain the matrix from any bilingual dictionary.

The monolingual corpora are parsed using Yara Parser Rasooli and Tetreault (2015), trained on the corresponding treebank from the Universal Dependency Treebank McDonald et al. (2013) (UDT-v1.4). Yara Parser was chosen as it is fast, and competitive with state-of-the-art parsers Choi et al. (2015). The monolingual corpora was POS-tagged using TurbogTarger Martins et al. (2013). We induce dependency contexts for words by first thresholding the language vocabulary to the top 50,000 nouns, verbs and adjectives. A co-occurrence matrix is computed over this vocabulary using the context types in SS3.1.

Inducing Dependency ContextsThe entries of the word-context co-occurrence matrix are reweighted using Positive Pointwise Mutual Information Bullinaria and Levy (2007). The resulting matrix is reduced to 1000 dimensions using SVD Golub and Kahan (1965).6 These vectors are used as \({\bf X}_{\mathbf{e}},{\bf X}_{\mathbf{f}}\) in the setup from SS3.3 to generate 100 dimensional sparse bilingual vectors.

Footnote 6: Chosen based on preliminary experiments with {500,1000,2000,3000} dimensional vectors for En-Fr.

EvaluationWe use accuracy as our evaluation metric, as it is easy to interpret when the classes are balanced Turney and Mohammad (2015). Both evaluation datasets - Hyper-Hypo and Hyper-Cohypo - are split into 1:2 dev/test splits. _BalAPinc_ has two tunable parameters - 1) a threshold that indicates the _BalAPinc_ score above which all examples are labeled as positive, 2) the maximum number of features to consider for each word. We use the tuning set to tune the two parameters as well as the various hyper-parameters associated with the models.

### Contrastive Approaches

We compare our BiSparse-Dep embeddings with the following approaches: Mono-Dep (**Translation baseline**) For word pair (\(p_{f}\), \(q_{e}\)) in test data, we translate \(p_{f}\) to English using the most common translation in the translation matrix. Hypernymy is then determined using sparse, dependency based embeddings in English.

BiSparse-Lex (Window context)Predecessor of the BiSparse-Dep model from our previous work Vyas and Carpuat (2016). This model induces sparse, cross-lingual embeddings using window based context.

Bivec+ (**Window context**)Our extension of the Bivec model of Luong et al. (2015). Bivec generates dense, cross-lingual embeddings using window based context, by substituting aligned word pairs within a window in parallel sentences. By default, Bivec only trains using parallel data, 

and so we initialize it with monolongually trained window based embeddings to ensure fair comparison.

Cl-Dep (Dependency context)The model from Vulic2017, which induces dense, dependency based cross-lingual embeddings by translating syntactic word-context pairs using the most common translation, and jointly training a word2vecf7 model for both languages. Vulic2017 showed improvements for word similarity and bilingual lexicon induction. We report the first results using Cl-Dep on this task.

Footnote 6: [https://github.com/](https://github.com/) 

Hyper-Hypo ResultsTable 2(a) shows the results on Hyper-Hypo. First, the benefit of cross-lingual modeling (as opposed to translation) is evident in that almost all models (except Cl-Dep on French) outperform the translation baseline. Among dependency based models, BiSparse-Dep (Full) and Cl-Dep consistently outperform both window models, while BiSparse-Dep (Joint) outperforms them on all except Russian. BiSparse-Dep (Joint) is the best model overall for two languages (French and Chinese), Cl-Dep for one (Arabic), with no statistically significant differences between BiSparse-Dep (Joint) and Cl-Dep for Russian. This confirms that dependency context is more useful than window context for cross-lingual hypernymy detection.

Hyper-Cohypo ResultsThe trends observed on Hyper-Hypo also hold on Hyper-Cohypo i.e. dependency based models continue to outperform window based models (Table 2(b)).

Overall, BiSparse-Dep (Full) performs best in this setting, followed closely by BiSparse-Dep (Joint). This suggests that the sibling information encoded in Joint is useful to distinguish hypernyms from hyponyms (Hyper-Hypo results), while the dependency labels encoded in Full help to distinguish hypernyms from co-hyponyms. Also note that all models improve significantly on the Hyper-Cohypo set, suggesting that discriminating hypernyms from cohypomys is easier than discriminating them from hyponyms.

While the BiSparse-Dep models were generally performing better than window models on both test sets, Cl-Dep was not as consistent (e.g., it was worse than the best window model on Hyper-Cohypo). As shown by Turney and Mohammad (2015), _BalAPinc_ is designed for sparse embeddings and is likely to perform poorly with dense embeddings. This explains the relatively inconsistent performance of Cl-Dep.

Besides establishing the challenging nature of our crowd-sourced set, the experiments on Hyper-Cohypo and Hyper-Hypo also demonstrate the ability of the BiSparse-Dep models to discriminate between different lexical semantic relations (viz. hypernymy and cohyponymy) in a cross-lingual setting. We will investigate this ability more carefully in future work.

### Ablating Directionality in Context

The context described by the Full and Joint BiSparse models encodes directional information (SS3.1) either in the form of label direction (Full), or using sibling information (Joint). Does such directionality in the context help to capture the asymmetric relationship inherent to hypernymy? To answer this, we evaluate a third BiSparse-Dep model which uses Unlabeled dependency contexts. This is similar to the Full context, except we do not concatenate the label of the relation to the context word (parent or children). For instance, for _traveler_ in Fig. 2, contexts will be _roamed_ and _tired_.

Experiments on both Hyper-Hypo and Hyper-Cohypo (bottom row, Tables 2(a) and 2(b)) highlight that directional information is indeed essential - Unlabeled almost always performs worse than Full and Joint, and in many cases worse than even window based models.

\begin{table}
\begin{tabular}{l c c c c c} Model \({}_{\downarrow}\) En With \(\rightarrow\) & Ru & Zh & Ar & Fr & Avg. \\ \hline \multicolumn{6}{c}{Translation Baseline} \\ \hline Mono-Dep & 50.1 & 52.3 & 51.8 & 54.5 & 52.2 \\ \hline \multicolumn{6}{c}{Win. Based} \\ \hline BiSparse-Lex & 56.6 & 53.7 & 50.9 & 52.0 & 53.3 \\ Bivec+ & 55.8 & 52.0 & 51.5 & 53.4 & 53.2 \\ \hline \multicolumn{6}{c}{Dep. Based} \\ \hline Cl-Dep & **60.2** & 54.4 & **56.7*** & 53.8 & **56.3** \\ BiSparse-Dep (Full) & 59.0 & 55.9 & 52.6 & 56.6 & 56.0 \\ BiSparse-Dep (Joint) & 53.8 & **57.0*** & 52.4 & **59.9*** & 55.8 \\ \hline BiSparse-Dep (Unlab) & 55.9 & 51.2 & 53.3 & 55.9 & 54.1 \\ \end{tabular}
\end{table}
Table 3: Comparing the different approaches from §5.2 with our BiSparse-Dep approach on Hyper-Hypo and Hyper-Cohypo (random baseline= 0.5). **Bold** denotes the best score for each language, and the * on the best score indicates a statistically significant (p < 0.05) improvement over the next best score, using McNemar’s test [1]. Across both datasets, BiSparse-Dep models outperform window based models and the translation baseline on an average.

 



### Evaluating Robustness of BiSparse-Dep

No TreebankWe run experiments (Table 4) for all languages with a version of BiSparse-Dep that use the Full context type for both English and the non-English (target) language, but the target language contexts are derived from a corpus parsed using a delexicalized parser (SS5.3).

This model compares favorably on all language pairs against the best window based and the best dependency based model. In fact, it almost consistently outperforms the best window based model by several points, and is only slightly worse than the best dependency-based model.

Further analysis revealed that the good performance of the delexicalized model is due to the relative robustness of the delexicalized parser on frequent contexts in the co-occurrence matrix. Specifically, we found that in French and Russian, the most frequent contexts were derived from amod, nmod, nsubj and dobj edges.8 For instance, the nmod edge appears in 44% of Russian contexts and 33% of the French contexts. The delexicalized parser predicts both the label and direction of the nmod edge correctly with an F1 of 68.6 for Russian and 69.6 for French. In contrast, a fully-trained parser achieves a F1 of 76.7 for Russian and 76.8 for French for the same edge.

Footnote 8: Together they make up at least 70% of the contexts.

Small Monolingual CorpusIn Figure 4, we use increasingly smaller monolingual corpora (10%, 20%, 40%, 60% and 80%) sampled at random to induce the monolingual vectors for BiSparse-Dep (Full) model. Trends (Figure 4) indicate that BisSparse-Dep models that use only 40% of the original data remain competitive with the BisSparse-Lex model that has access to the full data. Robust performance with smaller monolingual corpora is helpful since large-enough monolingual corpora are not always easily available.

Quality of Bilingual DictionaryBilingual dictionaries derived from smaller amounts of parallel data are likely to be of lower quality than those derived from larger corpora. Hence, to analyze the impact of dictionary quality on BiSparse-Dep (Full), we use increasingly smaller parallel corpora to induce bilingual dictionaries used as the score matrix \({\bf S}\) (SS3.3). We use the top 10%, 20%, 40%, 60% and 80% sentences from the parallel corpora. The trends in Figure 4 show that even with a lower quality dictionary, BiSparse-Dep performs better than BiSparse-Lex.

### Choice of Entailment Scorer

We change the entailment scorer from _BalAPinc_ to _SLQS_Santus et al. (2014) and redo experiments from SS6.1 to see if the conclusions drawn depend

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Model \({}_{1}\) En With \(\rightarrow\) Ru & Zh & Ar & Fr & Avg. \\ \hline \multicolumn{6}{c}{Hyper-Hypo} \\ \hline Best Win. & 56.6 & 53.7 & 51.5 & 53.4 & 53.8 \\ Delex. & 59.1* & 55.1* & 54.6* & 56.1* & 56.2 \\ Best Dep. & 60.2 & 57.0* & 56.7* & 59.9* & 58.5 \\ \hline \multicolumn{6}{c}{Hyper-Cobpo} \\ \hline Best Win. & 63.8 & 64.9 & 65.8 & 63.2 & 64.4 \\ Delex. & 59.4 & 65.7* & 67.5* & 66.3* & 64.7 \\ Best Dep. & 63.6* & 67.3* & 66.8* & 66.7 & 66.1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Robustness in absence of a treebank**: The delexicalized model is competitive to the best dependency based and the best window based models on both test sets. For each dataset, * indicates a statistically significant (p < 0.05) improvement over the next best model in that column, using McNemar’s test (McNemar, 1947).

Figure 4: **Robustness to Noisy Dictionary** For most languages, BiSparse-Dep outperforms the corresponding best window based model on Hyper-Hypo, with increasingly lower quality dictionaries.

Figure 3: **Robustness to Small Corpus** For most languages, BiSparse-Dep outperforms the corresponding best window based model for each language on Hyper-Hypo, with about 40% of the monolingual corpora.



on the choice of the entailment scorer. _SLQS_ is based on the distributional informativeness hypothesis, which states that hypernyms are less "informative" than hyponyms, because they occur in more general contexts. The informativeness \(E_{u}\) of a word \(u\) is defined to be the median entropy of its top \(N\) dimensions, \(E_{u}=median^{N}_{k=1}H(c_{k})\), where \(H(c_{i})\) denotes the entropy of dimension \(c_{i}\). The _SLQS_ score for a pair \((u,v)\) is the relative difference in entropies,

\[SLQS(u\to v)=1-\frac{E_{u}}{E_{v}}\]

Recent work Shwartz et al. (2017) has found _SLQS_ to be more successful than other metrics in monolingual hypernym detection.

The trends observed in these experiments are consistent with those in SS6.1 - both BiSparseDep models still outperform window-based models. Also, the delexicalized version of BiSparseDep outperforms the window-based models, showing that the robust behavior demonstrated in SS6.3 is also invariant across metrics.

We also found that using _BalAPinc_ led to better results than _SLQS_ . For both BiSparseDep models, _BaLAPinc_ wins across the board for two languages (Russian and Chinese), and wins half the time for the other two languages compared to _SLQS_ . We leave detailed comparison of these and other scores to future work.

## 7 Conclusion

We introduced BiSparseDep, a new distributional approach for identifying cross-lingual hypernymy, based on cross-lingual embeddings derived from dependency contexts. We showed that using BiSparseDep is superior for the cross-lingual hypernymy detection task, when compared to standard window based models and a translation baseline. Further analysis also showed that BiSparseDep is robust to various low-resource settings. In principle, BiSparseDep can be used for any language that has a bilingual dictionary with English and a "related" language with a treebank. We also introduced crowd-sourced cross-lingual hypernymy datasets for four languages for future evaluations.

Our approach has the potential to complement existing work on creating cross-lingual ontologies such as BabelNet and the Open Multilingual Wordnet, which are noisy because they are compiled semi-automatically, and have limited language coverage. In general, distributional approaches can help refine ontology construction for any language where sufficient resources are available.

It remains to be seen how our approach performs for other language pairs beyond simluated low-resource settings. We anticipate that replacing our delexicalized parser with more sophisticated transfer strategies Rasooli and Collins (2017); Aufrant et al. (2016) might be beneficial in such settings.While our delexicalized parsing based approach exhibits robustness, it can benefit from more sophisticated approaches for transfer parsing Rasooli and Collins (2017); Aufrant et al. (2016) to improve parser performance. We aim to explore these and other directions in the future.

#### Acknowledgments

The authors would like to thank the members of the CLIP lab at the University of Maryland, members of the Cognitive Computation Group at the University of Pennsylvania, and the anonymous reviewers from EMNLP/CoNLL 2017 and NAACL 2018 for their constructive feedback. YV and MC were funded in part by research awards from Amazon, Google, and the Clare Boothe Luce Foundation. SU and DR were supported by Contract HR0011-15-2-0025 with the US Defense Advanced Research Projects Agency (DARPA).

## References

* Ammar et al. (2016) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016. Many languages, one parser. _Transactions of the Association for Computational Linguistics_ 4:431-444.
* Aufrant et al. (2016) Lauriane Aufrant, Guillaume Wisniewski, and Francois Yvon. 2016. Zero-resource dependency parsing: Boosting delexelized cross-lingual transfer with linguistic knowledge. In _Proc. of COLING_. The COLING 2016 Organizing Committee, Osaka, Japan, pages 119-130. [http://aclweb.org/anthology/C16-1012](http://aclweb.org/anthology/C16-1012).
* Bansal et al. (2014) Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In _Proc. of ACL_. Association for Computational Linguistics, Baltimore, Maryland, pages 809-815. [http://www.aclweb.org/anthology/P14-2131](http://www.aclweb.org/anthology/P14-2131).
* Baroni et al. (2012) Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In _Proc. of EACL_. Association for Computational Linguistics, pages 23-32. [http://www.aclweb.org/anthology/E12-1004](http://www.aclweb.org/anthology/E12-1004).
* Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: 

a collection of very large linguistically processed web-crawled corpora. _Proc. of LREC_ .
* Baroni and Lenci (2010) Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. _Computational Linguistics_ 36(4):673-721.
* Baroni and Lenci (2011) Marco Baroni and Alessandro Lenci. 2011. How we BLESed distributional semantic evaluation. In _Proceedings of the GEMS 2011 Workshop_.
* Bond and Foster (2013) Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual wordnet. In _Proc. of ACL_. Association for Computational Linguistics, pages 1352-1362. [http://www.aclweb.org/anthology/P13-1133](http://www.aclweb.org/anthology/P13-1133).
* Bullinaria and Levy (2007) John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. _Behavior Research Methods_ pages 510-526.
* Carmona and Riedel (2017) Vicente Ivan Sanchez Carmona and Sebastian Riedel. 2017. How well can we predict hypernyms from word embeddings? a dataset-centric analysis. In _Proc. of EACL_. Association for Computational Linguistics, Valencia, Spain, pages 401-407. [http://www.aclweb.org/anthology/E17-2064](http://www.aclweb.org/anthology/E17-2064).
* Chersoni et al. (2016) Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache, and Chu-Ren Huang. 2016. Representing verbs with rich contexts: an evaluation on verb similarity. In _Proc. of EMNLP_. Association for Computational Linguistics, Austin, Texas, pages 1967-1972. [https://aclweb.org/anthology/D16-1205](https://aclweb.org/anthology/D16-1205).
* Choi et al. (2015) Jinho D Choi, Joel R Tetreault, and Amanda Stent. 2015. It depends: Dependency parser comparison using a web-based evaluation tool. In _Proc. of ACL_. Association for Computational Linguistics, Beijing, China, pages 387-396. [http://www.aclweb.org/anthology/P15-1038](http://www.aclweb.org/anthology/P15-1038).
* Faruqui and Dyer (2014) Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In _Proc. of EACL_. Association for Computational Linguistics, Gothenburg, Sweden, pages 462-471. [http://www.aclweb.org/anthology/E14-1049](http://www.aclweb.org/anthology/E14-1049).
* Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. _Psychological bulletin_ 76(5):378.
* Fu et al. (2014) Ruiji Fu, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In _Proc. of ACL_. Association for Computational Linguistics, Baltimore, Maryland, pages 1199-1209. [http://www.aclweb.org/anthology/P14-1113](http://www.aclweb.org/anthology/P14-1113).
* Geffet and Dagan (2005) Maayan Geffet and Ido Dagan. 2005. The Distributional Inclusion Hypotheses and Lexical Entailment. In _Proc. of ACL_.
* Goldstein et al. (2014) Tom Goldstein, Christoph Studer, and Richard Barainiuk. 2014. A Field Guide to Forward-Backward Splitting with a FASTA Implementation. _arXiv eprint_ abs/1411.3.
* Golub and Kahan (1965) Gene Golub and William Kahan. 1965. Calculating the singular values and pseudo-inverse of a matrix. _Journal of the SIAM_ .
* Graff (2007) David Graff. 2007. Arabic gigaword 3rd edition, LDC2003T40. LDC, University of Pennsylvania.
* Hearst (1992) Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In _Proc. of COLING_. Association for Computational Linguistics, pages 539-545. [https://doi.org/10.3115/992133.992154](https://doi.org/10.3115/992133.992154).
* Hill et al. (2014) Felix Hill, Roi Reichart, and Anna Karhonen. 2014. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _arXiv preprint arXiv:1408.3456_ .
* Moehn (2005) Philipp Moehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _Proc. of MT Summit_.
* Kotterman et al. (2009) Lili Kotterman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geftet. 2009. Directional distributional similarity for lexical expansion. In _Proc. of the ACL-IJCNLP_.
* Kotlerman et al. (2010) Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geftet. 2010. Directional distributional similarity for lexical inference. _Natural Language Engineering_ .
* Lenci and Benotto (2012) Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In _Proc. of the 6th Workshop on Semantic Evaluation_.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In _Proc. of ACL_. Association for Computational Linguistics, Baltimore, Maryland, pages 302--308. [http://www.aclweb.org/anthology/P14-2050](http://www.aclweb.org/anthology/P14-2050).
* Lin (1998) Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In _Proc. of ACL_. Association for Computational Linguistics, pages 768-774. [https://doi.org/10.3115/980691.980696](https://doi.org/10.3115/980691.980696).
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In _Proc. of the Workshop on Vector Space Modeling for NLP_.
* Martins et al. (2013) Andre Martins, Miguel Almeida, and Noah A. Smith. 2013. Turning on the Turbo: Fast Third-Order Non-Projective Turbo Parsers. In _Proc. of ACL_. Association for Computational Linguistics, pages 617-622. [http://www.aclweb.org/anthology/P13-2109](http://www.aclweb.org/anthology/P13-2109).
* McDonald et al. (2015) Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzeman Ganchev, Keith Hall, Slav Petrov, Hao 

Zhang, Oscar Tackstrom, Claudia Bedini, Noria Bertomeu Castello, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In _Proc. of ACL_.
* McDonald et al. (2011) Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In _Proc. of EMNLP_. Association for Computational Linguistics, pages 62-72. [http://www.aclweb.org/anthology/D11-1006](http://www.aclweb.org/anthology/D11-1006).
* McNemar (1947) Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. _Psychometrika_ 12(2):153-157.
* Melamud et al. (2016) Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The Role of Context Types and Dimensionality in Learning Word Embeddings. In _Proc. of NAACL-HLT_. Association for Computational Linguistics, San Diego, California, pages 1030-1040. [http://www.aclweb.org/anthology/N16-1118](http://www.aclweb.org/anthology/N16-1118).
* Munteanu and Marcu (2007) Dragos Stefan Munteanu and Daniel Marcu. 2007. ISI Arabic-English Automatically Extracted Parallel Text LDC2007T08. LDC, University of Pennsylvania.
* Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012. BugelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. _Artificial Intelligence_ .
* Negri et al. (2012) Matteo Negri, Alessandro Marchetti, Yashar Mehdad, Luisa Bentivogli, and Danilo Giampiccolo. 2012. Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization .
* Negri et al. (2013) Matteo Negri, Alessandro Marchetti, Yashar Mehdad, Luisa Bentivogli, and Danilo Giampiccolo. 2013. Semeval-2013 Task 8: Cross-lingual Textual Entailment for Content Synchronization .
* Pado et al. (2009) Sebastian Pado, Michel Galley, Dan Jurafsky, and Chris Manning. 2009. Robust machine translation evaluation with entailment features. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1_. Association for Computational Linguistics, pages 297-305.
* Pado and Lapata (2007) Sebastian Pado and Mirella Lapata. 2007. Dependency-based construction of semantic space models. _Computational Linguistics_ .
* Parker (2011) Robert Parker. 2011. Chinese Gigaword 5th Edition, LDC2011T13. LDC, University of Pennsylvania.
* Pavlick et al. (2015) Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. _Proceedings of ACL-IJCNLP 2015_ pages 425-430.
* Rasooli and Collins (2017) Mohammad Sadegh Rasooli and Michael Collins. 2017. Cross-lingual syntactic transfer with limited resources. _Transactions of the Association for Computational Linguistics_ 5:279-293. [https://transacl.org/ois/index.php/tacl/article/view/922](https://transacl.org/ois/index.php/tacl/article/view/922).
* Rasooli and Tetreault (2015) Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara Parser: A Fast and Accurate Dependencyarser. _CoRR_ abs/1503.06733.
* Roller and Erk (2016) Stephen Roller and Katrin Erk. 2016. Relations such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment. In _Proc. of EMNLP_. Association for Computational Linguistics, pages 2163-2172. [http://www.aclweb.org/anthology/D16-1234](http://www.aclweb.org/anthology/D16-1234).
* Santus et al. (2014) Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In _Proc. of EACL_. Association for Computational Linguistics, Gothenburg, Sweden, pages 38-42. [http://www.aclweb.org/anthology/E14-4008](http://www.aclweb.org/anthology/E14-4008).
* Santus et al. (2015) Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In _Proceedings of the 4th Workshop on Linked Data in Linguistics (LDL-2015)_. Association for Computational Linguistics, Beijing, China, pages 64-69. [http://www.aclweb.org/anthology/W15-4208](http://www.aclweb.org/anthology/W15-4208).
* Shwartz et al. (2017) Vered Shwartz, Enrico Santus, and Dominik Schlechtweg. 2017. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In _Proc. of EACL_. Association for Computational Linguistics, Valencia, Spain, pages 65-75. [http://www.aclweb.org/anthology/E17-1007](http://www.aclweb.org/anthology/E17-1007).
* Tiedemann (2012) Jorg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In _Proc. of LREC_.
* Turney and Mohammad (2015) Peter D Turney and Saif M Mohammad. 2015. Experiments with three approaches to recognizing lexical entailment. _Natural Language Engineering_ .
* Vossen et al. (2015) Piek Vossen, Egoitz Laparra, Itziar Aldabe, and German Rigau. 2015. Interoperability of cross-lingual and cross-document event detection. In _Proc. of the 3rd Workshop on EVENTS at the NAACL-HLT_.
* Vulic (2017) Ivan Vulic. 2017. Cross-lingual syntactically informed distributed word representations. In _Proc. of EACL_. Association for Computational Linguistics, Valencia, Spain, pages 408-414. [http://www.aclweb.org/anthology/E17-2065](http://www.aclweb.org/anthology/E17-2065).
* Vulic and Moens (2015) Ivan Vulic and Marie-Francine Moens. 2015. Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In _Proc. of ACL_. Association for Computational Linguistics, Beijing, China, pages 719-725. [http://www.aclweb.org/anthology/P15-2118](http://www.aclweb.org/anthology/P15-2118) 

Yogarshi Vyas and Marine Carpuat. 2016 Sparse Bilingual Word Representations for Cross-lingual Lexical Entailment. Association for Computational Linguistics, San Diego, California, pages 1187-1197. [http://www.aclweb.org/anthology/N16-1142](http://www.aclweb.org/anthology/N16-1142).
* Weeds and Wier (2003) Julie Weeds and David Wier. 2003. A general framework for distributional similarity. In _Proc. of EMNLP_. [http://www.aclweb.org/anthology/W03-1011](http://www.aclweb.org/anthology/W03-1011).
* Zeman and Resnik (2008) Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In _IJCNLP_. [http://www.aclweb.org/anthology/I08-3008](http://www.aclweb.org/anthology/I08-3008).