Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation

Abstract: Existing approaches to automatic VerbNet-style verb classification are heavily dependent on feature engineering and therefore limited to languages with mature NLP pipelines. In this work, we propose a novel cross-lingual transfer method for inducing VerbNets for multiple languages. To the best of our knowledge, this is the first study which demonstrates how the architectures for learning word embeddings can be applied to this challenging syntactic-semantic task. Our method uses cross-lingual translation pairs to tie each of the six target languages into a bilingual vector space with English, jointly specialising the representations to encode the relational information from English VerbNet. A standard clustering algorithm is then run on top of the VerbNet-specialised representations, using vector dimensions as features for learning verb classes. Our results show that the proposed cross-lingual transfer approach sets new state-of-the-art verb classification performance across all six target languages explored in this work.

Playing a key role in conveying the meaning of a sentence, verbs are famously complex. They display a wide range of syntactic-semantic behaviour, expressing the semantics of an event as well as relational information among its participants BIBREF0 , BIBREF1 , BIBREF2 .
Lexical resources which capture the variability of verbs are instrumental for many Natural Language Processing (NLP) applications. One of the richest verb resources currently available for English is VerbNet BIBREF3 , BIBREF4 . Based on the work of Levin Levin:1993book, this largely hand-crafted taxonomy organises verbs into classes on the basis of their shared syntactic-semantic behaviour. Providing a useful level of generalisation for many NLP tasks, VerbNet has been used to support semantic role labelling BIBREF5 , BIBREF6 , semantic parsing BIBREF7 , word sense disambiguation BIBREF8 , discourse parsing BIBREF9 , information extraction BIBREF10 , text mining applications BIBREF11 , BIBREF12 , research into human language acquisition BIBREF13 , and other tasks.
This benefit for English NLP has motivated the development of VerbNets for languages such as Spanish and Catalan BIBREF14 , Czech BIBREF15 , and Mandarin BIBREF16 . However, end-to-end manual resource development using Levin's methodology is extremely time consuming, even when supported by translations of English VerbNet classes to other languages BIBREF17 , BIBREF18 . Approaches which aim to learn verb classes automatically offer an attractive alternative. However, existing methods rely on carefully engineered features that are extracted using sophisticated language-specific resources BIBREF19 , BIBREF17 , BIBREF20 , ranging from accurate parsers to pre-compiled subcategorisation frames BIBREF21 , BIBREF22 , BIBREF23 . Such methods are limited to a small set of resource-rich languages.
It has been argued that VerbNet-style classification has a strong cross-lingual element BIBREF24 , BIBREF2 . In support of this argument, Majewska:2017lre have shown that English VerbNet has high translatability across different, even typologically diverse languages. Based on this finding, we propose an automatic approach which exploits readily available annotations for English to facilitate efficient, large-scale development of VerbNets for a wide set of target languages.
Recently, unsupervised methods for inducing distributed word vector space representations or word embeddings BIBREF25 have been successfully applied to a plethora of NLP tasks BIBREF26 , BIBREF27 , BIBREF28 . These methods offer an elegant way to learn directly from large corpora, bypassing the feature engineering step and the dependence on mature NLP pipelines (e.g., POS taggers, parsers, extraction of subcategorisation frames). In this work, we demonstrate how these models can be used to support automatic verb class induction. Moreover, we show that these models offer the means to exploit inherent cross-lingual links in VerbNet-style classification in order to guide the development of new classifications for resource-lean languages. To the best of our knowledge, this proposition has not been investigated in previous work.
There has been little work on assessing the suitability of embeddings for capturing rich syntactic-semantic phenomena. One challenge is their reliance on the distributional hypothesis BIBREF29 , which coalesces fine-grained syntactic-semantic relations between words into a broad relation of semantic relatedness (e.g., coffee:cup) BIBREF30 , BIBREF31 . This property has an adverse effect when word embeddings are used in downstream tasks such as spoken language understanding BIBREF32 , BIBREF33 or dialogue state tracking BIBREF34 , BIBREF35 . It could have a similar effect on verb classification, which relies on the similarity in syntactic-semantic properties of verbs within a class. In summary, we explore three important questions in this paper:
(Q1) Given their fundamental dependence on the distributional hypothesis, to what extent can unsupervised methods for inducing vector spaces facilitate the automatic induction of VerbNet-style verb classes across different languages?
(Q2) Can one boost verb classification for lower-resource languages by exploiting general-purpose cross-lingual resources such as BabelNet BIBREF36 , BIBREF37 or bilingual dictionaries such as PanLex BIBREF38 to construct better word vector spaces for these languages?
(Q3) Based on the stipulated cross-linguistic validity of VerbNet-style classification, can one exploit rich sets of readily available annotations in one language (e.g., the full English VerbNet) to automatically bootstrap the creation of VerbNets for other languages? In other words, is it possible to exploit a cross-lingual vector space to transfer VerbNet knowledge from a resource-rich to a resource-lean language?
To investigate Q1, we induce standard distributional vector spaces BIBREF39 , BIBREF40 from large monolingual corpora in English and six target languages. As expected, the results obtained with this straightforward approach show positive trends, but at the same time reveal its limitations for all the languages involved. Therefore, the focus of our work shifts to Q2 and Q3. The problem of inducing VerbNet-oriented embeddings is framed as vector space specialisation using the available external resources: BabelNet or PanLex, and (English) VerbNet. Formalised as an instance of post-processing semantic specialisation approaches BIBREF41 , BIBREF34 , our procedure is steered by two sets of linguistic constraints: 1) cross-lingual (translation) links between languages extracted from BabelNet (targeting Q2); and 2) the available VerbNet annotations for a resource-rich language. The two sets of constraints jointly target Q3.
The main goal of vector space specialisation is to pull examples standing in desirable relations, as described by the constraints, closer together in the transformed vector space. The specialisation process can capitalise on the knowledge of VerbNet relations in the source language (English) by using translation pairs to transfer that knowledge to each of the target languages. By constructing shared bilingual vector spaces, our method facilitates the transfer of semantic relations derived from VerbNet to the vector spaces of resource-lean target languages. This idea is illustrated by Fig. FIGREF2 .
Our results indicate that cross-lingual connections yield improved verb classes across all six target languages (thus answering Q2). Moreover, a consistent and significant boost in verb classification performance is achieved by propagating the VerbNet-style information from the source language (English) to any other target language (e.g., Italian, Croatian, Polish, Finnish) for which no VerbNet-style information is available during the fine-tuning process (thus answering Q3). We report state-of-the-art verb classification performance for all six languages in our experiments. For instance, we improve the state-of-the-art F-1 score from prior work from 0.55 to 0.79 for French, and from 0.43 to 0.74 for Brazilian Portuguese.
Our departure point is a state-of-the-art specialisation model for fine-tuning vector spaces termed Paragram BIBREF49 . The Paragram procedure injects similarity constraints between word pairs in order to make their vector space representations more similar; we term these the Attract constraints. Let INLINEFORM0 be the vocabulary consisting of the source language and target language vocabularies INLINEFORM1 and INLINEFORM2 , respectively. Let INLINEFORM3 be the set of word pairs standing in desirable lexical relations; these include: 1) verb pairs from the same VerbNet class (e.g. (en_transport, en_transfer) from verb class send-11.1); and 2) the cross-lingual synonymy pairs (e.g. (en_peace, fi_rauha)). Given the initial distributional space and collections of such Attract pairs INLINEFORM4 , the model gradually modifies the space to bring the designated word vectors closer together, working in mini-batches of size INLINEFORM5 . The method's cost function can be expressed as:
 DISPLAYFORM0 
The first term of the method's cost function (i.e., INLINEFORM0 ) pulls the Attract examples INLINEFORM1 closer together (see Fig. FIGREF2 for an illustration). INLINEFORM2 refers to the current mini-batch of Attract constraints. This term is expressed as follows:
 DISPLAYFORM0 
 INLINEFORM0 is the standard rectified linear unit or the hinge loss function BIBREF50 , BIBREF51 . INLINEFORM1 is the “attract” margin: it determines how much vectors of words from Attract constraints should be closer to each other than to their negative examples. The negative example INLINEFORM2 for each word INLINEFORM3 in any Attract pair is always the vector closest to INLINEFORM4 taken from the pairs in the current mini-batch, distinct from the other word paired with INLINEFORM5 , and INLINEFORM6 itself.
The second INLINEFORM0 term is the regularisation which aims to retain the semantic information encoded in the initial distributional space as long as this information does not contradict the used Attract constraints. Let INLINEFORM1 refer to the initial distributional vector of the word INLINEFORM2 and let INLINEFORM3 be the set of all word vectors present in the given mini-batch. If INLINEFORM4 denotes the L2 regularisation constant, this term can be expressed as:
 DISPLAYFORM0 
The fine-tuning procedure effectively blends the knowledge from external resources (i.e., the input Attract set of constraints) with distributional information extracted directly from large corpora. We show how to propagate annotations from a knowledge source such as VerbNet from source to target by combining two types of constraints within the specialisation framework: a) cross-lingual (translation) links between languages, and b) available VerbNet annotations in a resource-rich language transformed into pairwise constraints. Cross-lingual constraints such as (pl_wojna, it_guerra) are extracted from BabelNet BIBREF36 , a large-scale resource which groups words into cross-lingual babel synsets (and is currently available for 271 languages). The wide and steadily growing coverage of languages in BabelNet means that our proposed framework promises to support the transfer of VerbNet-style information to numerous target languages (with increasingly high accuracy).
To establish that the proposed transfer approach is in fact independent of the chosen cross-lingual information source, we also experiment with another cross-lingual dictionary: PanLex BIBREF38 , which was used in prior work on cross-lingual word vector spaces BIBREF52 , BIBREF53 . This dictionary currently covers around 1,300 language varieties with over 12 million expressions, thus offering support also for low-resource transfer settings.
VerbNet constraints are extracted from the English VerbNet class structure in a straightforward manner. For each class INLINEFORM0 from the 273 VerbNet classes, we simply take the set of all INLINEFORM1 verbs INLINEFORM2 associated with that class, including its subclasses, and generate all unique pairs INLINEFORM3 so that INLINEFORM4 and INLINEFORM5 . Example VerbNet pairwise constraints are shown in Tab. TABREF15 . Note that VerbNet classes in practice contain verb instances standing in a variety of lexical relations, including synonyms, antonyms, troponyms, hypernyms, and the class membership is determined on the basis of connections between the syntactic patterns and the underlying semantic relations BIBREF54 , BIBREF55 .
Given the initial distributional or specialised collection of target language vectors INLINEFORM0 , we apply an off-the-shelf clustering algorithm on top of these vectors in order to group verbs into classes. Following prior work BIBREF56 , BIBREF57 , BIBREF17 , we employ the MNCut spectral clustering algorithm BIBREF58 , which has wide applicability in similar NLP tasks which involve high-dimensional feature spaces BIBREF59 , BIBREF60 , BIBREF18 . Again, following prior work BIBREF17 , BIBREF61 , we estimate the number of clusters INLINEFORM1 using the self-tuning method of Zelnik:2004nips. This algorithm finds the optimal number by minimising a cost function based on the eigenvector structure of the word similarity matrix. We refer the reader to the relevant literature for further details.
Cross-Lingual Transfer Model F-1 verb classification scores for the six target languages with different sets of constraints are summarised in Fig. FIGREF29 . We can draw several interesting conclusions. First, the strongest results on average are obtained with the model which transfers the VerbNet knowledge from English (as a resource-rich language) to the resource-lean target language (providing an answer to question Q3, Sect. SECREF1 ). These improvements are visible across all target languages, empirically demonstrating the cross-lingual nature of VerbNet-style classifications. Second, using cross-lingual constraints alone (XLing) yields strong gains over initial distributional spaces (answering Q1 and Q2). Fig. FIGREF29 also shows that cross-lingual similarity constraints are more beneficial than the monolingual ones, despite a larger total number of the monolingual constraints in each language (see Tab. TABREF18 ). This suggests that such cross-lingual similarity links are strong implicit indicators of class membership. Namely, target language words which map to the same source language word are likely to be synonyms and consequently end up in the same verb class in the target language. However, the cross-lingual links are even more useful as means for transferring the VerbNet knowledge, as evidenced by additional gains with XLing+VerbNet-EN.
The absolute classification scores are the lowest for the two Slavic languages: pl and hr. This may be partially explained by the lowest number of cross-lingual constraints for the two languages covering only a subset of their entire vocabularies (see Tab. TABREF18 and compare the total number of constraints for hr and pl to the numbers for e.g. fi or fr). Another reason for weaker performance of these two languages could be their rich morphology, which induces data sparsity both in the initial vector space estimation and in the coverage of constraints.
This work has proven the potential of transferring lexical resources from resource-rich to resource-poor languages using general-purpose cross-lingual dictionaries and bilingual vector spaces as means of transfer within a semantic specialisation framework. However, we believe that the proposed basic framework may be upgraded and extended across several research paths in future work.
First, in the current work we have operated with standard single-sense/single-prototype representations, thus effectively disregarding the problem of verb polysemy. While several polysemy-aware verb classification models for English were developed recently BIBREF79 , BIBREF80 , the current lack of polysemy-aware evaluation sets in other languages impedes this line of research. Evaluation issues aside, one idea for future work is to use the Attract-Repel specialisation framework for sense-aware cross-lingual transfer relying on recently developed multi-sense/prototype word representations BIBREF81 , BIBREF82 .
Another challenge is to apply the idea from this work to enable cross-lingual transfer of other structured lexical resources available in English such as FrameNet BIBREF44 , PropBank BIBREF45 , and VerbKB BIBREF83 . Other potential research avenues include porting the approach to other typologically diverse languages and truly low-resource settings (e.g., with only limited amounts of parallel data), as well as experiments with other distributional spaces, e.g. BIBREF84 . Further refinements of the specialisation and clustering algorithms may also result in improved verb class induction.
We have presented a novel cross-lingual transfer model which enables the automatic induction of VerbNet-style verb classifications across multiple languages. The transfer is based on a word vector space specialisation framework, utilised to directly model the assumption of cross-linguistic validity of VerbNet-style classifications. Our results indicate strong improvements in verb classification accuracy across all six target languages explored. All automatically induced VerbNets are available at:
github.com/cambridgeltl/verbnets.
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909). The authors are grateful to the entire LEXICAL team, especially to Roi Reichart, and also to the three anonymous reviewers for their helpful and constructive suggestions.

Figure and table captions:
Figure 1: Transferring VerbNet information from a resource-rich to a resource-lean language through a word vector space: an English→ French toy example. Representations of words described by two types of ATTRACT constraints are pulled closer together in the joint vector space. (1) Monolingual pairwise constraints in English (e.g., (en_ruin, en_shatter), (en_destroy, en_undo)) reflect the EN VerbNet structure and are generated from the readily available verb classification in English (solid lines). They are used to specialise the distributional EN vector subspace for the VerbNet relation. (2) Cross-lingual English-French pairwise constraints (extracted from BabelNet) describe cross-lingual synonyms (i.e., translation links) such as (en_ruin, fr_ruiner) or (en_shatter, fr_fracasser) (dashed lines). The post-processing fine-tuning specialisation procedure based on (1) and (2) effectively transforms the initial distributional French vector subspace to also emphasise the VerbNet-style structure, facilitating the induction of verb classes in French.
Table 1: Example pairwise ATTRACT constraints extracted from three VerbNet classes in English.
Table 2: Statistics of the experimental setup for each target language: training/test data and constraints. Coverage refers to the percentage of test verbs represented in the target language vocabularies.
Figure 2: F-1 scores in six target languages using the post-processing specialisation procedure from Sect. 2.1 and different sets of constraints: Distributional refers to the initial vector space in each target language; Mono-Syn is the vector space tuned using monolingual synonymy constraints from BabelNet; XLing uses cross-lingual EN-TARGET constraints from BabelNet (TARGET refers to any of the six target languages); XLing+VerbNet-EN is a fine-tuned vector space which uses both cross-lingual EN-TARGET constraints plus EN VerbNet constraints. Results are provided with (a) SGNS-BOW2 and (b) SGNS-DEPS source vector space in English for the XLing and XLing+VerbNet variants, see Sect. 3.
Table 3: The effect of multilingual vector space specialisation. Results are reported for FR and IT using: a) cross-lingual constraints only (XLing); and b) the VerbNet transfer model (XLing+VN).
Table 4: Comparison of verb classification (VC) and verb semantic similarity (Sim) for English. VC is measured on the EN test set of Sun et al. (2008). Sim is measured on SimVerb-3500 (Gerz et al., 2016). The scores are Spearman’s ρ correlation scores. EN-Dist is the initial distributional English vector space: SGNS-BOW2; EN-VN is the same space transformed using monolingual EN VerbNet constraints only, an upper bound for the specialisation-based approach in EN.
Figure 3: F-1 scores when PanLex is used as the source of cross-lingual ATTRACT constraints (instead of BabelNet). EN Vectors: SGNS-BOW2.

Questions about this paper:
What are the six target languages?

Answers:
Answer with content missing: (3 Experimental Setup) We experiment with six target languages: French (FR), Brazilian Portuguese (PT), Italian (IT), Polish (PL), Croatian (HR), and Finnish (FI).