character and word n-gram similarity. The outcome of these experiments on English abbreviations showed that character and word n-gram features contribute the most to the final result. Work on monolingual clustering of acronym variants outside the biomedical domain and for altogether 22 different languages was carried out in Ehrmann et al. (2013). Ehrmann’s approach is based on hierarchical group-average clustering, where cluster homogeneity is set using an empirically determined threshold. The clustering depends on a pair-wise string similarity between expansions, using a normalised Levenshtein edit distance.

To the best of our knowledge, no work has been carried out for acronym clustering across languages. What comes closest to this or, more exactly, to its result, are multilingual lexical resources such as BabelNet (Navigli & Ponzetto 2012) or YAGO (Hoffart et al. 2013). Automatically built based on the mapping between WordNet and Wikipedia (and other resources), these resources provide (among others) multilingual variants of expansions for specific acronyms. They are inherited from cross-lingual and cross-script links provided in Wikipedia. In contrast, the work presented here starts from raw data extracted from real-life texts.

As regards learning resources for the recognition and classification of named entities and domain-specific multi-word expressions, a vast bulk of research has been reported on using weakly-supervised approaches. These are based, in particular, on the bootstrapping paradigm in which, starting from an initial set of annotated examples (or seeds), the learning process proceeds without further supervision, until a convergence criterion is reached. Some examples of the work in this field is presented in Riloff (1996); Collins & Singer (1999), and Yangarber et al. (2002).

With the emergence of large-scale knowledge bases and the availability of web-scale corpora, numerous efforts on exploiting such resources for developing named entity recognition and classification tools have been reported. For instance, Nothman et al. (2013) reports on a multilingual NER approach based on using Wikipedia links for automatically annotating a huge corpus for training purposes, whereas Downey et al. (2007) presents a novel method for detecting complex (multi-word) named entities using solely capitalisation information and n-gram statistics over a Web corpus. This approach outperformed standard supervised and semi-supervised approaches for named-entity recognition in cases of complex names of types not known in advance.

Our contribution complements prior work and focuses on exploiting the vast number of named entities contained in BabelNet (Navigli & Ponzetto 2012) for learning structurally simple and linguistically unsophisticated patterns for the recognition of multi-word named entities in various languages.
## 3 Creation of the multilingual MWE entity resource

In this section, we describe completed work (Jacquet et al. 2016) on recognising MWE entities and their corresponding acronyms in large volumes of text in 22 different languages, on identifying monolingual variants for the same entity and on linking the equivalent groups of variants across all languages. Figure 1 illustrates that task with an example of cross-lingual linking, which shows that we can neither assume that entities across languages have the same acronym, nor can we assume that the same acronym (within the same or across languages) refers to only one entity. The result of this work is a collection of currently 64,000 MWE entities plus their 600,000 multilingual lexical variants.

<figure><img src="image_95.png" /><figcaption>Figure 1: Example of multilingual MWE entity linking</figcaption></figure>

### 3.1 Starting point

The starting point of our work is a large set of multi-word entities and their corresponding acronyms in 22 Roman-script languages (Ehrmann et al. 2013). These acronym/expansion pairs were extracted from the news stream analysed by the EMM processing chain by applying patterns similar to those proposed by Schwartz & Hearst (2003). In a nutshell, the algorithm collects acronym/expansion pairs (such as *expansion* (*acronym*) and *acronym* (*expansion*)) by identifying short strings within parenthesis, along with candidate expansions in a side-window of a limited length. A filtering step is then applied, with the following main constraints: the first letter of the acronym must be upper-cased, and the length of the expansion must be smaller than (a) twice as many words as there
are characters in the acronym, or (b) the number of characters in the acronym plus five words, whichever is the smaller (i.e. min(|A| + 5, |A| * 2) words, with |A| being the number of characters of the acronym). We refer the reader to Schwartz & Hearst (2003) for more details. This process resulted in the extraction of 1.7 million expansions for 0.4 million different acronyms.

Applied on news articles, this method identified acronym/expansion pairs referring mostly to organisation names (e.g., *CP – Communist Party*), but also events (*WW2 – World War II*), names of drugs or of vaccines (*MMR – measles, mumps, rubella*), organisation types (*NGO – non-governmental organisation*), job titles (*MEP – Member of Parliament*), physical measurement units (*kmh – kilometres per hour*), and more. As one of the next steps, we will work on categorising the acronym/expansion pairs into various semantic categories.

To automatically determine which of the expansions are lexical variants of the same conceptual entity, a clustering step was carried out, on the basis of expansions having the same language and the same acronym. This monolingual clustering, based on a pair-wise string similarity, allowed to distinguish between sets of conceptually related expansions, such as those referring to the *International Space Station* and those referring to the *Institute for Security of Studies*, both clusters having the acronym *ISS* (cf. English part of Figure 1). Evaluated over the 10 most covered languages, this monolingual clustering has a micro-average precision of 95.2% (Jacquet et al. 2014).

Out of this monolingual clustering step, we selected only clusters having at least four expansions, resulting in 81,000 monolingual clusters with an average of 7.5 expansions per cluster, the biggest one having 232 expansions.

Based on this data, the objective is to go a step further by identifying cross-lingual multi-word entity lexical variants. More specifically, the goal is to link multilingual expansions referring to the same entity across languages and regardless of their acronyms. To this end, we leverage the previously computed monolingual clusters and attempt to link them across languages. Considering the previous example with the entity *International Space Station* (cf. Figure 1), this results in aggregating the monolingual clusters *SSI – Station spatiale internationale* (French), *ISS – International Space Station* (English) and *EEI – Estación Espacial* (Spanish). Additionally to linking expansions across languages and independently from their acronym, cross-lingual cluster aggregation can also revise monolingual clusters by aggregating those conceptually related but isolated because of their acronyms (both pairs *IMF – International Monetary Fund* and *FMI – Fondo Monetario Internazionale* occur in Italian texts).
## 3.2 Approach

Cluster aggregation can be cast as the problem of identifying connected components of a graph, where monolingual clusters represent vertices and where edges need to be computed. This section describes different cross-lingual aggregation strategies tested in our experiments (cf. Section 3.3) to link sets of monolingual clusters across languages.

### 3.2.1 Cluster aggregation based on common expansions

The most straightforward solution to link related acronyms in different languages (hereafter ExpAgg) is to merge those clusters that have more than *n* expansion forms in common, independently of whether their acronyms are identical or not (in our experiments, *n* was set to 1). This aggregation has been applied both to improve monolingual clusters (cf. IMF vs FMI case mentioned at the end of Section 3.1) and to aggregate clusters across languages.

### 3.2.2 Cluster aggregation based on tokens

#### 3.2.2.1 Cluster representation

For the two following aggregation strategies, monolingual clusters are no longer represented by vectors of expansions, but by a vector of all individual tokens appearing in the expansions.

C is the resulting (|C| × |T|) cluster-token matrix where $c_i : i = 1, ..., |C|$ is a monolingual cluster, and $t_j : j = 1, ..., |T|$ is a token. T contains all the tokens across languages which appear at least once in an expansion. If a token is present in different languages, such as *place* in English and *place* in French, it corresponds to different tokens in T.

Each token has its own importance to describe a cluster. In order to compare two clusters on the basis of their most relevant tokens, we consider the tf-idf value of each token $t_j$ where, in our context, each cluster $c_i$ is seen as a document and the whole set of clusters C as a corpus:

$$ (1) \qquad C(c_i, t_j) = tf(t_j, c_i) \times idf(t_j, C) $$

#### 3.2.2.2 Cluster aggregation based on similar tokens

This aggregation (hereafter TokAgg) addresses cases where monolingual clusters do not have identical expansions across languages, but they have a significant amount of highly similar tokens.
<table><caption>Table 1: Example of clusters aggregated on the basis of similar tokens</caption><thead><tr><th>Clusters</th><th>Expansion</th><th>Acronym</th><th>Language</th></tr></thead><tbody><tr><td>cluster 1</td><td>Social-Democratic Party<br>Social Democratic Party</td><td>SDP</td><td>en</td></tr><tr><td>cluster 2</td><td>Partito Social-Democratico<br>Partito di socialdemocratico<br>Partito socialdemocratico</td><td>PSD</td><td>it</td></tr></tbody></table>

We compute the matrix (|T| × |T|), hereafter *InvEdit*, which corresponds to the inverse of the normalised Levenshtein edit distance where $t_i : i = 1, ..., |T|$ and $t_j : j = 1, ..., |T|$ are tokens from all the addressed languages:

$$ (2) \qquad \mathrm{InvEdit}(t_i, t_j) = 1 - \frac{\mathrm{Lev}(t_i, t_j)}{\max(|t_i|, |t_j|)} $$

$\mathrm{Lev}(t_i, t_j)$ is the Levenshtein edit-distance between $t_i$ and $t_j$, and $|t_i|$ and $|t_j|$ are respectively the length of the tokens $t_i$ and $t_j$. We filter InvEdit using a threshold $\delta$ as follows:

$$ (3) \qquad \mathrm{InvEdit}(t_i, t_j, \delta) = \begin{cases} \mathrm{InvEdit}(t_i, t_j) & : \mathrm{InvEdit}(t_i, t_j) \ge \delta \\ 0 & : \mathrm{InvEdit}(t_i, t_j) < \delta \end{cases} $$

In this case, if $\delta = 1$, InvEdit only contains values for exact matching tokens. This matrix is then used to enrich the monolingual cluster representation. Given two languages $l_1$ and $l_2$, the corresponding monolingual clusters $C_{l_1}$ and $C_{l_2}$ do not have common tokens since in T tokens are language-dependent. The InvEdit matrix is used to identify common or similar tokens. We convert the obtained matrix $C_{\_Tok_{l_1}}$ to a binary matrix:

$$ (4) \qquad C_{\_Tok_{l_1}}(c_i, t_j) = \begin{cases} 1 : C_{l_1}(c_i, t_j) \times \mathrm{InvEdit}(c_i, t_j, \delta) > 0 \\ 0 : \text{otherwise} \end{cases} $$

This aggregation is particularly useful when comparing clusters from similar languages. Table 1 illustrates such cases, with the English-Italian tokens *Party/Partito* and *Democratic/Democratico*. This representation can also benefit from
the fact that it is possible to find multi-word entities of a given language in texts in another language (especially with names of international organisations such as *European Space Agency* which can be found in German text).

### 3.2.2.3 Cluster aggregation based on translated tokens

<table><caption>Table 2: Example of clusters aggregated on the basis of translated tokens</caption><thead><tr><th>Clusters</th><th>Expansion</th><th>Acronym</th><th>Language</th></tr></thead><tbody><tr><td>cluster 1</td><td>Russian Academy of Sciences<br/>Russian Academy of Sciences</td><td>RAS</td><td>en</td></tr><tr><td>cluster 2</td><td>russischen Akademie der Wissenschaften<br/>Russischen Akademie für Wissenschaften<br/>Russische Akademie der Wissenschaften</td><td>RAW</td><td>de</td></tr></tbody></table>

However, many entities have different written forms across languages so that a string-based comparison of tokens is not successful. We therefore complement the cluster aggregation by using token translation probabilities (hereafter *TransTokAgg*).

They are produced using statistical translation models trained on parallel corpora built from Wikipedia, by making use of redirection tables (i.e. several written forms redirecting to a specific page/entity) and of interlingual links between pages (implementation details of translation models are provided in Section 3.3.3). In order to separate training and test data, any variant name from these Wikipedia tables matching with one of the 1.7 million expansions or 0.4 million acronyms is removed from the parallel corpora (see Section 3.3).

Let TransMod be the resulting ($|T| \times |T|$) translation model matrix where $t_i : i = 1, ..., |T|$ and $t_j : j = 1, ..., |T|$ are tokens. As for InvEdit matrix, we filter TransMod using a threshold $\beta$:

$$ (5) \quad \text{TransMod}(t_i, t_j, \beta) = \begin{cases} \text{TransMod}(t_i, t_j) & : \text{TransMod}(t_i, t_j) \ge \beta \\ 0 & : \text{TransMod}(t_i, t_j) < \beta \end{cases} $$

This matrix is then used to enrich the monolingual cluster representation. Given a language *l* and its corresponding monolingual clusters $C_l, C_{\text{TransTok}_l}$ corresponds to the binary extended matrix based on a given translation model:
$$ (6) \quad C_{\text{TransTok}_l}(c_i, t_j) = \begin{cases} 1 : C_l(c_i, t_j) \times \text{TransMod}(c_i, t_j, \beta) > 0 \\ 0 : \text{otherwise} \end{cases} $$

Table 2 illustrates a case of such cluster aggregation, thanks to a high score in the TransMod matrix between tokens *Science* in English and *Wissenschaften* in German.

### 3.2.3 Aggregation strategies

We formulate cluster linking as the task of identifying connected components in a graph, where monolingual clusters are vertices and where edges represent links of related clusters across languages. Clusters are linked if their similarity is above a certain threshold $\alpha$. During preliminary experiments, we had also tested *pure* clustering algorithms, but it turned out that the graph approach was more efficient.

For the last two cluster aggregation methods (TokAgg and TransTokAgg), we applied two similarity measures: cosine and ComMNZ. The latter is actually a data fusion algorithm (Fox & Shaw 1994) which we assimilate, in this context, to a similarity measure. This algorithm aims at measuring the similarity between two objects having multiple comparison criteria. Specifically, the overall similarity score between two objects is better when those objects have reasonable similarity scores for all criteria than when they have a very good similarity score for one criterion, and less good or no value for the others. In our case, it would promote the similarity between two clusters $c_i$ and $c_j$ if they have many similar or translated tokens $t_k$ with a reasonable similarity score, and it would decrease the similarity between two clusters $c_i$ and $c_j$ if they have few similar or translated tokens $t_k$ with a very high similarity score:

$$ (7) \quad \text{CombMNZ}(c_i, c_j) = \sum_{t_k \in c_j} \frac{C(c_i, t_k)}{\sum_{t_l \in c_i} C(c_i, t_l)} \times \sum_{t_k \in c_j} 1_{\{C(c_i, t_k) \neq 0\}} $$

## 3.3 Evaluation

### 3.3.1 Evaluation dataset

As described in Section 3.1, the starting point of our experiments is a set of 81,000 monolingual clusters with one acronym per cluster, an average of 7.5 expansions per cluster, many of them having few expansions, and the biggest 232 expansions.
We evaluate cross-lingual cluster aggregation against Wikipedia data exclud-
ing the part used for the translations models (see previous section). The gold stan-
dard corresponds to a set of Wikipedia redirection tables and interlingual link-
ing tables, where we consider Wikipedia entities/pages as cross-lingual classes.
Each class contains all the expressions listed in the redirection tables in all the
languages linked via the interlingual linking tables. Only classes having at least
two expansions were selected, resulting in a gold standard of 10,000 classes. Con-
sidering Wikipedia information as a gold standard is disputable. The interlingual
linkings should be reliable but this is less the case for the redirection tables. How-
ever, a manual evaluation of the redirection table quality shows that, in over 160
randomly extracted classes in 4 different languages (fr, en, de, it), 93.4% of the
forms were correct (Jacquet et al. 2014).

3.3.2 Parameters

Parameters have to be set with regards to, first, the thresholds $\delta$ and $\beta$ applied to filter out some similarity values in the above-mentioned token matrices ($C_{\_Tok}_1$ and $C_{\_TransTok}_1$) and, second, the threshold $\alpha$ applied to the aggregation strategies, i.e. the one above which clusters are aggregated.

With respect to cluster representations based on similar tokens $C_{\_Tok}_1$, the threshold $\delta$ should be high in order to consider two tokens as similar only if they are close in terms of edit distance. Regarding representations based on translated tokens $C_{\_TransTok}_1$, the threshold $\beta$ can be low since even a weak token similarity could be a relevant indicator at the cluster level. For our experiments, the values of $\delta$ and $\beta$ were fixed to 0.7 and 0.3 respectively.

Cluster aggregation is allowed when the cluster similarity (either in terms of cosine or CombMNZ) is above a certain threshold $\alpha$. We experimented with different values for $\alpha$, ranging from 0.7 to 1 (cf. Section 3.3.5).

This aggregation step is further regulated with the addition of the following
constraints: two clusters $c_1$ and $c_2$ are linked if their similarity is above $\alpha$ and if
$c_1$ is in the $k$ most similar clusters of $c_2$ or $c_2$ is in the $k$ most similar clusters of
$c_1$. This additional constraints allow to rule out clusters having a high similarity
with a lot of other clusters. This is the case for short and frequent expansions, e.g.,
*Olympic Committee* which is highly similar to a cluster containing expansions
such as *Olympic Organizing Committee* or to another containing *games organising committee*, but as well to clusters containing more specific expansions such
as *Vancouver Olympic Committee*. In our experiments, $k$ equals 3.
### 3.3.3 Translation models

Cluster representations based on translated tokens correspond to lexical conditional translation probabilities computed for three language pairs, between English and French, German and Italian. The translation models were trained on parallel corpora built from Wikipedia, by making use of redirection tables (i.e. several written forms redirecting to a specific page/entity) and of interlingual links between pages. More specifically, given an entity/page *p* and two redirection tables *rt*₁ and *rt*₂ in languages *l*₁ and *l*₂, each written form from *rt*₁ can be seen as a translation *t* of each written form from *rt*₂. For a given language pair, the corresponding parallel corpus is the concatenation of all translations *t* from all the entities/pages *p*.

These Wikipedia tables are also used for evaluation purposes (see Section 3.3.1). As a consequence, the 1.7 million expansions and 0.4 million acronyms on which the approach is applied were removed from the parallel corpora.

There were about 300,000 training examples for German-English and French-English, and about 170,000 for Italian-English. Word alignments with many-to-one links were generated using the unsupervised fastCourt tool (Dyer et al. 2013) in both directions and combined with the grow-diag-final-and-symmetrisation heuristic (Koehn et al. 2003). Lexical translation tables for the three language pairs in both directions were extracted with a tool from the Moses translation toolkit (Koehn et al. 2007). Tables contain maximum likelihood probability estimated for the conditional word translation probabilities *p*(En|{Fr, De, It}) and *p*({Fr, De, It}|En). Our TransMod matrix is constructed based on the concatenation of these tables.

### 3.3.4 Evaluation measures

Clusters are evaluated against the gold standard using micro-average precision and recall, adopting the mapping between identified clusters and gold standard clusters which maximised the *F*₁ measure. Micro-average precision (MAV-P) and recall (MAV-R) are defined as follows:

$$ (8) \qquad \text{MAV-P}(C) = \frac{\sum_{c \in C} \text{EXP}(c)_{\text{true}}}{\sum_{c \in C} \text{EXP}(c)_{\text{true}}} + \sum_{c \in C} \text{EXP}(c)_{\text{false}} $$

$$ (9) \qquad \text{MAV-R}(C) = \frac{\sum_{c \in C} \text{EXP}(c)_{\text{true}}}{\sum_{c \in C} \text{EXP}(c)_{\text{true}}} + \sum_{c \in C} \text{EXP}(c)_{\text{miss}} $$
where C is the set of produced clusters, $EXP(c)_{true}$ is the set of expansions in a cluster c which also appear in the corresponding class of the gold standard, and $EXP(c)_{false}$ is the set of expansions in a cluster c which do not appear in the gold standard.<fnref n="2" />

### 3.3.5 Results and discussion

<table><caption>Table 3: Cluster aggregation strategies for 3 language pairs</caption><thead><tr><th></th><th>MAV-P</th><th>MAV-R</th><th>F1</th></tr></thead><tbody><tr><td>Baseline</td><td>97.7%</td><td>51.5%</td><td>67.4%</td></tr><tr><td>Monolingual ExpAgg</td><td>96.8%</td><td>54.8%</td><td>69.4%</td></tr><tr><td>Multilingual ExpAgg</td><td>96.9%</td><td>65.7%</td><td>78.2%</td></tr><tr><td>Cosine measure</td><td></td><td></td><td></td></tr><tr><td>TokAgg</td><td>97.7%</td><td>52.5%</td><td>68.3%</td></tr><tr><td>TransTokAgg</td><td>97.6%</td><td>51.8%</td><td>67.7%</td></tr><tr><td>All aggregations</td><td><strong>95.5%</strong></td><td><strong>71.4%</strong></td><td><strong>81.6%</strong></td></tr><tr><td>ComMNZ measure</td><td></td><td></td><td></td></tr><tr><td>TokAgg</td><td>97.7%</td><td>52.5%</td><td>68.3%</td></tr><tr><td>TransTokAgg</td><td>97.7%</td><td>51.6%</td><td>67.6%</td></tr><tr><td>All aggregations</td><td><strong>95.8%</strong></td><td><strong>71.2%</strong></td><td><strong>81.6%</strong></td></tr></tbody></table>

Table 3 reports the results obtained for the three language pairs for which we have a translation model, and Table 4 reports on a global evaluation for 22 languages. In both cases, values were computed with the aggregation similarity threshold $\alpha$ set to 0.9.

We defined the baseline as the concatenation of all monolingual clusters from all languages under consideration. It has a high precision (97.7% and 98.2% in Table 3 and 4 resp.) and a poor recall (51.5% and 40.5%) since none of the clusters is cross-lingual. The challenge is thus to improve the recall without affecting the precision too much.

In Tables 3 and 4, *monolingual ExpAgg* corresponds to the expansion aggregation strategy applied at the monolingual level, and *multilingual ExpAgg* at the multilingual level. The *TokAgg* and *TransTokAgg* lines correspond to results

<footnote n="2">²We tried two other metrics: macro-average and B-cubed measure (Bagga & Baldwin 1998) but since results are comparable we do not report them.</footnote>
<table><caption>Table 4: Cluster aggregation strategies on 22 languages</caption><thead><tr><th></th><th>MAV-P</th><th>MAV-R</th><th>F1</th></tr></thead><tbody><tr><td>Baseline</td><td>98.2%</td><td>40.5%</td><td>57.4%</td></tr><tr><td>Monolingual ExpAgg</td><td>97.0%</td><td>44.9%</td><td>60.5%</td></tr><tr><td>Multilingual ExpAgg</td><td>97.4%</td><td>54.6%</td><td>70.0%</td></tr><tr><td>Cosine measure</td><td></td><td></td><td></td></tr><tr><td>TokAgg</td><td>98.2%</td><td>45.3%</td><td>62.0%</td></tr><tr><td>TransTokAgg</td><td>97.7%</td><td>41.1%</td><td>57.9%</td></tr><tr><td>All aggregations</td><td><strong>93.1%</strong></td><td><strong>65.9%</strong></td><td><strong>77.2%</strong></td></tr><tr><td>ComMNZ measure</td><td></td><td></td><td></td></tr><tr><td>TokAgg</td><td>98.2%</td><td>45.3%</td><td>62.0%</td></tr><tr><td>TransTokAgg</td><td>98.2%</td><td>40.8%</td><td>57.6%</td></tr><tr><td>All aggregations</td><td><strong>95.8%</strong></td><td><strong>65.5%</strong></td><td><strong>77.8%</strong></td></tr></tbody></table>

with the corresponding token aggregation strategies using cosine similarity and CombMNZ fusion, and *All aggregations* to the ones obtained when using the four aggregation strategies in a joint way.

It can be observed that each aggregation strategy contributes to improving the quality of cross-lingual cluster aggregation, with multilingual ExpAgg providing the best improvement (+10.8 points for the 3 language pairs and +12.6 points for the 22 languages). The contribution of the TransTokAgg aggregation is slightly disappointing; it improves the baseline in both language configurations, but not significantly. Nevertheless, when all the aggregations are applied (bold lines), results are better than the addition of each single aggregation. It could mean that the TransTokAgg aggregation provides links between clusters which are not useful in isolation, but adds relevant bridges between sets of clusters when combined with other aggregations. Besides, one should notice that between the three language pairs and the 22 languages, improvements per aggregation strategy are comparable. Similarly, results obtained based on cosine similarity and CombMNZ fusion are comparable. This strengthens the reliability of the obtained results.

Figure 2 shows the impact of the threshold $\alpha$. When too low (0.7), the F1 measure can be below the baseline because too many links are established between clusters; when too high (1.0), aggregations based on similar and translated tokens are reduced to values close to zero. In between, it has a clear improvement impact.
Overall, all aggregations strongly improve the baseline by increasing the recall (+19.7 and +23.4 points resp.) with a small loss in precision (-1.9 and -2.4 points resp.). Eventually, there are 64,000 cross-lingual connected clusters across languages instead of 81,000 monolingual ones for the 22 languages.

<figure><img src="image_96.png" /><figcaption>Figure 2: F1 improvement per aggregation type on 22 languages given α, using cosine similarity</figcaption></figure>

# 4 Multi-word entity pattern learning

The previous section describes an approach which is useful to recognise frequently mentioned MWEntities and cluster them across languages, but it is limited to MWEntities mentioned at least once followed or preceded by its corresponding acronym. In this section we focus on a complementary approach to address the MWEntity recognition task. From the automatically obtained resource composed of 64,000 entities and their 600,000 multilingual lexical variants, we aim at learning MWEntity patterns in order to recognise new and not previously mentioned MWEntities. The described approach is ongoing work. Consequently, if the final goal is to learn these MWEntity patterns from the automatically extracted MWEntity resource, we must first control that our learning approach is reliable independently of the used MWEntity resource's quality. This section
describes the use of an existing and reliable resource to evaluate our pattern
learning approach.

## 4.1 Extraction of organisation names from BabelNet

For the sake of learning multi-word entity extraction patterns we have exploited BabelNet (Navigli & Ponzetto 2012), a large multilingual encyclopedic dictionary and semantic network, created by merging various publicly available linguistic resources, e.g., WordNet and Wikipedia. In particular, BabelNet contains circa 7.7 millions of named entity-related (NE-related) synsets. We used the BabelNet API<fnref n="3" /> to extract organisation names for English and Spanish, which were then used in the process of learning patterns in various ways. Since the NE-related BabelNet synsets are not tagged with a specific NE tag, the NE type was inferred through utilisation of the hypernym information provided in BabelNet (i.e., using WordNet hypernyms and Wikipedia categories). To be more precise, based on hypernym frequency information for the entire set of named entities a list of *positive* (circa 200) and *negative* (circa 20) hypernyms was manually created. These lists were subsequently used to extract organisation names, i.e., a given synset was extracted if: (a) there was at least one hypernym for the main sense of the synset in the list of positive hypernyms, and (b) no hypernym for the main sense of the synset was on the list of negative hypernyms. For instance, the list of positive hypernyms for extracting organisations includes terms like: *airline*, *enterprise*, *corporation*, *bank*, *local_government*, *political_organisation*, *law_enforcement_agency*, whereas the list of negative hypernyms includes terms like *person* and *human*. The main drive behind the usage of negative hypernym list was to filter out potentially ambiguous named entity candidates. In total, we have extracted 647,898 and 127,264 organisation names for English and Spanish respectively. We exploited only names that consisted of at least two tokens for the multi-word organisation name pattern learning, which resulted in maintaining only 87.0% (557,841) of the English and 86.1% (127,264) of the Spanish organisation names extracted. Noteworthy, the resource for English obtained in this manner includes a portion of organisation names in foreign languages, which is most likely due to the fact that some non-English name variants have been tagged in BabelNet as variants in English. Since the entire procedure for pulling out organisation names from BabelNet is automated such language-specific name variants have not been manually removed.

<footnote n="3">³http://babelnet.org/guide</footnote>
## 4.2 Learning multi-word entity patterns based solely on BabelNet resources

The first approach to learning multi-word organisation name extraction patterns exploits as the only resource the organisation names extracted from BabelNet (see Section 4.1). Therefrom, simple linear patterns are learned that consist of two types of elements, namely, surface forms (as they appear in the organisation names) and generic token classes, which will be referred to as token class elements. Example (10) illustrates the syntax of the patterns.

(10) University [] of [] the [] [UPP_W] [] in [] [UPP_W]
[ALLCAP] [] [UPP_W] [] Construction [] Group
The [] [NUM LET] [] Company
[UPP_W] [DASH] Institute

[] denotes a whitespace (not necessarily required to be included in the pattern as illustrated by the last pattern), whereas other token classes are delimited using square brackets. There are 28 generic token classes, out of which 8 cover natural language words (e.g., [UPP_W] – uppercase word, [LOW_W] – lowercase word, [ALLCAP] – all capital words), letters (e.g., [SINGCAP] – single capital letter), numbers (e.g., [NUM]) and combinations thereof (e.g., [NUM LET] – sequence of digits followed by a sequence of letters, etc.), whereas the remaining 20 classes are used to denote specific symbols (e.g., brackets, commas, dots, colons, etc.).

The pattern learning process consists of three main steps: (a) acquisition of candidate patterns, (b) filtering unreliable and ambiguous candidate patterns, and (c) ranking patterns. These are described in more detail below.

### 4.2.1 Acquisition of candidate patterns

First, each organisation name is transformed into a candidate pattern, i.e., each token which can be found in a set of predefined surface forms (consisting of keywords that trigger organisation names, e.g., *University*, and frequently occurring word forms, e.g., prepositions) remains unchanged, whereas all other tokens are mapped into a corresponding generic token class. Each candidate pattern must contain at least one surface form and at least one token-class element, otherwise it is discarded.

The set of predefined surface forms has been computed automatically and consists of word uni-grams that fulfill the following criteria: (a) it appears more than $\phi$ = 20 times as part of an organisation name, (b) it does not appear on a list of known toponyms,<fnref n="4" /> (c) it does not appear on the list of known first names and

<footnote n="4">⁴We used GeoNames resource at: http://www.geonames.org for this purpose.</footnote>
surnames,<fnref n="5" /> and (d) it is not an adjective (unless it appears very frequently). For instance, for the subset of English organisation names consisting solely of company names, the 10 top-most frequent word uni-grams that fulfill the aforementioned criteria are: *Company, and, of, The, Group, Corporation, Bank, de, Limited* and *Air*.

### 4.2.2 Filtering candidate patterns

In the subsequent step, a candidate pattern is discarded if:

1. its final element is the token class `[LOW_W]` (any lowercase word), or

2. it contains only surface forms which are single uppercase letters and it does not contain any token-class element representing words starting with an uppercase letter (e.g., `[UPP_W]`, `[ALLCAP]`), or

3. it starts with an initial uppercase letter, followed by an optional dot and a sequence of token classes corresponding to words starting with uppercase letters (and variations of this pattern), e.g., the following candidate pattern would be discarded: `A [] [DOT] [] [UPP_W] [] [ALLCAP]`

The filtering rules 1–2 are used in order to eliminate unreliable patterns, i.e., ones that are likely to overgenerate, whereas the filtering rule 3 aims at eliminating candidate patterns that are likely to match person names. The application of the filtering resulted in maintaining 47,496 (12,966) extraction patterns for English (Spanish), where 32.3% (41.9%) of these patterns were observed more than once. Interestingly, only 0.57% of English and 0.35% of the Spanish patterns occur more than 100 times.

### 4.2.3 Ranking patterns

In the final step candidate patterns are ranked with respect to their reliability based on the following general assumptions related to their structure:

* a pattern that contains either: (a) a larger fraction of surface forms vis-a-vis token-class elements, or (b) longer sequences of consecutive surface forms is deemed more reliable,

* a pattern whose final element is a lowercase surface form is deemed less reliable,

<footnote n="5">⁵We used for this purpose the *JRC Name Variant Database* and a huge list of first names extracted from Piskorski et al. (2011).</footnote>
* a pattern that contains either: (a) a larger fraction of token-class elements representing single capital letters and lowercase words, or (b) longer sequences of consecutive token-class elements representing lowercase words is deemed less reliable.

The formal definition of the reliability score (Rel($p$)) for a pattern $p$ is given below, where the expressions starting with `#` denote the number of elements in the pattern of a specific type<fnref n="6" /> and $\alpha = 0.2$, $\beta = 0.2$, $\gamma = 0.2$, $\delta = 0.15$, $\lambda = 0.1$ and $\kappa = 0.15$ are weighting coefficients for the various criteria used in the reliability ranking, whose values have been set based on empirical observations.

$$ \text{Rel}(p) = \frac{\frac{\#\text{SurfaceForms}(p) \cdot \alpha + \#\text{ConsecutiveSurfaceForms}(p) \cdot \beta}{\#\text{NonWhitespace}(p)} - \frac{(\#\text{LowerCTokens}(p) \cdot \gamma + \#\text{ConsecutiveLowerCTokens}(p) \cdot \delta)}{\#\text{NonWhitespace}(p)} - \frac{\#\text{SingleCapitalLetterTokens}(p) \cdot \lambda}{\#\text{NonWhitespace}(p)} + \gamma + \delta + \lambda + (1 - \text{LastElementIsLowerCToken}(p)) \cdot \kappa}{\gamma + \delta + \lambda + (1 - \text{LastElementIsLowerCToken}(p)) \cdot \kappa} $$

A few examples of patterns with various reliability scores (provided in brackets) are given in (11).

(11) Ministry [] of [] Foreign [] Affairs [] of [] [UPP_W] (0.97)
Institute [] of [] [UPP_W] [] Studies (0.95)
[UPP_W] [] [UPP_W] [] [ALLCAP] [] at [] [UPP_W] [] University (0.67)
St [DOT] [] [LOW_W] [] [LOW_W] [] [LOW_W] [] [LOW_W] [] school (0.24)
[UPP_W] [] [LOW_W] [] [LOW_W] [] [LOW_W] [] committee (0.22)

Figure 3 depicts the distribution of patterns with respect to their reliability scores.

<footnote n="6">⁶`#LowerCTokens(p)` denotes the number of lowercase tokens, while `#NonWhitespace(p)` denotes the number of elements in the pattern which are not whitespaces, i.e., it is a count of surface forms and token-class elements. `LastElementIsLowercaseToken(p)` denotes a function which returns 1 in case the last element of the pattern is a lowercase token class or 0 otherwise.</footnote>
<figure><img src="image_97.png" /><figcaption>Figure 3: Distribution of patterns with respect to their reliability score for English and Spanish. Each of the bars represents the fraction of patterns, whose reliability score is within the range (x, x + 0.5), where x ∈ {0, 0.5, ..., 0.95}.</figcaption></figure>

## 4.3 Evaluation

In order to evaluate the quality of the learned patterns we used two NE annotated corpora from the CoNLL shared task for English and Spanish<sup>7</sup>, which contains respectively 6,300 and 7,389 organisation occurrences, and tested the performance using different settings. In particular, we compared three settings: (a) using only BabelNet-derived patterns (denoted in the figures with PATTERNS), (b) using only an existing rule-based NER system as a baseline (denoted RULES), (c) combining rule-based NER system and BabelNet-derived patterns (denoted in the figures with RULES+PATTERNS). In our experiments, we used an in-house rule based NER system (Steinberger et al. 2011; Ehrmann et al. 2015) that is geared towards high precision. The choice of a specific NER system is not decisive for these experiments, but by combining an existing NER system with our BabelNet-derived patterns, we aim at testing how our automatically created patterns could be useful to improve the quality of the NER recognition.

Figures 4, 5, 6 and 7 depict the performance of applying the BabelNet-derived patterns for English and Spanish in terms of precision and recall. The precision

<footnote n="7">⁷https://www.clips.uantwerpen.be/conll2002/ner/ and https://www.clips.uantwerpen.be/conll2003/ner/</footnote>
and recall values were computed for the varying minimum pattern reliability threshold in the range of {0.10, 0.15, ..., 0.95}, i.e., patterns below the minimum reliability threshold were discarded. The figures are showing the results obtained for exact matching (denoted with EXACT-PATTERNS or EXACT-RULES+PATTERNS) and for fuzzy matching, e.g., when there is a matching but with a left or right boundary mismatch (denoted with FUZZY-PATTERNS or FUZZY-RULES+PATTERNS). We did not visualise the results corresponding to the RULE setting in the figures because they do not depend on the reliability threshold. However, the obtained scores for this setting are embraced in Table 5.

<table><caption>Table 5: Results obtained with pattern reliability threshold = 0.60</caption><thead><tr><th rowspan="2"></th><th colspan="3">EXACT matching</th><th colspan="3">FUZZY matching</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Spanish</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>PATTERNS</td><td>63.1%</td><td>10.2%</td><td>17.6%</td><td>81.4%</td><td>13.2%</td><td>22.8%</td></tr><tr><td>RULES</td><td><strong>79.8%</strong></td><td>24.2%</td><td>37.1%</td><td><strong>91.3%</strong></td><td>27.6%</td><td>42.4%</td></tr><tr><td>RULES+PATTERNS</td><td>73.5%</td><td><strong>31.0%</strong></td><td><strong>43.6%</strong></td><td>84.2%</td><td><strong>35.5%</strong></td><td><strong>50.0%</strong></td></tr><tr><td>English</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>PATTERNS</td><td>48.5%</td><td>11.4%</td><td>18.5%</td><td>69.2%</td><td>16.3%</td><td>26.4%</td></tr><tr><td>RULES</td><td><strong>69.0%</strong></td><td>25.5%</td><td>37.3%</td><td><strong>80.4%</strong></td><td>29.7%</td><td>43.4%</td></tr><tr><td>RULES+PATTERNS</td><td>55.9%</td><td><strong>28.7%</strong></td><td>37.9%</td><td>65.6%</td><td><strong>33.6%</strong></td><td><strong>44.5%</strong></td></tr></tbody></table>

Table 5 provides the results obtained with a pattern-reliability threshold equal to 0.60, which corresponds to the best results obtained in both languages. For the EXACT evaluation, an improvement in terms of F1 of 6.5 points for Spanish could be observed with the setting RULES+PATTERNS versus the baseline RULES. As regards FUZZY evaluation, one could observe an improvement of F1 of 7.6 and 1.1 points for Spanish and English respectively when comparing RULES+PATTERNS versus the baseline RULES setting.
<figure><img src="image_98.png" /></figure>

<table><caption>Figure 4: Experiments on English: Precision curves reflecting the performance of applying BabelNet-derived patterns and combining them with a rule-based NER system</caption><tr><td>FUZZY-RULES+PATTERNS</td><td>EXACT-RULES+PATTERNS</td><td>FUZZY-PATTERNS</td><td>EXACT-PATTERNS</td></tr></table>

<figure><img src="image_99.png" /></figure>

<table><caption>Figure 5: Experiments on English: Recall curves reflecting the performance of applying BabelNet-derived patterns and combining them with a rule-based NER system</caption><tr><td>FUZZY-RULES+PATTERNS</td><td>EXACT-RULES+PATTERNS</td><td>FUZZY-PATTERNS</td><td>EXACT-PATTERNS</td></tr></table>
<figure><img src="image_100.png" /><figcaption>Figure 6: Experiments on Spanish: Precision curves reflecting the performance of applying BabelNet-derived patterns and combining them with a rule-based NER system</figcaption></figure>

<figure><img src="image_101.png" /><figcaption>Figure 7: Experiments on Spanish: Recall curves reflecting the performance of applying BabelNet-derived patterns and combining them with a rule-based NER system</figcaption></figure>
## 4.4 Mistakes and fuzzy matchings

Using existing NE annotated corpora for our preliminary experiments was the most obvious choice to measure the quality of our patterns. Nevertheless, it appears that some MWEntities recognised by the BabelNet-derived patterns considered as incorrectly extracted according to the annotated corpora, could still be considered as correct extractions. Figure 8 provides the complete list of “wrong” MWEntities (first column) recognised by our patterns when pattern reliability threshold equals 0.90. Even if some cases are clear mistakes, like *Results of European Super* or *Bank on Thursday*, a large fraction of the extractions could be considered as valid organisation names. The two other columns show the partial matches with the same reliability threshold, which are considered as incorrect in the *Exact matching* evaluation, and correct in the *Fuzzy matching* evaluation. Again, if some of them are clear mismatches, like *European Commission on Wednesday* or *NATO and the European Union*, most of the extractions appear to be consistent entities.

We expect to achieve higher precision of the learned patterns through embracing in the computation of the reliability score additional external evidence, i.e., exploiting contextual information obtained from pattern matchings in a web-scale corpus to judge the correctness.

<table><caption>Figure 8: Complete list of “wrong” MWEntities and left or right boundary mismatching with pattern reliability threshold = 0.90</caption><thead><tr><th>"Wrong" patterns</th><th>LEFT boundary problems</th><th>RIGHT boundary problems</th></tr></thead><tbody><tr><td>Major League Baseball</td><td>Brazilian Foreign Ministry</td><td>American University in Washington</td></tr><tr><td>Union National Bank</td><td>Chong Hing Investment Ltd</td><td>Association for Relations Across</td></tr><tr><td>Bank on Thursday</td><td>County Board of Education</td><td>Association of Lloyd</td></tr><tr><td>Results of European Super</td><td>Court of Appeals for the Fourth Circuit</td><td>Awami League of Prime</td></tr><tr><td>Sachs and Co.</td><td>Court of Appeals for the Ninth Circuit</td><td>Bosnian Association for Refugees</td></tr><tr><td>Results of European Cup</td><td>Credit Corp of USDA</td><td>British Securities and Investments</td></tr><tr><td>Bank of Pakistan</td><td>Democratic Party of Iran</td><td>Bureau of Alcohol</td></tr><tr><td>Foreign and Commonwealth Affairs</td><td>Department of Humanitarian Affairs</td><td>Department of Humanitarian</td></tr><tr><td>The News Agency</td><td>Gas Chemical Co Ltd</td><td>DLI Merchant Banking Partners</td></tr><tr><td>Hospital School of Medicine</td><td>Illinois University at Carbondale</td><td>Economic Community of West</td></tr><tr><td>Police in Malta</td><td>Information Systems Co Ltd</td><td>European Commission on Wednesday</td></tr><tr><td>Brooks Investment Corp.</td><td>Life Insurance Co</td><td>European Union and the United States</td></tr><tr><td>Venture Partners of Menlo</td><td>Lodge of the University of the Witwatersrand</td><td>Government of National Unity</td></tr><tr><td>Department of Health</td><td>NATO and the European Union</td><td>Institute for Human Gene</td></tr><tr><td>Bank on Monday</td><td>Patriotic Union of Kurdistan</td><td>Irish Department of Enterprise</td></tr><tr><td>New England Telecommunications Corp</td><td>Rangoon Karen National Union</td><td>Kingston Technology Co.</td></tr><tr><td>Scottish League Cup</td><td>Rights Action League</td><td>Life Insurance Co</td></tr><tr><td>Police in Berovo</td><td>Staff Union of Universities</td><td>Lockhart Industries Inc.</td></tr><tr><td>Legislative Council on Wednesday</td><td>Taylor Made Co Ltd</td><td>Mallinckrodt Group Inc.</td></tr><tr><td>Yemeni Ministry of Petroleum</td><td>The Bank of Finland</td><td>Morgan Stanley and Co</td></tr><tr><td></td><td>The Bank of France</td><td>National Council of Christian Churches</td></tr><tr><td></td><td>The Foreign Ministry</td><td>NATO and the European Union</td></tr><tr><td></td><td>The Lebanese Association for the Democracy of Elections</td><td>Security Council on Friday</td></tr><tr><td></td><td>The National Basketball Association</td><td>Security Dynamics Technologies Inc</td></tr><tr><td></td><td>The Republican Party</td><td>Shanghai Posts and Telecommunications</td></tr><tr><td></td><td>Turkish Foreign Ministry</td><td>State National Bank<br>Ukrainian Academy of Agrarian<br>United in Turin<br>University of Oklahoma<br>University of Pennsylvania Medical</td></tr></tbody></table>
# 5 Conclusion and future work

The methods described in this chapter have produced a large 22-language resource containing multi-word entities of different types and a number of automatically learned patterns to recognise newly occurring MWEntities. We intend to integrate these recognition patterns, together with the variant matching techniques, into the workflow of the *Europe Media Monitor*. An interesting feature of this collection and the patterns is that all MWEntity forms were found in real-world text and that large numbers of variants were identified, including typos, simplifications of longer names, syntactic and morphological variants and translational equivalences.

The results obtained in MWEntity recognition with the patterns automatically derived from BabelNet are promising when applied to English and Spanish, although the reported approach and evaluation figures reflect only our preliminary research in this area. Expanding the pattern learning to other languages is part of our future work. We also envisage applying the same pattern learning approach from the automatically created MWEntity resource. This would require to categorise the MWEntity sets into some broad semantic classes (e.g., organisations, events, measurements, and others) which is a task we are currently working on.

Furthermore, we are also working on expanding the patterns we learned based on a distributional approach. It consists of replacing meaningful surface forms from each pattern by a cluster of surface forms that would belong to the same semantic class. In such a way, similar words like *company*, *firm*, *corporation*, etc., will be part of the same cluster because they have a high distributional similarity. Finally, the pattern reliability scoring could be extended through inclusion of additional statistics when applying the patterns on web-scale corpora.

# References

* Adar, E. 2004. SaRAD: A simple and robust abbreviation dictionary. *Bioinformatics* 20. 527–533.

* Bagga, Amit & Breck Baldwin. 1998. Algorithms for scoring coreference chains. *Proceedings of the First International Conference on Language Resources and Evaluation, Workshop on Linguistic Coreference*. 563–566.

* Chang, Jeffrey T., Hinrich Schütze & Russ B. Altman. 2002. Creating an online dictionary of abbreviations from MEDLINE. *Journal of the American Medical Informatics Association* 9(6). 612–620.
Collins, Michael & Yoram Singer. 1999. Unsupervised models for named entity classification. In *Proceedings of the joint SIGDAT conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP-VLC)*, 100–110. College Park, MD: University of Maryland.

Downey, Doug, Matthew Broadhead & Oren Etzioni. 2007. Locating complex named entities in web text. In *Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI’07)*, 2733–2739. Hyderabad, India: Morgan Kaufmann Publishers Inc. http://dl.acm.org/citation.cfm?id=1625275.1625715.

Dyer, Chris, Victor Chahuneau & Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In *Proceedings of the 2013 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)*, 644–648.

Ehrmann, Maud, Guillaume Jacquet & Ralf Steinberger. 2015. JRC-Names: Multilingual entity name variants and titles as linked data. *Semantic Web (Preprint)*. 1–13.

Ehrmann, Maud, Leo D. Rocca, Ralf Steinberger & Hristo Tanev. 2013. Acronym recognition and processing in 22 languages. In *Proceedings of the 9th conference on Recent Advances in Natural Language Processing (RANLP)*, 237–244. Hissar, Bulgaria.

Fox, Edward A. & Joseph A. Shaw. 1994. Combination of multiple searches. *NIST Special Publication*. 243–243.

Hahn, Udo, Philipp Daumke, Stefan Schulz & Kornél Markó. 2005. Cross-language mining for acronyms and their completions from the web. *Proceedings of the 8th international conference on Discovery Science (DS’05)* 9. 113–123.

Hoffart, Johannes, Fabian M Suchanek, Klaus Berberich & Gerhard Weikum. 2013. *YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia*. In *Proceedings of the twenty-third international joint conference on Artificial Intelligence*, 3161–3165.

Jacquet, Guillaume, Maud Ehrmann & Ralf Steinberger. 2014. Clustering of multi-word named entity variants: Multilingual evaluation. In *Proceedings of the 9th Language Resources and Evaluation conference (LREC 2014)*, 2548–2553. Reykjavik, Iceland.

Jacquet, Guillaume, Maud Ehrmann, Ralf Steinberger & Jaakko Väyrynen. 2016. Cross-lingual linking of multi-word entities and their corresponding acronyms. In *Proceedings of the 10th Language Resources and Evaluation Conference (LREC 2016)*, 528–535. Portorož, Slovenia.

James, J. Pustejovsky, José Castano, Brent Cochran, Maciej Kotecki & Michael Morrell. 2001. Automatic extraction of acronym-meaning pairs from MEDLINE databases. *Studies in health technology and informatics* 1. 371–375.
Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Mar-
cello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin & Eva Herbst.
2007. Moses: Open source toolkit for statistical machine translation. In *Proceed-
ings of the 45th annual meeting of the ACL on interactive poster and demonstra-
tion sessions*, 177–180. Prague, Czech Republic.

Koehn, Philipp, Franz J. Och & Daniel Marcu. 2003. Statistical phrase-based trans-
lation. In *Proceedings of the 2003 conference of the North American chapter of
the Association for Computational Linguistics on Human Language Technology*,
vol. 1, 48–54.

Kokkinakis, Dimitrios & Dana Dannélls. 2006. Recognizing acronyms and their
definitions in Swedish medical texts. In *Proceedings of the 5th conference on
Language Resources and Evaluation (LREC)*, 1971–1974. Genoa, Italy.

Kompara, Mojca. 2010. Automatic recognition of abbreviations and abbreviations'
expansions in multilingual electronic texts. In Chris Cummins, Chi-Hé Elder,
Thomas Godard, Morgan Macleod, Elaine Schmidt & George Walkden (eds.),
*Proceedings of the sixth Cambridge postgraduate conference in language research
(CAMLing)*, 82–91.

McNamee, Paul & Hoa T. Dang. 2009. Overview of the TAC 2009 knowledge base
population track. In *Text Analysis Conference (TAC)*, vol. 17, 111–113.

Nadeau, David & Peter D. Turney. 2005. A supervised learning approach to
acronym identification. In Balázs Kégl & Guy Lapalme (eds.), *Advances in arti-
ficial intelligence: Canadian AI 2005* (Lecture Notes in Computer Science 3501),
319–329. Berlin & Heidelberg: Springer.

Navigli, Roberto & Simone Paolo Ponzetto. 2012. BabelNet: The automatic con-
struction, evaluation and application of a wide-coverage multilingual semantic
network. *Artificial Intelligence* 193. 217–250. DOI:10.1016/j.artint.2012.07.001

Nothman, Joel, Nicky Ringland, Will Radford, Tara Murphy & James Curran. 2013. Learning multilingual named entity recognition from Wikipedia. *Artificial Intelligence* 194. 151–175. DOI:10.1016/j.artint.2012.03.006

Okazaki, Naoaki & Sophia Ananiadou. 2006. Building an abbreviation dictionary using a term recognition approach. *Bioinformatics* 22. 3089–3095.

Okazaki, Naoaki, Sophia Ananiadou & Jun'ichi Tsujii. 2010. Building a high-quality sense inventory for improved abbreviation disambiguation. *Bioinformatics* 26. 1246–1253.

Piskorski, Jakub, Martin Atkinson & Jenya Belyaeva. 2011. Exploring the usefulness of cross-lingual information fusion for refining real-time news event extraction: A preliminary study. In *Proceedings of the conference Recent Advances in Natural Language Processing*, 210–217. Hissar, Bulgaria.
Rao, D., P. McNamee & M. Dredze. 2013. Entity linking: Finding extracted entities in a knowledge base. In *Multi-source, multilingual information extraction and summarization*, 93–115. Springer.

Riloff, Ellen. 1996. Automatically generating extraction patterns from untagged text. In *Proceedings of thirteenth National Conference on Artificial Intelligence (AAAI-96)*, 1044–1049. The AAAI Press/MIT Press.

Schwartz, Ariel S. & Marti A. Hearst. 2003. A simple algorithm for identifying abbreviation definitions in biomedical text. In *Proceedings of the PAC on Bio-computing*, 451–462.

Siklósi, Borbála, Attila Novák & Gábor Prószéky. 2014. Resolving abbreviations in clinical texts without pre-existing structured resources. In *4th workshop on Building and Evaluating Resources for Health and Biomedical Text Processing (BioTxtM)*, LREC, 69–75. Reykjavik, Iceland.

Steinberger, Ralf, Maud Ehrmann, Júlia Pajzs, Mohamed Ebrahim, Josef Steinberger & Marco Turchi. 2013. Multilingual media monitoring and text analysis–Challenges for highly inflected languages. In *International conference on Text, Speech and Dialogue*, 22–33.

Steinberger, Ralf, Aldo Podavini, Alexandra Balahur, Guillaume Jacquet, Hristo Tanev, Jens Linge, Martin Atkinson, Michele Chinosiand, Vanni Zavarella, Yaniv Steiner & Erik van der Goot. 2015. Observing trends in automated multilingual media analysis. In *Proceedings of the symposium on New Frontiers of Automated Content Analysis in the Social Sciences (ACA’2015)*, 1–8.

Steinberger, Ralf, Bruno Pouliquen, Mijail Kabadjov & Erik van der Goot. 2011. JRC-Names: A freely available, highly multilingual named entity resource. In *Proceedings of the 8th International Conference Recent Advances in Natural Language Processing (RANLP’2011)*, 104–110. Hissar, Bulgaria.

Steinberger, Ralf, Bruno Pouliquen & Erik van der Goot. 2009. An introduction to the Europe Media Monitor family of applications. In *Proceedings of the SIGIR 2009 workshop (SIGIR-CLIR’2009)*, 1–8. Boston, USA.

Taghva, Kazen & Jeff Gilbreth. 1999. Recognizing acronyms and their definitions. In *International Journal on Document Analysis and Recognition* 1(4). 191–198.

Wren, Jonathan D. & Harold R. Garner. 2002. Heuristics for identification of acronym-definition patterns within text: Towards an automated construction of comprehensive acronym-definition dictionaries. In *Methods of Information in Medicine* 41(5). 426–434.

Yangarber, Roman, Winston Lin & Ralph Grishman. 2002. Unsupervised learning of generalized names. In *Proceedings of COLING: The 19th international conference on Computational Linguistics*. Taipei, Taiwan.
<figure><img src="image_102.png" /></figure>
Name index

Abeillé, Anne, 40
Ackerman, Farrell, 40
Adar, E., 272
Alegria, Iñaki, 218
Ananiadou, Sophia, 272
Angelov, Krasimir, v, 134, 135, 141, 143, 144, 151
Apidianaki, Marianna, 110
Arun, Abhishek, 147, 154, 157
Attia, Mohammed A., 110, 114
Bäckström, Linnéa, 139
Bagga, Amit, 282
Baldwin, Breck, 282
Baldwin, Timothy, 70, 83, 152
Bansal, Mohit, 168
Barbu, Ana Maria, 240
Bargmann, Sascha, 40, 49, 113
Baroni, Marco, 41, 51
Barz, Irmhild, 244
Bejček, Eduard, 150, 155
Besançon, R., 246, 247
Bilgram, Thomas, 114
Birch, Alexandra, 184
Black, E., 152, 171
Blank, Ingeborg, 240
Blunsom, Phil, 152
Bohnet, Bernd, 173
Bouamor, Dhouha, 246
Boulaknadel, Siham, 241
Boullier, Pierre, 163
Bresnan, Joan, 69, 74, 77

Brun, Caroline, 218
Buchholz, Sabine, 171
Burger, H, 244
Butt, Miriam, 70, 72
Cafferkey, Conor, 158
Calzolari, Nicoletta, 242
Candito, Marie, 148, 150, 152, 154, 158, 165, 166, 168, 170, 173
Carpuat, Marine, 242
Carroll, John, 169
Chafe, Wallace L., 36
Chang, Jason S, 241
Chang, Jeffrey T., 272
Charniak, Eugene, 149
Chomsky, Noam, 226, 228
Choueka, Yaacov, 241
Christodoulopoulos, Christos, 194
Church, Ken, 241, 248
Clark, Stephen, 188, 198
Collins, Michael, 149, 151, 158, 171, 273
Connine, Cynthia M., 38
Constable, James, 192
Constant, Mathieu, vi, 148, 152, 154–156, 158, 163–167, 170, 173, 190, 191, 219, 242
Copestake, Ann, 17, 110, 189
Crabbé, Benoît, 22, 115, 150, 152
Curran, James, 188, 192, 194
Daelemans, Walter, 4, 22
Dagan, Ido, 241, 248
Name Name index

Daille, Béatrice, 240, 241
Dalrymple, Mary, 69, 70, 112, 113
Dang, Hoa T., 270
Dannélls, Dana, 272
De Lhoneux, Miryam, vi
De Smedt, Koenraad, 22
DeNero, John, 240, 249, 251, 252
Denis, Pascal, 168
Deoskar, Tejaswini, 194, 197
Détrez, Grégoire, 131
Diab, Mona, 242
Doron, Edit, 49
Downey, Doug, 273
Dunning, Ted, 229
Durrett, Greg, 168
Dyer, Chris, 168, 281
Dyvik, Helge, v, 10, 30, 69, 72, 150

Ehrmann, Maud, 273, 274, 289
Eisner, Jason M., 150
Enache, Ramona, 139, 140
Erman, Britt, 36
Ernst, Thomas, 49, 50, 59
Eryiğit, Gülşen, 148, 154, 157, 165, 171, 173, 190
Evans, Roger, 17, 21

Fazly, Afsaneh, 160
Fellbaum, Christiane, 36
Finkel, Jenny Rose, 219
Finlayson, Mark Alan, 194
Fišer, Darja, 241
Foufi, Vasiliki, vi, 160
Fox, Edward A., 279
Francopoulo, Gil, 8
Fraser, Alexander, 240, 258, 263

Gao, Zhao-Ming, 252
Garner, Harold R., 272

Gaussier, Éric, 240, 249
Gazdar, Gerald, 17, 21
Germann, Ulrich, 257
Gibbs, Raymond W., 38, 39
Gilbreth, Jeff, 272
Green, Spence, 148, 153, 154, 164, 167, 184, 191, 219
Grégoire, Nicole, 12–14, 40, 114
Gross, Maurice, 5, 12, 113, 151
Grover, Claire, 169
Gruzitis, Normunds, 139
Głowińska, Katarzyna, 154

Habert, Benoît, 17
Hahn, Udo, 272
Hearst, Marti A., 272, 274, 275
Hedström, Björn, 142
Heid, Ulrich, 184, 191, 219, 228
Henderson, James, 173
Hoang, Hieu, 258
Hockenmaier, Julia, 188, 194, 198
Hoffart, Johannes, 273
Hoffmann, Thomas, 187
Hogan, Deirdre, 242

Ion, Radu, 241

Jackendoff, Ray, 36, 240, 241
Jacquet, Guillaume, vi, 274, 275, 280
James, J. Pustejovsky, 272
Jaquemin, Christian, 17

Kaplan, Ronald M., 70, 77
Kato, Akihiko, 164
Kay, Paul, 113
Keller, Frank, 147, 154, 157
Keson, Britt, 114
Kilbury, James, 17
Kilgarriff, Adam, 41, 51
Kim, Su Nam, 70, 83, 186, 191
Klein, Dan, 168, 240, 249, 251, 252

Koehn, Philipp, 257, 258, 281

Kokkinakis, Dimitrios, 272

Kompara, Mojca, 272

Koo, Terry, 168

Kordoni, Valia, 191, 218

Korkontzelos, Ioannis, 110, 158, 184,
189, 190, 194

Krishnamurthy, Jayant, 184

Kulkarni, Nidhi, 194

Kupiec, Julian, 241

Le Roux, Joseph, 163, 164, 167

Lehmann, Hans Martin, 161

Lewis, Mike, 184

Lichte, Timm, v, 22

Losnegaard, Gyri Smørdal, 72

Lyngfelt, Benjamin, 139

Lønning, Jan Tore, 72

Manandhar, Suresh, 110, 158, 184, 189,
190, 194

Manning, Christopher D., 219

Marchand, Morgane, 249

Marcu, Daniel, 240, 258, 263

Marcus, Mitchell P., 152

Markantonatou, Stella, v, 10, 15, 30,
110, 113–115

Marsi, Erwin, 171

Martens, Scott, 184, 191

Matsumoto, Yuji, 149

Maxwell, John, 70

McConnel, Stephen, 17

McDonald, Ryan, 150

McNamee, Paul, 270

Mel'čuk, Igor, 12

Melnik, Nurit, 53

Meurer, Paul, 72

Miller, George Armitage, 152

Minos, Panagiotis, 114

Mirroshandel, Seyed Abolghasem, 168

Mitchell, Tom M, 184

Moon, Rosamund, 41, 42, 50

Müller, Stefan, 17, 21, 26

Nadeau, David, 270, 272

Nagy T., István, 160

Nakagawa, Tetsuji, 150

Nasr, Alexis, 148, 163, 166, 168, 170,
219

Navigli, Roberto, 273, 285

Nayak, Nandini P., 39

Nerima, Luka, 218, 221, 226

Ney, H., 240, 257, 258

Nilsson, Jens, 147, 165, 171, 184, 189,
190, 192, 242

Nitoń, Bartłomiej, 16

Nivre, Joakim, 147–150, 152, 154–157,
164, 165, 167, 184, 189–192, 219,
242

Nordgård, Torbjørn, 72

Nothman, Joel, 273

Nunberg, Geoffrey, 35, 37, 38, 40–42,
49, 64, 66, 70, 185

Och, F. J., 240, 257, 258

Odijk, Jan, 14

Okazaki, Naoaki, 272

Okita, Tsuyoshi, 241

Osswald, Rainer, 24¹⁵

Padó, Sebastian, 203

Papageorgiou, Haris, 114

Papineni, Kishore, 260

Pawley, Andrew, 152

Petitjean, Simon, 22

Petrov, Slav, 224, 235

Piskorski, Jakub, 287
Pollard, Carl, 115
Ponzetto, Simone Paolo, 273, 285
Prins, Robbert, 169
Przepiórkowski, Adam, 12, 14, 16, 154

Ramisch, Carlos, 242
Ramshaw, Lance A., 152
Ranta, Aarne, 127–129, 131, 141, 144
Rao, D., 270
Ren, Z., 242, 248
Riehemann, Susanne, 39, 41, 244
Riloff, Ellen, 273
Ronan, Patricia, 161, 162
Rosén, Victoria, 72, 150, 153
Roth, Dan, 159

Sag, Ivan, 36, 70, 109, 111, 113, 115, 151, 184–186, 189, 209, 223, 240–244
Sagot, Benoît, 163, 168
Sailer, Manfred, 40, 49, 113
Salton, G., 248
Samaridi, Niki, 110, 113–115
Savary, Agata, 12, 113
Schenk, André, 40
Schneider, Gerold, 148, 152, 158, 161, 162, 169

Schneider, Nathan, 153, 192
Schwartz, Ariel S., 272, 274, 275
Seddah, Djamé, 154, 158, 165, 168
Seki, Hiroyuki, 135
Sekine, Satoshi, 171
Semmar, Nasredine, vi, 245, 249
Seretan, Violeta, 160, 191, 219, 229, 241
Shaw, Joseph A., 279
Sheinfux, Livnat Herzig, v, 11, 30
Shieber, Stuart M., 17
Sigogne, Anthony, 163, 164
Siklósi, Borbála, 272

Sinclair, John, 151
Singer, Yoram, 273
Steedman, Mark, 184, 186, 188
Steinberger, Ralf, 270, 271, 289
Syder, Frances Hodgetts, 152

Taghva, Kazen, 272
Tesnière, Lucien, 152
Tiedemann, Jörg, 260
Titone, Debra A., 38
Tjong Kim Sang, Erik F., 152
Tomasello, Michael, 152
Trawinski, Beata, 42
Trousdale, Graeme, 187
Tu, Yuancheng, 159
Tufiş, Dan, 241
Turney, Peter D., 270, 272

van der Linden, Erik-Jan, 4
Vandeghinste, Vincent, 184, 191
Vechtomova, Olga, 242
Villavicencio, Aline, 218, 219
Vincze, Veronika, 148, 152, 154, 160, 165
Vintar, Špela, 241

Warren, Beatrice, 36
Waszczuk, Jakub, 113
Webelhuth, Gert, 40
Wechsler, Stephen M., 17, 21, 26
Weeds, Julie, 158, 169
Wehrli, Eric, 218, 226, 241
Weller, Marion, 184, 191
White, Michael, 184
Wray, Alison, 152
Wren, Jonathan D., 272
Wu, Chien-Cheng, 241

Yamada, Hiroyasu, 149
Yangarber, Roman, 273
Yvon, François, 240, 249

Zhang, Yi, 189, 191, 218
<figure><img src="image_103.png" /></figure>
Language index

Arabic, 110, 167, 219

Bulgarian, 134, 139, 143

Dutch, 12, 29, 40

English, iv, vi, 11, 37, 40–43, 43⁴, 70,
71, 73³, 83, 119⁴, 129–132, 134–
140, 160, 164, 167, 219–221,
229–231, 241, 243, 246, 247,
249, 254, 257, 260, 264, 272,
273, 275–277, 279, 281, 285,
287, 289, 290, 294

French, iv, 17¹³, 30, 40, 119⁴, 130, 133,
140, 150, 152, 158, 164–167, 170,
174, 219, 221, 226, 231, 243,
246, 247, 249, 254, 257, 260,
264, 275, 276, 281

German, 40, 52, 102, 119⁴, 136, 137, 140,
219, 231, 272, 278, 279, 281

Hebrew, iv, v, 42, 42³, 43, 43⁴, 44–49,
51, 64, 66

Hungarian, 165, 272

Italian, 219, 231, 272, 275, 277, 281

Modern Greek, iv, v, 109, 110, 114, 119,
124, 219, 230

Norwegian, iv, v, 69–72, 73³, 76, 78,
83, 92, 97, 104, 106, 119⁴

Romanian, 219

Spanish, 219, 231, 272, 275, 285, 287,
289, 290, 294

Swedish, 134, 139–141, 143, 272
<figure><img src="image_104.png" /></figure>
Subject index

adjunct, 74, 82

alignment, 240, 244–250, 254, 255, 257–260, 264, 272

anaphora, 37, 226

bilingual lexicon, 254, 257–260, 263

CCG, vi, 184–188, 192, 194, 196, 198–200, 203, 209, 210, 212

co-occurrence metric, 255, 257

collocation detection, 232

collocation extraction, 161

collocation identification, 219, 222, 225, 227, 232, 235

compound word, 263

constituency parsing, 148, 149, 164, 167, 170, 171, 173, 210

constraint, 10, 80, 81, 94, 171, 251

decomposability, 37–42, 50, 66, 70, 71

dependency parsing, 147–150, 157, 165, 167, 168, 171, 172, 174, 189, 210

domain vocabulary, 260, 262

figuration, 60, 65, 66

fixed expressions, 70, 72, 78, 243

flexibility, v, 7, 7¹, 9, 11, 18, 28–30, 30¹⁹, 37, 39–43, 50, 51, 64–66, 70, 71, 104, 112, 113, 185, 194, 218

free word order, 109, 223

GF, 112, 113, 122, 124, 127–131, 133–135, 138–141, 143, 144

grammar, v, 3, 5, 12⁴, 13–15, 16⁸, 22, 37, 69–72, 74, 75, 91, 92, 104, 106, 110, 113–115, 119, 120, 124, 127–129, 135, 137–144, 149, 150, 186–189, 191, 210, 219, 222, 227, 263, 264

idiomaticity, 36, 41, 47, 51, 184, 185

idioms, iv, 11, 36, 36¹, 37–42, 42³, 43, 43⁴, 46, 47, 49–53, 58, 61, 64–66, 70–72, 80, 81, 87, 96, 98, 98⁷, 137, 151, 161, 164⁴, 217, 218, 220, 222, 223, 242–244, 248

lexical entry, 75, 79, 83, 85, 87, 89, 90, 94, 96, 131

LFG, v, 69, 70, 72–74, 92, 93, 96, 104, 106, 109–112, 112¹, 113, 114, 116, 119, 120, 122, 124, 151

linear programming, 249, 257

linguistic analysis, 242

long-distance dependencies, 48, 72, 87, 92, 165, 220, 222, 226

multilingual parser, 218, 219

Multiword Expressions, iii, 12, 183

MWE alignment, 258, 260

MWE parsing, 157

MWE recognition, 184, 189, 194, 208, 210, 211

MWE representation, 150, 151, 164, 173
MWE representations, 155

MWEs, vi, 184, 185

nominal collocation, 224, 229

parallel corpora, vi, 240, 241, 260, 264, 278, 281

passivization, 10, 11, 24, 36, 39–41, 70, 218, 225, 243

phrasal verbs, 72, 81, 98, 100, 104, 131, 151, 217, 218, 221

phrase, 3, 16, 25, 28, 36–39, 46, 49, 60, 64, 73, 74, 82, 87, 89, 92, 95, 97, 100, 102, 111, 124, 129, 132, 133, 137–141, 149, 158, 159, 161, 162, 225, 251, 252, 254, 258, 259, 263

predicate-argument structure, 79, 80, 97

prepositional verbs, 71, 81, 85, 90, 100, 102–104, 131, 135

pronominalization, 218, 225

relativization, 47, 48, 55, 65, 70, 87, 92, 218, 225

segment, 232, 233, 245

statistical machine translation, vi, 140, 257, 258, 263, 264

statistical parsing, 148–150, 173

subcategorization, 9, 11, 18, 21, 70, 79, 80, 99, 100, 103, 104, 162, 166, 170

syntactic parsing, 148, 159, 160, 165, 167, 172, 173, 184, 188, 190–193, 199, 210–212

syntactic structure, vi, 11–13, 15, 119, 147, 149, 150, 152, 154–156, 160, 165, 190, 241

template, 18, 21, 46, 47, 58, 75, 76, 79, 80, 83, 85, 87, 89, 90, 94–97, 99, 100, 102

transformations, 37, 40, 47, 51, 66

translation model, 257–259, 263, 264, 278, 282

translation quality, 262

transparency, 7, 14, 17, 21, 28, 30, 38, 65, 66

treebanks, iv, 72, 148–150, 153, 154, 163, 164, 164⁴, 167, 168, 174, 192

variations, iii, 15, 39–41, 47, 49–51, 62, 64–66, 106, 150, 243, 287

verb-particle constructions, 36, 70, 71, 81, 83, 85, 94, 100, 100⁸, 101–103, 160, 244

verbal collocation, 223

verbal MWEs, v, 12, 14, 36, 51, 70, 71, 80, 98

wh-questions, 226

word alignment, 140, 240, 248, 254, 258, 259, 263
<figure><img src="image_105.png" /></figure>
Did you like this
book?

This book was brought to you for
free

Please help us in providing free access
to linguistic research worldwide. Visit
http://www.langsci-press.org/donate to
provide financial support or register as
a community proofreader or typesetter
at http://www.langsci-press.org/register.

<figure><img src="image_106.png" /></figure>

language
science
press
<figure><img src="image_107.png" /></figure>
Representation and parsing of
multiword expressions

Deep parsing is the fundamental process aiming at the representation of the syntactic structure of phrases and sentences. In the traditional methodology this process is based on lexicons and grammars representing roughly properties of words and interactions of words and structures in sentences. Several linguistic frameworks, such as Head-driven Phrase Structure Grammar (HPSG), Lexical Functional Grammar (LFG), Tree Adjoining Grammar (TAG), Combinatory Categorial Grammar (CCG), etc., offer different structures and combining operations for building grammar rules. These already contain mechanisms for expressing properties of Multiword Expressions (MWE), which, however, need improvement in how they account for idiosyncrasies of MWEs on the one hand and their similarities to regular structures on the other hand. This collaborative book constitutes a survey on various attempts at representing and parsing MWEs in the context of linguistic theories and applications.

ISBN 978-3-96110-145-0

<figure><img src="image_108.png" /></figure>

9 783961 101450