Table 3: Bilingual Lexicon Induction performance from es, it, nl to en. Gouws and Søgaard (2015) + Panlex/Wikt is our reimplementation using Panlex/Wiktionary dictionary. All our models use Panlex as the dictionary. We reported the recall at 1 and 5. The best performance is bold.

es-en || rec1

es-en || rec5

it-en || rec1

it-en || rec5

nl-en || rec1

nl-en || rec5

Average || rec1

Average || rec5

Model || Gouws and Søgaard (2015) + Panlex

37.6

63.6

26.6

56.3

49.8

76.0

38.0

65.3

Model || Gouws and Søgaard (2015) + Wikt

61.6

78.9

62.6

81.1

65.6

79.7

63.3

79.9

Model || BilBOWA: Gouws et al. (2015)

51.6

-

55.7

-

57.5

-

54.9

-

Model || Vulic and Moens (2015)

68.9

-

68.3

-

39.2

-

58.8

-

Model || Our model (random selection)

41.1

62.0

57.4

75.4

34.3

55.5

44.3

64.3

Model || Our model (EM selection)

67.3

79.5

66.8

82.3

64.7

82.4

66.3

81.4

Model || Our model (EM selection) + Joint model

68.0

80.5

70.5

83.3

68.8

84.0

69.1

82.6

Model || Our model (EM selection) + combine embeddings (δ = 0.01)

74.7

85.4

80.8

90.4

79.1

90.5

78.2

88.8

Model || Our model (EM selection) + lemmatization

74.9

86.0

81.3

91.3

79.8

91.3

78.7

89.5

Quantitative evaluation. Table 3 shows our results compared with prior work. We reimplement Gouws and Søgaard (2015) using Panlex and Wiktionary dictionaries. The result with Panlex is substantially worse than with Wiktionary. This confirms our hypothesis in §2. That is the context might be corrupted if we just randomly replace the training data with the translation from noisy dictionary such as Panlex. Our model when randomly picking the translation is similar to Gouws and Søgaard (2015), using the Panlex dictionary. The biggest difference is that they replace the training data (both context and middle word) while we fix the context and only replace the middle word. For a high coverage yet noisy dictionary such as Panlex, our approach gives better average score. Comparing our two most basic models (EM selection and random selection), it is clear that the model using EM to select the translation outperforms random selection by a significant margin. Our joint model, as described in equation (3) which predicts both target word and the translation, further improves the performance, especially for nl-en. We use equation (5) to combine both context embeddings V and word embeddings U for all three language pairs. This modification during training substantially improves the performance. More importantly, all our improvements are consistent for all three language pairs and both evaluation metrics, showing the robustness of our models. Our combined model out-performed previous approaches by a large margin. Vulic and Moens (2015) used bilingual comparable data, but this might be hard to obtain for some language pairs. Their performance on nl-en is poor because their comparable data between en and nl is small. Besides, they also use POS tagger and lemmatizer to filter only Noun and reduce the morphology complexity during training. These tools might not be available for many languages. For a fairer comparison to their work, we also use the same Treetagger (Schmid, 1995) to lemmatize the output of our combined model before evaluation. Table 3 (+lemmatization) shows some improvements but minor. It demonstrates that our model is already good at disambiguating morphology. For example, the top 2 translations for es word lenguas in en are languages and language which correctly prefer the plural translation.