SlowGuess/ACL_2020_OCR huggingface dataset:
the observation that smaller context windows tend to produce syntax-oriented embeddings (Levy and Goldberg, 2014a). However, the performance of the small-window embeddings does not transfer to the test languages. The best context window for the English development data (the size of 1) performs the worst for all the test languages, and the transferred accuracy seems to benefit from larger context sizes, although it does not always correlate with the window size. This observation highlights the difficulty of transferring syntactic knowledge across languages. Word embeddings trained with small windows capture more grammatical aspects of words in each language, which, as different languages have different grammars, makes the source and target embedding spaces so different that it is difficult to align them.
In summary, a general trend we observe here is that good context windows in the source language task do not necessarily produce good transferrable bilingual embeddings. In practice, it seems better to choose a context window that aligns the source and target well, rather than using the window size that just performs the best for the source language.
Despite their obvious connection, the relation between the choice of context window and the structural similarity of two embedding spaces has not been fully investigated in prior work. In this study, we have offered the first thorough empirical results on the relation between the context window size and bilingual embeddings, and shed new light on the property of bilingual embeddings. In summary, we have shown that:
- larger context windows for both the source and target facilitate the alignment of words, especially nouns.
- for cross-lingual transfer, the best context window for the source task is often not the best for test languages. Especially for dependency parsing, the smallest context size produces the best result for the source task, but performs the worst for test languages.
We hope that our study will provide insights into ways to improve cross-lingual embeddings by not only mapping methods but also the properties of monolingual embedding spaces.
11https://github.com/facebookresearch/MLDoc
$^{12}$ https://universaldependencies.org/treebanks/en_ewt/index.html
13 https://universaldependencies.org/conll17/
We thank the anonymous reviewers for their valuable comments and suggestions. This work was supported by JST CREST Grant Number JP-MJCR1513, Japan.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289-2294, Austin, Texas. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, Vancouver, Canada. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia. Association for Computational Linguistics.
Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238-247, Baltimore, Maryland. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135-146.
Fabienne Braune, Viktor Hangya, Tobias Eder, and Alexander Fraser. 2018. Evaluating bilingual word embeddings on the long tail. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 188-193, New Orleans, Louisiana. Association for Computational Linguistics.
Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019. Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 974-983, Hong Kong, China. Association for Computational Linguistics.
Timothy Dozat and Christopher D Manning. 2017. Deep Biafine Attention for Neural Dependency Parsing. In Proceedings of the International Conference on Learning Representations.
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning Crosslingual Word Embeddings without Bilingual Corpora. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1285-1295, Austin, Texas. Association for Computational Linguistics.
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710-721, Florence, Italy. Association for Computational Linguistics.
Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. BilBOWA: Fast Bilingual Distributed Representations without Word Alignments. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 748-756, Lille, France.
Henry. Kucera and W. Nelson. Francis. 1967. Computational analysis of present-day American English. Brown University Press.
Guillaume Lample, Alexis Conneau, Marc Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2018. Word Translation without Parallel Data. In Proceedings of the International Conference on Learning Representations.
Omer Levy and Yoav Goldberg. 2014a. Dependency-Based Word Embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302-308, Baltimore, Maryland. Association for Computational Linguistics.
Omer Levy and Yoav Goldberg. 2014b. Linguistic Regularities in Sparse and Explicit Word Representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171-180, Ann Arbor, Michigan. Association for Computational Linguistics.
Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the Association for Computational Linguistics, 3:211-225.
Bofang Li, Tao Liu, Zhe Zhao, Buzhou Tang, Aleksandr Drozd, Anna Rogers, and Xiaoyong Du. 2017. Investigating Different Syntactic Context Types and Context Representations for Learning Word Embeddings. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2421-2431, Copenhagen, Denmark. Association for Computational Linguistics.
Wang Ling, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015a. Two/Too Simple Adaptations of Word2Vec for Syntax Problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1299-1304, Denver, Colorado. Association for Computational Linguistics.
Wang Ling, Yulia Tsvetkov, Silvio Amir, Ramón Fernández, Chris Dyer, Alan W Black, Isabel Trancoso, and Chu-Cheng Lin. 2015b. Not All Contexts Are Created Equal: Better Word Representations with Variable Attention. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1367-1372, Lisbon, Portugal. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual Word Representations with Monolingual Quality in Mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151-159, Denver, Colorado. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In Proceedings of the International Conference on Learning Representations.
Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting Similarities among Languages for Machine Translation. Computing Research Repository, arXiv:1309.4168.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111-3119. Curran Associates, Inc.
Peter Prettenhofer and Benno Stein. 2010. Cross-Language Text Classification Using Structural Correspondence Learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1118-1127, Uppsala, Sweden. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A Survey of Cross-lingual Word Embedding Models. Journal of Artificial Intelligence Research, 65(1):569-630.
Holger Schwenk and Xian Li. 2018. A Corpus for Multilingual Document Classification in Eight Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A Gold Standard Dependency Corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2897-2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
Samuel L Smith, David H P Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline Bilingual Word Vectors, Orthogonal Transformations and the Inverted Softmax. In Proceedings of the International Conference on Learning Representations, pages 1-10.
Anders Søgaard, Sebastian Ruder, and Ivan Vulic. 2018. On the Limitations of Unsupervised Bilingual Dictionary Induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778-788, Melbourne, Australia. Association for Computational Linguistics.
Peter D. Turney and Patrick Pantel. 2010. From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research, 37(1):141-188.
Ivan Vulic and Marie-Francine Moens. 2016. Bilingual Distributed Word Representations from Document-aligned Comparable Data. Journal of Artificial Intelligence Research, 55(1):953-994.
Takashi Wada, Tomoharu Iwata, and Yuji Matsumoto. 2019. Unsupervised multilingual word embedding with limited resources using neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3113-3124, Florence, Italy. Association for Computational Linguistics.
Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. 2018. Factors influencing the surprising instability of word embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2092-2102, New Orleans, Louisiana. Association for Computational Linguistics.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.
A The hyperparameters for training monolingual word embeddings
hyperparameter	Source embeddings (1M sentences)	Target embeddings (5M sentences)
embedding size	300
number of negative samples	15
alpha (learning rate)	0.025 (linearly decayed during training)
minimum word count	10	15
number of iterations	10	5
B The hyperparameters for downstream tasks
B.1 Document Classification and Sentiment Analysis
hyperparameters
CNN Classifier	number of filters	100
ngram_filter_sizes	2, 3, 4, 5
MLP hidden size	64
Training	optimizer	Adam
learning rate	0.001
lr scheduler	halved each time the dev score stops improving
patience	3
batch size	64
B.2 Dependency Parsing
hyperparameters
Graph-based Parser	LSTM hidden size	200
LSTM number of layers	3
tag representation dim	100
arc representation dim	500
pos tag embedding dim	50
Training	optimizer	Adam
learning rate	0.001
lr scheduler	halved each time the dev score stops improving
patience	3
batch size	32