Abstract: Word2vec is a popular family of algorithms for unsupervised training of dense vector representations of words on large text corpuses. The resulting vectors have been shown to capture semantic relationships among their corresponding words, and have shown promise in reducing a number of natural language processing (NLP) tasks to mathematical operations on these vectors. While heretofore applications of word2vec have centered around vocabularies with a few million words, wherein the vocabulary is the set of words for which vectors are simultaneously trained, novel applications are emerging in areas outside of NLP with vocabularies comprising several 100 million words. Existing word2vec training systems are impractical for training such large vocabularies as they either require that the vectors of all vocabulary words be stored in the memory of a single server or suffer unacceptable training latency due to massive network data transfer. In this paper, we present a novel distributed, parallel training system that enables unprecedented practical training of vectors for vocabularies with several 100 million words on a shared cluster of commodity servers, using far less network traffic than the existing solutions. We evaluate the proposed system on a benchmark dataset, showing that the quality of vectors does not degrade relative to non-distributed training. Finally, for several quarters, the system has been deployed for the purpose of matching queries to ads in Gemini, the sponsored search advertising platform at Yahoo, resulting in significant improvement of business metrics.

Introduction: Embedding words in a common vector space can enable machine learning algorithms to achieve better performance in natural language processing (NLP) tasks. Word2vec BIBREF0 is a recently proposed family of algorithms for training such vector representations from unstructured text data via shallow neural networks. The geometry of the resulting vectors was shown in BIBREF0 to capture word semantic similarity through the cosine similarity of the corresponding vectors as well as more complex semantic relationships through vector differences, such as vec(“Madrid”) - vec(“Spain”) + vec(“France”) INLINEFORM0 vec(“Paris”). More recently, novel applications of word2vec involving unconventional generalized “words” and training corpuses have been proposed. These powerful ideas from the NLP community have been adapted by researchers from other domains to tasks beyond representation of words, including relational entities BIBREF1 , BIBREF2 , general text-based attributes BIBREF3 , descriptive text of images BIBREF4 , nodes in graph structure of networks BIBREF5 , and queries BIBREF6 , to name a few. While most NLP applications of word2vec do not require training of large vocabularies, many of the above mentioned real-world applications do. For example, the number of unique nodes in a social network BIBREF5 or the number of unique queries in a search engine BIBREF6 can easily reach few hundred million, a scale that is not achievable using existing word2vec implementations. The training of vectors for such large vocabularies presents several challenges. In word2vec, each vocabulary word has two associated INLINEFORM0 -dimensional vectors which must be trained, respectively referred to as input and output vectors, each of which is represented as an array of INLINEFORM1 single precision floating point numbers BIBREF0 . To achieve acceptable training latency, all vectors need to be kept in physical memory during training, and, as a result, word2vec requires INLINEFORM2 bytes of RAM to train a vocabulary INLINEFORM3 . For example, in Section SECREF2 , we discuss the search advertisement use case with 200 million generalized words and INLINEFORM4 which would thus require INLINEFORM5 = 480GB memory which is well beyond the capacity of typical commodity servers today. Another issue with large vocabulary word2vec training is that the training corpuses required for learning meaningful vectors for such large vocabularies, are themselves very large, on the order of 30 to 90 billion generalized words in the mentioned search advertising application, for example, leading to potentially prohibitively long training times. This is problematic for the envisioned applications which require frequent retraining of vectors as additional data containing new “words” becomes available. The best known approach for refreshing vectors is to periodically retrain on a suitably large window comprised of the most recent available data. In particular, we found that tricks like freezing the vectors for previously trained words don't work as well. The training latency is thus directly linked to staleness of the vectors and should be kept as small as feasible without compromising quality. Our main contribution is a novel distributed word2vec training system for commodity shared compute clusters that addresses these challenges. The proposed system: As discussed in Section SECREF4 , to the best of our knowledge, this is the first word2vec training system that is truly scalable in both of these aspects. We have implemented the proposed word2vec training system in Java and Scala, leveraging the open source building blocks Apache Slider BIBREF10 and Apache Spark BIBREF11 running on a Hadoop YARN-scheduled cluster BIBREF12 , BIBREF13 . Our word2vec solution enables the aforementioned applications to efficiently train vectors for unprecedented vocabulary sizes. Since late 2015, it has been incorporated into the Yahoo Gemini Ad Platform (https://gemini.yahoo.com) as a part of the “broad” ad matching pipeline, with regular retraining of vectors based on fresh user search session data.

Question: What languages do they use?
Answer: Train languages are: Cantonese, Bengali, Pashto, Turkish, Vietnamese, Haitian, Tamil, Kurdish, Tokpisin and Georgian, while Assamese, Tagalog, Swahili, Lao are used as target languages.