Introduction	Embedding words in a common vector space can enable machine learning algorithms to achieve better performance in natural language processing (NLP) tasks. Word2vec  is a recently proposed family of algorithms for training such vector representations from unstructured text data via shallow neural networks. The geometry of the resulting vectors was shown in  to capture word semantic similarity through the cosine similarity of the corresponding vectors as well as more complex semantic relationships through vector differences, such as vec(“Madrid”) - vec(“Spain”) + vec(“France”) INLINEFORM0 vec(“Paris”). More recently, novel applications of word2vec involving unconventional generalized “words” and training corpuses have been proposed. These powerful ideas from the NLP community have been adapted by researchers from other domains to tasks beyond representation of words, including relational entities  ,  , general text-based attributes  , descriptive text of images  , nodes in graph structure of networks  , and queries  , to name a few. While most NLP applications of word2vec do not require training of large vocabularies, many of the above mentioned real-world applications do.

Network-Efficient Distributed Word2vec Training System for Large Vocabularies	Word2vec is a popular family of algorithms for unsupervised training of dense vector representations of words on large text corpuses. The resulting vectors have been shown to capture semantic relationships among their corresponding words, and have shown promise in reducing a number of natural language processing (NLP) tasks to mathematical operations on these vectors. While heretofore applications of word2vec have centered around vocabularies with a few million words, wherein the vocabulary is the set of words for which vectors are simultaneously trained, novel applications are emerging in areas outside of NLP with vocabularies comprising several 100 million words. Existing word2vec training systems are impractical for training such large vocabularies as they either require that the vectors of all vocabulary words be stored in the memory of a single server or suffer unacceptable training latency due to massive network data transfer. In this paper, we present a novel distributed, parallel training system that enables unprecedented practical training of vectors for vocabularies with several 100 million words on a shared cluster of commodity servers, using far less network traffic than the existing solutions.

Sponsored Search data set	We conducted qualitative evaluation in the context of sponsored search application described in Section SECREF2 . Figure FIGREF47 shows the queries whose trained vectors were found to be most similar (out of 133M queries) to an example ad vector, along with the respective cosine similarities to the ad vector. The figure shows the ten most and least similar among the 800 most similar queries, where we note that the ten least similar queries can still be considered to be fairly semantically similar. This particular set of vectors was trained for a vocabulary of 200M generalized words using the 300 dimensional vector, 15 PS shard settings described in Section SECREF41 . We found the vector quality demonstrated in Figure FIGREF47 to be the norm based on inspections of similar matchings of query vectors to a number of ad vectors. We also compared the cosine similarities for pairs of vectors trained using the proposed distributed system and for corresponding vector pairs trained using the open–source implementation of  , again on a large search session data set. The former was trained using a vocabulary of 200 million generalized words while the latter was trained using about 90 million words which is the most that could fit onto a specialized large memory machine.

As observed and partially theoretically justified in  (see also  ), in many applications involving sparse training data characterized by low average overlap between the model parameters associated with different minibatches, the model parameter updates arriving in parallel from multiple client nodes can be aggregated on the parameter server shards without locking, synchronization, or atomicity guarantees, and still result in a far better model accuracy versus training time latency trade-off than single threaded (i.e., sequential) training. The parameter server paradigm has been applied successfully to the training of very large models for logistic regression, deep learning, and factorization machines, and to sampling from the posterior topic distribution in large-scale Latent Dirichlet Allocation  ,  ,  ,  ,  ,  ,  ,  ,  . There have also been some attempts to extend the parameter-server approach to word2vec (e.g.,  ). These have followed the above computational flow, with each parameter server shard storing the input and output vectors for a subset of the vocabulary.

For a set of 7,560 generalized word pairs with words common to the vocabularies trained by the respective systems we found very good agreement in cosine similarities between the corresponding vectors from the two systems, with over 50% of word pairs having cosine similarity differences less than 0.06, and 91% of word pairs having differences less than 0.1.

For each INLINEFORM2 the word vectors are adjusted based on the gradient of the summation ( EQREF6 ) restricted to the input words belonging to INLINEFORM3 , as given by DISPLAYFORM0  The gradient of INLINEFORM0 with respect to the word vector components is 0 for all word vector components whose corresponding words do not appear as inputs, outputs, or negative examples in ( EQREF25 ). For the remaining components, the gradient is conveniently expressed in groups of components corresponding to specific word vectors. For example, consider a pair of indices INLINEFORM1 belonging to INLINEFORM2 . The gradient components corresponding to the word vector INLINEFORM3 can be expressed as DISPLAYFORM0  We see that evaluation of INLINEFORM0 requires computing the dot (or inner) products INLINEFORM1 appearing in the arguments to INLINEFORM2 and then computing linear combinations of the vectors INLINEFORM3 and INLINEFORM4 , with weights depending on the dot products. A similar expression and computation applies to the other gradient components corresponding to other word vectors appearing in INLINEFORM5 .

Another issue with large vocabulary word2vec training is that the training corpuses required for learning meaningful vectors for such large vocabularies, are themselves very large, on the order of 30 to 90 billion generalized words in the mentioned search advertising application, for example, leading to potentially prohibitively long training times. This is problematic for the envisioned applications which require frequent retraining of vectors as additional data containing new “words” becomes available. The best known approach for refreshing vectors is to periodically retrain on a suitably large window comprised of the most recent available data. In particular, we found that tricks like freezing the vectors for previously trained words don't work as well. The training latency is thus directly linked to staleness of the vectors and should be kept as small as feasible without compromising quality. Our main contribution is a novel distributed word2vec training system for commodity shared compute clusters that addresses these challenges. The proposed system: As discussed in Section SECREF4 , to the best of our knowledge, this is the first word2vec training system that is truly scalable in both of these aspects.

We have implemented the proposed word2vec training system in Java and Scala, leveraging the open source building blocks Apache Slider  and Apache Spark  running on a Hadoop YARN-scheduled cluster  ,  . Our word2vec solution enables the aforementioned applications to efficiently train vectors for unprecedented vocabulary sizes. Since late 2015, it has been incorporated into the Yahoo Gemini Ad Platform (https://gemini.yahoo.com) as a part of the “broad” ad matching pipeline, with regular retraining of vectors based on fresh user search session data.

In the next section, we present a different distributed system architecture for word2vec that requires significantly less network bandwidth for a given training throughput than the above conventional parameter server-based system, while continuing to accommodate large vocabularies and providing sufficient computational power to achieve the higher throughput allowed by the reduction in network bandwidth.

Implementation on Hadoop	We have implemented the system described in Section SECREF5 in Java and Scala on a Hadoop YARN scheduled cluster, leveraging Slider  and Spark  . Our end-to-end implementation of training carries out four steps: vocabulary generation, data set preprocessing, training, and vector export. We next review the details of each of these steps. Throughout, all data, including the initial training data, its preprocessed version, the exported vectors are all stored in the Hadoop Distributed File System (HDFS). We remark that although our compute environment is currently based on Hadoop and Spark, other distributed computational frameworks such as the recently released TensorFlow could also serve as a platform for implementing the proposed system.

The results of  were based on training the largest vocabulary that could fit into the large memory of a special purpose server, which resulted in learned vector representations for about 45 million words. The proposed training system herein enables increasing this by several fold, resulting in far greater coverage of queries and a potentially significant boost in query monetization, as indicated by Figure FIGREF5 .

Evaluation & Deployment	In this section, we provide evidence that the vectors trained by the proposed distributed system are of high quality, even with fairly aggressive parallelism during training. We also show bucket test results on live web search traffic that compare query-ad matching performance of our large-vocabulary model to the one trained using single-machine implementation, which led to the decision to deploy the proposed system in production in late 2015.

The ratio of the network bandwidths of the proposed system and a conventional parameter server based system is INLINEFORM0  For typical parameters of interest (we typically have INLINEFORM0 between 10 and 20, increasing with INLINEFORM1 between 300 and 1000), this is in the range of INLINEFORM2 to INLINEFORM3 , effectively eliminating network bandwidth as a bottleneck for training latency, relative to the conventional approach.

For example, the number of unique nodes in a social network  or the number of unique queries in a search engine  can easily reach few hundred million, a scale that is not achievable using existing word2vec implementations. The training of vectors for such large vocabularies presents several challenges. In word2vec, each vocabulary word has two associated INLINEFORM0 -dimensional vectors which must be trained, respectively referred to as input and output vectors, each of which is represented as an array of INLINEFORM1 single precision floating point numbers  . To achieve acceptable training latency, all vectors need to be kept in physical memory during training, and, as a result, word2vec requires INLINEFORM2 bytes of RAM to train a vocabulary INLINEFORM3 . For example, in Section SECREF2 , we discuss the search advertisement use case with 200 million generalized words and INLINEFORM4 which would thus require INLINEFORM5 = 480GB memory which is well beyond the capacity of typical commodity servers today.

The second A/B test was conducted to evaluate incremental improvement over the single machine solution, which was already launched in production. The model contained vectors for 133M queries. As it can be observed in the second row of Table TABREF48 , the distributed solution provided additional 2.44% query coverage and additional 9.39% revenue, without degrading user experience (CTR remained neutral). This strong monetization potential of our distributed system for training large vocabularies of query and ad vectors led to its deployment in our sponsored search platform. The model is being retrained on a weekly basis, automated via Apache Oozie  , and is currently serving more than INLINEFORM0 of all broad matches.

Distributed data-parallel	A similar drawback applies to distributed data-parallel training systems like those available in Apache Spark MLLib  and Deeplearning4j  . In the former, in each iteration the Spark driver sends the latest vectors to all Spark executors. Each executor modifies its local copy of vectors based on its partition of the training data set, and the driver then combines local vector modifications to update the global vectors. It requires all vectors to be stored in the memory of all Spark executors, and, similarly to its single machine counterparts, is thus unsuitable for large vocabularies. The Deeplearning4j system takes a similar approach and thus suffers from the same limitations, although it does enable the use of GPUs to accelerate the training on each machine.

Training step throughput	To give an idea of the kind of training throughput we can achieve with this system, the following is one configuration we have used for training the sponsored search application on our Hadoop cluster: Algorithm parameters: 200 million word vocabulary, 5 negative examples, maximum of 10 window size Training system parameters: 200 Spark executors, 8 threads per spark executor, minibatch size of 200 yields the following training throughputs in minibatch input words per second (see Section SECREF3 for the definition of input word), for varying PS shards and vector dimensions:  For this data set and algorithm parameters, each input word has associated with it an average of about 20 positive context words and negative examples, so that the system is effectively updating about 21 times the third column in the table number of vectors per second. For the first line of the table, for example, this is over 33 million 300 dimensional vector updates per second. The conventional parameter server approach would require a total bandwidth of about 300 Gbps (30 server shards would be needed for this) to and from the parameter server for similar training throughput. This is close to 10 percent of the fabric bandwidth in our production data center.

Conclusion	In this paper, we presented a novel scalable word2vec training system that, unlike available systems, can train semantically accurate vectors for hundreds of millions of vocabulary words with training latency and network bandwidth usage suitable for regular training on commodity clusters. We motivated the usefulness of large vocabulary word2vec training with a sponsored search application involving generalized “words” corresponding to queries, ads, and hyperlinks, for which the proposed system has been deployed to production. The results on both benchmark data sets and online A/B tests strongly indicate the benefits of the proposed approach. [ht] INLINEFORM0 .dotprod( INLINEFORM1 , INLINEFORM2 , long INLINEFORM3 )  INLINEFORM0 Random Number Generator initialized with INLINEFORM1 INLINEFORM2 ; INLINEFORM3 iterate over words in minibatch INLINEFORM4 INLINEFORM5 iterate over words in context INLINEFORM6 INLINEFORM7 INLINEFORM8 ; INLINEFORM9 generate INLINEFORM10 random negative examples for current output word INLINEFORM11 Array( INLINEFORM12 negative word indices INLINEFORM13 , generated using INLINEFORM14 ) compute partial dot products for positive and negative examples INLINEFORM15 INLINEFORM16 INLINEFORM17 send results back to client INLINEFORM18 Server side computation - dotprod.

Network bandwidth analysis	Using the same notation as in ( EQREF15 ), and letting INLINEFORM0 denote the number of shards, the average bytes transferred from all PS shards for each dotprod call is upper bounded by DISPLAYFORM0  That is, each shard transfers the partial dot product results between the input vector of each minibatch word and all context words (there are no more than an average of INLINEFORM0 of these per minibatch word) and negative examples (there are no more than INLINEFORM1 per context per minibatch word, or INLINEFORM2 per minibatch word). It is not hard to see that this is precisely the number of bytes transferred to all PS shards for the vector linear combination component of each adjust call. That is, there are two linear vector updates for each pair of vectors for which a dot product was computed, and these updates involve the same linear combination weight. Normalizing ( EQREF31 ) by the minibatch size, we have the following counterpart of ( EQREF15 ) for the bytes transferred, in each direction, per trained minibatch word, for the proposed scheme: DISPLAYFORM0  Notice that the vector dimension INLINEFORM0 has been replaced by the number of shards INLINEFORM1 .

Architecture	Our distributed word2vec training system (i.e., for maximizing ( EQREF6 )) is illustrated in Figure FIGREF18 , with pseudo code for the overall computational flow in Figures SECREF8 , SECREF8 , and SECREF8 in the Appendix. As can be seen in Figure FIGREF18 , the proposed system also features parameter-server-like components (denoted by “PS shards” in the figure), however they are utilized very differently and have very different capabilities from their counterparts in the conventional approach described above. We shall, however, continue to refer to these components as parameter server shards. The system features the following innovations, explained in more detail below, with respect to the conventional approach. Column-wise partitioning of word vectors among parameter server (PS) shards (as opposed to word-wise partitioning). No transmission of word vectors or vector gradients across the network. Server-side computation of vector dot products and vector linear combinations, distributed by column partitions. Distributed server-side generation of random negative examples via broadcasting of common random number generator seeds. In particular, avoiding the transmission of vectors and gradients greatly reduces network bandwidth requirements relative to the conventional approach.

The word vector portions are stored in each PS shard in arrays of primitive floats, and as mentioned, their indices in the arrays coincide with the indices of their corresponding words in the vocabulary. In the steady state, the PS allocates no new data structures to avoid garbage collection. Objects are created only during start-up, and possibly during the fairly infrequent connection setups, as managed by the Netty RPC layer. In this final step, carried out after training has completed, the partial vectors stored in each PS shard are aggregated and joined with their respective words in the vocabulary and stored together as a text file in HDFS. Again, we leverage Spark to carry out this operation in a distributed fashion, by creating an RDD from the vocabulary and using mapPartitions to launch clients that get the partial vectors from the PS shards for the respective partition of vocabulary words, combine the partial vectors and save corresponding word and vectors pairs to HDFS.

The formula arises from the fact that the input and output vectors for each term in the minibatch must be sent (this the '2' in the first factor in ( EQREF15 )), as must the output vectors for each random negative example. There are on average INLINEFORM0 of these per minibatch word. For INLINEFORM0 , values within the ranges recommended in  , this works out to INLINEFORM1 bytes transferred per word with each get and put. For 10 iterations of training on a data set of roughly 50 billion words, which is in the middle of the relevant range for the sponsored search application described in Section SECREF2 , attaining a total training latency of one week using the above system would require an aggregate bandwidth of at least 1300Gbits/sec to and from the parameter servers. This is impractically large for a single application on a commodity-hardware shared compute cluster. Moreover, one week training latency is already at the boundary of usefulness for our applications.

The first column shows results for the single machine implementation of  , the second for a 'low parallelism' configuration of our system using 50 Spark executors, minibatch size of 1, and 1 thread per executor, and the third column for a 'high parallelism' configuration again with 50 executors, but with minibatch size increased to 50 and 8 threads per executor. The various systems were run using the skipgram variant with 500 dimensional vectors, maximum window size of 20 (10 in each direction), 5 negative examples, subsample ratio of 1e-6 (see  ), initial learning rate of 0.01875, and 3 iterations over the data set. It can be seen that the vectors trained by the 'high parallelism' configuration of the proposed system, which is the closest to the configurations required for acceptable training latency in the large-scale sponsored search application, suffers only a modest loss in quality as measured by these tests. Note that this data set is more challenging for our system than the sponsored search data set, as it is less sparse and there is on average more overlap between words in different minibatches.

Parameter servers	A well-known distributed architecture for training very large machine learning models centers around the use of a parameter server to store the latest values of model parameters through the course of training. A parameter server is a high performance, distributed, in-memory key-value store specialized to the machine learning training application. It typically needs to support only fixed-size values corresponding to the model parameters, and also may support additive updates of values in addition to the usual key-value gets and puts. A parameter server-based training system also includes a number of worker/learner/client nodes that actually carry out the bulk of the training computations. The client nodes read in and parse training data in chunks or minibatches, fetch the model parameters that can be updated based on each minibatch, compute the updates (e.g., via gradient descent with respect to a minibatch restriction of the objective), and transmit the changes in parameter values to the parameter server shards which either overwrite or incrementally update these values in their respective in-memory stores.

Main steps	This step entails counting occurrences of all words in the training corpus and sorting them in order of decreasing occurrence. As mentioned, the vocabulary is taken to be the INLINEFORM0 most frequently occurring words, that occur at least some number INLINEFORM1 times. It is implemented in Spark as a straight-forward map-reduce job. In this step, each word in the training corpus is replaced by its index in the sorted vocabulary generated in the preceding phase (the ordering INLINEFORM0 referred to in Section SECREF5 ). This is also implemented in Spark using a low overhead in-memory key-value store to store the mapping from vocabulary words to their indices. Our implementation hashes words to 64 bit keys to simplify the key-value store. Referring to the system description in Section SECREF5 (and Figure FIGREF18 ), the parameter server portion is implemented in Java, with the RPC layer based on the Netty client-server library  . The RPC layer of the client is implemented similarly. The higher layers of the client (i/o, minibatch formation, partial dot product aggregation, linear combination weight computation) are implemented in Scala and Spark.

Sponsored search use case	Sponsored search is a popular advertising model  used by web search engines, such as Google, Microsoft, and Yahoo, in which advertisers sponsor the top web search results in order to redirect user's attention from organic search results to ads that are highly relevant to the entered query. Most search engines provide a self-service tool in which the advertisers can create their own ads by providing ad creative to be shown to the users, along with a list of bid terms (i.e., queries for which advertisers wish to show their ad). Due to a large number of unique queries it is challenging for advertisers to identify all queries relevant to their product or service. For this reason search engines often provide a service of “broad” matching, which automatically finds additional relevant queries for advertisers to bid on. This is typically implemented by placing queries and ads in a common feature space, such as bag-of-words using tf-idf weighting, and calculating similarity between ads and queries using a feature space metric in order to find good broad match candidates. In an unconventional application of word2vec to historical search logs, one could train query and ad vectors that capture semantic relationships and find relevant broad match candidates in the resulting feature space.

We refer to this as a 'column-wise' partitioning of the vectors, or more specifically, of the matrix whose rows correspond to the word vectors, as in INLINEFORM19  where INLINEFORM0 are the words in the vocabulary according to a fixed ordering INLINEFORM1 (e.g., by decreasing frequency of occurrence in the corpus). In the sequel, we shall equate each word INLINEFORM2 with INLINEFORM3 , its index in this ordering, so that INLINEFORM4 , and so on. For INLINEFORM5 shards, the vocabulary size can thus be scaled up by as much as a factor of INLINEFORM6 relative to a single machine. The vectors are initialized in the parameter server shards as in  . Multiple clients running on cluster nodes then read in different portions of the corpus and interact with the parameter server shards to carry out minibatch stochastic gradient descent (SGD) optimization of ( EQREF6 ) over the word vectors, following the algorithm in Figure SECREF8 (in the appendix). Specifically, the corpus is partitioned into disjoint minibatches with index sets INLINEFORM0 wherein each INLINEFORM1 is a subset of (sentence index, word index) pairs.

Note that although the search result hyperlinks clicked by the user are not needed for the sponsored search system, they are nevertheless important to include during training as they help propagate relevance between the queries and ads of interest. Given trained query and ad vectors, finding relevant queries for a given ad amounts to calculating cosine similarity between the ad vector and all query vectors. The INLINEFORM0 queries with the highest similarity are retrieved as broad matches. As illustrated in Figure FIGREF5 for representative search session data, the fraction of query occurrences in the search sessions for which vectors are available, and hence for which potential ads can be found using this vector-based approach, increases at a steady pace with the number of queries in the vocabulary, even with as many as 120 million queries, each occurring at least 5 times. This observation suggests that this application can benefit greatly from vocabularies of 200 million or more generalized words. Moreover, we found that there are around 800 million generalized words that occur 5 or more times in our largest data sets, indicating that additional scaling far beyond 200 million is well worth pursuing.

In fact, if we attempt to increase the parallelism to 200 executors as was used for the training of the vectors described in the next subsection, training fails to converge altogether. We are unsure why our system yields better results than the implementation of  on the wordsim test, yet worse scores on the analogies test. We also note that the analogies test scores reported here involve computing the closest vector for each analogy “question” over the entire vocabulary and not just over the 1M most frequent words, as in the script 'demo-train-big-model-v1-compute-only.sh' of  .

[ht] void INLINEFORM19 .adjust( INLINEFORM20 , INLINEFORM21 , INLINEFORM22 , INLINEFORM23 , INLINEFORM24 )  INLINEFORM0 Random Number Generator initialized with INLINEFORM1 INLINEFORM2 ; INLINEFORM3 ; INLINEFORM4 ; INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8 INLINEFORM9 INLINEFORM10 ; INLINEFORM11 regenerate random negative examples INLINEFORM12 Array( INLINEFORM13 negative word indices INLINEFORM14 , generated using INLINEFORM15 ) compute partial gradient updates and store in scratch area INLINEFORM16 ; INLINEFORM17 INLINEFORM18 INLINEFORM19 ; INLINEFORM20 add partial gradient updates to partial vectors in store all INLINEFORM21 INLINEFORM22 ; INLINEFORM23 Server side computation - adjust.  [ht] InputinputOutputoutput INLINEFORM0 : Vectors for vocabulary words INLINEFORM1 = # of parameter servers needed for INLINEFORM2 words Launch parameter servers INLINEFORM3 Initialize vectors in PS server iteration INLINEFORM4 INLINEFORM5 UnprocessedPartitions INLINEFORM6 INLINEFORM7 each executor,

Benchmark data set	To compare the proposed distributed system we trained vectors on a publicly available data set collected and processed by the script 'demo-train-big-model-v1-compute-only.sh' from the open-source package of  . This script collects a variety of publicly available text corpuses and processes them using the algorithm described in  to coalesce sufficiently co-occurring words into phrases. We then randomly shuffled the order of sentences (delimited by new line) in the data set, retaining order of words within each sentence. The resulting data set has about 8 billion words and yields a vocabulary of about 7 million words and phrases (based on a cut off of 5 occurrences in the data set). We evaluated accuracy on the phrase analogies in the 'question-phrases.txt' file and also evaluated Spearman's rank correlation with respect to the editorial evaluation of semantic relatedness of pairs of words in the well known wordsim-353 collection  . The results are shown in Table TABREF34 .

The word2vec training problem	In this paper we focus on the skipgram approach with random negative examples proposed in  . This has been found to yield the best results among the proposed variants on a variety of semantic tests of the resulting vectors  ,  . Given a corpus consisting of a sequence of sentences INLINEFORM0 each comprising a sequence of words INLINEFORM1 , the objective is to maximize the log likelihood: DISPLAYFORM0  over input and output word row vectors INLINEFORM0 and INLINEFORM1 with INLINEFORM2 ranging over the words in the vocabulary INLINEFORM3 , where: We follow  for setting INLINEFORM0 and select words occurring in the corpus a sufficient number of times (e.g., at least 5 times), or, if this results in too many words, as the most frequently occurring INLINEFORM1 words, where INLINEFORM2 is the largest number words that can be handled by available computational resources. We further also assume a randomized version of ( EQREF6 ) according to the subsampling technique of  , which removes some occurrences of frequent words. The algorithm for maximizing ( EQREF6 ) advocated in  , and implemented in its open–source counterpart, is a minibatch stochastic gradient descent (SGD).

The RPC calls are detailed in Figures SECREF8 and SECREF8 (in the Appendix), and, at a higher level, entail the following server/shard side operations: dotprod: Select negative examples INLINEFORM0 in ( EQREF26 ) according to a probability distribution derived from the vocabulary histogram proposed in  , but with the client thread supplied seed initializing the random number generation, and then return all partial dot products required to evaluate the gradient ( EQREF26 ) for all positive output, negative output, and input word vectors associated with the minibatch, wherein the partial dot products involve those vector components stored on the designated shard: INLINEFORM1 . adjust: Regenerate negative examples used in preceding dotprod call using the same seed that is again supplied by the client thread. Compute ( EQREF27 ) for vector components associated with the minibatch stored on the shard as a partial vector (restricted to components stored on shard) linear combination using weights received from the client. Between these two RPCs the client computes the linear combination weights needed for adjust by summing the partial inner products returned by the shards in response to the dotprod calls and evaluating the sigmoid function at values given by the aggregated dot products.

Both methods produced a separate query-ad match dictionary by finding INLINEFORM0 nearest ads in the embedding space for each search query from our vocabulary, and keeping only ads with cosine similarity above INLINEFORM1 . The threshold was chosen based on editorial results. To implement the bucket test the query-ad match dictionary is produced offline and cached in the ad server memory such that ads can be retrieved in real-time given an input query. Post retrieval, a click model is used to estimate the clickability of the ad for that query and the ad is sent into an auction, where it competes with ads retrieved by other broad match algorithms. It gets to be shown to the user in case it wins one of the ad slots on the page. The first A/B test was conducted to evaluate the value of query-ad dictionary produced by single-machine implementation. This implementation could scale up to a model with 50M query vectors. It was compared against control bucket that ran a production broad match module. Following positive A/B test metrics, with improvements in coverage and revenue, presented in the first row of Table TABREF48 , the dictionary was launched to production and incorporated into the existing broad match production model.

The idea of using word2vec to train query representations is not new and has been suggested by several researchers in the past  ,  . However, until now, it was not possible to use the algorithm to its fullest extent due to computational limitations of existing word2vec implementations. The sponsored search training corpus consists of billions of user search sessions each comprising generalized “words” corresponding to entire user queries (not the individual words in the queries), clicked hyperlinks, and clicked advertisements, ordered according to the temporal ordering of the corresponding user actions. Figure FIGREF1 shows a snippet from such a training corpus wherein the clicked ads and search link clicks are encoded as string IDs prefixed by “adid_” and “slc_”, respectively. The queries are highlighted in bold. The goal is to train vector representations for queries, hyperlinks, and advertisements, and to use the semantic similarity captured by these vectors to target advertisements to semantically relevant queries that might otherwise not be found to be relevant using more conventional measures, such as prior clicks or the number of constituent words common to the query and advertisement meta data (i.e., title, description, bid keywords).

Our training system is also based on minibatch SGD optimization of ( EQREF6 ), however, as described in Section SECREF5 , it is carried out in a distributed fashion in a manner quite different from the implementation of  . Any form of minibatch SGD optimization of ( EQREF6 ) involves the computation of dot products and linear combinations between input and output word vectors for all pairs of words occurring within the same window (with indices in INLINEFORM0 ). This is a massive computational task when carried out for multiple iterations over data sets with tens of billions of words, as encountered in applications described in the previous section.

in parallel UnprocessedPartitions is non-empty INLINEFORM8 INLINEFORM9 next partition in UnprocessedPartitions Launch client INLINEFORM10 connected to INLINEFORM11 INLINEFORM12 INLINEFORM13 minibatches in INLINEFORM14 INLINEFORM15 = randomly select a seed INLINEFORM16 INLINEFORM17 Array of word indices in INLINEFORM18 INLINEFORM19 INLINEFORM20 Array of Arrays of context word indices of words in INLINEFORM21 client broadcasts word indices to shards which compute partial dot products in parallel, returning results to client INLINEFORM22 INLINEFORM23 , in parallel INLINEFORM24 = INLINEFORM25 .dotprod( INLINEFORM26 , INLINEFORM27 , INLINEFORM28 ) aggregate partial dot products and compute linear coefficients for gradient update INLINEFORM29 INLINEFORM30 ; INLINEFORM31 client broadcasts coefficients to shards which compute partial vector linear combinations INLINEFORM32 INLINEFORM33 , in parallel INLINEFORM34 .adjust( INLINEFORM35 , INLINEFORM36 , INLINEFORM37 , INLINEFORM38 , INLINEFORM39 ) input vectors INLINEFORM40 } from INLINEFORM41 Grid based word2vec algorithm.

Single machine	Several existing word2vec training systems are limited to running on a single machine, though with multiple parallel threads of execution operating on different segments of training data. These include the original open source implementation of word2vec  , as well as those of Medallia  , and Rehurek  . As mentioned in the introduction, these systems would require far larger memory configurations than available on typical commodity-scale servers.

Multiple client nodes process minibatches of the training corpus, determining for each word in each minibatch the associated context words and random negative examples, issuing get requests to the parameter server shards for the corresponding vectors, computing the gradients with respect to each vector component, and issuing put or increment requests to update the corresponding vectors in the parameter server shards. Unfortunately, such a conventional parameter server-based word2vec training system requires too much network bandwidth to achieve acceptable training throughput. Using the skipgram training algorithm and denoting algorithm parameters as INLINEFORM0 for vector dimension, INLINEFORM1 for number of words per minibatch, INLINEFORM2 for average context size, and INLINEFORM3 for the number of random negative examples per context word, assuming negligible repetition of words within the minibatch and among the negative examples, and further assuming that vectors and their gradients are communicated and stored as arrays of single-precision floating point numbers at 4 bytes each, the amount of word vector data transferred for each get and put call from and to the parameter server, respectively, is on average INLINEFORM4 , or about DISPLAYFORM0  bytes per trained minibatch word.

We are not aware of any existing systems for training word2vec or its close relatives, matrix factorization and collaborative filtering (i.e., those systems cited in the previous section), that distribute vectors and compute in the manner of the proposed system. In our system, a number of parameter server shards each stores a designated portion of every input (row) vector INLINEFORM0 INLINEFORM1 INLINEFORM2 and output (row) vector INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 (dependence of components on INLINEFORM7 is suppressed). For example, assuming a vector dimension INLINEFORM8 , 10 parameter server shards, and equi-partitioned vectors, shard INLINEFORM9 would store the 30 components of INLINEFORM10 and INLINEFORM11 with indices INLINEFORM12 in the range INLINEFORM13 . We shall denote shard INLINEFORM14 stored portion of INLINEFORM15 and INLINEFORM16 as INLINEFORM17 and INLINEFORM18 , respectively.

The vector INLINEFORM6 (and, correspondingly, the other vectors as well) are updated according to the usual SGD update rule DISPLAYFORM0  where INLINEFORM0 is a (suitably small) learning rate. Once a client has assembled the indices (indexing according to the order INLINEFORM0 above) of positive output examples and input words corresponding to a minibatch INLINEFORM1 , it interacts with the parameter server shards to compute ( EQREF26 ) and ( EQREF27 ) using two remote procedure calls (RPCs), dotprod and adjust, which are broadcasted to all PS shards, along with an intervening computation to aggregate results from the dotprod RPC returned by each shard.

These weights are then passed to the adjust RPC, along with the seeds for regenerating the identical random negative example indices INLINEFORM0 that were generated during the dotprod RPC. The retransmission simplifies the server in that state need not be maintained between corresponding dotprod and adjust calls. Note that the same seeds are sent to all shards in both calls so that each shard generates the same set of negative example indices. The shards are multithreaded and each thread handles the stream of RPC's coming from all client threads running on a single node. In a typical at scale run of the algorithm, the above process is carried out by multiple client threads running on each of a few hundred nodes, all interacting with the PS shards in parallel. The data set is iterated over multiple times and after each iteration, the learning rate INLINEFORM0 is reduced in a manner similar to the open source implementation of  . Note that there is no locking or synchronization of the word vector state within or across shards or across client threads during any part of the computation. The only synchronization in effect is that the RPC broadcast ensures that all shards operate on the same set of word vector indices for computing their portion of the corresponding calls.

Additionally, the client threads independently wait for all responses to their corresponding dotprod calls before proceeding. The lack of synchronization introduces many approximations into the overall SGD computation, similar in spirit to the HOGWILD  and Downpour SGD  distributed optimization schemes. For example, here, in the worst case, the state of the vectors associated with a minibatch could change between the dotprod and adjust calls issued by a single client thread. Nevertheless, despite such approximations, our distributed algorithm incurs surprisingly little degradation in the quality of the trained vectors as compared to single machine solutions (in cases where the computation can be carried out on one machine), as shown in Section SECREF7 . Two details of our version of the algorithm and implementation are helpful for improving convergence/performance on some data sets. One is that in the adjust computation (Figure SECREF8 ) the word vectors belonging to the minibatch are not updated until the end of the call so that references to word vectors throughout the call are to their values at the start of the call.

The second is an option for interleaved minibatch formation, which can be used to ensure that indices INLINEFORM0 of input words belonging to a minibatch are sufficiently separated in the training corpus, and ideally, belong to different sentences. This allows input word vectors within a sentence (which are linked through their overlapping output word windows) to “learn” from each other during a single training iteration, as their respective minibatches are processed.

In particular, the clients are created and connect to the PS shards from within an RDD mapPartitions method applied to the preprocessed data set that is converted to an RDD via the standard Spark file-to-RDD api. At the start of training, the PS shards are launched from a gateway node onto Hadoop cluster nodes using the Apache Slider application that has been designed to launch arbitrary applications onto a Hadoop YARN scheduled cluster. The IP addresses and ports of the respective PS shards are extracted and passed to the Spark executors (which in turn use them to connect respective clients to the PS shards) as a file via the standard spark-submit command line executed on a gateway node. Each mapPartitions operation in the clients is multi-threaded with a configurable number of threads handling the processing of the input data and the interaction with the PS shards. These threads share the same connections with the PS shards. The PS shards are also multi-threaded based on Netty, wherein a configurable number of worker threads process incoming dotprod and adjust requests from multiple connections in parallel. Each shard has a connection to each Spark executor.

The proposed system requires only about 15 Gbps, making it far more practical for deployment to production in a shared data center, especially in light of the training latency for which this bandwidth must be sustained, which is about two days for data sets of interest. Even more extreme is the last line of the table (the 1000 dim. case), for which the equivalent throughput conventional system would require 800 Gbps vs. 20 Gbps for the proposed system. One important property of the training system is that its throughput at any given time is limited by the throughput of the slowest PS shard at that time. With this in mind, we use the YARN scheduler resource reservation capability exported through Slider to minimize resource contention on all of the machines to which the PS shards are assigned, thereby achieving higher sustained throughput. Another important property of the training system is that increasing the number of shards beyond some point is not helpful since the vector portions handled by each shard become so small that the random access memory transaction bandwidth (number of random cache lines per second) becomes the bottle neck. This explains the limited throughput scaling with PS shards for the 300 dimensional case above.

Further optimization of the vector-store of each PS shard with respect to caching and non-uniform memory access might be beneficial. We leave this for future investigation.

Online A/B tests	Following successful offline evaluation of the proposed distributed system, in the following set of experiments we conducted tests on live web search traffic. We ran two bucket tests, each on INLINEFORM0 of search traffic, where we compared query-ad matches produced by training query and ad vectors using search session data set spanning 9 months of search data. One model was trained using implementation from  and the other was trained using the proposed distributed system. Both buckets were compared against control bucket, which employed a collection of different broad match techniques used in production at the time of the test. Each of the online tests were run for 10 days, one after another, more than a month apart. The results of the tests were reported in terms of query coverage (portion of queries for which ads were shown), Auction Depth (number of ads per query that made it into an auction) click-through rate (CTR, or number of ad clicks divided by number of ad impressions), click yield (number of clicks), and revenue. Instead of the actual numbers we show relative improvement over control metrics.

We evaluate the proposed system on a benchmark dataset, showing that the quality of vectors does not degrade relative to non-distributed training. Finally, for several quarters, the system has been deployed for the purpose of matching queries to ads in Gemini, the sponsored search advertising platform at Yahoo, resulting in significant improvement of business metrics.

Question: What domains are considered that have such large vocabularies?
Answer: relational entities, general text-based attributes, descriptive text of images, nodes in graph structure of networks, queries