Title: No title found

Abstract: Word embeddings extract semantic features of words from large datasets of text. Most embedding methods rely on a log-bilinear model to predict the occurrence of a word in a context of other words. Here we propose word2net, a method that replaces their linear parametrization with neural networks. For each term in the vocabulary, word2net posits a neural network that takes the context as input and outputs a probability of occurrence. Further, word2net can use the hierarchical organization of its word networks to incorporate additional meta-data, such as syntactic features, into the embedding model. For example, we show how to share parameters across word networks to develop an embedding model that includes part-of-speech information. We study word2net with two datasets, a collection of Wikipedia articles and a corpus of U.S. Senate speeches. Quantitatively, we found that word2net outperforms popular embedding methods on predicting heldout words and that sharing parameters based on part of speech further boosts performance. Qualitatively, word2net learns interpretable semantic representations and, compared to vector-based methods, better incorporates syntactic information.

I:

Word embeddings are an important statistical tool for analyzing language, processing large datasets of text to learn meaningful vector representations of the vocabulary (Bengio et al., 2003; 2006; Mikolov et al., 2013b; Pennington et al., 2014) . Word embeddings rely on the distributional hypothesis, that words used in the same contexts tend to have similar meanings (Harris, 1954) . More informally (but equally accurate), a word is defined by the company it keeps (Firth, 1957) .

While there are many extensions and variants of embeddings, most rely on a log-bilinear model. This model posits that each term is associated with an embedding vector and a context vector. Given a corpus of text, these vectors are fit to maximize an objective function that involves the inner product of each observed word's embedding with the sum of the context vectors of its surrounding words. With useful ways to handle large vocabularies, such as negative sampling (Mikolov et al., 2013a) or Bernoulli embeddings (Rudolph et al., 2016) , the word embedding objective resembles a bank of coupled linear binary classifiers.

Here we introduce word2net, a word embedding method that relaxes this linear assumption. Word2net still posits a context vector for each term, but it replaces each word vector with a term-specific neural network. This word network takes in the sum of the surrounding context vectors and outputs the occurrence probability of the word. The word2net objective involves the output of each word's network evaluated with its surrounding words as input. The word2net objective resembles a bank of coupled non-linear binary classifiers. How does word2net build on classical word embeddings? The main difference is that the word networks can capture non-linear interaction effects between co-occurring words; this leads to a better model of language. Furthermore, the word networks enable us to share per-term parameters based on word-level meta-data, such as syntactic information. Here we study word2net models that share parameters based on part-of-speech ( ) tags, where the parameters of certain layers of each network are shared by all terms tagged with the same tag.

Figure 1a illustrates the intuition behind word2net. Consider the term . The top of the figure shows one observation of the word, i.e., one of the places in which it appears in the data. (This excerpt is from U.S. Senate speeches.) From this observation, the word2net objective contains the probability of a binary variable w n; conditional on its context (i.e., the sum of the context vectors of the surrounding words). This variable is whether occurred at position n. . The most similar word/tag pairs for the former include mostly verbs such as "decrease," while for the latter the most similar results are nouns such as "cut." neural network that outputs the probability of that word (Figure 1a ). If we are given the tags of the words, we may use parameter sharing instead in order to form a per-word per-tag neural network (Figure 1b ). Finally, we also propose a method for computing similarities between the neural network representations of the words and demonstrate that they capture semantic (and even syntactic) similarities (Figure 1c ).

In our empirical study, we show that parameter sharing in word2net performs better than applying word2vec or standard Benoulli embeddings on the augmented vocabulary of word/tag pairs. We also demonstrate that deep Bernoulli embeddings provide better predictive log-likelihood when compared to word2vec or standard Bernoulli embeddings. R fjrr: moved this to the introduction, needs rewriting here Word embedding models learn semantic features of words by exploiting the co-occurrence patterns of words in a collection of documents. There are many extensions and variants of word embeddings (Bengio et al., 2003; 2006; Mnih & Hinton, 2007; Mikolov et al., 2013a; b; c; Pennington et al., 2014; Mnih & Teh, 2012; Mnih & Kavukcuoglu, 2013; Levy & Goldberg, 2014; Vilnis & McCallum, 2015; Barkan, 2016; Bamler & Mandt, 2017) . Most of these approaches rely on a log-bilinear model, in which the emission probabilities depend on a dot product of the word embedding vectors and the context vectors, as opposed to the deep neural network architectures proposed by Bengio et al. (2003; 2006) and Mnih & Hinton (2007) . Our model di ers from these deep neural network architectures in two ways. First, we have a separate network for each vocabulary word, instead of a single network that outputs the logits for all words in the vocabulary. Our perspective of a bank of parallel binary classification problems allows for faster optimization of the networks. Second, our architecture enables incorporating side information (such as part of speech tags) in specific layers of the network. Recall that word embeddings (without any further structure) tend to capture semantic properties of the words, and the syntactic properties they encode are typically redundant (Andreas & Klein, 2014), so there is room for improvement with a model that allows for additional syntactic structure.

We adopt the perspective of exponential family embeddings (Rudolph et al., 2016) , which extend word embeddings to datasets beyond text. There are also some variants and extensions of exponential family embeddings (Rudolph & Blei, 2017; Rudolph et al., 2017; Liu & Blei, 2017; Liu et al., 2017) , but they all have in common an exponential family likelihood whose natural parameter is determined The idea behind word2net is that the conditional probability of w n;

is the output of a multi-layer network that takes the context as input. Each layer of the network transforms the context into a new hidden representation, reweighting the latent features according to their relevance for predicting the occurrence of . Note that not illustrated are the 0-variables, i.e., the negative samples, which correspond to words that are not at position n. In word2net, their probabilities also come from their corresponding word networks. Now suppose we have tagged the corpus with . Figure 1b shows how to incorporate this syntactic information into word2net. The network is specific to as a noun (as opposed to a verb). The parameters of the first layer (orange) are shared among all nouns in the collection; the other layers (blue) are specific to . Thus, the networks for / and / differ in how the first layer promotes the latent aspects of the context, i.e., according to which context features are more relevant for each tag. This model further lets us consider these two tags separately. Figure 1c shows the most similar words to each sense of ; the method correctly picks out tagged words related to the verb and related to the noun.

Below, we develop the details of word2net and study its performance with two datasets, a collection of Wikipedia articles and a corpus of U.S. Senate speeches. We found that word2net outperforms popular embedding methods on predicting held-out words, and that sharing parameters based on further boosts performance. Qualitatively, word2net learns interpretable semantic representations and, compared to vector-based methods, better incorporates syntactic information.

Related work.:

Word2net builds on word embeddings methods. Though originally designed as deep neural network architectures (Bengio et al., 2003; 2006; Mnih & Hinton, 2007) , most applications of word embeddings now rely on log-bilinear models (Mikolov et al., 2013a; b; c; Pennington et al., 2014; Mnih & Teh, 2012; Mnih & Kavukcuoglu, 2013; Levy & Goldberg, 2014; Vilnis & McCallum, 2015; Barkan, 2016; Bamler & Mandt, 2017) . The key innovation behind word2net is that it represents words with functions, instead of vectors (Rumelhart et al., 1986) or distributions (Vilnis & McCallum, 2015) . Word2net keeps context vectors, but it replaces the embedding vector with a neural network.

Previous work has also used deep neural networks for word embeddings (Bengio et al., 2003; 2006; Mnih & Hinton, 2007) ; these methods use a single network that outputs the unnormalized log probabilities for all words in the vocabulary. Word2net takes a different strategy: it has a separate network for each vocabulary word. Unlike the previous methods, word2net's approach helps maintain the objective as a bank of binary classifiers, which allows for faster optimization of the networks.

To develop word2net, we adopt the perspective of exponential family embeddings (Rudolph et al., 2016) , which extend word embeddings to data beyond text. There are several extensions to exponential family embeddings (Rudolph & Blei, 2017; Rudolph et al., 2017; Liu & Blei, 2017) , but they all have in common an exponential family likelihood whose natural parameter has a log-bilinear form. Word2net extends this framework to allow for non-linear relationships. Here we focus on Bernoulli embeddings, which are related to word embeddings with negative sampling, but our approach easily generalizes to other exponential family distributions (e.g., Poisson).

Finally, word embeddings can capture semantic properties of the word, but they tend to neglect most of the syntactic information (Andreas & Klein, 2014) . Word2net introduces a simple way to leverage the syntactic information to improve the quality of the word representations.

W N:

In this section we develop word2net as a novel extension of Bernoulli embeddings (Rudolph et al., 2016) . Bernoulli embeddings are a conditional model of text, closely related to word2vec. Specifically, they are related to continuous bag-of-words ( ) with negative sampling. 1 We first review Bernoulli embeddings and then we present word2net as a deep Bernoulli embedding model.

. B : B:

Exponential family embeddings learn an embedding vector v 2 R K and a context vector ˛v 2 R K for each unique term in the vocabulary, v D 1; : : : ; V . These vectors encode the semantic properties of words, and they are used to parameterize the conditional probability of a word given its context. Specifically, let w n be the V -length one-hot vector indicating the word at location n, such that w nv D 1 for one term (vocabulary word) v, and let c n be the indices of the words in a fixed-sized window centered at location n (i.e., the indices of the context words). Exponential family embeddings parameterize the conditional probability of the target word given its context via a linear combination of the embedding vector and the context vectors, p.w nv j c n / D Bernoulli . > v ˙n/ ; with ˙n ,

EQUATION

Here, .x/ D 1 1Ce x is the sigmoid function, and we have introduced the notation ˙n for the sum of the context vectors at location n. Note that Eq. 1 does not impose the constraint that the sum over the vocabulary words P v p.w nv D 1 j c n / must be 1. This significantly alleviates the computational complexity (Mikolov et al., 2013b; Rudolph et al., 2016) . This type of exponential family embedding is called Bernoulli embedding, named for its conditional distribution. In Bernoulli embeddings, our goal is to learn the embedding vectors v and the context vectors ˛v from the text by maximizing the log probability of words given their contexts. The data contains N pairs .w n ; c n / of words and their contexts, and thus we can form the objective function L. ; ˛/ as the sum of log p.w nv j c n / for all instances and vocabulary words. The resulting objective can be seen as a bank of V binary classifiers, where V is the vocabulary size. To see that, we make use of Eq. 1 and express the objective L. ; ˛/ as a sum over vocabulary words,

L. ; ˛/ D N X nD1 V X vD1 log p.w nv j c n / D V X vD1 0 @ X nW w nv D1 log . > v ˙n/ C X nW w nv D0 log . > v ˙n/ 1 A : (2)

If we hold all the context vectors ˛v fixed, then Eq. 2 is the objective of V independent logistic regressors, each predicting whether a word appears in a given context or it does not. The positive examples are those where word v actually appeared in a given context; the negative examples are those where v did not appear. It is the context vectors that couple the V binary classifiers together.

In practice, we need to either downweight the contribution of the zeros in Eq. 2, or subsample the set of negative examples for each n (Rudolph et al., 2016) . We follow the latter case here, which leads to negative sampling (Mikolov et al., 2013b) . (See the connection in more detail in Appendix B.)

.

W N B:

Word2net replaces the linear classifiers in Eq. 2 with non-linear classifiers. In particular, we replace the linear combination > v ˙n with a neural network that is specific to each vocabulary word v, so that p.w nv D 1 j c n / D f .˙nI ˇv/ ;

(3) where f . I ˇv/ W R K ! R is a feed-forward neural network with parameters (i.e., weights and intercepts) ˇv. The number of neurons of the input layer is K, equal to the length of the context vectors ˛v. Essentially, we have replaced the per-term embedding vectors v with a per-term neural network ˇv. We refer to the per-term neural networks as word networks.

The word2net objective is the sum of the log conditionals,

EQUATION

where we choose the function f . I ˇv/ to be a three-layer neural network,2 

EQUATION

Replacing vectors with neural networks has several implications. First, the bank of binary classifiers has additional model capacity to capture nonlinear relationships between the context and the cooccurrence probabilities. Specifically, each layer consecutively transforms the context to a different representation until the weight matrix at the last layer can linearly separate the real occurrences of the target word from the negative examples.

Second, for a fixed dimensionality K, the resulting model has more parameters. 3 This increases the model capacity, but it also increases the risk of overfitting. Indeed, we found that without extra regularization, the neural networks may easily overfit to the training data. We regularize the networks via either weight decay or parameter sharing (see below). In the empirical study of Section 3 we show that word2net fits text data better than its shallow counterparts and that it captures semantic similarities. Even for infrequent words, the learned semantic representations are meaningful.

Third, we can exploit the hierarchical structure of the neural network representations via parameter sharing. Specifically, we can share the parameters of a specific layer of the networks of different words. This allows us to explicitly account for tags in our model (see below).

Regularization through parameter sharing enables the use of tags. One way to regularize word2net is through parameter sharing. For parameter sharing, each word is assigned to one of T groups. Importantly, different occurrences of a term may be associated to different groups.

We share specific layers of the word networks among words in the same group. In this paper, all neural network representations have 3 layers. We use index `2 f1; 2; 3g to denote the layer at which we apply the parameter sharing. Then, for each occurrence of term v in group t we set ˇ.`/ v D ˇ.`/ t . Consider now two extreme cases. First, for T D 1 group, we have a strong form of regularization by forcing all word networks to share the parameters of layer `. The number of parameters for layer `has been divided by the vocabulary size, which implies a reduction in model complexity that might help prevent overfitting. This parameter sharing structure does not require side information and hence can be applied to any text corpus. In the second extreme case, each word is in its own group and T D V . This set-up recovers the model of Eqs. 4 and 5, which does not have parameter sharing.

When we have access to a corpus annotated with tags, parameter sharing lets us use the information to improve the capability of word2net by capturing the semantic structure of the data. Andreas & Klein (2014) have shown that word embeddings do not necessarily encode much syntactic information, and it is still unclear how to use syntactic information to learn better word embeddings. The main issue is that many words can appear with different tags; for example, can be both a and refer to the animal or a and refer to the activity of catching the animal. On the one hand, both meanings are related. On the other hand, they may have differing profiles of which contexts they appear in. Ideally, embedding models should be able to capture the difference. However, the simple approach of considering / and / as separate terms fails because there are few occurrences of each individual term/tag pair. (We show that empirically in Section 3.)

Exploiting the hierarchical nature of the network representations of word2net, we incorporate information through parameter sharing as follows. Assume that for location n in the text we have a one-hot vector s n 2 f0; 1g T indicating the tag. To model the observation at position n, we use a neural network specific to that term/tag combination,

EQUATION

That is, the neural network parameters are combined to form a neural network in which layer `has parameters ˇ.`/ t and the other layers have parameters ˇ.:`/ v . Thus, we leverage the information about the tag t by replacing ˇ.`/ v with ˇ.`/ t in layer `, resulting in parameter sharing at that layer. If the same term v appears at a different position n 0 with a different tag t 0 , at location n 0 we replace the parameters ˇ.`/ v of layer `with ˇ.`/ t 0 . Figure 1b illustrates parameter sharing at `D 1.

Even though now we have a function f . / for each term/tag pair, the number of parameters does not scale with the product V T ; indeed the number of parameters of the network with information is smaller than the number of parameters of the network without side information (Eq. 5). The reason is that the number of parameters necessary to describe one of the layers has been reduced from V to T due to parameter sharing (the other layers remain unchanged). Finally, note that we have some flexibility in choosing which layer is tag-specific and which layers are word-specific. We explore different combinations in Section 3, where we show that word2net with information improves the performance of word2net. The parameter sharing approach extends to side information beyond tags, as long as the words can be divided into groups, but we focus on parameter sharing across all words (T D 1) or across tags.

Semantic similarity of word networks. In standard word embeddings, the default choice to compute semantic similarities between words is by cosine distances between the word vectors. Since word2net replaces the word vectors with word networks, we can no longer apply this default choice. We next describe the procedure that we use to compute semantic similarities between word networks.

After fitting word2net, each word is represented by a neural network. Given that these networks parameterize functions, we design a metric that accounts for the fact that two functions are similar if they map similar inputs to similar outputs. So the intuition behind our procedure is as follows: we consider a set of K-dimensional inputs, we evaluate the output of each neural network on this set of inputs, and then we compare the outputs across networks. For the inputs, we choose the V context vectors, which we stack together into a matrix ˛2 R V K . We evaluate each network f . / row-wise on ˛(i.e., feeding each ˛v as a K-dimensional input to obtain a scalar output), obtaining a V -dimensional summary of where the network f . / maps the inputs. Finally, we use the cosine distance of the outputs to compare the outputs across networks. In summary, we obtain the similarity of two words w and v as dist .w; v/ D f .˛I ˇw / > f .˛I ˇv/ 

E:

In this section we study the performance of word2net on two datasets, Wikipedia articles and Senate speeches. We show that word2net fits held-out data better than existing models and that the learned network representations capture semantic similarities. Our results also show that word2net is superior at incorporating syntactic information into the model, which improves both the predictions and the quality of the word representations.

Data. We use word2net to study two data sets, both with and without tags:

Wikipedia: The text8 corpus is a collection of Wikipedia articles, containing 17M words. We form a vocabulary with the 15K most common terms, replacing less frequent terms with the token. We annotate text8 using the tagger and the universal tagset. 4 Table 7 in Appendix C shows a description of the tagset. We also form a tagged dataset in which each term/tag combination has a unique token, resulting in a vocabulary of 49K tagged terms.

Senate speeches: These are the speeches given in the U.S. Senate in the years 1916-2009. The data is a transcript of spoken language and contains 24M words. Similarly as above, we form a vocabulary of 15K terms. We annotate the text using the Stanford CoreNLP tagger (Manning et al., 2014), and we map the tags to the universal tagset. We form a tagged dataset with 38K tagged terms.

Table 1 summarizes the information about both corpora. We split each dataset into a training, a validation, and a test set, which respectively contain 90%, 5%, and 5% of the words. Additional details on preprocessing are in Appendix C.

Methods. We compare word2net to its shallow counterpart, the model (Mikolov et al., 2013b) , which is equivalent to Bernoulli embeddings ( -)5 (Rudolph et al., 2016) . We also compare with the skip-gram model.6 (Mikolov et al., 2013b) We run -/ and skip-gram on the data and also on the augmented data of -tagged terms. In detail, the methods we compare are:

-/ : Learns vector representations for each word (or tagged word) by optimizing Eq. 2.

Skip-gram: Learns vector representations for each word (or tagged word) by optimizing Eq. 12.

Word2net: Learns a neural network representation for each word by optimizing Eq. 4. We study the following parameter sharing schemes: 1. : layer `shared between terms with the same part-of-speech ( ) tag.

For word2net, we experiment with the context dimensions K 2 f20; 100g. The context dimension is also the dimension of the input layer. For K D 20, we use H 1 D 10 hidden units in the first hidden layer of each word network and H 2 D 10 hidden units in the second layer. For K D 100, we use H 1 D H 2 D 20 hidden units. Without parameter sharing, the number of parameters per word is

K C KH 1 C H 1 H 2 C H 2 .

The shallow models have 2K parameters per term (the entries of the context and word vectors). Since we want to compare models both in terms of context dimension K and in terms of total parameters, we fit the methods with K 2 f20; 165; 100; 1260g.

We experiment with context sizes jc n j 2 f2; 4; 8g and we train all methods using stochastic gradient descent ( ) (Robbins & Monro, 1951) with jS n j D 10 negative samples on the Wikipedia data and with jS n j D 20 negative samples on the Senate speeches. We use regularization with standard deviation 10 for the word and context vectors, as well as weight decay for the neural networks. We use Adam (Kingma & Ba, 2015) with Tensorflow's default settings (Abadi et al., 2016) to train all methods for up to 30000 iterations, using a minibatch size of 4069 or 1024. We assess convergence by monitoring the loss on a held-out validation set every 50 iterations, and we stop training when the average validation loss starts increasing. We initialize and freeze the context vectors of the word2net methods with the context vectors from a pretrained Bernoulli embedding with the same context dimension K. Network parameters are initialized according to standard initialization schemes of Table 2 shows the results for the Wikipedia dataset. We explore different model sizes: with the same number of parameters as word2net, and with the same dimensionality K of the context vectors. For word2net, we explore different parameter sharing approaches. Table 5 in Appendix A shows the results for other model sizes (including K D 100). In both tables, word2net without parameter sharing performs at least as good as the shallow models. Importantly, the performance of word2net improves with parameters sharing, and it outperforms the other methods.

Tables 2 and 5 also show that -/ and skip-gram perform poorly when we incorporate information by considering an augmented vocabulary of tagged words. The reason is that each term becomes less frequent, and these approaches would require more data to capture the cooccurrence patterns of tagged words. In contrast, word2net with parameter sharing provides the best predictions across all methods (including other versions of word2net).

Finally, Table 6 in Appendix A shows the predictive performance for the U.S. Senate speeches. On this corpus, skip-gram performs better than -/ and word2net without parameter sharing; however, word2net with sharing also provides the best predictions across all methods.

Qualitative results: Word2net captures similarities and leverages syntactic information. Table 3 displays the similarity between word networks (trained on Wikipedia with parameter sharing at layer `D 1), compared to the similarities captured by word embeddings ( -/ ). For each query word, we list the three most similar terms, according to the learned representations. The word vectors are compared using cosine similarity, while the word networks are compared using Eq. 7. The table shows that word2net can capture latent semantics, even for less frequent words such as .

Table 4 shows similarities of models trained on the Senate speeches. In particular, the table compares:

-/ without information, -/ trained on the augmented vocabulary of tagged words, and word2net with parameter sharing at the input layer (`D 1). We use Eq. 8 to compute the similarity across word networks with sharing. We can see that word2net is superior at incorporating syntactic information into the learned representations. For example, the most similar tag, but we also see some variation. One such example is in Figure 1c , which shows that the list of the 10 most similar words to the verb contains the adjective .

D:

We have presented word2net, a method for learning neural network representations of words. The word networks are used to predict the occurrence of words in small context windows and improve prediction accuracy over existing log-bilinear models. We combine the context vectors additively, but this opens the door for future research directions in which we explore other ways of combining the context information, such as accounting for the order of the context words and their tags.

We have also introduced parameter sharing as a way to share statistical strength across groups of words and we have shown empirically that it improves the performance of word2net That is, for each target term w nv , the objective has one term while the skip-gram objective has jc n j terms. Consider a term .n; v/ for which w nv D 1. We take the corresponding term from Eq. 11 and we apply Jensen's inequality to obtain the corresponding skip-gram term in Eq. 12:

EQUATION

Here, we have made use of the concavity of the log . / function. In general, this is a consequence of the convexity of the log-normalizer of the (Bernoulli) exponential family distribution.

This holds for the "positive" examples w nv . As for the negative examples (w nv D 0), the comparison is not as straightforward, because the choice of terms in Eqs. 11 and 12 is not exactly the same. In particular, Eq. 11 holds v 0 fixed and draws v from the noise distribution, while Eq. 12 holds v fixed and draws v 0 from the noise distribution.

C D:

In this paper we study Wikipedia articles (text8) and a corpus of U.S. Senate speeches. On both corpora, we restrict the vocabulary to the 15K most frequent words, replacing all the remaining words with a designated token. We annotate the data using tagger9 or the Stanford CoreNLP tagger (Manning et al., 2014) , using the universal tagset shown in Table 7 .

The Senate speeches contain a lot of boilerplate repetitive language; for this reason, we tokenize around 350 frequent phrases, such as or , considering the entire phrase an individual vocabulary term. We apply the tagger before this tokenization step, and then we assign the tag to all phrases.

We split the data into training (90%), testing (5%), and validation (5%) sets. We use the validation set to assess convergence, as explained in the main text. We subsample the frequent words following where frequency.w n / denotes the frequency of word w n , and t D 10 5 .

For each method, we use jS n j D 10 negative samples on the Wikipedia articles and jS n j D 20 negative samples on the Senate speeches. Following Mikolov et al. (2013b) , we draw the negative samples from the unigram distribution raised to the power of 0:75.

:

See Appendix B for more details on the connections.

Three layers performed well in our experiments, allowing for parameter sharing to include tags.

For fairness, in Section 3 we also compare to shallow models with the same number of parameters.

See http://nltk.org.

See Appendix B for the detailed relationship betweenand with negative

sampling.6 The skip-gram objective is related to /through Jensen's inequality (see Appendix B).

If we do not average, the held-out likelihood of skip-gram becomes worse.

This is consistent with the approaches in recommender systems(Hu et al., 2008).

See http://nltk.org

Figures and Tables:

Figure 1: An illustration of deep Bernoulli embeddings. (top left) In word2net, each vocabulary word v is represented by a neural network parameterized by ˇv. The word network takes as input the sum of the context vectors ˛v0 of the words in the context and outputs the occurrence probability of the target word. (bottom left) Word2net enables sharing one the layers across all occurrences of words with a given tag. (right) Using sharing, we can find the most similar word/tag pairs to a given query. In the example, the queries are /

Figure 1: An illustration of word2net. (a) In word2net, each term v is represented by a neural network with weights ˇ.`/ v . The word network predicts the probability of a target word ( , shown in blue) from its context (green). The input is the sum of the context vectors and the output is the occurrence probability of the target. (b) Word2net can incorporate syntactic information by sharing an entire layer (orange) between words with the same tag ( in this case). (c) The fitted word2net withsharing can be queried for semantic similarities of the word networks for each word/tag pair. In this example, we list the most similar networks to /

jjf .˛I ˇw /jj 2 jjf .˛I ˇv/jj 2

Table content: <table><tr><td/><td>:</td><td/><td>(7)</td></tr><tr><td colspan="4">If we are using parameter sharing, we can also compare similar is / to / . The two combinations will have different representations under -tagged words; e.g., we may ask how the word2net method trained with -tag sharing. Assuming that layer `is the shared layer, we compute the semantic similarity between the word/tag pair OEw; t and the pair OEv; s as</td></tr><tr><td>dist.OEw; t ; OEv; s/ D</td><td>f .˛I ˇ.:`/ w ; ˇ.`/ t / &gt; f .˛I ˇ.:`/ v jjf .˛I ˇ.:`/ w ; ˇ.`/ t /jj 2 jjf .˛I ˇ.:`/ ; ˇ.`/ s / v ; ˇ.`/ s /jj 2</td><td>:</td><td>(8)</td></tr></table>

Summary of the two corpora analyzed in Section 3.

Table content: <table><tr><td>corpus size vocabulary 17M words 15K terms Senate speeches 24M words Wikipedia 15K terms</td><td>tagger NLTK CoreNLP 11 tags tags 11 tags</td><td>tagged vocabulary 49K tagged terms 38K tagged terms</td></tr></table>

The word networks fitted using word2net capture semantic similarities. We compare the top 3 similar words to several query words (shaded in gray) for /and word2net, trained on the Wikipedia dataset. The numbers in parenthesis indicate the frequency of the query words.

Table content: <table><tr><td/><td>(3000)</td><td>(1000)</td><td/><td>(500)</td><td/><td>(70)</td></tr><tr><td>expectancy per increase</td><td>word2net capacity amount energy</td><td>word2net elect candidate election candidates votes voting</td><td>bananas potatoes pepper</td><td>word2net beans juice poultry</td><td>turtle beaver pratchett</td><td>word2net dolphin crow dodo</td></tr></table>

Word2net learns better semantic representations by exploiting syntactic information. The top 3 similar words to several queries are listed for different models fitted to the Senate speeches.

Table content: <table><tr><td colspan="6">We compare parameter sharing (right). The trained without capture better semantic similarities, while adding the tags (left), tags are noted in orange. sharing helps word2net with tags (center), and word2net with information to hurts its performance.</td></tr><tr><td>(pron)</td><td/><td/><td/><td>(sc)</td></tr><tr><td colspan="2">like senator senator from alabama n governor n just used adj</td><td>word2net myself pron my pron himself pron</td><td>but reason that</td><td>unemployed n annuity n shelled v</td><td>word2net as sc that sc through sc</td></tr><tr><td>(n)</td><td/><td/><td/><td>(v)</td></tr><tr><td>fatal consequences enough adv pro adj coupled positions n</td><td colspan="2">word2net consequences n clash n handicaps n</td><td>think what just</td><td>time v best adj favour n</td><td>word2net think v know v answer v</td></tr><tr><td colspan="3">networks to the pronoun are often similar to other word networks with the same are other pronouns such as</td><td>,</td><td>, and</td><td>. Word networks</td></tr></table>

. Another opportunity for future work is to explore other types of parameter sharing besides sharing, such as sharing layers across documents or learning a latent group structure together with the word networks.

Table content: <table><tr><td>R</td></tr><tr><td>Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.</td></tr></table>

Universal tagset.

Table content: <table><tr><td>Tag</td><td>Description</td></tr><tr><td colspan="2">adj adp adv conj conjuction adjective adposition adverb det determiner, article n noun num numeral prt particle pron pronoun sc preposition or subordinating conjuction v verb x other</td></tr></table>

Mikolov et al. (2013b); i.e., each word w n in the training set is discarded with probability Prob.w n is discarded/ D 1

Table content: <table><tr><td>s</td><td>t frequency.w n /</td><td>;</td><td>(14)</td></tr></table>

Peer review (iclr2018):
 The paper extends SGNS as follows. In SGNS, each word x is associated with vectors a_x and r_x. Given a set of context words C, the model calculates the probability that the target word is x by a dot product between a_x and the average of {r_c: c in C}. The paper generalizes this computation to an arbitrary network: now each word x is associated with some network N_x whose input is a set of context words C and the output is the aforementioned probability. This is essentially an architectural change: from a bag-of-words model to a (3-layer) feedforward model. Another contribution of the paper is a new form of regularization by tying a subset of layers between different N_x. In particular, the paper considers incorporating POS tags by tying within each POS group. For instance, the parameters of the first layer are shared across all noun words. (This assumes that POS tags are given.) While this is a natural extension to word2vec, the reviewer has some reservations about the execution of this work. Word embeddings are useful in large part because they can be used to initialize the parameters of a network. None of the chosen experiments shows this. Improvement in the log likelihood over SGNS is somewhat obvious because there are more parameters. The similarity between "words" now requires a selection of context vectors (7) which is awkward/arbitrary. The use of POS tags is not very compelling (though harmless). It*s not necessary: contrary to the claim in the paper, word embeddings captures syntactic information if the context width is small and/or context information is provided. A more sensible experiment would be to actually plug in the entire pretrained word nets into an external model and see how much they help. EDIT: It*s usually the case that even if the number of parameters is the same, extra nonlinearity results in better data fitting (e.g., Berg-Kirkpatrick et al, 2010), it*s still not unexpected. All of this is closely addressed in the following prior work: Learning to Embed Words in Context for Syntactic Tasks (Tu et al., 2017) Quality: Natural but questionable extension, see above. Clarity: Clear. Originality: Acceptable, but a very similar idea of embedding contexts is presented in Tu et al. (2017) which is not cited. Significance: Minor/moderate, see above.