# Assessment of Word-Level Neural Language Models for Sentence Completion +
## 3 Methods  
### Word-Level RNN LM  
RNN based word-level LMs [33] that exploit the advantage of RNN in modeling sequential data have been widely applied in natural language processing [22]. The goal of word-level language modeling is to estimate the probability of the next word based on a previous text. Let \(w_{1},w_{2},...,w_{n}\) denote a word sequence. Then, a typical word-level RNN LM approximates the conditional probability of the \(t\)th position, \(p(w_{t}|w_{1},w_{2},...,w_{t-1})\), by encoding the context words with RNN layers and decoding the output to the probability vector. In the following, we describe in turn a word-level RNN LM used in this work (referred to as word RNN henceforth), its bidirectional version, and another variant that adopts a training strategy called masked LM [16]. The unidirectional word RNN transforms each input word to a learnable embedding vector of size \(d_{emb}\). Then, recurrent layers take the word embedding and the previous hidden states where the past text has been encoded. We choose Long Short-Term Memory (LSTM) for the recurrent cell type and set the number of recurrent layers, \(l\) to 2, following the authors of [12; 34]. A fully-connected layer reduces the dimension of the output hidden vector of the topmost recurrent layer from \(d_{hid}\) to \(d_{out}\). After the linearly projected vector is multiplied by an output embedding matrix, the softmax operation produces the likelihood of the next word, \(P_{LM}(w_{i}|w_{1},w_{2},...,w_{i-1})\). The training objective for the network is to minimize the negative log-likelihood of target words, also known as categorical cross-entropy loss. Dropout is applied between recurrent layers [34]. For the bidirectional version, we design it to infer a target word based on not only the previous words but also the subsequent words, which means the output of the network estimates \(p(w_{i}|w_{1},w_{2},...,w_{i-1},w_{i+1},...,w_{n})\). Although this formulation cannot be applied for the language modeling in which a model is required to generate one word at a time from scratch, it fits in the sentence completion, which determines the missing word based on both sides. Some probabilistic interpretation of bidirectional modeling for sequences was suggested by [PERSON] et al. [35], who showed the effectiveness of bidirectional reconstruction on inferring omitted symbolic tokens such as characters in text and notes in midi-encoded music. It is noteworthy that our bidirectional model fuses forward and backward hidden layers before the softmax operation rather than just ensembles the output distributions of forward and backward LMs as in [14; 27]. To be specific, our bidirectional word RNN (Figure 1) has separate word embedding lookup tables for forward and backward directions, similar to context2 vec [9]. Each word embedding is connected to either forward or backward directional LSTM layers. The network then aggregates the output of the bidirectional hidden layers into a final output vector. The aggregation consists of conducting a separate linear projection for each hidden layer vector and then adding the two projected vectors. This way of aggregation was slightly better than concatenation followed by a single linear projection in our preliminary experiments. The main difference between our bidirectional word RNN and context2 vec is that our model computes the negative likelihood loss rather than an approximation of pointwise mutual information between the target word and the context during training. Lastly, we present a variant based on masked LM training, which is a crucial strategy to pre-train bidirectional Transformers for language understanding [16]. We apply this to our bidirectional word RNN to see if the training strategy has a synergy with RNN-based architecture in place of the transformer-based one. Instead of aggregating the one-step-ahead hidden state from the forward layer and the one-step-behind hidden state from the backward layer, the masked LM version aggregates the current step hidden states from both directions.