# Assessment of Word-Level Neural Language Models for Sentence Completion +
## 3 Methods
### LM-Based Scoring  
This subsection describes three scoring strategies with respect to how to apply a trained word RNN to sentence completion. Suppose the \(t\)th position is blank, \(w_{1},w_{2},...,w_{t-1},w_{t+1},...,w_{n}\) are context words, and \(c^{1},c^{2},...,c^{m}\) are candidate choices for the blank. Blank scoring strategy selects the choice word that maximizes the conditional likelihood, \(P_{LM}(c|w_{1},w_{2},...,w_{t-1})\) for a typical unidirectional LM or \(P_{biLM}(c|w_{1},w_{2},...,w_{t-1},w_{t+1},...,w_{n})\) for our bidirectional model, where \(c\in\{c^{1},c^{2},...,c^{m}\}\). This strategy is equivalent to picking the choice that minimizes the cross-entropy loss on the blank. In the case that a choice text spans more than a word, the strategy computes the score by summing the loss over the blank span and multiplying by \(-1\).  
In the unidirectional case, however, the blank scoring strategy does not reflect any contextual information from future words following the blank. To compensate for this defect, we adopt the full scoring strategy that minimizes the cross-entropy loss on the entire sentence instead of the blank. Let \(w_{1}^{j},w_{2}^{j}...,w_{n_{j}}^{j}\) be the word sequence obtained by replacing the blank with a choice word \(c^{j}\), or phrase composed of multiple words \(c_{1}^{j},c_{2}^{j},...,c_{r_{j}}^{j}\) where \(n_{j}=n-1+r_{j}\). Then, the strategy computes the score as \(\sum_{i=2}^{n_{j}}\log P_{LM}(w_{i}^{j}|w_{1}^{j},w_{2}^{j},...,w_{i-1}^{j})\) for the unidirectional LM or \(\sum_{i=2}^{n_{j}-1}\log P_{bi}(w_{i}^{j}|w_{1}^{j},w_{2}^{j},...,w_{i-1}^{j},w_{i+1}^{j},...,w_{n_{j}}^{j})\) for the bidirectional one, where \(j=1,2,...,m\). Even in the bidirectional case, we observed that the full scoring strategy performed better than the former, although the blank scoring strategy also considers the future context.  
The third strategy, referred to as partial scoring strategy [36], computes the score as the likelihood of the subsequent text conditioned on the choice word and its preceding text. For a unidirectional LM, the score of \(c\) is defined as \(\sum_{i=t+1}^{n}\log P_{LM}(w_{i}|w_{1},...,w_{t-1},c,...,w_{i-1})\), where the \(t\)th position is blank. Since \(P_{LM}(w_{i}|w_{1},w_{2},...,w_{i-1})\) for \(i=1,...,t-1\) does not depend on \(c\), the strategy is equivalent to picking the choice that minimizes \(\sum_{i\
eq t}\log P_{LM}(w_{i}|w_{1},...,w_{t-1},c,...,w_{i-1})\), which equals the full score minus the blank score. From this formulation, the partial score of \(c\) for the bidirectional model can be naturally derived as \(\sum_{i<t}\log P_{biLM}(w_{i}|w_{1},...,w_{i-1},w_{i+1},...,w_{t-1},c,...,w_{n})\) +  
\(\sum_{i>t}\log P_{biLM}(w_{i}|w_{1},...,w_{t-1},c,...,w_{i-1},w_{i+1},...,w_{n})\). While [PERSON] and [PERSON] [36] reported that the partial scoring strategy outperformed the full scoring strategy for pronoun disambiguation problems, we observed that it did not hold for the sentence completion.  
For networks based on masked LM training, we can apply the blank scoring strategy by defining the score of choice \(c\) as \(P_{mask}(c|w_{1},w_{2},...,w_{t-1},\) [MASK], \(w_{t+1},...,w_{n})\), where [MASK] is inserted in the  
Figure 1: Bidirectional word RNN (modified from Figure 1 in [11]).  
blank. If a choice text consists of multiple words, the corresponding number of [MASK] tokens are inserted. Since it is not straightforward to apply either the partial or full scoring strategy to masked LM based networks, we did not take them into account.