A Natural Bias for Language Generation Models
Year: 2023

Abstract:
After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. Yet around this point, these models have identified a simple, loss-minimising behaviour: to output the unigram distribution of the target training corpus. The use of such a heuristic raises the question: Can we initialise our models with this behaviour and save precious compute resources and model capacity? Here we show that we can effectively endow standard neural language generation models with a separate module that reflects unigram frequency statistics as prior knowledge, simply by initialising the bias term in a model’s final linear layer with the log-unigram distribution. We use neural machine translation as a test bed for this simple technique and observe that it: (i) improves learning efficiency; (ii) achieves better overall performance; and perhaps most importantly (iii) appears to disentangle strong frequency effects by encouraging the model to specialise in non-frequency-related aspects of language.

heading: 1 Introduction
text: Consider the structure of a number of core tasks in natural language processing (NLP): predicting the next word following a given context. What if you did not understand the context – for example, if you did not know the language? In the absence of such knowledge, the optimal prediction would be the language’s most frequent word. In fact, optimally one would predict each word according to its (unigram) frequency.1 This is precisely
∗Work done during internship at DeepMind. 1Notably, void of contextual clues, models of human language processing (Morton, 1969) would default to similar strategies. A word’s frequency also influences its age of acquisition (Gilhooly and Logie, 1980; Morrison et al., 1997), and the time taken to produce it in speech (Gerhand and Barry, 1998; Zevin and Seidenberg, 2002).
the strategy that neural language models have been empirically observed to employ during early training stages (Chang and Bergen, 2022) – before they have learnt a language’s syntax or semantics.
Although this strategy of predicting the unigram distribution emerges early in training, it still takes the model hundreds (or even thousands) of parameter updates to learn it from a cold start (see Fig. 1 or Chang and Bergen, 2022, Fig. 5). Yet a straightforward factorisation of a language model’s final linear layer shows that we can in fact encode this frequency-related knowledge prior to any optimisation,2 with the goal of bypassing this early stage of learning: Concretely, this is done by setting the bias term in a model’s final linear layer to the log-unigram distribution of the training data. Mathematically, this setup can be loosely interpreted as a modular “product of experts” (Hinton, 2002), where the bias term represents a simple unconditional distribution over the vocabulary, thus allowing the input-dependent logits to specialise in capturing contextual information. Indeed, we argue that a more modular design that disentangles word-frequency effects from contextual information may be desirable, given the recently-observed negative effects of word frequency statistics on models’ generalisation abilities (Wei et al., 2021;
2The unigram distribution of the training data is known before optimisation, as it is often computed when building vocabularies or tokenising; hence this approach should come at no extra cost.
243
Puccetti et al., 2022; Rajaee and Pilehvar, 2022). While this initialisation approach has been historically used in language models (Mnih and Hinton, 2007; Botha and Blunsom, 2014; Fang et al., 2015, inter alia), it has not seen widespread adoption within our current language generation architectures – an observation we attribute to uncertainty around whether the bias term automatically specialises to capture frequency without explicit encouragement to do so. We first observe that this is not the case – in fact, the final-layer bias term rarely changes from its random initialisation (see App. A.6), suggesting frequency is encoded elsewhere in the model parameters. We then empirically explore the impact of this initialisation on various aspects of model behaviour – within the context of current Transformer models for machine translation – including overall performance, learning efficiency, and the relationship between model-assigned probability and word frequency. We find this initialisation indeed leads to increased training efficiency: models achieve higher BLEU scores earlier on in training. More surprisingly, it also leads to improved overall performance. We discuss several potential reasons for these results, including changes to training dynamics and a mitigation of overfitting to surface statistics.

heading: 2 Probabilistic Language Generators

heading: 2.1 Preliminaries
text: We consider neural probabilistic models pθ for language generation. While there are a variety of architectural choices that can be made, most are autoregressive and follow a localnormalisation scheme. Explicitly, given prior context y<t def = ⟨y0, . . . , yt−1⟩, these models output a probability distribution pθ(· | y<t) over the next token y ∈ V def= V ∪ {EOS}, where V is the model’s predefined vocabulary and EOS is a special end-of-sequence token. To ensure that pθ provides a valid probability distribution, the output of the model is projected onto the probability simplex ∆|V|−1 using a softmax transformation after a (learnt) linear projection layer:3
pθ(y |y<t) = softmax (Wϕ(y<t) + b)y (1) def = eWy ϕ(y<t)+by ∑
y′∈V e Wy′ ϕ(y<t)+by′
(2)
3While ϕ is also conditioned on a source sentence x in the case of machine translation, we leave this implicit in our equations for notational simplicity.
where W ∈ R|V|×d denotes a weight matrix, b ∈ R|V| a bias vector, and ϕ : V∗ → Rd the model’s d-dimensional encoding for a given context.4
A number of prior studies have investigated whether – and if so, at what stage during the learning process – NLP models learn various linguistic phenomena (Alain and Bengio, 2017; Adi et al., 2017, inter alia). Among the key findings are that language models reflect the statistical tendencies exhibited by their respective training corpora (Takahashi and Tanaka-Ishii, 2017, 2019; Meister and Cotterell, 2021); some of which are learnt early on in training (Liu et al., 2021). For example, Chang and Bergen (2022) observe that, after only ∼ 1000 training updates, language models’ outputs are approximately equal to the unigram distribution, regardless of the context that they condition on. We similarly observe this for machine translation models (see Fig. 1).

heading: 2.2 A Natural Bias
text: These learning trends motivate trying to supply language generation models with a natural starting point: the unigram distribution. Fortunately, this form of prior knowledge can be modularly encoded in standard neural models using the bias term of the final, pre-softmax linear layer. Consider the standard operation for projecting the output of the model onto the probability simplex. Upon closer inspection, we see that eq. (2) has an interpretation as the product of two probability distributions, up to a normalisation constant:
pθ(· |y<t) ∝ eWϕ(y<t) · eb (3) ∝ pWϕ(· |y<t) · pb(·) (4)
i.e., one described by pWϕ(· |y<t) – which is contextual as it depends on the input y<t – and a separate, non-contextual term denoted by pb(·). Thus, we can qualitatively view this setup as factorising the model’s prediction into these two components.5 In this light, it makes intuitive sense that pb should be the unigram distribution – a distribution which optimally predicts (w.r.t. negative loglikelihood loss) the next-token when there is no contextual information to condition on. Note that such a setup – where a probability distribution is
4We index vectors and matrices using y, assuming an isomorphic mapping between y ∈ V and integers [1, . . . , |V|].
5Given this decomposition, one might expect that models learn to use the bias term to encode frequency on their own. Yet we do not find this to be the case empirically (App. A.6).
modelled using a product of several simpler distributions, each of which can specialise on modelling one aspect of the problem – is referred to as a product of experts (Hinton, 2002).6

heading: 3 Related Work
text: As previously mentioned, prior work has likewise taken advantage of the interpretation of the bias term as a frequency offset when initialising model parameters (Mnih and Hinton, 2007; Botha and Blunsom, 2014; Fang et al., 2015, inter alia). Yet such techniques have fallen to the wayside for a number of years now, as other more prominent determinants of model performance and training efficiency have dominated the community’s attention. We revisit this initialisation strategy in the context of today’s neural language models.
The practice of directly incorporating unigram probabilities into next-word predictions can be likened to the back-off methods proposed in the n-gram literature (Kneser and Ney, 1995; Chen and Goodman, 1999).7 Indeed, there is an entire class of methods built around learning deviations from some base reference distribution, some of which have been employed specifically for language modelling (Berger and Printz, 1998; Teh, 2006; Grave et al., 2017). More recently, Li et al. (2022) cast neural language modelling as the learning of the residuals not captured by n-gram models and Baziotis et al. (2020) use language models as a reference distribution for training low resource machine translation models.
Another class of prior work has similarly explored efficient strategies for model weight initialisation (Glorot and Bengio, 2010; Le et al., 2015; Mu et al., 2018, inter alia), including random variable choices and re-initialisation criterion. In a similar vein, Ben Zaken et al. (2022) investigate the usefulness of the bias term, albeit for efficient
6The comparison of mixtures and products of experts is well summarised by the phrase: a single expert in a mixture has the power to pass a bill while a single expert in a product has the power to veto it. Each paradigm has its advantages. Here, we argue that the latter is more suitable for language modelling, as the mixture formulation presents the issue that high-frequency tokens will be strongly “up-voted” by the expert corresponding to the unigram distribution. As these models already have a propensity to select high frequency tokens, even in improper contexts (Wei et al., 2021), this is arguably an undesirable trait.
7How to properly estimate the unigram distribution itself is an important, but often overlooked, question. In our work, we consider a predefined and finite vocabulary V; and estimate probabilities using their frequency in a training corpus. For a longer discussion on this see Nikkarinen et al. (2021).
fine-tuning techniques. They show that, often, modifying solely the bias parameters during finetuning provides comparable performance to updating the entire model. Both our results thus showcase the usefulness of this simple set of parameters for natural language processing tasks.
Other works have also embraced frameworks akin to product or mixture of experts in language modelling or generation tasks. For example Neubig and Dyer (2016) combine neural and countbased language models in a mixture of experts paradigm; Artetxe et al. (2022) take advantage of the mixture of experts structure to propose a compute-efficient language modelling architecture. In contrast, we suggest a simple initialisation method that does not require training additional models or major changes to model architectures.

heading: 4 Experiments
text: We explore the effects of the unigram bias initialisation strategy on neural machine translation systems in comparison to a standard initialisation technique: initialising the bias to all 0s (denoted as → 0 ) or omitting the bias term entirely.

heading: 4.1 Setup
text: We perform experiments with several language pairs: WMT’14 German-to-English (De→En; Bojar et al., 2014), IWSLT’14 German-toEnglish (De↔En; Cettolo et al., 2012), and Afrikaans/Rundi-to-English in the AfroMT dataset (Run↔En and Af↔En; Reid et al., 2021). These corpora span several language families and different sizes to demonstrate performance in higher, medium and lower resource domains (∼ 4.5M, ∼ 750K and ∼ 150K sentence pairs, respectively). All models use the standard Transformer encoder–decoder architecture (Vaswani
et al., 2017), with 6 layers in both. The IWSLT and AfroMT Run↔En models have 4 attention heads per layer (adjusted for the smaller size of these datasets) while all other models have 8 attention heads. Dropout is set to 0.1; the feedforward hidden dimension is set to 512 for the WMT model and 256 for all other models. Parameter estimation is performed using stochastic gradientdescent techniques, with the standard maximum likelihood objective and label smoothing (Szegedy et al., 2015) with hyperparameter α = 0.1. We use the Adam optimizer (Kingma and Ba, 2015) with (β1, β2) = (0.9, 0.997). Early stopping was performed during training, i.e., model parameters were taken from the checkpoint with the best validation set BLEU (Papineni et al., 2002).
We preprocess the data using subword tokenisation with the SentencePiece library (Kudo and Richardson, 2018)8 For initialisation, unigram frequencies are computed on respective training sets after tokenisation is performed. We do not hold bias term parameters fixed during training, although we found that they do not change perceptibly from their values at initialisation, even for the → 0 -initialised model (App. A.6). The projection matrix W in the final linear layer is initialised element-wise using N (0, 1/√d), where d is the embedding hidden dimension;the matrix is then scaled such that the matrix ℓ2 norm is approximately equal in magnitude to the bias ℓ2 norm. Decoding is done with length-normalised beam search with a beam size of 5, which was similarly chosen based on validation BLEU scores. All BLEU and chrF (Popović, 2015) scores are computed using the sacreBLEU library (Post, 2018).

heading: 4.2 Results
text: We present main results here, and defer additional experimental results that exhibit similar trends (e.g., using chrF as the evaluation metric, or training on WMT) to App. A. We also explore several extensions that build on the unigram initialisation, albeit with mixed results; again, see App. A.
Performance. Fig. 2 presents mean test BLEU scores with standard error estimates from 5 different random seeds per dataset–intitialisation strategy combination. On 5 out of the 6 datasets, the
8For WMT and IWSLT, we train joint SentencePiece models with vocabulary sizes 32000 and 20480, respectively. For AfroMT, we use the SentencePiece model provided with the dataset: https://github.com/machelreid/afromt.
unigram bias initialisation technique leads to comparable or better test set performance in comparison to standard bias term initialisation techniques.
Efficiency. In order to quantify training efficiency, we estimate9 the area under the validation BLEU learning curve (ALC) (Guyon et al., 2011; Liu et al., 2020) for the first 20k training updates;10 for the sake of interpretability, scores are renormalised by the interval span. From Fig. 3, we see that, on 5 out of the 6 datasets, higher BLEU is achieved earlier on in training. Hence, the unigram bias initialisation approach appears to reach better performance in fewer iterations than standard initialisation approaches, which would be beneficial in cases where training efficiency considerations are paramount (e.g., in low-resource languages or in compute-limited settings).
Analysis. The aim of this analysis is to investigate whether – and to what extent – the finallayer bias unigram initialisation leaves the contextual part of the network, pWϕ(· |y<t), to better capture non-frequency effects. To this end, we examine model-assigned log-probability as a function of token frequency. In Fig. 4, we plot a token’s unigram log-frequency against the average logprobability assigned to it (when it is the groundtruth token) by a model initialised with (left) a bias term of → 0and (right) a log-unigram bias term, binning them in equal-length intervals and averaging them for clarity. In Fig. 4a, the full model parameters are used. In Fig. 4b, the bias terms are not
9Explicitly, we use the composite trapezoidal rule. 10Models converged after ∼ 50k updates. We look only at the first 20k updates to focus on early training behaviours. Full training behaviours were similar (see App. A.1).
added in the linear projection, i.e., only the contextual part of eq. (4), pWϕ(· |y<t), is computed.
The upward trend in average model-assigned log-probability in Fig. 4a suggests that, in general, models are better (or at least more confident) when predicting more frequent tokens. This trend holds when the bias term is omitted from the final linear computation of the → 0 -initialised model. Interestingly though, when the bias term is omitted from the unigram-initialised model, the trend appears to reverse. This change suggests that for unigraminitialised models, frequency may instead be encoded in the bias term, providing evidence that for these models, pWϕ(· | y<t) may indeed specialise in non-frequency aspects of language.

heading: 5 Discussion
text: NLP models have been observed to overfit to surface cues in their training data, impeding their ability to generalise at inference time (Warstadt et al., 2020; Wei et al., 2021). Thus, one could argue that learning or encoding the superficial statistical tendencies of language is not necessarily a good thing. Yet, empirical results suggest that it may in fact be an important part of model learning dynamics (see App. A.4, for example). Indeed, Takahashi and Tanaka-Ishii (2019) find evidence that more powerful language models have a natural bias for learning them. Here we ask if – when
initialising model parameters – we can explicitly endow our models with prior knowledge about one such statistical tendency: frequency.
While the result that this initialisation strategy improves training efficiency is perhaps not surprising, the relatively consistent improvement in overall performance is. We offer two possible explanations for this improvement. The first is that this initialisation beneficially alters model learning dynamics at the beginning of training, especially as early learning dynamics can have an outsized impact on final model performance (Achille et al., 2019). A second possible explanation is that it disentangles frequency in the modelling of contextual probabilities. If pb (eq. (4)) explicitly models the unigram distribution, then our model does not need to capture this component of the conditional distribution in its other parameters, which frees up model capacity to focus on more complex phenomena within natural language. Its success thus motivates exploring the use of higher-order statistical models, such as a bigram or trigram model, in an attempt to further disentangle surfaces statistics from more nuanced components of natural language in a modular fashion.

heading: 6 Conclusion and Future Work
text: In this work, we revisit a simple initialisation technique in the context of modern neural language generation models: setting the bias term in the final linear projection layer to the log-unigram distribution of (sub)words within the training corpus. This strategy leads to more efficient training; perhaps more surprisingly, it also leads to better overall performance in our machine translation experiments. We offer analysis and discussion as to the cause of these trends. An interesting direction for future work could be determining the effects that this initialisation procedure has on various model properties, e.g., its embedding space, and its benefits specifically in low-resource settings. Furthermore, extensions of this work could explore potential uses of this strategy in the mitigation of problems with lexically infrequent words, e.g., by analysing via the decomposition in eq. (4) whether a model’s probability estimate for a word is being driven by frequency or contextual components. Finally, this technique is not limited to models of distributions over strings; it is in fact applicable to any neural classification setting, the exploration of which is left to future work.

heading: 7 Acknowledgements
text: We would like to thank the members of the DeepMind Language Team for insightful discussions during the course of this work and specifically, Chris Dyer and Kris Cao for helpful feedback on the initial version of this paper and John Hale for pointers to references on human language acquisition. We would also like to thank Clément Guerner for detailed feedback on clarity and presentation.

heading: 8 Limitations
text: Perhaps the main limitation of this work is that we only explore the approach within the context of machine translation benchmarks, although we conduct extensive experiments within this task that cover different training data scales and diverse pairs of languages, including low-resource ones. Nevertheless, we remark that the proposed approach is entirely general-purpose, and can be applied to any other language generation or even any neural classification tasks. We leave it to future work to investigate whether the same gains would apply in those settings. Furthermore, we have not yet explored how this technique would interact with other modelling choices, such as different optimizers, training objectives, or subword tokenisation algorithms. Lastly, our unigram initialisation of the bias term is currently done at the level of subword units, which do not always correspond to lexically or morphologically meaningful linguistic units. We leave the extension of this approach to more meaningful linguistic units, such as words or morphemes, to future work.

heading: 9 Ethical Considerations
text: We foresee no ethical issues that could arise from the findings presented in this work.

heading: A Additional Experiments
text: A.1 Additional Training Trends
A.2 WMT Experiments
A.3 chrF Scores
A.4 Regularising Away from the Unigram Distribution
Prior work has suggested that models’ learning of surface statistics, such as the unigram distribution, may harm their generalisation abilities (Warstadt et al., 2020; Wei et al., 2021). Under this premise, it seems feasible that the learning trends observed in Fig. 1 could have downstream negative side-effects, e.g., the inappropriate preference for higher frequency words observed in (Wei et al., 2021). Given the importance of early stage training dynamics (Achille et al., 2019), it may even be the root cause of such behaviour. In the effort to test this hypothesis, we try to regularise a model’s output away from the unigram distribution in early stages of training. Specifically, we instead minimise the objective KL(p || pθ)−λKL(ω(p) || pθ) for empirical distribution p and the unigram distribution of this empirical distribution ω(p). λ is a hyperparameter. We use this objective for the initial steps of training, then switching back to the standard objective KL(p || pθ). In Fig. 11, we observe that this form of regularisation leads to worse (or equivalently performing) models by the time of convergence. Results were similar when evaluated on out-of-distribution data.
A.5 Out-of-Domain Performance
A.6 Change in Bias Term over Training
In Figs. 13 and 14, we see the divergence of the bias term from the unigram distribution and the magnitude of the bias term, respectively. Interestingly, we see that neither value changes perceptibly from the time of initialisation onward, suggesting the bias term itself does not change much from its initialised value. This trend is consistent across seeds and datasets.
A.7 Initialisation with Bias Term from Large-Scale Dataset
We additionally explore the effects of initialising the bias term with the log-unigram distribution, as estimated from a larger dataset in a more general purpose domain. We hypothesise that this strategy could be useful in low resource settings. We find that this indeed improves the generalisation performance of a model trained on IWSLT when evaluated on an OOD dataset (see Fig. 16).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
7 A2. Did you discuss any potential risks of your work? We do not foresee any potential risks

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: abstract, section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Model parameters are provided in appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave it to future work to investigate whether the same gains would apply in those settings. Furthermore, we have not yet explored how this technique would interact with other modelling choices, such as different optimizers, training objectives, or subword tokenisation algorithms. Lastly, our unigram initialisation of the bias term is currently done at the level of subword units, which do not always correspond to lexically or morphologically meaningful linguistic units. We leave the extension of this approach to more meaningful linguistic units, such as words or morphemes, to future work