Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Authors: Nathan Godey
Éric Villemonte de la Clergerie
Benoît Sagot
Venue: ICLR 2024 poster (2024)
Keywords: representation learning
NLP
language modeling
pretraining
contrastive
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Our models get better performance by learning to recover masked input embeddings instead of predicting masked input tokens.

Abstract: Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative method that shifts away from probability prediction and instead focuses on reconstructing input embeddings in a contrastive fashion via Constrastive Weight Tying (CWT). We apply this approach to pretrain Headless Language Models in both monolingual and multilingual contexts. Our method offers practical advantages, substantially reducing training computational requirements by up to 20 times, while simultaneously enhancing downstream performance and data efficiency. We observe a significant +1.6 GLUE score increase and a notable +2.7 LAMBADA accuracy improvement compared to classical LMs within similar compute budgets.

Published as a conference paper at ICLR 2024
HEADLESS LANGUAGE MODELS: LEARNING WITHOUT PREDICTING WITH CONTRASTIVE WEIGHT TYING
Nathan Godey!.3 Éric de la Clergerie
Benoît Sagot'
Inria, Paris, France
Sorbonne Université, Paris, France
boothan.godey. eric. de la_ allaaclleeeeeeeeerrrrrrrr deengeriee.beoitteo. sagot fr
ABSTRACT
Self-supervised pre- treiinaning of language models usually consists in predicting prob- ability distributions over extensive token vocabularies. In this study, we propose an innovative method that shifts away from probability prediction and instead focuses on reconstructing input embeddings in contrastive fashion via Constrastive Weight Tying (CWT). We apply this approach to pretrain Headless Language Models in both monolingual and multilingual contexts. Our method offers practical advan- tages, substantially reducing training computational requirements by up to 20 times, while simultaneously enhancing downstream performance and data efficiency. We observe significant +1.6 GLUE score increase and notable +2.7 LAMBADA accuracy improvement compared to classical LMs within similar compute budgets.
INTRODUCTION
Natural Language Processing (NLP) has seen tremendous progress in recent years thanks to the development of large-scale neural language models. These models have been shown to be effective in wide range of NLP tasks such as text classification, question answering, and machine translation, either fine-tuning, few-shot and zero-shot settings. These approaches usually involve a self-supervised pre-training step. based tasks requiring predictions of contextual probability distributions over large vocabulary of tokens.
However, the need for language modeling projection head can be limitation it requires additional memory, slows down training and impedes scaling up to large token vocabularies. In this paper, we propose novel pretraining approach called Headless Language Modeling, which removes the need to predict probability distributions and rather focuses on leveraging contrastive learning to reconstruct sequences of input embeddings. Instead of adding a projection head towards high-dimensional vocabulary space in order to make a prediction about given token, we teach those models to contrastively output static embeddings corresponding to this token. The static embeddings we use for this are the model's own input embeddings. Due to its resemblance with the well-established weight- tying trick (Press Wolf, 2017 He et al., 2023). we call this pre-training technique Contrastive Weight Tying (CWT).
Controstive Weight Tying
Batch tokens
the
cat
cat
smasko
Madel
the
for
Figure I: Masked Headless Language Modeling (HLM) using Contrastive Weight Tying. The CWT objective aims contrastively predict masked input representations using in-batch negative examples.

---

Published as a conference paper at ICLR 2024
We find that our approach outperforms usual language modeling counterparts in several aspects and by substantial margins. First, it drastically speeds up training by freeing up GPU memory and avoiding the costly language modeling projection, thus allowing up 2 acceleration of the training throughput, and up to 20x less compute requirements to achieve similar performance. Moreover, given the same amount of training tokens, headless language models (HLMs) significantly outperform their classical counterparts on downstream tasks, as shown by 2.7 gain in LAMBADA accuracy for our headless generative model. Finally, given similar compute budgets, HLMS bring substantial gains for NLU tasks, with our BERT reproduction scoring 1.6 points above its classical counterpart on the GLUE benchmark. We also show that headless models can benefit from larger token vocabularies at much more reasonable cost than classical models.
In terms of implementatior(] our approach can be used as drop-in replacement in usual pretraining codebases, as it only requires a change in the loss computation that can be applied to any kind of language model.
Overall, we make several contributions in this article:
We introduce a pretraining objective that replaces cross-entropy, thus removing the need to project on the vocabulary high- tigh-dimensional space and instead learning to contrastively predict latent representations of tokens;
Using this technique, we pretrain encoder and decoder models for English, and a multilingual encoder model;
We show the various benefits of headless training in terms of data-efficiency, compute- efficiency, and performance;
We explore the effects of micro-batch size and vocabulary size on downstream performance, and provide an ablation study of our contrastive objective.
RELATED WORK
Efficient pre-training
With the dawn of pretrained language models, such as BERT DDvlin
et al. 2019). RoBERTa (Liu al. 2019). GPT-2 (Radford et al. 2019) or T5 (Raffel et al. improving training efficiency has become an important stake in NLP. Subsequent works have focused on changing the training objectives to improve performance. ELECTRA (Clark et al. 2020b) uses Replaced Token Detection as the unsupervised training task, and substantially improves data- efficiency, compute- -ompute-efficieeeyy and downstream performance. Their work has also been extended using energy-based models (Clark et al. 2020a) or disentangled weight sharing (He al. 2020).
2020).
Contrastive approaches in NLP The idea of relieving language models of the need to predict probabilities over the whole token vocabulary has been explored in the importance sampling literature (Bengio Senecal 2003) Mnih & Teh [2022] JJaa et al. 2015] Ma Collins 2018). These methods approximate the denominator of the softmax by using only subset of the possible tokens. Those approaches usually rely on variants of the CostrsCont Estimation objective (Gutmann & Hyvärinen 2010) that use unique negative samples, contrary our approach that samples represen- tations uniformly from the batch. Kumar & Tsvetkov (2019) and Tokarchuk & Niculae (2022) use contrastive objectives based on cosine-similarity to match pre-trained static embeddings for Machine Translation. We instead use the model's input embeddings as trainable target representations.
Contrastive self-supervised learning The Contrastive Predictive Coding loss (van den Oord et al. 2019) initiated the use of pretraining approaches based on contrastive learning objective, idea that has obtained success in many modalities over the years (Sermanet et al.) 20188) Schneider et al.
2019 Baevski et al. 2020 Algayres al.] 2022). In NLP, contrastive learning has proven efficient in the training of sentence-lever models (Gao et al. 2021 Yan et al.] 2021 Klein & Nabi 2023). Token- level approaches rely on contrastive auxiliary objectives that are added to the usual cross-entropy
loss. SimCTG (Su et al. 2022a) introduces a token-level contrastive objective using in-batch output representations as negative samples, and adds this objective to a sentence-level contrastive loss and a
regular causal LM loss. TaCL (Su et al. 2022b) relies on a similar technique for encoder models,
'Our pretraining and fine-tuning code is published in hnttps://gt thub. http.c////Mettaaaccceeeeeeeeeels
2

---

Published as a conference paper at ICLR 2024
where teacher model is used produce negative samples. ContraCLM Jain et al. 2023) uses an auxiliary contrastive loss for code generation.
Tokenization and frequency The importance of tokenization for language models has been discussed by several works (Rust et al. 2021: Zouhar et al. 2023). As discussed in Zouhar et al. (2023). tokenization choices impact token probability distributions both contextual and general scales. It has been shown that skewed token distributions can impact the quality of representations
(Gao et al. 20199| Zhou et al., 202T; Puccetti et al., [022] Yu al. (022). Removing the language modeling head could mitigate these issues. In the case of multilingual models, [Lingg et al. (2023)
have shown that increasing the vocabulary size leads better performance, at the cost of added time
and memory complexity.
3 METHOD
3.1 CLASSICAL FRAMEWORK
We consider a batch X (Ir,)))ee of token sequences length L. We also produce slightly altered version of these sequences X (EL..)) optionally using masking or random replacement for instance, some pretraining objectives require. We introduce an embedding matrix Eg R XD where V is the token vocabulary size and D the hidden dimension, and seuuuence-tossqqueee model To NxLD - RR Nx both based on set of parameters 0 R.P.
classical language modeling approach consists selecting a subset of tokens Xs = (Z).))..Jes and then estimating probability distribution over the token vocabulary for these tokens from the (Zi.) sequences, using eg and To. Learning occurs as X is partially altered (āig) (e.g. in Masked Language Modeling) or internally in To (e.g. decoder models), and contextual information is essential for and Te to accurately estimate the tokens in Xs.
trick that has been used in many such approaches relies on using eg's transpose (e a projection from the output space of To to TR This approach, called weight tying, can be written for given sequence at index E [1, N] as:
Pr. softmar (eT (To(e)(E.)))))
where Pi,j is the estimated distribution for the j-th word of the sequence. Weight tying has been shown to improve performance while reducing the number of parameters (Clark et al. 2020b). Cross-entropy loss is then used as an objective function:
((0,X,X) =
Σ Lx,, log(fr.)
[S]
iJES
3.2 HEADLESS MODELING
While weight tying does not additional parameters, the projection og actually has non negllggle computational cost, which increases as the token vocabulary grows. Like Gao et al. (2019a), we advocate that the weight tying approach tends maximize the scalar product between the input embedding of the original token eo(zi.) and the output representation at the same position of (e)e)()))) under the contrastive regularization of the softmax function.
Based this understanding, we design an objective that directly optimizes this scalar product while not requiring the computation of the e/ projection. As we do not use this projection, we cannot rely on softmax regularization anymore, and instead introduce a contrastive loss using the in-batch samples from S as negatives. All in all, our contrastive loss can be written as:
C, =
1
Σ iJES
S
Extes Eeeeuua
We call this objective Contrastive Weight Tying (CWT), as weight sharing is not used per se but is set contrastive objective. Across the paper, do not combine this loss function with the classical
3

---

Published as a conference paper at ICLR 2024
cross-entropy objective in Su et al. (2022a), and rather use it as the only pretraining objective. To the best of our knowledge, this work stands as the first attempt to pretrain language models in a self-supervised fashion using explicit contrastive loss as the sole objective.
3.3 THE CASE OF DECODERS: CAUSAL FINE-TUNING
We can easily adapt the Causal Language Modeling (CLM) objective using the Contrastive Weight Tying approach. Negative samples correspond every input embedding at different position in the batch. However, the resulting model is not directly able to generate text, as it has no projection head towards R. A way to retrieve language generation capacities is to use the input embedding matrix transpose eg as projection head (Kumar & T'vetkov 2019: Tokarchuk & Niculae 2022). Nevertheless, we observe that this approach yields poor performance (see Table 3). Instead, we fine-tune the headless model and language modeling head initialized with co using the predictive
CLM objective on a small portion (<2%) the pre-training dataset. This method allows recovering an effective language model.
3.4 THEORETICAL CONSIDERATIONS
In terms of time and memory complexity, Headless Language Models (HLMs) are more efficient than classical language models under usual conditions. If we focus the computation of the loss a single device from || = K output representations, a neural probabilistic LM requires D(KDV) operations while our headless approach performs O(K² D) operation Hence, when K V. which is very common for micro-bate bato- sizes that fit on one device, our CWT T loss is more computationally efficient than cross-entropy. With regard to memory requirements, our CWT loss also more efficient than its classical counterpart. On the one hand, the cross- loss with weight tying stores the outputs of the eÿ projection dimension x V the forward pass. On the other hand, our CWT loss stores the scalar product matrix of dimension K x N, which is again smaller when K < V.
headless
Traniing speed (maitoloe)
80
varilla
verilla
20
memory used
50-
60
a0
40
30
20
20
10
(a) Training latency
(b) Memory use
Figure 2: Comparison of time and memory complexities of BERT-baar model on single RTX 8000 GPU.
In Figure 2, we provide preliminary empirical analysis of the speed and memory improvements when training BERT-base model using original hyperparameters, i.e. sequences of 512 tokens and 15% masking. We use HuggingFace's implementation for the Transformers blocks, and run experiments on a single RTX 8000 GPU. We observe that training latency is significantly reduced by roughly 25% for all batch sizes, and that the engine can handle larger batch size due to the improvement memory consumption.
EXPERIMENTS
We use the Contrastive Weight Tying objective for rediuum-scak pre-training experiments in different contexts. We focus on monolingual encoder and decoder architectures, but we also train one
One could extend our CWT loss by picking separate set Sy of negative samples. This allows to tune the number of negative samples, which is important in Contrastive Learning. However, for the sake of simplicity, and avoid extensive hyperparameter tuning, Sx S.
4

---

Published as a conference paper at ICLR 2024
MLM type
Tokens
GPU hours
MRPC
COLA
STS-B
SST2
QNLI
QQP
MNLI
Avg.
Vanilla
85.87
54.66
83.7
92.45
82.4
82.43
(#10.12)
Headless
4.1
444
85.31
58.35
84,54
93.23
89.49
89.62
82.54
83,29
(=0:15)
Headless
8.2
888
86.89
60.72
85.98
92.56
89.75
89.81
82.87
84.08 (2:0.14)
Table 1: Results of Masked Language Models (MLMs) on the dev sets of the GLUE benchmark. Best results bold and second best are underlined. We report Matthews' correlation for COLA, Spearman correlation for STS-B, B, and accuracy elsewhere. MNLI validation datasets are concatenated. All scores averaged over different seeds.
MLM type
BoolQ
COPA
WiC
Avg.
Vanilla
60.2
64.9
67.9
(2:0.4
Headless
69.8
74.7
62.7
67.2
68.6 (::00))
Table 2: Results of Masked Language Models (MLMs) on the dev sets of datasets from the Super- GLUE benchmark. We report accuracy for all tasks. Scores are averaged over 10 fine-tuning runs.
multilingual encoder as we believe the uniformity brought by our contrastive objective may improve cross-lingual alignment. We compare our HLMs with classical language models that we pretrain on the same data with roughly similar compute budgets.
4.1 HEADLESS MONOLINGUAL ENCODER
We pretrain BERT-base architectures (10MM parameters) for English on the Open dataset extracted from The Pile (Gao et al., 2020). We use the tokenizer from the Pythia suite (Biderman ctt al., 2023), which was trained on The Pile and uses a 50k tokens vocabulary. We mostly use hyperparameters from BERT (Devlin et al. 2019), although RoBERTa (Liu et al. 2019). For the sake of simplicity, we use sequence length of 128 for the whole training. We give detailed overview of the hyperparameters in Appendix D.I
remove the NSP objective in
We pretrain all models using A GPUs, with budget of roughly 1,000 hours each. To optimize training, we use memory-efficiee self-attention as implemented in xFormers (Lefaudeux al. 2022) for all experiments. For the vanilla MLM, we set micro- batch size of 32 for each A100 GPU, then accumulate to the original 256 batch size at optimization level, and train on million batches. For our headless approach, we observed that we could remain within compute budget when using micro-batch size 64, Hence, we use an effective batch size of 512 for the headless MLM (HMLM). Although the HMLM uses more pretraining sequences, it does not gain additional information compared to the vanilla MLM as both models perform several epochs on the Open WebTex12 dataset.
We evaluate on the GLUE benchmark, where we exclude the RTE dataset due to high standard deviations in the obtained scores. We fine-tune our models for 10 epochs on every dataset, and compute validation metries
every fine-tuning epoch. We use the AdamW optimizer with learning rate of 10-5, a weight decay of 0.01 and balanced cross-entropy loss objective. See Appendix E for more details.
In Table I we compare our headless MLM with the classical MLM on the GLUE benchmark. To ensure fair comparison, we display evaluations at similar amounts of tokens seen during pre-training, and at similar training durations on the same hardware. In both cases, the headless MLM outperforms the vanilla MLM by significant margins, showing that our CWT loss is both more data-efficient and compute-efi -fficcieeeeii in this setup. We extend this analysis at various intervals along pretraining, and plot results in Figure It shows that the headless MLM outperforms the downstream performance of its vanilla counterpart after using 25% of its training compute. We notice that the performance gap is near constant across pretraining steps.
4.2 HEADLESS MONOLINGUAL DECODER
We pretrain Pythia-70x 70M architectures for English, sticking to the Pythia procedure (Bierrman et al. 2023) much possible. We OpenneeTexxt2 pretraining dataset. We train on 143,000
5

---

Published as a conference paper at ICLR 2024
headless varilla
84
headless vanilla
83
QLUR averege
GLUE average
68
81
79
79
1000
Pretraining
Pretraining
(a) Pretraining hours
(b) Pretraining tokens
Figure 3: Comparison of GLUE average scores along pretraining.
heedleess (w! fine-turing)
20
LAMBADA peplosity
800
haadless Iw/o fine tuning)
600
LAMDADA
16-
400
headless (w/ fine tuning)
headless (w/o tine turing)
200
50
100
150
200
250
300
100
150
200
Pretraining tokens (billions)
Pretraining tokens (billions)
(a) Accuracy
(b) Perplexity
Figure 4: Comparison of LAMBADA metrics along pretraining. We display results for vanilla causal language modeling and headless models before and after causal LM fine-tuning. The pretraining token count for the fine-tuned HLM takes fine-tuning tokens into account.
batches of 1,024 sequences of length 2,048 split over 16 V100 GPUs. We use exactly the same hyperparameters as in the Pythia suite. The micro-batch size is set to 32 in both cases.
fine-tune our headless models for CLM with an LM head initialized with cg for 10000 steps using effective batch size of 256 (4x smaller that during pretraining), learning rate 10 and constant learning rate schedule with 2000 linear warm-up steps. All other hyperparameters are kept similar to pretraining. We evaluate our models on the LAMBADA dataset and report accuracy and perplexity for zero-shot generation Figure
As mentioned in Subsection 3.3
We find that the HLM fine-tuned for predictive language modeling outperforms the vanilla model by a significant margin along training. We report language generation results in Table 3 We observe that despite having higher validation perplexity even after fine-tuning, the HLM is improving the zero-shot perplexity on the LAMBADA dataset.
We also study the zero-shot performance of the causal models datasets taken from the LM Evaluation Harness. At this model scale, many tasks are not relevant and thus discarded, as the results do not always significantly outperform random baseline. We also discarded tasks where the
Validation
LAMBADA
LM type
Ppl.
Ppl.
Azc.
Vanilla
170.23
Headless
524.44
18.26
Headless +
3,283
153.5
22.2
Table 3: Results of the causal language models on the validation set after training, and on the LAMBADA dataset.

---

Published as a conference paper at ICLR 2024
LM type
GPU hours
BoalQ
ChowS-Paine
RACE
SaQ
PubMedQA
QASPER
Vanilla
(4:09)
(4-3
23.7 (4:13)
(6.)))
(4-86)
41.9
HLM
1052 (618)
53.0 (20.9)
56.0 (.2))
(::14)
64.5
47,5' (216))
66.0 (1:3.)
Table 4: Zero-shot evaluation of monolingual causal language models on datasets from the LM Evaluation Harness. We report the stereotype percentage for CrowS-Paie and accuracy elsewhere. 1 best scores that are significantly better than the second best score according to a one-tailed t-test with power 0.95.
sample size was below 1000 or where comparison was not meaningful due to low performance gaps compared to the variance level. Hence, only subset of the tasks shown in Table
we find that the fine-tuned HLM outperforms the vanilla causal model by significant margins on BoolQ (Clark et al., 2019), PubMedQA (Jin et al.) 2019) and QASPER (Dasigi al. 2021). Although we observe less statistically significant gaps for the other datasets, we still note that our HLM performs at least comparably to the vanilla baseline. We also note that the HLM seems slightly less prone to stereotypes measured by the CrowS-Pairs benchmark (Nangia et al. 2020).
In Table 4
Overall, using the Contrastive Weight Tying loss the context of causal LM allows obtaining models on par with vanilla counterparts at lower compute cost. We notice that the resulting models get surprisingly good results in challenging datasets, hence showing language understanding capabilities, while being outclassed in language generation benchmarks (before predictive fine-tuning). We believe that this study shows that language generation needs to considered as a downstream task for HLMs, as they are designed to generate representations instead of words.
MULTILINGUAL ENCODER
In this section, we pretrain small multilingual MLMs and evaluate their performance on the XNLI dataset (Conneau et al. 2018). ). Due to compute limitations, consider architectures similar to the distilled multilingual BERT trained by |anh et al. (2019). This model has 137M parameters, and uses vocabulary of 119 tokens. As in [Subsection 4.1 train vanilla MLM and headless counterpart. However, we share training hyperparameters such as batch size and total number of steps between both models, without compute considerations. For both experiments, we pretrain models on 400k batches of 64 sequences of 128 tokens taken from the multilingual Wikipedia dataset using a single RTX8000 GPU. We select 90 million entries from 10 languages (Arabic, German, English, Spanish, French, Hindi, Italian, Japanese, Korean, and Chinese). Training hyperparameters can be found in Appendix D.3
Models are then fine-tuned on the XNLI dataset, for both cross-lingual zero-shot transfer from English and target language fine-tuning. Fine-tuning hyperparameters can be found in Appendix E.4
MLM type
hi
zh
Avg.
Fine-tuned on Ennglish only
Vanilla Hendless
46.83
56.71
71.66
59.93
58.34
43.16
50.99
55.37 (1:0.11)
57.32
74.03
62.72
45.25
52.15
(1:0.)
Fine- tuned on target language
Vanilla Headless
$1.32
64.09
70.4
66.98
65 68.88
55.95
64.63
62.87 (1:0.)
54.25
66.95
73.96
69.14
67.22
60.04
67.22
65.54 (1:0.22)
Table 5: Evaluation of multilingual models on the XNLI benchmark. averaged over 3 runs.
We report dev accuracy,
We display final results in Figure 5
We find that the headless approach leads to significantly better performance for every language in both cross-lingual transfer and language-speciff fine-tuning. In average, the headless MLM outperforms its vanilla counterpart by accuracy points in the cross-lingual scenario, and by 2.7 points in the language-specific fine-tuning experiments.
Availlale btw.:
http:///gggiggaaaccc.ociittleeea base- tiateeiuaalllllnllllaaaa -cased

---

Published as a conference paper at ICLR 2024
headless vanilla
56
vanilla
XNL sccuracy
XNLI accuracy
54
52
50
48
25
50
75
100
125
150
175
25
50
75
100
125
150
175
Pretraining hours
Pretraining hours
(a) Translate-Train. target language fine-tuning
(b) Translate-Test English fine-tumi fine-tuning
Figure 5: Comparison of XNLI average scores along pretraining for different setups. Models are iie--tuned/evlluaee in Arabie, German, English, Spanish, French, Hindi and Chinese.
we evaluate the models at intermediate pretraining checkpoints and plot the XNLI average score as function of used GPU hours. We observe that our HLM finishes training within 45% of the time required by the vanilla mode, and that its performance level outperforms the fully trained vanilla model after only using 5% as much compute in Figure 5a and 22%
In Figure 5
Figure 5b
DISCUSSION
Token vocabulary Training language models without output vocabulary projection makes using large vocabularies more affordable in terms of compute. As a matter of fact, the time complexity of HLMs during training is theoretically constant as we increase the vocabulary size. With input embedding lookup tables that do not require fully loading the еạ weights, the memory complexity can also be kept constant with respect to the size of the vocabulary. This property could be useful for multilingual models relying considerable vocabulary sizes, such as XLM-V (Liang et al. 2023).
To verify this hypothesis, we pretrain models for different vocabulary sizes using the BERT-Small architecture from Ture al. (2019) and the CC-News dataset (Hamborg al. 2017). Hyperparameter details can be found in inlppeenixx D.4] For each vocabulary size, we train a BPE tokenizer similar to the BERT tokenizer, and pretrain vanilla MLM and headless MLM. We then compare average GLUE results, excluding RTE, MRPC and COLA, due to high variance at that model scale.
headless vanills
175
headlees venilla
83.0
speeed (mnstaken)
150
82.5
125
GIUE
02.0
25
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Vocabulary 9t2 (k)
Vocabulary size (k)
(a) GLUE average score
(b) Training speed
Figure 6: Comparison of downstream performance and training speed for small models trained using
different token vocabulary sizes.
Figure showss that HLMs can actually benefit from larger token vocabularies up to a certain extent, and that they outperform their vanilla counterparts for every vocabulary size. Figure 6b demonstrates that increasing the vocabulary size comes at almost no decrease training speed for the HLMss contrary to vanilla MLMs. However, observe sudden throughput increase between 85k and 100k tokens vocabularies for both vanilla and headless models, which we attribute to a different handling of GPU memory and operations the models get bigger.
8

---

Published as a conference paper at ICLR 2024
Batch size As discussed in [Susscctionn 3.4, the micro-batch size used to compute the CWT loss is important as it impacts the training complexity by increasing the number of negative samples. Recent work on Contrastive Learning shows that there usually exists an optimal number of negative samples in terms of model performance (Awasthi et al. 2022 Ash et al. 2022). As a consequence, increasing the batch size when using CWT may not always be beneficial.
To study the impact of batch size on downstream performance, we pretrain small decoder models using different batch sizes. Our models inspired from the smallest architecture of GPT2 (Radford et al. 2019) where many hyperparameters are divided by 4. More details about the pretraining procedure of HLMs are fine-tuned similarly to [Susecction 4.2)
these models can be found Appendix D.5
17.0
16.5
LAMIADA
vanill 64)
14.5
headless (bs=844
headless (bsm 128)
14.0
40
60
100
training hours
Figure 7: LAMBADA accuracy along pretraining for different batch sizes.
In Figure 7 we observe that increasing batch size leads better performance for our HL.Ms. While smaller batch sizes train even faster, the headless model with the greatest batch size (128) is the only one that able to significantly outperform its vanilla counterpart at the end training.
we conduct ablation study by training small models using the hyperparameters described in Appendix D.4 different objectives. We observe that adding Cross- Entropy to CWT leads to slightly worse performance, at the cost of reduced throughput. We also notice that using a contrastive objective without using input embeddings as targets decreases performance, despite adding parameters during training. This shows the relevance of weight tying approach.
Ablation study In Table 6
Objective
Parameters
Throughpu 1
GLUE avg.
Cross Fintrp Cross-Entrops CWT
x1
xl
82.45
xl
82.93
NCB (wo/ WT)
x1.57
X2.47
82.91
CWT
x1
x2.13
83.37
Table 6: Ablation study using variants of the CWT objective. In CWT + Cross-Entropy, we add the objectives without specific weights. In NCE (wo/ WT), adapt CWT objective with additional static embedding matrix instead of the model's input embeddings, which resembles Ma & Collins (2018).
CONCLUSION
In this paper, present new pretraining approach called headless language modeling. that removes the need to predict probability distributions over token vocabulary spaces and instead focuses on learning to reconstruct representations in contrastive fashion. Our method only relies on changing the objective function, allowing for straightforward adaptations of classical language modeling pretraining objectives.
Using our contrastive objective, we pretrain headless monolingual and multilingual encoders, and headless monolingual decoder. We demonstrate that headless pretraining is significantly more compute-efficient data-efficient, and performant than classical predictive methods.
A major advantage of approach is that it enables the use of very large token vocabularies at virtually no increased cost. We believe that this paper paves the way for the exploration of contrastive techniques replacement of cross-entropy based pretraining objectives for NL..

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank our colleagues Arij Riabi and Roman Castagné for their advice and for the helpful discussions. We are grateful to Robin Algayres for his enlightening question "But what is the difference with softmax?", in the hope that this paper is a satisfying answer.
This work funded by the last author's chair in the PRAIRIE institute, itself funded by the French national agency ANR as part of the "Investissements d' d'avenir" programme under the reference ANR-19-P31A-000
This work was granted access to the HPC resources of IDRIS under the allocation 2023- AD011013680RI made by GENCI.
REFERENCES
Robin Algayres, Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, and Emmanuel Dupoux. DP-parse: Finding word boundaries from raw speech with an instance lexicon. Transactions of the Association for Computational Linguistics, 10: 1051-1065, 2022. doi: 1 (tte:ttalla_0OS000 a0005
wrgp//o00....//22222/2 1
61
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. In Gustau Camps-Vall Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 7187-7209. PMLR, 28-30 Mar 2022. URL https://prooeeddnggs mlr, pwtpss/v11/ass222//2 html
Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1101-1116. PMLR, 17-23 Jul 2022. URL http:: h/tp://ed///ccoecnngg//e .ww.innsssmm.....mmm www...pees//vl2222 awasthi225 html
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for supervsue learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12449 1 12460. Curran Associates, Inc., 2020. URL https::/prooeeddngs..eeee paper_ fit.es//ppper/20/////////////// http:///ee........aaa22222266000
Yoshua Bengio and Jean-Sébastien Senecal. Quick training of probabilistic neural nets by im- portance sampling. In Christopher M. Bishop and Brendan J. Frey (eds.), Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, volume R4 of Pro- ceedings of Machine Learning Research, pp. 17-24. PMLR, 03-06 Jan 2003. URL httpps //proceedings..
httpa:ss......rssseem htw../prrees////ennnioo.. html Reissued by PMLR on 01 April 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingnistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/ N19-1300. URLlhttps://accantholooo wrw.y/orggg///// 1300
Toutanova. BoolQ:
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D. Manning. Pre-training trans- formers as masey-based cloze models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 285-294, Online, November 2020a.
Association for Computational Linguistics. doi: 10. 10.116633v112022.emmm main..0.. URL
https://acl.antooogg..rrr. org/2//0 www.::emmlpmmm.....m.....m mainmaii..0
10

---

Published as a conference paper at ICLR 2024
Kevin Clark, Minh-Thang Luong. Quoe V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020b. URL htttttps::/opereeeiiiieee
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seelle seekiing questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4599,4610, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 maacl-main.36 URL htttps://cclanthoolgg. org/2221
maacl-maing 365
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp- 2019, Association for Computational Linguistics. doi: 118653/v1/N191111 URL https //aclantholoog.orr///W
Minneapolis, Minnesota, June
Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database. Language, Speech, and Communication. MIT Press, Cambridge, MA, 1998. ISBN 978-0-262-06197-1.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a. URL http:ttttt///eeeeee//ee http:///ppneeeeeeeeee//oom/mmmm/ het/Foormmidd-SEEojjRRRe
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Tie-Yan Liu. Representation degeneration problem in training natural language generation models. 2019b. doi: 10.48550ARXXI.1907..20 URL //axxi wrg./bs//ggo/oooooooooooo ab9/ 12009
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002 2020.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE:
Simple contrastive learning of sentence embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011 http://0022mmmmmmmm555522223 URL https //aclanthology. grg/ 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. httl/https://prceeeiinn mlr wwtppsereessueieeeeeeee 00. html
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. Proceedings of the 15th International Symposium of Information Science, pp. 218-223, March 2017. doi: 10.528 1 ht//////oooo000100 4120316.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced BERT with disentangled attention. CoRR, abs/2006.03654, 2020. URL httt::::/aaxiiv.or////////000 03654
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2023.
11

---

Published as a conference paper at ICLR 2024
Nihal Jain, Dejiao Zhang, Wasi Uddin Ahmad, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray. Parminder Bhatia, Xiaofei Ma, and Bing Xiang. ContraCLM: Contrastive learning for causal language model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 6436-6459, Toronto, Canada, July 2023. Association for Computational Linguistics. URL htttts:://allaatllllooo org/ 2023 acl 1ong.lon...... 355
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1 1-10, Beijing, China, July 2015, Association for Computational Linguistics. doi: 10.3115/1/P15-10001 URL https waw.laahhooogy.rr/Pgo
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 2567-2577, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: N0.18633/1/DD19-122 URL https f/aclanthology.om wrwplloggyor//11---// 1259.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the As- sociation for Computational Linguistics, 8:64-77, 2020. doi: 1111622taclaa000300 URL
https://aclanthologg www.hhooogg..oggg/...o org/ tacl-1.5
Tassilo Klein and Moin Nabi. miCSE:
Mutual information contrastive learning for low-shot sentence embeddings. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 0179-6177, Toronto, Canada, July 2023. Association for Computational Linguisties.
hwwp://awwtooooooooo//ooo2// org///233.33332/ acl- long.
Sachin Kumar and Yulia Tsvetkov. Von mises-fisher loss for training sequence sequence models with continuous outputs. In Proc. of ICLR, 2019. URL htt.:ttps///arii.oo/////////222
pdf
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https:://iithb... com/ facebookresearchh wwwppr/s//s////////o/o//oooos//s 2022.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models. arXiv e-prints, art, arXiv:2301.104777 January 2023. doi: 10.48550/
arXiv.2301.1042 10472.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zeutlemoeer and Madian Khabsa. XIm-v: Overcoming the vocabulary bottleneck in multilingual masked language models, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11699 1692, 2019.
11692
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3698-3707, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18. 1405. URL https 7/aclanthology.orgg org/ 14055
Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models, 2012.
12

---

Published as a conference paper at ICLR 2024
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953-1967, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/11/2020
emnlp-main. emnlp-main. 154. http:tts::/// JURL/htttp:///.lanntoooooo.orrr oatpp/a.......o....mmmmmmmaaaaaa 154
Ofir Press and Lior Wolf. Using the oulput embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 2, Short Papers, pp. 157-163, Valencia, Spain, April 2017. Association for Computational Linguisties. URL htpLttss////alaatooooog wrt.aayyrrg//17//2/...
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Outlier dimensions that disrupt transformers are driven by frequency. In Findings of the Association for Computa- fional Linguistics: EMNLP 2022, pp. 1286-1304, Abu Dhabi. United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. URL /tLLttts////calattoooo wttp:ttthwoooooooo//....o00 org/2022 ffidinng-eemnlp.. 93
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://jmlr. http://mmmmorrrr///// wwtggppper//22/2200744444
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 31 18-3135, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 lcng-long. acl-long 243
243. URL httt:p::/aalantthooog.oooo
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC2 Workshop, 2019.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised Pre- Tee-Trainng for Speech Recognition. In Proc. Interspeech 2019, pp. 3465-3469 2019. doi: 10.2 00221447hnnerpeehh2219--91
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video, 2018.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. contrastive framework for neural text generation, 2022a.
Yixuan Su, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Collier. TaCL: Improving BERT pre-training with token-aware contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 2497-2501 Seattle, United States, July 2022b. Association for Computational Linguistics. doi: 1/211/5////222...fml httts:///aclanttoolog..oo org.o///oooor////2222///// http://2222...fnnnngee findings-naacl.11
naacl.191. URL
Evgeniia Tokarchuk and Vlad Niculae. On target representation in continuous-output neural machine translation. In Proceedings of the 7th Workshop on Representation Learning for NLP, pp. 227-235, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 rep14nlp-1.24, URL httphhtsss//caaaahhlllooo htt.aa/rhhooooooooo//orrrrerrrrr
Tulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint wwX.v::908.0866200 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
13

---

Published as a conference paper at ICLR 2024
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A contrastive framework for self self-supervised sentence representation transfer. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume Long Papers), pp. 5065-5075, Online, August 2021. Association for Computational Linguistics. doi: (0.18653/v1/2022 acll-Long (ta/22211cccll111 acl-long......l .1o....n.3 393
393. URL
Sangwon Yu, Jongyoon Song. Heeseung Kim, Seongmin Lee, Woo-Jong Ryu, and Sungroh Yoon. Rare tokens degenerate all tokens: Improving neural text generation adaptive gradient gating for rare token embeddings. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 29 49,45, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 20.18653/1//022 20215//1/2.......ooo....... URL https
//aclanthology.orr/2222 2ctp:/2222...l.ogg...3
Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. Frequency-basee distortions contextualized word embeddings, 2021.
Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. Tok- enization and the noiseless channel. Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. $184-5207, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. N0.1863/11/2022 acc-002....alllmoo acl-long.284. URL
http:://eclantthoooo..rr////2 attp:/2222...lloooggg 284
A MODELING CONSIDERATIONS
From a linguistic point of view, we hypothesize that an important difference between our approach and classical predictive modeling is the fact that headless modeling mostly pushes for discrimination between co-occurring tokens, instead of imposing a contextual hierarchy over the whole vocabulary. For instance, in the case of synonyms and B, each occurrence of A (or B) is pushing the input representations of A and B apart for predictive modeling, due to weight tying. For headless modeling, an occurrence of A will only push the representations apart B appears in the same batch. Hence, the CWT objective could let models identify A and B as synonyms more casily. This argument is already mentioned in Jeann et al. (2015).
To provide empirical evidence of this behavior, we study the representation similarity for pairs of synonyms for classical and headless models. We use WordNet (Fellbaum) 1998) to extract synonym pairs and we then compute the cosine-similarity between the input embeddings corresponding to the two synonyms. Resulting cosine-similarity distributions are displayed Figure
3.5
headless vanilla
3.5
headless+FT
3.0
3.0
vanilla
headless
2.5
2.5
Density
2.0
Density
2.0
1.5
1.5
0.5
0.5
0.0
0.0
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Cosine similarity
Cosine-similadn similaritty
(a) Monolingual encoders
(b) Monolingual decoders
Figure 8:
Cosine-similanne simiia-immllrity distributions for pairs of WordNet synonyms.
In Figure 8, we observe that HLMs tend to generally represent synonyms more similar way than vanilla LMs, cosine-similarity distributions slightly drift towards higher values. In average,
14

---

Published as a conference paper at ICLR 2024
cosine-similarte between synonyms is 1.4 points higher for the encoder and roughly 7 points higher for both the original HLM decoder and its fine-tuned version.
However, we do not observe radical difference between HLMs and classical LMs in this analysis of the input representations. A more thorough analysis of the latent spaces of both types models could be relevant. For instance, comparing contextual representations of similar words across examples could help clarify this matter. We leave such analyses for future work.
Another advantage of pushing discrimination between co-occurring tokens only may be an improved feedback quality, as we expect distinguishing between co-occurring tokens to be more linguistically relevant than distinguishing between all tokens.
Finally, we believe that our method avoids the issue of cross-entropy regarding rare and unused tokens. Gao al. (2019b) prove that cross-entropy pushes the representations of rare and unused tokens in shared direction, thus distorting the resulting embedding space. The CWT objective only updates these embeddings when they appear in the negative samples, which should result more meaningful representations.
B LIMITATIONS
One key limitation of this paper is the scale of the used architectures. In recent months, the dawn of Large Language Models using billions of parameters reshaped the language modeling paradigm. The research process that led this paper is empirical and required extensive experimentation that could not be done at large scale in our academic compute budget. We believe that the results presented in this paper are still sufficiently promising to be communicated and useful to the community. We leave the scaling of these techniques to future work.
could be opposed this paper that as architectures grow in size, the proportion of compute that is associated with the output vocabulary projection shrinks. While we acknowledge that this effect may reduce the advantage of HL.Ms terms of training throughput, our experiments show that HLMss are more performant for given number of pretraining steps.
We chose not to compare with other efficient encoder architectures such as ELECTRA or DeBERTa in this paper. We also chose not to apply our method to encoder-decoder architectures, or subtle masking methods such as SpanBERT (Joshi et al. 2020). As a matter of fact, we argue that our work could be combined to these methods, and thus believe that comparison is not relevant these works are orthogonal to ours. We leave the intersection these approaches for future work.
Finally, decided to pick English for all monolingual experiments. Different behaviors could be ob- served for other languages, although our multilingual experiments gave no sign such discrepancies.
C ETHICS STATEMENT
To the best of our knowledge, this paper does not raise any specific ethical concern that is not already inherent to the open-data pre-training paradigm. Our results on the CrowS-Pairs dataset indicate that headless language modeling may mitigate some of the biases that are measured in this task. Due to considerations that are discussed in Zhou et al. (202I), and for reasons evoked in Secciion 6 believe that alternatives to cross-entropy as objective for language modeling could mitigate some of the biases that are observed in LLMs, and hope that our work can pave the way for such alternatives.
15

---

Published as a conference paper at ICLR 2024
D PRETRAINING HYPERPARAMETERS
D.1 MONOLINGUAL ENCODERS
Dataset
OpenWebText
Architecture
bert-base-uncased
Tokenizer
pythia-70m 70m deduped
Optimizer
AdamW
Learning rate
le-4
Precision
16
Weight decay
0.01
Gradient clipping
Device batch size
32/ /4
Batch size
256 / 512
Sequence length
128
LRR schedule
Triangular
Warmup steps
10000
Nb. steps
1000000
Table 7: Pre-training hyperparameterss used for the monolingual encoders. When they differ between vanilla and headless models, we provide separate values formatted (vanilla / headless). Model names written as model-name refer to their HuggingFace release.
D.2 MONOLINGUAL DECODERS
Dataset
TebTe22
Architecture
7
Tokenizer
pythia-70m
Optimizer
Adam W
Adam
Ic-8
Adam (3i, (2))
(0.9, 0.95)
Learning rate
Ic-3
Precision
16
Weight decay
0.1
Gradient clipping
Device batch size
8/8
Batch size
1024 1024
Sequence length
2048
LRR schedule
Cosine
Warmup steps
1430
Nb. steps
143000
Table 8: Pre-training hyperparameters used for the monolingual encoders. When they differ between
vanilla and headless models, we provide separate values formatted as (vanilla / headless).
16

---

Published as a conference paper at ICLR 2024
D.3 MULTILINGUAL ENCODERS
Dataset
Wikipedia (multilingual)
Architecture
Tokenizer
bistibbet-beese
Optimizer
Adam W
Learning rate
2c-4
Precision
16
Weight decay
0.01
Gradient clipping
Device batch size
Batch size
64
Sequence length
128
LR schedule
Triangular
Warmup steps
10000
Nb. steps
400000
Table 9: Pre-training hyperparameters used for the multilingual encoders.
D.4 SMALL MONOLINGUAL ENCODERS
Dataset
CC-News
Architecture
---4-12_AA888
Tokenizer
google/bert http///brttttuutceeeeeee L-
Optimizer
Adam W
Learning rate
2c-4
Precision
16
Weight decay
0.01
Gradient clipping
Device batch size
64
Batch size
64
Sequence length
128
I.R schedule
Triangular
Warmup steps
10000
Nb. steps
400000
Table 10: Pre-training hyperparameters used for the small monolingual encoders used in Figure 6
17

---

Published as a conference paper at ICLR 2024
D.5 SMALL MONOLINGUAL DECODERS
Dataset
CC-News
Architecture
gpt2
Hidden size
192
Number heads
Number layers
Tokenizer
gpt2
Optimizer
Adam
carning rate
2.5e-4
Precision
16
Weight decay
0.01
Gradient clipping
Sequence length
128
LR schedule
Cosine
Warmup steps
2000
Nb. steps
1000000
Table 11: Pre-training hyperparameters used for the small monolingual decoders used in Figure These models rely on the GPT-2 architecture with few changes. These changes scale down the model size to 11M parameters.
E FINETUNING HYPERPARAMETERS
E. BALANCED CROSS-ENTROPY
We have noticed that using balanced cross-entropy loss for fine-tuning could further improve the performance all our monolingual encoders, and increase the gap between headless models and their vanilla counterparts. We also noticed empirically that it helped stabilize results for smaller datasets such as MRPC and COLA.
Let's consider classification problem where the class distribution is described by frequencies (wo)cE(LCC). We can group the cross entropy loss Loe as such:
-x) Σ CX
where
C.(X,) = Σ yuze Loe (ai, W:)
Using this notation, the balanced cross-entropy loss can be defined as:
L
Lice (X,Y)
Wr
In practice, we approximate the (we) using the batch labels. The purpose of the balanced cross-entropy loss is mitigate general and in-batch class imbalance.
We reproduce fine-tuning experiments with the more usual categorical cross-entropy loss only, and using moderately optimized hyperparameters for this loss (see Table 12).
18

---

Published as a conference paper at ICLR 2024
Optimizer
AdamW
Learning rate
5e-6
Weight decay
0.01
Batch size
32
LR schedule
Constant
Lincar warm-up
10%
Epochs
10
Table 12: Fine-tuning hyperparameters for monolingual encoder models trained with regular cross- entropy on the GLUE benchmark.
MLM type
MRPC
COLA
STS-B
SST2
QNLI
QQP
MNLI
Avg.
Vanilla 86.27
49.33
82,06
92.37
88.62
89.49
82,35
81.5 (2:014)
Headless
85.8
56
84.85
93.23
89.67
89.77
83.05
83.19 (#0.09)
Table 13: Results of Masked Language Models (MLMs) on the dev sets of the GLUE benchmark for the regular cross-entropy loss. Results are averaged over 3 runs.
E.2 MONOLINGUAL ENCODERS
Optimizer
AdamW
Learning rate
le-5
Cross-entropy
Balanced
Weight decay
Batch size
32
LR schedule
Constant
Linear warm-up
Hpachs 10%
10
Table 14: Fine-tuning hyperparameters for monolingual encoder models trained with balanced cross- entropy on the GLUE benchmark.
E.3 MONOLINGUAL DECODERS
Dataset
OpenWeeTex12
Optimizer
AdamW
Learning rate
Ic-5
Cross-entropy
Regular
Weight decay
0
Batch size
256
LR schedule
Constant
Lincar warm-up
2000
Nb. steps
10000
Table 15: Fine-tunm tone-tuning hyperparameters for the headless monolingual decoder model using the causal language modeling objective.
19

---

Published as a conference paper at ICLR 2024
E.4 MULTILINGUAL ENCODERS
Optimizer
AdamW
Learning rate
2c-5
Cross-entropy
Regular
Weight decay
0
Batch size
128
LR schedule
Constant
Linear warm-up
10%
Table 16: Fine-tuning hyperparameters for the multilingual encoder models Translate- Train and Translate-Test scenarios.
F REPRESENTING SYNONYMS
In this section,
G IMPLEMENTATION
def cwt_
cwt_losst loss( input_embs, target_eniln _entte )
# input_embs: :mbs: nb_embs x hidden_ Lim
# target_enmat embs: nb embs
andden_din
exp_cosine _ns_ssneesiiiiiieeoeeeooiiliolee sm = torch. exp( torch. mm input_embs, aarge__embs T
= exp cos ine_sne_s_o_ im. iil.diaann diagonal()
neg_dist exp_cosine sim. sum(11)
return (self_distt(neg__die + 1e-9)) log( ) .eaan )
Figure 9: PyTorch implementation of the Contrastive Weight Tying loss.
The figures generated using the Carbon tool Intpp:://caroon.no..hh/
20

---

Published as a conference paper at ICLR 2024
def compute ( lm model, input_ input_batche batch):
input_batch: batch: batch_ size x senglength (LongTensor)
labels input_batchl.... 1:
Get model output tm_result = lm [__modde((
gest_hidden_state = _m_ result. Wwddden_states____ :-1]
# Get input embedd ings emb .mpppnnngggg...... lm model. get_ inpuiiipttttt ingss
target_ wwwpt_nnpeeeettteeeeeeee ings emb_ mappingg labels)
# Compute CWT loss
batch_ loss cwt wor_loos((
emb emb_prediction flatten (0, 1),
target_
flatten(O, 0, 1)
return batch_loss
Figure 10: PyTorch implementation of the computation of the training loss for headless causal LMs. The implementation the MLM equivalent is straightforward.
21


Reviews:
[{"review_id": "X9ytLn66RI", "reviewer": "Reviewer_A6os", "summary": "This study introduces an innovative approach to self-supervised pre-training of language models. Instead of predicting probability distributions over token vocabularies, the method focuses on reconstructing input embeddings contrastively using Contrastive Weight Tying (CWT). This approach is applied to pretrain Headless Language Models in both monolingual and multilingual contexts, offering practical advantages. It is significantly more compute-efficient, data-efficient, and performant than classical predictive methods. The paper's contributions include the introduction of a new pretraining objective, the pretraining of encoder and decoder models for English and a multilingual encoder model, demonstrating the benefits of headless training, and exploring the effects of pretraining hyperparameters on downstream performance.", "questions": "1. I'm curious about the experimental results in Table 2. I wonder why there is a significant improvement on all test sets except the CB test set. Why does the Headless model perform so much worse than the vanilla model on the CB test set? Could the authors provide a detailed analysis and explanation for this? Is there anything special about the CB test set?\n\n2. In Table 3, why is there no output for the Headless model's perplexity (ppl) on the validation set?\n\n3. I believe it's reasonable that the Headless model is more memory-efficient and exhibits higher training and inference efficiency on the XNLI benchmark compared to the Vanilla model. However, why is there such a significant improvement in quality as well? Where do these quality improvements primarily come from? Could the authors provide a more specific explanation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper maintains a high standard of quality. The methodology is well-structured and explained clearly, making it accessible to readers. The empirical results demonstrate substantial reductions in training computational requirements, improved downstream performance, and increased data efficiency. The study includes a significant increase in GLUE score and LAMBADA accuracy, indicating a quality improvement over classical LMs.\n\n2. The paper is well-written and clear in its explanations. It effectively communicates the core concepts and methodologies involved in the proposed approach. The objectives and benefits of headless training are well-articulated, and the exploration of pretraining hyperparameters adds to the clarity of the study.\n\n3. The paper's significance lies in its departure from traditional language model pre-training by introducing a contrastive objective. Sufficient experiments validate the advantages of the proposed contrastive learning loss over traditional cross-entropy loss, and the experimental results are relatively convincing.\n\nIn summary, it is a valuable contribution to the field of self-supervised language model pre-training. The proposed approach has the potential to impact the efficiency and effectiveness of language models in various applications.", "weaknesses": "1. The proposed method is not very novel and original. Although it departs from traditional probability prediction, offering a unique perspective on language model pre-training, there is lots of similar work that leverages the similar contrastive learning method. Thus, the use of Contrastive Weight Tying (CWT) in this context almost combines the existing ideas into the pretraining model, and it is not a very novel and creative methodology;\n\n2. For the currently most popular CLMs, the proposed method still requires fine-tuning on a small amount of training data; otherwise, the performance would be significantly worse than the vanilla approach, adding complexity to the training process;\n\n3. With the current trend of scaling up language models, it would be even more valuable to validate the proposed method on larger generative LLMs, such as ChatGLM or LLaMa.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z4T376hwqH", "reviewer": "Reviewer_ccZo", "summary": "This paper proposes replacing the standard cross-entropy-based language modeling objective (classification over token vocabulary) with a \"contrastive weight tying\" objective, which is more memory and compute efficient during training. Computing the full cross-entropy loss requires loading and multiplying the final model embeddings by a $d_{model} \\times V$ matrix, where $V$ is the vocabulary size (often $\\geq 30k$), which can be quite expensive (especially for small models, with few layers and $d_{model} \\ll V$). Instead, this paper proposes using a contrastive objective, where the targets are the input token embeddings for the masked tokens (\"weight tying\"), and the negative samples are taken from the other \"masked\" tokens. It shows in experiments that the proposed method is generally more efficient, and attains better performance, than the standard cross-entropy-based training. Lastly, to allow for generating text (using next token prediction) with these \"headless\" models, the paper proposes performing fine-tuning with a LM head for a small number of tokens ($< 2$% of pre-training dataset).", "questions": "- Am I understanding correctly that for the monolingual decoder experiments, the proposed method offers no speed advantages, because  \"negative samples correspond to every input embedding at a different position in the batch\"? Would it make sense to decouple the number of negative samples from the number of \"masked\" samples, to allow for controlling these two hyperparameters separately?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method can reduce the number of trainable parameters meaningfully for small models with large vocabulary sizes (e.g., by my calculations, reduce # parameters of BERT-base by ~18%).\n- The proposed method generally yields better accuracy on downstream tasks than standard LM training (across mono-lingual encoder experiments with BERT-base, mono-lingual decoder experiments with Pythia-70m, and multi-lingual encoder experiments with distilled multi-lingual BERT).\n- This method decouples the training speed from the number of tokens in the vocabulary, which allows for choosing the vocabulary size that attains best performance.", "weaknesses": "- The proposed method is not very novel --- straightforward application of contrastive loss function to LM training.\n- The experiments are relatively small-scale, only considering models < 140m parameters. Unclear if any of the benefits of this method hold for larger models.\n- The paper does not discuss that for larger models, the $d_{model} \\times V$ classification matrix corresponds to a tiny percentage of the total number of model parameters (and thus, total memory and computation during training). For example, it corresponds to roughly 2.5% and 0.4% of the Llama2 (7b) and Llama2 (70b) model parameters, respectively.\n- The proposed method adds a step to decoder-only LM training, as the model must first be pre-trained in a \"headless\" way, and then fine-tuned with a LM classification head. This adds complexity to the training process.\n- The paper does not perform careful ablations to help understand why the proposed method attains better model performance than standard cross-entropy (CE) LM training. For example, it could have considered weight-tying with the CE loss, or a contrastive loss without weight tying. This latter option could still be implemented in a compute-efficient way during training (since only a subset of columns of the classification matrix would need to be used). It could have also tried using different numbers of negative samples for the contrastive loss function (assuming # negative samples was decoupled from # masked samples), such that on one extreme (# neg samples = vocab size) the loss would be equivalent to the cross-entropy loss function (modulo the weight tying). Overall, I was confused by why the proposed method would yield better-performing models than cross-entropy.\n\nNote that I was torn on whether to assign this paper \"marginal accept\" or \"marginal reject\", but chose marginal accept given that I see the efficiency benefit of this approach for smaller models, and was intrigued that this approach could improve model quality. The reasons I was considering marginal reject were that I felt the method was not very novel, only leads to meaningful efficiency gains for small models (and no experiments with medium/large models were performed), and because I felt the paper didn't do a good job explaining why the proposed method performs better. Open to being swayed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6O61GTdSgH", "reviewer": "Reviewer_CHGU", "summary": "This paper proposes to replace the cross-entropy minimization by a new pre-training objective in language models: Contrastive Weight Tying (CWT). It consists in contrastively predicting the representation of the missing word. Instead of projecting the representation output by the model into the vocabulary, and using a softmax, the goal is to predict directly this static embedding of the reference word, which the model is trained for by (1) using a contrastive objective with the rest of the batch as negative examples (2) using input static embeddings as reference output embeddings, which is traditionally referred as *weight tying*. After a presentation of previous work, the paper introduces the method, consisting in weight tying, the contrastive objective with no projection, and how to adapt it for text generation, plus computational requirements. Experiments are performed on a Monolingual Encoder (GLUE), Monolingual Decoder (LAMBADA/LM Evaluation Harness), and Multilingual Encoder (XNLI Benchmark), with CWT showing improvements on all settings compared to the model trained classically. Lastly, the paper presents experiments on vocabulary size, as expanding the vocabulary is less costly and may provides some advantages with CWT.", "questions": "- On the missing related work: I will detail what I estimate to be essential. I believe you could find most of these references (and many other) in papers you have yourself cited (the papers describing ELECTRA and ELECTRIC, if only).\n    - You write \"This work stands as the first attempt to train language models using an explicit contrastive loss\".  The first that I am aware of would be [1], but there are many more recent references, especially those making use of Noise Contrastive Estimation; with [2] which, while binary in nature, has been adapted into a loss resembling your own in [3], though I believe the first use of a direct softmax-like objective with negative examples coming from the batch is from [4].\n    - Though I believe input/output embeddings weight sharing was a common approach that was very certainly implemented in most early languages models prior to 2017, your work combines this approach with a contrastive objective in order to get rid of the output vocabulary, which is a very nice idea. On that matter, I will mainly cite [5], a dedicated loss aiming at predicting static embeddings too; another paper [6] studies how to choose the target embeddings (which you use weight tying for).     \n    - Language modelling without an output vocabulary (contrastive objective or not) has been explored quite a lot in the past, with the specific problem you address in subsection 3.3 with fine-tuning having had many proposed solutions. I believe a nearest neighbour approach has been used in [5] and [6], but [7] proposes an energy-based model to repurpose a well-trained encoder (outputting *embeddings*) into a generative model.\n    - I believe you will find in this previous work many insights which will help you improve the design of your experiments and directions for improvement.\n\n- On experiments:\n    - Do you have any hypothesis regarding the better performance of CWT than the vanilla model *with the same number of training token* on most GLUE tasks ? \n    - In fine-tuning a causal LM for text generation, is the supplementary amount of compute received by the fine-tuned model compared to the vanilla model significant ? \n\n\n\n\n- [1] Quick Training of Probabilistic Neural Nets by Importance Sampling (Bengio and Sénacal, 2003)\n- [2] A fast and simple algorithm for training neural probabilistic language models (Mnih and Teh, 2013)\n- [3] Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (Ma and Collins, 2018)\n- [4] On Using Very Large Target Vocabulary for Neural Machine Translation (Jean et al, 2014)\n- [5] Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs (Kumar and Tsekov, 2019)\n- [6] On Target Representation in Continuous-output Neural Machine Translation (Tokarchuk and Niculae, 2022).\n- [7] Residual Energy-Based Models for Text Generation (Deng et al, 2019)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- This paper presents a simple and well motivated method for pre-training a language model, CWT.\n- The paper implements various experiments showing that CWT consistently improves performance, training efficiency, or both, over the classical version of the model.", "weaknesses": "The main issue I found with this paper is a huge lack of bibliographic work. It mainly keeps the paper from proper contextualization but I believe a proper related work would have greatly helped in making the experiments less shallow. \n- The lack of reference to relevant previous work keeps the paper from being well-positioned within the literature, especially when it comes to comparison with the numerous pre-existing similar methods,\n- It also keeps it from investigating relevant experimental settings - for example, comparison with similar objectives and ablation studies which could shed some light on what makes CWT work better than some previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "122jHxwB4W", "reviewer": "Reviewer_QSPQ", "summary": "The paper proposes a simple method for pre-training LMs with contrastive loss. Instead of evaluating logits for all vocabulary, only embeddings from the batch are used. Such an approach allowed us to pre-train LMs more effectively by reducing memory constraints. Surprisingly, this approach also performed well for autoregressive language modeling. For both autoregressive and masked language modeling, the proposed method showed improved results on downstream tasks, as well as improved training efficiency with respect to training hours.", "questions": "Please, refer to the Weaknesses section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method is straightforward yet shows promising results. \nPerformance on down-stream tasks is marginally improved with respect to vanilla models\nPre-training is more efficient with respect to training time and the total number of tokens\nThe paper is easy to follow, and the motivation is clear.", "weaknesses": "I would like to see more insights from the paper. It shows that HLMs provide better efficiency and down-stream performance, while such results are slightly contr-intuitive. More analysis could make the paper better. I.e., it would be beneficial to answer the question, \"what makes HLMs special, and why it outperforms vanilla LMs?\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)