Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024)

Abstract: The natural language processing field has been evolving around language models for the past few years, from the usage of n-gram language models for re-ranking, to transfer learning with encoder-only (BERT-like) language models, and finally to large language models (LLMs) as general solvers. LLMs are dominated by the decoder-only type, and they are popular for their efficacy in numerous tasks. LLMs are regarded as having strong comprehension abilities and strong capabilities to solve new unseen tasks. As such, people may quickly assume that decoder-only LLMs always perform better than the encoder-only ones, especially for understanding word meaning. In this paper, we demonstrate that decoder-only LLMs perform worse on word meaning comprehension than an encoder-only language model that has vastly fewer parameters.

heading: 1 Introduction
text: Large language models (LLMs) are highly effective tools for solving different kinds of problems in natural language processing (Qorib and Ng, 2023; Zhou et al., 2023), computer vision (Liu et al., 2023a), robotics (Zeng et al., 2023), and more. Due to their fascinating abilities to solve a myriad of tasks, large language models, which are dominated by the decoder-only type, are often considered general problem solvers (Mirchandani et al., 2023; Yao et al., 2023). Moreover, large language models are able to perform tasks outside of what they were trained on with few to no examples, a phenomenon referred to as emergent abilities (Wei et al., 2022).
Large language models have been shown to have strong comprehension abilities (Liu et al., 2023b). As such, it is natural to think that they are the best for lexical semantics. However, Zhu et al. (2024) reported that decoder-only language models struggle with understanding more nuanced contextual features. This motivates us to investigate the seman-
tic understanding of decoder-only language models compared to the encoder-only ones.
We evaluate the semantic understanding capability of language models through the word sense disambiguation (WSD) and word-in-context (WiC) tasks. Word sense disambiguation is the task to determine which sense of a word is meant in a particular context, while the word-in-context task is to identify whether the word senses of the same word in two sentences are the same or not (Figure 1). We investigate both open-source decoder-only models, Mistral (Jiang et al., 2023) and Llama 2 (Touvron et al., 2023), and a closed-source model, GPT-41 (OpenAI, 2023). We compare the results against a strong encoder-only model, DeBERTa-V3-Large (He et al., 2023).

heading: 2 Methods

heading: 2.1 Encoder-Only Models
text: We use encoder-only language models as a binary classifier for the word sense disambiguation and word-in-context tasks. For the WSD task, we follow the ESR (Song et al., 2021) method. The input to the model is the concatenation of the sentence s with the enhanced sense representation (ESR) of
1The architecture of GPT-4 is not transparent, but the model is trained with the next token prediction task. In addition, previous GPT versions are decoder-only models.
16339
one candidate sense c ∈ Cw for the target word w. The model then calculates the probability p(c) of the candidate sense c to be the correct sense, by projecting the hidden representation of the target word w to the binary classes. Lastly, we choose the sense c with the highest probability. With LM representing an encoder LLM, σ representing the softmax function, and the square bracket representing concatenation, the method can be written as follows:
p(c,¬c) = σ(W × LM([s; ESR(c)])w + b) (1) ŷ = arg max
c∈Cw p(c) (2)
p(c,¬c) is a 2-dimensional vector denoting the probabilities that w belongs to sense c or not. The predicted sense ŷ is the sense with the highest probability among all senses of w.
For the WiC task, the input to the model is the concatenation of the two sentences s1 and s2, prepended with a [CLS] pseudo-token. The answer is computed from the projection of the [CLS] pseudo-token to the binary classes.
p(yes, no) = σ(W × LM([s1; s2])CLS + b) (3) ŷ = arg max
c∈{yes,no} p(c) (4)

heading: 2.2 Decoder-Only Models
text: We perform WSD and WiC prediction by providing a natural prompt x as input to the decoderonly model and choosing the option y ∈ Y which is the most probable continuation of the prompt as the selected answer. For WSD, the prompt is in a multiple-choice question style with the option number Y = {1, 2, ..., k} as the expected answer. For the WiC task, the prompt is a question about whether the target word in the two sentences has the same sense and the answer options are Y = {yes, no}. The prompt examples are given in Figure 2.
ŷ = argmax y∈Y
p(y|x) (5)

heading: 3 Experiments
text: We evaluate the models’ performance in the zeroshot, few-shot (in-context learning), and fine-tuned settings. All experiments of decoder-only models use natural language prompts. For the few-shot setting, we prepend the prompt with four examples from the training set.

heading: 3.1 Dataset
text: For WSD, we utilize SemCor (Miller et al., 1994) as the training dataset. SemCor was manually annotated with WordNet senses and predominantly used in the literature for training supervised WSD systems (Zhong and Ng, 2010; Hadiwinoto et al., 2019; Song et al., 2021). The evaluation framework proposed by Raganato et al. (2017) incorporates five evaluation datasets from the Senseval/SemEval series: Senseval-2 (SE2) (Edmonds and Cotton, 2001), Senseval-3 task 1 (SE3) (Snyder and Palmer, 2004), SemEval-07 task 17 (SE07) (Pradhan et al., 2007), SemEval-13 task 12 (SE13) (Navigli et al., 2013), and SemEval-15 task 13 (SE15) (Moro and Navigli, 2015). We evaluate the models by calculating the F1 score on the concatenation of the five test sets (ALL) standardized to the same format and sense inventory of WordNet 3.0 (Table 1).
WiC (Pilehvar and Camacho-Collados, 2019) is a dataset to evaluate a model’s capability in differentiating word senses in a binary fashion. The input is two sentences that contain the same target word, and the label is yes if the word sense in the two sentences is the same, and no otherwise. We evaluate the accuracy of the models’ prediction on
the standard test split (Table 2).

heading: 3.2 Fine-Tuning
text: Parameter-Efficient Fine-Tuning (PEFT) (Liu et al., 2022) is a set of techniques where only a small subset of added or selected parameters are trained to allow for efficient fine-tuning of LLMs. For all the fine-tuning experiments, we use LoRA (Hu et al., 2022), one of the most widely used PEFT methods. We use the HuggingFace (Wolf et al., 2020) framework to fine-tune the models with LoRA. Additionally, we utilize 4-bit quantization to optimize memory usage and speed up inference.

heading: 3.3 Evaluation
text: With regard to our in-context learning experiments, we randomly select four examples from the training dataset. This set of four examples is subsequently fixed to ensure the consistency of the experimental settings, mitigating any possible influence on the scores that may arise from variations in the quality of in-context learning examples.
Since decoder-based models are predominantly utilized to generate text (Fu et al., 2023), it is imperative to constrain their output in order to render them suitable for classification tasks. In this regard, we employ the Language Model Evaluation Harness (Gao et al., 2023) framework to develop a methodology for calculating the generation probability to predict each class as the subsequent token based on the given prompt. Consequently, the class
with the highest generation probability is chosen as the predicted response.

heading: 4 Results
text: On the WiC task, the zero-shot scores of Mistral and Llama are close to random chance, probably because the models do not understand the expected output without any example (Table 3). On the WSD task, Llama zero-shot performance is relatively better even though the task seems harder (multiple possible answers instead of binary). Hu and Levy (2023) raised concerns that the zero-shot performance of language models on metalinguistic prompts, such as in our tasks, may not be representative of the model’s capability. Therefore, it is more important to focus on the few-shot and fine-tuned results.
Mistral 7B was reported to outperform Llama 7B and 13B across a wide range of benchmarks (Jiang et al., 2023), but we find that Llama performs better in almost all experimental settings. On the other hand, GPT-4 in the zero-shot setting outperforms both Mistral 7B and Llama 13B in their few-shot settings. GPT-4 performs exceptionally well in the zero-shot setting of the WSD task but only gains very slight improvement with in-context learning. As GPT is a closed-source model, information on its model training procedure and data is not accessible. We cannot rule out the possibility that GPT may have been trained on the WSD task or with WSD training and test data.
On the WiC task, fine-tuned Llama 2 performs close to DeBERTa-V3-Large. However, the score difference is still large when compared to the bigger DeBERTa 1.5B, which achieves an accuracy of 76.4 on the WiC task (He et al., 2021).

heading: 5 Analysis

heading: 5.1 Importance of Encoders
text: We hypothesize that an encoder can help language models understand word meanings better. As such, we also run the WSD experiment on an encoderdecoder LLM, Flan-T5-XL (Chung et al., 2022). The encoder-decoder model is evaluated using the same method as the decoder-only models (Section 2.2). Since the Flan-T5 model was instructiontuned, we compare it with the instruction-tuned versions of Mistral (Mistral 7B Instruct 0.1) and Llama 2 (Llama 2 13B Chat).
With much fewer parameters (3B), Flan-T5-XL outperforms Mistral 7B and even Llama 2 13B by
substantial margins (Figure 3). Flan-T5-XL also scores higher than the non-instruction-tuned versions of Mistral and Llama (Table 4). Raffel et al. (2020) also previously reported that an encoderdecoder model performs better than a comparable decoder-only model on the WiC and other natural language understanding tasks. This may suggest that having an encoder architecture helps in understanding word meaning.
Another interesting observation is that the instruction-tuned versions of Mistral and Llama perform worse than their base versions without instruction tuning (Table 3). After being fine-tuned on the WSD training data, the instruction-tuned Mistral achieves an F1 score of 65.2 (versus 78.0 without instruction tuning), while the instructiontuned Llama achieves an F1 score of 78.1 (versus 79.1 without instruction tuning). We believe this phenomenon can be explained by the “alignment tax” law (Ouyang et al., 2022), which states that reinforcement learning with human feedback (RLHF) comes at the cost of lower performance on certain downstream tasks.

heading: 5.2 Prompt Styles
text: For the same goal of finding the correct sense of a word in a given context, there are different ways of prompting a language model. We investigate various types of prompts to utilize decoder-based language models for the WSD task.
Multiple-choice questions (MCQ) – Inspired by the question answering task, we formulate the WSD task as choosing the correct sense among the possible senses for the target word w. The prompt starts by specifying w and a list of sense definitions of w, followed by the sentence containing w. One difference from the usual prompt for the multiplechoice question answering task is that we list the options in numbers (e.g., 1, 2, ...) instead of letters since the number of possible senses can exceed 26. This is the prompt style used for our main results.
Sentence completion (COMP) – Decoder-based LLMs are pre-trained with the sentence completion task, so it is imperative to try formulating the downstream task as a sentence completion task too. The COMP prompt asks for the sense definition of w given the sentence, and expects the model to give the appropriate sense definition following the definition from WordNet.
Binary classification (BIN) – Encoder-only language models solve WSD by formulating it as a binary classification, predicting whether each sense definition of w is appropriate in the given con-
text. The encoder models are effective in solving the WSD task with this formulation, so we are intrigued to investigate whether using this formulation can make the decoder-based models perform better.
The prompt examples are given in Appendix A. We run experiments by fine-tuning Mistral and Llama 2 on a subset of the training data that contains 10,000 target words. Despite the intuitive motivation, we find that the other prompt styles do not increase the performance of Mistral or Llama 2 (Figure 4).

heading: 5.3 Low-Resource Settings
text: LLMs are known to be good at adapting to a new task with few to no examples. In this experimental setting, we investigate whether decoder-based language models can outperform encoder-only ones when the amount of training data is limited. We fine-tune all models in the WSD task with subsets of the training data containing 10,000 and 20,000 target words. We found that even with less data, the encoder-only model still outperforms the decoderonly models (Figure 5).

heading: 6 Related Work
text: Prior work has reported performance comparisons of decoder-only models with encoder-only models on different tasks. Li et al. (2023) investigate the performance of decoder-based LLMs on financial text analytics tasks and report that GPT-4 with incontext learning has comparable performance to fine-tuned encoder-only models on financial sentiment analysis and lower performance on financial headline classification and relation extraction.
Yu et al. (2023) investigate Llama 2 and GPT4 on named entity recognition, political ideology prediction, and misinformation detection. They also find that GPT-4 can achieve comparable performance on some of the tasks and perform worse on others. These works support our findings that LLMs still have limited capabilities on some downstream tasks, but they have not investigated LLMs’ capabilities in understanding word meaning.

heading: 7 Conclusion and Future Work
text: In this paper, we demonstrate that decoder-only LLMs perform worse on word meaning comprehension than an encoder-only language model with vastly fewer parameters. We report the performance of Mistral 7B, Llama 2 13B, and GPT-4 on the WSD and WiC tasks and show that DeBERTaV3-Large with vastly fewer parameters outperforms all aforementioned decoder-only models on both tasks. We discuss the importance of the encoder in understanding word meaning by running the same experiment on an encoder-decoder model, Flan-T5-XL, and find that it also outperforms the decoder-only models.
This work provides a concrete task on which encoder-only language models can still outperform decoder-only language models, but it is not meant as a study on neural network architecture. Future research can explore the optimal architecture and the roles of the encoder and decoder in natural language understanding. Additionally, investigating methods to enhance the capabilities of decoderonly models in comprehending word meaning is also a valuable research direction.
Limitations
Our research and analysis are focused on the English language. We only evaluated recent language models that fitted into our compute budget. Our experiments were carried out using the current stateof-the-art LLMs in both encoder-only and decoder-
only architectures with their respective model sizes and training data as is. As previously stated, the decoder-only models were trained with quantization and parameter-efficient fine-tuning, following the common practice for decoder-only LLMs in the literature. Our finding is based on the optimal prompting method currently known for classification tasks for decoder-only models. Given these limitations, it is possible that a different conclusion can be reached in the future with improved language models, training methods, or prompting techniques.

heading: A Prompt Examples
text: A.1 MCQ Style for WSD
• Prompt : Given the following list of definitions of the
word “art”: 1. the products of human creativity; works of art collectively 2. the creation of beautiful or significant things 3. a superior skill that you can learn by study and practice and observation 4. photographs or other visual representations in a printed publication Which definition is used by the word “art” in the bracket in the following sentence? Sentence: “The (art) of change-ringing is peculiar to the English , and , like most English peculiarities , unintelligible to the rest of the world .” Answer:
• Expected Answer : 3
A.2 COMP Style for WSD
• Prompt : The definition of [art] in “The [art] of changeringing is peculiar to the English , and , like most English peculiarities , unintelligible to the rest of the world .” is
• Expected Answer : “a superior skill that you can learn by study and practice and observation.”
A.3 BIN Style for WSD
• Prompt : Is it correct that the definition of the word “art” in “The (art) of change-ringing is peculiar to
the English , and , like most English peculiarities , unintelligible to the rest of the world .” is “the products of human creativity; works of art collectively”?
• Expected Answer : no

heading: B Compute Budget
text: All zero-shot and few-shot computations are done on a single NVIDIA A100 40GB GPU, while all fine-tuning computations are done on a single NVIDIA A100 80GB GPU.

heading: C Experimental Details
text: We describe our hyper-parameters’ search intervals for the decoder-only and encoder-decoder models
in Table 7. For the encoder-only model, we follow the final hyper-parameters of ESR (Song et al., 2021). The hyper-parameter search was done manually, so not all possible hyper-parameter values were investigated for all models. We describe the final hyper-parameters of the models in Table 5. All our results are from single-run experiments.

heading: D Resources
text: We utilize open source code for our experiments. For the decoder-only and encoder-decoder models, our fine-tuning code is based on the Alpaca-Lora code base2. For the encoder-only model, we use the ESR code base3. For the evaluation, including the zero-shot and few-shot settings, we use the Language Model Evaluation Harness (LM-Eval) framework4.
2https://github.com/tloen/alpaca-lora 3https://github.com/nusnlp/esr 4https://github.com/EleutherAI/
lm-evaluation-harness/

heading: Future Work
text: Future research can explore the optimal architecture and the roles of the encoder and decoder in natural language understanding. Additionally, investigating methods to enhance the capabilities of decoderonly models in comprehending word meaning is also a valuable research direction. Limitations
Our research and analysis are focused on the English language. We only evaluated recent language models that fitted into our compute budget. Our experiments were carried out using the current stateof-the-art LLMs in both encoder-only and decoder-
only architectures with their respective model sizes and training data as is. As previously stated, the decoder-only models were trained with quantization and parameter-efficient fine-tuning, following the common practice for decoder-only LLMs in the literature. Our finding is based on the optimal prompting method currently known for classification tasks for decoder-only models