Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
Authors: Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Boxing Chen, Sarath Chandar
Accepted at: acl25

Do Robot Snakes Dream like Electric Sheep?
Investigating the Effects of Architectural Inductive Biases on Hallucination

Jerry Huang
Chandar Research Lab
Mila & Université de Montréal

Prasanna Parthasarathi*
Noah’s Ark Lab

Mehdi Rezagholizadeh†
Advanced Micro Devices

Boxing Chen
Noah’s Ark Lab

Sarath Chandar
Chandar Research Lab
Mila & Polytechnique Montréal & CIFAR AI Chair

Abstract
The growth in prominence of large language models (LLMs) in everyday life can be largely attributed to their generative abilities, yet some of this is also owed to the risks and costs associated with their use. On one front is their tendency to hallucinate false or misleading information, limiting their reliability. On another is the increasing focus on the computational limitations associated with traditional self-attention based LLMs, which has brought about new alternatives, in particular recurrent models, meant to overcome them. Yet it remains uncommon to consider these two concerns simultaneously. Do changes in architecture exacerbate/alleviate existing concerns about hallucinations? Do they affect how and where they occur? Through an extensive evaluation, we study how these architecture-based inductive biases affect the propensity to hallucinate. While hallucination remains a general phenomenon not limited to specific architectures, the situations in which they occur and the ease with which specific types of hallucinations can be induced can significantly differ based on the model architecture. These findings highlight the need for better understanding both these problems in conjunction with each other, as well as consider how to design more universal techniques for handling hallucinations.

1 Introduction
Large language models (LLMs) have rapidly emerged as a every-day tool in modern life (OpenAI, 2024), with many relying on their abilities to accomplish a variety of specific tasks. However, this opened up concerns relating to their propensity to hallucinate (Huang et al., 2023), with no concrete reasons for this behavior (Dziri et al., 2022b; Rawte et al., 2023; Chen et al., 2024b), hindering the ability to directly train LLMs that are consistently factual or able to explain themselves through their knowledge (Madsen et al., 2024a; Prato et al., 2023, 2024; Huang et al., 2024).

In parallel, as LLMs evolve and existing limitations are discovered, alternative architectures have become increasingly common and grow in popularity. In particular, Transformer LLMs (Vaswani et al., 2017) and linear sequence models (Gu et al., 2022; Gu and Dao, 2024) present contrasting methods of encoding sequences, with the Transformer using attention (Bahdanau et al., 2015) to form lossless representations of the context, while linear sequence models follow the recurrent neural network (Rumelhart et al., 1986; Jordan, 1986) in their use of a compressed state representation.

With the intensifying focus in both directions, a notable void exists in verifying how each can affect the others in conjunction. For example, existing works in hallucination detection and mitigation focus almost exclusively on Transformer-based models (Maynez et al., 2020; Longpre et al., 2021a; Guerreiro et al., 2023; Shi et al., 2023; Ji et al., 2023b; Farquhar et al., 2024; Wei et al., 2024), without extension to recurrent-style models, despite the use of a unified hidden representation potentially acting as an information bottleneck that can induce more common hallucinations. This lack of unified understanding on both topics prompts the need for a more explicit investigation.

In this work, we comprehensively explore the differences between pure attention LLMs and recurrent LLMs, specifically with respect to the propensity to hallucinate. Using a set of 20 different hallucination tasks, categorized into 6 groups that evaluate both faithfulness and factuality hallucinations, we evaluate across numerous open-source LLMs that range in scale from under 1B parameters to 70B parameters all the while covering a variety of different architecture choices such as self-attention, recurrent and hybrid models. We further evaluate across factors such as instruction-tuning, all to build a more comprehensive picture of architecture-
specific phenomena with respect to hallucination. From this, we observe the following:

(1) Viewed very broadly over various different tasks and settings, neither Transformer-based nor recurrent/hybrid LLMs appear to induce hallucinations more often than others.

(2) However, shifting to individual tasks, it becomes evident that they result in disparate tendencies on specific tasks evaluating for unique criteria such as recalling long-tailed factual knowledge and falling into memorization traps, highlighting that model architecture may promote specific behavior that renders some types of hallucinations more common.

(3) Evaluating the effects of scale and instruction-tuning, we observe that though factuality remains dependent on model size, recurrent/hybrid architectures are often more faithful at smaller sizes and observe significantly fewer faithfulness benefits from instruction-tuning and scaling compared to self-attention models.

These results highlight that while some hallucinations may be quite consistent across architectures, others are the direct results of specific model design choices that go into the LLM construction. This hints towards the need for more careful consideration on this front, as different techniques for addressing this problem can potentially be highly catered towards specific models, bringing to the forefront the need for better consideration of both these problems in the face of each other.

2 Related Work

Hallucinations in LLMs. Hallucination broadly refers to when LLMs generate information that does not directly follow from the context, such as nonsensical or irrelevant answers to questions (Ji et al., 2023a). While it has grown in importance due to its direct relationship with ensuring the safe and responsible use of LLMs, both classifying and quantifying the hallucinations is challenging. In particular, it is difficult to ascertain if the divergence occurs because of specific data heuristics (Lebret et al., 2016; Wiseman et al., 2017) or because of the innate lack of similarity between pre-training and downstream tasks (Rashkin et al., 2021), while measuring for hallucinations automatically introduces various biases (Reiter, 2018; Tian et al., 2020; Ganguli et al., 2022) that may not fully capture the scope of the errors. However, a variety of task-specific benchmarks (Li et al., 2020; Pagnoni et al., 2021; Zhou et al., 2021; Santhanam et al., 2022) have shown various LLMs to struggle with factual inconsistencies, highlighting a need to render them safer for every-day use.

LLM Architectures. General LLM architecture can be broadly thought to be composed of two components: token mixers which serves to model transformations between time steps (such as attention and recurrent layers) and channel mixers, such as multi-layer perceptrons and mixtures-of-experts (MoE) (Jacobs et al., 1991; Shazeer et al., 2017), which allow communication between different channels within a single time step. Accordingly, token mixing often forms a computational bottleneck in terms of time complexity while channel mixers consist of a memory bottleneck. Though the contemporary standard for token mixing remains self-attention, alternatives are becoming increasingly common as they begin to display more promise. These include the use of linear attention (Katharopoulos et al., 2020) to compensate for the quadratic memory complexity of vanilla self-attention, to new recurrent models (Gu and Dao, 2024; Gemma Team, 2024b) that function as a linear recurrent neural network but can process all elements of a sequence in parallel, and well as hybrid mixtures of recurrent mechanisms and attention (AI21, 2024; Dao and Gu, 2024) that have emerged as a meaningful competitor.

Architecture and Hallucination. Despite growing research in new architectural components, their effects on hallucination have yet to be studied. While some works (Madsen et al., 2024b; Hu et al., 2024; Schimanski et al., 2024) have proposed modifications to either the learning/generation pipeline as a way of reducing hallucinations, proposals for hallucination reduction through structural modification have yet to be suggested. Additionally, though some works (Elhage et al., 2021; Fu et al., 2023; Lutati et al., 2023; Poli et al., 2024) have demonstrated self-attention to aptly solve synthetic tasks that form an essential component of language modeling, these fail to remain faithful on more realistic datasets, punctuating a major limitation of existing LLMs. Finally, though tangential work demonstrates that recurrent models may suffer from issues with information retention (Vardasbi et al., 2023), formally defining a link with hallucination remains necessary. These issues lead to a lack of clarity
on this front, accentuating the need for a formal investigation comparing hallucination alongside architectural paradigms. In particular, the choice of token mixer is important, as this is the component that governs how information is shared between different elements of the sequence to form a holistic representation. As such, an effective token mixer will adequately enable enough information from the context to propagate forward during the generation phase (Olsson et al., 2022; Arora et al., 2024), potentially enabling models to hallucinate less.

3 Background

Attention for Sequences. Vanilla self-attention as used in Transformers is powerful but costly. When provided an embedded text representation as a sequence of tokens \( x \in \mathbb{R}^{L \times d} \), each Transformer layer in the network applies a function

\[
T_\ell(x) = \mathrm{FF}_\ell(A_\ell(x) + x) + A_\ell(x)
\]

where \( A_\ell \) is the self-attention of the \( \ell \)-th layer and \( \mathrm{FF}_\ell \) is the following feed-forward network. Self-attention computes, for a token at position \( i \) in a sequence, a weighted average of the feature representations of all tokens (the values \( V_\ell \)) in the sequence with a weight proportional to a similarity score between \( i \) (the query \( Q_\ell \) at position \( i \)) and the rest of the sequence (the keys \( K_\ell \)). In particular, these can be computed for all positions in parallel

\[
\begin{align*}
Q_\ell &= xW^Q_\ell & K_\ell &= xW^K_\ell & V_\ell &= xW^V_\ell \\
A_\ell(x) &= V_\ell' = \operatorname{softmax}(Q_\ell K_\ell^T / \sqrt{d}) V_\ell
\end{align*}
\]

providing the model a lossless representation of the complete past context. This can be seen as equivalent to search and retrieval within a database, where search is defined using query-key parameterizations and retrieval with value parameterization. In this setting, the database from which the information is being retrieved is equivalent to the model parameters, which store information from a training corpus that the weights \( W^K, W^Q, W^V \) parameterizing the model attempt to mimic.

Multi-head attention, a variant of self-attention popularized by the Transformer (Vaswani et al., 2017), has become the dominant variant used in LLMs such as LLaMA (LLaMA Team, 2024) and Gemma (Gemma Team, 2024a). Additional variants, such as multi-query attention from Falcon (Almazrouei et al., 2023) and grouped-query attention from Mistral (Jiang et al., 2023), adapt multi-head attention but remain build upon the same underlying principle of self-attention. Consequently, such methods are considered to fall under the family of self-attention token mixers.

Recurrent LLMs. A concern for Transformers is the quadratic complexity of attention with sequence length, leading a focused on improving this bound as it directly affects the ability to learn from long sequences. Instead of directly modifying attention (Katharopoulos et al., 2020; Kitaeve et al., 2020; Choromanski et al., 2021; Zeng et al., 2025), Gu et al. (2020) motivated a novel paradigm using state-space models (SSMs) from control theory. SSMs map an input \( x(t) \in \mathbb{R}^d \) to an intermediate state \( h(t) \in \mathbb{R}^n \) that is then projected to an output \( y(t) \in \mathbb{R}^d \):

\[
h'(t) = Ah(t) + Bx(t), \quad y(t) = Ch(t) + Dx(t)
\]

where \( A, B, C \) and \( D \) are trainable parameters and \( h'(t) \) represents the rate at which \( h(t) \) changes. Gu et al. (2021) use this paradigm to define a recurrent model to work on discrete signals, in which case the input can be regarded as discretized data sampled from a continuous signal with a step size \( \Delta \), for which the corresponding SSM is defined by:

\[
\begin{align*}
h_t &= \overline{A}h_{t-1} + \overline{B}x_t & y_t &= \overline{C}h_t + \overline{D}x_t \\
\overline{A} &= \frac{(I + \Delta A/2)}{(I - \Delta A/2)} & \overline{B} &= \frac{\Delta B}{(I - \Delta A/2)}
\end{align*}
\]

and \( \overline{C} = C \) (\( \overline{D} \) is equivalent to a residual connection and set to 0.) Thus

\[
\overline{K} = (\overline{CB}, \overline{CAB}, \ldots, \overline{CA}^{L-1}\overline{B}) \quad y = \overline{K} * x
\]

where \( \overline{K} \) is the SSM kernel. As \( y \) can be computed in \( O(L \log L) \) with a Fast Fourier Transform (Cormen et al., 2009), the entire output can be computed in tandem, given the matrices that parameterize the system. Furthermore, setting \( A \) as a Hurwitz matrix, SSMs can preserve long-term information, overcoming a long-standing issue (Bengio et al., 1994) with prior recurrent models (Hochreiter and Schmidhuber, 1997; Cho et al., 2014).

Further works have modified this structure; Gu and Dao (2024) use input-dependent \( B \) and \( C \) information filtering. LRU/Hawk (Orvieto et al., 2023; De et al., 2024) remove the discretization step from a continuous signal and instead learn discrete matrices for \( A, B, C \) and \( D \) directly. RWKV (Peng et al., 2023, 2024) uses a novel WKV
sequence mixing operator that acts like an RNN. However, these models all share the use of hidden states from which token-level information must pass through in order to interact with future tokens. This has also led to the creation of hybrid mixtures of recurrence and attention (Dao and Gu, 2024; De et al., 2024; AI21, 2024) to form new classes of models meant to capture the benefits of both.

Different sequence (or time) mixing modules attempt to encode the history of tokens in a fixed representation. Recurrent architectures maintain this representation and learn to update it during pre-training, but this can fall short of attention (Vardasbi et al., 2023; Jelassi et al., 2024; Huang, 2025; Wang et al., 2025) as they are constrained by its size (Arora et al., 2024). Meanwhile, attention has a theoretically infinite context window (albeit being costly). As recurrent structures are constrained to make the predictions based on the recent context (which in practice can be long, but remains shorter than what self-attention can attend to), the expressivity of the hidden representation directly affects task performance (Sun et al., 2024) and parametric knowledge can be predisposed to favor the prediction of certain tokens. We model this effect on tasks under the lens of hallucination: faithfulness and factuality. Faithfulness expects the model to follow an instruction (that is largely recent), while factuality assumes the use of token-to-token relations learned during pre-training for a broader contextual representation. Following this, we ask two questions: (1) Do self-attention and recurrent architectures hallucinate differently? (2) Does instruction-tuning and/or size benefit the different classes of LLMs equally?

4 Experiments and Results

Models. To investigate, we use models that vary in size and architecture, with a particular focus on different sequence mixing methods. These include self-attention models (Pythia, LLaMA2/3 (LLaMA Team, 2024), Falcon (Almazrouei et al., 2023), Mistral (Jiang et al., 2023), Gemma (Gemma Team, 2024a) and Mixtral (Jiang et al., 2024)), recurrent models (Mamba (Gu and Dao, 2024; Dao and Gu, 2024), FalconMamba and RWKV/Finch (Peng et al., 2023)), as well as hybrid models (RecurrentGemma (Gemma Team, 2024b), Jamba (AI21, 2024)). We use base and instruction-tuned variants when available. Additional details are available in Appendix A.

Datasets. We evaluate on the Hallucination Leaderboard (Hong et al., 2024), consisting of tasks

1) Closed-book Open-domain QA♣: NQOPEN (Kwiatkowski et al., 2019), TRIVIAQA (Joshi et al., 2017), TRUTHFULQA (Lin et al., 2022), POPQA (Mallen et al., 2023)

2) Summarization♣: XSUM (Narayan et al., 2018), CNN/DM (See et al., 2017)

3) Reading Comprehension♣: RACE (Lai et al., 2017), SQUADV2 (Rajpurkar et al., 2018), NQ-SWAP (Longpre et al., 2021b)

4) Instruction Following♣: MEMOTrap (Liu and Liu, 2023), IFEVAL (Zhou et al., 2023)

5) Hallucination Detection: FAITHDIAL♣ (Dziri et al., 2022a), HALUEVAL♣ (Li et al., 2023a), TRUE-FALSE♣ (Azaria and Mitchell, 2023)

6) Fact Checking♣. FEVER (Thorne et al., 2018)

For tasks, a higher score (ranging from 0 to 100) indicates better performance. Tasks are further divided into faithfulness (♣), i.e. whether the generation adheres to the given context, and factuality (♦), i.e. whether the generation is factually correct.

4.1 Investigating Task Biases

Our first interest is to verify whether specific choices in architecture can lead to highly evident patterns in performance1. While direct differences can be difficult to quantify due to differences in how each model is trained, some pattern are consistent, described as follows.

O1: Sequence models miss rare knowledge.

Interestingly, recurrent and hybrid models all significantly underwhelm on POPQA, a task that tests for uncommon knowledge (left plot in Figure 1). For example, FalconMamba achieves an exact-match (EM) score of 0.7 compared to 17.5 by a similar Falcon. Similarly, 2B and 9B RecurrentGemma achieve 7.6 and 16.0 EM compared to 14.6/21.3 EM for 2B/7B Gemma and 13.6/18.2 by 2B/9B Gemma2, while Jamba (0.4 EM) performs worse than a similarly sized Mixtral (31.9 EM).

Previous work (Vardasbi et al., 2023) suggests that hidden states can become dense and difficult to extract information from. However, the brevity and simplicity of the prompt here begs the question whether the hidden representation has the opportunity to become significantly dense. Additional

1Numbers provided in Table 4, 5 and 8 in Appendix B.
Figure 1: Performance on POPQA (left) and MEMOTrap (right). Recurrent and hybrid models significantly outperform similar pure attention-based alternatives on MEMOTrap, but the opposite is true on POPQA.

works (Dao and Gu, 2024) instead posit that using recurrent models with additional mechanisms enabling input filtering can enhance performance on domains such as language; however, the relevance of this claim here is again questionable. Yet this result is significant as it indicates recurrent models, despite the task’s simplicity, struggle when compared to attention-based equivalents, suggesting a link with the sequence mixing. Instead, the reason may be that recurrent/hybrid models do not learn by explicitly memorizing and acting as a retrieval system (as discussed in Section 3), where the learning mechanism may not have led to said information being stored within parameters.

O2: Recurrent models may rely less on memory, leading to more reliance on context.
There are also cases where recurrent models consistently outperform attention equivalents, namely on MEMOTrap (right plot in Figure 1). Here, the LLM is prompted to complete a well-known proverb with an ending that deviates from the commonly used ending, testing for over-reliance on knowledge memorized from a training corpus. This suggests that although recurrent models might struggle with long-tail knowledge, they are less prone to ignoring contextual cues not stored within parametric memory, or they more model the dynamics based on the context as opposed to acting as a retrieval system. Noticeable drops exist from FalconMamba (58.8) to Falcon (38.9) and from RecurrentGemma 2B/9B to Gemma2 2B/9B (53.7/43.4 vs 42.7/34.4), suggesting that attention can lead to more hallucinations. It further reveals a benefit of recurrent layers; by not memorizing information directly, there may be a reduced tendency to repeat previous observations (Jelassi et al., 2024) and greater focus on the context.

O3: Scale is required for emergent qualities.
However, some scale to the data and model remain important. Mamba, Finch and Pythia fail to draw the same distinctions as the other models, but given the scale of training for these (300B tokens compared to >1T tokens for other models) and the fact that the discussed tasks relate directly to information memorization, it is possible that these phenomena fail to emerge at this data scale. This is particularly evidenced by how these models all perform better with size on MEMOTrap, which is designed in a way such that larger models should normally perform worse.

4.2 On the role of Instruction-Tuning.
The use of instruction-tuning appears to show inconsistent improvements across categories (Table 1) and tasks2. There exists no strict pattern for overall hallucination; on some tasks, its use is effective for all (ex. TRUTHFUL QA) or no (ex. TRIVIAQA) models. Smaller trends also exist, such as instruction-following observing the strongest gains.

However, changes in faithfulness are generally positive for pure attention models and negative for models with recurrent layers, whereas changes in

2See Figure 5.
<table>
  <tr>
    <th>Model Name</th>
    <th>Hallu.</th>
    <th>Detection</th>
    <th>Instr. Following</th>
    <th>Closed-Book QA</th>
    <th>Reading Comp.</th>
    <th>Sum.</th>
    <th>Fact-Checking</th>
    <th>Faithfulness</th>
    <th>Factuality</th>
  </tr>
  <tr>
    <th colspan="10">Attention-Only Models</th>
  </tr>
  <tr>
    <td>Gemma-2B</td>
    <td>49.65 (± 11.51)</td>
    <td>30.25 (± 12.84)</td>
    <td>38.20 (± 6.92)</td>
    <td>38.96 (± 13.98)</td>
    <td>24.93 (± 4.53)</td>
    <td>54.14 (± 1.68)</td>
    <td>35.85 (± 1.92)</td>
    <td>32.56 (± 4.56)</td>
  </tr>
  <tr>
    <td>Gemma-7B</td>
    <td>53.83 (± 5.67)</td>
    <td>31.94 (± 6.79)</td>
    <td>27.43 (± 4.13)</td>
    <td>31.35 (± 5.83)</td>
    <td>20.63 (± 1.84)</td>
    <td>43.28 (± 5.22)</td>
    <td>36.48 (± 2.71)</td>
    <td>44.50 (± 6.84)</td>
  </tr>
  <tr>
    <td>Gemma2-2B</td>
    <td>58.33 (± 8.64)</td>
    <td>26.73 (± 10.80)</td>
    <td>29.59 (± 4.83)</td>
    <td>32.95 (± 5.26)</td>
    <td>16.04 (± 0.47)</td>
    <td>39.57 (± 18.96)</td>
    <td>35.61 (± 0.65)</td>
    <td>34.96 (± 8.88)</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>65.64 (± 6.31)</td>
    <td>23.76 (± 30.87)</td>
    <td>39.36 (± 3.13)</td>
    <td>39.74 (± 7.87)</td>
    <td>21.38 (± 3.35)</td>
    <td>62.06 (± 6.47)</td>
    <td>40.56 (± 7.55)</td>
    <td>46.85 (± 3.13)</td>
  </tr>
  <tr>
    <td>Gemma2-7B</td>
    <td>62.03 (± 12.43)</td>
    <td>26.62 (± 33.58)</td>
    <td>47.19 (± 2.86)</td>
    <td>43.04 (± 5.02)</td>
    <td>28.92 (± 0.74)</td>
    <td>68.28 (± 1.88)</td>
    <td>41.54 (± 13.82)</td>
    <td>53.92 (± 2.36)</td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>53.63 (± 4.38)</td>
    <td>28.94 (± 8.39)</td>
    <td>37.64 (± 1.89)</td>
    <td>27.58 (± 3.74)</td>
    <td>25.19 (± 0.69)</td>
    <td>51.38 (± 5.88)</td>
    <td>35.14 (± 4.81)</td>
    <td>42.51 (± 8.12)</td>
  </tr>
  <tr>
    <td>LLaMA2-13B</td>
    <td>67.80 (± 3.68)</td>
    <td>26.57 (± 9.72)</td>
    <td>39.65 (± 8.65)</td>
    <td>32.17 (± 3.64)</td>
    <td>27.33 (± 0.23)</td>
    <td>62.35 (± 1.64)</td>
    <td>40.83 (± 0.88)</td>
    <td>46.27 (± 0.49)</td>
  </tr>
  <tr>
    <td>LLaMA2-70B</td>
    <td>62.34 (± 18.78)</td>
    <td>30.24 (± 17.11)</td>
    <td>47.85 (± 2.53)</td>
    <td>39.48 (± 8.21)</td>
    <td>28.00 (± 0.55)</td>
    <td>66.63 (± 1.45)</td>
    <td>41.74 (± 5.82)</td>
    <td>53.69 (± 2.34)</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>60.76 (± 18.12)</td>
    <td>22.06 (± 26.76)</td>
    <td>41.55 (± 1.38)</td>
    <td>33.52 (± 3.62)</td>
    <td>26.62 (± 1.60)</td>
    <td>60.84 (± 4.82)</td>
    <td>37.60 (± 10.89)</td>
    <td>48.14 (± 1.60)</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>71.78 (± 8.69)</td>
    <td>21.59 (± 25.68)</td>
    <td>48.07 (± 3.43)</td>
    <td>46.38 (± 7.73)</td>
    <td>28.91 (± 1.68)</td>
    <td>69.57 (± 0.99)</td>
    <td>45.92 (± 6.23)</td>
    <td>54.80 (± 2.81)</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>60.48 (± 3.36)</td>
    <td>28.39 (± 16.85)</td>
    <td>41.24 (± 5.21)</td>
    <td>32.03 (± 1.92)</td>
    <td>26.77 (± 0.77)</td>
    <td>58.59 (± 6.78)</td>
    <td>38.47 (± 4.67)</td>
    <td>47.42 (± 4.17)</td>
  </tr>
  <tr>
    <td>Mistral-8x7B</td>
    <td>73.51 (± 9.15)</td>
    <td>26.84 (± 17.69)</td>
    <td>48.99 (± 4.33)</td>
    <td>40.66 (± 3.99)</td>
    <td>27.91 (± 1.87)</td>
    <td>68.35 (± 9.49)</td>
    <td>46.16 (± 3.89)</td>
    <td>55.15 (± 3.58)</td>
  </tr>
  <tr>
    <td>Falcon-7B</td>
    <td>52.40 (± 2.90)</td>
    <td>25.54 (± 12.62)</td>
    <td>33.03 (± 4.42)</td>
    <td>29.18 (± 1.64)</td>
    <td>20.70 (± 1.88)</td>
    <td>46.91 (± 7.71)</td>
    <td>34.91 (± 7.71)</td>
    <td>36.92 (± 7.75)</td>
  </tr>
  <tr>
    <th colspan="10">Recurrent and Hybrid Models</th>
  </tr>
  <tr>
    <td>RecurrentGemma-2B</td>
    <td>52.88 (± 2.33)</td>
    <td>33.43 (± 3.12)</td>
    <td>27.36 (± 1.85)</td>
    <td>30.66 (± 1.77)</td>
    <td>18.41 (± 2.86)</td>
    <td>42.97 (± 3.63)</td>
    <td>36.55 (± 0.14)</td>
    <td>31.46 (± 1.39)</td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B</td>
    <td>55.75 (± 1.87)</td>
    <td>31.67 (± 12.96)</td>
    <td>36.79 (± 2.83)</td>
    <td>36.57 (± 5.66)</td>
    <td>22.99 (± 1.84)</td>
    <td>51.25 (± 13.36)</td>
    <td>37.61 (± 0.17)</td>
    <td>42.96 (± 3.28)</td>
  </tr>
  <tr>
    <td>Jamla</td>
    <td>57.66 (± 2.90)</td>
    <td>43.36 (± 16.76)</td>
    <td>39.50 (± 8.38)</td>
    <td>33.00 (± 1.88)</td>
    <td>23.72 (± 15.48)</td>
    <td>59.88 (± 2.60)</td>
    <td>39.80 (± 0.31)</td>
    <td>45.95 (± 8.91)</td>
  </tr>
  <tr>
    <td>FalconMamba-7B</td>
    <td>55.80 (± 0.73)</td>
    <td>42.97 (± 5.19)</td>
    <td>39.94 (± 0.43)</td>
    <td>23.76 (± 8.85)</td>
    <td>23.89 (± 0.89)</td>
    <td>61.85 (± 1.88)</td>
    <td>35.92 (± 0.26)</td>
    <td>47.02 (± 0.38)</td>
  </tr>
</table>

Table 1: Changes in performance from the use of instruction-tuning. While factuality does not exhibit a specific trend, faithfulness is shown to improve within attention-based models but not recurrent or hybrid models.

factuality show no consistent trends. This suggests that instruction-tuning for LLMs may enable particular learning patterns which become irrelevant when using recurrent layers. Their inductive biases therefore can have a particular effect on specific forms of hallucinations emerging, with instruction-tuning being a manner to mitigate these.

Recalling from Section 3, instruction-tuning can be understood as shifting the projection weights towards a new space through the additional fine-tuning (which is of itself simply an additional step of language modeling). This leads to the overwriting of some of the pre-training corpus with that used for fine-tuning, effectively replacing the keys and values from the underlying retrieval storage. While effects are ambiguous on factuality, as some potentially relevant facts can be removed, faithfulness often improves from this process, as the fine-tuning corpus is more likely to contain information relevant to context following and thereby make it easier for the attention mechanism to retrieve relevant information for such a purpose. In addition to highlighting how enhancing the ability to follow instructions does not necessarily lead to factually correct results, another conclusion stemming from the fact that recurrent models generally observe no benefits in faithfulness after instruction-tuning is that such models’ inductive biases render them inherently more faithful or the process of making them more faithful is distinct from self-attention.

![Line chart showing changes in factuality and faithfulness as model size increases.](page_1012_1342_495_377.png)

Figure 2: Changes in factuality (top) and faithfulness (bottom) as models are increased in size. Score range between 0 and 100. Factuality always increases with the number of parameters, however faithfulness increases are only meaningful for pure-attention models.

4.3 Impact of Model Size on Hallucinations.

With the large amount of prior work (Kaplan et al., 2020; Wei et al., 2022; Hoffmann et al., 2022) concluding that model size can play a significant role in model reasoning abilities, we provide this additional axis of variation in Figure 2.

Immediately, we observe that increased size
Figure 3: Performance to scale values for factuality and faithfulness. Colors differentiate model families while shapes differentiate the model type. Factuality improves with model size (indicated by values >0) and the model type does not have a link with this relative improvement. Recurrent/hybrid models show low values for faithfulness, indicating that size increases generally do not benefit them, unlike attention models.

leads to improvements in factuality regardless of model. This is partially expected, considering the range of work that suggest a link between size and the amount of factual knowledge that can be encoded within parametric space. However, faithfulness is different. Recurrent and hybrid architectures tend to saturate at a small size whereas pure attention models do not. For example, Mamba stagnates in faithfulness at around 130M parameter, whereas Pythia continues to increase up to the billions. Finch and RecurrentGemma also exhibit similar behavior. Furthermore, even the smallest Mamba are equally faithful as a 7B LLaMA2 and 1.4B Pythia, indicating a meaningful manner in which their hallucinations may differ. To further illustrate, we show a performance to scale metric, defined as

\[
\text{perf\_to\_scale}(m, \mathcal{M}, \mathcal{B}) = \frac{m(\mathcal{M}) - m(\mathcal{B})}{P(\mathcal{M})/P(\mathcal{B}) - 1}
\]

where \( m(\cdot) \) is a metric of interest, \( \mathcal{M} \) is a model and \( \mathcal{B} \) is a base model of the family as \( \mathcal{M} \). \( P(\cdot) \) counts the number of parameters in a given model. Namely, this measures the relative increase in \( m \) by linear scaling the model, with a lower value indicating that linearly increasing the number of parameters yields no changes while larger values indicate scaling leads to significant improvements. More positive values indicate that the score increases greatly with linear increases in size and negative values the opposite. Consistently negative or near-zero values hence mean that size has little effect on the score. Figure 3 depicts the performance to scale changes for factuality and faithfulness, where it becomes apparent that increasing the number of parameters in hybrid and recurrent models yields a significantly lower increase in faithfulness compared to pure attention models.

This suggests that recurrent layers enable an inherent ability to follow contexts not improved by model size. Their inductive biases can be perceived as encouraging context following, supported by how the hidden state incorporates information from the context with the current input to control the generation. This perspective aligns these results with our instruction-tuning observations, where such models do not exhibit a clear benefit compared to their attention-based counterparts. In particular, we can interpret this as indicating that the base model learns to perform this output control even without direct instruction-tuning; hence applying these techniques is ineffective as we previously observed.

4.4 Controlling for Data/Model Differences

Models are often trained on varying amounts of data, both for pre-training and fine-tuning, while also potentially possessing additional architectural differences in components, such as a channel mixer that exchanges information within each individual token. These can each influence the representation and potentially have a downstream effect on hallucination. To verify whether or not this may be the case, we control for each of these differences in order to quantify the direct effects that exist from replacing attention with sequence layers.

We inspire ourselves from Wang et al. (2024); we start with a pre-trained base model and replace some attention with sequence layers, namely Mamba, while retaining a similar parameter count and keeping the channel mixing Gated MLPs. For fair comparison, we also consider a scenario where the replaced layers are simply re-initialized. These layers are then trained using a standard next-token prediction objective. Specifically, we use a 8B LLaMA3 model and replace/re-initialize 25% or 50% of the attention layers in the model. These layers are then trained, providing us with a synthetically pre-trained model for which both the pre-training data and the channel mixer are controlled for, en-
<table>
  <tr>
    <th>Category</th>
    <th>Baseline (LLaMA3-8B)</th>
    <th>25% Reset</th>
    <th>50% Reset</th>
    <th>25% Replaced</th>
    <th>50% Replaced</th>
  </tr>
  <tr>
    <td>Hallucination Detection</td>
    <td>60.76 (± 10.12)</td>
    <td>50.27 (± 3.24)</td>
    <td>44.25 (± 8.53)</td>
    <td>53.64 (± 0.91)</td>
    <td>53.29 (± 0.33)</td>
  </tr>
  <tr>
    <td>Instruction Following</td>
    <td>22.06 (± 26.76)</td>
    <td>14.29 (± 6.85)</td>
    <td>13.11 (± 6.50)</td>
    <td>20.81 (± -0.92)</td>
    <td>20.18 (± 0.32)</td>
  </tr>
  <tr>
    <td>Closed-Book QA</td>
    <td>41.55 (± 1.30)</td>
    <td>36.79 (± -0.09)</td>
    <td>31.69 (± -0.07)</td>
    <td>36.26 (± -0.42)</td>
    <td>33.74 (± 0.14)</td>
  </tr>
  <tr>
    <td>Reading Comprehension</td>
    <td>33.52 (± 3.62)</td>
    <td>30.87 (± 1.85)</td>
    <td>29.65 (± 2.25)</td>
    <td>32.16 (± 0.51)</td>
    <td>30.68 (± 0.03)</td>
  </tr>
  <tr>
    <td>Summarization</td>
    <td>26.62 (± -1.60)</td>
    <td>21.12 (± 1.06)</td>
    <td>20.86 (± 1.67)</td>
    <td>23.84 (± 0.54)</td>
    <td>22.59 (± 0.79)</td>
  </tr>
  <tr>
    <td>Fact-Checking</td>
    <td>60.84 (± 4.82)</td>
    <td>54.93 (± 2.69)</td>
    <td>50.82 (± 3.17)</td>
    <td>55.45 (± -1.06)</td>
    <td>49.66 (± 2.16)</td>
  </tr>
  <tr>
    <td><b>FAITHFULNESS</b></td>
    <td>37.60 (± 18.09)</td>
    <td>31.40 (± 4.29)</td>
    <td>30.30 (± 4.48)</td>
    <td>35.74 (± 0.17)</td>
    <td>34.32(± 0.13)</td>
  </tr>
  <tr>
    <td><b>FACTUALITY</b></td>
    <td>48.14 (± 1.60)</td>
    <td>43.16 (± 0.83)</td>
    <td>38.98 (± 1.60)</td>
    <td>42.85 (± -0.31)</td>
    <td>39.08(± 0.56)</td>
  </tr>
</table>

Table 2: Controlling for the pre-training and post-training data as well as the presence of a channel mixer. Baseline is a LLaMA3-8B model. Colored values in brackets represent change in category performance after an additional phase of supervised instruction tuning. Note that the SFT dataset for the baseline differs from other models.

abling us to directly compare the effects of the use of sequence layers compared to attention layers. As a next step, we also perform an additional end-to-end instruction tuning. Thus we control for:

(1) The pre-training phase, through the base model and dataset used for re-training.

(2) The instruction fine-tuning process, through the use of the same instruction dataset mix.

(3) Additional architectural components through when they are frozen and trained.

We use a filtered SlimPajama (Soboleva et al., 2023) to re-train layers with a sequence length of 4096. For the instruction tuning phase, we use the same datasets as Wang et al. (2024) (Chen et al., 2024a; BAAI, 2024; Teknium, 2023). We follow hyper-parameters provided by Wang et al. (2024).

Results in Table 2 demonstrate that our previous observations persist. Replacing attention layers with sequence layers and then re-training appears to regain the losses from the initial baseline model, whereas simply resetting the attention layers still appears to observe a noticeable gap. Findings regarding the effects of instruction fine-tuning hold as well, which is that pure-attention models observe a much greater increase in faithfulness relative to factuality, while hybrid models do not.

5 Discussion

Mitigating Hallucinations in LLMs. Facts such as sequence models saturating in faithfulness in a manner not resolved through instruction-tuning indicates that existing techniques for hallucination mitigation can differ in their effectiveness due to a bias towards specific architectures. Current techniques therefore deserve a look, in particular in terms of their effectiveness across the different inductive biases of models. While some methods evidently might be specifically catered to specific models, e.g. using attention (Li et al., 2023b; Zhang et al., 2024; Chuang et al., 2024), other methods may implicitly also possess biases due to the underlying assumptions being made or specific requirements that may be unevenly addressed through different architectures, such as the reliance on data-refinement or additional context (Shi et al., 2024).

Inherent Faithfulness and Factuality. Designing inherently faithful (Herman, 2019; Wiegrefe and Pinter, 2019) or factual models remains an important issue. While much work suggests attention-based LLMs lack an inherent tendency to be faithful (Jacovi and Goldberg, 2020; Wiegrefe and Marasovic, 2021), methods have been designed to render models more interpretable (Madsen et al., 2024b) for specifically evaluating this. Similarly, some of these methods have been shown to also adapt to Mamba models (Sharma et al., 2024). However, while interpretability is useful for evaluating models as being faithful or factual, it does not ensure an inherent tendency towards being either.

6 Conclusion

Are hallucinations the direct result of how token-mixing takes place across time? We study whether different inductive biases of LLM architectures can increase this propensity by observing how hallucinations change across numerous tasks. Patterns emerge demonstrating that some architectural biases can lead to either improvements or degradation in performance compared to others, both for individual tasks and categorizations. Additionally, some model types lead to inherent behaviour when comparing the effects of instruction-tuning and scaling model size. In sum, model-specific induc-
tive biases can have a direct effect on the type of hallucinations they are faced with. However, the data used and the specific model construction can also affect learning. We hope that future work can build on our findings by exploring both the types of hallucinations that can occur and techniques meant to mitigate them and how to design more universal techniques to make models more robust or reliable.

7 Limitations

Evaluating faithfulness and factuality. A major limitation existing in hallucination detection and mitigation research is the lack of metrics that provide explicit information regarding hallucination. While the development of such metrics is an active area of research, limitations still exist, such as biases when involving additional models within the evaluation process.

Limits on tasks. Another limitation of this work is the non-exhaustive set of tasks and domains in which we measure hallucinations. Due to the exhaustive ways in which hallucinations can be measured, we limit ourselves to tasks that are well motivated and frequently used in practice.

8 Ethical Concerns

This paper provides an analysis on the effects of different architectural inductive biases on the propensity to hallucinate within large language models. As such, mistakes in methodology can lead to unsupported confidence or skepticism regarding their performance with the explored task or related ones. While skepticism may not be an ethical issue, unsupported confidence can be problematic. However, the overall message is that all LLMs have specific tasks and settings in which they will display a greater propensity to hallucinate, hence we do not believe that the ideas expressed in this work will explicitly lead to unsupported confidence.

9 Acknowledgements

Jerry Huang was supported by a National Science and Engineering Research Council (NSERC) Canada Graduate Scholarship, a Fonds de Recherche du Québec Nature et technologies (FRQNT) Training Scholarship and a Hydro-Québec Excellence Scholarship. Sarath Chandar is supported by a Canada CIFAR AI Chair, the Canada Research Chair in Lifelong Machine Learning and a NSERC Discovery Grant.

References

AI21. 2024. Introducing jamba: Ai21’s groundbreaking ssm-transformer model.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language models. Preprint, arXiv:2311.16867.

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. 2024. Zoology: Measuring and improving recall in efficient language models. In International Conference on Learning Representations.

Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, Singapore. Association for Computational Linguistics.

BAAI. 2024. Infinity instruct.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.

Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166.

Jiahui Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. 2024a. Genqa: Generating millions of instructions from a handful of prompts. Preprint, arXiv:2406.10323.

Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li. 2024b. Is bigger and deeper always better? probing llama across scales and layers. Preprint, arXiv:2312.04333.

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Beleanger, Lucy J Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In International Conference on Learning Representations.
Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. 2024. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. Preprint, arXiv:2407.07071.

Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to Algorithms, Third Edition, 3rd edition. The MIT Press.

Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning.

Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. Preprint, arXiv:2402.19427.

Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022a. FaithDial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473–1490.

Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022b. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5271–5285. Seattle, United States. Association for Computational Linguistics.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html.

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625–630.

Daniel Y Fu, Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, Armin W Thomas, Benjamin Frederick Spector, Michael Poli, Atri Rudra, and Christopher Re. 2023. Monarch mixer: A simple sub-quadratic GEMM-based architecture. In Thirty-seventh Conference on Neural Information Processing Systems.

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. Preprint, arXiv:2209.07858.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Gemma Team. 2024a. Gemma.

Gemma Team. 2024b. Recurrentgemma: Moving past transformers for efficient open language models. Preprint, arXiv:2404.07839.

Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 2020. Hippo: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, volume 33, pages 1474–1487. Curran Associates, Inc.

Albert Gu, Karan Goel, and Christopher Ré. 2022. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations.

Albert Gu, Isys Johnson, Karan Goel, Khaled Kamal Saab, Tri Dao, Atri Rudra, and Christopher Re. 2021. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems.

Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1059–1075. Dubrovnik, Croatia. Association for Computational Linguistics.

Bernease Herman. 2019. The promise and peril of human evaluation for model interpretability. Preprint, arXiv:1711.07414.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. Preprint, arXiv:2203.15556.

Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-Beltrachini, Max Ryabinin, Xuanli He, Clémentine Fourrier, and Pasquale Minervini. 2024. The hallucinations leaderboard – an open effort to measure hallucinations in large language models. Preprint, arXiv:2404.05904.

Minda Hu, Bowei He, Yufei Wang, Liangyou Li, Chen Ma, and Irwin King. 2024. Mitigating large language model hallucination with faithful finetuning. Preprint, arXiv:2406.11267.

Jerry Huang. 2025. How well can a long sequence model model long sequences? comparing architectural inductive biases on long-context abilities. In Proceedings of the 31st International Conference on Computational Linguistics, pages 29–39, Abu Dhabi, UAE. Association for Computational Linguistics.

Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, and Sarath Chandar. 2024. Towards practical tool usage for continually learning llms. Preprint, arXiv:2404.09339.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. Preprint, arXiv:2311.05232.

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87.

Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics.

Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and eran malach. 2024. Repeat after me: Transformers are better than state space models at copying. In Forty-first International Conference on Machine Learning.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezhen Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023a. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.

Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezhen Yu, Bryan Wile, Min Zeng, and Pascale Fung. 2023b. RHO: Reducing hallucination in open-domain dialogues with knowledge grounding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4504–4522, Toronto, Canada. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. Preprint, arXiv:2401.04088.

Michael I. Jordan. 1986. Serial order: a parallel distributed processing approach. Technical report, University of California, San Diego: Institute for Cognitive Science.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5156–5165. PMLR.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob
Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Linguistics.

Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203–1213, Austin, Texas. Association for Computational Linguistics.

Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a. HaltuEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. Association for Computational Linguistics.

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems.

Yangming Li, Kaisheng Yao, Libo Qin, Wanxiang Che, Xiaolong Li, and Ting Liu. 2020. Slot-consistent NLG for task-oriented dialogue systems with iterative rectification network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 97–106, Online. Association for Computational Linguistics.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.

Alisa Liu and Jiacheng Liu. 2023. The memotrap dataset.

LLaMA Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021a. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021b. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Shahar Lutati, Itamar Zimerman, and Lior Wolf. 2023. Focus your attention (with adaptive IIR filters). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12538–12549, Singapore. Association for Computational Linguistics.

Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024a. Are self-explanations from large language models faithful? Preprint, arXiv:2401.07927.

Andreas Madsen, Siva Reddy, and Sarath Chandar. 2024b. Faithfulness measurable masked language models. In Forty-first International Conference on Machine Learning.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context learning and induction heads. Preprint, arXiv:2209.11895.

OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26670–26698. PMLR.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812–4829, Online. Association for Computational Linguistics.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huaiqin Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzhen He, Haowen Hou, Przemyslaw Kazienko, Jan Kokocin, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanislaw Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. 2023. RWKV: Reinventing RNNs for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14048–14077, Singapore. Association for Computational Linguistics.

Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kokocin, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr. au2, Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanislaw Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. 2024. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. Preprint, arXiv:2404.05892.

Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnuasamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, and Stefano Massaroli. 2024. Mechanistic design and scaling of hybrid architectures. Preprint, arXiv:2403.17844.

Gabriele Prato, Jerry Huang, Prasanna Parthasarathi, Shagun Sodhani, and Sarath Chandar. 2023. EpIK-eval: Evaluation for language models as epistemic models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9523–9557, Singapore. Association for Computational Linguistics.

Gabriele Prato, Jerry Huang, Prasanna Parthasarathi, Shagun Sodhani, and Sarath Chandar. 2024. Do large language models know how much they know? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6054–6070, Miami, Florida, USA. Association for Computational Linguistics.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics.

Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 704–718, Online. Association for Computational Linguistics.

Vipula Rawte, Swagata Chakraborty, Agnihb Pathak, Anubhav Sarkar, S.M Towhidul Islam Tomoy, Aman Chadha, Amit Sheth, and Amitava Das. 2023. The troubling emergence of hallucination in large language models - an extensive definition, quantification, and prescriptive remediations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2541–2573, Singapore. Association for Computational Linguistics.

Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401.

David E. Rumelhart, James L. McClelland, and PDP Research Group. 1986. Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations. The MIT Press.

Sashank Santhanam, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2022. Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation. Preprint, arXiv:2110.05456.

Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, and Markus Leippold. 2024. Towards faithful and robust LLM specialists for evidence-based question-answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1913–1931, Bangkok, Thailand. Association for Computational Linguistics.

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.

Arnab Sen Sharma, David Atkinson, and David Bau. 2024. Locating and editing factual associations in mamba. In First Conference on Language Modeling.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning.

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783–791, Mexico City, Mexico. Association for Computational Linguistics.

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. 2024. Learning to (learn at test time): Rnns with expressive hidden states. Preprint, arXiv:2407.04620.

Teknium. 2023. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.

Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2020. Sticking to the facts: Confident decoding for faithful data-to-text generation. Preprint, arXiv:1910.08684.

Ali Vardasbi, Telmo Pessoa Pires, Robin Schmidt, and Stephan Peitz. 2023. State spaces aren’t enough: Machine translation needs attention. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 205–216, Tampere, Finland. European Association for Machine Translation.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. 2024. The mamba in the llama: Distilling and accelerating hybrid models. In Thirty-eighth Annual Conference on Neural Information Processing Systems.

Xinyu Wang, Linrui Ma, Jerry Huang, Peng Lu, Prasanna Parthasarathi, Xiao-Wen Chang, Boxing Chen, and Yufei Cui. 2025. Resona: Improving context copying in linear recurrence models with retrieval. Preprint, arXiv:2503.22913.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification.

Jiaheng Wei, Yuanshun Yao, Jean-Francois Ton, Hongyi Guo, Andrew Estornell, and Yang Liu. 2024. Measuring and reducing llm hallucination without gold-standard answers. Preprint, arXiv:2402.10412.

Sarah Wiegreffe and Ana Marasović. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).

Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for Computational Linguistics.

Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.

Qiuqiao Zeng, Jerry Huang, Peng Lu, Gezheng Xu, Boxing Chen, Charles Ling, and Boyu Wang. 2025. ZETA: Leveraging $k$-order curves for efficient top-$k$ attention. In The Thirteenth International Conference on Learning Representations.

Shaolei Zhang, Tian Yu, and Yang Feng. 2024. TruthX: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8908–8949, Bangkok, Thailand. Association for Computational Linguistics.

Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, Online. Association for Computational Linguistics.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.
A  Technical Implementation Details

A.1 Models and Datasets
All models and datasets used in this paper are public and directly available on the HuggingFace Hub.

A.2 Experimental Setup
Experiments were run using the Language Model Evaluation Harness (Gao et al., 2024).

A.3 Datasets
We evaluate on the Hallucination Leaderboard (Hong et al., 2024), consisting of tasks

1) Closed-book Open-domain QA*: NQ-OPEN (Kwiatkowski et al., 2019), TRIVIAQA (Joshi et al., 2017), TRUTHFULQA (MC1, MC2, Generative) (Lin et al., 2022), POPQA (Mallen et al., 2023)
2) Summarization*: XSUM (Narayan et al., 2018), CNN/DM (See et al., 2017)
3) Reading Comprehension*: RACE (Lai et al., 2017), SQUADv2 (Rajpurkar et al., 2018), NQ-SWAP (Longpre et al., 2021b)
4) Instruction Following*: MEMOTrap (Liu and Liu, 2023), IFEVAL (Zhou et al., 2023)
5) Hallucination Detection: FAITHDIAL* (Dziri et al., 2022a), HALUEVAL* (QA, Summarization, Dialogue) (Li et al., 2023a), TRUEFALSE* (Azaria and Mitchell, 2023)
6) Fact Checking*: FEVER (Thorne et al., 2018)

For tasks, a higher score (ranging from 0 to 100) indicates better performance. These are

• EM: HALUEVAL (all sets), POPQA, TRUTHFULQA (MC1 and MC2), SQUADv2, NATURALQUESTIONS (NQ-OPEN), TRIVIAQA
• Accuracy: MEMOTrap, TRUTHFULQA (Gen), RACE, IfEVAL
• Rouge-L: CNNDM, XSUM, FAITHDIAL, TRUEFALSE, FEVER10

Tasks are further divided into two categories: faithfulness (\(\blacklozenge\)) hallucinations, i.e. whether an LLM generation adheres to the given source of information, and factuality (\(\blacktriangle\)) hallucinations, i.e. whether LLMs generate factually correct content according to world knowledge based on knowledge acquired during training. To compute scores across such categories, the scores for each task in the category are averaged.

A.4 Models and Baselines
For our comparison, we use various models (Table 3) that vary in size and architecture, with a particular focus on different time-mixing methods. These include:

• All Pythia models
• All publicly available Mamba (Gu and Dao, 2024; Dao and Gu, 2024) models
• All LLaMA2 and LLaMA3 (LLaMA Team, 2024) models
• Falcon-7B (Almazrouei et al., 2023) and FalconMamba-7B
• Mistral-7B (Jiang et al., 2023)
• All Gemma (Gemma Team, 2024a) models
• RWKV (Peng et al., 2023) models, namely Finch (RWKV-v6)
• RecurrentGemma (Gemma Team, 2024b)
• Mixtral-8x7B (Jiang et al., 2024)
• Jamba (AI21, 2024)

For models with instruction fine-tuned versions, we use both the base version as well as the instruction fine-tuned variant.

A.5 Computing Resources Used
All results were obtained using a server of 8 NVIDIA V100 32GB or 4 NVIDIA RTX A6000 48GB GPUs. The accelerate package was used for model sharding in instances where a single GPU was insufficient to store the entire model.

B  Complete Results
<table>
  <tr>
    <th>Model</th>
    <th>Public Link</th>
    <th>HuggingFace Model</th>
  </tr>
  <tr>
    <td>Pythia-160M</td>
    <td>EleutherAI/pythia-160m</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pythia-410M</td>
    <td>EleutherAI/pythia-410m</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pythia-1B</td>
    <td>EleutherAI/pythia-1b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pythia-1.4B</td>
    <td>EleutherAI/pythia-1.4b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pythia-2.8B</td>
    <td>EleutherAI/pythia-2.8b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pythia-6.9B</td>
    <td>EleutherAI/pythia-6.9b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pythia-12B</td>
    <td>EleutherAI/pythia-12b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mamba-130M</td>
    <td>state-spaces/mamba-130m</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba-370M</td>
    <td>state-spaces/mamba-370m</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba-790M</td>
    <td>state-spaces/mamba-790m</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba-1.4B</td>
    <td>state-spaces/mamba-1.4b</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba-2.8B</td>
    <td>state-spaces/mamba-2.8b</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba2-130M</td>
    <td>state-spaces/mamba2-130m</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba2-370M</td>
    <td>state-spaces/mamba2-370m</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba2-780M</td>
    <td>state-spaces/mamba2-780m</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba2-1.3B</td>
    <td>state-spaces/mamba2-1.3b</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba2-2.7B</td>
    <td>state-spaces/mamba2-2.7b</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Mamba2Attention-2.7B</td>
    <td>state-spaces/mamba2attn-2.7b</td>
    <td>×</td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B</td>
    <td>google/recurrentgemma-2b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B (IT)</td>
    <td>google/recurrentgemma-2b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B</td>
    <td>google/recurrentgemma-9b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B (IT)</td>
    <td>google/recurrentgemma-9b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma-2B</td>
    <td>google/gemma-2b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma-2B (IT)</td>
    <td>google/gemma-2b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma-9B</td>
    <td>google/gemma-9b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma-9B (IT)</td>
    <td>google/gemma-9b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma2-2B</td>
    <td>google/gemma2-2b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma2-2B (IT)</td>
    <td>google/gemma2-2b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>google/gemma2-9b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma2-9B (IT)</td>
    <td>google/gemma2-9b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>google/gemma2-27b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Gemma2-27B (IT)</td>
    <td>google/gemma2-27b-it</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Falcon-7B</td>
    <td>tiiuae/falcon-7b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Falcon-7B (IT)</td>
    <td>tiiuae/falcon-7b-instruct</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>FalconMamba-7B</td>
    <td>tiiuae/falcon-mamba-7b</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>FalconMamba-7B (IT)</td>
    <td>tiiuae/falcon-mamba-7b-instruct</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>mistralai/Mistral-7B-v0.3</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mistral-7B (IT)</td>
    <td>mistralai/Mistral-7B-Instruct-v0.3</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mistral-8x7B</td>
    <td>mistralai/Mistral-8x7B-v0.1</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mistral-8x7B (IT)</td>
    <td>mistralai/Mistral-8x7B-Instruct-v0.1</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Jamba</td>
    <td>ai21labs/Jamba-v0.1</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Jamba (IT)</td>
    <td>ai21labs/AI21-Jamba-1.5-Mini</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>meta-llama/Llama-2-7b-hf</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA2-7B (IT)</td>
    <td>meta-llama/Llama-2-7b-hf</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA2-13B</td>
    <td>meta-llama/Llama-2-13b-hf</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA2-13B (IT)</td>
    <td>meta-llama/Llama-2-13b-hf</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA2-70B</td>
    <td>meta-llama/Llama-2-70b-hf</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA2-70B (IT)</td>
    <td>meta-llama/Llama-2-70b-hf</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>https://huggingface.co/meta-llama/Meta-Llama-3-8B</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA3-8B (IT)</td>
    <td>https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>https://huggingface.co/meta-llama/Meta-Llama-3-70B</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LLaMA3-70B (IT)</td>
    <td>https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct</td>
    <td>✓</td>
  </tr>
</table>

Table 3: Models used and public links to their weights.
Figure 4: Performance of various base models on tasks within the Hallucination Leaderboard.

<table>
  <tr>
    <th>Name</th>
    <th>hilu±Oul</th>
    <th>hilu+OA</th>
    <th>Syn</th>
    <th>Mente-TraP</th>
    <th>PPOA</th>
    <th>Turbo4QA-Gan</th>
    <th>Turbo4QA-MC1</th>
    <th>Turbo4QA-MC2</th>
    <th>R4CE</th>
    <th>Pytha</th>
    <th>Vcw-Q1</th>
    <th>Kim</th>
    <th>Rejo</th>
    <th>Squad-v2</th>
    <th>HF@hid</th>
    <th>TrunGUI@hid</th>
    <th>FafMsI-10</th>
    <th>NQ@hid(hid)</th>
    <th>TrunkUI4C(random)</th>
    <th>Gefex-QI@Ack</th>
    <th>Bass</th>
    <th>FA%</th>
    <th>KP</th>
    <th>NH+CM</th>
    <th>Gefex-QI@hdf5</th>
    <th>Hrls/Spr</th>
    <th>Selscen</th>
    <th>Kd%</th>
    <th>Bass</th>
    <th>NH+</th>
    <th>KP</th>
    <th>NH,NoFM</th>
  </tr>
  <tr>
    <td>Falcon-7B</td>
    <td>41.8</td>
    <td>46.6</td>
    <td>49.0</td>
    <td>38.9</td>
    <td>17.5</td>
    <td>25.0</td>
    <td>22.4</td>
    <td>34.3</td>
    <td>37.4</td>
    <td>12.2</td>
    <td>20.6</td>
    <td>20.8</td>
    <td>26.0</td>
    <td>22.9</td>
    <td>58.7</td>
    <td>66.4</td>
    <td>58.1</td>
    <td>46.9</td>
    <td>24.3</td>
    <td>59.1</td>
  </tr>
  <tr>
    <td>FalconMamba-7B</td>
    <td>49.8</td>
    <td>39.5</td>
    <td>42.5</td>
    <td>58.0</td>
    <td>0.7</td>
    <td>47.2</td>
    <td>52.4</td>
    <td>43.0</td>
    <td>27.2</td>
    <td>25.1</td>
    <td>27.7</td>
    <td>24.4</td>
    <td>26.0</td>
    <td>44.1</td>
    <td>58.2</td>
    <td>88.8</td>
    <td>61.8</td>
    <td>27.8</td>
    <td>60.4</td>
  </tr>
  <tr>
    <td>Finch-6.2</td>
    <td>42.1</td>
    <td>40.4</td>
    <td>50.3</td>
    <td>45.9</td>
    <td>1.6</td>
    <td>26.1</td>
    <td>22.2</td>
    <td>34.7</td>
    <td>35.6</td>
    <td>16.3</td>
    <td>6.4</td>
    <td>1.6</td>
    <td>14.9</td>
    <td>11.0</td>
    <td>10.3</td>
    <td>56.7</td>
    <td>63.1</td>
    <td>31.5</td>
    <td>39.5</td>
    <td>13.0</td>
    <td>37.0</td>
  </tr>
  <tr>
    <td>Finch-3E</td>
    <td>44.6</td>
    <td>48.7</td>
    <td>53.3</td>
    <td>38.8</td>
    <td>0.0</td>
    <td>27.1</td>
    <td>23.6</td>
    <td>37.0</td>
    <td>35.3</td>
    <td>16.8</td>
    <td>0.0</td>
    <td>0.1</td>
    <td>18.1</td>
    <td>11.9</td>
    <td>45.6</td>
    <td>44.3</td>
    <td>54.4</td>
    <td>37.9</td>
    <td>14.6</td>
    <td>45.9</td>
  </tr>
  <tr>
    <td>Finch-7B</td>
    <td>52.5</td>
    <td>49.4</td>
    <td>48.6</td>
    <td>32.1</td>
    <td>3.4</td>
    <td>38.7</td>
    <td>27.4</td>
    <td>42.2</td>
    <td>35.5</td>
    <td>19.0</td>
    <td>0.7</td>
    <td>10.2</td>
    <td>27.3</td>
    <td>21.5</td>
    <td>60.0</td>
    <td>60.4</td>
    <td>44.6</td>
    <td>43.3</td>
    <td>23.2</td>
    <td>60.2</td>
  </tr>
  <tr>
    <td>Finch-14B</td>
    <td>61.5</td>
    <td>57.2</td>
    <td>50.2</td>
    <td>32.5</td>
    <td>4.4</td>
    <td>40.8</td>
    <td>31.6</td>
    <td>45.4</td>
    <td>34.5</td>
    <td>20.7</td>
    <td>0.3</td>
    <td>5.6</td>
    <td>21.6</td>
    <td>249</td>
    <td>63.5</td>
    <td>64.2</td>
    <td>74.8</td>
    <td>52.0</td>
    <td>25.7</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td>Gemma-28</td>
    <td>41.7</td>
    <td>47.5</td>
    <td>45.1</td>
    <td>47.7</td>
    <td>13.3</td>
    <td>25.5</td>
    <td>23.0</td>
    <td>30.5</td>
    <td>14.0</td>
    <td>23.9</td>
    <td>18.3</td>
    <td>24.8</td>
    <td>14.7</td>
    <td>47.8</td>
    <td>54.5</td>
    <td>49.3</td>
    <td>16.2</td>
    <td>16.7</td>
    <td>16.2</td>
    <td>46.1</td>
  </tr>
  <tr>
    <td>Gemma-37B</td>
    <td>54.4</td>
    <td>26.8</td>
    <td>28.3</td>
    <td>41.2</td>
    <td>18.2</td>
    <td>42.0</td>
    <td>31.6</td>
    <td>45.1</td>
    <td>41.5</td>
    <td>23.1</td>
    <td>25.6</td>
    <td>22.2</td>
    <td>30.3</td>
    <td>28.3</td>
    <td>63.1</td>
    <td>37.4</td>
    <td>52.1</td>
    <td>38.5</td>
    <td>25.5</td>
    <td>55.4</td>
  </tr>
  <tr>
    <td>Gemma2-28</td>
    <td>46.9</td>
    <td>50.9</td>
    <td>41.5</td>
    <td>43.2</td>
    <td>14.6</td>
    <td>22.2</td>
    <td>24.1</td>
    <td>36.2</td>
    <td>37.3</td>
    <td>17.5</td>
    <td>11.7</td>
    <td>2.6</td>
    <td>17.7</td>
    <td>2.2</td>
    <td>69.1</td>
    <td>37.3</td>
    <td>18.3</td>
    <td>20.2</td>
    <td>19.1</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td>Gemma2-7B</td>
    <td>48.6</td>
    <td>57.7</td>
    <td>37.0</td>
    <td>34.4</td>
    <td>21.3</td>
    <td>51.6</td>
    <td>43.6</td>
    <td>42.0</td>
    <td>13.1</td>
    <td>17.0</td>
    <td>25.7</td>
    <td>37.1</td>
    <td>6.6</td>
    <td>69.7</td>
    <td>91.6</td>
    <td>62.1</td>
    <td>25.5</td>
    <td>63.0</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>75.9</td>
    <td>24.2</td>
    <td>41.5</td>
    <td>27.4</td>
    <td>8.2</td>
    <td>40.9</td>
    <td>33.3</td>
    <td>47.8</td>
    <td>42.5</td>
    <td>17.6</td>
    <td>25.2</td>
    <td>12.6</td>
    <td>18.5</td>
    <td>39.8</td>
    <td>78.4</td>
    <td>66.1</td>
    <td>63.4</td>
    <td>88.3</td>
    <td>39.6</td>
    <td>77.1</td>
  </tr>
  <tr>
    <td>RecurrentGemm-28</td>
    <td>49.9</td>
    <td>35.9</td>
    <td>47.2</td>
    <td>54.7</td>
    <td>7.6</td>
    <td>25.7</td>
    <td>21.1</td>
    <td>35.1</td>
    <td>37.9</td>
    <td>14.1</td>
    <td>17.3</td>
    <td>27.3</td>
    <td>11.4</td>
    <td>47.1</td>
    <td>67.5</td>
    <td>69.1</td>
    <td>38.4</td>
    <td>46.3</td>
    <td>31.1</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>RecurrentGemm-98</td>
    <td>50.0</td>
    <td>38.1</td>
    <td>44.9</td>
    <td>42.4</td>
    <td>16.0</td>
    <td>39.8</td>
    <td>26.3</td>
    <td>38.3</td>
    <td>37.3</td>
    <td>19.0</td>
    <td>22.3</td>
    <td>31.2</td>
    <td>8.4</td>
    <td>45.9</td>
    <td>55.2</td>
    <td>57.2</td>
    <td>38.7</td>
    <td>66.5</td>
    <td>28.8</td>
    <td>68.8</td>
  </tr>
  <tr>
    <td>Jamba</td>
    <td>56.0</td>
    <td>30.7</td>
    <td>50.0</td>
    <td>6.9</td>
    <td>0.4</td>
    <td>59.9</td>
    <td>28.8</td>
    <td>45.1</td>
    <td>19.6</td>
    <td>3.7</td>
    <td>(case replacement)</td>
    <td>375.9</td>
    <td>72.4</td>
    <td>73.5</td>
    <td>38.2</td>
    <td>73.6</td>
    <td>34.2</td>
    <td>55.5</td>
    <td>39.4</td>
    <td>70.9</td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>50.6</td>
    <td>2.6</td>
    <td>64.0</td>
    <td>36.2</td>
    <td>0.7</td>
    <td>47.6</td>
    <td>25.1</td>
    <td>59.0</td>
    <td>19.7</td>
    <td>23.7</td>
    <td>26.7</td>
    <td>4.6</td>
    <td>64.6</td>
    <td>54.1</td>
    <td>7.1</td>
    <td>42.0</td>
    <td>29.0</td>
    <td>54.1</td>
    <td>29.2</td>
    <td>72.7</td>
  </tr>
  <tr>
    <td>LLaMA2-139</td>
    <td>76.1</td>
    <td>24.5</td>
    <td>43.9</td>
    <td>28.5</td>
    <td>24.9</td>
    <td>26.4</td>
    <td>16.9</td>
    <td>18.2</td>
    <td>34.5</td>
    <td>51.0</td>
    <td>30.7</td>
    <td>1.8</td>
    <td>84.8</td>
    <td>92.8</td>
    <td>52.3</td>
    <td>38.2</td>
    <td>65.6</td>
    <td>38.7</td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>LLaMA2-70B</td>
    <td>69.7</td>
    <td>31.4</td>
    <td>46.7</td>
    <td>13.0</td>
    <td>9.2</td>
    <td>29.6</td>
    <td>24.9</td>
    <td>39.0</td>
    <td>16.7</td>
    <td>21.6</td>
    <td>24.9</td>
    <td>7.7</td>
    <td>(case replacement)</td>
    <td>31.9</td>
    <td>37.4</td>
    <td>80.1</td>
    <td>67.4</td>
    <td>76.7</td>
    <td>81.5</td>
    <td>40.8</td>
    <td>77.9</td>
  </tr>
  <tr>
    <td>LLaMA3-BB</td>
    <td>66.2</td>
    <td>34.0</td>
    <td>46.9</td>
    <td>13.8</td>
    <td>41.1</td>
    <td>37.1</td>
    <td>7.2</td>
    <td>22.7</td>
    <td>21.5</td>
    <td>7.9</td>
    <td>1.4</td>
    <td>3.1</td>
    <td>81.3</td>
    <td>92.8</td>
    <td>52.7</td>
    <td>30.3</td>
    <td>57.8</td>
    <td>17.6</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>77.0</td>
    <td>20.5</td>
    <td>51.9</td>
    <td>24.4</td>
    <td>16.6</td>
    <td>20.2</td>
    <td>21.9</td>
    <td>52.4</td>
    <td>35.1</td>
    <td>12.2</td>
    <td>20.7</td>
    <td>19.9</td>
    <td>17.7</td>
    <td>81.1</td>
    <td>61.1</td>
    <td>68.6</td>
    <td>94.0</td>
    <td>77.7</td>
    <td>48.8</td>
    <td>97.4</td>
  </tr>
  <tr>
    <td>Mamba-1030M</td>
    <td>41.89</td>
    <td>2.1</td>
    <td>54.7</td>
    <td>20.6</td>
    <td>31.2</td>
    <td>52.3</td>
    <td>5.1</td>
    <td>29.0</td>
    <td>34.1</td>
    <td>82.4</td>
    <td>50.0</td>
    <td>13.6</td>
    <td>79.5</td>
    <td>53.3</td>
    <td>391</td>
    <td>79.9</td>
    <td>97.4</td>
    <td>92.4</td>
    <td>80.7</td>
    <td>93.1</td>
  </tr>
  <tr>
    <td>Mamba-370M</td>
    <td>47.3</td>
    <td>2.8</td>
    <td>45.3</td>
    <td>43.5</td>
    <td>29.3</td>
    <td>57.6</td>
    <td>17.4</td>
    <td>18.2</td>
    <td>31.7</td>
    <td>19.7</td>
    <td>1.4</td>
    <td>20.4</td>
    <td>16.3</td>
    <td>11.3</td>
    <td>67.9</td>
    <td>81.1</td>
    <td>91.0</td>
    <td>92.8</td>
    <td>15.0</td>
    <td>214.1</td>
  </tr>
  <tr>
    <td>Mamba2ATM-77B</td>
    <td>41.6</td>
    <td>37.0</td>
    <td>49.3</td>
    <td>31.2</td>
    <td>26.2</td>
    <td>56.2</td>
    <td>32.0</td>
    <td>31.7</td>
    <td>21.7</td>
    <td>10.7</td>
    <td>18.7</td>
    <td>7.5</td>
    <td>20.8</td>
    <td>79.5</td>
    <td>53.0</td>
    <td>57.8</td>
    <td>87.9</td>
    <td>16.8</td>
    <td>69.7</td>
    <td>105.0</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>57.3</td>
    <td>19.6</td>
    <td>40.8</td>
    <td>38.7</td>
    <td>
35
</td>
    <td>42.5</td>
    <td>9.8</td>
    <td>15.7</td>
    <td>12.6</td>
    <td>17.5</td>
    <td>30.2</td>
    <td>15.1</td>
    <td>40.1</td>
    <td>67.5</td>
    <td>49.7</td>
    <td>39.2</td>
    <td>16.5</td>
    <td>36.7</td>
    <td>72.6</td>
    <td>105.1</td>
  </tr>
  <tr>
    <td>MixtAldR9XB-68B</td>
    <td>66.6</td>
    <td>69.5</td>
    <td>77.3</td>
    <td>33.1</td>
    <td>39.3</td>
    <td>45.4</td>
    <td>48.6</td>
    <td>39.9</td>
    <td>17.5</td>
    <td>25.5</td>
    <td>20.3</td>
    <td>19.7</td>
    <td>11.8</td>
    <td>69.8</td>
    <td>81.9</td>
    <td>90.1</td>
    <td>3.9</td>
    <td>32.7</td>
    <td>49.1</td>
    <td>70.3</td>
  </tr>
  <tr>
    <td>Pythia-1460M</td>
    <td>51.6</td>
    <td>45.3</td>
    <td>34.1</td>
    <td>74.2</td>
    <td>0.0</td>
    <td>52.5</td>
    <td>24.7</td>
    <td>46.5</td>
    <td>27.0</td>
    <td>8.7</td>
    <td>3.3</td>
    <td>27</td>
    <td>4.1</td>
    <td>0.1</td>
    <td>3.0</td>
    <td>50.3</td>
    <td>60.9</td>
    <td>0.1</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Pythia-2040M</td>
    <td>51.</td>
    <td>47.0</td>
    <td>54.9</td>
    <td>49.0</td>
    <td>35.7</td>
    <td>40.2</td>
    <td>30.2</td>
    <td>23.3</td>
    <td>27.1</td>
    <td>1.3</td>
    <td>18.7</td>
    <td>21.6</td>
    <td>26.0</td>
    <td>27.7</td>
    <td>20.4</td>
    <td>50.6</td>
    <td>42.9</td>
    <td>22.6</td>
    <td>11.4</td>
    <td>36.9</td>
  </tr>
  <tr>
    <td>Pythia-3140M</td>
    <td>51.7</td>
    <td>48.4</td>
    <td>48.9</td>
    <td>49.3</td>
    <td>25</td>
    <td>27.0</td>
    <td>24.0</td>
    <td>15.8</td>
    <td>21.4</td>
    <td>0.3</td>
    <td>9.6</td>
    <td>1.8</td>
    <td>47.2</td>
    <td>81.1</td>
    <td>61.2</td>
    <td>52.0</td>
    <td>28.8</td>
    <td>14.7</td>
    <td>19.4</td>
    <td>17.4</td>
  </tr>
  <tr>
    <td>Pythia-21-8K</td>
    <td>53.8</td>
    <td>48.5</td>
    <td>51.6</td>
    <td>47.7</td>
    <td>9.6</td>
    <td>25.9</td>
    <td>22.9</td>
    <td>34.4</td>
    <td>16.3</td>
    <td>13.2</td>
    <td>5.2</td>
    <td>18.7</td>
    <td>7.1</td>
    <td>7.1</td>
    <td>5.4</td>
    <td>52.9</td>
    <td>39.2</td>
    <td>6.4</td>
    <td>20.3</td>
  </tr>
  <tr>
    <td>Pythia-2199</td>
    <td>50.0</td>
    <td>42.4</td>
    <td>51.4</td>
    <td>6.2</td>
    <td>25.5</td>
    <td>43.7</td>
    <td>3.7</td>
    <td>8.3</td>
    <td>50.6</td>
    <td>21.2</td>
    <td>8.8</td>
    <td>34.4</td>
    <td>70.6</td>
    <td>44.4</td>
    <td>9.1</td>
    <td>29.7</td>
    <td>459</td>
    <td>14.7</td>
    <td>21.9</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>Pythia-699</td>
    <td>49.1</td>
    <td>48.7</td>
    <td>45.0</td>
    <td>47.2</td>
    <td>5.2</td>
    <td>25.7</td>
    <td>21.7</td>
    <td>34.3</td>
    <td>27.6</td>
    <td>18.4</td>
    <td>12.8</td>
    <td>9.6</td>
    <td>34.4</td>
    <td>70.4</td>
    <td>31.9</td>
    <td>42.9</td>
    <td>11.4</td>
    <td>13.6</td>
    <td>6.2</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>D_one-L12B</td>
    <td>49.7</td>
    <td>45.3</td>
    <td>34.9</td>
    <td>42.4</td>
    <td>7.7</td>
    <td>25.7</td>
    <td>20.4</td>
    <td>38.0</td>
    <td>18.1</td>
    <td>22.7</td>
    <td>38.8</td>
    <td>7.6</td>
    <td>21.9</td>
    <td>1.0</td>
    <td>0.2</td>
    <td>5.8</td>
    <td>3.7</td>
    <td>8.9</td>
    <td>6.3</td>
    <td>8.1</td>
  </tr>
</table>
<table>
  <tr>
    <th>Model</th>
    <th>Instruction Model</th>
    <th>Instruction Split</th>
    <th>Instruction Size</th>
    <th>Minibatch Size</th>
    <th>Epochs</th>
    <th>Learning Rate</th>
    <th>Learning Rate Decay</th>
    <th>Learning Rate Schedule</th>
    <th>Batch Norm</th>
    <th>Dropout</th>
    <th>Weight Decay</th>
    <th>Regularization</th>
    <th>Regularization Weight</th>
    <th>Regularization Type</th>
    <th>Regularization Args</th>
    <th>Reg.</th>
  </tr>
  <!-- Table data omitted for brevity -->
</table>

Table 4: Complete results on individual tasks.

Figure 5: Change in task performance from base model to instruction fine-tuned model, for all tasks.

![Heatmap showing change in task performance from base model to instruction fine-tuned model, for all tasks.](page_128_1347_1392_627.png)
<table>
  <tr>
    <th></th>
    <th>Hallu.</th>
    <th>Detection</th>
    <th>Instr.</th>
    <th>Following</th>
    <th>QA</th>
    <th>Reading Comp.</th>
    <th>Sum.</th>
    <th>Fact-Checking</th>
  </tr>
  <tr>
    <td>Falcon-7B</td>
    <td>52.40</td>
    <td>25.54</td>
    <td>33.03</td>
    <td>29.18</td>
    <td>20.70</td>
    <td>46.91</td>
  </tr>
  <tr>
    <td>Falcon-7B (IT)</td>
    <td>49.50</td>
    <td>38.16</td>
    <td>28.61</td>
    <td>25.54</td>
    <td>19.70</td>
    <td>39.20</td>
  </tr>
  <tr>
    <td>FalconMamba-7B</td>
    <td>55.80</td>
    <td>42.97</td>
    <td>39.94</td>
    <td>23.76</td>
    <td>23.89</td>
    <td>61.85</td>
  </tr>
  <tr>
    <td>FalconMamba-7B (IT)</td>
    <td>56.53</td>
    <td>41.78</td>
    <td>39.51</td>
    <td>22.91</td>
    <td>23.80</td>
    <td>60.05</td>
  </tr>
  <tr>
    <td>Gemma-2B</td>
    <td>53.01</td>
    <td>30.94</td>
    <td>27.43</td>
    <td>30.35</td>
    <td>19.63</td>
    <td>43.28</td>
  </tr>
  <tr>
    <td>Gemma-7B</td>
    <td>50.45</td>
    <td>31.25</td>
    <td>38.20</td>
    <td>39.96</td>
    <td>25.93</td>
    <td>54.14</td>
  </tr>
  <tr>
    <td>Gemma-2B (IT)</td>
    <td>49.74</td>
    <td>40.73</td>
    <td>23.30</td>
    <td>27.52</td>
    <td>23.67</td>
    <td>38.06</td>
  </tr>
  <tr>
    <td>Gemma-7B (IT)</td>
    <td>61.16</td>
    <td>44.09</td>
    <td>31.28</td>
    <td>25.00</td>
    <td>20.40</td>
    <td>55.74</td>
  </tr>
  <tr>
    <td>Gemma1.1-2B (IT)</td>
    <td>50.61</td>
    <td>41.27</td>
    <td>25.43</td>
    <td>29.38</td>
    <td>21.00</td>
    <td>34.65</td>
  </tr>
  <tr>
    <td>Gemma1.1-7B (IT)</td>
    <td>55.27</td>
    <td>46.50</td>
    <td>36.66</td>
    <td>26.63</td>
    <td>22.32</td>
    <td>58.96</td>
  </tr>
  <tr>
    <td>Gemma2-2B</td>
    <td>58.33</td>
    <td>26.73</td>
    <td>29.59</td>
    <td>32.95</td>
    <td>16.04</td>
    <td>39.57</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>65.64</td>
    <td>23.76</td>
    <td>39.36</td>
    <td>39.74</td>
    <td>21.38</td>
    <td>62.06</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>62.03</td>
    <td>26.62</td>
    <td>47.19</td>
    <td>43.04</td>
    <td>28.92</td>
    <td>68.28</td>
  </tr>
  <tr>
    <td>Gemma2-2B (IT)</td>
    <td>58.97</td>
    <td>37.63</td>
    <td>34.42</td>
    <td>27.69</td>
    <td>15.57</td>
    <td>58.53</td>
  </tr>
  <tr>
    <td>Gemma2-9B (IT)</td>
    <td>71.95</td>
    <td>53.83</td>
    <td>42.49</td>
    <td>39.04</td>
    <td>18.03</td>
    <td>68.53</td>
  </tr>
  <tr>
    <td>Gemma2-27B (IT)</td>
    <td>74.46</td>
    <td>60.20</td>
    <td>50.05</td>
    <td>48.06</td>
    <td>28.18</td>
    <td>69.28</td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B</td>
    <td>53.68</td>
    <td>34.43</td>
    <td>27.36</td>
    <td>31.66</td>
    <td>19.41</td>
    <td>42.97</td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B</td>
    <td>54.95</td>
    <td>30.67</td>
    <td>36.79</td>
    <td>35.57</td>
    <td>21.99</td>
    <td>51.25</td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B (IT)</td>
    <td>55.21</td>
    <td>36.55</td>
    <td>25.51</td>
    <td>28.89</td>
    <td>21.27</td>
    <td>46.60</td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B (IT)</td>
    <td>53.28</td>
    <td>43.63</td>
    <td>38.82</td>
    <td>28.91</td>
    <td>23.83</td>
    <td>64.61</td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>53.63</td>
    <td>28.94</td>
    <td>37.64</td>
    <td>27.58</td>
    <td>25.19</td>
    <td>51.38</td>
  </tr>
  <tr>
    <td>LLaMA2-13B</td>
    <td>67.00</td>
    <td>25.57</td>
    <td>39.65</td>
    <td>31.17</td>
    <td>26.33</td>
    <td>62.35</td>
  </tr>
  <tr>
    <td>LLaMA2-70B</td>
    <td>62.34</td>
    <td>30.24</td>
    <td>47.85</td>
    <td>39.48</td>
    <td>28.00</td>
    <td>66.63</td>
  </tr>
  <tr>
    <td>LLaMA2-7B (IT)</td>
    <td>57.93</td>
    <td>37.93</td>
    <td>35.75</td>
    <td>31.32</td>
    <td>25.88</td>
    <td>57.26</td>
  </tr>
  <tr>
    <td>LLaMA2-13B (IT)</td>
    <td>64.92</td>
    <td>37.29</td>
    <td>39.00</td>
    <td>29.53</td>
    <td>28.10</td>
    <td>60.71</td>
  </tr>
  <tr>
    <td>LLaMA2-70B (IT)</td>
    <td>73.12</td>
    <td>47.35</td>
    <td>45.32</td>
    <td>31.27</td>
    <td>27.45</td>
    <td>65.18</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>60.76</td>
    <td>22.06</td>
    <td>41.55</td>
    <td>33.52</td>
    <td>26.62</td>
    <td>60.84</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>71.78</td>
    <td>21.59</td>
    <td>48.07</td>
    <td>46.38</td>
    <td>28.91</td>
    <td>69.57</td>
  </tr>
  <tr>
    <td>LLaMA3-8B (IT)</td>
    <td>70.88</td>
    <td>48.82</td>
    <td>42.85</td>
    <td>37.14</td>
    <td>25.02</td>
    <td>65.66</td>
  </tr>
  <tr>
    <td>LLaMA3-70B (IT)</td>
    <td>80.47</td>
    <td>47.19</td>
    <td>51.50</td>
    <td>38.65</td>
    <td>27.23</td>
    <td>70.56</td>
  </tr>
  <tr>
    <td>Jamba</td>
    <td>57.66</td>
    <td>43.36</td>
    <td>39.50</td>
    <td>33.00</td>
    <td>23.72</td>
    <td>59.88</td>
  </tr>
  <tr>
    <td>Jamba (IT)</td>
    <td>54.76</td>
    <td>37.60</td>
    <td>38.52</td>
    <td>27.12</td>
    <td>8.24</td>
    <td>57.28</td>
  </tr>
  <tr>
    <td>Finch-1.6B</td>
    <td>53.09</td>
    <td>31.10</td>
    <td>22.69</td>
    <td>16.60</td>
    <td>11.77</td>
    <td>39.47</td>
  </tr>
  <tr>
    <td>Finch-3B</td>
    <td>53.29</td>
    <td>27.80</td>
    <td>25.96</td>
    <td>17.89</td>
    <td>0.03</td>
    <td>39.26</td>
  </tr>
  <tr>
    <td>Finch-7B</td>
    <td>55.04</td>
    <td>25.55</td>
    <td>34.58</td>
    <td>21.05</td>
    <td>5.42</td>
    <td>45.26</td>
  </tr>
  <tr>
    <td>Finch-14B</td>
    <td>60.65</td>
    <td>26.64</td>
    <td>37.54</td>
    <td>22.40</td>
    <td>1.89</td>
    <td>51.98</td>
  </tr>
  <tr>
    <td>Mamba-130M</td>
    <td>55.24</td>
    <td>44.29</td>
    <td>13.31</td>
    <td>12.87</td>
    <td>12.84</td>
    <td>33.84</td>
  </tr>
  <tr>
    <td>Mamba-370M</td>
    <td>52.41</td>
    <td>39.30</td>
    <td>15.25</td>
    <td>12.97</td>
    <td>15.04</td>
    <td>38.55</td>
  </tr>
  <tr>
    <td>Mamba-790M</td>
    <td>50.81</td>
    <td>35.35</td>
    <td>18.14</td>
    <td>18.75</td>
    <td>16.96</td>
    <td>38.73</td>
  </tr>
  <tr>
    <td>Mamba-1.4B</td>
    <td>51.88</td>
    <td>33.11</td>
    <td>19.98</td>
    <td>20.44</td>
    <td>17.11</td>
    <td>39.08</td>
  </tr>
  <tr>
    <td>Mamba-2.8B</td>
    <td>53.10</td>
    <td>33.74</td>
    <td>23.91</td>
    <td>20.80</td>
    <td>20.00</td>
    <td>42.99</td>
  </tr>
  <tr>
    <td>Mamba2-130M</td>
    <td>54.02</td>
    <td>46.13</td>
    <td>14.05</td>
    <td>12.73</td>
    <td>13.88</td>
    <td>39.17</td>
  </tr>
  <tr>
    <td>Mamba2-370B</td>
    <td>53.41</td>
    <td>38.58</td>
    <td>16.19</td>
    <td>19.26</td>
    <td>17.05</td>
    <td>39.08</td>
  </tr>
  <tr>
    <td>Mamba2-780B</td>
    <td>52.97</td>
    <td>36.58</td>
    <td>17.42</td>
    <td>21.44</td>
    <td>19.15</td>
    <td>38.35</td>
  </tr>
  <tr>
    <td>Mamba2-1.3B</td>
    <td>53.77</td>
    <td>34.63</td>
    <td>19.45</td>
    <td>22.20</td>
    <td>17.98</td>
    <td>41.26</td>
  </tr>
  <tr>
    <td>Mamba2-2.7B</td>
    <td>53.62</td>
    <td>31.40</td>
    <td>22.82</td>
    <td>27.90</td>
    <td>20.66</td>
    <td>40.35</td>
  </tr>
  <tr>
    <td>Mamba2Attn-2.7B</td>
    <td>53.60</td>
    <td>29.57</td>
    <td>19.64</td>
    <td>36.83</td>
    <td>21.72</td>
    <td>43.37</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>60.48</td>
    <td>28.39</td>
    <td>41.24</td>
    <td>32.03</td>
    <td>26.77</td>
    <td>58.59</td>
  </tr>
  <tr>
    <td>Mistral-7B (IT)</td>
    <td>63.84</td>
    <td>45.24</td>
    <td>46.45</td>
    <td>33.95</td>
    <td>26.00</td>
    <td>65.37</td>
  </tr>
  <tr>
    <td>Mixtral-8X7B</td>
    <td>73.51</td>
    <td>26.84</td>
    <td>48.99</td>
    <td>40.66</td>
    <td>27.91</td>
    <td>68.35</td>
  </tr>
  <tr>
    <td>Mixtral-8X7B (IT)</td>
    <td>73.36</td>
    <td>44.44</td>
    <td>53.32</td>
    <td>36.76</td>
    <td>26.84</td>
    <td>68.84</td>
  </tr>
  <tr>
    <td>Pythia-160M</td>
    <td>49.80</td>
    <td>40.56</td>
    <td>10.42</td>
    <td>8.14</td>
    <td>3.00</td>
    <td>37.96</td>
  </tr>
  <tr>
    <td>Pythia-410M</td>
    <td>53.18</td>
    <td>40.28</td>
    <td>11.72</td>
    <td>20.58</td>
    <td>15.20</td>
    <td>39.49</td>
  </tr>
  <tr>
    <td>Pythia-1B</td>
    <td>53.81</td>
    <td>35.88</td>
    <td>13.78</td>
    <td>27.11</td>
    <td>17.26</td>
    <td>39.22</td>
  </tr>
  <tr>
    <td>Pythia-1.4B</td>
    <td>54.29</td>
    <td>36.45</td>
    <td>17.90</td>
    <td>22.16</td>
    <td>17.98</td>
    <td>39.22</td>
  </tr>
  <tr>
    <td>Pythia-2.8B</td>
    <td>53.15</td>
    <td>33.71</td>
    <td>20.18</td>
    <td>22.89</td>
    <td>20.19</td>
    <td>44.43</td>
  </tr>
  <tr>
    <td>Pythia-6.9B</td>
    <td>52.62</td>
    <td>33.62</td>
    <td>22.47</td>
    <td>24.86</td>
    <td>23.14</td>
    <td>42.91</td>
  </tr>
  <tr>
    <td>Pythia-12B</td>
    <td>54.32</td>
    <td>30.26</td>
    <td>23.84</td>
    <td>28.05</td>
    <td>21.75</td>
    <td>42.14</td>
  </tr>
</table>

Table 5: Results separated by task categories.
<table>
  <tr>
    <th></th>
    <th>Hallu.</th>
    <th>Detection</th>
    <th>Instr. Following</th>
    <th>QA</th>
    <th>Reading Comp.</th>
    <th>Sum.</th>
    <th>Fact-Checking</th>
  </tr>
  <tr>
    <td>Falcon-7B</td>
    <td>49.53</td>
    <td>29.11</td>
    <td>43.50</td>
    <td>26.18</td>
    <td>20.70</td>
    <td>46.91</td>
    <td></td>
  </tr>
  <tr>
    <td>Falcon-7B (IT)</td>
    <td>45.22</td>
    <td>44.21</td>
    <td>29.13</td>
    <td>21.13</td>
    <td>19.71</td>
    <td>39.20</td>
    <td></td>
  </tr>
  <tr>
    <td>FalconMamba-7B</td>
    <td>52.15</td>
    <td>47.19</td>
    <td>42.70</td>
    <td>19.98</td>
    <td>23.90</td>
    <td>61.85</td>
    <td></td>
  </tr>
  <tr>
    <td>FalconMamba-7B (IT)</td>
    <td>52.20</td>
    <td>45.63</td>
    <td>41.77</td>
    <td>19.44</td>
    <td>23.80</td>
    <td>60.05</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma-2B</td>
    <td>50.86</td>
    <td>36.43</td>
    <td>34.54</td>
    <td>28.08</td>
    <td>20.64</td>
    <td>43.28</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma-2B (IT)</td>
    <td>49.29</td>
    <td>43.60</td>
    <td>19.19</td>
    <td>23.57</td>
    <td>21.71</td>
    <td>38.06</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma-7B</td>
    <td>45.01</td>
    <td>32.90</td>
    <td>44.58</td>
    <td>34.76</td>
    <td>24.93</td>
    <td>54.14</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma-7B (IT)</td>
    <td>60.66</td>
    <td>47.33</td>
    <td>29.69</td>
    <td>23.04</td>
    <td>20.41</td>
    <td>55.74</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma1.1-2B (IT)</td>
    <td>50.52</td>
    <td>44.41</td>
    <td>21.27</td>
    <td>26.53</td>
    <td>20.02</td>
    <td>34.65</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma1.1-7B (IT)</td>
    <td>51.32</td>
    <td>47.12</td>
    <td>37.28</td>
    <td>26.31</td>
    <td>22.32</td>
    <td>58.96</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-2B</td>
    <td>55.10</td>
    <td>31.01</td>
    <td>37.82</td>
    <td>32.13</td>
    <td>16.04</td>
    <td>39.57</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-2B (IT)</td>
    <td>54.83</td>
    <td>40.62</td>
    <td>35.33</td>
    <td>28.15</td>
    <td>15.57</td>
    <td>58.53</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>62.73</td>
    <td>26.61</td>
    <td>49.30</td>
    <td>38.02</td>
    <td>21.35</td>
    <td>62.06</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-9B (IT)</td>
    <td>71.21</td>
    <td>53.89</td>
    <td>47.89</td>
    <td>37.78</td>
    <td>18.02</td>
    <td>68.53</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>58.23</td>
    <td>29.05</td>
    <td>58.76</td>
    <td>40.52</td>
    <td>28.90</td>
    <td>68.28</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-27B (IT)</td>
    <td>74.87</td>
    <td>57.08</td>
    <td>57.62</td>
    <td>44.46</td>
    <td>28.17</td>
    <td>69.28</td>
    <td></td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B</td>
    <td>49.05</td>
    <td>38.86</td>
    <td>34.90</td>
    <td>27.59</td>
    <td>18.42</td>
    <td>42.97</td>
    <td></td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B (IT)</td>
    <td>51.02</td>
    <td>40.49</td>
    <td>25.93</td>
    <td>27.11</td>
    <td>21.28</td>
    <td>46.60</td>
    <td></td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B</td>
    <td>51.49</td>
    <td>34.80</td>
    <td>46.83</td>
    <td>34.27</td>
    <td>22.99</td>
    <td>51.25</td>
    <td></td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B (IT)</td>
    <td>50.33</td>
    <td>47.66</td>
    <td>45.24</td>
    <td>29.00</td>
    <td>24.83</td>
    <td>64.61</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>51.62</td>
    <td>31.69</td>
    <td>47.96</td>
    <td>25.65</td>
    <td>25.18</td>
    <td>51.38</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-7B (IT)</td>
    <td>55.95</td>
    <td>41.30</td>
    <td>40.15</td>
    <td>29.34</td>
    <td>25.89</td>
    <td>57.26</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-13B</td>
    <td>66.34</td>
    <td>28.98</td>
    <td>51.85</td>
    <td>31.01</td>
    <td>27.32</td>
    <td>62.35</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-13B (IT)</td>
    <td>62.10</td>
    <td>37.64</td>
    <td>46.84</td>
    <td>27.44</td>
    <td>27.10</td>
    <td>60.71</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-70B</td>
    <td>60.75</td>
    <td>33.58</td>
    <td>61.27</td>
    <td>36.44</td>
    <td>27.98</td>
    <td>66.63</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-70B (IT)</td>
    <td>74.25</td>
    <td>49.63</td>
    <td>52.59</td>
    <td>30.43</td>
    <td>27.43</td>
    <td>65.18</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>57.88</td>
    <td>25.19</td>
    <td>51.81</td>
    <td>32.13</td>
    <td>26.60</td>
    <td>60.84</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA3-8B (IT)</td>
    <td>71.37</td>
    <td>51.59</td>
    <td>49.53</td>
    <td>35.39</td>
    <td>25.02</td>
    <td>65.66</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>70.61</td>
    <td>24.64</td>
    <td>61.02</td>
    <td>45.04</td>
    <td>28.88</td>
    <td>69.57</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA3-70B (IT)</td>
    <td>81.70</td>
    <td>45.77</td>
    <td>59.18</td>
    <td>37.54</td>
    <td>27.21</td>
    <td>70.56</td>
    <td></td>
  </tr>
  <tr>
    <td>Jamba</td>
    <td>53.44</td>
    <td>50.30</td>
    <td>43.45</td>
    <td>29.15</td>
    <td>23.72</td>
    <td>59.88</td>
    <td></td>
  </tr>
  <tr>
    <td>Jamba (IT)</td>
    <td>50.28</td>
    <td>47.46</td>
    <td>38.92</td>
    <td>25.42</td>
    <td>8.23</td>
    <td>57.28</td>
    <td></td>
  </tr>
  <tr>
    <td>Finch-1.6B</td>
    <td>51.43</td>
    <td>35.07</td>
    <td>24.94</td>
    <td>10.35</td>
    <td>11.73</td>
    <td>39.47</td>
    <td></td>
  </tr>
  <tr>
    <td>Finch-3B</td>
    <td>51.38</td>
    <td>30.74</td>
    <td>30.36</td>
    <td>14.33</td>
    <td>0.03</td>
    <td>39.26</td>
    <td></td>
  </tr>
  <tr>
    <td>Finch-7B</td>
    <td>53.22</td>
    <td>27.29</td>
    <td>41.06</td>
    <td>20.56</td>
    <td>5.39</td>
    <td>45.26</td>
    <td></td>
  </tr>
  <tr>
    <td>Finch-14B</td>
    <td>58.69</td>
    <td>28.23</td>
    <td>43.98</td>
    <td>23.51</td>
    <td>1.88</td>
    <td>51.98</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba-130M</td>
    <td>52.91</td>
    <td>51.59</td>
    <td>4.08</td>
    <td>5.25</td>
    <td>12.84</td>
    <td>33.84</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba-370M</td>
    <td>49.43</td>
    <td>44.82</td>
    <td>9.81</td>
    <td>6.45</td>
    <td>15.04</td>
    <td>38.55</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba-790M</td>
    <td>47.69</td>
    <td>39.27</td>
    <td>15.91</td>
    <td>11.06</td>
    <td>16.96</td>
    <td>38.73</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba-1.4B</td>
    <td>48.63</td>
    <td>36.97</td>
    <td>20.59</td>
    <td>12.91</td>
    <td>17.12</td>
    <td>39.08</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba-2.8B</td>
    <td>51.38</td>
    <td>37.37</td>
    <td>27.53</td>
    <td>15.26</td>
    <td>20.00</td>
    <td>42.99</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba2-130M</td>
    <td>50.59</td>
    <td>51.86</td>
    <td>4.21</td>
    <td>4.28</td>
    <td>12.87</td>
    <td>39.17</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba2-370B</td>
    <td>50.90</td>
    <td>42.72</td>
    <td>9.26</td>
    <td>12.18</td>
    <td>17.05</td>
    <td>39.08</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba2-780B</td>
    <td>49.55</td>
    <td>39.81</td>
    <td>14.22</td>
    <td>14.34</td>
    <td>18.16</td>
    <td>38.35</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba2-1.3B</td>
    <td>51.44</td>
    <td>38.46</td>
    <td>19.64</td>
    <td>16.15</td>
    <td>17.99</td>
    <td>41.26</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba2-2.7B</td>
    <td>50.96</td>
    <td>33.72</td>
    <td>26.81</td>
    <td>21.93</td>
    <td>20.67</td>
    <td>40.35</td>
    <td></td>
  </tr>
  <tr>
    <td>Mamba2Attn-2.7B</td>
    <td>49.56</td>
    <td>32.84</td>
    <td>24.12</td>
    <td>30.08</td>
    <td>21.72</td>
    <td>43.37</td>
    <td></td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>58.98</td>
    <td>31.48</td>
    <td>51.43</td>
    <td>30.90</td>
    <td>26.75</td>
    <td>58.59</td>
    <td></td>
  </tr>
  <tr>
    <td>Mistral-7B (IT)</td>
    <td>62.47</td>
    <td>45.43</td>
    <td>50.98</td>
    <td>31.94</td>
    <td>25.98</td>
    <td>65.37</td>
    <td></td>
  </tr>
  <tr>
    <td>Mixtral-8X7B</td>
    <td>73.44</td>
    <td>29.32</td>
    <td>60.87</td>
    <td>38.99</td>
    <td>27.89</td>
    <td>68.35</td>
    <td></td>
  </tr>
  <tr>
    <td>Mixtral-8X7B (IT)</td>
    <td>74.02</td>
    <td>43.87</td>
    <td>59.64</td>
    <td>34.24</td>
    <td>26.83</td>
    <td>68.84</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-160M</td>
    <td>49.68</td>
    <td>49.09</td>
    <td>1.16</td>
    <td>5.03</td>
    <td>3.01</td>
    <td>37.96</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-410M</td>
    <td>51.34</td>
    <td>46.65</td>
    <td>7.76</td>
    <td>16.13</td>
    <td>15.20</td>
    <td>39.49</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-1B</td>
    <td>51.18</td>
    <td>40.83</td>
    <td>11.60</td>
    <td>22.24</td>
    <td>17.28</td>
    <td>39.22</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-1.4B</td>
    <td>51.67</td>
    <td>41.84</td>
    <td>14.72</td>
    <td>15.96</td>
    <td>17.99</td>
    <td>39.22</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-2.8B</td>
    <td>50.36</td>
    <td>38.32</td>
    <td>20.82</td>
    <td>19.26</td>
    <td>20.20</td>
    <td>44.43</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-6.9B</td>
    <td>47.25</td>
    <td>34.26</td>
    <td>24.80</td>
    <td>19.11</td>
    <td>20.14</td>
    <td>42.91</td>
    <td></td>
  </tr>
  <tr>
    <td>Pythia-12B</td>
    <td>52.20</td>
    <td>33.51</td>
    <td>28.72</td>
    <td>23.42</td>
    <td>21.75</td>
    <td>42.14</td>
    <td></td>
  </tr>
</table>

Table 6: Results separated by task categories, where scores are weighted by the size of the tasks.
<table>
  <tr>
    <th rowspan="2">Model Name</th>
    <th colspan="2">Hallu.</th>
    <th colspan="2">Detection</th>
    <th colspan="2">Instr. Following</th>
    <th colspan="2">Closed-Book QA</th>
    <th colspan="2">Reading Comp.</th>
    <th colspan="2">Sum.</th>
    <th colspan="2">Fact-Checking</th>
    <th colspan="2">FAITHFULNESS</th>
    <th colspan="2">FACTUALITY</th>
  </tr>
  <tr>
    <th>Hallu.</th>
    <th>Detection</th>
    <th>Instr. Following</th>
    <th>Closed-Book QA</th>
    <th>Reading Comp.</th>
    <th>Sum.</th>
    <th>Faithfulness</th>
    <th>Factuality</th>
  </tr>
  <tr>
    <td colspan="21"><b>Attention-Only Models</b></td>
  </tr>
  <tr>
    <td>Gemma-2B</td><td>53.81 (± 5.67)</td><td>31.94 (± 7.59)</td><td>27.45 (± 4.13)</td><td>31.35 (± 5.83)</td><td>20.63 (± 1.64)</td><td>43.28 (± 5.22)</td><td>34.32 (± 8.83)</td><td>39.38 (± 11.29)</td>
  </tr>
  <tr>
    <td>Gemma-7B</td><td>49.65 (± 11.51)</td><td>30.25 (± 12.84)</td><td>38.20 (± 6.92)</td><td>38.96 (± 13.59)</td><td>24.93 (± 4.53)</td><td>54.14 (± 1.68)</td><td>33.06 (± 4.92)</td><td>50.44 (± 8.42)</td>
  </tr>
  <tr>
    <td>Gemma-2-2B</td><td>58.33 (± 8.64)</td><td>26.73 (± 16.90)</td><td>29.59 (± 4.83)</td><td>32.95 (± 5.26)</td><td>16.04 (± 0.47)</td><td>39.57 (± 18.96)</td><td>33.97 (± 2.01)</td><td>40.54 (± 11.17)</td>
  </tr>
  <tr>
    <td>Gemma-2-9B</td><td>65.64 (± 6.31)</td><td>23.76 (± 30.77)</td><td>39.36 (± 3.13)</td><td>39.74 (± 6.70)</td><td>21.38 (± 3.35)</td><td>62.06 (± 6.47)</td><td>41.41 (± 3.93)</td><td>56.42 (± 1.52)</td>
  </tr>
  <tr>
    <td>Gemma-2-7B</td><td>62.03 (± 12.43)</td><td>26.62 (± 33.58)</td><td>47.19 (± 2.86)</td><td>43.04 (± 5.82)</td><td>28.92 (± 0.74)</td><td>68.28 (± 1.88)</td><td>41.82 (± 10.80)</td><td>64.25 (± 0.31)</td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td><td>53.63 (± 4.38)</td><td>28.94 (± 8.99)</td><td>37.64 (± 1.89)</td><td>27.58 (± 3.74)</td><td>25.19 (± 0.69)</td><td>51.38 (± 5.88)</td><td>35.47 (± 2.90)</td><td>50.65 (± 5.91)</td>
  </tr>
  <tr>
    <td>LLaMA2-13B</td><td>67.80 (± 3.68)</td><td>26.57 (± 9.72)</td><td>39.65 (± 0.65)</td><td>32.17 (± 3.64)</td><td>27.33 (± 0.23)</td><td>62.35 (± 1.64)</td><td>41.08 (± 3.11)</td><td>57.51 (± 3.38)</td>
  </tr>
  <tr>
    <td>LLaMA2-70B</td><td>62.34 (± 18.78)</td><td>30.24 (± 17.11)</td><td>47.85 (± 2.53)</td><td>39.48 (± 8.21)</td><td>28.00 (± 0.55)</td><td>66.63 (± 1.45)</td><td>42.47 (± 5.96)</td><td>64.76 (± 5.64)</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td><td>60.76 (± 18.12)</td><td>22.06 (± 26.76)</td><td>41.55 (± 1.30)</td><td>33.52 (± 3.62)</td><td>26.62 (± 1.69)</td><td>60.84 (± 4.82)</td><td>39.32 (± 7.82)</td><td>57.22 (± 6.49)</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td><td>71.78 (± 8.69)</td><td>21.59 (± 25.66)</td><td>48.07 (± 3.43)</td><td>46.38 (± 7.73)</td><td>28.91 (± 1.68)</td><td>69.57 (± 8.99)</td><td>49.24 (± 4.80)</td><td>66.05 (± 6.72)</td>
  </tr>
  <tr>
    <td>Mistral-7B</td><td>60.48 (± 3.36)</td><td>28.39 (± 16.85)</td><td>41.24 (± 5.21)</td><td>32.03 (± 1.92)</td><td>26.77 (± 0.77)</td><td>58.59 (± 6.78)</td><td>39.96 (± 1.85)</td><td>56.04 (± 2.40)</td>
  </tr>
  <tr>
    <td>Mistral-8XB</td><td>73.51 (± 8.15)</td><td>26.84 (± 17.60)</td><td>48.99 (± 4.33)</td><td>40.66 (± 3.90)</td><td>27.91 (± 1.07)</td><td>68.35 (± 8.45)</td><td>48.49 (± 8.82)</td><td>65.37 (± 9.54)</td>
  </tr>
  <tr>
    <td>Falcon-7B</td><td>52.40 (± 2.99)</td><td>25.54 (± 12.62)</td><td>33.03 (± 4.42)</td><td>29.18 (± 1.64)</td><td>20.70 (± 1.48)</td><td>46.91 (± 7.71)</td><td>30.37 (± 3.89)</td><td>45.60 (± 18.79)</td>
  </tr>
  <tr>
    <td colspan="21"><b>Recurrent and Hybrid Models</b></td>
  </tr>
  <tr>
    <td>RecurrentGemma-2B</td><td>52.88 (± 2.33)</td><td>33.43 (± 3.12)</td><td>27.36 (± 1.85)</td><td>30.66 (± 1.77)</td><td>18.41 (± 2.86)</td><td>42.97 (± 3.63)</td><td>34.07 (± 1.11)</td><td>38.93 (± 2.45)</td>
  </tr>
  <tr>
    <td>RecurrentGemma-9B</td><td>55.75 (± 1.87)</td><td>31.67 (± 12.90)</td><td>36.79 (± 2.03)</td><td>36.57 (± 0.60)</td><td>22.99 (± 1.84)</td><td>51.25 (± 13.36)</td><td>35.89 (± 1.29)</td><td>50.63 (± 1.11)</td>
  </tr>
  <tr>
    <td>Jamha</td><td>57.66 (± 2.98)</td><td>43.36 (± 5.76)</td><td>39.50 (± 0.88)</td><td>33.00 (± 5.88)</td><td>23.72 (± 15.48)</td><td>59.88 (± 2.68)</td><td>36.28 (± 7.38)</td><td>51.78 (± 2.49)</td>
  </tr>
  <tr>
    <td>FalconMamba-7B</td><td>55.80 (± 0.73)</td><td>42.97 (± 1.19)</td><td>39.94 (± 0.43)</td><td>23.76 (± 0.85)</td><td>23.89 (± 0.89)</td><td>61.85 (± 1.88)</td><td>33.31 (± 0.28)</td><td>52.37 (± 1.11)</td>
  </tr>
</table>

Table 7: Changes in performance from the use of instruction-tuning, weighted by the number of samples present in each task.

![Change in task category performance from base model to instruction fine-tuned model.](page_1042_1342_768_384.png)

Figure 6: Change in task category performance from base model to instruction fine-tuned model.
<table>
  <tr>
    <th></th>
    <th>Faithfulness</th>
    <th>Factuality</th>
  </tr>
  <tr><td>Falcon-7B</td><td>34.91</td><td>36.92</td></tr>
  <tr><td>Falcon-7B (IT)</td><td>34.20</td><td>33.17</td></tr>
  <tr><td>FalconMamba-7B</td><td>35.92</td><td>47.02</td></tr>
  <tr><td>FalconMamba-7B (IT)</td><td>35.66</td><td>46.64</td></tr>
  <tr><td>Finch-1.6B</td><td>31.62</td><td>27.41</td></tr>
  <tr><td>Finch-3B</td><td>29.21</td><td>30.14</td></tr>
  <tr><td>Finch-7B</td><td>30.52</td><td>38.65</td></tr>
  <tr><td>Finch-14B</td><td>32.08</td><td>42.69</td></tr>
  <tr><td>Gemma-2B</td><td>35.85</td><td>32.56</td></tr>
  <tr><td>Gemma-7B</td><td>36.48</td><td>44.50</td></tr>
  <tr><td>Gemma-2B (IT)</td><td>36.77</td><td>28.00</td></tr>
  <tr><td>Gemma-7B (IT)</td><td>39.19</td><td>38.46</td></tr>
  <tr><td>Gemma1.1-2B (IT)</td><td>36.60</td><td>30.12</td></tr>
  <tr><td>Gemma1.1-7B (IT)</td><td>37.11</td><td>43.79</td></tr>
  <tr><td>Gemma2-2B</td><td>36.61</td><td>34.96</td></tr>
  <tr><td>Gemma2-9B</td><td>40.56</td><td>46.85</td></tr>
  <tr><td>Gemma2-27B</td><td>41.54</td><td>53.92</td></tr>
  <tr><td>Gemma2-2B (IT)</td><td>36.26</td><td>41.94</td></tr>
  <tr><td>Gemma2-9B (IT)</td><td>48.11</td><td>49.98</td></tr>
  <tr><td>Gemma2-27B (IT)</td><td>54.56</td><td>56.28</td></tr>
  <tr><td>RecurrentGemma-2B</td><td>36.55</td><td>31.46</td></tr>
  <tr><td>RecurrentGemma-2B (IT)</td><td>36.41</td><td>32.85</td></tr>
  <tr><td>RecurrentGemma-9B</td><td>36.61</td><td>42.96</td></tr>
  <tr><td>RecurrentGemma-9B (IT)</td><td>36.44</td><td>46.24</td></tr>
  <tr><td>Jamba</td><td>39.80</td><td>45.95</td></tr>
  <tr><td>Jamba (IT)</td><td>32.89</td><td>45.04</td></tr>
  <tr><td>LLaMA2-7B</td><td>35.14</td><td>42.51</td></tr>
  <tr><td>LLaMA2-13B</td><td>40.83</td><td>46.27</td></tr>
  <tr><td>LLaMA2-70B</td><td>41.74</td><td>53.69</td></tr>
  <tr><td>LLaMA2-7B (IT)</td><td>39.15</td><td>42.39</td></tr>
  <tr><td>LLaMA2-13B (IT)</td><td>41.71</td><td>45.78</td></tr>
  <tr><td>LLaMA2-70B (IT)</td><td>47.56</td><td>51.35</td></tr>
  <tr><td>LLaMA3-8B</td><td>37.60</td><td>48.14</td></tr>
  <tr><td>LLaMA3-70B</td><td>45.92</td><td>54.80</td></tr>
  <tr><td>LLaMA3-8B (IT)</td><td>47.69</td><td>49.74</td></tr>
  <tr><td>LLaMA3-70B (IT)</td><td>52.15</td><td>57.61</td></tr>
  <tr><td>Mamba-130M</td><td>34.38</td><td>19.13</td></tr>
  <tr><td>Mamba-370M</td><td>32.56</td><td>21.21</td></tr>
  <tr><td>Mamba-790M</td><td>32.95</td><td>23.63</td></tr>
  <tr><td>Mamba-1.4B</td><td>33.44</td><td>25.23</td></tr>
  <tr><td>Mamba-2.8B</td><td>34.47</td><td>29.05</td></tr>
  <tr><td>Mamba2-130M</td><td>34.42</td><td>20.13</td></tr>
  <tr><td>Mamba2-370B</td><td>34.93</td><td>22.04</td></tr>
  <tr><td>Mamba2-780B</td><td>35.40</td><td>22.90</td></tr>
  <tr><td>Mamba2-1.3B</td><td>35.31</td><td>24.91</td></tr>
  <tr><td>Mamba2-2.7B</td><td>36.39</td><td>27.86</td></tr>
  <tr><td>Mamba2Attn-2.7B</td><td>38.91</td><td>26.49</td></tr>
  <tr><td>Mistral-7B</td><td>38.47</td><td>47.42</td></tr>
  <tr><td>Mistral-7B (IT)</td><td>43.14</td><td>52.59</td></tr>
  <tr><td>Mistral-8X7B</td><td>46.16</td><td>55.15</td></tr>
  <tr><td>Mistral-8X7B (IT)</td><td>48.05</td><td>58.65</td></tr>
  <tr><td>Pythia-160M</td><td>25.13</td><td>15.23</td></tr>
  <tr><td>Pythia-410M</td><td>33.29</td><td>18.69</td></tr>
  <tr><td>Pythia-1B</td><td>35.43</td><td>18.21</td></tr>
  <tr><td>Pythia-1.4B</td><td>35.89</td><td>23.44</td></tr>
  <tr><td>Pythia-2.8B</td><td>35.48</td><td>25.79</td></tr>
  <tr><td>Pythia-6.9B</td><td>36.30</td><td>27.46</td></tr>
  <tr><td>Pythia-12B</td><td>36.85</td><td>28.73</td></tr>
</table>

Table 8: Results categorized by faithfulness and factuality.

<table>
  <tr>
    <th></th>
    <th>Faithfulness</th>
    <th>Factuality</th>
  </tr>
  <tr><td>Falcon-7B</td><td>30.37</td><td>45.60</td></tr>
  <tr><td>Falcon-7B (IT)</td><td>34.26</td><td>34.82</td></tr>
  <tr><td>FalconMamba-7B</td><td>32.75</td><td>52.37</td></tr>
  <tr><td>FalconMamba-7B (IT)</td><td>33.03</td><td>51.26</td></tr>
  <tr><td>Finch-1.6B</td><td>29.36</td><td>31.87</td></tr>
  <tr><td>Finch-3B</td><td>26.54</td><td>35.01</td></tr>
  <tr><td>Finch-7B</td><td>29.70</td><td>43.97</td></tr>
  <tr><td>Finch-14B</td><td>31.41</td><td>48.68</td></tr>
  <tr><td>Gemma-2B</td><td>34.32</td><td>39.38</td></tr>
  <tr><td>Gemma-2B (IT)</td><td>35.15</td><td>28.18</td></tr>
  <tr><td>Gemma-7B</td><td>33.06</td><td>50.44</td></tr>
  <tr><td>Gemma-7B (IT)</td><td>37.98</td><td>42.02</td></tr>
  <tr><td>Gemma1.1-2B (IT)</td><td>34.55</td><td>28.59</td></tr>
  <tr><td>Gemma1.1-7B (IT)</td><td>33.85</td><td>48.00</td></tr>
  <tr><td>Gemma2-2B</td><td>33.97</td><td>40.54</td></tr>
  <tr><td>Gemma2-2B (IT)</td><td>35.98</td><td>46.71</td></tr>
  <tr><td>Gemma2-9B</td><td>41.41</td><td>56.42</td></tr>
  <tr><td>Gemma2-9B (IT)</td><td>45.34</td><td>57.94</td></tr>
  <tr><td>Gemma2-27B</td><td>41.82</td><td>64.25</td></tr>
  <tr><td>Gemma2-27B (IT)</td><td>51.82</td><td>63.94</td></tr>
  <tr><td>RecurrentGemma-2B</td><td>34.07</td><td>38.93</td></tr>
  <tr><td>RecurrentGemma-2B (IT)</td><td>32.96</td><td>36.48</td></tr>
  <tr><td>RecurrentGemma-9B</td><td>35.89</td><td>50.63</td></tr>
  <tr><td>RecurrentGemma-9B (IT)</td><td>34.60</td><td>54.74</td></tr>
  <tr><td>Jamba</td><td>36.28</td><td>51.78</td></tr>
  <tr><td>Jamba (IT)</td><td>28.90</td><td>48.29</td></tr>
  <tr><td>LLaMA2-7B</td><td>35.47</td><td>50.65</td></tr>
  <tr><td>LLaMA2-7B (IT)</td><td>38.37</td><td>48.74</td></tr>
  <tr><td>LLaMA2-13B</td><td>41.08</td><td>57.51</td></tr>
  <tr><td>LLaMA2-13B (IT)</td><td>44.19</td><td>54.13</td></tr>
  <tr><td>LLaMA2-70B</td><td>42.47</td><td>64.76</td></tr>
  <tr><td>LLaMA2-70B (IT)</td><td>48.43</td><td>59.12</td></tr>
  <tr><td>LLaMA3-8B</td><td>39.32</td><td>57.22</td></tr>
  <tr><td>LLaMA3-8B (IT)</td><td>47.14</td><td>57.71</td></tr>
  <tr><td>LLaMA3-70B</td><td>49.24</td><td>66.05</td></tr>
  <tr><td>LLaMA3-70B (IT)</td><td>53.24</td><td>65.33</td></tr>
  <tr><td>Mamba-130M</td><td>29.78</td><td>17.64</td></tr>
  <tr><td>Mamba-370M</td><td>28.72</td><td>22.70</td></tr>
  <tr><td>Mamba-790M</td><td>29.28</td><td>26.35</td></tr>
  <tr><td>Mamba-1.4B</td><td>30.14</td><td>29.23</td></tr>
  <tr><td>Mamba-2.8B</td><td>32.78</td><td>34.83</td></tr>
  <tr><td>Mamba2-130M</td><td>28.44</td><td>19.58</td></tr>
  <tr><td>Mamba2-370B</td><td>31.37</td><td>22.59</td></tr>
  <tr><td>Mamba2-780B</td><td>31.50</td><td>25.16</td></tr>
  <tr><td>Mamba2-1.3B</td><td>32.76</td><td>29.41</td></tr>
  <tr><td>Mamba2-2.7B</td><td>34.30</td><td>33.42</td></tr>
  <tr><td>Mamba2Attn-2.7B</td><td>34.45</td><td>33.51</td></tr>
  <tr><td>Mistral-7B</td><td>39.98</td><td>56.04</td></tr>
  <tr><td>Mistral-7B (IT)</td><td>41.83</td><td>58.44</td></tr>
  <tr><td>Mistral-8X7B</td><td>48.49</td><td>65.37</td></tr>
  <tr><td>Mistral-8X7B (IT)</td><td>49.31</td><td>64.83</td></tr>
  <tr><td>Pythia-160M</td><td>22.83</td><td>12.62</td></tr>
  <tr><td>Pythia-410M</td><td>30.48</td><td>18.94</td></tr>
  <tr><td>Pythia-1B</td><td>32.71</td><td>20.29</td></tr>
  <tr><td>Pythia-1.4B</td><td>32.92</td><td>25.81</td></tr>
  <tr><td>Pythia-2.8B</td><td>33.60</td><td>31.22</td></tr>
  <tr><td>Pythia-6.9B</td><td>31.84</td><td>32.97</td></tr>
  <tr><td>Pythia-12B</td><td>35.72</td><td>35.11</td></tr>
</table>

Table 9: Results categorized by faithfulness and factuality, weighted by the number of examples per task.