HistAlign: Improving Context Dependency in Language Generation by Aligning with History
Authors: David Wan
Shiyue Zhang
Mohit Bansal
Venue: EMNLP 2023 Main (2023)
Keywords: generation
language models
summarization
data-to-text
TL;DR: Improve context dependency in language generation by aligning with history

Abstract: Language models (LMs) can generate hallucinations and incoherent outputs, which highlights their weak context dependency. Cache-LMs, which augment LMs with a memory of recent history, can increase context dependency and have shown remarkable performance in diverse language generation tasks. However, we find that even with training, the performance gain stemming from the cache component of current cache-LMs is suboptimal due to the misalignment between the current hidden states and those stored in the memory. In this work, we present HistAlign, a new training approach to ensure good cache alignment such that the model receives useful signals from the history. We first prove our concept on a simple and synthetic task where the memory is essential for correct predictions, and we show that the cache component of HistAlign is better aligned and improves overall performance. Next, we evaluate HistAlign on diverse downstream language generation tasks, including prompt continuation, abstractive summarization, and data-to-text. We demonstrate that HistAlign improves text coherence and faithfulness in open-ended and conditional generation settings respectively. HistAlign is also generalizable across different model families, showcasing its strength in improving context dependency of LMs in diverse scenarios.

HISTALIGN:
Improving Context Dependency in Language Generation by Aligning with History
David Wan
Shiyue Zhang UNC Chapel Hill shiyue, mbansal (cs. unc edu
Mohit Bansal
(davidwan,
Abstract
Top king, woman
Beseline Top Predictions
king
Language models (LMs) generate hallu- cinations and incoherent outputs, which high- lights their weak context dependency. Cache- LMs, which augment LMs with memory of recent history, can increase context dependency and have shown remarkable performance in di- verse language generation tasks. However, we find that even with training, the performance gain stemming from the cache component of current cache-LMs is suboptimal due to the misalignment between the current hidden states and those stored in the memory. In this work, we present HISTALIGN, new training ap- proach to ensure good cache alignment such that the model receives useful signals from the history. We first prove our concept on simple and synthetic task where the mem- ory is essential for correct predictions, and we show that cache component HistALiGN is better aligned and improves overall perfor- mance. Next, we evaluate HISTALIGN on di- verse downstream language generation tasks, including prompt continuation, abstractive sum- marization, and data-to-text. We demonstrate that HISTALIGN improves coherence and faithfulness in open-ended and conditional gen- eration settings, respectively. HISTAI,IGN is also generalizable across different model fami- lies, showcasing its strength in improving con- text dependency of LMs diverse scenarios.
Combine Probabilties
Combine Probabilties
Softmax Probability
Our Cache Probability
Baseline Cache Probability
Token
Prob
Token
Prob
Token
Prob
king
0.70
king
0.60
to
0.40
queen
0:15
3.30
woman
0.05
jester
0.05
jester
0.20
king
Similarity
Model
Local Cache
Input
woman
bow
After debating whether bow to the woman or king. the decided
king
Figure 1: An illustration of HISTALIGN and baseline cache-LM. The input example is from Chang and Mc- Callum (2022). Our HISTALIGN is able to assign high probabilities to both king and woman, and thus is able to tune down the weight of the hallucinated token queen from the softmax probability. Current cache language models (baseline) give high probabilities to irrelevant tokens the cache and thus are at risk of producing hallucinated incoherent tokens.
2020; Cao et al.. 2018; Maynez et al., 2020), which suggests the weak context dependency of LMs.
Cache language model (Grave et al., 201 7b, Cache-LM) is a simple yet effective method improve context dependency by equipping LM with an additional memory of recent history (lo- cal context) and enabling it to directly "copy" from the history. Such models showed considerable im- provement in language modeling and downstream generation tasks (Merity et al., 2017; See et al., 2017). However, since the introduction of Trans- formers (Vaswani al., 2017), local memory has been less used due the powerful self-attention mechanism, and more works have been focusing on leveraging long-term or external memory (Khandel- wal et al., 2020; Yogatama et al., 2021). Nonethe-
1 Introduction
Language modeling (LM), language generation, requires decent context dependency. For both open- ended and conditional generation tasks, we want the model generation to be consistent with its pre- vious generation or the input context. However, incoherence and hallucination problems are perva- sive in current model generations (Holtzman et al.,
'Our code is publicly available at https: //g thub. com/mmettde www.daawwwwwwaaaoooooo www./hiiiee stallig

---

less, Zhong et al. (2022) showed that using local memory on top of Transformer still beneficial.
showcase its strength in improving coherence for open-ended prompt continuation and faithfulness for abstractive summarization and data-to-text. To summarize, our contributions are as follows:
In this paper, we focus on applying local cache Transformer-baser LMs and show that better alignment of the cache component leads to stronger gains. First, we show that cache-LM thcoretically breaks the softmax bottleneck (Yang et al.. 2018) that limits the capacity of any parametric LM to model highly context-dependent natural language. Then, find that, in current cache-LMs, the sig- nals provided by the memory component are minor, even when using the cache component during train- ing (Zhong et al., 2022). We hypothesize that the main bottleneck comes from the misalignment of the current hidden states and those in the mem- ory, because of which more relevant memories are not given higher weights than less relevant ones. We demonstrate this problem through synthetic task: Ambiguous Template (Chang and McCallum, 2022), an example of which is shown in Figure 1, When asking the model to predict the next word given the context "After debating whether to bow to the woman or the king, the jester decided to bow to the "" current cache-LM does not give the highest probabilities to the desired words king and woman. Instead, we find that irrelevant words, such to and jester have high cache probabilities. When combining these probabilities with the origi- nal softmax, the desired words cannot be ranked as top tokens. We find that this problem exists pre- trained LMS of various sizes, fine-tuned models, as well as models with cache augmented.
We discuss why cache-LM with local memory can improve context dependency through a softmax bottleneck lens.
We show the misalignment problem present in current cache language models and their training strategy.
We propose new training method, HISTAL- IGN, based on order-informed contrastive learning, which alleviates the misalignment problem and makes better use of memories.
We demonstrate that HISTALIGN improves the coherence of open-ended generation as well as the faithfulness of conditional genera- tion, and it works across different model fami- lies and adds little computational overhead.
2 Related Work
Cache-LM and Pointer Network. Adding a cache component to language model (LM) was first introduced for speech recognition (Kuhn and De Mori, 1990). Grave et al. (2017c) extended this idea to RNN- neural LM, which they call neu- ral LL.. Cache-LM predicts the next token by combining the RNN model's outputs with the similarities between the cache and the current hid- den state. The cache saves tuples of hidden state and next token prediction, i.e., (hi, 2i+1), from recent history (see Section 3.2). Essentially, the cache component enables the model to copy tokens from the history. Similar to cache-LM, pointer network (Vinyals et al., 2015; Merity al., 2017) also combines generating and copying of tokens but uses h; as a representation of I; (instead of 21+1). This means that a pointer network requires learn- ing additional transformations between the current representation and those in the past and a gating component for interpolation (Merity et al., 2017; See et al., 2017).² ? In contrast, cache-IM doesn't 't need extra parameters be learned and can be ap- plied directly at testing time. It is more efficient to
Next, we address this misalignment issue by proposing a new fine-tuning scheme, HISTALIGN, in which we augment the LM training objective with contrastive loss to encourage the model to align the current hidden states with those in the history. As shown in Figure our cache compo- nent gives higher probabilities for king and woman than other less relevant words in the cache. Unlike the typical contrastive loss that treats all negative examples equally, we propose to learn a ranking of negative tokens, i.e., more semantically similar tokens are ranked higher. As shown in Figure 2, when we align the space for the token housing, we want words such as accommodations to be closer than less relevant words like children. Hence, the cache can also be useful even when the exact target word is not present in the history. We demonstrate the stronger cache performance of HISTALIGN through the synthetic ambiguous template task and
Depending on the implementation, the model can have additional numbers of parameters that are quadratic to the number of hidden size for the projection matrix (for example, See et al. (2017) uses the concatenation of four hidden states for the gating module).

---

be used for larger cache sizes (i.e., extending cache- LM long-term and external memory), and has been shown to perform better than pointer-network (Grave et al., 201 7b; Zhong et al., 2022).
mnodll-generated summaries contain hallucinations for two popularly used summarization datasets, re- spectively. Similar unfaithfulness problems have also been seen in data-to-text generation (Chen et al., 2020a), machine translation (Weng et al., 2020), etc. Though many approaches have been in- troduced to alleviate incoherence (Li et al., 2022a) or unfaithfulness (Cao and Wang, 2021; Wan and Bansal, 2022), in this work, we explore sim- ple yet general cache-LM method increase con- text dependency for diverse tasks. The concurrent work (Chang et al., 2023) uses pointer network type of architectures to improve next-word distri- bution and summarization factuality. They mod- ify the softmax head by using additional context- dependent embeddings. In contrast, we simply ap- ply the original rache-LM architecture and improve with a novel training objective.
While cache-LM can be directly applied at test time, a recent work (Zhong et al., 2022) showed that leads more improvement when using cache during training time as well. Nonetheless, such pro- posed learning objectives for cache-LMs usually only provide distant supervision to the cache com- ponent. In contrast, we introduce direct supervision to the cache, which aligns the current representa- tion with its history.
LM with Local or External Memory. Cache- LM and pointer network were originally proposed to only use hidden states from the local context, i.e., previous tokens in the input context. Though this technique has been proven to be helpful for language modeling and other language generation tasks (Gulcehre et al., 2016; Grave et al., 2017c; Merity et al., 2017: See et al., 2017), has been less used after the Transformer architecture became popular, because the aelf-attention mechanism can attend to any token in the input context. There- fore, many works (Grave al., 201 7a; Khandelwal al., 2020; Yogatama et al., 2021: Zhong al., 2022: Min et al., 2022) proposed to use long-term or external memory beyond local context by ap- plying retrieval techniques. Though our work can be extended to the external cache setting, we fo- cus only on incorporating local memory, and we show that local memory is still helpful on top of Transformer because it breaks the softmax bottle- neck (Yang et al., 2018) of parametric language models. concurrent work (Chang al., 2023) also demonstrates how a pointer network breaks softmax bottleneck by examples and empirical re- sults, while we discuss this in a more mathematical way in Section 4.1.
Preliminaries
3.1 Language Modeling
We focus on autorcgressive language modeling (LM). Here, for simplicity, we assume that the LM is decoder-only, i.c., the context of the current step is the generated tokens of previous steps. We show that the same approach can easily be generalized encoder-decoder models in Section 4.3. Given the context C = 21. It 1. the probability of next token 2₁ = w is predicted by a softmax head:
Pim (ulee) x exp(h, ew)
(1)
where ew is the output embedding of token w and h4 is the output context vector (hidden state) from the model at the t-th step. The model is trained by minimizing the cross-entropy loss: Ize E, log (z:
3.2 Cache Language Models
Context Dependency in Language Generation. Existing language generation models demonstrate weak context dependency. For open-ended gen- eration tasks, Holtzman et al. (2020) pointed out that strong LMs can produce very incoherent text following an input prompt. This incoherence issue has also been long observed in the story generation literature (Rashkin et al., 2020; Alabdulkarim et al., 2021). For conditional generation tasks, for ex- ample, summarization, Cao et al. (2018); Maynez et al. (2020) showed that around 30% and 70%
Cache language models augment memory com- ponent to language models. Following Grave et al. (2017c), we consider cache be list of tuples of context vector and target token, (h4, 2z). Assume we only consider the history of the local context, then the local memory of t-th step is written as:
Miocal {(,, 24)
(2)
Then, the next-token prediction aggregates the log- its from the softmax head and the similarities be-

---

tween h and those saved in the memory:
To break this bottleneck, one simple solution is to increase d, as we see larger models usually have better performance. Another solution pro- posed by Yang et al. (2018) and extended by Kanai et al. (2018); Yang et al. (2019); Chang and Mc- Callum (2022) is to use multiple softmax heads
Palm(u (w]er) x exp(hy ew)+
1 (xx=x))xp((simhhe he))
(3)
where sim(..-) can be arbitrary similarity func- tion. Here, we follow Zhong al. (2022) and use the scaled dot product: sim(h1, h2) du, where va
mixture of softmax (MoS), e.g., P(w[h)
cxp(hg) (1)T Cw) + exp(h (2) Cw) + exp(h, (3) Cw).
d is the hidden dimension size.
Each h is a different context vector. However, adding softmax hcads is fairly computationally ex- pensive. Comparing MoS to Eq. 3, we can see that adding exp(hy ew) and exp(sim( (h,, hs)) resembles MoS without adding extra softmax heads. Another way to understand this connection is that when using local memory, A is roughly equivalent to HE HH where H. are the hidden states in the local context.3 Assuming E.- E H.. A becomes HE,,. Different from E, E, is no longer a static output embedding matrix of size x d but a context-dependent embedding tensor of size N x V x d. Hence, the rank of A is no longer upper bounded by d. Note that this connection also holds for using long-term external memories.
While Grave et al. (2017c) only incorporated cache during evaluation, TRIME (Zhong et al., 2022) showed that it brings more benefits when also incorporated during training, i.e., minimizing Itime - E, log (22l())))) Here, we also use cache in both training and evaluation, but we im- prove the training objective by introducing direct supervision on the cache (see Section 4.2).
4
Our Methodology
4.1 Breaking Softmax Bottleneck
We first want to connect using local memory with the softmax bottleneck problem (Yang et al., 2018) and show that Transformer's self-attention cannot break this bottleneck, while the local cache can. Parametric autoregressive language models (Sec- tion 3. including Transformer-based LMs, use a softmax function operating on context vectors (or hidden states) H E Nxd and output embedding matrix E E Vxd, N is the number of contexts, assuming every token in the training set has differ- ent context, then N is the number of tokens in the training set. V the vocabulary size, and is the hidden dimension size. Then, the next token proba- bilities form a log-probatne probbabbiyy matrix A E IR VVV (At = log )). Ideally, since every context is unique, the rank of A should be as large as V (as- suming V < NN. However, A is roughly equiv- alent HE its rank is strictly upper bounded by hidden size (please refer Yang et al. (2018) for the formal proof). This low-rank problem greatly limits the LM's capacity to model highly context- dependent natural language. This can be seen in Figure| 1, where queen achieves higher probabil- ity than woman. The reason for LM's difficulty in such bimodal distribution, as explained in Chang and McCallum (2022), is that the four words king, woman, man, queen tend to form a parallelogram in the embedding space, and if the model's 's hidden state wishes to be close to the output embeddings of king and woman, will also be close to those of man and queen.
4.2 HISTALIGN
Cache-LM combines the original softmax probabil- ities with the cache probabilities by aggregating the similarity scores between the current hidden state and those in the cache. To use the cache module effectively, the similarity function sim(:-) plays an important role in Eq. 3. If the similarities be- tween the current hidden state and less relevant memories are higher than more relevant ones, it would steer the model away from selecting the most useful information from the cache. By assign- ing high probability to the correct local memo- ries, e.g., those corresponding to king and woman in the example of Figure 1, we can ensure that when the probabilities are combined, they will be scored higher than irrelevant and hallucinated to- kens. However, we find that even when directly maximizing log Pam (Zhong et al., 2022), there is no guarantee that the current representations are well aligned with relevant information stored in the memory, as shown by the baseline probabilities in Figure 1 (see Section 6.1 for more details).
Hence, deal with this misalignment, we pro-
"This is because the logits under the cache-LM setting are the sum of the original token logits HE (first term of Eq. 3) and the cache logits H H (the second term of Eq. 3), and property 2 of Yang et al. (2018) showed that the log- prohhbibyyylityy matrix and logit matrix have similar ranks.

---

1. Get Local Cache
Rank Memories by Embedding Similarity
Align Cache with Margin
For predicting label: housing
Vector Space
negotives
shildrea
Emb.
rentedd flat
Token
Sim.
Decoder- Onco Model
in housing
1.00
T
T
T
houssng problems...
1.00
suitable accommodations..
0.20
X4
rented
0.05
hausing problems
children
0.01
suitable g
Y4
accommodations
Local Cache
Encoder Decoder Model
remted fut
housing problems shildton ...
hausing
1
T
t
f
housiinn......
more housing
autubbl
Ya
accommodoaaong..
Figure 2: Mustrrtiion of HISTALIGN training approach. We first get local cache by combining the hidden states in local context with their target tokens, and then rank them according to embedding similarity. The ranked memories are then used to train with the margin loss. This ensures that negative yet similar words (c.g. accommodations) will be closer in the vector space than irrelevant words (e.g. children).
pose a new contrastive objective that encourages higher similarities between the hidden states of sim- ilar target tokens. During training, given the current hidden state h₂ and the corresponding next token Tt, we construct positive set Pe from caches by selecting memories with the same target token:
the following max- max-marginn loss (Liu et al., 2022c):
Leont. = =ΣΣ Σ
max (0, sim h2)
VEP: j>ijEEP
sim((e, hi) + dis)
(5)
where A = (j 1 and À is the margin tuned based on validation loss.
Pi (b)
The final objective of HISTALIGN is combina- tion of the original LM cross-entropy loss lae and this ranking- based contrastive loss:
All other memories are taken as negative examples. An example is shown in step 2 of Figure 2. For predicting the token housing, we have two previous mentions of the word housing, and the other words, including flat, children, accommodations,, etc., considered negative.
histalign
-ure + aleamt.
(6)
where a a tunable weight of the contrastive loss. Note that during the inference time, we use Eq. 3.
In the typical contrastive loss, such as InfoNCE (van den Oord et al., 2019), all negative examples are treated equally. However, we hope to learn an ordering of the negative examples more similar examples ranked higher than less similar ones. In the example in Figure 2, accommodations is more similar housing than children. This ensures that even when predicting words that do not have previous mentions the local cache, our model can still output reasonable alternative.
4.3 Extension to Encoder-Decoder Models
HISTALIGN can be easily adapted to encoder- decoder models. For conditional generation tasks, the target text usually short, hence, coherence is not a big issue. What more crucial is whether the target generation stays true to the input context, e.g., the input document for summarization or the input table for data-to-text. Therefore, we define the local cache to be the input tokens and their corresponding encoder hidden states, as opposed to the output tokens and decoder hidden states for decoder-only models. We then calculate the simi- larity between the current decoder hidden state with those encoder hidden states stored in the cache.
To achieve this, construct ranking of mem- ories by computing the cosine similarities be- tween the embedding of the current target word and the embeddings of words in the cache, i.e., cosim(et, ei). After sorting tokens from the most similar w.r.t. semantic similarity to the least, we use

---

5 Experimental Setup
first baseline we compare fine-tuning with the original cross-entropy loss (lz in Section 3.1). which named by the original model name our result tables. Then, we also compare to the most re- cent cache-LM learning objective, TRIME (Zhong et al., 2022) (Irime in Section 3.2).
Here, we describe the tasks and the experimental setups. Please refer to Appendix A for more details.
5.1 Tasks and Datasets
Ambiguous Template is useful synthetic dataset collated by Chang and McCallum (2022), in which each example is generated using templates with diagonal words* from semantic analogy relations in the Google (English) analogy dataset (Mikolov et al., 2013). This is a simple yet effective setting to examine whether the model can copy the correct tokens from history and not hallucinate semanti- cally similar tokens, c.g., queen and man of the example in Figure 1. Since the target words can al- ways be found in the context, we can also evaluate the performance only with the cache component. Open-Ended Generation evaluates the lan- guage modeling capability by asking the model to generate a continuation given prompt (Holtzman et al., 2020; Su et al., 2022; Li et al., 2022b). We use WritingPrompts (Fan et al., 2018), and treat the first 50 tokens as the prompt and allow the model to generate up to 256 tokens using the canonical nu- cleus sampling (p- 0.95) (Holtzman et al., 2020). Abstractive Summarization is the task of pro- viding an abridged version of the input document. One crucial problem is "hallucination', where the generated summaries contain facts entities that are wrong or not present in the document (Cao al., 2018; Maynez al., 2020). We evalu- ate on two widely-used English News summariza- tion datasets, XSum (Narayan et al., 2018) and CNN/DM (Hermann et al., 2015).
5.3 Evaluations
Ambiguous Template. As a proof-of-concept ex- periment, we evaluate under both full setting, using the combined probability in Eq. 3, as well as cache-only setting, only using the cache similar- ity scores to predict the next token. We evaluate the performance via the accuracy of having the two diagonal words within the top-k predictions (Aco@k) k), where = {2, 5, 10, 25). Ideally, we want to see 100% accuracy with k = 2, which in- dicates that the two diagonal words are the top 2 choices. Note that when only using the cache, k value of 50 would achieve perfect accuracy, as it would include the entire local history. In addition, want to empirically verify that cache LM with local memory can break the softmax bottleneck. To this end, we calculate the rank of bog-probabilityy matrix E N (Section 4.1) using 500 exam- ples (concretely, N = 1750 and V = 50257 for GPT-2 based models) under the full setting.
Open-Ended Generation. We mainly evaluate the coherence of model-generatte continuations. Following Su et al. (2022), coherence is approxi- mated by the cosine similarity of the SimCSE (Gao et al., 2021) sentence embeddings of the prompt and the continuation. In addition, following pre- vious works, ter et al., 2022) and MAUVE (Pillutla et al., 2021) scores for a more general evaluation. We hope HISTALIGN not to harm diversity and MAUVE. We also run human evaluation on Amazon MTurk to ask workers to compare the continuations gener- ated by TRIME and HISTAL.IGN More details can be found in Appendix B.I.
report n-gram diversity (Meis-
Data-to-Text is the task of describing structured data, where faithfulness is extremely important, humans do not tolerate any hallucinations in cases such as describing medical reports or finan- cial statistics (Thomson and Reiter, 2020). We evaluate on LogieNLG (Chen et al., 2020a).
5.2 Systems
A bstractev Summarization. We mainly eval- uate the faithfulness of generated summaries by three widely-used automatic metrics: FactCC (Kryscinski et al., 2020) and DAE (Goyal and Dur- rett, 2021), which entailment-baat metric; and Entity Precision (Nan al., 2021, PENT), which calculates the percentage of entities in the sum- mary that are present in the document. We also report ROUGE-L (Lin, 2004) for general content selection evaluation. Similarly, we conduct human
We use GPT2-small and GPT2-large (Radford et al., 2019) for ambiguous template and prompt continu- ation, and we use BART-large (Lewis et al., 2020) for both summarization and data-to-text. For all tasks, choose to finetune pre-trained LMs. The
'This refers to words that lie on the diagonal of a paral- lelogram in the embedding space. For example, for the tuple (queen, king, woman, man). the diagonal words are (king, woman) and (queen, man).

---

Full
Cache-Only
Full
Model
Acc@2
Acc@5
@cc@ 10
Acc@25
Acc@2
Acc@5
Acc@ 10
Acc (cc@2 25
Rank
GPT2-Small
50.00
62.20
68.80
76.96
0.00
35,57
50.56
75.71
762
TRIME
46.43
56.41
76.47
97.96
0.00
39.89
66,51
100.00
836
HISTALIGN
63.47
72.26
89.71
100.00
58.62
70.62
79.59
94.31
854
GPT2-Large 75.43
84.76
87.93
91.26
0.05
37.83
73.88
100.00
1280
TRIME 77.40
91.10
94.56
97.17
0.11
22.32
84.89
100.00
1377
HISTALIGN 82.22
92.84
96.57
98.34
82.15
92.51
99.94
100.00
1377
Table 1: Results on Ambiguous Template. HISTALIGN achieves the best performance in both full and cache-only settings. We also empirically show that TRIME and HISTALIGN break the softmax bottleneck.
Model
Acc@2
Acc@10 A
pected because the original model is fine-tuned us- Acc@25
LLaMA2-7B
0
0
100
ing the cross-entropy loss without the cache compo- nent involved, and thus applying cache at test time may not be helpful. Second, though TRIME (Zhong et al., 2022) generally outperforms the original model in the full setting, its cache-only Acc@ @ and Acc @5 are similar to the original model. Consider- ing that all target words are present in the history, this result indicates that despite the fact that TRIME uses cache during training, its cache component is still misaligned and has limited contributions to the final performance.
TRIME
100
HISTALIGN
100
100
100
100
Table 2: Cache-Only results Ambiguous Template with LLaMA2-7B model.
evaluation, where we ask crowd workers to judge whether each summary (of 100 randomly selected examples) faithful and informative. Please refer Appendix B.2 for more details.
Data-to-Text, We mainly evaluate the faithful- ness of model generations by NLI-Ace and SP- Acc (Chen et al., 2020a) and two more recent met- rics- TAPEX-Acc and TAPAS-Acc (Liu et al., 2022a). NLI-Ace is entailment-basee metric pre-trained on TabFacc dataset (Chen al., 2020b) using TaBERT (Yin al., 2020), and SP-Ace first parses the sentence into a logical program and eval- uates the execution accuracy. TAPEX-Ace and TAPAS-Acc are entailment-based metrics trained with TAPEX (Liu et al., 2022b) and TAPAS (Eisen- schlos et al., 2020), respectively. Same as previous works (Chen et al., 2020a), we report BLEU (Pap- ineni et al., 2002) for surface-level evaluation.
In contrast, HISTALIGN achieves high Acc@2 with only the cache module, substantially outper- forming the original model and TRIME on both model sizes, which demonstrates the effectiveness of our contrastive loss for aligning memories better. As a result, HISTALIGN outperforms both base- lines across all A in the full setting. And the im- provement holds for both model sizes, though with smaller gaps for the large model. This observa- tion is consistent with our discussion in Section 4.1 that larger model with larger hidden dimension suffers less from the softmax bottleneck, while lo- cal memory can help break this bottleneck of any parametric LM. This also empirically verified by the rank of the log-probability matrix reported Table I, where we see that the rank of the original model is upper-bounded by its hidden dimension (768 for GPT2-small and 1280 for GPT2-large), and having local cache breaks this bottleneck. Finally, we present two qualitative examples in Ta- ble See detailed discussions in Appendix C.
6 Results
We verify the strength of HISTALIGN aligning the cache component and thus improve the next- token prediction on ambiguous template in Sec- tion 6,1, coherence in open-ended prompt continu- ation in Section 6.2, and faithfulness in abstractive summarization and data-to-text in Section 6.3 and Section 6.4, respectively.
Experiment on recent LLM. LLaMA2 7B model (Touvron et al., 2023). Interest- ingly, find that LLaMA2 achieves 0% accuracy for Acc@ (2,5,10) when evaluated zero-shot. Af- ter fine-tuning, the model achieves 100% accuracy without any cache. This expected, as the task is a simple synthetic task, and the model, compared to GPT2-large, is 10x larger, and the hidden size
We also fine-tune
6.1 Importance Cache on Ambiguous Template
We show the results of the Ambiguous Template in Table 1. First, it can be seen that the original GPT2 model has pretty bad performance in the cache-only setting, especially considering @cc@2. This is ex-

---

Model
diversity
MAUVE
coherence
Fluency
Coherence
GPT2-small
88.13
86.6211.10
53.77:10.29
Wint
Tie
Lose|
Wint
Tie
Losel
TRIME
88.53
86.76 58
57.58
46.33
36.33
17.33
48.33
32.00
19.66
HISTALIGN
90.07 10.19
87.46 90
61.30++o.1
GPT2-large
88 88.82.2..0
96.18. L.0.44
12.39
Table 4: Human evaluation results of open-ended gen- eration. We conduct a pairwise comparison between HISTALIGN with TRIME ("Win" means humans pre- fer our HISTÁLIGN over TRIMe) and show the per- centage of passages that are judged as coherent and fluent. HISTALIGN is statistically significantly better (p < 0.05) than TRIME on fluency and coherence.
TRIME
90.70
87.27 1:0
53.1 10.19
HISTALIGN
89.41 20000 86.83
$3.51
Table 3: Automatic evaluation results of open-ended generation. Numbers are 3-run averages + the 95% confidence intervals.
is 3.2x larger (1280 + 4096). Thus, as mentioned in Section 4.1, the model alleviates the softmax bottleneck due to its larger hidden size.
performance, HISTALIGN still improves over the original model by 0.7 points. A similar trend can be observed for diversity. Holistically, HISTAL- IGN improves coherence while maintaining similar diversity and MAUVE.
However, we still observe the two problems with LLaMA2. First, the problem of softmax bottleneck still exists, as the rank of its output log-probability matrix A is still upper- of 4096, we find that its empirical rank is 3332. This means that it is still thcoretically less ex- pressive than highly context-dependent natural lan- guage. Second, TRIME is still not able to make good use of the cache, i.e., misalignment still ex- ists. As shown the Table 2, TRIME achieves 0% accuracy for Acc@ @ (2,5.10) under the cache-only setting, which shows that the issue of misalignment is even more apparent for larger language models: Since the token logits perform well enough, the model does not learn to use the cache anymore. Nevertheless, as shown in the table, our training objective can enforce the use of the local cache and achieve 100% accuracy, which is consistent with our findings from smaller models.
Besides automatic evaluations, we also conduct a human evaluation, the results of which are shown in Table 4. On both fluency and coherence, human raters prefer the continuations by HISTALIGN more than that by TRIME. This confirms the observation from the automatic evaluations that HISTALIGN does improve especially on coherence.
upper-bounded bounded by its hidden size
6.3 Faithfulness in Abstractive Summarization
The summarization results are shown in Table 5. TRIME improves faithfulness over the baseline on XSum, but the improvement is not clear on CNN/DM. In contrast, our HISTALIGN method greatly improves over the baseline, especially on DAE and Pem Pent, which specifically targeted to- wards hallucinations. Concretely, we improve FactCCC by 0.91 points, DAE by 4.78 points, and Pent by 3 points on the XSum dataset. HISTALIGN improves the metrics on CNN/DM as well though to a smaller degree. This shows that allowing the model to pay specific attention to previous contexts in the input is helpful in reducing hallucinations.
The presence of these two issues showcases that there is still room for improvement on LM's context dependency, as HISTALIGN outperforms TRIME in making good use of cache.
6.2 Coherence in Open-Ended Generation
The results of prompt continuation can be found in Table 3. Across both sizes of the model, we observe an improvement in coherence with TRIME and a larger improvement with HISTALIGN. The effect of HISTALIGN is especially prominent for the smaller model, where coherence increases by 7.5 points compared to the original model, and 3.7 points over TRIME. This validates hypothe- sis that HISTALIGN can improve the coherence of LMS. When looking at MAUVE, HISTALIGN im- proves by 0.8 points and 0.7 points over GPT2 and TRIME respectively when using small models. On the large model, while TRIME achieves the best
We note that the ROUGE-L score for HISTAL- IGN lower than the original model. This ROUGE- faithfulness tradeoff has been observed by many previous works (Chen et al., 2021; Kryscinski et al., 2020; Wan and Bansal, 2022; Wan et al., 2023), where the reference summary inherently contains hallucinations and thus does not overlap highly with the more faithful generated summaries.
To confirm this, we conduct a human evalua- tion. The results are shown in Table 6. HISTALIGN achieves the best faithfulness score, which sta- tistically significantly better than BART. This con- firms our observation from automatic metric results

---

XSum
CNN/DM
Model
Rouge-L.
FactCC
DAE
Por
Rouge-L
FactCC
DAEL
Pmni
BART
36.41
22.16
67.96
72.72
30.63
72.63
6.98
93.53
TRIME
36.50
22,94
66.34
74.25
30.60
72.65
7.08
93.39
HISTALIGN 35.45
23.07
63.18
75.71
29.96
74.93
5.73
93.80
Table Performance on abstractive summarization tasks. HISTALIGN consistently improves faithfulness over the two baseline methods both datasets.
Model
Faithfulness
Informativeness
Model
BLEU-(I/2/3)
NA
SA
TX
TS
BART
19.33
63.67
56.27/37.07/25.66
85.46
53.45
63.74
TRIME
20.00
66.33
TRIME 56.12/36.84/23 29
84,55
$2.85
63.74
65.11
HISTALIGN 26.33
65.33
HISTALIGN
56.65/37.56266.55
85.67
53.12
64.58
65.48
Table 7: Performance on LogieNLG (data-to-text gener- ation) evaluated by BLEU scores, NLI-Ace (NA), SP- Acc (SA), TAPEX-Ace (TA), and TAPAS-Ace (TS). HISTALIGN improves over two baselines on BLEU and three faithfulness metrics: NA, TX, and TS.
Table 6: Human evaluation results XSum. indi- cates that it is statistically significantly better (p < 0.05) than BART. Krippendorff's as are 0.52 and 0.34 for faithfulness and informativeness, respectively.
in Table 5. Though there is small drop in informa- tiveness, the difference between the three methods has no statistical significance. 5 This shows that the drop in automated metrics such as ROUGE-L does not necessarily mean a decrease informativeness.
more powerful models. Firstly, attention mech- anisms alone cannot break the softmax bottleneck, as shown in Table 2. Secondly, while increasing the model size can mitigate this bottleneck, the prob- lem will persist unless we reach size that truly encapsulates the complexity of human language. Cache-LM is a light alternative for breaking soft- max bottleneck theoretically and improving context dependency empirically.
6.4 Faithfulness in Data-to- Text Generation
The results on LogieNLG are shown in Table 7. Similar to abstractive summarization, HISTALIGN improve faithfulness LogieNLG. Out of the four faithfulness metrics, HISTALIGN achieves the highest NLI-Acc, TAPEX-Acc, and TAPAS-Acc: HISTALIGN achieves 0.6 and 0.8 point improve- ments on TAPEX-Acc Ace over BART and TRIME respectively, and a 1.74 point improvement on TAPAS-Acc over the BART model. In the mean- time, HISTALIGN obtains the best BLEU scores.
Acknowledgments
We thank the reviewers and Haw-Shiuan Chang for helping with providing the Ambiguous Template data. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL- 21 12335, DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031, and a Bloomberg Data Science Ph.D. Fellowship. The views contained in this article are those of the authors and not of the funding agency.
7 Diseussion and Conclusion
In this work, we improve the context dependency of LMs by introducing a novel cache-LM training ob- jective, HISTALIGN, which improves the existing cache-LM objective by adding an order-informed contrastive loss for the cache component. On a syn- thetic dataset, we show that HISTALIGN effective at retrieving the desired memories from the cache and breaking the softmax bottleneck. Furthermore, we demonstrate the effectiveness of HISTA LIGN at improving the coherence of open-ended generation and improving faithfulness of ubstractive summa- rization and data-to-text generation.
Limitations
While we focus on the local memory to show that current LMs still benefit from better local context dependency, our method is also compatible with external memories, which can potentially further improve the performance of HISTALIGN in future work. We evaluate HISTALIGN using GPT2 and BART that at most consist of 774M parameters, which smaller than the latest large LMs that have billions of parameters. On the Ambiguous Template task, we do show that this problem exists for recent LLMs with LLaMA2 7B models and our method improves the cache alignment, but we
We want to emphasize couple of salient points with the recent trend of pushing for larger and
Wo use honstrap test (Ffron and Tibshirani, 1993) to de- termine statistical significance in our paper.

---

hope that in the future we can explore scaling up the approach on large LMs various tasks. We believe that our method is still helpful for larger models. But as larger models suffer less from soft- max bottleneck (Section 4. how much it can help is an interesting problem to study in the future. An- other current limitation of this work is that due to the additional hyper-parameters (the À of the margin and the weight a of the contrastive loss), it becomes less straightforward to incorporate our HISTALIGN objective into pre-training compared to TRIME. The training objective also considers that each token has a fixed margin (and thus as- sumes that each token is equally different), which can be improved by dynamically adjusting the mar- gins. Although fine-tuning is cheaper and we show effective gains using HISTALIGN in fine-tuning, how to use HISTALIGN to pre-train LMs is also an interesting future work direction.
Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial In- telligence Conference and Fighth AAAI Symposium on Educational Advances in Artificial Intelligence, ISAIAA// 18. AAAI Press.
Hlw-Shhuan Chang and Andrew McCallum. 2022. Soft- max bottleneck makes language models unable to represent multi- mode word distributions. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 8048-8073, Dublin, Ireland. Association for Computational Linguistics.
Haw-Shiuan Chang, Zonghai Yao, Alolika Gon, Hong Yu, and Andrew McCallum. 2023. Revisiting the architectures like pointer networks to efficiently im- prove the next word distribution, summarization fac- tuality. and beyond. In Findings of the Association for Computational Linguistics: ACL 2023 (Findings of ACL).
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving faithfulness in abstractive sum- marization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 5935-5941, Online. Association for Computational Linguistics.
Ethical Considerations
As the OpenAl team pointed out, GPT-2 does not distinguish fact from fiction, so it can not support cases that require the generated text to be true. In addition, GPT-2 reflects the biases inherent the data they were trained on, so it can not be deployed unless the deployers first carry out a study of bi- ases relevant to the intended use case. Though our HISTALIGGN improves the coherence of GPT-2 gen- erations, the above statement still holds. Similarly, despite that HISTALIGN improved the faithfulness of BART-large generations for abstractive summa- rization and data-to-text generation, such systems cannot be directly deployed and used in factuality- sensitive scenarios without further checks in place.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural lan- guage generation from open-domain tables. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7929- 7942, Online. Association for Computational Lin- guistics.
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for table-hased fact verification. In Interna- tional Conference on Learning Representations.
References
Bradley Efron and Robert J. Tibshirani. 1993, An Intro- duction to the Bootstrap. Number 57 in Monographs Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, Florida, USA.
Amal Alabdulkarim, Siyan Li. and Xiangyu Peng. 2021. Automatic story generation: Challenges and attempts. In Proceedings of the Third Workshop on Narrative Understanding, pages 72-83, Virtual. Association for Computational Linguistics.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with interme- diate pre-training. In Findings of the Asvociation for Computational Linguisties: EMNLP 2020, pages 281-296, Online. Association for Computational Lin- guistics.
Shuyang Cao and Lu Wang. 2021. CLIFF; Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact-aware neural abstrac- tive summarization. In Proceedings of the Thirty-

---

Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9332-9346, Online. Association for Computa- tional Linguistics.
Roland Kuhn and Renato De Mori. 1990. A cache- based natural language model for speech recognition. IEEE transactions on pattern analysis and machine intelligence, 12(6):570-583.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1462, Online. Association for Computa- tional Linguisties.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seqqeene pre-training for natural language generation, translation, and com- prehension. In Proceedings the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 7871-7880 Online. Association for Computa- tional Linguisties.
Edouard Grave, Moustapha M Cisse, and Armand Joulin. 2017a. Unbounded cache model for online language modeling with open vocabulary. Advances in neural information processing systems, 30.
Quentin Lhoest, Albert Villanova del Moral, Yaeine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu. Lewis Tunstall, Joe Davison, Mario Šaško. Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger. Clément Delangue, Théo Matus- sière. Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017b. Improving neural language models with a continuous cache. In International Conference on Learning Representations.
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017c. Improving neural language models with a continuous cache. In International Conference on Learning Representations.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 140-149, Berlin, Germany. Association for Computational Lin- guistics.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. 2022a. Contrastive decod- ing: Open-ended text generation as optimization.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang. Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. 2022b. Contrastive decod- ing: Open-ended text generation as optimization.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Chin-Yer Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Sekitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi. 2018. Sigsoftmax: Reanalysis of the softmax bottleneck. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Ao Liu, Haoyu Dong, Naoaki Okazaki, Shi Han, and Dongmei Zhang. 2022a. PLOG: Table-to-logic pre- training for logical table-to-text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5531- 5546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022b. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In

---

Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022c. BRIO: Bringing order to abstractive summarization, In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. In Advances in Neural Information Pro- cessing Systems, volume 34, pages 48 4816- Cur- ran Associates, Inc.
Nyy Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline- conditioned generation with dynamic plot state track- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 4274-4295 Online. Association for Computational Linguisties.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Locally typical sampling. Transac- tions of the Association for Computational Linguis- tics, abs/2202.00066 00066..
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (FMNNP-IICNLP)) pages 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionallty. In Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc.
Abigail See, Peter Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Longg Papers), pages 1073- 1083.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Nonparametric masked language modeling.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems.
Feng Nan, Ramesh Nallapati, Zhiguo Wang. Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang. Kathleen McKeown, and Bing Xiang. 2021. Entity- level factual consistency of abstractive text summa- rization. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 2727-2733, Online. Association for Computational Linguistics.
Craig Thomson and Ehud Reiter. 2020. A gold standard methodology for evaluating accuracy in data-to-text systems. Proceedings of the 13th International Conference on Natural Language Generation, pages 158-168, Dublin, Ireland. Association for Computa- tional Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details. just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Di- ama Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten,
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

---

Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, liyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Zhilin Yang, Thang Luong, Russ R Salakhutdinov, and Quoe V Le. 2019. Mixtape: Breaking the softmax bottleneck efficiently. In Advances in Neural In- formation Processing Systems, volume 32. Curran Associates, Inc.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data, In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, On- line. Association for Computational Linguistics.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz
Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. 2021. Adaptive semiparametric lan- guage models. Transactions of the Association for Computational Linguistics, 9:362-373.
Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages 5657-5673, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in neural infor- mation processing systems, 28.
David Wan and Mohit Bansal. 2022. FactPEGASUS:
-waree pre-training and fine-tuning for ab- stractive summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1010-1028, Seattle, United States, Association for Computational Lin- guistics.
Factuality- aware
A Experimental Setup Details
Unless specified, we use Huggingface's Transform- ers library (Wolf et al., 2020) to train the mod- els. We use the trainer's default setting, including AdamW optimizer (Loshchilov and Hutter, 2019) and a linear rate scheduler. We use mixed preci- sion and deepspeed. We use RTX A6000 GPUs with 48GB memory and A A100 GPUs with 80GB memory.
David Wan, Mengwen Liu, Kathleen McKeown, Markus Dreyer. and Mohit Bansal. 2023. Faithfulness-wware decoding strategies for ab- stractive summarization. In Proceedings of the 17th Conference of the European Chupter of the Association for Computational Linguisties, pages 2864-2880, Dubrovník, Croatia. Association for Computational Linguistics.
For hyperparameter tuning, we try learn- ing rate of (1c-5.3e-,,5e-5) and À between (0.001.000001000000001 and contrastive weight (0.5.1.0) for all tasks. For HISTALIGN, we use of 0.001 and the contrastive weight a = 1, un- less otherwise specified.
Rongxiang Weng. Heng Yu, Xiangpeng Wei, and Wei- hua Luo. 2020. Towards enhancing faithfulness for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2675-2663 Online. Association for Computational Linguistics.
A.1 Ambiguous Template
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
The dataset consists of 122k, 250k, and 122k ex- amples for train, dev, and test sets, respectively. The test set has no overlap of diagonal words with the training set. Following Chang and McCallum (2022), we freeze output vocab to prevent over- fitting, and get loss only from the last token (the target token). We select the model based on valida- tion loss. For GPT2-large-basee models, training the original model took around 30 minutes, TRIME and HISTALIGN took around an hour with 4 RTX A6000. On GPT2-small-base based models, training took 10 minutes, 15 minutes, and 15 minutes, for orig, TRIME, and HISTALIGN, respectively.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bot- tlneekk: A high-rank RNN language model. In Inter- national Conference on Learning Representations.

---

Task
Datoset
Learning rate
Stepo/Epochs
Warmup ratio
Butch size
Max input tokens
Mox output tokens
Ambiguous Template
le-5
cp.
0.0
256
1024
Open endeeeddddd Generation
WritingPrompt
Se-5
ep.
0.2
512
1024
XSum
Se-5
15,000 steps
0.0
128
512
64
Summarization
CNN/DM
3e-5
20,000 steps
0.0
128
512
128
Data-to-text
LogieNLG
5e-5
eps
0.0
64
500
200
Table Hapereparamnters for all tasks.
Context
Houston and Pennsylvania are my favorites, and I especially love
The brother and the granddaughter are my favorites, and especially love the
GPT2-large
Pennsylvania, Harris, Pittsburgh Pennsylvania, Philadelphia, Pittsburgh Pennsylvania. Houston, Harris
brother, niece, granddaughter brother, niece. granddaughter brother, granddaughter, niece
Full
TRIME
HISTALIGN
GPT2-large
and, I. Houston
the, granddaughter, I the grunddaughter, favorites granddaughter, brother, the
Cache- only
TRIME
I. and, Houston
HISTALIGN
Pennsylvania, Houston, and
Table 9: Qualitative examples from Ambiguous Template. For both the full and cache-only settings, HISTALIGN retrieves the correct two tokens from the context as the top predictions.
A.2 Prompt Continuation
library (Lhoest al., 2021) for loading the XSum (Narayan al., 2018) and CNN/DM (Hermann et al., 2015) datasets. And we use Huggingface's s Metrics library for calculating ROUGE scores. Training the original model, TRIME, and HISTAL- IGN all took around 5 hours for XSum and training orig, TRIME and HISTALIGN all took around 4 hours for CNN/DM on A6000s.
WritingPrompts (Fan et al., 2018) contain 273k, 16k, and 15k examples in the train, dev, and test sets. We use the full train and dev sets, while we sample 5000 examples from the test set for final evaluation to save time. We first train the mod- els using the different objectives on the training set. We split the text into blocks of 512 tokens. For generation, we decode with nucleus sampling with p 0.95 and three random (0,1ss) and average the scores. Training the original small model takes around 1.5 hours, TRIME takes around 2 hours, and HISTALIGN takes around hours. Training GPT-2 large, TRIME and HISTALIGN takes around hours, 7 hours, and 11.5 hours on A100s, respectively.
A.4 Data-to-text
We follow Liu et al. (2022a) for pre- pre-processing dataset, such as adding numerical pre-computation to the tables. We use contrastive weight a = 0.5. LogicNLG (Chen et al., 2020a) consists of 28k training, 4k validation, and 4k test examples. We use original evaluation scripts for the faithfulness metrics, and the BLEU calculation script provided by the original dataset. Training the original model and TRIME took hours, and HISTALIGN took around 5 hours on 2 A100.
A.3 Summarization
XSum a news summarization dataset consisting of BBC articles and contains 204k/1 1 k exam- ples in the train/dev/test set. CNN/DM consists of Dailymail and CNN articles and the dataset con- sists of 287k/1334 IK examples in the train/dev/test set. We use the official packages for the faithful- ness metrics.? We calculate Pont by using spacy extract entities and only consider [PERSON, FAC, GPE, ORG, NORP, LOC, EVENT] the al- lowed entity types. We use Huggingface's 's Dataset
B Human Evaluation Details
For both human evaluations, we use Amazon Me- chanical Turk to do the annotation. We have the same set of requirements: The workers need be from the United States, have more than 10,000 number of HITS approved, and an approval rate greater than 98%.
'FacCCC: https: http::///gihhuu.ccoo... fact DAE: http::////ggtttttttttteeaaaeeeet Factuality-datasee
Phttps://gh thub. wtw.:ttw....ccceeee www.:/enucceeen///////emmmm http:c/..//blooommateeeeeeeeeeee

---

B.1 Open-ended Generation
the get high probabilities. In fact, the cache simi- larities of the original model are similar those of TRIME, again indicating that there is no guarantee of well-aligned memories, despite training with the cache. HISTALIGG nevertheless returns the two target words as the top two choices for both the full and cache-only settings, showing that the model benefits from the well-aligned memories through contrastive objective.
We use Amazon Mechanical Turk to annotate whether human prefers the continuation by TRIME by HISTALIGN. We do not include the original model, since TRIME shows better performance on the automatic metries. We select examples where the difference between their characters is less than 200 characters to ensure that the length is similar (since shorter texts will naturally be more coher- ent). We collect annotations per example for 100 randomly selected examples, yielding 300 annota- tions. We take the percentuge of passages that judged coherent and/or fluent.
D Sample Outputs
We show sample outputs for prompt continuations in Figure 5, summarization in Figure 6 and Figure 7, and data-to-text in Figure 8.
We pay 0.5 USD per HIT, and the average time it takes around 2.5 minutes, which yields an hourly rate of 2 $12 per hour. An example of the annotation page is shown in Figure 3.
B.2 Summarization
We follow the same setup as Wan and Bansal (2022), and also use qualification test where rate the faithfulness of the selected generated sum- maries. Only workers with the correct annotation can perform the actual task.
We select the most important sentences and re- place the less relevant sentences with an ellipsis to reduce the overload for the workers. We select ten most relevant sentences from the document by cosine similarity of the sentence embedding ing
(Reimers and Gurevych, 2019) for each summary and then combine and show all the selected relevant sentences from each summary.
Each task consists of three unique workers, where we take the mean the scores for this docu- ment. The final score is the mean factuality score across all documents. The average time for each task is around 2.5 minutes and we pay 0.5 USD per task, hence an hourly rate of 2 $12 per hour. An example of the annotation page shown in Figure 4.
Qualitative Results on Ambiguous Template
We present two qualitative examples in Table 9. We see that both the original model and TRIME have difficulty in outputting the two correct words as the top two choices. This is also reflected by the cache- only results, where irrelevant words, such and, I,
"We use the all-mpnet-hase-y2 model.

---

possible, which text better. The outputs evakuated with respect to following features: Coherence; Fluency. the following, we provide some guidelines regarding how to judge quality the system's result different features.
Given human-writth tarmtn- compare
whether the systemia sementicaly semantic meaning delined by the prefix And all is content is factuaity supported by can be logically inferned from the prett.
sumann-rrillnn
systien's perfectly line with
Fluenay: This metric
system's 's resuit human- grammaticclly
sasy understand.
Please note that:
1. is NOT nocessary that one paragroph is better than the other. You should choose "A and B are the same" If you think they are both very poorigood.
you farilllr 3. Paragrophs are excerpts from longer toxt, so the onding part may not be complete. But incompleteness should NOT affect your judgment.
Phenggrphhs may
NOT verify
facturlity
mvsluntion.
Prompt:
MM]) fighting haviing a patty argument middle fighting the villain. You watened the credit this one"" Yeeled
eachother gatling
A:
vigi with smir, My ourtosity ty peaked when them badass!"
my world with hile face. "The who killed us the again.
towards me. "Just you looiding q. "Yo... you'ro my sort" This timo it had to be a boy, he wasn't too big for that
noining
contenoe!" thought oeen was still a young girl and a child. He wore the samo attire. Ao watched that man maki my fate. "IIl have
green not
kind thing! "Whst the... you/nentt ren't boy! Both lileis began yell each other,
deall they seid
tomorro:" II told him when you whispared
my
the ane
laughing,
running with
put my fingor on the knitfo. "You ware too slow. You got too many things to bo done. This is partner white.
looked my reflection,
going
up, bitchi Just got the crodit" yolloo, and turned to hor, who just startod running. She threw
wher him. "What the... what the hell are you doin 'wit
revenge:"" The women in purple came forward and tumned right day!"
towel?",
was gone.
on thought, did not have to run. With that as my face was completaly open, throw my fist in fece, and shot the man on the back with his hands and head. Hi wimoort people
"Look,
placed my
came apross room with two maskked mon in One of the masked man was big hairy
brack beast with larga, dark face, while other was simill, tull black figure. "Whtd!
you
yelled, they
making my
moment playground. You've ruined those poople. All you vo done kill them
crawe. "Whhat the hotl are you doing
Coheronce:
A better
la better
and are the same
Flueney:
O A better
better
and
Submitt
Figure 3: Human annotation page for evaluating coherence and fluency for prompt continuation.

---

The article has been separited into serdences and displayed ine- -ine. We have shortared the text for your converience by removing the least misted sentiences, indicitted by ellpelt (-))
Note summaries shown difierent not any particutar order.
Please evaluate the summary the following three categories:
Coherence
document,
fluent. The summary better
follow.
more logical
ConsistencyFacttuliyy
Please avoid using general knowledge, and only consider in the contest the provided document.
supported
The summary contradicts information the document. The summary might say "A fhee broke out in Seattle", but an document says it broke out Portland. Or the summary mighht "the Republicana the election", but the document indicates the Democrats won instead. esample, summary might say
anywhere document doesn't menioon the time when the fire broke out.
summary (hallucinatee)
Informativeness
Please select informative if the summary expresses the main points the document, Summary should contain relevant and Important information and few unimportant details. If summary consistent document, please only consistent information evalusting this category.
Document:
First- coach Andy Smith, appointed
coach Maykel Moreira
Park, club confimed..
"We thank them for their time with us." a cub statement said.
"They heiped the club
best home start to seison Vale and progression FA Gup thire round." Monclay, playing Championship their last three league games to sipp to 17th the table just six points above the relegation zone.
game
Cup January, They
Ribero'ss a former assistant, Michael Brown, has been placed in temporary charge the Bursiem based club.
Summary:
manager
by Consistency:
charge.
Coherence:
Informatiennse:
Informative
consistent
not informative
Summary:
by Consistency:
charge.
Coherence:
Inforatiinnss:
Informative
consistent
not informative
Summary:
manager
charge.
Coherence:
Consistancy:
Informatvenos:
Informative
consistent
not informative
Figure 4: Human annotation page for evaluating faithfulness and informativeness for abstractive summarization.

---

Every time you die, you transported to the time of the first event that ultimately caused your death. One day, you are hit by a car and sent years into the past. It's been a strange life; constantly setting
Prompt:
GPT2-Large: up and killing other people. But it's done, and now it s time to settle down in this new world. I' 'd never actually thought about my past life before, 1 just knew that went back and lived, but this time there complication. I 've been to many places in my life, so why is there such big problem? It s the weirdest thing. I could anywhere in time and never get the feeling, but this time it weird. The cars are just like the last time, the ones that hit me, there" just hole in the middle of them. 1 m in a small, abandoned office, and the walls aren tt quite made out of cement. think that nice touch...
TRIME:
up and killing other people, But it's done, and now it's time to put the final touches on this latest creation. I'll be frank, don't I really know what I'm supposed to do. A couple months ago, I was my way to my parents, and there was an accident on the side of the road. It was some kind of accident; the car was hit by a drunk driver, and it fell on my head. Luckily, I was alive. I was given few months to live. by the hospital. I can live with that, at least until can figure out what happened. Anyway, I'm now waiting for the day when I meet my next death, and have to be prepared for the worst. I can tr to avoid death
HISTALIGN: up and stopping things, trying to prevent every mistake that might happen. If it weren't I for the fact that was always so careful with everything, I might be a normal person who just goes about their business. A normal person like you wouldn't I have the ability to go back in time to prevent yourself from ever having died in the first place. Every mistake that might have happened could have killed me then and there, and it wasn't II an easy road to travel down. I was lucky enough to have only been born on July 4th, 1996. I've been trying to do the right thing all these years. I've avoided death on more than one...
Prompt: Crown Minotaur Expedition The first week of the expedition was slow going. The terrain was rocky and unforgiving. as the Mother Nature had forgotten smooth out this patch of land. The Minotaurs had the most trouble due the
steep slopes of the mountain pass. When a Minotaurel came up the mountain pass, she would pass straight through, just passing through the pass. It was just one of many impassable ridges, that the Miatarel would face to traverse the pass. The only reason it was never impassable was due to the massive, hairy beasts, which covered half of the pass, and caused many people to leave the mountains. This, plus the numerous Minotaurel who passed through the pass would create an increasingly high attrition rate on the expedition. Still, the Miatarel had learned of the pass through the passing... ...
GPTz-Large:
TRIME: steep slopes of the mountain pass. When mountain pass was at its highest, the Minotaurs were no longer able to pass into the valley below. It was just too steep. Luckily there was plenty of rock that they could use as ramp and they would be able to crawl up with their arms, legs, and wings intact. The trip hegan on the third day after lunch. The trail was still rough, as many of the boulders were nearly a foot high. By now they could barely lift the weight of their bodies, much less their bags full of provisions and equipment. The road ahead of them was now covered in small puddles. This was typical of these...
HISTALIGN steep slopes of the mountain pass. When a mountain pass was at its highest, the Minotaurs were no longer able to pass into the valley below. It was just ton steep. Luckily, they were able to set up camps and set out in the first few days. This was only going to be matter of time until they realized where they were going. This place was far away from anything they knew of. The only light was the lanterns on their back, and the lanterns were only good for a few minutes. As for how they got there, no one is quite sure. They have not been able to find any of the equipment they carried when they left...
Figure 5: Sample outputs for open-ended prompt continuation.
Document: David I. Lipton, second in command at the IMF, outlined some of these risks in speech to the National A Association for Business Economics in Washington on Tuesday. "The IMF's latest reading of the global economy shows once again weakening baseline," he said, "We are clearly at a delicate juncture." The comments come after taaker-than-expectte figures from China showing that exports plunged hy a quarter from year ago. The IMF has alrcady said it is likely it will downgrade its current forecast of 3. 3.4% for global growth when it next releases its economic predictions in April. The dismal picture is one that has on-going ramifications for businesses and industrics that bet on China's 's growth story. Read more from Karishma: Why story about bulk shipping matters
trade
BART: The International Monetary Fund (IMF) has warned that the global economy is at "critical juncture". TRIME: The International Monetary Fund (IMF) has warned that the global economy is a "dangerous situation". "
HISTALIGN: The International Monetary Fund (IMF) has warned that the global economy is in "a delicate juncture".
Coventry University's 's Scarborough campus has been built on the town's 's former Weaponness Park and Ride site. About 200 students have hegun courses at the site, though it is expected to eventually he home to more than 2,000 students. The building, which includes engineering and science labs, mock law court and library, is part of f50m sports and education facility. Professor Craig Gaskell said: "Launccing our new state-of-the-as building is huge milestone for us and demonstrates our commitment to Scarborough and the Yorkshire coast area." A new University Technical College has been built nearby and Scarborough Athletic FC's new 5.000-seater stadium is also under construction on the site. Coventry University also has campus near London's s Liverpool Street Station and recently announced it will open a campus in Dagenham in September 2017.
Document:
BART:A university has officially opened its first campus in North Yorkshire.
TRIME: A new university campus has been officially opened in North Yorkshire. HISTALIGN: A university campus on the Yorkshire coast has opened to the public.
Figure 6: Sample outputs for XSum summarization.

---

Bayern Munich will be without versatile back David Alaba for the closing stages of the season after he was ruled out for around seven weeks with a torn knee ligament. Alaba, out for three months earlier this season after partial ligament tear in his right knee, was injured while playing for Austria in their I-1 draw against Bosnia on Tuesday. 'I am bitterly disappointed? the 22-year-old told the club's website. 'But I have big aim and that is to be there at the season finale.' David Alaba is tackled by Emir Spahic's 's tackle during Austria's 1-1 draw against Bosnia- Herzegovina. Mare Janko (left) celebrates with Martin Harnik and Alaha after he scnred against Licchtenstein last Friday. He is set to start rehabilitation work in about month, Bayern said. Alaba had been in superb form for Bayern since his comeback, helping them stay firmly on track for a treble of titles with some spectacular goals. Bayern, who are top of the Bundesliga with eight games left and through to the German Cup last eight, face Porto in the Champions League quarter-fina finals later this month. Alaba (left) has been in excellent form for Bayern this season as the Bavarian giants chase a treble of titles.
Document:
BART: David Alaba has been ruled for around seven weeks with lorn knee ligament. Alaba was injured while playing for Austria in their 1-1 draw against Bosnia on Tuesday. Bayern Munich face Porto the Champions Leugue quarter-finals later this month.
TRIME:David Alaba has been ruled out for around seven weeks with a torn knee ligament. Alaba was injured while playing for Austria in their 1-1 draw against Bosnia on Tuesday. Bayern Munich face Porto the Champions League quarter-final later this month. The 22-year-old is set
rehabilitation work in about month. David Alaba has been ruled for seven weeks with torn knee ligament. Austria defender was injured in their 1-1 draw against Bosnia- Herzegovina. Bayem Munich face Porto in the Champions League quarter-finals later this month.
HISTALIGN:
Demry Mathews has won the interim WBA lightweight title after being handed a unanimous points decision over Tony Luis in Liverpool. Mathews saw Richar Abril twice pull out of a fight due to illness to be striped of the belt and just this week replacement Ismael Barroso was forced to pull out after failing to ohtain a visa. Derry Mathews celebrates after victory over Tony Luis at Echo Arena on Saturday night. Mathews celebrates with his family after claiming the interim WBA lightweight title. Luis was drafted in at the last minute and while the Canadian will have won a lot of fans with his relentless style, he did not win over the judges and Mathews was handed a I 114-112, 12, 114-112, 115-112 decision. Mathews made the brighter start and kept Luis contained behind the jab while landing a decent uppercut at the end of the second round. Mathews throws a right punch during interim WRA lightweight title fight in Liverpool. Luis (right) was drafted in after Richar Abril and Ismacl Barroso both pulled out. The Liverpudlian continued to still largely control the fight but Luis showed plenty of aggression himself, coming forward and making life difficult for Mathews, landing a good uppereut of his own at the of the sixth, Luis' pressure and all-action style was beginning to tell on Mathews, who was more and more often going to the ropes and both men were deducted a point in the eighth, Mathews for a low blow and Lais for use of the head, Luis was in the ascendency in the closing rounds and looked to be pulling some points back on the cards but it was not enough as Mathews was handed a tight decision after a thrilling encounter.
Document:
BART: Demry Mathews beat Tony Luis Liverpool on Saturday night. Mathews Richar Abril twice pull of fight due to illness to be striped of the belt. Ismael Barroso was forced to pull out after failing to obtain visa. TRIME: Derry Mathews beat Tony Luis in Liverpool Saturday night. Mathews saw Richar Abril twice pull out of fight due to illness. Ismael Barroso was forced pull out after failing to obtain visa. Luis drafted in at the last minute but did not win over the judges.
HISTALIGN: Derry Mathews beat Tony Luis by unanimous points decision Echo Arena. Mathews saw Richar Abril and Ismael Barroso pull out of the fight due to illness and illness. The Canadian was handed 114-112, and 115-112 decision by the judges. L Liverpudlian Lais was drafted in the last minute after the pair pulled out.
Figure 7: Sample outputs for CNN/DM summarization.

---

home team score
away team
away team score
11.12 (78)
st kilda
21.11 (137)
8.9 (57)
geelong
8.8 (56)
Table for "1928 vfl season":
11.15 (81)
richmond
10.13 (73)
22.17 (149)
hawthorn
11.13 (79)
18.18 (126)
fitzroy
11.13 (79)
11.17 (83)
essendon
18.11 (119)
Reference: St Kilda had the highest Score Away Team in the 1928 VH Scason
BART: Hawthorn had the lowest Away Team Score of any team in the 1928 Vfl Season
TRIME: Geelong had the lowest Score 8.8 (56) while Hawttorn had the highest Score 11.13 (79)
HISTALIGN: St Kilda was the Away Team with the highest Score in the 1928 Vfl Season
matches
john hearne
29
Table for "1893 english cricket season":
tom richardson
23
johny briggs
28
arthur mold
28
bill lockwood
27
Reference: John Hearne, played more Match than any other Player, with 20 9
BART: Bill Lockwood and Arthur Mold had same number of Match
TRIME: Bill Lockwood and Arthur Mold both played 27 Match the 1893 English Cricket Scason HISTALIGN: John Hearne the Match with
Figure 8: Sample outputs for data-to-text generation.


Reviews:
[{"review_id": "qlXIpmrqhK", "reviewer": "Reviewer_DVaq", "summary": "", "questions": "A) Line 291-310, can you add some more details on why $A$ is roughly equivalent to $HE^T + HE_c^T$ and what $E_c$ (hidden states in the local context) means here?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful and detailed comments and for appreciating the novelty and efficacy of our method as well as the strong experiment results! Please see below for responses to your questions:\n\n**Clarification on breaking the softmax bottleneck**\n\n> A) Line 291-310, can you add some more details on why $A$ is roughly equivalent to $HE^T + HE_{c}^{T}$ and what $E_c$(hidden states in the local context) means here?\n\nWe would like to clarify that $A$ is roughly equivalent to $HE^T + HH_c^T$ (line 294), and not $HE^T + HE_{c}^{T}$. We will use equation 3 in Line 233 to explain these terms. $A$ is the log-probability matrix. It is the log_softmax of output logits. Under the cache-LM setting, output logits are the sum of the original token logits $HE^T$ (first term of equation 3) and the cache logits $HH_c^T$ (the second term of equation 3). Essentially $A=$ log_softmax$(HE^T + HH_c^T)$. Property 2 of Yang et al. 2018 showed that the log-probability matrix and logit matrix have similar ranks, and so we say $A$ is roughly equivalent to $HE^T + HE_{c}^{T}$. As for $E_c$, it is equal to $E+H_c$, and we use it to show how the static output embedding matrix ($E$) becomes a context-dependent tensor ($E_c$) (line 295-300).\n\nWe will expand on this point in the final version, thanks.\n\n\n\n\n\n**Performance on recent LLM**\n\n>The baselines are limited to GPT2 and BART. It would be interesting to see the performance of more recent works of LLM such as LLaMA.\n\nDue to academic computational constraints, we were not able to use very large models (note that we also finetune these models and not just zero-shot), but now we also run the LLaMA2 7B model on the ambiguous template. Interestingly, we find that LLaMA2 achieves 0% accuracy for Acc@{2,5,10} when evaluated zero-shot. After fine-tuning, the model achieves 100% accuracy without any cache. This is kind of expected, as the task is fairly simple, and the model, compared to gpt2-large, is 10x larger, and the hidden size is 3.2x larger (1280-> 4096), and we discussed how a larger model can alleviate softmax bottleneck in Section 4.1 (Line 279-281).\n\nHowever, we still observe the two problems with LLaMA2. First, the problem of softmax bottleneck still exists, as the rank of its output log-probability matrix ($A$) is still upper-bounded by its hidden size of 4096. We find that its empirical rank is 3332. This means that it is still theoretically less expressive than highly context-dependent natural language because ideally its rank can be as large as the vocabulary size (see Section 4.1). Second, TRIME on top of LLaMA2 is still not able to make good use of the cache (Line 319-323), i.e., misalignment still exists. As shown in the table below, TRIME achieves 0% accuracy for Acc@{2,5,10} under the cache-only setting, which shows that the issue of misalignment is even more apparent for larger language models: Since the token logits perform well enough, the model does not learn to use the cache anymore. Nevertheless, as shown in the table, our training objective can enforce the use of the local cache and achieve 100% accuracy, which is consistent with our findings from smaller models.\n\nThe presence of these two issues showcases that there is probably still room for improvement on LM’s context dependency and our method still outperforms TRIME in making good use of cache. Our method thus may lead to gains in more complex downstream tasks, e.g., faithfulness for abstractive summarization that are not fully resolved by large models like LLaMA2.\n\n\nWe will include this in the final version given the extra page.\n\nCache-only results for LLaMa2 7B-based models:\n\n| **Model** | **Acc@2** | **Acc@5** | **Acc@10** | **Acc@25** |\n|-----------|-------|-------|--------|--------|\n| LLaMA2-7B | 0 | 0 | 0 | **100** |\n| TRIME | 0 | 0 | 0 | **100** |\n| HistAlign | **100** | **100** | **100** | **100** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- In general, the paper is well-motivated. With a clear analysis, the paper proceeds to identify the root cause of the lack of context dependency and proposes the contrastive loss proposed is novel and logically sound. The design is simple but effective, bringing inspiration that the community can benefit from. \n- The experiment results are strong, indicating the efficacy of the methodology. \n- The paper is well written. The logic and design details are presented in a way that is easy to follow. ", "reasons_to_reject": "- The baselines are limited to GPT2 and BART. It would be interesting to see the performance of more recent works of LLM such as LLaMA.", "questions_to_authors": "A) Line 291-310, can you add some more details on why $A$ is roughly equivalent to $HE^T + HE_c^T$ and what $E_c$ (hidden states in the local context) means here?", "paper_topic_and_main_contributions": "This paper proposes to improve the context dependency of cache LMs. The authors connect the cache LM approach and the softmax bottleneck in the Transformer architecture. The main technical contribution is a contrastive learning objective encouraging higher similarity between histories corresponding to the same output token. Empirical results show a significant improvement in retrieving suitable memory and in downstream task coherence and faithfulness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LcC2BhUw3t", "reviewer": "Reviewer_gaqX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful and detailed comments and for appreciating the soundness of our method as well as the strong experiment results! Please see below for responses to your questions:\n\n**Diminishing performance with larger model**\n\n> b1. In LLM-based models such as ChatGPT, attention is effectively doing the caching operation (by attending to relevant tokens in the context, it generates the next word less according to the whole training memory) as well as contrastive training (the weight updates on less attended tokens are smaller). Therefore, I am afraid when both the LM and the training data are sufficiently large, the benefit of cached LM and contrastive training might diminish.\n\nWe agree that local cache will become less useful when the model becomes larger. That being said, we want to emphasize a couple of salient points. Firstly, as highlighted in Lines 250-278, attention mechanisms alone cannot break the softmax bottleneck. Secondly, while increasing the model size can mitigate this bottleneck, the problem will persist unless we reach a size that truly encapsulates the complexity of human language (lines 279-281). Hence, we think that as the model becomes larger, its context dependency gets better but there will still be room for improvement. Cache LM is a light alternative for breaking softmax bottleneck theoretically and improving context dependency empirically. How to make it stay useful for larger models is an interesting future research direction. Besides, smaller (e.g., distilled) models by far remain to be useful in real-world applications. We hope our method can be helpful in these cases.\n\nWe will make this point clear in the camera-ready version.\n\n\n**Training objective**\n\n> b2. In Equation (5), the definition of $\\lambda_{i,j}$ is a bit ad-hoc. It assumes nth-ranked tokens are equally spaced in the probability space which might not be the case.\n\nWe agree with the reviewer regarding the potential enhancement of the training objective. Our primary rationale for adopting this training objective from Liu et al., 2022c centers on its emphasis on the ranking of tokens, rather than the margins between tokens.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a1. The methodology is somewhat novel; the rationale is clear and reasonable.\na2. The author has evaluated LM performance on 4 different LM-based tasks, and achieve significant improvement in all of them.\na3. See b1, this technique might be particularly effective on smaller models or dataset.", "reasons_to_reject": "b1. In LLM-based models such as ChatGPT, attention is effectively doing the caching operation (by attending to relevant tokens in the context, it generates the next word less according to the whole training memory) as well as contrastive training (the weight updates on less attended tokens are smaller). Therefore, I am afraid when both the LM and the training data are sufficiently large, the benefit of cached LM and contrastive training might diminish.\nb2. In Equation (5), the definition of lambda_{i,j} is a bit ad-hoc. It assumes nth-ranked tokens are equally spaced in the probability space which might not be the case.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of incoherence and hallucination in state-of-the-art language models (LM). Since the language model is trained on huge text, it generates the next word based on the entire training memory rather than giving the relevant word within the current context. State-of-the-art uses cached language model to address this issue. On top of this, this paper proposes a discriminative training criteria (with a max-margin loss that adjusts the probability score of not only the correct token but also similar tokens ranked according to similarity) to further improve the performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YAywAgiizx", "reviewer": "Reviewer_GMge", "summary": "", "questions": "A) What is an advantage of the Cache-LM compared to the pointer network, and have you considered including a model equipped with a pointer network as one of the baselines?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful and detailed comments and for appreciating the robustness (on multiple NLG tasks) and flexibility (in adapting to different architectures) of our method! Please see below for responses to your questions: \n\n**Difference between pointer network and cache-lm**\n> I am missing a stronger distinction between the Cache-LM and the pointer network. As far as I understood from §2 (l. 138-142), the formulations are fairly similar. There is also a concurrent work using a pointer network for solving similar problems (l. 198-201). I thus believe that the authors should devote more space for explaining the differences between these two architectures. (I am willing to increase the soundness score if this point is made more clear.)\n\n> A) What is an advantage of the Cache-LM compared to the pointer network, and have you considered including a model equipped with a pointer network as one of the baselines?\n\nTo expand on our points from Lines 138-142: As noted in prior studies such as Grave et al., 2017a and Zhong et al., 2022, pointer network uses the current hidden activation ($h_i$) as a representation of the current input ($x_i$), while cache-LM use it to represent the $x_{i+1}$. Hence, a pointer network requires additional learning of a transformation between the current representation and those in the past. Also, it usually requires a learned gating component for interpolation (Merity et al., 2017; See et al., 2017). Therefore, adding a pointer network to a model means adding quite a number of new parameters. Depending on the implementation, the model will have additional numbers of parameters that are quadratic to the number of hidden size for the projection matrix (for example, See et al., 2017 uses the concatenation of four hidden states for the gating module).\n\nIn contrast, cache-LM doesn't need extra parameters to be learned and can be applied directly at testing time. It is more efficient to be used for larger cache sizes (i.e. extending cache-LM to long-term and external memory). \n\nFurthermore, previous works (Grave et al., 2017a, Zhong et al., 2022) have shown that the pointer network performs worse than cache-LM, and we thus mainly compare to cache-LM baselines.\n\nWe will make this point clear in the camera-ready version.\n\n\n**Diminishing performance with larger model**\n\n> As the authors point out, the gains from the proposed method are diminishing as LMs grow in size, since the problem can be also tackled by increasing the size of the context vector.\n\nWe agree that local cache will become less useful when the model becomes larger. That being said, we want to emphasize a couple of salient points. Firstly, as highlighted in Lines 250-278, attention mechanisms alone cannot break the softmax bottleneck. Secondly, while increasing the model size can mitigate this bottleneck, the problem will persist unless we reach a size that truly encapsulates the complexity of human language (lines 279-281). Hence, we think that as the model becomes larger, its context dependency gets better but there will still be room for improvement. Cache LM is a light alternative for breaking softmax bottleneck theoretically and improving context dependency empirically. How to make it stay useful for larger models is an interesting future research direction. Besides, smaller (e.g., distilled) models by far remain to be useful in real-world applications. We hope our method can be helpful in these cases.\n\nWe will make this point clear in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper demonstrates that the proposed method robustly improves coherence of text generated by mid-sized Transformer models, both for open-ended and conditional generation.\n- The method can be simply applied to both decoder-only and encoder-decoder architectures.\n- The authors clearly explain the reasoning behind the design choices and arrive to reasonable conclusions on the basis of the performed experiments.\n- The paper itself is coherent and easily readable.\n- The code for the experiments is provided in the supplementary.", "reasons_to_reject": "- I am missing a stronger distinction between the Cache-LM and the pointer network. As far as I understood from §2 (l. 138-142), the formulations are fairly similar. There is also a concurrent work using a pointer network for solving similar problems (l. 198-201). I thus believe that the authors should devote more space for explaining the differences between these two architectures. (I am willing to increase the soundness score if this point is made more clear.) [After rebuttal: I have increased the soundness score to 4.]\n- As the authors point out, the gains from the proposed method are diminishing as LMs grow in size, since the problem can be also tackled by increasing the size of the context vector.", "questions_to_authors": "A) What is an advantage of the Cache-LM compared to the pointer network, and have you considered including a model equipped with a pointer network as one of the baselines?", "paper_topic_and_main_contributions": "The paper presents an approach for improving coherency and reducing hallucinations in a text generated by Transformer language models (LMs). The approach is based on Cache-LM (Grave et al., 2017b), a method for equipping a RNN with an additional memory of recent history. Cache-LM combines the standard softmax probabilities with probabilities of copying a token from cache, using similarity of hidden states to determine the relevancy of cached token for the current context.\n\nHere, the authors adapt Cache-LM for Transformer decoder and encoder-decoder architectures. On a variety of tasks (including a synthetic task for semantic analogy, open-ended generation, abstractive summarization, and data-to-text generation), the authors show that LMs (GPT-2 and BART) equipped with the caching mechanism achieve better factuality and coherency while retaining text diversity and lexical similarity to the reference.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper innovatively integrates the cache language model (Cache-LM) approach with the Transformer architecture to address incoherence and hallucination in state-of-the-art LMs. Its key contributions encompass bridging the cache LM strategy with the softmax bottleneck of Transformers, introducing a novel contrastive learning objective to enhance the similarity between histories associated with the same output token, proposing a discriminative training criterion with a max-margin loss, and adapting Cache-LM for both Transformer decoder and encoder-decoder architectures.
The reviewers commend the paper for being well-motivated and presenting a clear rationale. They highlight the innovative nature and logical soundness of methods such as the contrastive learning objective. The paper's strength is further bolstered by comprehensive evaluations that demonstrate significant improvement across multiple tasks. Additionally, the versatility of the proposed technique is emphasized, noting its applicability to various architectures, including decoder-only and encoder-decoder. Finally, the paper's coherence, ease of understanding, and the inclusion of supplementary code are appreciated.
The consolidated concerns from reviewers primarily focus on the paper's baseline model choices, emphasizing its reliance on older models like GPT-2 and BART and suggesting the inclusion of newer models like LLaMA. Additionally, there's a strong call for a more pronounced distinction and deeper discussion regarding the differences between Cache-LM and pointer networks. Most reviewers also raised questions about the proposed techniques' effectiveness as language models become larger and more sophisticated. Furthermore, there are requests for clarification on certain technical aspects of the paper, especially around specific lines and equations. Lastly, concerns were highlighted about some assumptions in the paper, particularly regarding the definition of terms in the equations and the general perspective on token probabilities.
Decision: Accept-Main