Context Tokens are Anchors: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
Authors: Qiyan Zhao
Xiaofeng Zhang
Shuochen Chang
Qianyu Chen
Xiaosong Yuan
Xuhang Chen
Luoqi Liu
Jiajun Zhang
Xu-Yao Zhang
Da-Han Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Multimodal Large Language Models; Information flow
Primary area: foundation or frontier models, including LLMs
TL;DR: Understanding and Mitigating the “Repeat Curse” from the Perspective of Information Flow.

Abstract: Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the Repeat Curse. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model’s growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present CoTA, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA

Published as a conference paper at ICLR 2026
CONTEXT TOKENS ARE ANCHORS: UNDERSTANDING THE REPEAT CURSE IN DMLLMS FROM AN INFORMA- TION FLOW PERSPECTIVE
Qiyan Zhao 1.2,3
Xiaofeng Zhang 11 Shuochen Chang' Qianyu Chen', Xuhang Chen' Luoqi Liu Jiajun Zhang Xu-Yao Zhang", Da-Han Wang"*
Xiaosong Yuan-
SJTU "XMUT
CASIA *NTU MMeitu (China) Limited
Equal contribution
*Corresponding author
ABSTRACT
Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accel- erate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the Repeat Curse. To better investigate underlying mechanism behind this issue, we analyze repeti- tion generation through the lens of information flow. Our work reveals three key findings; (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges deeper layers, reflecting the model's growing predic- tion certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing penalty term the confi- dence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alle- viating repetition and achieves consistent performance improvements on general tasks. Code is available at /ttttts://qitttt com/Erik2719/cot
present CoTA, plug-and-play method for
INTRODUCTION
Recent advances in diffusion-based large language models (dLLMs) Nie al. (2025b); Zhu et al. (2025a); Ye et al. (2025a) have demonstrated impressive reasoning and parallel decoding capabili- ties. Unlike autoregressive large language models Touvron al. (2023a) Yang al. (0024a), which generate text by sequentially predicting the next token, dLLMss produce tokens in paral lel by framing response generation an iterative denoising process over fully masked discrete token sequence. Building on this foundation, diffusion-based Multimodal Large Language Models (dMLLMs) You al. (2025); Yang et (2025b); Li et al. (2025b) have emerged powerful multimodal systems that integrate vision- instruccion tuning with dLLMs, and have evolved into promising alternatives to autoregressive architectures Liu al. (2023a) :
Li et al. (2025a).
Current dMLLMs face notable inference latency, stemming from their multi-step denoising proce- dure and the use of bidirectional attention. Existing efforts Liu al. (2025b); Wei et al. (0025a); Wu et al.] (2025) mainly exploit customized caching strategies for both prefix and suffix tokens to accelerate dMLLM inference. However, our experiments reveal that these methods often introduce severe side effect: the generated text exhibits substantial repetition, as illustrated in Figure ! phenomenon we refer to as the Repeat Curse. This repetition significantly reduces the performance and readability of the model outputs. We conduct statistical analysis of the repeat curse across different dMLLMs and caching techniques, as shown in Figure 2 The empirical results reveal the widespread presence of the repeat curse when dMLLMMs equipped with current caching methods.

---

Published as a conference paper at ICLR 2026
Normal Outputs
Repeat Curse
Normal Outputs
landscape, cominattng the scene
The image capteres a serone rural
weathered
fandecape. bam, its rentic charm
The image cuptures a serene rural
image captures serene rural a nustio weoden barn, its weathered
exterior testament to the passage of time. The barn nestled In fllld to most the horissn, the siatenee,
unhunced by the weathered texture
exterier teafament ot to years use.
range mountains rises, their
out towards the the bortzon, The the
flald tail which stretches
grass, which strutahes out towardss the hortsen, the distance. Beyand
clowds. The sky ahove soft pink, sugpesting aither down or dusk, The
mujjsttcc range of mountains rises,
fas cloude. the distance,
mounteins ris risouutioally against the aby, thatr stopss cleaked in shaden ahove is soft pink of down, dusk, earty morning, casting warm
low engle, making the born appear grand and Impoatin. The overall
sceno. The born's position of the
the overall tranquility the the
placamest mountains create harmanioun composiition, leading the the eye the
sense of deepth and tranquility inviiig the viewer to Imagiee
perspective the image taken from lew angle, adding the barn's
setting.
Please describe the imago in detail.
LLaDA-V
LLaDA-V+Cache
Figure I: Motivation. When cache is applied to accelerate dMLLMs, the generated responses often exhibit excessive token repetition-a phenomennn we term the Repeat Curse.
However, the inherent black-box nature of dMLLMs poses challenge to uncovering the under- lying mechanisms behind the repeat curse. Recent advances in information flow Yu et al. (2024); Wang et al. (2023): Chen et al. (2024a) have introduced an interpretable analysis paradigm for un- derstanding the relationship between model outputs and internal mechanisms, which has motivated the development of numerous methods for mitigating abnormal generation behaviors, such as hallu- cination Tang et al. (2025b) and degeneration Yona et al. (2025). Inspired by these successes, this work conducts de-thhhh analysis from the perspective of information flow to reveal the connection between the internal mechanisms of dMLLMs and the repeat curse.
By visualizing the attention interaction patterns among tokens (as shown in Figure that in dMLLMs, the context tokens adjoining the query act as anchors that progressively aggregate semantic information across layers, thereby causing attention distribution to gradually concen- trate on these context tokens. We further introduce information entropy to quantitatively analyze the impact of context tokens on the decoding dynamics, as illustrated in Figure 4 (a). We find that the entropy of context tokens typically converges in deeper layers, indicating that as information progressively aggregated across the layers, the model's 's predictive certainty consistently increases. However, when repetition arises under caching mechanisms (e.g.. dLLM-Cachh or SlowFast), the model tends to exhibit a highly random attention distribution, as illustrated in Figure 3 Moreover, the generated context tokens with repetition during decoding exhibit persistently abnormally high entropy in deeper layers, as shown in Figure We compute the per-layer entropy over multiple samples and report the averaged results in Figure 5 where the resulting statistics further validate the above findings. Together, these findings suggest that caching disrupts the inherent mechanisms of dMLLMs, thereby triggering the repeat curse. We attribute this phenomenon two key factors: 1. The introduction of caching disrupts the attention distribution and the inherent information flow patterns of context tokens; 2. Some context tokens whose entropy fails converge in deeper layers induce the model to generate uncertain tokens, which are often accompanied by repetition.
observe
Building these insights, we propose T plug-and-play approach that addresses the ab- normal patterns underlying the repeat curse and mitigates repetition. CoTA is built on two key components: (I) Context-token Attention Enhancement (CTAE): a distance-aware attention inter- vention that strengthens attention to context tokens, thereby preserving the intended information flow during token interactions; (2) Context-token
Voting (CTEV): a mechanism that leverages the aggregated deep-layer entropy of context tokens as penalty term in the confi- dence score, discouraging the model from generating uncertain and repetitive outputs. CoTA can be seamlessly integrated with bascline dMLLMs and existing caching strategies in training-frie free man- ner, while incurring only modest computational overhead. The experimental results demonstrate the effectiveness of CoTA in mitigating the repetition curse, as well as its robustness across different dMLLMS and caching methods. Moreover, across several multimodal benchmarks, CoTA consis- tently achieves performance gains over the baseline models, further validating its generalization. Our contributions are three-fold:
First, we identify the repeat curse that emerges when caching is applied in dMLLMs and uncover its underlying causes through information flow analysis.
'The name CoTA is derived from our key finding: Context Takens are Anchors.
2

---

Published as a conference paper at ICLR 2026
Aepeer
Cach
Lavga Taad (h)
Lua
MPADA
Figure 2: The statistical anal- ysis of the repeat curse on LLaDA-V and MMaDA.
Figure 3: The information-flow visualizations from left right cor-
respond to LLaDA-V, LaDA-V+dLLae Cache Liu et al. (2025b), and LLaDA LLaDA-V+Slow Wei et al. (2025a).
Second, we introduce CoTA, a plug-and-plaa method specifically designed to alleviate repetition. Third, we validate the effectiveness of CoTA through extensive experiments, demonstrating con- sistent performance improvements on general multimodal tasks.
2 RELATED WORK
dMLLMs and Caching Methods. The latest dLLMs Nie et al. (2025);; Zhu et al. (2025a); Ye
al. (2025a); Gong al. (2025) have been successfully scaled to 8B parameters, achieving perfor- mance comparable to state-of-the oftt-h--thhrrt autoregressive large language models (LLMs)
(LLLM))/raaane Grattafiori et al. 2002 ; DeepSeek-Al et al. (2024); Yang et al. (2025a). By introducing visual encoders Tschannen ct al. (202);; Bai al. (2025) into dLLMs and incorporating visual instruction tuning Liu et al. (2023a). recent studies You et al. 2025); Yang et al. (2025b); Li et al. (2025b); Yu et al. (2025) have successfully developed dMLLMs. Despite their strong performance across diverse tasks, high inference latency remains a major bottleneck, underscoring the need for effective acceleration meth- ods. In autoregressive models, KV caching Pope et al. (2023) has been widely adopted accelerate inference by reusing intermediate representations. However, due to the bidirectional, non-causal attention in diffusion models, conventional caching techniques are difficult to apply directly. Recent studies, such as Fast-dLLM Wu et al. (2025), Prefix-DLM. et al.) (2025b), ILLM--aahhr Liu et al. (2025b), and SlowFast Wei et al. (2025a). have successfully adapted the caching paradigm to dLLMs and dMLLMMs. This work reveals critical limitation of these caching approaches: the repeat curse.
Information Flow. Recent research on information flow has provided an interpretable approach for probing the internal mechanisms of black-box models. Common approaches mainly include saliency scores /Zhang et al.) (2025): Wang al.] (2023); |Huo et al. (2025); Zhuang al. (2025)
Deng et al. (2025), attention maps Tong et al. (2025b); HHang et al. (2023); Zhao et al. 2025ct:
Kang et al. 2025);; Qiu et al. (2025); Song et al. (2025a): Chen et al. (2025) Ye et al. Chen Grad-CAM Zhang et al. (2024b); Li et al. (2025c); Wang et al. (2025b); Xing al. (025 infor-
(025b),
mation entropy Yao et al. (2025): Nguyen et al. (0025). and massive values Jin et al. (2025); An
al. (2025): Kaul et al. (2025); Gan et al. (025a); Gu al. (2025); Gan et al. (225b)), among analysis, prior works have observed that, in autoregressive MLLMs, certain tokens as "anchors" receive disproportionately high atten- tion, and have further analyzed their impact on model reasoning Zhang et al. ) Chen et al. (2024a) and hallucination TTng al. (2025a); Kang et al. (2025). In contrast, this work presents the in-depth information-foom flow analysis of dMLLMs to uncover the underlying mechanism behind the repeat curse.
others Zou et al. (2025). Through information
Token Repetition in Generation. Early research on token repetition primarily focused on n-gram penalties Zhu et al. (2023), decoding strategies [Su al. (2022): Fu et al. (202T), and training- time optimization Lagutin et al.) (202T): Xu et al. (2022). More recent studies have extended the investigation to autoregressive LLMs Li et al. (2023b); Yang et al. (2024b); Ginart et al. 2025)). However, these approaches largely focus on mitigation rather than explaining the underlying causes of repetition. In contrast. DUC Yao et al. (2025) and model editing Wang et al. (2024) analyze the origins of repetition in LLMs from the perspectives of feature and neurons, respectively. To the best of our knowledge, we are the first to reveal the underlying causes of token repetition in dMLLMS from information-flor perspective.
3

---

Published as a conference paper at ICLR 2026
Cosss ortregy dotriburic somples)
No noticeable difference
Coaslty estopy suress of ofneest takens noreall resporuss
Signifieant difference!!
(b) Caus-layer entrppy surves of cottect tokens repeilion resuisscs
Figure 4: (a) and (b) respectively show the cross-layer en- tropy curves local context tokens under non-repetitive and repetitive cases. We observe that neighboring tokens with- out repetition exhibit a sharp decay in deep-layer entropy. whereas neighboring tokens with repetition tend to maintain persistently high entropy in deeper layers.
Figure 5: Statistical results of cross- layer entropy distributions across multiple samples (randomly selected from the COCO Val Lin et al. (2014)) We compute the mean en- tropy at each layer.
MOTIVATION AND ANALYSIS
In this section, we begin with brief overview of baseline dMLLMs and the cache mechanism D We then investigate the repeat curse through both quantitative and qualitative analyses. Finally, we compare the information flow in dMLLMs with and without cache, shedding light on the underlying cause of the repeat curse.
3.1 PRELIMINARY
Architecture of dMLLMs Typically, a dMLLM consists three main components: a pretrained vision encoder Fu,, dLLM Fr. and projector / that maps visual features into the text embed- ding space. Given an image imput La and an instruction prompt input 4p (c.g.. "Please describe the image in detail"), the model converts them into tokens and concatenates them into multimodal sequence: {Su,Sr where S. = ((F.(TT)) = {w:){= and S = F((p)) = {we; } represent vi- sual and instruction tokens of lengths V and T, respectively. Subsequently, fully masked sequence Sn = | where w; = [AAS]] and [MASK] denotes the special mask token, is initialized the starting state of the response sequence with maximum generation length M. Finally, Sm is concatenated with (S,, } to form the complete sequence S = (S,, St, J. which is fed into the dLLMs for iterative denoising. In this sequence, all tokens share the same dimensionallty.
Inference of d.MLLMs. Let S = St, denote the input sequence state at decoding step where N I,..., 1}. Starting from S S, where the response sequence Sm is fully masked, the model performs iterative denoising of St over N discrete steps to predict the final clean response sequence Sh. progressing from = N to k 1. At each step f, the model estimates a probability distribution po (8 | St) for each masked token based the noisy state St. Subsequently, the prediction of original clean sequence at step is obtained via greedy decoding as:
S/ = arg max pa (S)) = v S Vi (1,
(1)
eEV
where V is the token vocabulary. Although the model can predicts all masked tokens each step, to enable finer-grained decoding, dMLLMS adopt remasking strategy R to the unmasked tokens. Therefore, to obtain the next state St from St, an unmasking process is required (e.g., random or by confidence). Taking the confidence-based remasking strategy an example") the corresponding confidence scores Cis) based on S/.) can be obtained as:
(())
= po ( = S/ St),
It I
(2)
where It I represents the index set of masked token positions at step t. Finally, the model selects the k positions in It Tyask with the highest confidence scores 6() and obtains the set of indices It
Tupdate
We adopt I.LDDAVV You et al. 2025) and di M-C.Mee Liu al. 2025b as the haseline dMI.LMs and cache method.
'The discussion of the random remasking strategy is provided in the Appendix
4

---

Published as a conference paper at ICLR 2026
LLaDA-V, Layer I, Step
LLaDA-V, Layer 22, Step
MMaDA, Layer I, Step 1.
MMaDA, Layer 32, Step 1
LLaDA-V, Layer Step 128 LLaDA-V, Layer 22, Step 128
MMaDA, Layer 1, Step
MMaDA, Layer 32, Step 46
Figure 6: Context Tokens are Anchors. We visualize the attention maps of two dMLLMs, LLaDA- V and MMaDA, to analyze their information fllo patterns. The x-axis and y-axis correspond to key and query tokens, respectively, where brighter colors indicate higher attention weights during query-key interactions. For LLaDA-V, the maximum generation length is 128, while for MMaDA it 512. We present attention maps across different depths and decoding steps, with additional results provided in Appendix B We observe that neighboring context tokens act as anchors that aggregate information across layers and absorb disproportionate amount of attention.
to update. The l: selected tokens are unmasked at this step, producing the updated sequence:
if I updotee
())
(3)
otherwise.
())
The indices of the masked tokens at step f are updated by It-1 = I
It
update'
Caching in dMLLMs. During the iterative unmasking process of dMLLMs, the states of all tokens must be recomputed at each step, leading to significant inference latency. Unlike the causal attention in autoregressive models, which allows direct reuse of past KV states Xiao & et al. (2024). dMLLMs adopt bidirectional attention over all tokens, which fundamentally limits the reuse of intermediate states. Recent works Liu et al. (2025b); Huang et al. (2025); Wei et al. (2025a) reveal a similar ob- servation: the features of prompt tokens {S., S.) and parts of the output tokens S. across adjacent decoding steps exhibit high similarity, suggesting opportunities for token-state caching and reuse. Based on this insight, caching in dMLLMs can be organized according to token types follows:
Recompute (S St)), if mod Ep O,
Rccompute(S2,),.f if mod E, 0,
Recompute(S ), St argmin, (Sim 1)
where Recompute() denotes the recomputation. For prompt tokens, their states are recomputed every Ep steps, and the cached states are reused within each iteration interval. For output tokens, the set of tokens to be recomputed is determined according to two principles: First, similar to prompt caching, token states are recomputed periodically according to a fixed interval &g, Second, the cosine similarity between token features at adjacent steps is computed via Sim(), and the bottom proportion of tokens, whose feature vectors vary most significantly, are selected for recomputation
Unless otherwise specified, all hyperparameters are fixed to 0.25, E, 25, and E. 7 to ensure fair comparison.
5

---

Published as a conference paper at ICLR 2026
Hor Conte- Wiphrdiecah
Table 1: Ablation study on cache mechanism design components.
Interval
5
Interval
SRR. 0 0
SRRI
0 79.9 89.7
(a) Prompt interval.
(b) Output interval.
Threshold
0.25
0.75
Policy
Prefix
dLLLM Cache
SKRI
29.7
SKR)
Figure 7: Radar charts of the repeat curse, where L and S denote long and short responses, respectively.
(c) Similarity thresholds.
(d) Reuse policies.
3.2 REPEAT CURSE
Quantitative Metries. As illustrated in Figure [ we define the redundant tokens repetition in model responses as the Repeat Curse. To quantitatively evaluate this phenomenon, we introduce the Adjacent Repetition Rate (ARR), which measures the proportion of repeated tokens within response sequence {y: J of lengths M:
M-1 1(y
1
ARR
v++1)
M
where 1() an indicator function that takes the value when the condition satisfied and 0 other- wise. In addition, we adopt the sample repetition rate (SRR) to quantify the proportion of samples exhibiting repetition the test set:
SRR- NR/N,
(6)
where N denotes the total number of samples in the test set, and NR denotes the number of sam- ples whose responses contain repeated tokens (i... ARR > 0). To further assess the severity of repetition, we additionally introduce three metries: maximum repetition length (MRL), average rep- etition length (ARL), and the 95th-percentile repetition length (95pRL... The detailed computation procedures are presented in Appendix D
Quantitative Analysis of Repeat Curse. Figure 7 presents the quantitative results of the Repeat Curse two dMLLMs under both long-response and short-response settings. The experiments conducted on a VQA task using 500 randomly sampled images from MSCOCO et al. (2.14)) We observe that, after applying two advanced caching techniques, both LLaDA-V and MMaDA exhibit catastrophic token repetition. Furthermore, we ablate three key components of the cache mechanism (prompt recomputation interval, output recomputation interval, and similarity threshold) analyze the underlying causes of repetition. As shown in Table aa), starting with the prompt token recom- putation interval while fixing the output token recomputation interval to and setting the similarity threshold to 0, we observe that periodically caching prompt tokens has negligible impact repeti- tion. Next, by fixing the prompt interval to 25 and the similarity threshold to we find that varying the output interval leads to severe token repetition (Table In contrast, increasing the similarity threshold gradually mitigates the repetition (Table cc). Finally, we compare dLLM-Ca with prefix KV cache Li et al. (2025b), which reuses only prompt tokens, analyze the impact of dif- ferent reuse policies. Across all experiments, we find that token repetition is primarily driven by the caching of output tokens, with more comprehensive analysis provided in Appendix E.2
3.3 INFORMATION FLOW ANALYSIS
As mentioned earlier, applying caching techniques leads severe repeat curse in dMLLMs. To uncover the underlying mechanisms driving repeated token generation, we analyze and interpret this phenomenon from information-fio perspective. For each output token in the sequence, we visualize its attention matrix to analyze the information-flow patterns among tokens during the interaction process.
As shown in Figure 6) neighboring context tokens consistently receive high attention throughout the decoding process. The positions which these tokens attract high attention remain stable across lay-

---

Published as a conference paper at ICLR 2026
and decoding steps, predominantly corresponding adjacent query-key pairs. This information- flow pattern is consistently observed across different dMLLMs, including LLaDA-V and MMaDA. Interestingly. this behavior resembles the "attention sink" phenomenon reported in autoregressive models Chen et al. (2024)); Wang et al. (2023) /Zhag et al. (2024b), where certain special tokens anchors that aggregate information and absorb disproportionately high attention. Under the bidirectional attention mechanism of dMLLMs, these context tokens similarly function as anchors that guide and shape the model's information interactions.
Finding 1: In dMLLMs, context tokens serve as anchors that aggregate information and ab- sorb attention across layers and guide the final prediction.
We further plot the cross-layer entropy curves of neighboring context tokens under normal decoding, as shown in Figure 4aa). We observe high entropy in the shallow layers, followed by a gradual convergence depth increases-with a particularly sharp drop around layers 26-30. This indicates that as information is aggregated across layers, the model's prediction for the final output becomes increasingly stable and confident. This observation is also consistent with findings reported in prior studies Ali et al. 2025).
For context tokens under normal decoding, the information entropy gradually con- verges in the deeper layers.
Finding 2:
However, after applying caching, the model exhibits patterns that differ markedly from the above observations, shown in Figure FigureBand and Figure b). Specifically, caching leads randomized at- tention distribution, disrupting the original information flow from context tokens. Moreover, context tokens involved in repetition exhibit persistently high entropy in deeper layers, failing converge as normal context tokens do during normal tokens decoding.
Finding 3: Repetition typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers.
4 METHOD
Building on the findings in Section 3] we propose CoTA, a training-free method to mitigate repeat curse, which has two key components: (1) Context tokens attention enhancement, which preserves the intrinsic information flow pattern of context tokens, and (2) Context tokens entropy entropy-guided vot- ing, which prevents the decoding of uncertain tokens arising from non-convergent deep-layer en- tropy in context tokens.
4.1 CONTEXT TOKENS ATTENTION ENHANCEMENT (CTAE)
As analyzed in Section 3 context tokens in dMLLMs typically serve as anchors to aggre- gate information and absorb attention, but the use cache disrupts this information flow pattern. We hypothesize that tokens in close proximity exhibit stronger semantic correla- tions; therefore, this context--oeen-driven at- tention pattern arises naturally to promote lo- cal semantic coherence and prevent abnor- mal repetition caused by attention being as- signed to random tokens. Therefore, we pro- pose Context Tokens Attention Enhancement (CTAE), as shown in Figure g. whose key idea is to introduce distance-based decay term that intervenes in the attention matrix element-wiee Given query token 9, at posi- tion and a key token ky at position j. the de- cay term Gi for each query-key pair is com-
Contest anchees
Randors
Gameian
M
information
from contest telons
dease infortattion flow from random inkoua
Figure 8: Illustration of context tokens attention enhancement. Example values are annotated on the decay matrix for clarity, while actual values are computed using Equation Manioon//nnd and The workflow of the CTAE algorithm is provided in Appendix E.4

---

Published as a conference paper at ICLR 2026
puted as follows:
91,5 = Ymin + (1 - -Yminnga
Vunin E (0, 1].
(7)
- ( (1(44)
9 = exp
exp
(8)
where Pijj denotes the Gaussian decay term computed by the exponential function exp() and the relative position distance V | between query-key pairs. T denotes the temperature factor, which fixed to 5 in the experiments. We introduce lower-bound constant Ymin for stabilization, yielding the final decay term G... The G. values for all query-key pairs together form the Gaussian decay matrix shown in Figure 8 Assume the attention between a query token di and key token ky is jtt We enhance the attention to context tokens by applying Attng,j * Gi.j. thereby preserving the native information flow pattern of context tokens in dMLLMs.
4.2 CONTEXT TOKENS ENTROPY-GUIDER VOTING (CTEV)
As observed in Section neighboring con- text tokens associated with repetition typ- ically exhibit persistently high entropy in deeper layers, reflecting the model's uncer- tainty toward these tokens during decoding. Yet baseline dMLLMs rely solely on confi- dence scores to vote for candidate decoding
Algorithm I Context Tokens Entropy-Guided Voting
Require: Deep-layer logits z!." base confidences c(j). co- efficient a
1:
for i 1 to M do
2:
Esum (i) +0 for E (26,....
3:
4:
Pe (1) exp(= EV exp(= (,u
tokens, ignoring this uncertainty. To address this, we propose Context Tokens Entropy-
5:
Exum (i) t- Esum (i) Tog
Ev Po log !))
Guided Voting (CTEV), which incorporates the aggregated deep-layer entropy of context
6: end for
7: end for
tokens as penalty term on the confidence score to prevent decoding under uncertain context tokens. The complete algorithm is formalized Algorithm ??. First, we com- pute the entropy E of each candidate token based on its softmax probability distribution Pa follows:
8: Build C(t) = {t} U fcontext tokens of candidate t
9:
for C(t) do
10: Eetxx (1) E Eeum (
Score()) +- ((3 + a Egai (i)
12: end for
13: return context tokens entropy-guided voting score
EVE Py log Pv
exp(zx) Eu=i exp(zu)
E
Pe
log V
where denotes the logits of the v-th word in the vocabulary, and V is the vocabulary size. On this basis, the entropy of candidate decoding tokens deeper layers is accumulated layer by layer as follows:
30
30
v= Pe logpe"
Exun
ΣE
E
(10)
log V
L=26
26
Let Esum (i) denote the Esum of token x, position in sequence of length M. The accumulated decp-layer entropy of the context tokens, Entr (i). , is then computed as follows:
Estz (i)
Σ
Esum()),
(11)
JEC(()
c(i)= [2.JU 23
| je arg ming lj -il }
(12)
(1))([M]
Finally, the weighted Este (i) with coefficient is added into the confidence score (() from Equa- tion resulting in the new candidate tokens voting score Score(i) as follows:
Score(i) ((3) Fettz (i)-
(13)
'ee define layers 26-30 the deep layers, and candidate tokens together with two nearest tokens as the context tokens.
8

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1
EXPERIMENTAL SETUP DETAILS
Our baseline dMLLMs are LLaDA-V You et al. (2025) and MMADA Yang et al. (2025b), and the baseline caching method is dLLM-Cache Liu al. (2025b). The comparison methods include Fast- dLLM Wu et al. (2025), prefix KV Cache Li ct al. (2025)), and SlowFast Wei et al. (2025a). The repeat curse evaluated on the VQA task of the COCO dataset Lin et al. (2014). while general capa- bilities are assessed on DocVQu Mathew et al. (2020), ChartQA et al. (2022), MME Fu et al. (025), Seed Li et al. (2023a), MMBench Liu et al. (202), MMMU Yue et al. (2024). LLa
Liu
et al. 2023a) and Math Vista Lu et al. 2024).
Table 2: Repeat Curse Evaluation Results. 512 and 64 denote the maximum generation length.
Table 3: Repeat curse across different settings.
Method
64
Method
ARRL
SRR.
Method
ARR1
SRRJ
ARRI
SKR
Je1
ARKL
SRRL
Ien
MMLDA
LL4DA-V
LLADA V
dLLM Cloke
JLLLM-Cache
ALLMM-Cache+ CCAA
14.3
0.2
K0.3
10.6
6.9
5.0
438.5
43010
404.2
0.1
7.1
05.6
3.3
02.1
61.4
1 4LLMI Cucbe i CoTA
-LLM--Caaee
M--. CTEN
dLLM--Cucee CTAE
08
0.6
35.0
30.0
Slow
Slow e Coin
Slo + CTTV
Sloo CTAE
1.4
199
5.6
MLLMM-Cacte CTAAE
(a) Different dMLLMs.
(b) Different cache methods.
Table 4: Comparison with additional caching methods.
Method
ARR4
SRRJ
Method
ARR_
SRR.
LLaDA-V
0.7
6.0
MM&DA
0.7
6.0
+Prefix-KV
0.6
5.0
+Prefix KV
0.7
6.0
+D*
1.2
8.0
LaViDa
+FasdLLM
0.7
6.0
+Picfix-KV
0
5.2 RESULTS ON REPEAT CURSE EVALUATION
Sableepresentte the effectiveness of our method in mitigating the repeat curse on VQA captioning task using 500 randomly sampled images from COCO Lin et al. (2014). SRR measures the likeli- hood of repetition occurring, while ARR evaluates its severity. We report the mean values of both metrics across samples. The experiments reveal catastrophic Repeat Curse after applying caching techniques, with long-text responses being particularly prone to repetition. The results demonstrate that our method, CoTA, along with its two key components, CTAE and CTEV, effectively mitigates the repetition issues introduced by caching. Specifically, under the long-response setting, CoTA reduces ARR and SRR by 96% and 92%, respectively; under the short-response setting, it reduces ARR and SRR by 85% and 95%, respectively. Furthermore, we report the Repeat Curse results on MMaDA. As shown in Table a)). our method exhibits strong generalization ability, achieving improvements of 86% and 45% on ARR and SRR, respectively.
To evaluate the generalizability of CoTA across different caching methods, we further test its performance on SlowFast. As shown in Table 3(b), the CoTA, along with its components CTAE and CTEV, demonstrates consistent effectiveness. However, we find that some caching methods, such as ToM, Fast-dLLM, and Prefix KV Cache, do not introduce the Repeat Curse, as shown in Table By comparing these methods, we identify a common characteristic: they almost not cache the states of output tokens. Prefix KV Cache periodically caches prefix tokens (the prompt part) while recomputing the states of output tokens at every step. This setting corresponds to the experimental configuration in Table 1(a) and is consistent with the conclusions drawn in Section 3.2. DoM builds upon Prefix KV Cache by introducing visual-token merging, which also applics only the prompt portion. In contrast, Fast-dL_LM recomputes the states of all output tokens after the block's 's starting position at every decoding step.
repetition-mitigen
5.3 GENERALIZATION AND EFFICIENC STUDY
We report the generalization and efficiency analysis of CoTA in Table Our method demonstrates consistent performance improvements and repetition mitigation across open-domain natural and mathematical tasks. When the baseline LLaDA-V employs caching techniques, it achieves sub- stantial efficiency gains; however, it also exhibits the Repeat Curse and drop in accuracy on both

---

Published as a conference paper at ICLR 2026
Table 5: Generalization and Efficiency Analysis of CoTA.
Method
LLaVA m
Vista
Scoret
ARR4
TPSt
FLOPs.
Scoret
ARR
TPSt
FLOPs
LLaDA-V
70.1
1.3
7.3
16.1
59.7
8.3
14.1
Ours
70.5
1.1
5.1
17.8
59.8
7.5
15.3
LLaDA-V w/ dLLM-Cache
63.2
7.3
23.1
3.2
54.9
6.9
31.0
3.9
Ours w/ dLLM-Cache
69.9
1.4
20.3
5.1
59.7
1.3
28.7
5.0
benchmarks. In contrast, our method effectively overcomes this limitation of caching, yielding im- provements of 11% in Score and 80% ARR on LLaVA and 9% Score and 81% ARR on Vatta For efficiency evaluation, tational overhead: on LLaVA", Tokens Per Second (TPS) decreases by 2.8, while Floating Point Operations (FLOPs) increase by 1.9.
method introduces only small and acceptable compu-
5.4 ABLATION AND QUALITATIVE STUDY
Table 6 presents the ablation study of the hy- perparameters involved in COTA, including the weighting factor in Equation 13 the minimum gain Ymin in Equation 7 and the number of con- text tokens. CoTA achieves its best performance when = 0.75, Tmin 0.5, and the number of context tokens is set 3, yielding an ARR of 1.2% and an ACC of 23.1. We also report more detailed ablation analyses of CTEV and CTAE Appendix E.4 Figure
9 illlstrrtes the impact of CTAE on the model's intrinsic information -llow pattern. After applying caching, the model exhibits random attention distribution, whereas CoTA restores the "context tokens as anchors" pattern by strengthening attention toward contextual tokens.
Figure The intervention effect of CTAE.
Table Ablation results on hyperparameters evaluated on the Math Verse benchmark.
7mmi
ARR
ACC1
Tmin
ARR
ACC1
7min
ARR
ACC1
0.25
2.8
21.4
0.25
2.1
22.1
0.25
3.5
19.6
1
0.5
2.2
21.9
0.5
2.0
22.5
0.5
3.8
19.2
0.75
2.3
22.1
0.75
1.7
22.3
0.75
3.1
19.2
0.25
2.9
20.0
0.25
1.6
22.2
0.25
3.9
19.0
0.5
0.5
2.5
20.2
0.5
0.5
1.4
22.0
0.5
0.5
3.4
18.2
0.75
2.1
20.1
0.75
1.2
23.1
0.75
3.3
19.1
(a) Context tokens number
(b) Context tokens number =
(c) Context tokens number =
6 CONCLUSION AND LIMITATIONS
From an information-Aee perspective, this work reveals the underlying reasons why caching tech- niques in dMLLMs lead to repeated token generation, known as the Repeat Curse: (1) caching output tokens disrupts the intrinsic "context tokens as anchors' information-flow pattern of dM- LLMs, resulting in randomized attention distributions: and (2) neighboring context tokens exhibit persistently high deep-layer entropy during decoding, causing the model to generate uncertain and repeated outputs. Building igate the repetition. Due to the limited availability of baseline dMLLM architectures and caching methods, CoTA has not yet been validated broader range of models and caching strategies.
these insights, we propose CoTA, a plug-and-play approach to mit-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was conducted at the Fujian Key Laboratory Pattern Recognition and Image Under- standing, Xiamen University of Technology. It was supported by the National Natural Science Foundation of China (Grant No. 62576301) and the Major Science and Technology Plan Project on Future Industry Fields of Xiamen City (Grant No. 3502220241027).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Riccardo Ali, Francesco Caso, Christopher Irwin, and Pictro Liò. Entropy-lens: The information signature of transformer computations, 2025.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Systematic outliers in large language models. In The Thirleenth International Conference on Learning Representations, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing 2021.
systems,
Shuai Bai, Peng Wang. Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409.12199 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, et al. Qwen2.5-vl technical report, 2025.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 1692- 1717. PMLR, 23-29 Jul 2023.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Amaud Doucet. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022.
Shuochen Chang, Xiaofeng Zhang, Qingyang Liu, and Li Niu. Dotom: socider-guided dynamic token merging for accelerating diffusion milmss arXiv preprint arXiv:25/1.12280, 2025.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference Computer Vision, 2024a.
Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations, 2023.
Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Helen" Li, and Yiran Chen. Dpad: Efficient diffusion language models with suffix dropout, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See htps://vvicuaa Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
11

---

Published as a conference paper at ICLR 2026
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models, 2023.
DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, et al. Deepseek llm: Scaling open-source
language models with longtermism, 2024.
Jinhong Deng, Wen Li, Joey Tianyi Zhou, and Yang He. SCOPE: Saliency-coverage oriented to- ken pruning for efficient multimodel LLMs. The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 4171-4186. Association for Computational Linguistics, 2019.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. MME: A comprehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12848-1222 12856, 2021.
Chaofan Gan, Yuanpeng Tu, Xi Chen, Tieyuan Chen, Yuxi Li, Mehrtash Harandi, and Weiyao Lin. Unleashing diffusion transformers for visual correspondence by modulating massive activations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a.
Chaofan Gan, Zicheng Zhao, Yuanpeng Tu, Xi Chen, Ziran Qin, Tieyuan Chen, Mehrtash Harandi, and Weiyao Lin. Massive activations are the key to local detail synthesis in diffusion transformers, 2025b.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Antonio A. Ginart, Naveen Kodali, Jason Lee, Caiming Xiong, Silvio Savarese, and John R. Em- mons. Lz penalty: An inforeition-thhorrticc repetition penalty for autoregressive language mod- els, 2025.
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learming Representations, 2023.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, et al. The llama 3 herd of models, 2024.
12

---

Published as a conference paper at ICLR 2026
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks, 2025.
Xiangming Gu, Tianyu Pang. Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025.
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood diffusion language models. Ad- vances in Neural Information Processing Systems, 36:116693-6715, 2023.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann. Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Haiyang Guo, Fanhu Zeng. Ziwei Xiang, Fei Zhu, Da-Han Wang. Xu- Yao Zhang, and Cheng-Lin Liu. HiDe-LLaVA: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 13572-13586. Association for Computational Linguisties, July 2025.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 11575-11599 11596, 2023.
Guangxin He, Shen Nie, Fengqi Zhu, Yuankang Zhao, Tianyi Bai, Ran Yan, Jie Fu, Chongxuan Li, and Binhang Yuan. Ultrallada: Scaling the context length to 128k for diffusion large language models, 2025.
Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuan-Jing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Pro- ceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 4521-4534, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances Neural Information
Processing Systems, 2021.
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S. Abdelfattah, Jae sun Seo, Zhiru Zhang, and Udit Gupta. Flashdlm: Accelerating diffusion language model inference via efficient kv caching and guided diffusion. 2025.
Jianuo Huang, Yaojie Zhang, Yicun Yang, Benhao Huang, Biqing Qi, Dongrui Liu, and Linfeng Zhang. Mask tokens as prophet: Fine-grained cache eviction for efficient dllm inference, 2025.
Qidong Huang, Xiao wen Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weim- ing Zhang, and Neng H. Yu. Opera: Alleviating hallucination in multi-modal large language mod- els via over-trust penalty and cetispecctionallloaatiee 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13418 13427, 2023.
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large langugee models. In The Thir- teenth International Conference on Learning Representations, 2025.
Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du Zirui Liu, and Yongfeng Zhang. Massive values self-attention modules are the key to contextual knowledge understanding. In Forty-second International Conference on Machine Learning, 2025.
13

---

Published as a conference paper at ICLR 2026
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual at- tention sink in large multimodal models. In The Thirteenth International Conference on Learning Representations, 2025.
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, and Arman Cohan. TESS: Text-to-text conditiond simplex diffusion. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2347-2361. Association for Computational Linguistics, March 2024.
Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. From attention to activation: Unraveling the enigmas of large language models. In The Thirteenth International Conference Learning Representations, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Evgeny Lagutnn, Daniil Gavrilov, and Pavel Kalaidin. Implicit unlikelihood training: Improving neural text generation with reinforcement learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1432- 1441, 2021.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li. Ziwei Liu, and Chumyuan Li. ALLVA-ooevisionn Easy visual task transfer. Transactions on Machine Learning Research, 2025a. ISSN 2835-8856.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. ArXiv, 0os/2307.16125, 2023a.
Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. In Thirth-seventh Conference on Neural Information Processing Systems, 2023b.
Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: A large diffusion model for vision-language understanding. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Yi Li, Hualiang Wang, Xinpeng Ding, Haonan Wang, and Xiaomeng Li. Token activation map to visually explain multimodal 1ims. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 48-58, October 2025c.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang- Dual diffusion for unified image generation and understanding, 2025d.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. ICMI., pp. 21051-21064, 2023a.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In International Conference on Machine Learning, pp. 21051-21064, PMLR, 2023b.
14

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023b.
Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Longllada: Unlocking long context capabilities in diffusion Ilms, 2025a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024.
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelcrating diffusion large language models with adaptive caching. ArXiv, abs/2506.06295, 2025b.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q. Weinberger. Latent diffusion for language generation. In NeurlPS, 2023.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dKV-cache: The cache for diffusion language models. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025a.
Yiyang Ma. Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang. Xingkai Yu, et al. Janusflow: Harmonizing autoregression and rec- tified flow for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7739-7751, 2025b.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pp. 2263-2279, 2022.
Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. 2021 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1199-2208, 2020.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34545, 2022.
Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman. Beyond semantic entropy: Boosting LLM uncertainty quantification with pairwise semantic similarity. In Annual Meeting of the Association for Computational Linguistics, 2025.
Shen Nie, Fengqi Zhu, Chao Du Tianyu Pang. Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations, 2025a.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In The Thirty-ninte Anmual Conference on Newral Information Processing Systems, 2025b.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference Learning Representations 2025.
15

---

Published as a conference paper at ICLR 2026
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolttion image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, volume 5, pp. 606-624, 2023.
Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and iirention-sink-feeee In The Thirty- ninth Anmual Conference on Neural Information Processing Systems, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2022.
Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Anchored diffusion language model. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103162 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. pmlr, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Sparse-dllm: Accelerating diffusion lims with dynamic cache eviction, 2025a.
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang. Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference, 2025b.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:2211.044 2022.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Proceedings of the 36th International Conference on Neural Information Processing Systems. Curran Associates Inc., 2022.
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragki- adaki. Unified multimodal discrete diffusion, 2025.
Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim, Intervening anchor token: Decoding strategy in alleviating hallucinations for MLLMs. In The Thirteenth International Conference on Learning Representations, 2025a.
Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zelin Peng. Zhiwei Yang, Jionglong Su, Minquan Lin, Yifan Peng, Xuelian Cheng, Imran Razzak, and Zongyuan Ge. Seeing far and clearly: Mitigating hallucinations in milms with attention causal decoding. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26147-26159, 2025b.
Runchu Tian, Junxia Cui, Xueqiang Xu, Feng Yao, and Jingbo Shang. Finish first, perfect later: Test-time token-level cross-validation for diffusion large language models, 2025.
Bingkui Tong, Jiaer Xia, and Kaiyang Zhou. Mitigating hallucination multimodal Ilms with layer contrastive decoding, 2025a.
16

---

Published as a conference paper at ICLR 2026
Jintao Tong, Wenwei Jin, Pengda Qin, Anqi Li. Yixiong Zou, Yuhong Li, Yuhua Li, and Ruixuan Li. Flowcut: Rethinking redundancy via information flow for efficient vision-language models. In The Thirty-minth Annual Conference Neural Information Processing Systems, 2025b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13377, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, et al. Llama 2: Open foundation and fine-tund chat models, 2023b.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim M. Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H'enaff, Jeremiah Harmsen, Andreas Steiner, and Xiao-Qi Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features. ArXiv, abs/2502.147886 2025.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 9840-9855, 2023.
Weichuan Wang, Zhaoyi Li, Defu Lian, Chen Ma, Linqi Song, and Ying Wei. Mitigating the lan- guage mismatch and repetition issues in Ilm-based machine translation via model editing. In EMNLP, pp. 15681-15700, 2024.
Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, and Chunhua Shen. Time is a feature: Exploiting temporal dynamics diffusion language models, 2025a.
Yuji Wang, Jingchen Ni, Yong Liu, Chun Yuan, and Yansong Tang. Iterprime: Zero-shot referring image segmentation with iterative grad-cam refinement and primary word emphasis. Proceedings of the AAAI Conference on Artificial Intelligence, 39(8):81 19(8)8155-8116) 59-88168, Apr. 2025b.
Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang, Accelerating dif- fusion large language models with slowfast: The three golden principles. arXiv preprint arXiv:2506. 10848, 2025a.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. VidcoroPE: What makes for good video rotary position embedding? In Forty-second International Conference on Machine Learn- ing, 2025b.
Scan Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Rep- resentations, 2020.
Zichen Wen, Jiashu Qu, Dongrui Liu, Zhiyuan Liu, Ruixi Wu, Yieun Yang, Xiangqi Jin, Haoyun Xu, Xuyang Liu, Weijia Li, Chaochao Lu, Jing Shao, Conghui He, and Linfeng Zhang. The devil behind the mask: An emergent safety vulnerability of diffusion 1lm,, 2025.
Chengyue WU, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-fres acceleration of diffusion Ilm by enabling kv cache and parallel decoding. ArXiv, abs/2505.22 222611 2025.
Guangxuan Xiao and et al. Efficient streaming language models with attention sinks. In Interna- tional Conference on Learning Representations (ICLR), 2024.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijic Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In The Thirteenth International Conference Learning Representations, 2025a.
17

---

Published as a conference paper at ICLR 2026
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream-coder 7b: An open diffusion language model for code, 2025b.
Xiaoying Xing, Chia-War Kuo, Li Fuxin, Yulei Niu, Fan Chen, Ming Li, Ying Wu, Longyin Wen, and Sijie Zhu. Where do large vision-language models look at when answering questions?, 2025.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: analyzing and mitigating repetitions for neural text generation. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, JUN ZHOU, and Chongxuan Li. Unifying bayesian flow networks and diffusion models through stochastic differential equations. In Forty-first International Conference Machine Learning, 2024.
An Yang, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, et al. Qwen2.5 technical report. CoRR, (bs/2442.15115, 2024a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al. Qwen3 technical report, 2025a.
Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. A frustratingly sim- ple decoding method for neural text generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics. Language Resources and Evaluation LLLCCCOLING 2024), pp. 536-557, 2024b.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. MMaDA: Multimodal large diffusion language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Yicun Yang, Cong Wang, Shaobo Wang, Zichen Wen, Biqing Qi, Hanlin Xu, and Linfeng Zhang. Diffusion IIm with native variable generation lengths: Let [eos] lead the way, 2025c.
Junchi Yao, Shu Yang, Jianhua Xu, Lijie Hu, Mengdi Li, and Di Wang. Understanding the repeat curse large language models from feature perspective. Annual Meeting of the Association for Computational Linguistics, 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng github wt/bil//loo/o 2205 /dream
Kong. Dream 7b, 2025a. URL
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. Diffusion language models can perform many tasks with scaling and cestuucion-fiietunige ArXiv, abs/2308. 12219, 2023.
Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-fres free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 22128-22136. 2025b.
Koon-Ting Yip, Qiyan Zhao, Wenhao Yu, Liangyu Yuan, Mingkai Li, Xiaofeng Zhang, Jianmin Ji, Yanyong Zhang, Qing Jiang, and Ka- Veng Yuen. Sope: Spherical coordinate-based positional embedding for enhancing spatial perception of 3d Ivlms. CVPR 2026, 2026a.
Koon-Ting Yip, Qiyan Zhao, Wenhao Yu, Xiaofeng Zhang, Jianming Ji, Yanyong Zhang, and Ka- Veng Yuen. C'2rope: Causal continuous rotary positional encoding for 3d large multimodal- models reasoning. In 2026 IEEE International Conference on Robotics and Automation (ICRA), 2026b.
Itay Yona, Ilia Shumailov, Jamie Hayes, and Yossi Gandelsman. Interpreting the repeated token phenomenon in large language models. In Fecty- International Conference Machine Learning. 2025.
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu. Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint 2xiv:2505.16933 2025.
18

---

Published as a conference paper at ICLR 2026
Rumpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding, 2025.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Un- veiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In Forty-first International Conference on Machine Learning, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9556-9567, 2024.
Ruixiang ZHANG, Shuangfei Zhai, Yizhe Zhang, James Thornoon, Zijing Ou, Joshua M. Susskind, and Navdeep Jaitly. Target concrete score matching: holistic framework for discrete diffusion. In Forly-second International Conference on Machine Learning, 2025.
Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in Ivlms. arXiv preprini ar arXiv:2411 1.0996,, 2024a.
Xiaofeng Zhang. Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Information flow in Ivims across reasoning tasks. In North American Chapter of the Association for Compu- Iutional Linguistics, 2024b.
Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. The 39th Annual AAAI Con- ference on Artificial Intelligence, 2024c.
Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Xiaosong Yuan, Qiyan Zhao, Jiawei Cao, Feilong Tang, Sinan Fan, Yaomin Shen, Chen Shen, et al. Hallucination begins where saliency drops. In The Fourteenth International Conference on Learning Representations (ICLR' 26 Oral), 2026.
Yihua Zhang, Changsheng Wang, Yiwei Chen, Chongyu Fan, Jinghan Jía, and Sijia Liu. The fragile truth of saliency: Improving Ilm input attribution via attention bias optimization. In The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025.
Qiyan Zhao, Lanying Liang, Xiaofeng Zhang, Tiange Zhang, Jiuze Li, and Yuefeng Liu. Few-shot modal text detection via clip. The Visual Computer, pp. 1-15, 2025a.
Qiyan Zhao, Yue Yan, and Da-Han Wang. Textmamba: Scene text detector with mamba. In 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2025b.
Qiyan Zhao, Xiaofeng Zhang, Yiheng Li, Yun Xing, Xiaosong Yuan, Feilong Tang, Sinan Fan, Xuhang Chen, Da-Han Wang, and Yao Zhang. Mca-llava: Manhattan causal attention for reducing hallucination in large vision- languala models. In Proceedings of the 33rd ACM Inter- national Conference on Multimedia, pp.
2025c.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In The Thirteenth International Conference on Learning Representations, 2025.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.05737. 2023.
Chunting Zhou, LILI YU, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Ja- cob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-moda. model. In The Thirteenth International Conference on Learning Representations, 2025.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu. Jun Hu, Jun Zhou, Jianfei Chen. Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models, 2025a.
19

---

Published as a conference paper at ICLR 2026
Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang, Xudong Wang, Lanning Wei, Hongrui Jiqii Hu, Wentao Ye, Tieyuan Chen, Chenchen Li, Chengfu Tang, Haibo Feng, Jun Hu, Jun Zhou, Xiaolu Zhang, Zhenzhong Lan, Junbo Zhao, Da Zheng, Chongxuan Li, Jianguo Li, and Ji-Rong Wen. Llada-moe: A sparse moe diffusion language model, 2025b.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, et al. Internv13; Exploring advanced training and test-time recipes for open-source multimodal models, 2025c.
Wenhong Zhu, Hongkun Hao, and Rui Wang. Penalty decoding: Well suppress the self- reinforcement effect in open-ended text generation. In The 2023 Conference Empirical Meth- ods in Natural Language Processing, 2023.
Xianwei Zhuang, Zhihong Zhu, Yuxin Xie, Liming Liang, and Yuexian Zou. Vasparse: Towards efficient visual hallucination mitigation via visual-aware token sparsification. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Xin Zou, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijie Jiang, Jia Liu, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. In Forty-second International Conference on Machine Learning, 2025.
20

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Diffusion Language Models. Motivated by the remarkable success of diffusion models in image generation Ho ct al. 20200; Rombach Rombach ct al. (2022); Podell ct al. (2024); Esser et al. (2024); [Sohl]
Dickstein et al. (2015); ; Song et al. (2021). recent studies have introduced the diffusion process into
language modeling. including both continuous diffusion models Lovelace et al. (2023): : Li et al.
2002
ZHANG et al. (2025); ; Xue et al. (2024); Lin al. (2023a); Gong et al. (2023); Han et al.
2023
Strudel et al. ;022): Dieleman et al. (2022] Chen et al. (2023): Karimi Mahabadi et al.
22024
Graves et al. ( Lin et al. (2023b); Gulrajani & Hashimoto (023) and discrete diffu-
sion models|Rou et al. (2025 3CCampeel et al. (2022) Zheng et al. (2025): Gat et al. a (2024); Ye et al.
(2023); Zheng ct al. (2023); Austin et al. (2021); Hoogeboom et al. (2021 1) He et al. (2023) Meng
ct al. (2022); Shi et al. (2024); Ou et al. (2025); : Nie et al. (2025a); Gong al. (2025). 5) The latest
masked diffusion models Nie et al. (025b): /huu al. (025a): Ye et al. (2025a): Gong et al. (0255) )
He et al. (025): ; Liu al. (2025a) have successfully scaled dLLMs to 8B parameters and demon-
strated performance comparable to autoregressive large language models Touvron et al. (2023a b):
Devlin et al. 2019); Grattafiori al. (2024); DeepSeek-AT et al. (2024): Yang et al. (022a)) Gu-
nasekar et al. 20033: : Chiang et al. (2023).
Multimodal Large Language Models (MLLMss). Building upon the unprecedented generative
pabilities of large language models (LLMs Touvron et al. (2023a Devlin et al.] (2019): Grattafiori
al. 2024); DeeepSeckAA et al.] 2024 Yang et al. (2024a): Gunasekar
al. (2023); Chiang
et al. 2023). MLLMs have emerged as powerful systems that extend LLM architectures to process
over multimodal inputs. Probabilistic modeling approaches for MLLMs generally fall into three
paradigms: (i) autoregressivéL et al. (2023b) a): Bai et al. (2025) 2024); (Zhu et al. (2025c); Chen
(2025a): Guo et al. (2025); Zhang et al. (2024a); Wei cet al. 2025b); Zhang et al. ( Yip et al. (026b); Zhao et al. (2025b a Yip ct al. (2026a). (I)
jet al. 2024b);; Achiam et al. (2023); Li et al.
autoregressive diffusion hybrid [ao et al. (2023 Xie et al. (2025a); Zhou et al. (2025); Ma et al. (2025b): Yu et al. (2025), and (ii) pure diffusion Swerdlow al. (2025); Li et al. (202dd) Building
on the recent breakthroughs in dLLMs, the latest dMLLMs You et al. (2025): Yang et al. (2025b)
iet al. (2025b) exploit the language modeling capabilities of dLLMs, coupled with effective training pipelines, to deliver performance on par with leading autoregressive and hybrid counterparts.
Diffusion-based Multimodal Large Language Models (dMLLMs). The latest diffusion-basee
large language models (dLLMs) Nie et al. (2025b); Zhu et al.] (2025a); ; Ye et al. (2025a); [Gong al. (2025); He et al. (2025); Liu et al (2025a); Yang et al. (2025c); Wen et al. (2025); Xie et al. (2025b) Zhu et al. (2025b): Tian et al. (2025); Dieleman et al. (2022) have been successfully scaled to 8B parameters, achieving performance comparable to of art autoregressive large lan- guage models Touvron et al. 22023a by: Devlin et al. (2019) 9 Grattafiori et al. [2024); DeepSeek-A) al.] (2024); Yang et al. (2024a); Gunasekar et al. (2023); Chiang et al. (2023). Furthermore, by in- tegrating visual instruction tuning and architectural extensions, dLLMs have recently been extended to dMLLMs Yang et al. (2025b); Li et al. (2025b); You et al. (2025) Yu et al. (2025), demonstrating
promising multimodal capabilities.
Caching Methods of dLLMs and dMLLMs. Although state-of-the-arr dLLMs Nie et al. (005b));
Zhu et al. (2025)); Ye et al. (2025a) and dMLLMs Yang et al. (2025b); Li ct al. (2025b); You et al. (2025): Yu et al. (2025) demonstrate promising performance, their substantial inference latency remains key limitation for practical deployment. Owing to their intrinsic bidirectional attention mechanism, these models cannot directly adopt the KV caching strategies Pope et al. (2023) used in autoregressive models. Several effective caching methods have been proposed accommodate the characteristics of bidirectional attention. Specifically, Prefix KV Cache Li et I (2025b) transfers the KV caching mechanism from autoregressive models to dLLMs by periodically caching prompt tokens, while recomputing the states of output tokens at every decoding step. dLLM-Cache Liu et al. (2025b) introduces V-vary caching strategy. It observes that prompt features exhibit high similarity over long sampling intervals and therefore applies longer caching interval to prompt tokens, reusing their states to avoid redundant computation. In contrast, output tokens remain stable only over shorter sampling intervals and are thus cached with a shorter interval. Moreover, based on the similarity of output tokens across adjacent steps, a certain proportion of tokens with low similarity is selectively recomputed. SlowFast Wei et al. (2025a) builds upon dLLM-Cache by introducing a dynamic sampling strategy that divides decoding into a cautious, exploratory decoding phase followed by fast, deterministie parallel decoding phase. MaskKV Huang et al. (2025)
21

---

Published as a conference paper at ICLR 2026
further improves upon dLLM-Cache by addressing the memory overhead introduced by caching, incorporating a fine-grane geeeggaannd cache eviction strategy on top of dLLM-Cache. Do Chang et al. (225) extends Prefix KV Cache by introducing adaptive visual-token merging to alleviate token redundancy. Fast-dI.LM Wu et al.] (2025) proposes a caching strategy based on the block-wise decoding process of dLLLMs: at each step, recomputes the states of all tokens after the start of the current block, while caching the states of tokens preceding it. Other recent works include Ma et al. (2025a)
Hu et al. (2025); Song et al. (025bb: Wang et (2025a).
Repeated Token Generation. Early research on text repetition and degeneration in model pre- dictions primarily focused on improving sampling strategies for language models Holtzman et al. (2020), such as nucleus sampling and top-k sampling. Several studies treat repetition control as a &training objective and introduce various training-timer strategies to mitigate repetitive generation Welleck et al. (2020); Lagutin et al. (202T); Xu et al. (2022). Other methods focus on incorporating repetition-awan penalties during the sampling stage Zhu et al. 2023).
In recent years, research on repetitive text generation has expanded from conventional language gen- eration models to modern LLMs. Yang et al.] (2024b) estimates the likelihood of future repetition using n-gram LM constructed from the generated prefix and imposes penalties based on this es- timate. Ginart et al. (2025) introduces a repetition penalty from a compression bassddd---bddda-d perspective, while et al. (2023b) highlights how repeated tokens in training corpora exacerbate repetition during inference. Some recent work further investigates the underlying mechanisms behind repe- tition. DUC Yao al. (2025) employs Sparse Autoencoders (SAEs) [Cunningham et al. (2023) to identify latent features that become highly activated when a model produces repeated tokens, re- ferred to repetition features. They attribute repetition to the activation of these features specific layers and mitigate repetition by suppressing them. Wang et al. (2024) employs model-editing tech- niques to locate FFN neurons strongly associated with repetition, as well as neurons strongly related to the main task. By intersecting these neuron sets, they filter out neurons purely tied to repetition and perform targeted edits.
Distinet from the above approaches, our work examines repetition from an information per- spective. By comparing the information-flow patterns that emerge during repetitive generation with those observed under normal conditions, we identify the underlying mechanisms that lead to rep- etition. Our analysis shows that in dMLLMs, context tokens act as anchors that aggregate and propagate information. However, the introduction of cache disrupts this aggregation pattemn and consequently triggers repetitive outputs. In addition, our layer-wise entropy analysis reveals that context tokens involved in local repetition exhibit failure of entropy convergence in deeper layers. It is important to note that existing research on repetitive outputs in MILMS and IMI.LMs remains limited. Prior studies MIL.Ms discuss repetition primarily in the context of hallucination and demonstrate that certain decoding strategies can reduce this effect Tong et al. (2025a): Huang et al. 2023 Recent dMLLM work has reported the presence of repetition but has not conducted deeper investigation into its causes Yu et al. (2025).
B ADDITIONAL INFORMATION-FLOON VISUALIZATIONS AND ANALYSES
We present additional information- -floo visualizations to further substantiate our key finding that context tokens act as anchors. As shown in Figure 10, LLaDA-V exhibits clear attention pattern, indicating that context tokens in close relative positions aggregate information and absorb attention across layers and decoding steps. similar phenomenon is observed in MMaDA, as shown in Figure 11 Moreover, as discussed earlier, introducing caching techniques significantly pattern of LLaDA-V. We further confirm this effect on MMADA (as shown in Figure 12), where locally guided, context-driven attention under standard decoding degen- erates into randomized attention distributions when caching is applied.
disrupts the information-Ile
C RANDOM REMASKING
As discussed Section 3.1 although dMLLMs can predict masked tokens at all positions in each decoding step, the model progressively refines its predictions by remasking subset of tokens and re- predicting them in subsequent denoising steps. We have previously introduced the low-confidence remasking strategy. Random remasking is another token-state transition sampling strategy. Specif-
22

---

Published as a conference paper at ICLR 2026
Layer 1, Step 1
Layer 22, Step
Layer 31, Step
Layer 32, Step 1
Layer I, Step 128
Layer 22, Step 128
Layer 31. Step 128
Layer 32, Step 128
Figure 10: Information-flow visualization of LLaDA-V.
Layer 1, Step
Layer 16, Step
Layer 29, Step
Layer 32, Step
Layer 1, Step 46
Layer 22, Step 46
Layer 29, Step 46
Layer 32, Step 46
Figure 11: Information-fior visualization of MMADA.
ically, for cach token state Si.) at decoding step t, if St * [MASK], the state remains unchanged, (2) with probability 1 (A: 1)/k, kk, and
i... ( = (i). If S = [MASK], then S/;) is set to St remains masked with probability (k 1)/k.
D MAXIMUM REPETITION LENGTH, AVERAGE REPETITION LENGTH, AND 95TH- PERCENILE REPETITION LENGTH
Given an input sequence of length M, MT. we record the length of each consecutive identical segment r'k as follows:
F = 1 T = T,1 =... T1 k == =,1,,..., K,
(14)
where K denotes the total number of segments. Furthermore, obtain all token repetition segments rep_rums as follows:
rep runs {rk rk > 2, k: 1,2,.... K}.
(15)
23

---

Published as a conference paper at ICLR 2026
MMaDA
MMaDA-Cache
Figure 12: Information- flow visualization of MMaDA.
Model
Type
LLM Tower
DocVQA
ChartQA
MMStar
MME
Seed'
MMBench
LLaVA1.5
AR
Vicuna-78 7B
1510
66.1
64.3
Qwen2-VL
AR
Qwen2-7B
83.0
60.7
DeepSeck-Vi VL
AR
DeepSeck-7B
70.4
73.2
LLaVA-OV
AR
Qwen2-7B
80.0
61.7
1580
75.4
80.8
MetaMorph
AR+Diff.
LLaMA3,1-88
37.1
71.8
75,2
JanusFlow
AR+DifI.
DeepSeek-1.38
64.6
1333
70.5
74.9
LLaMA3-V
LLaMA3-8D
86.2
77.8
56.5
76.6
LLaDA-V
Diff.
LLaDA-8B
83.9
60.1
LLaDA-V widllm-cache
Diff.
LLaDA-8B
82.1
78.1
58.3
1410
72.1
83.0
Ours
Dift. LLaDA-88
84.1
78.4
59.3
1523
73.9
83.1
Table 7: Benchmark results of different MLLMs on multiple multimodal evaluation datasets.
Based on rep runs, the maximum repetition length (MRL), average repetition length (ARL), and 95th-percentile repetition length (95pRL) can be obtained from Equations T6 tively.
17 and 18 respec-
MRL mrx
(16)
ARL
r.
(17)
rep runs
rÊrep runs
95pRL- = Quantilep.og (rep runs).
(18)
E ADDITIONAL EXPERIMENTAL ANALYSIS
E.1 COMPARISON WITH OTHER MLLMS
Table 7 presents a performance comparison with other paradigm MLLMs. Our goal is not absolute performance maximization, but rather to mitigate the output degradation of baseline models when using cache. Results across six benchmarks demonstrate the effectiveness our method.
E.2 ANALYSIS OF CACHE MECHANISM AND REPEAT CURSE
In this section, we conduct a more detailed analysis of the repeat curse induced by caching, exam- ining how different components and design choices in caching techniques relate to repetition. We begin by providing a supplementary analysis of the experimental results in Table T
(a) We begin by examining the prompt token recomputation interval, while fixing the output token recomputation interval to and setting the similarity threshold to 0. As shown Table a), periodic caching of prompt tokens has negligible impact repetition.
24

---

Published as a conference paper at ICLR 2026
Algorithm 2 Context Tokens Attention Enhancement
Require: Attention weights A R temperature > 0, lower bound Tmin (0. 1]
Ensure: Enhanced attention Á E R
1: A + A
2: for = 1 to do
3:
(layer index)
4:
for h = 1 to do
5:
(head index)
6:
Da.j - li jl Vi,j E ( (1,...,,
7:
96.i + (xp(-(Di./T))
8:
Gi, t Ymin + (1 Ymin) ) 9b.j
9: A + Ae, 0g
end for
11: end for
return Á
(b) We then fix the prompt token recomputation interval to 25 and set the similarity threshold to O, varying only the output token recomputation interval. As reported in Table [b), we observe that periodic caching of output tokens induces repetition, and longer recomputation intervals lead to higher repetition rates.
(c) In the dLLM-Cache framework, subset of output tokens is adaptively recomputed at each step based on a similarity threshold. To analyze this further, we fix the prompt token recomputation interval to 25 and the output token recomputation interval to 7, while varying the similarity threshold. As shown Table [c), the threshold plays a critical role: lower thresholds correspond to higher repetition rates.
(d) Finally, we compare dLLM-Cache with prefix KV cache Kwon et al. 2023 Li et al. 20255) to analyze the effect of different reuse policies. We fix the prompt token recomputation interval to 25, the output token recomputation interval to 7, and the similarity threshold to 0.25. As shown in Table d)), prefix KV cache, which only reuses cached states for prefix tokens, does not trigger repetition.
These four experiments reveal that the prompt recomputation interval has little effect on repetition behavior, while the output token recomputation interval, similarity threshold, and caching policy ex- ert substantial influence. In general, the fewer output tokens that are recomputed at each decoding step, the more likely the model is fall into repetition.
We hypothesize that this is because the token states within the prompt sequence remain relatively static, so caching them has minimal impact on future predictions. contrast, output tokens evolve across decoding steps. Caching these dynamic tokens forces the model to rely on outdated states, reducing uncertainty during prediction-ultimanee contributing to the non-convergence of context- token entropy in deeper layers.
E.3 INFORMATION FLOW ACROSS DIFFERENT DMLLMS
As illustrated in Figure we visualize the attention maps of MMaDA to analyze the flow of infor- mation between tokens. We observe similar pattern to LLaDA-V, where context tokens function as anchors, continually attracting attention across layers. Interestingly, we also detect vertical bands of concentrated attention in Layer 22 at Step 46, resembling the "attention sink Huang et al. (2023) phenomenon commonly seen autoregressive MLLMs. However, after introducing the cache, the original attention pattern of MMaDA is also disrupted (as shown in Figure 12
E.4 ADDITIONAL ANALYSES OF COTA
We first present the workflow of CTAE in Algorithm 2 CTAE applies a distance-basee decay term to attention values, thereby enhancing attention toward context tokens. CTEV further introduces deep-layer context-token entropy as a penalty to the voting scores. Together, these two components jointly alleviate the repeat curse. N-gram penalties Holtzman et al. (2020), simple and widely
25

---

Published as a conference paper at ICLR 2026
Methods
ACC1
ARR
LLaDA-V + cache
54.9
6.9
Methods
ARR!
SRR!
n-gram (n 2)
54.0
2.4
Decay Matrix
1.8%
8%
n-gram (n = 3)
54.3
3.2
AliBi Bias Matrix
3.9%
10%
CoTA
59.7
1.3
(b) Comparison of different attention biasing methods.
(a) Comparison between CoTA and n-gram
penalties.
Table 8: Experimental results of CoTA and related techniques.
used decoding-time technique, are commonly applied to control repetition. We further compare the performance of CoTA and n-gram penalties in Table 8 We find that although n-gram penalties can reduce repetition, their effectiveness is inferior to that of CoTA. Moreover, applying n-gram penalties often leads to degradation in model aceuracy, likely due forced token substitution, which disrupts semantic coherence and lowers output quality. In addition, we replace the decay matrix used in CTAE's attention biasing design with an Alibi bias penalty matrix [Tang al. (2025b)to evaluate its effect. As shown in Table the results confirm that using the Decay Matrix for attention intervention in CTAE is more effective.
Methods
16
32
64
128
LLaDA-V
2%
1%
1%
LLaDA- LLaDA-V+cache
88%
85%
79%
75%
LLaDA-V+CoTA
17%
14%
10%
8%
Table 9: Repeat comparison under different block lengths. The evaluation metric is the sample repetition rate.
We further analyze the sensitivity of CTEV through additional experiments, As shown in Table 9 we evaluate the CoTA performance under different block lengths during decoding. We observe that our method consistently reduces the repetition rate across all tested block configurations, demonstrating its robustness to different window sizes.
512
128
Layer range
ARRJ
SRR
ARR.
SRRI
LLaDA-V+cache
13.9%
85%
9.3%
75%
1-10
12.1%
81%
8.4%
73%
11-20
12.4%
83%
9.1%
74%
21-25
11.3%
73%
7.9%
65%
25-30
5.6%
23%
4.3%
19%
26-30
2.9%
10%
1.8%
8%
26-31
3.2%
12%
2.1%
10%
26-32
3.3%
12%
2.1%
9%
Table 10: Performance comparison under different layer ranges.
We further present results across different layer ranges in Table 10 We find that computing the cumulative entropy using layers 26-30 yields the best performance. We speculate that this because the entropy values in this depth range exhibit the largest separation between normal and abnormal modes, while incorporating additional layers introduces unnecessary noise, leading to diminished performance.
26

---

Published as a conference paper at ICLR 2026
ALaaAA-+caacl
Repeated word Repetition ratio
the
of
98%
76%
56%
Table 11: Statistics of repeated words generated by LLaDA-V with cache.
E.5 LINGUISTIC ATTRIBUTE ANALYSIS OF REPEATED TOKENS
In this section, we analyze the linguistic attributes of repeated tokens by computing their frequency statistics over 200 samples that exhibit repetition. As shown in Table [T] the top-3 most frequently repeated words are "the," "of," and "a." " We observe that repeated tokens are predominantly function words carrying low semantic content, which is also consistent with the case study presented in Figure This indicates a common tendency of the model when generating uncertain or repetitive text.
Method
throughput (tok/s) f
latency (s/sample) +
LLaDA-V
1.8
17.4
+prefix KV cache
2.7
13.3
+dllm-cache
4.9
6.1
CCEI
4.1
7.2
CTII
4.6
6.4
-Ollm-cache+CoTAA
3.8
7.7
MMMDA
0.5
32.1
+dllm-cache
2.0
19.6
CCVV
1.5
22.0
CCLl
1.7
21.4
AAIlm-cache+CoTAA
1.3
24.0
Table 12: Throughput and latency comparison on Voc.
E.6 MORE EFFICIENCY ANALYSIS
We further perform a comprehensive efficiency analysis on DOA, covering prefix-KV, dLLM- cache, CoTA, and its two components (CTAE and CTEV). As shown in Table 12 integrating CoTA introduces only a modest and acceptable increase in latency, along with a slight reduction in throughput, yet still delivers substantial improvements over the baseline. Moreover, provide a more holistic assessment of efficiency, we additionally report latency and throughput results on both LLaDA-V and MMaDA.
Evaluation data
DocVQA
ChartQA
MMStar
MME
SEED
Max generation length
32
16
Block length
32
16
2
Decode steps
16
16
Batchsize
1
Table 13: Evaluation configuration details of different datasets.
F CONFIGURATION DETAILS
F.1 EVALUATION CONFIGURATION DETAILS OF DIFFERENT DATASETS
We report detailed information about the evaluation setup in Table 13 including the maximum generation length, block length, decoding steps, and batch size for different benchmarks.
27

---

Published as a conference paper at ICLR 2026
F.2 ARCHITECTURE OF LLADA-V
Following the baseline dMLLM LLaDA-V You et al. (2025). the language tower adopts LLaDA- 8B-Instruct Nie et al. (2025b), the vision employs higlin2--o4400m-paacch4-----3--- et al. (025). and the projector is a two-layer MLP.
Tschannen
G DECLARATION OF THE USE OF GENERATIVE AI (LARGE LANGUAGE MODELS)
Generative Al tools, including Grammarly and ChatGPT, were used solely for grammar checking and language polishing. All technical content, experimental design, data analysis, and conclusions were generated and verified exclusively by the human authors. The use of Al tools does not affect the originality or authorship of this work.
H ETHICS STATEMENT
This work seeks to uncover the underlying causes of the Repeat Curse in dMLLMs with cache from an information-fie perspective and to propose mitigation strategies that enhance the output performance of iiffusion-based multimodal large language models (dMLLMs). The proposed meth- ods, CoTA, developed using only publicly available dMLLMs (LLaDA-V), caching approaches (dLLM-Cache), and benchmark datasets (e.g.. MME, MMBench, SEED), without relying on any private, sensitive, or numan-subjert data. These techniques do not introduce biases beyond those in- herent the base models, and are designed supplement rather than replace human supervision in critical applications. Nevertheless, while our methods can effectively mitigate response repetition, they cannot guarantee the elimination of errors misleading outputs. Hence, caution is advised in practical deployments.
I REPRODUCIBILITY STATEMENT
To ensure full reproducibility, we provide the following resources: (1) Code: The complete imple- mentation of CoTA, including Context Tokens Attention Enhancement (CTAE) and Context Tokens Entropy-Guided Voting (CTEV), will be publicly released on GitHub upon publication. (2) Hyper- parameters: For dLLM-Cache, all experiments fix the hyperparameters
25%. & 25. and E, 7. The parameter details for CoTA are described in Section [5. while those for the evaluation setup are provided in Section F (3) Evaluation: We use standard, publicly available benchmarks
(e.g., MME, MMBench, SEED) along with their official evaluation scripts. (4) Compute: Experi- ments are conducted on NVIDIA A800 80GB GPUs Inference latency reported using TPS and FLOPs (Table 5 (5) Models and Cache Methods: We evaluate open-source dMLLMs, specifi- cally LLaDA-V (8B). The cache method is based on the official implementation dLLM-Cache
for LLaDA-V, V. No proprietary data models are used in this work.
28

---

Published as a conference paper at ICLR 2026
Figure 13: Full Attention Maps of Each Layer of LLaDA-V.
29


Reviews:
[{"review_id": "LsiZM1Q1kR", "reviewer": "Reviewer_qq7Y", "summary": "This paper identifies the \"Repeat Curse,\" a text repetition issue caused by caching in diffusion-based Multimodal Large Language Models (dMLLMs). The authors analyze this from an information flow perspective, finding that caching disrupts the role of \"context tokens\" as information anchors and prevents their entropy from converging in deeper layers. To solve this, they propose CoTA, a training-free method with two components: Context-token Attention Enhancement (CTAE) to restore information flow, and Context-token Entropy-guided Voting (CTEV) to penalize uncertain predictions during decoding. Experiments show CoTA effectively reduces repetition and restores model performance on various multimodal benchmarks.", "questions": "1. How do you expect your findings to generalize to other dMLLM architectures (e.g., JanusFlow) and different model sizes?\n2. Would the \"Repeat Curse\" still occur with alternative caching strategies, perhaps those with more adaptive update policies?\n3. Why a fixed local window for \"context tokens\"? Have you investigated if the \"anchor\" role is dynamic or could involve non-adjacent tokens?\n4. Did your analysis reveal if certain types of words (e.g., common function words like \"the\") are more prone to exhibiting high entropy and causing repetition?\n5. What was the rationale for choosing layers 26-30 for the entropy calculation in CTEV? How sensitive is the performance to this specific layer range?\n6. Could you comment on how CoTA compares to simpler decoding-time baselines like n-gram blocking, particularly regarding the trade-off between repetition control and text quality?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Addresses a critical and practical issue (inference acceleration side-effects) in the emerging field of dMLLMs.\n2. Provides a technically sound analysis of the problem's root cause using information flow and entropy, which is more insightful than a purely phenomenological study.\n3. Proposes a novel, training-free, and plug-and-play method (CoTA) that is shown to be highly effective.\n4. The experiments are well-designed, featuring custom metrics to quantify the problem, extensive benchmark validation, and thorough ablation studies.", "weaknesses": "1. The study's conclusions are drawn from a single dMLLM architecture (LLaDA-V 8B), making it unclear if the \"Repeat Curse\" is a universal issue.\n2. The analysis is tied to a single caching method (dLLM-Cache). The findings may not hold for other acceleration strategies.\n3. Defining \"context tokens\" as a fixed-size local window feels empirical. The paper lacks a deeper theoretical justification for why these specific tokens act as anchors.\n4. The paper does not investigate whether \"anchor\" tokens or repeating tokens possess specific linguistic properties (e.g., function words vs. content words).\n5. The experiments do not compare CoTA against classic, simpler methods for controlling repetition, such as n-gram penalties or blocking.\nThe CTAE component is conceptually similar to attention biasing methods. A discussion of these related techniques is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X56ExzT360", "reviewer": "Reviewer_bzXt", "summary": "This paper studies the repetition curse that arises when caching is applied to accelerate inference in multimodal diffusion large language models (dMLLMs). While caching improves efficiency, it leads to repetitive text generation. The authors propose **CoTA (Context Tokens are Anchors)** to mitigate this issue by preserving attention coefficients of context tokens (typically 3 neighboring tokens) and downweighting attention for keys that are far from the current position. Context tokens are heuristically chosen based on proximity to the query position. Experiments on 500 randomly sampled images from the COCO dataset for image captioning demonstrate that CoTA alleviates repetition while maintaining generated caption quality.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "This paper makes an interesting observation that enabling caching in multimodal diffusion large language models can distort attention coefficients, leading to degraded generation quality.  \nTo address this, the authors propose an attention reweighting mechanism that enhances the attention coefficients of a few key context tokens (typically 3), termed **context anchors**, to preserve semantic consistency while maintaining the efficiency benefits of caching.", "weaknesses": "1. **Line 82:** The text refers to Figure 2.c, but there is no subfigure (c) in Figure 2.  \n\n2. **Generalization:** Is the proposed approach generic, or is it specific to LLaDA? Can it also be applied to other diffusion large-language models?  \n\n3. **Line 139:** The authors mention prior studies showing the existence of certain anchor tokens in autoregressive models. However, the related works section does not discuss anchors in diffusion language models, such as ADLM (https://openreview.net/pdf?id=E8adS5srds). It is important to include related work on anchoring in diffusion models since this paper primarily focuses on context tokens as anchors in diffusion language models, which was first proposed in ADLM.   \n\n4. **Figure 3:** The font is too small and difficult to read.  \n\n5. **Line 280 Typo:** “Equation 5 6” → “Equations 5 and 6.”  \n\n6. **Figure 4 Incorrect labeling:** Subfigure “(a), (a)” → “(a), (b).”  \n\n7. **Applicability of Attention Enhancement:** The main idea of attention enhancement is relevant only when caching changes the original attention pattern. Recent models such as BD3LM can better handle caching, so this problem may not arise. Can the authors verify that their observations hold for such models?  \n\n8. **Equation 7:** Why is the entropy normalized by $\\log(V)$?  \n\n9. **Ablation Study (Table 4):** Increasing the number of context tokens from 3 to 5 drastically reduces performance, which seems counterintuitive and contradicts the principle of attention mechanisms that capture long-range dependencies. Could the authors explain this behavior?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eeopqy8HOz", "reviewer": "Reviewer_XZLN", "summary": "This paper studies repetition in diffusion-based MLLMs with cache (“Repeat Curse”) and analyzes it via information flow. The authors observe that (i) context tokens serve as “anchors” that aggregate information and guide predictions; (ii) under normal decoding, the entropy of context tokens decreases and converges across deeper layers; (iii) with cache enabled, attention becomes randomized and context-token entropy fails to converge, which correlates with repetition. Building on these observations, they propose CoTA, consisting of Context-Token Attention Enhancement (CTAE) and Context-Token Entropy-guided Voting (CTEV). Experiments on several multimodal benchmarks and a 500-image captioning evaluation show reduced repetition and preserved task performance when using cache.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear empirical characterization of repetition with interpretable metrics and visualizations.\n2. A training-free, plug-in style method (CoTA) that targets the identified failure mode and is simple to implement.\n3. Results suggest repetition can be mitigated without large performance drops.", "weaknesses": "1. Mechanism not adequately explained.\nThe paper does not convincingly explain why cache introduction leads to non-convergence of context-token entropy in deeper layers. Current evidence is largely correlational (entropy trajectories + repetition metrics) without ablations that isolate which cache component (e.g., recomputation interval, similarity thresholds, reuse policy) drives the phenomenon. A causal link is missing.\n2. Lack of generality (single model + single cache).\nAll main claims—the “Repeat Curse,” the information-flow analysis, and CoTA efficacy—are validated on a single model (LLaDA-V 8B) and a single cache method (dLLM-Cache). This undermines the generality of the conclusions. Although the conclusion section acknowledges this, the paper still reads as over-claimed relative to its evidence base.\n3. CTEV design choices under-justified.\nCTEV aggregates cumulative entropy from layers 26–30. The choice appears ad-hoc and may be tightly coupled to a 32-layer backbone. No sensitivity analysis (different windows, learned weights, different depths) is reported, so robustness remains unclear.\n4. Presentation/editing issues.\nPage 2, paragraph 2: figure references are incorrect/mixed, making the early narrative confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9qZxKb2EXo", "reviewer": "Reviewer_bFZ4", "summary": "The paper successfully identifies and names the phenomenon of cache-induced repetition in dMLLMs during inference, referred to as the “Repeat Curse,” demonstrating a strong awareness of the problem setting.\nBy examining information entropy and attention flow, the paper reveals the mechanism behind repetitive generation, showcasing solid theoretical explanatory power. The paper proposes the insight that “Context Tokens are Anchors,” uncovering how context tokens act as anchors across layers and stabilize the decoding process. The proposed CoTA module (CTAE + CTEV) is a plug-and-play solution that requires no additional training, making it highly practical for real-world deployment. The work not only validates the mitigation effect on repetition but also demonstrates strong generalization across different tasks and compatibility with existing cache strategies.", "questions": "The types of repetition I've seen roughly include local word repetition, sentence pattern looping, and semantic looping. This article seems to only observe local word repetition. Is it because dLLMs only have this type of repetition?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "From the full workflow of \"Problem Definition → Mechanism Analysis → Method Design → Experimental Validation → Detail Supplementation\", this work features a complete structure, rigorous logic, clear innovation, and strong practicality. Its core contributions—revealing the repetition mechanism of dMLLMs + caching and proposing CoTA, a lightweight deduplication method—hold both academic and application value.", "weaknesses": "1. First of all, this should not be the first time that the “repeat curse” problem has been raised. It would be better to include some citations and add an analysis of previous methods that address the repeat curse in the Related Work section. In particular, please highlight solutions proposed for LLMs and MLLMs. This can help readers understand that your approach is not just a direct adaptation from LLMs to dMLLMs, but rather an innovation tailored specifically to the structure of dMLLMs.\n2. You could also consider adding another attention map after applying CoTA as a supplement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)