Title: Attention (as Discrete-Time Markov) Chains
Venue: NeurIPS 2025
Accepted: yes
Keywords: attention

transformers

Markov chain
Abstract: We introduce a new interpretation of the attention matrix as a discrete-time Markov chain. Our interpretation sheds light on common operations involving attention scores such as selection, summation, and averaging in a unified framework. It further extends them by considering indirect attention, propagated through the Markov chain, as opposed to previous studies that only model immediate effects. Our key observation is that tokens linked to semantically similar regions form metastable states, i.e., regions where attention tends to concentrate, while noisy attention scores dissipate. Metastable states and their prevalence can be easily computed through simple matrix multiplication and eigenanalysis, respectively. Using these lightweight tools, we demonstrate state-of-the-art zero-shot segmentation. Lastly, we define TokenRank---the steady state vector of the Markov chain, which measures global token importance. We show that TokenRank enhances unconditional image generation, improving both quality (IS) and diversity (FID), and can also be incorporated into existing segmentation techniques to improve their performance over existing benchmarks. We believe our framework offers a fresh view of how tokens are being attended in modern visual transformers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to interpret the attention matrix as the transition matrix of a markov chain. The multi-hop attention can then be computed to obtain a token importance measure of higher order. It then proposes TokenRank----the steady state vector of the markov chain that offers a better global token importance measure and can be used to improve several downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. This method is novel in a sense that no one has interpreted the attention matrix as a transition matrix before. It attempts to provide a framework to unify previous interpretations and further improve the quality of the representation, using well-studied knowledge in the markov chain.\n2. The method is simple yet effective: By considering information from multiple hop of tokens, a better global representation can be built through simply multiplying the adjusted attention matrix iteratively. \n\nWeakness\n1. Efficiency: Although the operations seem simple, I would still like to know whether computing the TokenRank vector will have a cost in efficiency. I wonder how many iterations are needed on average to compute the steady state vector and how much time will they cost. When TokenRank is used in downstream tasks like zero-shot segmentation and unconditional image generation with SAG, how much time is added when using TokenRank?\n2. Minor point not affecting judgement: The work only demonstrates application on vision transformer models. It would be nice if we can see the application of TokenRank on natural language processing tasks like sentiment analysis etc to see if there is any caveat or new discovery. \n3. Exploration of TokenRank with multi-modal cross attention: I think this work itself is rather complete. Potential directions to explore would be seeing whether the representation generated by TokenRank can locate specific objects in local regions, instead of a global representation of the image presented in this work. Potential downstream tasks can be visual grounding or image and sentence matching with models like “Multi-Modality Cross Attention Network for Image and Sentence Matching”.\n\nQUESTIONS:\nSee weakness 1.\n\nLIMITATIONS:\nSee weakness 2 and weakness 3", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel interpretation of attention matrices in transformers as discrete-time Markov chains (DTMCs), allowing for principled analysis of attention dynamics. By treating the attention matrix as a transition matrix, the authors are able to introduce concepts such as multi-bounce attention to capture higher-order token interactions, and TokenRank, a PageRank-inspired steady-state vector measuring global token importance. The framework leads to improved zero-shot segmentation performance, sharper visualizations of attention, and enhancements in generative tasks like diffusion-based image synthesis.\n\nSTRENGTHS AND WEAKNESSES:\nI like this paper and found parts of it to be intuitive. However, there are areas in which I struggled to get sufficient detail. See below!\n\n# Strengths\n1.  The paper is clear, well-structured, and grounded in both theoretical insight and empirical validation.\n2. The interpretation of attention as a DTMC unifies many existing operations (row select, column sum, averaging) under a cohesive framework (L1-14, L36-43) and cleanly ties into the well-understood theory of PageRank.\n3. The proposed techniques yield state-of-the-art performance in zero-shot segmentation (Table 1), improved visualization fidelity (Figure 4), and boosts in generative tasks like SAG (Table 3).\n\n# Weaknesses\n1. L7-8: \"set of metastable states where the attention clusters while noisy attention scores tend to disperse.\" This phrasing is awkward and could benefit from clarification.\n2. L12-13: \"We demonstrate that using it brings improvements in unconditional image generation.\" It would be more informative to include specific quantitative results (e.g., improvement in FID/IS scores).\n3. L37: \"(non-negative)\". I'm not sure why this is in parentheses!\n4. L55/223: \"2nd\" should be spelled out\n5. L42-43: \"we argue that considering direct token relationships is not necessarily best for evaluating importance.\" This claim would benefit from a few more words. What characteristic should be considered instead/what is your method highlighting that these others are not?\n6. Section 3.1 discusses convergence to $v_{ss}$​, but it would be nice to tie this into the formalized equations where $v_{ss}$​ is explicitly shown as a left-hand side variable.\n7. L225-227: \"we hypothesize that larger values are statistically correlated with the amount of important information...\" No statistical test is presented to evaluate this hypothesis. Rephrase this as a nullable hypothesis, and then show the statistical significance of your findings to see if this hypothesis holds or does not hold.\n8. L255 + others: \"See supplementary for an illustrative example.\" So many critical details are offloaded to the supplement. The supplement is not even part of the main PDF, which drastically reduces the ease at which I can understand what's going on in the paper. Further, please reference specific sections in the supplement rather than vaguely referencing \"the supplementary\".\n9. L252-253: \"indicating that our method is on par and arguably better than the previous state-of-the-art...\" All of the comparisons in this paper lack error bars. Going back to weakness 7, how can anything be \"arguably\" better? It either is or isn't, and you can show that with error bars and a statistical test. These experiments results seem compelling but need significance to underline the claims.\n10. In general, the authors attempt to show too many use cases, which dilutes the focus. Section 5.3, which was highlighted in the abstract (!) gets minimal exposition. I would consider trimming less central experiments to allow for deeper analysis of key results. Right now the paper reads pretty thin.\n11. This work is not the first to relate attention and Markov Chains. It's worth citing Ildiz et. al [0] and discussing in the related works where this work differs.\n12. I'm not sure if I agree with an answer of \"NA\" to question 3 on the checklist. The authors are providing new theoretical results (tying attention to DTMC), but because they've left themselves no space, they are not able to clearly present this theory in the paper.\n13. Checklist question 4 is marked \"yes\", but even in the supplement, there is not enough detail about how the authors ran their experiments to redo them. Which experiments ran on which GPU (of the 4 types used)?\n\n\n[0] Ildiz, M. E., Huang, Y., Li, Y., Rawat, A. S., & Oymak, S. (2024). From self-attention to markov models: Unveiling the dynamics of generative transformers. arXiv preprint arXiv:2402.13512.\n\nQUESTIONS:\nThe authors need to improve the readability of the work. My central concern is just how many results the authors are trying to show, but they end up falling short of satisfaction in each section.\n\n1. The work currently lacks a lot of details in the experiments section. Please pull all of this out of supplementary purgatory and put it into the main paper.\n2. No error bars are provided, therefore no claims about the significance of the results can be made. The authors claim in the checklist that experiments with error bars are **not** provided, but then justify this by saying that \"we run the segmentation results several times...\" So the authors have a way to provide error bars but choose not to? I don't agree with the \"computational expense\" excuse to not provide error bars. At least, discuss this in the main paper, not the checklist!\n\nIf the authors can address these two claims in their rebuttal (and therefore an edit of their paper), that will be satisfactory.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present a new interpretation for the attention matrix by modeling it as a discrete-time Markov chain, providing fresh insights into the mechanism of attention. Specifically, this paper treats the attention matrix as a state transfer matrix and consider the stable state of it. In this way, the proposed method brings improvements in unconditional image generation and zero-shot image segmentation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1. This paper is easy to follow, with a clear description of the proposed method and the problem it aims to address. The solution presented by the authors is also reasonable.\n2. The results presented by the authors show that their method has clear advantages over other approaches.\n3. To my knowledge, this is the first work to consider about the stable state of attention matrix.\n\nWeaknesses：\n1. In Section 4.1, the authors describe existing interpretations of attention mechanisms. However, I find this description to be incomplete, as it does not mention [1], which merges attention maps using KL divergence. Moreover, the comparative experiments do not include a comparison with this method. If the authors believe that this method is not relevant to the present work, they may provide a rebuttal.\n\n2. The experimental results shown in Figure 6 of the supplementary material are convincing. However, I hope the authors can provide some visualizations of the feature maps or attention maps to further enhance the credibility of their results.\n\n3. In Figure 3, the authors only present some basic attention visualization methods, while most of the quantitative comparisons are provided in the supplementary material. Although this does not affect the review process, if the paper is accepted for publication, it would be necessary to include more qualitative results directly in Figure 3. Moreover, the comparative methods included in the supplementary material appear to be insufficient.\n\n4. In Figure 5, the authors demonstrate the effectiveness of the SAD method on the SD1.5 architecture. However, since SD1.5 is based on a UNet architecture, it would be valuable to include qualitative results on methods utilizing DiT-based architectures, such as PixArt or Flux.\n\n[1] Diffuse, Attend, and Segment: Unsupervised Zero‑Shot Segmentation using Stable Diffusion\n\nQUESTIONS:\nSee Weaknesses. If the authors are able to address all or some of the weaknesses, I would consider raising my score.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents a novel and theoretically grounded interpretation of attention matrices as discrete-time Markov chains (DTMCs), introducing operations like multi-bounce attention, TokenRank, and \lambda_2-weighted head averaging. The approach demonstrates state-of-the-art performance in zero-shot segmentation and improves unconditional image generation. The work is well-motivated, methodologically sound, and empirically validated.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	Framing attention matrices as DTMCs is elegant and insightful. This allows modeling indirect attention paths (via Markov transitions), moving beyond prior work limited to direct interactions.

2.	Extensive experiments are conducted to show its SOTA performance on zero-shot segmentation and the effectiveness of TokenRank.

Weaknesses:

1. Experiments are limited to image tasks (image segmentation/generation). More validation in NLP or multimodal settings will be helpful to illustrate its effectiveness and domain generality.

2.	More ablation studies to evaluate the contributions of individual components (e.g., \lambda_2 vs. multi-bounce) are suggested.


3.	The discussions about the limitations are insufficient.

QUESTIONS:
See weaknesses.

LIMITATIONS:
More limitations should be discussed.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05