Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
Authors: Itamar Zimerman
Ameen Ali Ali
Lior Wolf
Venue: ICLR 2025 Poster (2025)
Keywords: Explainability
Interpretability
Gated-Linear RNNs
Attention-free
Mamba
Primary area: interpretability and explainable AI
TL;DR: Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation

Abstract: Recent advances in efficient sequence modeling have led to attention-free layers, such as Mamba, RWKV, and various gated RNNs, all featuring sub-quadratic complexity in sequence length and excellent scaling properties, enabling the construction of a new type of foundation models. In this paper, we present a unified view of these models, formulating such layers as implicit causal self-attention layers. The formulation includes most of their sub-components and is not limited to a specific part of the architecture. The framework compares the underlying mechanisms on similar grounds for different layers and provides a direct means for applying explainability methods. Our experiments show that our attention matrices and attribution method outperform an alternative and a more limited formulation that was recently proposed for Mamba. For the other architectures for which our method is the first to provide such a view, our method is effective and competitive in the relevant metrics compared to the results obtained by state-of-the-art Transformer explainability methods. Our code is publicly available.

Published as a conference paper at ICLR 2025
EXPLAINING MODERN GATED-LINEAR RNNS VIA A UNIFIED IMPLICIT ATTENTION FORMULATION
Itamar Zimerman"
Ameen Ali
Lior Wolf
The Blavatnik School of Computer Science, Tel Aviv University {z imeemanl, ameenali @mmail 1. tau. ac. wolfecs.tau.ac.It
ABSTRACT
Recent advances in efficient sequence modeling have led to attention-fre free lay- ers, such Mamba, RWKV, and various gated RNNs, all featuring sub-quadratic complexity in sequence length and excellent scaling properties, enabling the con- struction of new type of foundation models. In this paper, present unified view of these models, formulating such layers as implicit causal self-attention lay- ers. The formulation includes most of their sub-components and is not limited a specific part of the architecture. The framework compares the underlying mech- anisms on similar grounds for different layers and provides direct means for ap- plying explainability methods. Our experiments show that our attention matrices and attribution method outperform an alternative and more limited formulation that was recently proposed for Mamba. For the other architectures for which our method is the first to provide such a view, our method is effective and compet- itive in the relevant metrics compared to the results obtained by kfuteoof-the-at Transformer explainability methods. Our code is publicly available.
INTRODUCTION
The very recent State Space Model (SSM) named Mamba by Gu & Dao (2023) has attracted con- siderable attention since its recent debut (Lieber et al. 2024 Liu et al.) 2024 Zhu et al. 2024 Xu et al. 2024), further establishing it as efficient and accurate general-purpose model. Like other SSM models (Gu et al. 202 b). Mamba is autoregressive during inference and trains efficiently in parallel. Recently, Ali et al. (2024) have highlighted a third aspect of the Mamba model; namely, that it is also an attention model, since implicitly computes attention.
Attention models be defined models that linearly combine the values associated with different elements to create the next set of such associated values. When discussing sequences of tokens, an attention operator considers the values obtained for each token separately, as hidden representation, and mixes these to obtain a new set of values for each token. The mixing coefficients are also a function of the hidden representations.
Let X be the matrix whose columns are the hidden values associated with each token, and let be the matrix of mixing coefficients. The of values of the next layer is initially obtained as Y = ax and it can then undergo other forms of processing, such as nonlinear activations and per- token processing. Given neural architecture, one can always linearize the mixing operators and write them in the form Y = aX via their first-order approximation. However, to be considered an attention model it is required that a a function of X, which means that the linear operator is data-dependent. This property is shown by Ali et al. (2024) to hold only for the recent selective SSM (S6), but not for most earlier SSMs. Specifically, for standard state-space layers, it has been demonstrated that they can be linearized into constant operator, represented by constant matrix a, which is solely controlled by the layer's parameters. However, in the S6 layers, a is influenced by both the input and the layer's parameters.
Thheee authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
The implicit attention matrix [fli al. (2024) considers the S6 mechanism and ignores the in- fluence of other critical mixer components, such as ConvID, gate branch, linear layers, and SiLU activations. The formulation we propose in this work incorporates these additional elements and, as we show empirically, leads to improved interpretability results in both computer vision and NLP.
Furthermore, using a similar holistic formulation, show that S6 is not the only sequence model that implicitly computes attention and that an implicit attention representation can also describe other recent layers, such as RWKV (Peng et al., 2023). Griffin (De al. 2024) ,HGRN (Qin al., 2024b) and more, as illustrated in Figure
To achieve more accurate representation that better reflects the model's behavior, employ composition of multiple components. The concept of composing non-attention layers and represent- ing them data-controlled linear operators was initially introduced in Hyena (Poli et al., 2023), which attempts to replicate attention capabilities through a composition of two sub-quadratie opera- tors (long convolutions and multiplicative gating). Our formulation differs from this approach in two main ways. First, instead of focusing replicating attention capabilities, we take the reverse step by demonstrating that, through a sequence of algebraic manipulations, several existing modern gated linear RNNs can be viewed as single implicit attention layers. Second, our goal is to find the most accurate implicit attention representation possible, as it crucial for applications like interpretabil- ity. This leads to significant extension over Hyena's work. For example, while Hyena's 's matrices are constructed from the two components mentioned above, our implicit attention representation in- corporates additional layers, including non-linear operators such linear layers, activations, short convolutions, and normalization layers. For instance, our formulation for Mamba-2 is built upon six different layers, some of which appear multiple times, resulting in much more complex out- come. Additionally, while other works explore the relations between non-attention layers and linear attention (Arora al., 2023), we not aware to any work extending the concept of composition of components similar to us, or apply it to existing modern RNN such as Griffin, Mamba, or link in to interpretability or similar domains.
Our main contributions are as follows: (i) We introduce the implicit self-attention representation, unifying Transformers with non- Transformer layers, such as Griffin, RWKV, ReNet, and others. (ii) We refine the approach Afl et al. (2024) to produce more accurate attention matrices. The previous work focused exclusively on the S6 layer, without considering the gating and Conv ID sub-layers in Mamba, while representation incorporates all these factors (and additional peripherals in other models) (iii) While "Attention is not Explanation" " (Jain & Wallace 2019). Transformer explainabil-- ity relies heavily attention matrices. We demonstrate that our implicit attention representation of non-Transformer models can be used to develop new explainability and interpretability techniques for non-Transformer models, enhancing the community's 's ability understand, explore, and manage aspects of robustness, bias, fairness, and safety. As a sample downstream application, we demon- techniques. (iv)
strate excellent out-of-the-box results for attribution-bass. Finally, our framework facilitates comparisons between Transformers and other recent architectures, by providing unified attention view and setting the stage for further improvements and insights.
performance-enhane
2 RELATED WORK
This section describes the scientific context and provides the necessary terminology and symbols for discussing self-attention and selective SSM layers.
Self-Attention. Self-attention, a cornerstone of Transformer architectures (Vaswani 2017), has profoundly influenced recent developments in NL.P and computer vision. This mechanism leverages pairwise token interactions dynamically allocate focus across different parts of the input sequence, assessing the relevance of each token in relation to others. The computational formula is given by:
QKT
Self - Attention(Q) Q,K,V) = aV, a = softmax
(I)
VoL.
Here, Q.K, and V denote the queries, keys, and values respectively, with di representing the key dimension. Transformers enhance this mechanism by incorporating H parallel attention heads, thus capturing a wider range of dependencies.
Applications of Attention Matrices. Attention matrices play a crucial role in Transformers, as multiplying these matrices with value vectors is the core operation that captures interactions be- tween tokens. Beyond this essential role in computing self-attention, they are also used for various
2

---

Published as a conference paper at ICLR 2025
purposes: (i) Explainability and Interpretability: Although attention itself is not inherently
plainable Jain & Wallace| 2019), many methods in these domains rely on attention matrices to understand and analyze model behavior (Abnar & Zuidemma, 2020; Chefer al. 202 a Ali al. 2024). (ii) Multi-modal I Learning: Numerous multi-modal learning schemes are based on varia- tions of cross-attention, enabling dependencies to be learned between any pair of tokens of different modalities (Lu et al. [2019] Tan & Bansal 2019). (iii) Weakly Supervised Tasks: Attention ma- trices can provide valuable source of supervision, highlighting relevant regions relationships within the data to guide model learning. These techniques are popular in semantic segmentation (Ru al., 2022 Wang et al. 2020 Ru et al. 2023), and robustness enhancement (Chefer et al., 2022). Finally, (iv) Inductive Bias and Regularization Methods: Since attention matrices represent inter- actions between tokens, they inherently carry semantic meaning. Therefore, they can be manipulated to incorporate domain knowledge or regulate the model effectively (Li et al. 2018 Attanasio et al.,
2022 Bonaldi et al., 2023 Zimerman & Wolf 2024a).
S6 Layers and Mamba. The recently presented selective SSM (Gu & Dao. 2023 (S6) outper-
forms the previous SSMs and various other architectures in NLP (Anthony et al. [024|] Wang et al.)
2024b), vision (Liu et al. 2024 Zhu al. 2024), graph classification (Wang al. [024a]] Behrouz & Hashemil 2024). and more. S6 incorporates a dynamic input-dependent form of the discrete matrices A, B, and C, such that for every time-step the SSM employs different recurrent rule. This technique differs from the previous state-space layers, which use the same set of matrices and recurrent rules for each time step.
Denoting the input sequence by 2 == ($1.... ... .L) E R where &, E IR the discrete matrices for time step i, namely Ai, Bi, and C; are defined as:
B. Smtt), C = So:),,, A; SoftllusSSaaA.), À exp((AAA) B. = A.B.
where Sc. SA are linear projection layers, and Softplus is asmooth approximation of ReLU.
The usage of nependee time- variant layers adds to the expressivity of the layer CChhen-Karite et al. 2025), allowing it to adapt to the input, and potentially captures more complex dependen- cies. While other input-dependent time-variant mechanisms have been proposed previous works through gated RNNs, the S5 layer (Smith et al.) (022),, or adaptive filtering via input-dependent IIR filters (Lutati et al.] 2023). S6 also presents an efficient I0-aware implementation, which is paral- lelized GPUs via work-efficient parallel scanners (Blelloch) 1990, Martin & Cundy 2017).
The Mamba block combines the S6 layer, Conv ID and other elementwise operators. It borrows elements from Gated MLP, and given an input æ == (II,.--- IL), (I),), is computed by:
f SiLU( Conv ID Linear(x))) )
2 SiLU Linear(())) j' Lineer(Selecctve SSM(2) @ é)
(3)
where @ denotes elementwise multiplication.
The entire Mamba model contains stacked Mamba blocks with D channels per block. Below, the tensors of the j-th channel in the i-th block are denoted by superscript indices of the form i.
The vision Mamba architectures (Liu et al. 2024; Zhu et al. 2024) (ViM) follow the vision Trans- former (VTT) (Dosovitskiy et al. 2020) but replace the Transformer's self-attention mechanism by two bidirectional Mamba layers, These vision models outperform the standard ViT in terms of ac- curacy and efficiency, for models of similar parameter counts.
Gated-Linear RNNs. RNNs, along with their advanced versions, such as GRU (Chung et al., 2014) and LSTM (Hochreiter & Schmidhuber 1997). play fundamental role in deep sequence modeling. Their auto-regressive design decouples es sequence length from computational complexity per step, making them highly efficient at decoding. However, they don't scale as effectively as Transformers and often face challenges, such as slow training and vanishing gradients. Recently, linear RNNs have shown improved abilities in capturing long-range dependencies (Gu et al., 2021a Orvieto et al., 2023) and enhanced scalability (Peng et al. [2044 al. (222)). Furthermore, gated linear RNNs deliver surprisingly strong language modeling performance (Mehta et al.) 2022 Wang
ct al.
2022 Peng et al. 2023 Qin et al. 2024b). The most advanced gated linear RNNs include the following variants: (1) RWRKV V-6 (Peng et al. 2023). which draws inspiration from attention -fee Transformers (AFT) (Zhai et al. 2021), (ii) Mamba (Gu & Dao 2023), which employs selective
SSM, (iii) HGRN2 (Qin et al. 2024a). which utilizes state expansion, and (iv) Hawk (De et al.
2024], which is built upon an enhanced variant of the LRU (Orvieto et al. 2023). Other notable
3

---

Published as a conference paper at ICLR 2025
examples include GLA Yang al. 2023). GateLoop (Katsch 2023). , and RenNet (Sun al. 2023 These layers achieve results comparable to Transformers on larger scales, matching well known models, such as Pythia (Biderman et al. 2023) and LLaMA 2 (Touvron et al. 2023). Moreover, several studies show that hybrid models combining attention mechanisms with gated linear RNNs can be complementary (De al., 2024 [Leber al. 2024 Poli al. 2024 Ma et al. 2022 Baron iet al., [2023; Fu et al. 22222, enhancing both approaches. Despite these successes, interpretability and explainability techniques for these models remain relatively unexplored.
3 METHOD
In this section, we present a general and holistic data-control linear operator representation that can be applied to (at least) many of the recent non-Transffrmer architectures and which incorporates all components of the architecture. Our objective is to describe each layer in the form of y á such that x and is the input and output respectively, and â f(z; Qarch) is attention matrix controlled by the parameters of the model and the input. Sec. 3.1 formulates the entire Mamba and Mamba- 2 (Dao & Gu. 2024 architectures
data-control linear operator, incorporating subcomponents such as Conv ID, gate branches, normalizations and activations. Subsequently, Sections. 3.2 and 3..3 extend our approach other architectures. such as Griffin (De al. (2024) and RWKV (Peng al., 2023). Additionally, in Appendix [A] we present how to extract holistic data-controlled attention
matrices for RetNet (Sun al. 2023) and HGRN (Qin et al. 2024b).
Mamba
Griffin
RWKV
Unified Formulation
Interpretable reproseniztion
y ((rroRadt))
39 SSM
RB LRU
Vi
Sigmad
WKV Upemtor
BLUU
Gomv1U
GeLU
Temporol Comvll
Linear Ir Ierepoaation
Li
Tokan
Unified and Interpretable Formulation of Attention-Fres Architectures via Attention Ma- trices: (Left) Schematic overview of the architectures of Mamba, Griffin, and RWKV. (Right) A new view of those layers that rely on implicit attention. Our perspective enables the generation of implicit attention maps, offering valuable applications in areas such Explainable AL.
Figure 1:
3.1 FORMULATION OF MAMBA VIA ATTENTION MATRICES
Mamba can be formulated in a way that separates the components that mix channels from those that mix tokens:
Mamba (r) Linears (SILUU(Linearg(Linean)
(LnnannSIIA((LnLannnnLLeeeeeee (x))) @ SGISILU (x))))
(4)
Since Linear, and Lineary do not mix tokens, they are less relevant to representation (similar to the MLP layers in Transformers), and we consider the following simplified expression:
Mamba(x) x) = (ILU((Lnearr(:))) @ (S6(SILU(Conv
(Dx)))
(5)
Replacing the element-wise gating multiplication with matrix multiplication leads to:
Mamba(z) diag (ILU((Lneerr(=))) (S6(SILU(ConvID(()))
(6)
The S6 layer can formalized as a data-control linear operator (see Eq. 12 in (Ali et al. 2024
S6(x)) âx, رGi C. (1i
(7)
4

---

Published as a conference paper at ICLR 2025
By plugging Eq. 7 into Eq. 6 and since SILU(x) Sigmoid(()
Mamba(z)
diag (ILU((inearr(z))) à diag (Ceno (Conv1D(x)) (8)
WIER
(gate)
ZRIoR,, (Conv Act)
Recall that causal Conv ID layer with filter f= (f1, ft) can be converted into matrix form by arranging shifted copies of the filter into rows, forming a convolution matrix M. This matrix is then multiplied by the input sequence produce output, where each element represents the dot product of the filter and corresponding segment of the input.
By plugging the convolution matrix M and the gate matrix W into Eq.
get:
Mamba(r) = (rar))WaAaZzee Wr Hx, H = W
Therefore, the entire Mamba layer can be viewed as a data-control linear operator, which implicitly parameterizes the per-channel implicit attention matrices through the parameters of the S6 layer, the Conv ID filter, the linear layer in the gate branch, and is controlled by the input
Mamba-2 This architecture builds upon Mamba by introducing two key enhancements relevant to our formulation: (i) incorporating the concept of multiple heads via multi-input SSM, and (ii) applying additional normalization (GroupRMSNorm) after the multiplicative gating.
The first modification can be handled by broadcasting parts of the equations across different attention heads. For the second modification, we first compute the per-head statistics necessary for Group Normalization and pack them into diagonal matrix.
1
Ph Σ
[1) oh =h=4t =
(an mn)3, diag
(10)
0h
0H
where [ denotes the i-th feature of head [H]. d is the dimensionality of each head, and e is small constant added for numerical stability in GroupRMSNorm.
The matrix N allows us to represent the GroupRMSNorm operator via matrix multiplication such that Nr GroupRMSNorm( (where N is augmented across groups). Thus, plug these mod- ifications into our formulation Mamba (Equation 9), obtaining the following implicit-attention formulation for Mamba-2:
Mamba-2(z) NW Hr
(11)
3.2 FORMULATION OF GRIFFIN VIA ATTENTION MATRICES
The component that captures interactions between tokens Hawk and Griffin (regardless self- attention) is the temporal mixing block, which is built top of a Real-Gated Linear Recurrent Unit (RG-LRU), Conv and gating. can be formalized as follows:
= Linearg (GEL.U(Linear, ({))) e I.G-LRRU(Com/IDDLLinn..tttt
(12)
We first rearrange the linear layers and replace elementwise gating with matrix multiplication:
I Linearg(x'),
=
Lineary (iag (LLneeLinear(( (x))) (RG-)))RRGGG (RG-LRU(ConvID(=))
(13)
Note that Linear := Linear, Linearg and Lineary do not mix tokens and can therefore be omitted.
By substituting Conv ID with matrix multiplication using a causal convolution matrix M, we derive:
=
diag (ELUU(Lnnear((( (x)) (RG-LRU(Mx))
(14)

---

Published as a conference paper at ICLR 2025
RG-LRU is defined by the following recurrent rule:
= oW ba), = ( ot bR), = =
h2 10
v1
a
s0 (de 00))))
(15)
This linear recurrent rule can be converted to matrix form as follows:
h1
v1 - a1 @ i
0
ha
a2v 1 a1² &
vI as²
0
h = 0xr,
0
de
vI
ai 6n
= vI
az²
az?
@ in.
(16)
By plugging Eq 16 into Eq 14 data-control linear operator:
that the entire temporal mixing block can be formalized
y = diag (FELLU((inear((x))) aMx Ha,, H = diag (GELU(LLinerr (x)) aM
(17)
3.3 FORMULATION RWKV ATTENTION MATRICES
The time-mixing block of RWKV includes three components: the WKV operator gate branch, and a token shift. For simplicity, we will ignore the token shift operation over the values. The simplified RWKV, which maps the input to the output can be formulated as follows:
Te Wr (u, + (1 up) rt-1), he Wk (ux @ at + (1 ux) x2-1), 2t=2 2t (18)
wky, Σ ij & + euthee @
Wor(r)) @ wkv,
(19)
(t-1-ittt+ + euthe
where W,, WR. W. are linear projections, and u, w, ur Ug learnable parameters.
Now, we will reformulate the wkv; operator into a form of causal self-attention:
the 2im=1 (( mow t e --(1-1-))0+
if j holds,
Ōs.j
if j holds,
āz = wkk
(20)
Di-1
otherwise.
Note that W. does not mix tokens and can therefore be omitted. By plugging Eq. 20 into Eq. and replacing element-wise gating with matrix multiplication, obtain:
18
diag(o
(21)
3.4 SHARED PROPERTIES
The proposed formulation for Griffin, Mamba, and RWKV is based on the similarities in the struc- ture of the architecture. Our formulation focuses on three main components: (i) the core of the linear attention mechanism (S6 for Mamba, RG-LRU for Griffin, or the WKV operator for RWKV), (ii) short filter operation implemented via Conv ID in Griffin and Mamba and token shift in RWKV, and (iii) the gate branch, as illustrated in Fig. ! Additionally, our formulation builds on the follow- ing key components: (1) rearranging linear layers and omitting operators that do not influence the mixer components, (2) representing the gate branch, activations, and normalization layers as data control linear operator via diagonal matrices, (3) unrolling the linear recurrent layer to obtain token-to-token map, and (4) fusing several cascaded linear operators and ignore biases.
In Appendix E elaborate on several insights derived from our formulation of gated-linear RNNs. including the inner dynamics of their parameterization, their expressivity, and the categorization of attention models.
EXPERIMENTS
To assess the effectiveness of our implicit attention formulation, perform comprehensive set of experiments. In Sec. [.], we begin by visualizing the implicit attention matrices and the cor- responding explainability maps built upon them. In Sec 4.2 demonstrate that integrating

---

Published as a conference paper at ICLR 2025
improved attention matrices into existing attribution methods results in SoTA interperablity tech- niques. We further conduct ablations to analyze the contribution of each architectural component to the overall representation. Finally, in Sec. 4.3 show how our formulation enables the transfer of performance-enhaniinn techniques, originally designed for other architectures, to gated RNNs.
4.1 VISUALIZATIONS
Transformer
Mamba
Griffin
RWKV
In Fig. 2 we present a comparative visualization of the attention matrices from Mamba, RWKV, Griffin, and Transformers. To enhance clarity, we applied the Softmax function to each row the attention matrices from the Transformers and conducted min-max nor- malization on the absolute values the matrices from the non-Transformer models. In every instance, we used uniform prompt of size 32. For each model, examined the attention matrices derived from the standard pre-trained models available in the Hugging Face, including the Recurrent Gemma-2B, RWKV- 430M trained on the Pile, and a Mamba-hased LLM with 2.8B parameters also trained on the Pile.
Hidden Attention Matrices: Atten- matrices of LLMs. Each row represents a different layer within the LLMs, showcasing the evolution of the attention matrices at 25% (top), 50%, and 75% (hottom) of the layer depth.
Figure 2:
As illustrated, the implicit attention matrices of Mamba, Griffin, and RWKV exhibit similarities to those derived from traditional Transformers. Echoing findings from (Ali et al., 2024). we note that depen- dencies between distant tokens become more apparent in the deeper layers, as shown in the lower rows. Additionally, the matrices from RWKV are characterized by distinct horizontal tiles, whereas those from Mamba display more continuous structure.
Visualization Explainability Maps. Sample explainability maps built on top of our implicit at- tention formulation are resented in Figure This visualization focuses the rows of the attention maps that are associated with the [CLS] token, as is traditionally done for interpretability purposes. We explore the attention matrices with three explanation methods: raw attention, attention roll- out (Abnar & Zuidemal 2020), and attribution following Ali et al. (2024), along with a comparison attention formu-
to the VIT counterparts. Evidently, the explanation methods that are based lation (columns e, f, and g) depict much more accurate and sharp maps compared to those of (Ali et al. 2024) and the ViT counterparts. In Fig. 4 we show similar visualizations in the NLP domain. More qualitative results for the NLP domain can be found in Appendix.
[b) (c)
(d)
(6)
(
(b)
(
Figure 3: Qualitative results for the different explanation methods for the ViT and VIM, both of small size. (a) The original image, (b) Raw-Attention over ViM, (c) Attention-Rolleut over ViM, (e) Raw-Attention with our proposed attention over ViM, (f) Attention-Rollon with our proposed attention over ViM, (g) Mamba-Attributin with our proposed attention over ViM, (h) Raw-Attention of VIT, (i) Attention-Rollou for VIT. (i) Maansfoomee-Atrbbuuuti for VIT. Results for columns (b), (c), and (d) are based the method of (li al. 2024). and the VIT results (i), and (k) rely on Chefer et al. (202lb)
ViM, (d) Mamba-Atrihation
7

---

Published as a conference paper at ICLR 2025
Baety- pare
Sacry, gare +lB. which raking asleop
which asloop minntes frow and, ceally, zoally beced and not caring at
saleep
give to morias as shich walt
seleep
this fall
this case
unlwep miestes from
really.
and, really, seally becwd and nst caring
happesed sest
about shat bagpased sext)
sboat shat hagpened nest
Bii of Eleecite tilm all tim
foveciie filns time.
Facies arting the beet
mscalients Toe simn well
well
nocellen.. bod sidet
His firss eelonsed classic
didet
fiest relanned
didai
didm
chince esperiance tais plassic
chance
(a)
(c)
Figure 4: Qualitative results for NLP. samples are taken from IMDB movie sentiment classification. (a), we show the results for the previously proposed Mamba's attention (Ali et al. (2024). (b) our proposed Mamba's attention, and in (c) we show our proposed method over RWKV. In the upper row, we show a negative sentiment, and in the lower row, we show positive sentiment.
4.2 IMPLICIT ATTENTION- BBSSED ATTRIBUTION
Although empirical evaluation of attribution methods is challenging, this section demonstrates that off-the-shelf techniques, when built on top of our implicit attention formulation, produce SoTA explainability tools. We provide empirical analysis via perturbation and segmentation tests.
Perturbation Tests. To assess the faithfulness of explanations, we adopted an input perturbation scheme similar to (Chefer et al. 2)) This method involves systematically masking image pixels based on their predicted relevance from the explanation method. We conducted experiments with both positive and negative perturbations on both NLP and Vision domains. For positive pertur- bation, a good explanation prioritizes relevant pixels. We expect the model's accuracy (specifically, top-1 accuracy) to gradually decrease as we mask pixels in descending order of relevance (most relevant first). As for negative Perturbation, robust explanation should maintain model accuracy even when irrelevant pixels are masked. Here, we mask pixels in ascending order relevance (least relevant first). In both scenarios, we evaluate the explanation quality using the Area-Under-Cure (AUC) metric. AUC considers the model's 's accuracy as a function of the percentage of masked pixels (ranging from 10% to 90%).
The perturbations results for vision models are summarized in foo various explanation meth- ods under both positive and negative perturbation scenarios the ImageNet validation set. In the positive perturbation scenario, where lower AUC values indicate better performance, our proposed Mamba's attention method consistently outperforms the other methods. Specifically, our method achieves the lowest AUC values across all explanation methods, with an AUC of 13.264 for Raw- Attention, 12.830 for Attn-Rollout, and a notably low 11.350 for Attribution. In the negative per- turbation scenario, where higher AUC values are better, our method shows the best performance, with AUC values of 47. 77. for Raw-Attention. 50.035 for Attn- atln-Rolluut, and 51.310 for Attribution, outperforming both the method of (Ali et al.) 2024) and the counterpart XAI methods for VIT.
Table 1: Perturbation Tests for Vision. We present the AUC results (percentages) for the predicted class the ImageNet validation set. For positive perturbation lower is better, and for negative perturbation higher is better. Previous results by (Ali et al., 2024) denoted by
Positive Perturbation
Negative Perturbation
Mamba
Mamba Ours
Transformer
Mamba
Mamba Ours
Transformer
Raw-Attention 17.268
13,264
20.687
34,025
47.705
40.766
Attn-Rollout 18.806
12.830
20.594
41.864
50.035
43.525
Auribution 16.619
11.350
15.351
39.632
51.310
48.089
In the NLP domain, we conducted perturbation tests in both the zero-shot and fine-tuned settings. In the zero-shot setting, we utilized pre-trained Mamba-based LLMSs with sizes of 1.3B and 2.8B on the ARC-E dataset (Clark et al. 2018), which evaluates the reasoning abilities of LLMs. Results are presented in Table and contain activation and pruning perturbations, as described by Ali et al. (002). It is shown that our explainability method improves upon the baseline of Ali et al. (2024) for both model sizes. Specifically, in the activation scenario, method improves results by least 2.2%, and by over 10% in the pruning settings. Similar trends are also evident in the fine-tuned scenario, see Oppendix[[for these results for both Mamba and RWKV. Taken together, these results
8

---

Published as a conference paper at ICLR 2025
Table 2: Perturbation Tests for NLP. For activation perturbation lower is better, and for pruning perturbation higher is better. Previous results by (Ali et al., (024) denoted by
Activation Perturbation (AUAC) f
Pruning Perturbation (AU-MSE) +
Mamba 1.3B
Mamba 2.8B
Mamba 1.3B
Mamba 2.8B
Attribution of (Ali et al. 2024 Our Attribution
0.915
0.918
1.765
1.239
0.934
0.939
1.588
1.082
demonstrate that our attention formulation is much more precise and better reflects the model's behavior compared to the formulation proposed by Ali et al. (2024). The same phenomenon occurs with the RWKV model, consistently showing that our formulation can lead to SoTA attribution methods for an entire family of models.
Segmentation Tests. We evaluated our proposed Mamha's implicit attention mechanism by com- paring its generated foreground segmentation maps against ground truth from the ImageNet- Segmentation dataset (Guillaumin et al., 2014). We employed established metrics (pixel accuracy, mean Dneesecttio-ovve-Uniioo (mloU), and mean Average Precision (mAP)) aligning with prior works (Chefer et al. 2021b Nam et al. 2020 Gur et al.) 2021). We compared our method against techniques specifically designed for ViM, including the attention- approach offAli et al. (2024) and the recent LRP-based method introduced by Jafari et al. (2024). Additionally, as reference point, present the results of standard methods applied VIT.
Results presented in Table demonstrate that our proposed Mamba's implicit attention outperforms both the VIT and the previous method designed for Mamba across all metries for the three different XAI methods. This superior performance suggests the potential of these maps for downstreaming tasks such as weakly supervised semantic segmentation, and mitigating background bias in classi-
fiers (Chefer et al. 2022).
Table 3: Segmentation results the ImageNett-Segmentttion dataset (percent). Higher is better.
Model
Method
Pixel accuracy f
mAP t
mloU
DeiT S
Raw-Attention
59.69
77.25
36.94
ViM-S
Raw-Attention et al.
2024
67.64
74.88
45.09
ViM-S
Raw-Attention Ours
67.66
80.00
47.28
DeiT-S
Attn-Rollout Abnar Zuidema
2020
66.84
80.34
47.85
ViM-S
Attn-Rollout Ali ct al. 2024)
71.01
80.78
51.51
ViM-S
Attn-Rollout Ours
76.40
83.90
58.48
DeiT-S
Transformer- Attr (Chefer et al
202 bb
79.26
84.85
60.63
VIM-S
Mamba-Attr (LRP) (Jatan et al
2024
71.19
77.04
49.98
VIM-S
Mamba-Attr (Ali et al. 2024
74.72
81.70
$4.24
VIM-S
Mamba-Attr Ours
79.60
86.40
62.51
M.
M. w/o Conv
M. w/o Gate
Table 4: Ablation. ViM-small for Ima-
geNet Segmentation. Higher is better.
Method
Pixel Acc. mAP mloU
Full Model
79.60
86.40
62.51
w/o SilU
79.32
86.22
62.41
w/o Conv
70.01
78.87
50.64
Gate
75.11
80.12
55.78
Figure Comparative visualization of ablated hidden matrices. 'M' for Mamba.
w/o S6
42,99
64.13
26.01
Only S6
72.39
80.09
53.19
Ablation study. The architectures we explored implicitly parametrize attention matrices through a composition of several different sub-layers, see Eq 21 and 17 Examples of these sub-layers include linear recurrent layers, gate mechanisms, activations, normalization and other components, such as token-shift or depth-wise convolutions. To better understand the contribution of each of these components, conduct sequence of ablation studies. Initially, in Fig. visualize the

---

Published as a conference paper at ICLR 2025
Table 5: Performance of various Mamba-based LLMs Snarks, CommonsenscQA, and Formal
Fallacies datasts. We compare the vanilla model performance, and models employ the Amplified method, with attribution method or the attribution method of al. (2024) (denoted by )
Model Size
Method
Snarks (%)
CommonsenseeQ (%)
Formal Fallacies ( (%)
1.3B
Vanilla
44.54
52.15
40.13
1.3B
AMPLIFY
53.11
53.55
44.28
1.3B
AMPLIFY (Ours)
56.15
54.72
45.22
2,8B
Vanilla
48.75
53.11
44.67
2.8B
AMPLIFY
58.10
56,10
47.80
2.8B
AMPLIFY (Ours)
60.02
56.08
47.86
implicit attention of Mamba, ablating the Conv1D or the gate branch, or focusing solely on the S6 layer. As expected, it seems that the ConvID causes a smoothing effect, and the Mamba matrices are significantly sharper, with more pixels having non-negligible values compared to those of S6.
In T we compare several ablation variants of our method. As can seen, our method, which utilizes all the components of Mamba, achieves a much better score than the ablated versions, illus- trating the importance of all components. This experiment reveals that including the S6, Conv ID and gating mechanism is crucial for high performance and reliable representation. However, the activation has a relatively low impact on these aspects. A similar ablation study was conducted for RWKV and presented in Appendix C demonstrating similar trends.
4.3
IETRIBUTION-BASIE PERFORMANCE: CETEORMMANC--ENIINNIIIO TECHNIQUES
To further demonstrate the practical impact of our representation, we show it can enhances model performance. While various attention- -bsed and explainability-based techniques was previously proposed for improving model performance, focus is on in-context learning (ICL) and weakly supervised semantic segmentation tasks.
To improve ICL, we adopt the AMPLIFY method of Krishna al. (2024). prompt engineering technique designed for few-shot ICL, which leverages post-hoc explanation methods. In our exper- iments, use the Mamba-790n model PLIFY, but with an attribution method that relies on attention matrices. We report the vanilla model performance, and the performance with AMPLIFY with attribution method and the attribution method of Ali et al. (2024). The results are depicted in Tab. 5] show that our method outperforms the baseline across all tested scenarios except one, with an average margin of 1.2 accuracy points over the amplify baseline, and 9.8% over the vanilla baseline.
proxy, following the same evaluation protocol as AM-
Detailed results and experimental settings for weakly supervised semantic segmentation are pre- sented in Appendix B
5 CONCLUSIONS
In this study, we have extended the use of self-attention from its traditional role as the core mech- anism Transformers to a representation of neural sequence layers. Our unified framework facil- itates the exploration of similarities and differences among non-attention layers, such Mamba, RWKV, and Griffin, and their interconnections with Transformer architectures. Additionally, en- ables the development of innovative explainability techniques for the latest attention-fret frree architec- tures. Our contributions provide the research community with new tools for analyzing the perfor- mance, fairness, and robustness of gated-linear RNN variants, while also identifying their potential vulnerabilities. These advancements set the stage for future improvements and support the imple- mentation of weakly supervised downstream tasks.
Looking ahead, we aim to incorporate additional layers, such as Hyena (Poli et al. 2023), and HGRN2 (Qin et al. 2024a) into our framework, including their vision-specific variants DDua al. 2024 Fan et al.) 2023) Zimerman & Wolf, 2024b) Spravil et al.) 2024). Furthermore, we plan examine how differences in these architectures are reflected their self-attention matrices and explore whether such insights can reveal more about the inductive bias inherent in each architecture.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
All of our experiments are conducted using the PyTTrrch framework on public datasets. Addition- ally, our code for some of the experiments is included as supplementary, along with user- sser-friendly interface and notebook demos. Therefore, we consider our empirical results to be reproducible.
ACKNOWLEDGMENTS
This work was supported by a grant from the Tel Aviv University Center for Al and Data Science (TAD). This research was also supported by the Ministry of Innovation, Science & Technology Israel (1001576154) and the Michael Fox Foundation (MJFF-022407). The contribution of the first author is part of PhD thesis research conducted at Tel Aviv University.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 4190-4197, 2020.
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. Interna- tional Conference on Machine Learning, pp. 435-451. PMLR, 2022.
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590. 2024.
Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01771, 2024.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language mod- els. arXiv preprint arXiv:2312.04922 2023.
Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. Entropy-based attention regular- ization frees unintended bias mitigation from lists. arXiv preprint arXiv:2203.09/92 2022.
Ethan Baron, Itamar Zimerman, and Lior Wolf. 2-dimensional state space layer for spatial induc- tive bias. In The Twelfth International Conference on Learning Representations, 2023.
Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space models. arXiv preprint arXiv:2402.088 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth. Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Guy E Bielloch. Prefix sums and their applications. Technical Report, 1990.
Helena Bonaldi, Giuseppe Attanasio, Debora Nozza, and Marco Guerini. Weigh your own words: Improving hate speech counter narrative generation via attention regularization. arXiv preprint arXiv:2309.023/1. 2023.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-mode explainability for interpreting bi- modal and encoder-decoded transformers. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 397-406, 2021a.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 782-791, 2021b.
11

---

Published as a conference paper at ICLR 2025
Hila Chefer, Idan Schwartz, and Lior Wolf. Optimizing relevance maps of vision transformers
improves robustness. Advances in Neural Information Processing Systems, 35:33618-3388 2022.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling- arXiv preprint arXiv: 2014.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: /803.05457 2018.
Edo Cohen-Karlik, Itamar Zimerman, Liane Galanti, Ido Atad, Amir Globerson, and Lior Wolf. On the expressivity of selective state-space layers: A multivariate polynomial approach. arXiv preprint arXiv:2502.02209, 2025.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.21060, 2024.
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Al- bert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.1 19427, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures. arXiv preprint arXiv:2403.023088 2024.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. Rmt: Retentive networks meet vision transformers. arXiv preprint wrXiv:2309.11533, 2023.
Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint 1xXiv:211.003300 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585, 2021b.
Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmen- tation propagation. International Journal of Computer Vision, 110:328 348, 2014. URL
https://api www.stttttnccchoceeeeeeee
org/ http:cuc:::1100555::
Shir Gur, Ameen Ali, and Lior Wolf. Visualization of supervised and self-supervised neural net- works via attribution guided factorization. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11545-11554, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
12

---

Published as a conference paper at ICLR 2025
Farnoush Rezaci Jafari, Grégoire Montavon, Klaus-Rober Müller, and Oliver Eberle. Mambalrp: Explaining selective state space sequence models. The ehihty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of NAACL-HLI, pp. 3543-3556, 2019.
Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling. arXiv preprint arXiv:2311.01 1.01927, 2023.
Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations of language models can improve language models. Advances in Neural Information Processing Systems, 36, 2024.
Jian Li, Zhaopeng Tu, Baosong Yang. Michael R Lyu, and Tong Zhang. Multi-head attention with disagreement regularization. arXiv preprint arXiv: 1810. 10183, 2018.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403.19885 2024.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint 1rXiv:2401.10166. 2024.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-lange and-langaage tasks. Advances neural information processing systems, 32, 2019.
Shahar Lutati, Itamar Zimerman, and Lior Wolf. Focus your attention (with adaptive iir filters). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 12538-12549, 2023.
Xuezhe Ma, Chunting Zhou, Xiang Kong. Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209. 10655, 2022.
Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. arXiv preprint arXiv:/709 7005544 2017.
Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language model- ing via gated state spaces. arXiv preprint arXiv:2206. 13947, 2022.
Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong- Whan Lee. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural net- works. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 2501- 2508, 2020.
Seong-Whan
Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pas- canu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp. 26670-26698. PMLR, 2023.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer arXiv preprint arXiv:2305.13048 2023.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemystaw Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint mxXv:22444058892 2024.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao. Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint ar0iv:2302. 10866, 2023.
13

---

Published as a conference paper at ICLR 2025
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bjöm Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint arXi::2403.17844, 2024.
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrm2: Gated linear mns with state expansion. arXiv preprint arXiv: 2404.004000407444 2024a.
Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for se- quence modeling. Advances in Neural Information Processing Systems, 36, 2024b.
Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16846-16855, 2022.
Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du. Token contrast for reakly-supervised seman- tic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3093-3102, 2023.
Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.049933 2022.
Julian Spravil, Sebastian Houben, and Sven Behnke. Hyenapixel: Global image context with
volutions. arXiv preprint arXiv:2402.19300 2024.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.0800 2023.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprini arXiv: (E0::::990.0799999 2019.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykow. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint trXiv:2307.09288 2023.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint wrXiv:2402.007883 2024a.
Junxiong Wang. Jing Nathan Yan, Albert Gu, and Alexander M Rush. Pretraining without attention. arXiv preprint arXiv:2212.10544 2022.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint 2.kiv:2401113660 2024b.
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivari- ant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12275-12284, 2020.
Rui Xu, Shu Yang, Yihui Wang, Bo Du, and Hao Chen. A survey on vision mamba: Models, applications and challenges. arXiv preprint arXiv:2404. 18861, 2024.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint 1xXiv233/2.06635, 2023.
Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer. arXiv preprint arXiv:2 14103, 2021.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09// 0944177 2024.
Itamar Zimerman and Lior Wolf. Viewing transformers through the lens of long convolutions layers. In Forty-first International Conference Machine Learning, 2024a.
Itamar Zimerman and Lior Wolf. multi-diimensioall hyena for spatial inductive bias. In Interna- fional Conference on Artificial Intelligence and Statistics, pp. 973-981. PMLR, 2024b.
14

---

Published as a conference paper at ICLR 2025
A REPRESENTING ADDITIONAL ARCHITECTURES VIA IMPLICIT ATTENTION
In sec. we present the formulation of Griffin, RWKV, and Mamba via attention matrices. this section, extend method to other layers, such RetNet (Sun et al. 2023) and HGRN QQn
et al. 2024b).
RetNet. The Retention Network is composed of two primary blocks: (i) the Multi-Scale Retention (MSR) layer and the (ii) FFN layer, which operates independently across tokens. The MSR layer, responsible for token mixing, is built on top of the retention sub-layer and defined as follows:
head, Retention X, %) = 1 - 2-5-; Y GroupNorm, (Conca((head,, ... head,, ))
(22)
Furthermore, the outputs are scaled using data-control gate branch, parameterized by matrix Wg E RDXD;
MSR(X) (swish(XWo) @ @
(23)
To refine this formulation, we can represent the element-wise multiplication matrix multipli- cation using a diagonal matrix G (ing(swis((XWec) Additionally, per-head statistics can be integrated into G. Given that the parallel representation of Retention can be depicted via an attention matrix R (see Eq. 5 in (Sun et al. 2023)), the entire MSR block simplifies to:
Retention(z) GRz
(24)
HGRN.
The Hierarchically Gated RNN (HGRN) first defined with the following recurrent rule:
fr = Sigmoid (x, 5 + by) E R 1xd
i, = Sigmoid (x; W; + b,) E 1xd
(25)
C₁ = SIL.U X W. + b.) E R lxd
h, = f & h + i, @ C₁ E IR 1xd
(26)
where the output of the recurrent h2 multiplied by 9: SiLU(Linear(zz)) produce the output:
gt he
(27)
Note that the recurrent rule of the HGRN layer can be computed via an implicit attention represented by matrix ar (see Eq. 5 in QQin et al. 2024b)), as follows:
H (hi,'.. hL); C (C1,
CL),
(28)
Hence, by define G = (iig(S(SULiina(z))) Gaot diag (z)) Furthermore, we can rearrange the linear layer such that W, and by will be omitted, and obtain:
GACT = diag (Sggmo)iio)))
Ga,GA
(29)
which is linear operator characterized by an input-dependent matrix, defined as G diag(Sig diag(Sigmoid(z)). The output o is given by Ga.-
=
fr = Sigmoid (x, W, + by) E R glxd
i, = Sigmoid (x: W; + b.) E Rixd
(30)
= SiLU (x, W. + b.) E R Ixd
h, = f @ h + i, @ C₁ E R Ixd
(31)
where the output of the recurrent h; multiplied by g: = SiLU(Linar(r)) to produce the output:
0 9t h
(32)
Note that the recurrent rule of the HRGU layer can be computed via an implicit attention represented by matrix a, (see Eq. 5 in (Qin et al. 2024b)), as follows:
H (h,, ht), C (o,," Ct), H = a.c
(33)
15

---

Published as a conference paper at ICLR 2025
Hence, by define G ding(SiLU(Linar(z))), Gact diag(sigmoid(z)).
Furthermore, we can rearrange the linear layer such the Wub, will be omitted, and obtain:
GACT = diag (Sigmoi((x)
= GaCT
(34)
as requested.
B WEAKLY SUPERVISED SEMANTIC SEGMENTATION
In weakly supervised semantic segmentation (WSSS), common approach involves first training classifier on image-level labels and then extracting Class Activation Maps (CAMs) for individual images, which highlight regions that the classifier deems relevant specific classes. The SoTA methods then employ these CAMs as pseudo-masks to train a segmentation decoder.
In this context, we adopt our proposed Mamba-Attr XAI method for vision-Mamba models. We assess its competitiveness against the well-established CAMs in generating pseudo-labels for Trans- formers. To ensure a fair comparison, we fine-tune both DeiT-Small and ViM-Small models under identical conditions over the Pascal-voc 2012 (Everingham et al.) 2010) dataset, excluding multi- scale training, inference, or any other modifications. This controlled setting isolates the influence of our Mamba-Attr method on the quality of the generated pseudo-labels.
The results are presented in Table 6 Evidently, Mamba-Attr XAI method achieves competitive results surpassing the baseline approach of Class Activation Maps (CAMs) without any additional modifications. This is evident in the mean dntersection-over-uree Attr (1%) outperforms the CAM of DeiT-Small (35.99%) by sizable gap. While Mamba-Attr does not reach the state-of-the-ar performance Toco (Ru et al. 2023) (61.10%). it achieves, out CAM and comes surprisingly close to this much more
(mIoU) score, where Mamba-
of the box, substantial improvement elaborate multi-phase learning method which utilizes multiple loss terms specifically designed to enhance the quality of the initial CAM map. These results suggest that Mamba-Attr XAI offers powerful and efficient solution for WSSS tasks with vision-Mamba models.
Table 6: Evaluation and comparison of the pseudo-labels for the different classes in Pascal-voc
2012 [Everingham et al. 2010). Results are mloU.
Mig
3070
Meyyde
bird
cluitr
dag
train
Deit-S Tooo
(86.00 56.49 37.27 76.80 45.73 58.31 82.56 71.95 76.75 30,66 79.13 71.44 81.31 71.11 59.51 44,87 80,8 41.81 60.73 29.72
69.12
61.10
Deit-S
49.12
17.53 25.01
14.49
19.43
55.55
49.33
37.92
61.32
15.85
34,90
55.27
32.36
29.83
64.82
33.24
23.84
32.53
32.76
36.40
35.99
Mamba-Alir
37.23
20.76
56.57
52.11
C PERTURBATION EXPERIMENTS FOR NLP
In this section, we present results for the perturbation test in the NLP domain with fine-tuned classi- fiers. In this setting, we fine-tune the last layers of various LLMs and append the [CLS] token to all samples to generate explanation maps, similar to the methods used in vision models.
we present perturbation results for both positive and negative settings. We utilize Mamba-] RWKV WPaad BERT models as the models of interest and fine-tune them on the IMDB sen- timent classification dataset, employing the [CLS] token for all samples. Subsequently, we evaluate the explanation quality using similar procedure proposed in (Ali et al. 2024) for the perturbation experiment.
In Figure 6
The results reveal that Mamba-attr, based on attention formulation, achieves superior AUC
for both negative and positive perturbations compared to the previous attention formulation by (Ali
httee///ugguutsee.otttrnmaaaaaaa text classification
16

---

Published as a conference paper at ICLR 2025
al. 2024). Additionally, unified attention formulation is effective for RWKV models, yielding comparable results to those of Mamba and BERT.
Moreover, ablation, the first column of Figure 6 demonstrates that including the gate branch, as presented in our full method, consistently improves performance.
RWKV Activation
Mamba Activation
BERT Activation
1.0
1.0
1.0
0.9
0.9
0.9
0.8
0.8
0.8
p
0.7
Q 0.7
2 0.7
0.6
0.6
0.5
0.5
0.5
0.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
% of words added
% of words added
% of words added
Random RWKV-Attr
RWKV-Attr w/o Gate
Mamba-Attr et al., '24 Mamba-Attr Ours
TransformerAAtr
RWKV Pruning
Mamba Pruning
BERT Pruning
10
28
12
8
24
6y0 mt)
mt)
10
8
16
6
I
12
4
S
2
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
% of words removed
of words removed
% of words removed
Random
RWKV-Attr w/o Gate
Mamba-Attr Ali et al., '24 Mamba-Attr Ours
RWKV-Attr
Transformer-Attr
Figure 6: Evaluation of explanations using input perturbations. Results for IMDB activation task (top row) in which the most relevant words are added first, and for IMDB pruning task (lower row) in which the words of least relevance are removed first. Results shown for different models: RWKV, Mamba, and BERT, respectively.
D ADDITIONAL QUALITATIVE RESULTS FOR NLP
Additional NI.P results obtained on IMDB dataset are presented Figure In panel (a), we show the results for the previously proposed Mamba's 's attention (Ali et al. 2024 Panel (b) shows proposed Mamba's is attention. Lastly, panel(c) presents our proposed method over RWKV. In red, we show negative sentiment, and in blue, show positive sentiment.
As can be seen from these qualitative results, the explanation maps generated by our new attention formulation exhibit sparser and more accurate heatmaps of relevant words than those of Ali et al. (2024), aligning with the desired properties of XAI methods. Similarly, the results for RWKV models show comparable success those of Mamba.
17

---

Published as a conference paper at ICLR 2025
barzox fiis the fosis darneaar typimi horrer coviss Fonaee annoyin fush 1tee ane
are looking for, I suggest you aes Browese corrible
en very dimppointe vith this sovie bacause : like these E rench actors and i liked Buffet Froid fren this Disector bertraad bite) but the script of Acteurs vSa 1008.
Rhat can I say, dus if you atiis haven C.. Great canEca works and lighting tachaiques Ransora, just merors Orkon Nolles is incrodibie The Lady From shanghas can cert certainly take the place of Citiien Kane
very good movis wish wished could sind is in vig to Juy reall enjoy od this movie detinket would 1ike ade
just wnbeleavala
mhe worst movie have seen in waile. Yeak its Fun to iastasiso, but if that is what you
to watch but never rind in tv it would be well wozth the time
Table 7: Additional qualitative results on the IMDB dataset. (a) The Mamba attention of Ali et al. 2024). (b) Our Mamba attention method. (c) Our RWKV attention.
Thuze's ase that ado it to 2e0t for hugel
sise can't why sapody would not love Boar
actore they agrend to play prot rcenario
Boog movis
tntrihre novi non's sen this K omin
fans: Everyose eire this is just rery bad
ass he descrise those see why this non aa sent In the top 250.
larre: rouLe
it a mut mn
Euale charecte
are
corrible
axesone. DEson Aves ona, Shanghak
thig wag very good novia I wi ghod I could find is in the to buy really anjoy Ehis nopie Maten rocom rocommond wou would iiee to cbe. it again but ean nover find sould be well worth
dinapolnted with this novis bacause 1ike these actors and Liked Buffet
The voEst movia I have seen in a while M Xeak its run to fantaciso ael but if that is what you are looking for, suggest you
Reg it if you still have t. ureat canors WOzKS lighting ques
or 1ike turrible movies, don't t see this oneg
time
There one 1i0e that sakes it warts to rent for Angol
from this pirector (bertrand biler) but the script of Les Actaurs is VERY POOR. why these
glad when they kiled off. Long live the stzong fomale character in harror noviell
actors they agrued to play this pooz somaario
what can eay, it's s dann good morid:
fins. Everyone eise: this is just very BM
dumb. are
place 'Eitiren
deneribad thees best rovis SRE nade jut
see uny thie noris in inss in san top 188)
face, Reyeoys is you havnt sees it
(b)
chiracter,
Uniess
in
incredible
is a mot bay
woodes
InTOR fllck
being
it in tv,
rais nouis
females
arowsters cerrible
what can say, it's a damn good movic see it it you still have wnn't t Great camera works and lighting techni ques. Awesone, just awes avesone Okson Holles incredible 'hhe Lady From Shanghai can cortainly take the place of Citisen Kane Ts
this was very wood movie I wi shed I could find is the bey, realiy enjcya rocommend
disapelttedd WIth DI becauze like these Buffet
The worst movie have seen in while Yeah Its fun to fantasino aiioo but if that is what you
wateh weul would like to sac. is aga in but can ROVOr find in
looking for, suggest you corsy like
There one 1ine that makes worta rent for Angel
movia
time
glad shes they are killed off. Long live the stzong
ible movies. den't t aue this onez
Just & very
The They are vooden, ennoyion and dunb, Yor
satch" would be well vorth
see sky tasa sovie lnst in the top 1501 al oas's wky unghnay facel
those
fomale chrrectur in a horrer noviell
madey
it it SURME buyd seas
af Les Actours is VERY FOOR. actors agreed
Unlens
Director bertrand blier)
bast
fans: peerone
actors
at being
18

---

Published as a conference paper at ICLR 2025
E UNDERSTANDING THE INNER DYNAMICS OF GATED-LINEAR RNNS
24 34 reveals that several modern gated-linear RNNs can be viewed as data-controlled linear operators. Specifically, these models can be expressed in the form Y = AX where A is an implicit attention matrix dynamically controlled by the input data. This matrix layers, Conv ID, linear layers, gating mechanisms, activations, and normalization layers.
Our unified formulation, presented in Equations
parameterized through various sub-layers, including linear recurrent
Beyond the applications discussed in Section 2 this formulation sheds light on the inner dynam- ics and design choices inherent in the parameterization of these layers. It suggests that the entire mixing block in these models, including several sub-layers, implicitly parameterizes complex and expressive attention matrices. This implies that these components collectively enable the model to efficiently and implicitly express thousands of data-dependent (where transformers of the same size typically have around few dozen), complex attention heads.
Given the distinct behaviors of these sub-layers, we propose that they incorporate specific induc- tive bias into the model. In particular, the Conv ID layers promote the inductive bias for attention matrices to be relatively local and smooth. Similarly, the gating mechanisms, linear layers, and activations allow the model to capture sparse and complex (non-linear) features that depend on the input data. Finally, the LayerNorm can be reinterpreted as a per-row data-dependent scaling of the attention matrix, analogous the role of the softmax function in Transformers.
Our unified view also enables us to categorize attention models into three distinct types based on their expressivity and efficiency: (i) Explicit Attention Models: Models like the Transformer com- pute the attention matrix A explicitly during the forward pass, resulting in quadratic space complex- ity with respect to sequence length. The attention weights are directly calculated from the input data, allowing straightforward interpretability of how each element attends to others. (ii) Implicit Atten- tion Models: Models such as RWKV, Mamba, and Griffin fall into this category. Although they do not compute the attention matrix explicitly during the forward path, our formulation demonstrates that their operations can be interpreted as implicit attention mechanisms. This implicit computation allows for greater efficiency with sub-quadratic complexity. To further enhance the expressivity and inductive bias these implicit attention matrices, several sub-layers can be utilized. (iii) Attention Tensor Models: In these models, the operation cannot be represented simply as Y AX with being a matrix. Instead, A is a higher-order tensor, capturing more complex interactions beyond pairwise relationships. The S5 model serves an example, where the data-control linear operator does not mix tokens for each channel independently.
This categorization highlights the trade-offs and design considerations in different architectures. In particular, explicit attention models are relatively more expressive but may suffer from higher computational costs. In contrast implicit attention models that balance between efficiency and expressivity. Lastly, the attention tensor based models provide the capacity to model more complex interactions the expense of increased computational complexity and less interpretable representa- tion.
This perspective not only enhances our theoretical understanding but also has practical implications. It suggests that critical design choices in these models, such as the inclusion of Conv ID layers, gat- ing mechanisms, and normalization techniques are critical for shaping the implicit attention patterns. Future work can leverage this insight to optimize architectures further, potentially leading to more efficient and expressive sequence models.
19


Reviews:
[{"review_id": "FMgXdF5GXS", "reviewer": "Reviewer_xsWF", "summary": "The paper studies the problem of sequence modelling. The authors aim to provide a unified framework of the recent attention-free methods such as Mamba and RWKV. The paper presents empirical results to validate the proposed unified framework.", "questions": "- What is the best method to evaluate the unified framework beside the interpretability analysis?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The unified framework makes it easier to study and compare the different sequence modelling algorithms.\n- It is important for the community to learn about such work.\n- The experimental results are interesting.", "weaknesses": "Although it is hard to evaluate such approaches empirically, interpretability-based metrics are not very conclusive in general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3FIWHBesc", "reviewer": "Reviewer_JKNZ", "summary": "This paper introduces a unified framework that reformulates various gated recurrent neural network (RNN) architectures, such as Mamba, RWKV, and Griffin, into implicit causal self-attention layers. This reinterpretation aims to make these models more interpretable by constructing attention-like matrices for use in visual and NLP explainability tasks. Experimental evaluations demonstrate that this approach achieves competitive performance in robustness and attribution metrics, though prior work has already suggested that certain gated models, including Mamba, are attention-based.", "questions": "A critical question is: given that previous work has noted the attention-like properties of models such as Mamba, what specific benefits does the implicit attention framework offer over these prior interpretations?\n\nWhat are the current explainability methods or metrics for modern gated-linear RNNs and how's the comparison between them and attention matrices?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper offers a clear explanation of how implicit attention could be used to interpret gated RNNs, making it accessible to readers interested in explainability across model types.\n\nBy applying the framework to both NLP and vision tasks, the authors demonstrate its cross-domain relevance.", "weaknesses": "Previous work has already conceptualized models like Mamba as attention-like, meaning that simply reinterpreting these gated RNNs under an implicit attention framework may not be largely novel.\n\nThe paper does not thoroughly compare its implicit attention framework with existing interpretability tools for gated RNNs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j4aTQYkpH4", "reviewer": "Reviewer_rDjF", "summary": "This paper writes out gated recurrent architectures such as Mamba, RKWV, and Griffin as causal self-attention layers. The main objective of this is to increase interpretability, which is tested through perturbation and segmentation tests.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my view, this is a pretty complete paper. From my understanding, the authors present an extension of the paper Ali et al. (2024) to include additional architectural components in the linearization and not just the S6 mechanism considered before. The authors show that the model improves interpretability through visualizations and a set of both perturbation and segmentation tests. The ablation gives quite a lot of strength to their arguments, but I am not so familiar with these types of explainability results so I am not able to comment on the details.", "weaknesses": "While the authors have explored the interpretability side of things extensively, I was wondering if it would be worth comparing the performance of the linearized models compared to its recurrent counterparts when trained on some small datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9hrM5tNmH", "reviewer": "Reviewer_PBPq", "summary": "This paper tries to provide an implicit self-attention formulation for most state-of-the art non-transformer sequence models (known as Gated Linear RNNs) such as Mamba, Griffin, RWKV, and ReNet. In this way, it can exploit techniques used in attention explainablity to explain these new models.\n\nCompared to the closest work (Ali et al, 2024), which only formulates the S6 layer in Mamba, the main contribution of paper is:\n- Formulating more layers with implicit self-attention and propose a unified and more accurate attention-based representation for all the SOTA gated linear RNNs.\n\nOther contributions include:\n- Introducing new explainablity technique for these models leveraging the self-attention formulation\n- Showing performance of their explanations and attributions by purturbation testing and segmentation.\n- Showing their proposed formulation can give attributions which can further used in some performance-enhancing techniques (based on in-context learning) for large language models.", "questions": "mentioned above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Covering multiple models including most popular modern non-transformer sequence models.\n- Evaluating the performance of the resulting attributions in multiple quantitate experiments and down-stream tasks (across both vision and NLP).\n- Showing the impact of various sub-layers in the ablations study", "weaknesses": "- In their ablation study, the authors could discuss the trade-off between the time explainability and more accurate formulation/explainability.\n\n- The main baseline paper (Ali et al, 2024) has not been published yet. So, it is hard to evaluate this paper. Actually, the performance of the model in downstream tasks such as segmentation and attribution-based performance-enhancement helped me to have better evaluation of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)