Focus Your Attention (with Adaptive IIR Filters)
Authors: Shahar Lutati
Itamar Zimerman
Lior Wolf
Venue: EMNLP 2023 Main (2023)
Keywords: IIR Filters
dynamic Filtering
Transformers
TL;DR: We present a novel layer that leverage dynamic IIR filters to focus the attention mechanism, strengthening its long-range modeling capabilities.

Abstract: We present a new layer in which dynamic (i.e., input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. The input is split into chunks, and the coefficients of these filters are determined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus attention on the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-the-art networks, with a fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable
to layers such as Heyna, GPT2, and Mega, both with respect to the number of parameters and the obtained level of performance on multiple long-range sequence problems.

Focus Your Attention (with Adaptive IIR Filters)
Shahar Lutati and Itamar Zimerman and Lior Wolf
The School of Computer Science, Tel Aviv University 7 com z zerrmann @mail. tau. acc il
wolf tau.ac. il
Abstract
that could be efficiently computed via a global con- volution. Recent research has explored alterna- tive variations of global convolutions, including implicit (Poli et al., 2023) and regularized parame- terization (Fu et al., 2023a; Li et al., 2022). These methods have demonstrated improved performance on tasks involving long-range dependencies and with sub-quantitative complexity. They have also shown effectiveness in enhancing long-range trans- former capabilities (Ma et al., 2022; Zuo al., 2022; Saon et al., 2023). However, remains un- certain whether these models can scale up or func- tion similarly to transformers across a diverse range of tasks (Vardasbi et al., 2023).
We present a new layer in which dynamic (i.e., input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conven- tional attention. The input is split into chunks, and the coefficients of these filters are deter- mined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus atten- tion the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-te oftte--ffthheeeee art networks, with fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable to layers such as Heyna, GPT2, and Mega, both with respect the number of parameters and the obtained level of performance on multiple long-range sequence problems.
This work strives to efficiently integrate convolution-basel sequence models and transform- ers, to provide model that is capable of handling both short and long dependencies. The attempt to combine these components was first presented by Ma al. (2022), who used a simple global convolution before each transformer block. This convolution is parameterized by the Exponential Moving Average (EMA) recurrent rule and can be seen as an IIR filter. In this work, instead of using first-order IIR filters, we introduce learnable adap- tive IIR filters, which allow to propose Focus, layer that combines local attention and novel type of regularized global convolution grounded on hypernetwork that produces adaptive IIR filters.
1 Introduction
Designing sequence models that capture short- and long-term dependencies a central goal of se- quence modeling. Besides performance, compu- tational complexity also plays part when deal- ing with long sequences. Although transform- ers (Vaswani et al., 2017) excel at tasks that involve short-range dependencies, their performance data with long-range dependencies can be poor. For example, regardless of the high time complexity, on the Long Range Arena benchmark (LRA) (Tay et al., 2020) transformers perform poorly compared to other sequence models.
Our main contribution is the focus layer, which has several unique properties: (i) We are the first to use data-dependent global filters, which are im- plemented by a global hyper-network mechanism that focuses local attention. (ii) In contrast to other works in the domain that employ FIR filters, it re- lies on IIR filters. (iii.. We present efficient and stable computation of those IIR filters. (iv) Theoretically, our layer is grounded in the theory of control systems, similar to state-space layers, which are built on the state-space model (SSM) of control theory. Furthermore, in Sec. 4 we show
Another approach that has emerged to address long-range data processing is the utilization of reg- ularized implicit global (long) convolutions. In this technique, convolutions are employed along the sequence dimension, enabling convolutions with a global receptive field. Initially, this approach was implemented through state-space layers (Gu et al., b,a), which introduced a recurrent layer

---

that 1IR filters are generalization of SSMs and diagonal-linear RNNs, which have recently been recognized as remarkable long-range learning ar- chitectures (Gupta et al., 2022a; Gu et al., 2022; Orvieto et al., 2023; Gupta et al., 2022h; Saon et al., 2023; David et al., 2023). By drawing upon the extensive research conducted on IRR filters, our findings can provide additional insights into the effectiveness, stability, expressiveness, and initial- ization of those layers.
and train it on guitar signal.
Global Convolutions also known as a long convolution, is a layer that applies scalar convolutions along the sequence di- mension, enabling the handling of unrestricted I-D sequences. Empirically, these layers have shown strong performance in tasks involving long range dependencies, particularly in domains such NLP (Dao al., 2022b; Mehta et al., 2022; Wang at. 2022), audio (Goel al., 2022), speech (Saon et al., 2023), video (Islam et al., 2022; Wang et al., 2023), time-series analysis (Zhang et al.. 2023) and more. Moreover, they exhibit computational effi- ciency their cost is sub-quadratic. However, to achieve SOTA results, appropriate regularization is necessary. The approach of (Gu et al., 2021 Ma et al., 2022; Li al., 2022) incorporates param- eterization that inherently regularizes the kemnel and decoupling sequence length from parameter count. (Romero et al., 2021; Poli et al., 2023) uti- lizes an implicit parameterization learned by FFNs operating on positional encodings, while (Fu et al., 2023a) explicitly regularizes the convolution ker- nels using squash or smooth operators.
The global convolution.
2 Background and related work
IIR filters, known as infinite impulse response fil- ters, are digital filters that utilize feedback to gen- erate an output signal. Their primary applications involve signal smoothing, filtering. and signal mod- ification. These filters are extensively employed in various fields, such as audio processing, speech processing, and image processing. One notable advantage of IIR filters their ability to achieve a significantly sharper roll-off the transition region compared to an FIR filter of the same order. This is made possible by the presence of complex poles in the IIR filters, which enable them to attenuate frequencies more rapidly.
Transform- ers (Vaswani et al., 2017) have emerged as highly effective models for various tasks, but their widespread adoption has been limited by the quadratic cost of the self-attention mechanism and poor performance on long-range tasks. Researchers have pursued diverse approaches to overcome this challenge and to create efficient transformer archi- tectures (Fournier et al., 2021; Tay et al., 2022). From the perspective of efficiency, techniques such as sparse attention (Child et al., 2019), low-rank attention (Wang et al., 2020; Winata et al., 2020), kernel- based attention (Choromanski et al., 2020), recurrent mechanisms (Hutchins et al., 2022; Dai et al., 2019), and efficient IO- (O-awareness-ban based implementation (Dao et al., 2022a) proved efficient. From the perspective of effectiveness, (Yu et al., 2023; Ivgi et al., 2023) combines local and global attention models hierarchically, enhancing the model's ability to handle extensive context Other techniques employ global memory-based Attention (Gupta and Berant, 2020; Al Adel, 2022; Al Adel and Burtsev, 2021), and (Zhou et al.. 2022) applies attention in the frequency domain to expand long-range capabilities.
Long Range Transformers
The state-space representation of an IR filter a convenient way to represent the filter's dynamics and to implement it in software. It consists of three parts: the state vector, the state transition matrix, and the output matrix. The state vector contains the filter's internal state variables, the state transition matrix describes how the state vector changes over time, and the output matrix describes how the output signal computed from the state vector. Such representation are described in (Zhang et al., 2023)
Learnable IIR Filters Since IIR filters are com- putationally efficient, yet expressive, it is natural to design IIR filters with Deep Learning. (Kuznetsov al., 2020) proposes an approach to using tra- ditional digital IR filter structures inside deep- learning networks trained using backpropagation. The authors establish the link between such struc- tures and recurrent neural networks and present three different differentiable IIR filter topologies. They compare the proposed topologies against each other and an established baseline and show that the proposed topologies can achieve better perfor- mance in some cases. Additionally, the authors present simple Hemner-Hammerttenn model, us- ing differentiable IIRs as its filtering component,
Hyper Networks A hypernetwork (Ha et al., 2016) is a function that maps a set of inputs to

---

set of weights, which are used as the parame- ters of a "primary network". Hypernetworks have been shown to effective for a variety of tasks, including, for example, image classification (Lutati and Wolf, 2023), natural language processing (He et al., 2022), and speech recognition (Szatkowski et al.. 2022). They have also been shown to be able to improve the performance of neural net- works on meta-learning tasks, such as few-shot learning (Bertinetto et al., 2016), continual learn- ing (Von Oswald al., 2019), and neural architec- ture search (Zhang et al., 2019).
global models Given the quadratic complexity of transformers, chunking is a common practice for computing short-range attention efficiently. How- ever, despite excelling in short-range tasks, full- length transformers often struggle to handle long- range dependencies and often perform comparably to local-attention-basee transformers (Xiong et al., 2021). Recent studies have demonstrated that a combination of local and global transformers achieve state-of-the-as performance on long-range tasks (Ivgi et al., 2023; Yu et al., 2023; Hutchins et al., 2022). Inspired by these findings, we intro- duce local attention as the local model, which is combined with a novel type of global convolution as the global model. Furthermore, in contrast to (Hutchins et al., 2022; Bulatov et al., 2023), our global model does not use recurrent computations, since it severely restricts parallelization.
Adaptive Filtering Adaptive filtering is a tech- nique used to improve the quality of a signal by removing noise or interference. Adaptive filters are able to adapt to changes in the signal or the environment, making them well-suited for variety of applications.
One common technique used in adaptive filter- ing is the short-time Fourier transform (STFT), which provides a time-frequency representation of a signal. It enables the analysis of time-varying properties of signal by dividing it into short-time windows and applying the Fourier transform to each window. The STFT reveals the distribution of frequency content over time, which allows the adaptive filter to track the frequency content of the signal and adapt its coefficients accordingly. How- ever, STFT introduce non-causal implementation due to overlapping time-bing bins. To mitigate it, we introduce FFI, degenerate form of the STFT.
Adaptive IRR Filters In MEGA (Ma et al., 2022), it was demonstrated that incorporating EMA at the beginning of each transformer block improves transformer performance in long-range tasks. EMA can be viewed as a convolution opera- tion using simple first-order IIR filters. Motivated by this finding, we adopt a more versatile and ex- pressive convolution approach that utilizes adaptive filters generated by a hypernetwork. Since the hy- pernetwork is an integral part of our global model, it employs global convolutions. Specifically, the regularized global convolution of (Fu al., 2023a) used, the most straightforward option. A com- mon challenge with hypemetworks is ensuring rela- tively small output sizes. this regard, leveraging IRR filters, which have only a few parameters, reasonable choice.
Recent research in Al has focused on using deep learning improve the performance of adaptive filters. For example, deep learning has been used to improve the performance of adaptive filters for noise cancellation (Zhang and Wang, 2021), 1) echo cancellation (Haubner and Kellermann, 2022), and cqualization (Zhou et al., 2020). Deep learning has also been used to develop new adaptive filter architectures that are more robust to noise and inter- ference (Alwan and Hussain, 2022). Revach et al. (2022) demonstrate how deep learning can be used to improve the performance of Kalman filtering (Kalman, 1960), a classical control algorithm.
3.2 The Focus Layer
In this section, we describe the focus attention head, our primary contribution. This head is in- tegrated into the MEGA backbone, as visualized in Fig 1. Let x be the input for the focus layer, where E R L is the sequence length, and D is the input's dimension. Our method, termed Focus, utilizes the foundations of adaptive filtering theory to cope with very long stochastic sequences. Given the seasonality of the sequence, the resolution of the FFT is determined by its size in each time-bin. Denote the size of the FFT in single time-bin as FF...
3 Method
3.1 Overview
We start by discussing the main design choices of our architecture.
The first component of the Focus layer is the hypernework, H. The output of H is θ which
Chunking and the combination of local and

---

(a) Focus Head
S2P
Q
H Looo
K
P2S
y
FFTehunked chunked
X
I1R
X
IFFT chunked
S2P
V
Attention Head
(b) Focus Layer
Focus Model
Focus Head & Residual
H globel
Gate & Normalization
Focus
Focus Layer
MLP
Layer
Layer
Normalization
Figure 1: Focus Architecture: (a) The architecture of single head. (b) The obtained layer. (c) The entire model. The architecture of the model and layer defined similarly MEGA (Ma al., 2022). Blocks in blue not learned, while blocks red are learned parameters. S2P (serial to parallel) and P2S (parallel to serial) are the chunking and the de-chunking operations, respectively.
is the set of UR kernels used for the forward pro- cessing of the sequence. @ has dimension of Nbins D x F x 2, denoting F kernels, each with kernel size of two for D feature channels. The kernel is unique per time-bin, Nbins, which makes the filter adaptive changes over time.
The second component of H is a 2-layer MLP with sigmoid activations that maps the embedding to a tensor with size Nbins D x F x 2. With mapping of latent dimension O to 2 F
θ = MLP(e),
(3)
where θ is the IIR kernel, with size Nbins x D F 2. MLP is the forward MLP mapping, described above. Since H is hypernetwork, the initialization of the last layer of the MLPs follows (Chang al., 2020). The rest of the layers follow the Xavier initialization (Glorot and Bengio, 2010). The input x is split into non-overlapping time bins, where each time bin is passed through the FFT of the size NFFT. Denote the input in the r-th time bin as Ty.
==))).
(1)
H has two main components. The first is shallow global convolution (Fu et al., 2023b) based sub- model that is followed by adaptive max pooling (over each feature channel) (Pytorch, 2023) with a size of O x Nbins, where O is the oversampling factor.
e =
(2)
X X[wrr] [..r)=FFT(rr,)
(4)
where e is the embedding from processing & us- ing the global convolution layer. This computation can be shared across multiple Focus layers and can be split into local (Hoeaa) and global (Hglobal) sub- components, in most of our experiments, thus reducing substantially the computational cost. Fur- thermore, the embedding is permuted such that the feature space has the size while Nbins is added to the batch dimension for parallel computing.
where w is the normalized frequency variable, sam- pled evenly on 2T range, and is the index over the different time bins.
A note about causality To maintain fully causal model that is applicable auto-regressive tasks, each time bin processed on its own and is not overlapped with other time bins. In addition, is shifted right by one time bin, such that the

---

sequence time bin is processed by the kernel computed from time bin 1. For each time-bin index, the corresponding IR filter is applied. The IIR filter of order 2 has the following frequency response, denote as Rimp
IR Impulse Resporase
4..0
IRRamp ( 1 0000 2- 1 [[1] A-f 1
10
20
30
40
80
60
[semplea]
(5) Since a Sigmoid activation is used for the last layer of H, it is guaranteed that O's elements are positive real numbers smaller than 1. Further anal- ysis and reasoning behind the specific selections made are presented in Sec. 4.
Figure 2: Filter responses for three random filters with the specific denominator structure Eq.
where Q, K, V are the query, key, and value matri- ces that map each chunk to their latent correspond- ing space. Note that the attention head is causal, as it uses lower triangle masking. The chunks, y² are rearranged to form a complete signal, with sequence length L, denote it as y- Following (Ma et al., 2022), the output of the Focus layer, y. passes through reset gate the update gate vb. Specifi- cally.
Recall that in the frequency domain, the equiva- lent of filtering is multiplying with the conjugated impulse response.
y X I1Rimp'
(6)
where the conjugation is denoted by a star and is the elementwise (Hadamard) multiplication. The hyper-dimension F defined earlier as the filter-bank size is collapsed via regular sum operation, denote the collapsed tensor as X
7 SiLU(x, W, + b,)
(11)
Φ = sigmoid ( W + b&)
(12)
SiLU(x, + (7 U) + by).
(13)
X 1
where Wy, Wa and W,, are learned matrices with size 1) D. by, bor and bn are learned biases with size D. SiLU stands for the sigmoid lincar unit (Elfwing et al., 2018).
where an all-ones vector with size 1 x F. The collapsed tensor is the short-time Fourier represen- tation of the original sequence filtered with adap- tive filter kernels. The frequency representation goes through the inverse chunked Fourier trans- form (IFFT), obtain the time-domain sequence.
The final output is the gated summation of the gated attention and the input sequence,
ー δ 0 +
(1
(14)
f = IFFT(X))
(8)
Analysis
The time-domaie sequence has the same dimen- sions as the original sequence, yet, by applying an adaptive filter to it, we furnish with an induction bias that helps smaller context attention head to cope with complicated tasks.
IR and FIR Filters IRR (Infinite Impulse Re- sponse) and FIR (Finite Impulse Response) filters two commonly used types of digital filters with distinct characteristics. The main difference be- tween them lies in their impulse response and fil- tering properties. FIR filters have a finite impulse response, meaning that the filter output is based solely on a finite number of past input samples. In contrast, IIR filters have an infinite impulse re- sponse, allowing the filter output to depend on both past and future input samples.
The sequence is split into C' separate non- overlapping chunks. Denote the chunk length as M, such that L MC. We denote the chunk of signal with uppercase i, and use square brackets for indexing, starting with index O, follows
x. (
((+1))-11) (9)
One advantage of UI filters is their ability achieve desired frequency response with fewer coefficients compared to FIR filters. This makes IRR filters more computationally efficient, requiring fewer calculations and lower memory requirements.
All chunks are processed in parallel with the same small attention head,
y' = AAtten(Qu' Krj, Vay),
(10)

---

This can achieved if b is positive. In the see- nario where A < O, the exponents have an imagi- nary part, causing it to oscillate. This is called an under-damped response. This response is stable, yet more expressive than Exponential Moving Av- erage (EMA) filters, found in MEGA (Ma et al., 2022). The frequency of the sine, w, in this case is Consequently, in control feedback systems where real-time operation and compulational efficiency are crucial, IIR filters are often preferred.
and the time domain impulse response reads,
ITR(() -. 2za sin(wt + Φ), (21)
Additionally, IIR filters can exhibit higher selec- tivity and sharper roll-off in the frequency domain compared to FIR filters. This characteristic can be advantageous in control feedback systems, where precise control over specific frequency components is necessary.
where ? is the normalizing factor, and e is the phase from aggregating both sine and cosine func- tions with the same frequency. Note that for orders above 2, there is no simple condition that guar- antees that the real part will be negative and the response stable.
Stability IRR filters can more sensitive to quan- tization errors and can be prone to instability not properly designed. The presence of feedback loops in control systems can further impact stability considerations. Therefore, careful attention must be given to stability analysis and appropriate filter design techniques to ensure reliable performance.
To demonstrate the oscillating behavior of the generated I1R filters, the time-domain impulse re- sponses of some random kernels are drawn in Fig. 2. In this plot, the purple kernel acts as EMA, while other kernels have more complicated response. However, all filters decay as time increases, which leads to stable response and has been identified by Li et al. (2022) as an essential property for cap- turing long-range dependencies.
The filter can be described in the frequency do- main as a rational polynomial function of the com- plex exponent e - Denote the complex expo- nent as S.
Time Complexity The global conv time complex-
ity is
S= c
(15)
GlobalConv se O(Llog(L) D)
(22)
A second-order system can be described as follows,
This is due to the FFT and IFFT that the signal is passed through. This computation done only once and is shared through multiple layers of Focus. The MLP mapping between D and
1
IIR(S) - as² + + +1
(16)
Specifically solving for general b, gives,
IIR(L)= ace
4n) 2a
+.3 20
MLP N O(DF)
(23)
(17)
The chunked-FFT complexity depends on the size of the FFT used (NFFT). Denote the size of a single time bin R.
where a and f3 are normalizing factors. Denote the term under the square root as discriminant, A. For any 2 and 2 0 the following holds:
R L
(24)
A < 62 V(a b} 20. 0.
(18)
Nbins
The time complexity of a single time bin is The total time complexity of the chunked FFT reads,
In order to guarantee stability, both exponents should decay, which leads to the requirement that the real part must be negative.
FFF R (LLog((R)
(25)
Ref
2a +
) 50.
(19)
2a
Next, the time complexity of the attention head depends on the size of the context length M,
Atten OCCM2 CMD*)
(26)
The total time complexity of the Focus layer is, therefore.
Focus = O(Llog(L) D CC
(27)
where we neglected smaller terms when the se- quence length is large (greater than dimensions).
A
(20)
2a

---

that the IRR filter can be constrained to linear SSM. In machine learning. D is often omitted in SSMs or diagonal RNNs, since it can be seen as a parameter- based skip-connection. In this case, the SSM can be represented by an IIR filter of order 1. As mentioned earlier, higher-order filters can introduce stability issues. Therefore, our decision utilize IIR filters of order 2 is justified, as we opt for the most expressive IIR filters that still maintain stability during training.
10
10"
Torus contect
104
Poces contact 512
Fozus context 1024
attentoon
10000
20000
30000
10000
S6000
60000
Experiments
Sequenci Length
Figure 3: Time Complexity of the Focus layer and of Attention, increasing sequence length
Below we present experimental results for the pro- posed Focus layer. In addition our full method, introduce ablation evaluate the importance of adaptive filtering, in which instead of the hyper- network H, the IIR filters are conventional learned parameters. This ablation denoted by "Focus-H".
Recalling that MC, and rearranging terms, we have,
Focus R O(DI (L) M))
(28)
5.1 In-context learning
obtaining sub-quadratic time complexity with re- spect to input sequence length. A visual compari- son of overall complexity versus the standard atten- tion head is depicted in Fig. 3.
In order to evaluate our method relative to other state-of-the-an long-range architectures, such as (Poli et al., 2023), (Dai et al., 2019), the associative recall synthetic task is evaluated. The associative recall task was first introduced in (Elhage et al.. 2021) and is part of number of simple yet infor- mative tasks that test the capabilities of the model in processing long-range sequences. In the associative recall task, each string is formed by concatenating key-value pairs sampled randomly from dictionary. The model should output the correct value given singular key, re- gardless of whether the key is in the long sequence. Similarly to Poli al. (2023), we employ the as- sociative recall task in order to explore the memory capabilities of our model.
Expressiveness An emerging class of diagonal linear RNNs (Orvicto et al., 2023; Gupta et al., 2022b) recently achieved near SOTA results in sev- eral long-range tasks. They include complex and real variants, as well as diagonal state-space lay- ers (Gupta et al., 2022a; Gu et al., 2022). The following recurrent rule describes each channel of those layers:
s|[] As(t-
Br[(], u((] Cs( D
(29)
where s[t] is the recurrent state time t. By isolat- ing s[t - 1]. we can rewrite Eq. 29 as follows:
In all synthetic data experiments the same shared hyperparameters are used, with the exception of the sequence length. The hyperparameters de- picted in in Appendix A. The AdamW W optimizer (Loshchilov and Hutter, 2018) used.
8/t - 1| v/t - -
D x
(30)
y/t] CAslt -1 (CB + D)x(t] (31)
As can be seen in Tab. 1, our model is able to ob- tain an accuracy of 100% for all sequence lengths, without overfitting, despite the low number of ex- amples (2000), and with no memory explosion thanks to linear scaling with input size. These results show that the Focus mechanism is able improve the performance of regular transformers to be on par with Heyna (Poli et al., 2023), with smaller footprint. In addition, the ablation exper- iment shows the importance of adaptive filtering,
Aylt 1] (CB+ D)+ ADa(t 1]
Recall that the differential equation of an IRR filter of order can be represented as follows:
yft] boort) + bialt 1] + b2 - 2] (32)
any(t 1] a
By substituting the values bo = CB+D,br
AD, 01 = AA by = a2 = 0, it becomes evident

---

Table 1: Test accuracy (%) for associative recall on long sequences of length and a vocabulary size of 30. NF- not feasible to test. NR not reported.
L
Focus
Focus-H
Hyena
Transfrrerre
Transformer
30
100.0
100.0
100.0
100.0
100.0
1K
100.0
98.0
100.0
95.0
100.0
8K
100.0
85.3
100.0
NR
NF
32K
100.0
34.6
100.0
32.4
NF
64K
100.0
28.0
100.0
26.7
NF
5.2 Language Modeling
The enwiki8 dataset is a byte-level dataset consist- ing of the first 100 million bytes of Wikipedia XML dump. It is commonly used dataset for benchmarking character-level language models.
The Text8 dataset corpus of text used for training and evaluating language models. It is a subset the Wikipedia dump from March 2006 and consists of 90 million characters. The text is tokenized and lowercased, and each token is assigned unique id.
The metric used to evaluate language models on enwiki8 and Text8 is bits per character (BPC). The lower the BPC, the better the language model. To compute the BPC the average cross-entropy is computed in the log2 basis
[ime-bin]
Figure 4: Frequency Response of IIR kernels, for 1K sequence split into time bins. The important key found in the 12th time bin.
BPC = L
Logz CrossEntropy P); (33)
where P is the target distribution, and P is the output distribution. I is the sequence length. To maintain the same capabilities such Mega (Ma et al., 2022), we used layers of the Focus layer, with a hidden dimension of 1024 and an input di- mension of 512.
i.e. estimating the filter kernels online to focus the attention mechanism important sub-sequences where the results degrade in the ablation due to the greater sequence length.
The associative recall task is used not only for benchmarking, but also to gain insights into the adaptive filtering mechanism. As can be seen in Fig. 4, the frequency response of the adaptive filter- ing plotted for this task. In this specific run, the important key is found in the 12th time bin. The frequency response of the IIR filters is almost orders of magnitude higher than for nearby time bins. This effect demonstrates the "Focus" mecha- nism. Note that using only 2 parameters for the IIR kernel, the filters able to differentiate between important and unimportant time bins with orders of magnitude. This supports our design choice of IIR filter, sceing that with kernel size as low as the filter is still sharp enough.
The enwiki8 results are reported in Tab. 2. Ev- idently. Focus outperforms both Mega (Ma et al., 2022) and Transformer XL (Dai et al., 2019), de- spite much lower number of parameters, perform- ing on-par with GPT2(Radford et al., 2019) (zero- shot) but with fraction of its parameters and sub- stantially less FLOPS. The same occurs for the Text8 dataset, reported in Tab. 3. While the abla- tion is inferior to Transformer-X XL, with the full method Focus is on par with GPT2.
5.3 1-D image classification
We evaluated our model on the sequential MNIST task, a challenging problem that requires models

---

Table 2: BPC for enwiki8 dataset.
Model
Inference time Memory
Model
#params BPC
Transformer
x
x
S4
x1.58
x0.43
Transformer XL (Dai et al., 2019)
277M
0.99
MEGA
x1.49
x0.57
Mega (Ma et al., 2022)
39M
1.02
Focus
x1.75
x0.38
GPT2 (Radford et al., 2019)
1542M
0.94
Focus-H (ablation)
21M
1.06
Table 5: Comparison of inference speed and peak mem-
Focus
22M
0.94
ory consumption for related models.
Table 3: BPC for text8 dataset.
Conclusions
Model
#params BPC
Attention models are extremely powerful for mod- eling sequences, demonstrated by the seminal work of Vaswani al. (2017). Indeed, transform- ers have revolutionized the way deep learning is practiced, leading to unprecedented performance across almost all studied AI domains.
Transformer XL (Dai et al., 2019)
277M
1.08
GPT2 (Radford et al., 2019)
1542M
0.98
Focus-H (ablation)
21M
1.10
Focus
22M
0.98
Table 4: Accuracy for SMNIST, PMNIST
However, transformers have quadratic complex- ity in the sequence length, which can impact their efficiency, and often struggle to perform optimally in tasks that involve long-range dependencies (Tay et al., 2020). In this work, present dynamic filtering approach that enables employ attention within much shallower architec- tures. As our ablation shows, the dynamic nature of these filters is crucial to the success of the layer. Similarly crucial the use of IIR filters, and we analyze the regime in which these are stable.
Model
sMNIST PMNIST
Transformer
98.9
97.9
$4 (Gu et al., 2021a)
99.6
98.7
subsequently
Focus-H (ablation)
98.9
98.0
Focus
99.7
98.8
capture long-range dependencies. Permuted MNIST is a variant of MNIST where the order pixels in each image scrambled, making the task more challenging. Following S4, we use a hidden dimension of 512 but to save resources, we use 6 layers and not more. The results are listed in Tab. 4. Focus has accuracy of 99.7% (98.8%) unpermuted (permuted) MNIST, outperforming the transformer and S4.
7 Limitations
While this paper presents promising results, there are few limitations to consider. Firstly, although we are the first to utilize IRR filters for long-range tasks, we have not examined sequence models solely based on IIR filters. Additionally, we have not investigated the impact of different types of hyper-global convolution on performance. Further- more, IIR filters can be computed using recurrent rules convolution. While the convolutional view more natural for training, the recurrent view pre- sented in Eq. 32, can be leveraged for efficient auto-regressive generation. This can lead to a sig- nificant reduction the time and space complexity of the layer during inference, which is beneficial for real-time applications.
5.4 Efficiency Comparison
To assess the efficiency of the Focus layer, we measure peak memory usage and inference speed. We compare several related models on the associ- ation recall task. This task involves processing sequence of IK tokens, which represents the maxi- mal fit in memory for transformers. The results presented in Tab. 5. As can be seen, Focus exhibits the lowest peak memory consumption, using only 38% of the memory consumed by the Transformer model. However, its inference speed is slightly slower than the other methods.
8 Acknowledgments
This work was supported by grant from the Tel Aviv University Center for AT and Data Science (TAD). It is part of a PhD research conducted by the first author.

---

References
Stefan Elfwing. Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighee linear units for neural network function approximation in reinforcement learning.
Arij Al Adel. 2022. Global memory transformer for pro- cessing long documents. In Advances in Neural Com- putation, Machine Learning, and Cognitive Research VI: Selected Papers from the XXIV International Con- ference on Neuroinformaties, October 17-21, 2022, Moscow, Russio, pages 343-352. Springer.
Neural Networks, 107:3-11.
N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread.
Arij Al Adel and Mikhail S Burtsev. 2021. Memory trasffrmer with hierarchical attention for long docu- ment processing. In 2021 International Conference Engineering and Telecommunication (En&T), pages 1-7. IEEB.
Quentin Fournier, Gaétan Marceau Caron, and Daniel Aloise. 2021. A practical survey faster and lighter transformers. ACM Computing Surveys
Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. 2023a. Simple hardware-efficieen long convolutions for sequence modeling. arXiv preprint arXiv:3302.06646
Nuha A. S. Alwan and Zahir M. Hussain. 2022. Deep learning for robust adaptive inverse control nonlin- ear dynamic systems: Improved settling time with an autoencoder. Sensors, 126)
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. 2016. Learning feed-forward one-shot learners. In Advances in Neu- ral Information Processing Systems, pages 523-531.
Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Re. 2023b. Simple hardware-efficient long convolutions for sequence modeling. In Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10373-10391. PMLR.
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. 2023. Scaling transformer Im tokens and beyond with rmt. arXiv preprint arXiv:2304, 1062.
Oscar Chang, Lampros Flokas, and Hod Lipson. 2020. Principled weight initialization for hypernetworks. In Int. Conf. Learning Representations.
Xavier Glorot and Yoshua Bengio. 2010. Understand- ing the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth Interna- tional Conference on Artificial Intelligence and Statis- tics, volume of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sar- dinia, Italy. PMLR.
Rewon Child, Scott Gray, Alec Radford, and Πly Sutskever. 2019.
Generating long se- quences with sparse transformers. arXiv preprint arXiv:1904. 105099
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint 2rXiv:2009. 14794.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. 2022. It's raw! audio generation with state-space models. In International Conference Machine Learning, pages 7616-7633. PMIR.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. 2022. On the parameterization and initialization of diagonal state space models. Advances Neural Information Processing Systems, 35:35971-335983
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl Attentive language mod- els beyond a fixed-length context. arXiv preprint arXiv: NKYi:1901.02866
Albert Gu, Karan Goel, and Christopher Ré. 2021a. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00330
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022a. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 1216334-163559
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. 2021b. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-588 585.
Tri Dao, Daniel Fu, Khaled K Saah, Armin W Thomas, Atri Rudra, and Christopher Ré. 2022b. Hungry hungry hippos: Towards language mod- cling with state space models. arXiv preprint arXiv:2212.14052
Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006.03274
Shmuel Bar David, Itamar Zimerman, Eliya Nach- mani, and Lior Wolf. 2023. Decision s4: Efficient sequence-basee rl via state spaces layers. In The Eleventh International Conference on Learning Rep- rexentations.
Ankit Gupta, Albert Gu, and Jonathan Berant. 2022a. Diagonal state spaces effective structured state spaces. Advances in Neural Information Pro- cessing Systems, 35:22982-22994,

---

Ankit Gupta, Harsh Mehta, and Jonathan Berant. 2022b. Simplifying and understanding state space models with diagonal linear mnns. arXiv preprint arXiv:2212.00768.
Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neu- ral networks for long sequences. arXiv preprint arXiv:2303.06349
David Ha, Andrew Dai, and Quoe V Le. 2016. Hyper- networks. arXiv preprint arXiv:1609,0911
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Ste- fano Ermon, and Christopher Ré. 2023. Hyena hierar- chy: Towards larger convolutional language models. arXiv preprint arXiv:2302. 10866.
Thomas Haubner and Walter Kellermann. 2022. Deep soaming-bbsser joint control of acoustic echo cancel- lation, beamforming and postfiltering.
Pytorch. 2023. Adaptivemaxpool22f..
Yun He, Steven Zheng. Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Don- ald Metzler, et al. 2022. Hyperprompt: Prompt-based lask-conditioning of transformers. In International Conference on Machine Learning, pages 8678-8690. PMLR.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9
Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud G. van Sloun, and Yonina C. Eldar. 2022. KalmanNet: Neural network aided kalman filtering for partially known dynamics. JEEE Trans- actions on Signal Processing. 10:1532-1547.
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. recurere transformers. arXiv preprint arXiv:2203.00 078552.
Md Mohaiminul Islam, Mahmudul Hasan, Kis- han Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. 2022. Efficientt movie scene detection using state-space transformers. arXiv preprint
David W Romero, Anna Kuzina, Brik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. 2021. Ckconv: Continuous kernel convolution for sequen- tial data. arXiv preprint arXiv:2 202.0221)
George Saon, Ankit Gupta, and Xiaodong Cui. 2023. Diagonal state space augmented transformers for speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. Ef- ficient long-text understanding with short-text mod- Transactions of the Association for Computa- tional Linguistics, 1::84-299,
Rudolph Emil Kalman. 1960. new approach to linear filtering and prediction problems.
Filip Szatkowski, Karol J Piczak, Przemystaw Spurek. Jacek Tabor, and Tomasz Trzciński. 2022. Hyper- sound: Generating implicit neural representations of audio signals with hypernetworss. arXiv preprint urXiv:2211.0123
Boris Kuznetsov, Julian D Parker, and Fabián Esqueda. 2020. Differenniable iir filters for machine learning applications. In Proc, Inf. Conf. Digital Audio Effects (eDAF=-20), pages 297-303.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. 2020. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.044
Yuhong Li, Tianle Cai, Yi Zhang. Deming Chen, and Debadeepta Dey. 2022. What makes convolutional models great on long sequence modeling? arXiv preprint 1...v:22200092290
Yi Tay, Mostafa Dehghami, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Computing Surveys. 55(6):1-28.
Ilya Loshchilov and Frank Hutter. 2018, Fixing weight decay regularization in adam.
Ali Vardasbi, Telmo Pires, Robin M. Schmidt, and Stephan Peitz. 2023. State spaces aren't enough: Machine translation needs attention. ArXiv, abs/2304.12776.
Shahar Shlomo Lutati and Lior Wolf. 2023. Ocd: Learn- ing to overfit with conditional diffusion models. In ICML.
Xuezhe Ma. Chunting Zhou, Xiang Kong, Junxian He. Liangke Gui, Graham Neubig, Jonathan May. and Luke Zettlemoyer. 2022. Mega: moving av- crage equipped gated attention. arXiv preprint wrXiv2229.10055.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances neural information processing systems, 30.
Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. 2022. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.1 13947.
Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. 2019. Contin-
ual learning with hypernetworks. arXiv preprint arXiv:1906.00665.

---

Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. 2023. Se- lective structured state-spaces for long-form video understanding. arXiv preprint arXiv:2303. 14526.
A Hyperparameters
The hyperparameters for the associative recall task are provided in Tab. 6. Hyperparameters that differ in other experiments are reported in the respective sections. For example, in Sec. 5.2, eight layers of the Focus layer are used, with hidden dimension of 1024 and input dimension of 512, in order compare with Mega on similar terms.
Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander Rush. 2022. Pretraining without at- tention. arXiv preprint arX1::2212.1054.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0 04768.
Table 6: Hyperparameter settings for the synthetic asso- ciative recall task
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, and Pascale Fung. 2020. Lightweight and efficient end-to-end speech recognition ICASSP 2020-2020
Optimizer
AdamW
ing low-rank transformer. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6144-6148. IEEE.
Optimizer momentum
B1, 1, 0.9, 0.98
Vocabulary Size
30
NFFT
L/4
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, and Yashar Mehdad. 2021. Simple local attentions remain competitive for long-context tasks. arXiv preprint arXiv:2112.07210.
32
Learning Rate
1E-4
Batch Size
32
Num Samples
2000
Warmup epochs
10
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte se- quences with multiscale transformers. arXiv preprint arXiv:2305.07185.
Number of Layers Width
2
64
Chris Zhang, Mengye Ren, and Raquel Urtasun. 2019. Graph hypernetworks for neural architecture search. 7th International Conference on Learning Repre- sentations, ICTR 2019.
Hao Zhang and DeLiang Wang. 2021. Deep anc: deep learning approach to active noise control. Neu- ral Networks, 141:1-10.
Michael Zhang, Khaled Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Ré. 2023. Ef- fectively modeling time series with simple discrete state spaces. arXiv preprint arXiv:2303.00 09489.
Qingyi Zhou, Fan Zhang, and Chuanchuan Yang. 2020. AdaNN: Adaptive neural network- based equalizer via online semi-supervised learning. Journal of Light- Technology, 38( 16):4335-43224.
Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Fre- quency enhanced decomposed transformer for long- term series forecasting. In International Conference on Machine Learning, pages 17268-272888 PMLR.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles. Fren Manavoglu, Tuo Zhao, and Jianfeng Gao. 2022. Efficient long sequence modeling via state space augmented transformer. arXiv preprint arXiv:2212.0873:


Reviews:
[{"review_id": "qkulj9LIJm", "reviewer": "Reviewer_K69V", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive review and the constructive feedback.\n\n\n1. (i) The results of all baselines in Tab. 2 are taken from Mega [Xuezhe et al., 2023]. The training procedure for Focus (our layer) is exactly \n    the same as for Mega and Transformer XL [Dai et al., 2019]. GPT2 is trained with extra data, and performs zero-shot, as detailed in     [Radford et al., 2019], see L611-L612. \n\n    (ii) FLOPs metric for Tab. 2 is not published for other methods, and thus we are not able to compare between the baselines along this \n     metric. In Fig. 3 the complexity of the Focus method is shown, and we also provide a worst-case run time analysis in L462-L488. \n\n    (iii) We will add training/inference times with the full 9-page version. See for example this graph https://imgur.com/RPvcAD9, which \n     depicts training progression for the associative recall task by epoch, and this one https://imgur.com/uvSJLt9, which is by \n     time in seconds and further emphasizes the clear advantage of our layer.\n\n\n2. Indeed, the scalability of the transformer is one of its best properties. In terms of scalability in sequence length, the Focus layer is \n    similar to that of transformers, due to its attention layer. As for effectively scaling when the model size (and amount of data) increases, \n    this remains an open question that should be answered by empirical experiments with models containing billions of parameters. \n    Unfortunately, we cannot afford such experiments. However, since the Focus layer contains self-attention layers that are known to \n    scale well, there is no reason to believe it will not scale.\n    On the other end of the spectrum, as we show, the focus layer is much less data-hungry than the transformer.\n\n\n3. The permuted MNIST is considered a hard task, since spatial information is important when dealing with classification from images. \n    This benchmark has become a must-pass test for long-range models. The results show that our method does not fall behind other \n    state-of-the-art long-range methods (and even slightly better as the reviewer noted). In other benchmarks such as the BPC, our method \n    presents remarkable results given the small model size compared to other methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method are new and well motivated. Section 4 give good analysis of stability of IIR.\nIf proposed method solid, the potential impact are huge.", "reasons_to_reject": "(1) Experimental results missing some details. I'd suggest adding flops in table 2 and 3. Also adding inference/training cost. For the reported GPT-2 and other paper's results, does it trained on the same data as focused layers?\n\n(2) Easy to scale is one of best property for transformer. Does the new focused layer had similar properties?\n\n(3) Section 5.3 was unclear. The accuracy is 98.8, does it still make sense to compare on this dataset? How should I interpret it? It seems only 0.1% better than previous method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a new input dependent layer, which use IIR filters. The experimental results shows it's on par with GPT-2 but with much less parameters on Text8 using BPC metric. The paper also claim it the layer is grounded in the theory of control systems, similar to state-space layers.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tmbyLBeDl0", "reviewer": "Reviewer_Wq8Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive review and the constructive feedback.\n1. Our method builds upon the key insights of recent state-of-the-art methods,\n    each of which is a seminal work in its own right but it is also the first method to\n    offer a control-theory theorem showing that the best stable and causal\n    state-space filter for the long-range setting is an IIR filter of order 2. Such filters\n    were not considered before in the construction of attention or other layers.\n    While we do not claim to have invented exponential moving average or\n    transformers, the insights and mathematical framework presented in our paper\n    shed new light on these methods and demonstrate, for the first time, how\n    state-space models can be incorporated in a stable and causal way with the\n    expressive power of transformers.\n2. We would follow your advice, and try to reduce some of the space allocated to\n    equations, e.g., by moving equations to be inline. We would also make use of\n    most of the ninth page for adding experimental details and results, especially\n    those that the reviewers requested.\n3. Attached here are graphs that show the training progression of the focus layer\n    vs. transformers and Mega in the associative recall task with a context length of\n    1K, both by epoch https://imgur.com/RPvcAD9 and by clock time\n    https://imgur.com/uvSJLt9. These graphs show a very clear advantage.\n    We note that (i) the tasks we have used are considered extremely challenging.\n    The associative recall experiments depict a clear case in which transformers fail\n    and the recent long-range methods are able to perform better. (ii) we have\n    much fewer resources than researchers at Meta (Mega), Stanford University,\n    Mila, and Université de Montréal (Hyena [Poli et al., 2023]), but are still able to\n    outperform. (iii) in terms of real-world data we show that our method is able to\n    surpass other state-of-the-art language modeling models that have two orders\n    of magnitude more parameters.\n    With regards to peak memory consumption, following the reviewer’s request,\n    see below a comparison over the association-recall task with 1K tokens (maximal fit in\n    memory for transformers)\n| Model | Inference Speed | Memory |\n|----------------------------|-----------------|---------|\n| Transformer | x1 | x1 |\n| S4 [Gu et al., 2022] | x1.58 | x0.43 |\n| MEGA [Xuezhe et al., 2023] | x1.49 | x0.57 |\n| Focus | x1.75 | x0.38 |\n4. We will make sure to fix all citations to a standard format.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction of the background and related work is quite comprehensive.\n2. The analysis of method stability, time complexity, and expressiveness is detailed and comprehensive.", "reasons_to_reject": "1. The novelty of the method is not easy to determine, as it appears to be a combination of existing approaches.\n2. The excessive number of equations in the paper makes it challenging to comprehend, and it occupies significant space, reducing the presentation of experiments and ablation studies. Consider minimizing the use of equations to improve readability and allocate more space for experimental results and ablation studies.\n3. The experimental section lacks sufficient validation, as it utilizes relative simple tasks and datasets, and a limited number of baseline models, which results in unreliable and insufficiently convincing results. Additionally, the evaluation metrics are too simplistic, focusing solely on performance and parameter count. It is recommended to refer to Mega's experimental section[1], where they employ five different data types for sequence modeling tasks, and incorporate diverse baseline models and evaluation metrics (including training speed and peak memory consumption comparisons, in addition to performance assessment).\n4. The citation format is not standardized (please try to cite formal published versions and avoid citing Arxiv, such as Mega [1], and others, please check accordingly).\n\n[1] MEGA: MOVING AVERAGE EQUIPPED GATED ATTENTION   ICLR2023", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a novel layer that incorporates dynamic Infinite Impulse Response (IIR) filters of order two to process input sequences before applying conventional attention. The filters are input-dependent and causal, preserving causality by determining their coefficients based on previous chunks. Despite their low order, these adaptive filters effectively focus attention on relevant sequence elements. The layer performs comparably to state-of-the-art networks while using significantly fewer parameters and boasting a time complexity that is sub-quadratic with input size. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O3SFVMAejb", "reviewer": "Reviewer_EHXv", "summary": "", "questions": "(A) What are the impact of the new parameters brought by your approach e.g. Nbins, F, H,...?\n\n(B) Why are you applying signal processing technique only to the key and the value, and not to the query? How does it impact the value  output of the attention layer?\n\n(C) How does your approach compare in terms computational complexity to MegaBytes, and sparse transfromer used in GPT-3?\n\nYu, Lili, et al. \"Megabyte: Predicting million-byte sequences with multiscale transformers.\" arXiv preprint arXiv:2305.07185 (2023).\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive review and the constructive feedback.\nAddressing the issues raised as weaknesses one by one:\n1. Advanced knowledge: In section 3 (“Methods”; L334) we refer to section 4 (“Analysis”) for\nfurther explanations, intuitions, and proofs, since, as the reviewer states, there is an advanced\nusage of knowledge from signal processing and control theory. We want the paper to be\napproachable to all members of the AI community, and thus we wrote section 4 in a way\nthat delves into causality, stability, and IIR general formulation. In the final version of the\npaper (where an extra page is available), we will add text about intuition, to make the paper\neven more readable.\n2. Statistical analysis: While previous work did not provide data beyond mean performance\nnumbers, we will add the standard deviation to the experiments. Below is a sample table of\nPMNIST benchmark with five different seeds. As written in lines L623-L625, we were able\nto be on par with a Transformer with half the number of layers and half the number of\nparameters.\n| Model | sMNIST | pMNIST |\n|-----------------------|------------|------------|\n| Transformer | 98.90 | 97.90 |\n| S4 (Gu et al., 2021a) | 99.60 | 98.70 |\n| Focus-H (ablation) | 98.90±0.05 | 98.00±0.11 |\n| Focus | 99.70±0.08 | 98.80±0.12 |\n3. Hyperparameters: The number of hyperparameters is similar to Mega, for example. The\nmeaning of each hyperparameter is rather intuitive. When $Nbins\\rightarrow1$ our method\ndegenerates into the Mega method, when $Nbins\\rightarrow T$ (where T is the sequence length) the\nmethod degenerates to a Transformer (since no temporal filtering is applied). The trade-off\nbetween being adaptive to fast changes in the sequence and maintaining context along very\nlong sequences is controlled by this parameter.\nThe “F” parameter controls the dimensionality of the network, with sufficient expansion that\nleads to generalization.\nThe hypernetwork H adds adaptivity to the network, which, as we show, is important in long\nsequences since single-static filters as in Mega fail to produce meaningful features with\ndifferent time scale events. This is shown in the ablation “-H” in tables 2-4.\n4. Newer GPT baselines: The results of all baselines in Tab. 2 are taken from Mega [Xuezhe\net al., 2023]. The training procedure for Focus (our layer) is exactly the same as for Mega\nand Transformer XL [Dai et al., 2019]. GPT2 is trained with _extra data_, and performs\nzero-shot, as detailed in [Radford et al., 2019], see L611-L612. It is given to put the number\nis context, not as a direct comparison.\nWhile there are a lot of challenges associated with scaling up models to GPT3, GPT3.5\nsizes, this is not our focus. In our work, we show, for the first time, how signal processing\ntools and theorems, combined with recent advances in long-range modeling (tackling long\nsequences with state-space models) can yield slim and effective models that surpass more\nthan 2 orders of magnitude larger models. Thus the comparison to bigger, higher-capacity\nnetworks is out of the scope of this work.\nNeedless to say, we compare to the state-of-the-art layers in the field of long-range\ndependencies such as Mega and Hyena.\n5. More variants: In order to maintain the time resolution of the input sequence, at least one of\nthe variables of the transformer (key or value) needs to be unchanged, otherwise, the result\nwill be distorted. In section 4, we show that the most general filter to represent state-space\nmodeling, with simple guarantees for causality and stability, is IIR of order 2. Other\nformulations, if not equivalent, simply do not converge (exploding gradients) or lead to\nnon-causal reactions. In the full 9-page version we will add a few examples of those failed\nalternatives and show why it is important to have this specific formulation and order of\nfilters.\n\nTo the three questions asked by the reviewer:\n\nA. The signal is chunked to Nbins and this is the minimal window that the focus adaptive\n     filtering addresses. For $Nbins\\rightarrow1$ the method degenerates into Mega formulation. For\n     $Nbins\\rightarrow T$  the filter kernel size is one, and the method degenerates to a vanilla transformer. By\n     using the division of Nbins one controls the degree of adaptivity to non-stationary signals of\n     the algorithm. F is the number of channels, similar to the number of channels in CNN.\n\nB. The reasoning behind this choice (see also point 5 above) is to conserve at least one input\n     of the transformer to be with the same time resolution as the original signal, without\n     changing its spectrum. By using a multiresolution approach, our method effectively utilize the full bandwidth of the original data.\n\nC. In terms of complexity, our method is more efficient than these methods. MegaBytes is the\n     most efficient out of those mentioned by the reviewer and still has twice the complexity than\n     focus (MegaBytes was published on arXiv right before the EMNLP deadline).\n     The underlying cause for our efficiency is that for each example, we compute its\n     corresponding FFT representation only once. We note that this computational burden would\n     become negligible for deeper architectures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strength of the paper are:\n\n* The idea is novel and bridge the gap between system/signal processing to transformer. While text-based task will benefit from such layers. I believe this is likely to be more influencial for audio-based (ASR, TTS,...) task.\n* Applies approach in two domains: (text, images_ accross 4 tasks.\n* Computation complexity of the approach is given.\n* A stability proof is provided, but unclear overall how it impact the algorithm in practice.\n\n", "reasons_to_reject": "The weakness of the paper are:\n\n* The method section is expecting advanced knowledge on various system and signal processing. The link between the signal processing and the impact within the attention network is not necessarily obvious. There is a lack of interpretations of why some design choices have been made.\n* Lack of statistical analysis of the results. There is not even a standard deviation of the metrics.\n* Unclear impact of the parameter brought by this new approach.\n* Quite a few of the baseline are old: most are from 2019, one from 2021 and one from 2022. For instance, it uses GPT-2 as a baseline, where GPT3, GPT3.5 or equivalent would be more suitable.\n* We are missing ablations studes on the components added by the IIR filter, e.g. applied on the key or only the value or added parameters, other type of filters,...\n\n", "questions_to_authors": "(A) What are the impact of the new parameters brought by your approach e.g. Nbins, F, H,...?\n\n(B) Why are you applying signal processing technique only to the key and the value, and not to the query? How does it impact the value  output of the attention layer?\n\n(C) How does your approach compare in terms computational complexity to MegaBytes, and sparse transfromer used in GPT-3?\n\nYu, Lili, et al. \"Megabyte: Predicting million-byte sequences with multiscale transformers.\" arXiv preprint arXiv:2305.07185 (2023).\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019.\n", "paper_topic_and_main_contributions": "The paper presents a new attention layer in which dynamic Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. It lead to a strong reduction of parameter and sub-quatric computational complexity through chunking of the sequence. The aims of the author to combine system and signal processing insights to attention layers. It provide stability proof, low computational complexity and experimental evaluation on several datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I would like to have more interpretation of the presented method, and if possible ablation study to highlight the benefit of the approach."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a preprocessing layer with IIR filters to efficiently handle the inputs. Reviewers appreciate the novelty and the structure of this work, with comprehensive backgrounds and results. Therefore, the AC recommends acceptance of this paper to EMNLP.
Decision: Accept-Main