DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations
Authors: Zuowen Wang
Longbiao Cheng
Pehuen Moure
Niklas Hahn
Shih-Chii Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: implicit layers
deep learning acceleration
deep equilibrium model
Primary area: deep_learning_architectures

Abstract: Implicit neural networks including deep equilibrium models have achieved superior task performance with better parameter efficiency in various applications. However, it is often at the expense of higher computation costs during inference. In this work, we identify a phenomenon named $\textbf{heterogeneous convergence}$ that exists in deep equilibrium models and other iterative methods. We observe much faster convergence of state activations in certain dimensions therefore indicating the dimensionality of the underlying dynamics of the forward pass is much lower than the defined dimension of the states. We thereby propose to exploit heterogeneous convergence by storing past linear operation results (e.g., fully connected and convolutional layers) and only propagating the state activation when its change exceeds a threshold. Thus, for the already converged dimensions, the computations can be skipped. We verified our findings and reached 84\% FLOPs reduction on the implicit neural representation task, 73\% on the Sintel and 76\% on the KITTI datasets for the optical flow estimation task while keeping comparable task accuracy with the models that perform the full update.

DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations
Zuowen Wang Longbiao Cheng Institute of Neuroinformatics, University of Zurich and ETH Zurich {zuowen, shih]@ini. uzh. ch
Pehuen Moure Niklas Hahn Shih-Chii Liu
Abstract
Implicit neural networks including deep equilibrium models have achieved superior task performance with better parameter efficiency in various applications. However, it is often at the expense of higher computation costs during inference. this work, we identify phenomenon named heterogeneous convergence that exists in deep equilibrium models and other iterative methods. We observe much faster convergence of state activations in certain dimensions therefore indicating the dimensionality of the underlying dynamics of the forward pass is much lower than the defined dimension of the states. We thereby propose exploit heterogeneous convergence by storing past linear operation results (e.g., fully connected and convolutional layers) and only propagating the state activation when its change exceeds threshold. Thus, for the already converged dimensions, the computations can be skipped. We verified our findings and reached 84% FLOPs reduction on the implicit neural representation task, 73% on the Sintel and 76% on the KITTI datasets for the optical flow estimation task while keeping comparable task accuracy with the models that perform the full update.
Introduction
Implicit neural networks [2] 13 have gained much attention in recent years. They are capable of matching or surpassing waateof-ttheaar performance in domains such as computer vision [ 17 32
41 language modeling 2 and audio processing 32 335. Implicit networks typically model these tasks with certain dynamics represented by the evolution of the intermediate hidden states and are often described as fixed-point equations or differential equations. The implicit models, represented by the deep equilibrium (DEQ) model family, often achieve better parameter efficiency for reaching similar level of task accuracy as their regular neural network counterparts. Furthermore, the implicit function theorem assists implicit models in avoiding the need for direct differentiation through numerous forward steps, like those required in propppropagttion through time, which demands substantial memory for training.
Despite their compact architectural design with fewer parameters, implicit models frequently require extensive computations during inference to better approximate the fixed point. The fixed point is typically computed with fixed-point iteration methods 4 or other root-solving techniques such as Broyden 77] Anderson [54] methods, all of which require multiple forward passes of the model. The substantial computational cost at inference time significantly hinders the deployment of implicit models. Various strategies have been developed to reduce this computational burden, including fixed-point reuse 4 41 to accelerate the convergence of the forward pass for temporally correlated inputs like consecutive video frames, and global early stopping criteria that halt the forward pass once the absolute or relative change of the hidden states norm is below threshold. However, these approaches uniformly control the termination of updates across all dimensions the hidden states and they fail consider the internal structure of the dynamics.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This work makes finer-gran grained inspection of the dynamics of the hidden states update. By amalyzing the element-win trajectory and dimensionality of hidden states update, we observed the heteroge- neous convergence phenomenon among implicit models. That is, there exists a significant variance in convergence speeds across different dimensions of the hidden states; and that certain dimensions converge substantially faster than others in the forward pass of an implicit model. Based on this finding, we modified the forward pass update of the DEQ models with the delta updating rule that stores the intermediate results of computationally intensive linear operators and only calculates at dimensions where their changes between two sequential updates exceed a threshold. Our method is orthogonal to other acceleration techniques including fixed-point reuse 4 32] and early stop-
ping 44 41 of iterations. Unlike fixed seedtpoint reuse acceleration, our method does not assume temporal correlation of inputs i.e. effective even for static. While primarily applied to DEQ-based methods, our technique can be adapted to any iterative method that uses a fixed-point search in the forward pass. We empirically tested our method for two tasks: implicit neural representation (INR) for images and optical flow (OF) estimation. Compared to previous DEQ-based methods, Our approach achieved an 84% reduction in FLOPs for the INR task and 73% and 76% reduction for the OF task using the Sintel and KITTI datasets, respectively, without significant loss in task accuracy. The code is Deaat--Deee----iiiiiee MModees.rum--MMelss.
available at http::/ggtttbboom///ooavaaaaggno
2 Background
The majority of the findings in this study are derived from deep equilibrium models (DEQ which represent a specific category of implicit models. We offer essential annotations and background
information on DEQ in this section. A DEQ layer is modeled as fixed-poine equation:
.z),
(1)
where z is the input and the layer is parameterized by 6. Notice that fo does not need to be single- layered but could also be a block of layers. Thus, =" could be activations of a block of layers. This
fixed-point equation is solved with Broyden method [7] or Anderson acceleration (54) by finding the root of fa(=", - z² = 0 in the initial DEQ works
Alternatively, the fixed-point equation can be solved by iterative methods. Picard's method is an iterative method for approximating the fixed point 2". By simply substituting the state 2f at i-th iteration the input and 22 as initialization, the state at iteration step i 1 is computed as
2 2++1 fa .),
(2)
and 2d + 2n as + oo when fe is contraction mapping. Empirically the 2n can be approximated in limited numbers of iterations, in other words, the fixed-point iteration (FPI) converges empirically in limited steps.
For a temporal input sequence dt where t = I....,T, T, processing each input s, at time t requires solving a corresponding fixed-point 2 fo(2;, sx). This means that for every input timestep t, the forward pass of DEQ involves solving a fixed-point equation which could be computationally intensive. From previous studies 4 41 56), reusing the fixed point z as the initialization (2f ={=1) for processing the next timestep input wt, could accelerate the convergence speed to z
3 Observations and motivation
Our methods are based on several key observations on the forward pass of DEQ models. We first demonstrate the observations with simple DEQ model which formulated as a deep equilibrium layer x² tanh W- =n" + W x + b) and a linear readout layer = Wz*, where W; e R20020 W2 E R 20 W 1 and b E R20011 are the weights and bias respectively. We generated a dataset that consists of data points (I. (r.)))(z))) with x being evenly sampled 200 points in [-2x,, 2xr.. We train the network to fit the sin(2)) function learning from these data samples. The forward pass is implemented with Picard's 's method in Eq. 2 with 15 iterations and the backward pass implemented with prokppropagttion through time (BPTT). The loss function used for training is
'In the original works 2 26 DEQ refers to the models trained with implicit function theorem (IFT) or approximation based on IFT. In this work we relax the definition of DEQ to any network that models a fixed-poin equation and the forward pass is solved by fixed- fixed-point iteration or other root-finded

---

Cumuuast undulsveeeeee Explained Vartance
Principal Component 2
55% Enp
Hoem ttmatton
0.38
sin.e)
02-
0.0
0.84
0.0
5150
-14
-044
(8)
-25
Principal Component
50
18
15
Number Components
1.02
Converges Dim Dimeensons 1ith Iter.
Hidden State value
0.33
0.21
Wean Deta Activation
23
0:02
20
0.39
-0.75
-142
(d)
Iteration
(e)
Iteration
(f)
14th Iteration
Figure 1: (a) Reconstruction evolution when increasing inference iterations. (b) Hidden states
trajectory for 5 consecutive input points with the first two principal components. Details in A.I (c)
Cumulative explained variance for all hidden states. (d) Evolution of different dimensions of hidden states (represented by colors) (represented by colors). The colored solid areas indicate the standard deviation from different inputs. (f) Histogram of converged dimensions (blue) i-th iteration evolution of the model prediction MSE (red).
iterations. (e) Mean delta activation for different dimensions
l = MSE - 111112 where MsE is the mean squared error. The second term f11-112 = A [1I is the spectral norm regularization on the matrix W₂ and is applied when ||W=||2 > 1 in each iteration. This regularization points as stated in the Banach Fixed Point Theorem [6]. As shown in Fig. a the model gradually converges the ground truth sin(x) values when the iterations at inference increased from I 15.
ensure the model realizes contrastive mapping and guarantees stable fixed
In Fig. [c), we concatenated hidden states from different iterations for all the input x the dataset and conducted the principal component analysis (PCA) them. We can see that although the hidden states are defined with 20 dimensions, with only 3 principal components we could explain over 99% of the variance in all hidden states trajectory. This observation shows that for DEQ models trained on specific tasks, the underlying dynamics could depicted with much lower dimensionalities than the defined state =², which was observed similarly in (50) for continuous RNNs. Details of the PCA method are in Appendix. A.1
Heterogeneous convergence The underlying dynamics of DEQ models, as well other forms of recurring architectures [37]. often have lower intrinsic dimensions than their explicitly defined dimensionality. Fig. (d) shows how this low intrinsic dimension characteristic is reflected in the original hidden state space. We can see that some dimensions of 2 experience much larger fluctuation with the hidden state trajectory while certain dimensions converge within few iterations. Fig. (e) presents the evolution of the mean delta activation |-^-| 20|k at dimension l for 1000 uniformly sampled input points. We can observe very different convergence speed for different dimensions and large variances in all dimensions. The averaged histogram statistics for the number of dimensions that converged at th iteration is shown in Fig. f We observe that although globally the model reaches an MSE plateau at around 8 iterations, many dimensions have already converged long before that. We name this phenomenon heterogeneous convergence, indicating that different dimensions of the state converge at uneven speeds for a fixed-point iteration implemented forward pass of a DEQ. natural question arises from observations of this phenomenon:
Can we exploit the heterogeneous convergence phenomenon in the dynamics of hidden state updates for accelerating deep equilibrium models and other iterative methods?
Indeed, identifying and leveraging the rates at which dimensions stabilize could lead to more efficient computational strategies. In Sec. 4 we describe our DeltaDEQ method that exploits the heterogeneous convergence for reducing computation of the DEQ forward pass together with details for other associated methods used in this work.
3

---

Methods
We propose DeltaDEQ to leverage the heterogeneous convergence phenomenon by inducing the delta activation sparsity in ixeetpoint iterations. DeltaDEQ can be instantiated with mainstream network architectures including recurrent neural networks (RNNs), convolutional neural networks (CNNs) and transformers just like DEQ. Computation savings for DeltaDEQ do not require the strong assumption 44 45]
that input data has to be minimally changing temporal sequences. Even when processing static input, DeltaDEQ can still save computation in comparison to DEQ due to the convergence nature of equilibrium evolution.
Iteri : output of previous layer I
Iter H+1 output of previous layer 1t*
sparse convolution
convolution kernel K
(a)
atf 1 4
Art1
11t2 11
Al gets sparser during fixed-point iterations
C
DD
K+AI
K *
K
sparse activation
conv.
cached:
K + /
cached: K 1
kernel
II K 1
update cache
II K 11
update cache
partial
spa rse
fully
sparse
sparse
cached
sparse
cached
of alk, * Alf + K₂ (!-
of+1 = o(
A1
+ K2
Iter i: input for the next layer Of
Iter 1+1: input for the next layer 0
(b)
Figure 2: (a) Convolution type of DeltaDEQ. The imput 1 from the previous iteration stored and subtracted to create the sparse AT; White represents zero. (b) For sparse convolution, in theory, all zero entries in the feature map can be skipped; practice, this is more feasible on hardware [ [5] when the entire activation patch is fully sparse. The complete formulation and pseudo-code are given A.2 and RNN type DeltaDEQ is illustrated in A.5.
4.1
Delta deep equilibrium layer (DeltaDEQ)
We start the introduction of the DeltaDEQ with simple instantiation of single RNN layer:
2, ++ = oW,, -W 2 + W. x2),
(3)
where va E Rdd is input at timestep I, of E Idd is the hidden states vector at fixed-point iteration We e gdxd and Wg E gdxdd are the weight matrices and a represents the activation function. Due to the linearity of W- projection, Eq. 3 can be reformulated as follows:
2t .+1 ( (zf - W. 2 2 Ws W.y).
(4)
Assuming that =f 2 is sparse, namely many dimensions zeros, then the computation of the matrix-vector multiplication can be greatly reduced (see Fig-5). 5). This sparsity assumption holds due the fact that DEQ models trained converge in the forward pass. In order achieve higher sparsity, we further apply an element-wiss delta threshold r to zero out small changes. We define the thresholded delta hidden states vector and apply the delta rule on Eq. as follows:
if 22 2f otherwise
A
(Delta Rule)
(5)
0
#+
(W2
Az
W
W Lz ) cached in C.
(6)
sparse
cached in C.
C2
- W
A z + C2
(update cache)
(7)

---

Notice that Eq. 5 opperates element-wise. The assignment Eq. states that after finishing the sparse matrix-vector multiplication, we update the stored value C² for the next iteration. There no need to update the cached value Cz within the same input timestep / since the input 2y does not change. But when the computation moves to 1+++1, a similar update rule as in Eq. 5 can be applied on input and thus also saving computation. An illustration for the RNN type DeltaDEQ is in Sec. A.5.1 Fig. 5
Convolutional DeltaDEQ
The convolution layers, which are most commonly used for vision tasks, are in principle also composed of linear transformation followed by non-linearities. Thus, we can also reformulate a convolutional DEQ block as a convolutional DeltaDEQ Assuming the input features to layer in a DeltaDEQ block at iteration i is Iī and the output is O
block.
O o(K I;) (K ( I + 1 1)
(AC AT; + 1C
(8)
sparse
cached
where denotes the convolution operation and K. is the convolution kernel. The convolution KC Alj is conducted on the sparse delta feature map AT;. The multiply-accumulaa (MAC) operations could be skipped on the zero entries in AT; illustrated in Fig. 2
Theoretical complexity analysis and hardware practicality Assuming the sparsity level (% zero entries) in zf is Osp. For every fixed-point iteration for the RNN type of network layer as in Eq, 5 the floating point operations (FLOPs) spent on computing W- z is 2d2, while the delta thresholded version costs approx. 2(1 Osp 0sp)a² + d. For convolution kernel with kernel size and stride 8, and assuming 22j, z Rdlxd and padding so that the output feature map still has size x d. the original convolution costs approx. 2d FLOPs while the delta version costs approx. 2(1 - axp d + d2 FL.OPs. large level of Oup could reduce FI.OPs greatly for both the RNN type of layer as well as a convolution in theory. However, while the computation cost reduction for the RNN type of layer is easy to realize in practice (20] 43), the sparse convolution is in general more difficult due to its fine granularity and its realization often requires special hardware library support [1 15).
4.2 Fixed-point iteration instead of root solving
The original DEQ work adopted soot-solving, including Broyden 77 and Anderson 54] methods, in the forward pass. These methods contain overhead computation other than the model inference fo(=), .r) itself. We found that fixed-point iterations, which include Picard's iteration (Fq. 2 and Krasnoselskii-Man (KM) iteration, suffice for approximating the fixed-point in the forward pass and has good (global) convergence speed (fewer iterations needed to converge) in comparison to root-solving techniques. KM iteration maintains a history of one past iteration and conducts weighted
sum: 2, = fe 2.) + (1 - at)²/, where ag is the coefficient for the weighted sum and in work choose fixed namely a, a. The KM method helps to accelerate the overall convergence of the fixed-point iteration with better asymptotic and stabilize the trajectory of the forward pass and improve task performance. It does not induce an additional memory footprint with DeltaDEQ since the storage of two steps of states is already required. We provide a detailed comparison of fixed-point iterations and root-solving techniques in A.7
4.3 Other acceleration techniques for DeltaDEQ
We adopt another two major acceleration techniques from previous works 4
56 for
DeltaDEQ: (1) fixed-point reuse and (2) global early stopping. The fixed-point reuse method is often used for processing temporal sequences such as video frames. It initializes the hidden states = with the fixed-point from last input timestep i, namely 2 = 2 This technique is based on the assumption that small distances between consecutive inputs d(x,, 11+1) will induce small distances of fixed-point d(z;, z < d(=f+1: z[+1) under their corresponding metric spaces. In the work [32] the authors recycle the fixed point from the last training epoch as initialization for accelerating the training. Distinct from heterogeneous convergence, global early stopping is used to terminate the exed-point iteration when it has converged globally. After computing the new hidden state 2/,, calculate its absolute || 2t 2/ or relative ||æ 2/- l| 1 I|2 distance to the previous hidden state z/-1, If the distance is smaller than the tolerance, the forward pass is terminated. We emphasize that global early stopping requires storing the previous hidden state for calculating the difference =t 2/- f-1 which also required for the delta rule. This additional computation and memory usage can be shared for both techniques.
5

---

4.4 Training and fine-tuning DeltaDEQ
One of the biggest advantages of deep equilibrium models is the constant memory and computation costs for training in comparison to explicit iterative methods with (trunatedd-BBPT) The loss value & evaluated fixed-point in Eq. with respect the function parameters is given by the implicit
function theorem (IFT) 22 36
(gg. rT((")) (e(ug r2(2")) 00
02-
(E(ygr nz(2")) 0z
(I
a Ofa(""
Ofa(=", xz)
(9)
0zn
00
Oz*
00
where ra is the final read-off' layer with parameters /3 and yet is the ground truth label. The inversion in Eq. could be expensive when the dimensionality of 2" large. However, various methods have been proposed to avoid computing the inversion [99 26 28 without hurting task performance much.
The Eq. showss that the backward pass is independent of the forward pass regardless. Thus, applying the delta rule (Eq. 5) in the forward pass will not impose any additional cost in the backward pass.
5 Experiments
In this section, we present the experimental results of the INR task and the OF task with the DeltaDEQ method. We showcase the two instantiations: RNN and CNNNbbaee DeltaDEQs and present the computation reduction we could achieve without hurting the task accuracy.
5.1 Implicit neural representation
In this section, we present empirical results of DeltaDEQ for implicit neural representation (INR) 32
48 [1]. INR is type of method that represents the data with coordination-based neural network. For example, for encoding a 2D image, the network is trained with input coordinate (2r, y) and to predict the corresponding (R. or greyscale value of the pixel (x: y). The pixels in an image are the training set for fitting the network to represent such
data. Despite the simplicity of the task setting, training INR models to reconstruct the original data with good quality is not easy.
Two of the representative works
are Siren [48] and multiplicative filter networks with Fourier fil-
Ground
Recon. Fourler
Recon. (ourer
ters [5T]] later work [32] modi-
fied these two methods with a deep equilibrium paradigm and achieved better task performance with the same amount of parameters. The experimental setup of DeltaDEQ for INR is largely based on [32]. Methods and hardware details and Figure 3: Original image and reconstructions with INR network. additional results are in A.5
PS P5M 32, 1LOPs
PSMB-31 30. (iratt
DeltaDEQ architectures for INR We apply the delta rule on the Implicit Sine-activate Networks and Implicit Multiplicative Filter Networks with Fourier filters, which we name DEQ-Siren and DEQ-Fourier respectively and their delta version DeltaDEQ-Siren and DelraDEQ-Fourier The DEQ-Siren (Eq. 10p and DEQ-Fourier (Eq. TT] are formulated as follows:
Sin (W + W'innVVr + Ux
(10)
x* = (W=* + Wy(a;V) + b) g(x; U)
(11)
where g(æ; U)= == Sin(U'x) represents the Fourier filters [51). The dimensionalities of the hidden state z used in the experiments is 512. The most computationally heavy part is the dense metrix-vector multiplication Wz* We apply the delta rule described in Eq. to convert the forward pass to reduce FLOPs (detailed formulations specific for each architecture are given in A.3.3)
Inference acceleration with DeltaDEQ We first train vanilla DEQ-Siren and DEQ-Fourier models without incorporating the delta rule in the training stage. The hidden state size is 512 and spectral normalization 42 is used for better stability and to ensure the non-diverging behavior of the network. The input image size is 512 x 512. As [2, we use phantom gradient (6 accelerate DEQ
training. We use Adam (44] optimizer and cosine annealing learning rate schedule (38) with an initial

---

learning rate of 0.001. The fixed point from the end of one training epoch is used for the initialization of the next epoch, and only one forward iteration used the fixed point is reused. This training technique [32] greatly accelerates the training speed and saves lot of FLOPs. For global early stopping, we use absolute distance with a tolerance of 0.001 and a maximum of 40 forward iterations.
the relationship of com- putation saving (FLOPs Reduction (%) t), re- construction quality (peak signal-to-noise ratio, PSNR 1) w.r.t. the choice of inference delta threshold as in Eq. [5] Mean values of three 60
FLOPS Reduction (%)
80
PS PSNR
runs are given and the standard deviations are
25
shown as colored areas around the mean, We evenly sample 1000 inference delta threshold values between O and 0.1, and 100 threshold values between 0.1 to 0.5. Both DeltaDEQ instantiations show great computation savings (55% to 80% approx.) with a robust range of delta threshold levels (10-3 change in the task performance aceuracy. Fig- ure shows qualitative comparison of the reconstructed image with DeltaDEQ- with threshold 0.5. We can DeltaDEQ-Fouric achieved x 84% FLOPs reduction with a minor drop in PSNR.
40
DetaDLU-1 ourter
DetaDDQ Siran
10
10
Inference Delta Threshold
3 to 0.5) with little
Figure 4:
FLOPs reduction and task accuracy
(PSNR) at different inference delta threshold.
and our
Learning with delta for the forward pass of DEQ The forward pass of training DeltaDEQ is decoupled from the gradient calculation as described in Sec. 4.4 Thus, the saving of the forward pass could directly contribute to the training time reduction of DEQ-based INR methods. We study in this section the model behaviors when incorporating the delta rule during training. Tab. I shows the benefits of training with DeltaDEQ. For both Fourier and Siren variants of DEQ instantiations, applying the delta rule with training delta threshold 5 = 1 - reduces the training FLOPs in the forward pass for both phantom gradient and implicit function theorem used for the backward pass. Notice that the training FLOPs reduction level is obtained at the circumstances of fixed-point reuse (Sec. 4.33 for each training epoch and only fixed-point iteration is executed except for the first epoch. This approach [32 has already significantly reduced the FLOPPs required during the training forward pass an asymptotically constant level [3). Our method is orthogonal this technique and can be implemented in conjunction with it. Furthermore, a single fixed-point iteration often does not suffice to accurately approximate =* for other complicated tasks, even when the fixed-point is reused. By applying our method, a greater reduction in training computations can be achieved.
Training Method
Phantom Gradient (26)
Implicit Function Theorem Tr. FLOPs*
Tr. FLOPs*
PSNR
PSNR1
DEQ-Fourier
32
140
31.31
278
33.22
FourreeQ-Fourie
110
(-21%)
31.06
212 (-24%)
32.98
DEQ-Siren 32
140
29.94
278
34.96
DeltaDEQ-Siren 118 (-16%)
29.64
190 (-32%)
34.99
Comparison of PSNR and training FLOPs w/o vs. w the delta rule during training. For reference, with same hyperparameters, the original Fourier and Siren (non-DEQ) networks recorded PSNRs of 30,06 and 3,.31, respectively. All FLOPs values presented in Tera-FLOPs (1e12). *This table only includes FLOPs for the forward pass of training; the computation cost of backward pass is independent of the forward pass.
Table 1:
5.2 Optical flow estimation
In this section, we study the application of DeltaDEQ on the classical computer vision task of optical flow (OF) estimation [8 31
39
53 58) OF estimation is task for predicting the pixel
displacement between two frames. Modern OF estimation is dominated by learning-based approaches
and one of the representatives is the RAFT 53 network, which iteratively refines the OF estimation
with recurrent structure. In the work
4 the authors formulated the recurrent iterative structure of
7

---

RAFT as a DEQ block which consumes 4 to 6 times less training memory than the original RAFT with BPTT and achieved SOTA on the MPI Sintel
8 and KITTI 2015 [24 optical flow datasets.
DEQ-RAFT architecture We use the RAFT x network as the architectural backhone and the
DEQ flow estimator version 4 (DEQ-RAFT) as starting point for the conversion De RAFT consists of mainly two parts: (1) the correlation and context encoders (feature extraction stage) and (2) the update block (iterative flow refinement stage). Although the feature extraction stage also compatible with the delta rule and can be used to exploit the temporal correlation between the accelerate the iterative
input frames, it is not the focus of this study since our main contribution computation in DEQ and other models. Thus, we omit the FLOPs of the first stage in results.
Assuming the feature extraction stage gives context embedding and correlation tensor C, the flow refinement stage conducts the fixed-point iteration simultaneously in two parts 53
h H(h", o', C), o F(h++1 .. C):
(12)
where h is the hidden representation and o is the optical flow estimation. Putting these notations into the fixed-point iteration framework in [ the extracted features q, C' are the input (q. C), the iterative functions H.F are the function Jø and the hidden states and the optical flow make the fixed-point iteration 2² = (h', )²). Notice marks the fixed-point iteration step and all the terms input time dependent when we move to process the next frame pair. Like in RAFT the optical flow refinement module is implemented with ConvGRU [5] for hidden states updates and other convolutional layers for the optical flow updates:
c² -Conv(e of Corr(o' + C)]), h ConvGRU(h [c', ()),)), 0²-1 = Chnv (13)
where Corr is the correlation lookup. Design details of DEQ-RAFT are in Fig. 10 and A.6.2
DeltaDEQ conversion of DEQ-RAFT We convert the layers in the update block (optical flow refinement stage, Fig. 10p of the DEQ-RAFT (DEQ in Tab. 2 for simplicity). We use pretrained models provided in [4] and show even without any fine-tuning,
achieve good computation reduction without hurting the task accuracy. We also observed the heterogencous convergence phenomenon in the DEQ-RAFT network (Fig. 9 and during the convergence of fixed-point iteration, the delta activation sparsity also increases (Fig. 8). indicating the DEQ-RAFT network is suitable for
applying the delta rule. DeltaDEQ conversion, pertaining and fine-tuning settings given in A.6.2
inf.
Sintel (train) J
A
FLOPs
KITTI (train) AEPE
A
FLOPs per pair
Method
Clean
Final
Sp.
per pair
F1-all
Sp.
PWC-Net
2.55
3,93
10.39
28.5
VCN
57
2.21
3.68
8.36
25.1
GMA
33
1.30
2.74
4.69
17.10
RAFT
53
1.43
2.71
5.04
17.40
DEQ(picard)
1.26
2.51
782G
3.73
13.42
814G
DeltaDEQ
0.0
1.22
2.55
0.52*
394G
3.77
13.47
0.54
396G
60 iters
DeltaDEQ
0.001
1.24
2.52
0.69
264G
3.79
13.49
0.69
274G
DeltaDEQ
0.005
1.48
2.76
0.80
177G
4.00
14.04
0.80
183G
DeltaDEQ
0.01
2.08
3.32
0.83
147G
4.33
15.85
0.84
150G
(FQ(picard)
1.27
2.51
520G
3.73
13.45
621G
early stop
DeltaDEQ
0.0
1.22
2.55
0.53*
225G
3.78
13.54
0.54*
260G
DeltaDEQ
0.001
1.23
2.51
0.61
186G
3.80
13.55
0.63
213G
DeltaDEQ
0.005
1.31
2.63
0.74
141G
3.90
13.75
0.74
152G
DeltaDEQ
0.01
2.04
3.31
0.83
147G
4.19
14.73
0.82
140G
DeltaDEQ-ft 0.005
1.25
2.59
0.73
127G
3.86
13.64
0.74
144G
Table 2: Even for 7 0.0 the sparsity level already 0.53. This high level of sparsity is favor of our method since it is not exploited in the vanilla DEQ. For DEQ and DeltaDEQ models, the default forward method is KM with a = 0.9. Notice with early stopping the total FLOPs could lower even when the ASp is lower since different number of global fixed -oint iterations could be executed.
In Tab. we show the results of DeltaDEQ for the OF task in comparison with the original DEQ method and other methods. The models were all pretrained on the FlyingChair [6 and Fly- Thinggn [40] datasets and tested the training splits of Sintel (8) and KITTI 24 datasets,
8

---

which is a common evaluation approach for the zero-shot generalization in OF. We report average end point error (AEPE) and Fl-all (%) scores. We also report the delta activation sparsity ASp. which calculates della activations activations across all the iterations executed in the update block. We can see that applying the delta rule substantially reduced the averaged FLOPS needed in general. When using a fixed schedule of 60 iterations to process every input pair, the DeltaDEQ with = 0.001 could save 77% and 66% of FLOPs on Sintel (train) and KITTI (train) in theory without substantially hurting the task accuracy. It is worth mentioning that even when the threshold is set to O, we still got over 50% of ASp.. This is due to the fact of usage of ReLU activations. In combination with early stopping with relative distance criteria and tolerance of 0.001, further reduce the average FLOPs to below 20% of originally needed. We also found that tne-ttne the model (DeltaDEQ with T = 0.005 could improve the task accuracy while reducing the FLOPs and accelerating the convergence.
6 Related work
Delta activation sparsity For accelerating network inference, one can exploit the temporal corre- lation, which naturally arises in time-dependent data such as audio and video. Most related to our work is that of 43 where the authors proposed to use a delta update rule, which zeros out small hidden states change between two input timesteps skip computation. Later works realized this structured delta activation sparsity on custom hardware
20 22] and turned the theoretical FLOPs reduction into runtime acceleration. Another line of work studies similar temporal- -ependent
[2 [4]
delta activation sparsity but with CNNs. In [T] (2 29 44 45| the authors propose to exploit the
linearity of the convolution operator and apply the delta rule to skip zero delta activities for processing consecutive video frames.
Several challenges arise with these approaches. To implement the delta rule in CNN architectures, it is necessary to store the intermediate feature maps of all convolution layers for computing the delta activations. If the network incorporates many layers and has low parameter efficiency, this will lead to substantial memory consumption. Although DeltaDEQ also requires storing feature maps, its property of high parameter efficiency could alleviate this issue by requiring fewer feature maps to be stored. Secondly, there is no guarantee that input data from consecutive time steps will produce temporally correlated features. Consequently, employing the delta rule might result in more overhead, offering no computational advantages. However, due to the -xedtpoint nature of DeltaDEQ, it is guaranteed to have a certain degree of sparsity from the delta activations between two FP iterations.
Deep equilibrium models Deep equilibrium (DEQ) models 2
formulate stacking multiple layers of conventional network layers as a fixed-point solving problem of a single layer block of layers transformation. This could result in a smaller memory footprint and better parameter efficiency. DEQ models have also shown state-of-art task performance in various practical applications including object recogmition or detection 3
25
optical flow estimation 4. face landmark detection [41.. implicit neural representation [32 and many others [10] 46). One of the major drawbacks of DEQ models is its computation cost: for finding the fixed point, many fixed-point iterations or iterations in root-solving are usually required and every iteration requires one inference of the DEQ layer.
Differences to other network acceleration algorithms In comparison to layer skipping methods [ which skip certain layers or halts the recurrence early in inference, our method exploits a finer-grained delta activation sparsity while the other methods can only skip an entire layer or recurrent step. Another line of works dynamically prunes activation, including channel selection 23 for CNNs and patch selection for transformer architectures [52 [5 These works usually set certain channels tokens of the activations to zeros discard them so the network enjoys activation sparsity. Our method allows the activations to maintain non-zero values, while still generating delta activation sparsity, which could be exploited for computation saving. The non-zero activations could potentially make the network more expressive.
27 29
7 Conclusion
In this work, we introduce DeltaDEQ, a method designed to enhance computational efficiency for implicit models represented by deep equilibrium models. This method is inspired by our observation of the heterogeneous convergence phenomenon prevalent in implicit neural networks, where different dimensions of DEQ hidden states converge at varying speeds. DeltaDEQ leverages this disparity by

---

calculating the delta activation between consecutive fixed-point iterations and utilizing the activation's ' sparsity to omit unnecessary computations for the already converged dimensions. We have tested DeltaDEQ across tasks involving implicit neural representation and optical flow, employing both RNN and CNN architectures. Our findings confirm that DeltaDEQ maintains accuracy while reducing computational demands across these network types. We provide detailed theoretical FLOPs reductions from our empirical research. This technique has the potential societal impact to decrease energy usage significantly in DEQ models [2] 4
41 46) and other iterative methods, including 46]. However, translating these theoretical savings into actual wall-clock inference latency reductions often demands carefully designed implementation [15] and special hardware 1
TO 17
those used in iterative refinement [5 and emerging diffusion models [30|46]
Acknowledgments and Disclosure of Funding
This work was partially funded by the European Union's Horizon 2020 research, innovation programme under grand agreement No 899287 and the Swiss National Science Foundation CA- DNNEdge project (208227).
References
[1] Alessandro Aimar, Hesham Mostafa, Enrico Calabrese, Antonio
Ricardo Tapiador- Iulia-Alexandra Lungu, Moritz B. Milde, Federico Corradi, Alejandro Linarcs-Barranco, Shih-Chii Liu, and Tobi Delbruck. Nullhop: A flexible convolutional neural network accelerator based sparse representations of feature maps. IEEE Transactions Neural Networky and Learning Systems, 30(3):
644-656, 2019, doi: 10.1 109/TNNSS 201 2018,285 (Cited on pages and 10
[2] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neurul Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL neurips cc/ aeppappr___
221///119//// 21t00///01..00000666600000000010 10 and 15]
Paper.
pdf (Cited on pages I
[3] Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In Advances in Neural Information Processing Systems (NeurlPS), 2020. (Cited pages] Mand
[4] Shaojie Bai, Zhengyang Geng, Yash Savani, and J. Zico Kolter. Deep equilibrium optical flow estimation. In 2022 JEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 610-620. Los Alamitos, CA. USA. jun 2022. IEEE Computer Society. doi: 10.1109//VPS22265 www.P/2222882222.000 2022.00070. URL
https: / /oi. weeeeccppueessccityyooom
1009/C1 2022. 00070
(Cited on pages
25
and
[5] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. arXiv preprint arX04:1511.00432, 2015. (Cited on page 8
[6] Stefan Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fundamenta mathematicae, 3(1): 3(1):133-181, 1922. (Cited on page
[7] Charles George Broyden. class methods for solving nonlinear simultaneous equations. Mathematics of Computation, 19:577 593, 1965. URL (Cited on pages 22
28029772
5 and I5)
[8] Daniel Butler. Jonas Wulff. Garrett Stanley, and Michael J Black. naturalistic open source movie for optical flow evaluation. In Computer ExsionEECC 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12, pp. 61 611-625. Springer, 2012. (Cited on
pages 8 and
[9] Victor Campos, Brendan Jou, Xavier Giró-i-Nieto, Jordi Torres, and Shih-Fu Chang- Skip RNN: learning to skip state updates in recurrent neural networks. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. https
wwt.:eeeeeotnnnoem witprrunnnddiiwWAyyyyCC HkwVAXyCW)
(Cited page 9
[10] Qi Chen, Yifei Wang, Zhengyang Geng. Yisen Wang. Jiansheng Yang. and Zhouchen Lin. Equilibrium image denoising with implicit differentiation. IEEE Transactions on Image Processing, 32:1868 1881, 2023, doi: 10.11099112200233555500 (Cited pages and 10
10

---

[11] Qinyu Chen, Zuowen Wang, Shih-Chi Liu, and Chang Gao. 3et: Efficient event-based eye tracking using change-based convistm network. In 2023 IEEE Biomedical Circuits and Systems Conference (BioCAS).
pp. 1-5, 2023. doi: 10.1 10.1100//ICAASS449..20000 2023. 10389906.. (Cited on page 9
[12] Qinyu Chen, Kwantae Kim, Chang Gao, Sheng Zhou, Tackwang Jang, Tobi Delbruck, and Shih-Chii Liu. A 65mm 36nJ/Decision bio-inspired caampora--ppasiitee -aare aware digital keyword spotting IC with 0.6
near- sear-threshold SRAM. arXiv preprint arXiv:2405.03900 2024. (Cited on page 9)
[13] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in Neural Information Processing Systems, volume 31, Curran As- sociates, Inc., 2018. URL httpp:://proceedingg mww..pp....ppp.. wwwprip..cccaapppppeeeeee cc/paper files/paper/21111/ httpppppprre////eppeppepp/////// /file// (r55ffffbb..ffoo6666222224288833 Papeseea999pppppppp....e pdf
(Cited on page
[14] Xi Chen, Chang Gao, Zuowen Wang, Longbiao Cheng, Sheng Zhou, Shih-Chii Liu, and Tobi Delbruck. Exploiting symmetric temporally sparse BPTT for efficient RNN training. Proceedingy of the AAAI Conference on Artificial Intelligence, (8(10))111999111)1)))1 URLptttss://oj awa..aaai....................... oxg/index,,.. php hhtp/AAAA//rtiileeriieeee 29020
Mar. 2024. doi: 10.1609/aai,, 388110.29020.
(Cited on page
[15] Spcony Contributors. SpConv: Spatially sparse convolution library.
https://gi gigguu......... comb
traveller59/spcony 2022. (Cited on pages 4 S and 10)
[16] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference computer vision, pp. 2758-2766, 2015.
(Cited on pages and 20
[17] Can Ufuk Ertenli, Emre Akbaş, and Ramazan Gökberk Cinbis. Streaming multiscale deep equilibrium models. In /7th European Conference on Computer Vision, ECCV 2022, 2022. URL hhtpps:/ //www ecva. www.:neeepppeeecccceee wttpp/2222pppppe__pe_e Bwt.:///aapp./117111111--1-----1 and 10]
(Cited on pages
[18] David Fleet and Yair Weiss. Optical flow estimation. In Handbook of mathematical models computer vision, pp. 237-257. Springer, 2006. (Cited on page 7
[19] Samy Wu Fung. Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. JFB: Jacobian-fre free backpropagation for implicit models. Proceedings of the AAAI Conference on Artificial Intelligence, 2022. (Cited page(6)
[20] Chang Gao. Daniel Neil, Enea Ceolini, Shih-Chii Liu, and Tobi Delbruck. DeltaRNN: A power-efficient recurrent neural network accelerator. In Proceedings of the 2018 ACM/SIGDA International Symposium on PoelddProggammabee Gate Arrays, FPGA '18, pp. 21-30, 2018. doi: 10.1 10.1145/314444333144261. (Cited on pages and
[21] Chang Gao, Stefan Braun, Ilya Kiselev, Jithendar Anumula, Tobi Delbruck, and Shih-Chir Liu. Real-time speech recognition for loT purpose using a delta recurrent neural network accelerator. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-5, 2019. doi: 10.1 109/SCAA.22019 8702290. (Not cited.)
[22] Chang Gao, Tobi Delbruck, and Shih-Chii Liu. Spartus: A 9.4 Top/s FPGA-based LSTM accelerator exploiting geatio-temporal sparsity. IEEE Transactions Neural Networks and Learning Systems, 35(1): 1098 1112, 2024. doi: 10. 10.1109/TNLSS.C 2022..3.2022221111 1180299 (Cited on page 9
[23] Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng zhong Xu. Dynamic channel pruning: Feature boosting and suppression. In International Conference on Learning Representations. 2019. URL /tttts:://opnrreeee
Bdd (Cited on page }
[24] Andreas Geiger. Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32 22(1))1231-1231 2013. (Cited on page 8)
[25] Zhengyang Geng and Zico Kolter. TorchDEQ: A library for deep equilibrium models. https://
github. catiun...ommooouuu www///ocslaa//ooooooooooooooeeee torchdeq 2023. (Cited on page
[26] Zhengyang Geng, Xin- Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Advances in Neural Information Processing Systems, 34:24247-242600 2021. (Cited on pages 2
and 20
[27] Alex Graves, Adaptive computation time for recurrent neural networks, 2017, URL https:// //aaxiv org/ abs/ / 1601. 08983 (Cited on page g)

---

[28] Swaminathan Gurumurthy. Shaojie Bai. Zachary Manchester, and J Zico Kolter. Joint inference and input optimization in equilibrium networks. In Thirty-Fifth Conference Nearal Information Processing Systems, 2021. URL
(Cited on page
[29] Amirhossein Habibian. Davide Abati, Taco Cohen, and Babak Ehteshami Bejnordi. 3Xip-convolutions for efficient video processing. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2694-2703, Los Alamitos, CA. USA. jun 2021. IEFE Computer Society,
doi: 10. 109//CPR44437.. 20..37..221..00 1.0022.. tRI.https://do. deeeccoppteeraccieey.oorg wrwp:/t..oo..ooo. 1109/ (Cited on page D
CVPR46437 2021.00272
[30] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. (Cited on page [T]
[31] Berthold KP Horn and Brian G Schunck, Determining optical flow. Artificial intelligence, 175--3): 1981. (Cited on pagel 7
[32] Zhichun Huang, Shaojie Bai, and J. Zico Kolter, (implicit)): Implicit layers for implicit representations. In Advances in Neural Information Processing Systems (NeurIPS), 2021. (Cited on pages] 22 5 and
[33] Shihao Jiang, Dylan Camphell, Yao L.,, Hongdong Li, and Richard Hartley. Learning estimate hidden motions with global motion aggregation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9772-9781, 2021. (Cited on page(8)
[34] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:://p////// wrgg/bs/////////// http:://aarr..orrr///////...... 6980
(Cited page
[35] Yuichiro Koyama, Naoki Murata, Stefan Uhlich, Giorgio Fabbro. Shusuke Takahashi. and Yuki Mitsufuji. Music source separation with deep equilibrium models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 296-300. IEEE, 2022. (Cited on
page
[36] Steven George Krantz and Harold R Parks, The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002. (Cited on p
[37] Renate Krause, Matthew Cook, Sepp Kollmorgen, Valerio Mante, and Giacomo Indiveri. Operative dimensions in unconstrained connectivity of recurrent neural networks. In Ad- vances Neural Information Processing Systems, volume 35, pp. 17073-17085. Curran Asso- ciates, Inc., 2022. URL 2aadlece8330220001111100ee222200
wttps/faaeepppaa//2222222 200p///ppr/2/2//1llll Compere-e pdf (Cited on page
[38] Ilya Loshchilo and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint 160iv: 03983,, 2016. (Cited on 6
[39] Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application stereo vision. In IJCA/'81: 7th international joint conference on Artificial intelligence, volume pp. 674-679. 1981. (Cited on page?)
[40] Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEFF conference computer vision and pattern recognition, pp.
4040-4048, 2016. (Cited on pages X and 20
[41] Paul Micaclli, Arash Vahdat, Hongxu Yin, Jan Kautz, and Pavlo Molchanov. Recurrence without recurrence: Stable video landmark delection with deep equilibrium models. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22814-22825, Los Alami- URL
tos, CA. USA, jun 2023. IEEE Computer Society. doi: 10. VD.109222.
https: //doi www.eccoouterssocceeyyooooo org/10 1199///VP922299 02188
(Cited on pages
and 10
[42] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.055: 2018. (Cited on p)
[43] Daniel Neil, Junhaeng Lee, Tobi Delbrück, and Shih-Chii Liu. Delta networks for optimized recurrent network computation. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017. pp.
2017. (Cited pages 5 and 9
12

---

[44] Mathias Purger, Chengcheng Tang, Christopher Twigg, Cem Keskin, Robert Wang, and Markus Stein-
berger. DeltaCNN: End-to-end CNN inference of sparse frame differences in videos. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12487-12496, Los Alamitos, CA, USA, jun 2022. LEEE Computer Society. doi: 10. 10.1109//VPR5266600 2TT.2226822222..2.. 202..012222222 0127..1 URL (Cited on pages
https: ieeecomputersocleeyy wttp://ssoocttttooooo1o. 10. 1109/CVPR52688 2022. 01217
[45]
Mathias Parger, Chengeheng Tang. Thomas Neff, Christopher D. Twigg Twigg, Cem Keskin, Robert Wang, and Markus Steinberger. MotionDeltaCNNN Sparse CNN inference of frame differences in moving camera vidcos with spherical buffers and padded convolutions. In 2023 TEEE/CVF International Conference on Computer Vision (ICCV), pp. 17246-17255 Los Alamitos, CA. USA, oct 2023, IEEE Computer Society. doi: 10. N0.100//CCV1000002002..0
URL httpss//doi. htw.cccttccttttttrcccttt../// wttp:/aaaaiiitttrrr...o. org/10. 1109/
ICCV51070. 2023. 01586
(Cited on pages 4 and 9
[46] Ashwini Pokle, Zhengyang Geng, and J. Zico Kolter. Deep equilibrium approaches to diffusion mod- els. In Advances in Neural Information P Processing Systems, volume 35, pp. 37975-37999 Curran As- sociates, Inc., 2022. URL https: p/tpp//ei/s/ooeeeengsssee www.eeewdddinnn.nnnesese.e...eee
2022/ file/ and [S)
FFF44a77766311044046677778883833 Pappeee------prrr-----------ee-e Conffrenoneeeeceee.om pdf (Cited on pages
[47] Jack Sherman and Winifred J Morrison. Adjustment of inverse matrix corresponding to change in one element of a given matrix. The Annals of Mathematical Statistics, 21(1): 127, 1950. (Cited on page [5)
[48] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Weizstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 53:7462-7473, 2020. (Cited on page O
[49] Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8934 8943, 2018. (Cited on page 8
[50] David Sussillo and Omri Barak. Opening the black box: low-dimesiional dynamics in high- sech-dimensional recurrent neural networks. Neural Computation. 25(3):626 649, March 2013. ISSN 0899- 7667. doi: 10. 10.1162/NECO_F a_ -000099. MRl.hhtpa:///aa. org/ 10. 1162/NECD a_ 00409
(Cited on page
[51] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Tridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances in neural information processing systems, 33: 7537-7547, 2020. (Cited on page
[52] Ychui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12165-12174, 2022. (Cited on page
[53] 7achary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer #isio-FECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 402-419. Springer. 2020. (Cited on pages 7 8 TO and 20
[54] Homer F. Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis, (715511715-7735. 2011. doi: 10.1137/10783566. URL https://: orreii.or//1 1137/1007836XX (Cited on pages] I 5))
[55] Zuowen Wang, Yuhuang Hu, and Shih-Chii Liu. Exploiting spatial sparsity for event cameras with visual transformers. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 41 411-415, 2022. doi: 10.1 00.1109/CCPP467772222999999
(Cited on page 9
[56] Zuowen Wang, Longhiao Cheng, Joachim Ott, Pehuen Moure, and Shih-Chii Liu. Bio-inspired parameter reuse: Exploiting inter-frame tepresentation similarity with recurrence for accelerating temporal visual processing. In Proceedings of UniReps: the First Workshop on Unifying Representutions in Neural Models, volume 243 of Proceedingy of Machine Leurning Research (PMLR), pp. 209-222. PMLR. 15 Dec 2024.
URL httl:ttttt:///rcedding
mlr. press/v243/ 2at4////aan2a..... html (Cited on pages 2 and 5)
[57] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Advances in neural information processing systems, 32, 2019. (Cited on page 3
[58] Mingliang Zhai, Xuezhi Xiang, Ning Lv, and Xiangdong Kong. Optical flow and seene flow estimation: A survey. Pattern Recognition, 11:1107861, 2021. (Cited on page
13

---

A Appendices
A.1 PCA details
In this section we describe the details for the methods used in Fig. (b) and (c).
For Fig. bb) we randomly selected 5 neighboring r in 2-r, 2x] from the training set model inputs. The inference fixed-point iteration 15 steps, same as in training. Thus, this results in 5 15 75 hidden states, each of dimension 20. We conduct PCA on the R 75x20 hidden states matrix and plot the first 2 sorted principle components and visualize the hidden states in 2D. The colors of the dots are selected so the hidden state from the i-th iteration has the same color. We demonstrate with Fig. [ that if imput points are close in the input space, their distances in the hidden states also close, which justifies the acceleration effect of fixed-point reuse.
For Fig. ()) we infer the model 200 data samples in [-2xr, 2nr] and store the 200 x 15 3000 hidden states. The resulting hidden state matrix is of dimension ii 76000020 We conduct PCA on this matrix and plot the cumulative explained variance the top 8 principal components.
A.2 Complete delta formulation for a convolution layer
section, we give the complete formulation convolution layer a DeltaDEQ when applying the delta rule, For the input features to a layer in a DeltaDEQ block at itcration is Iद and the output is O
0 al * 1 = o(K (I 1 1))))
(14)
o(K (Ji
1, K 1)
(15)
I 11-1 if if - 12T
AC
(Delta Rule, applied elementwise)
(16)
0
otherwise
O
(K. AC )
(17)
cached
C. + K CC (update cache, already calculated 17
(18)
where denotes the convolution operator and K. represents the convolution kernel. The illustration is given in Here we provide the escudo-code for convolution layer with delta rule in DeltaDEQ:
Fig.
Algorithm 1: An convolution layer with delta rule Eq. 17
Input
:Oupput feature map I; from the previous layer, could also be input data if this convolution layer is the first layer of the network. Convolution kemel K. Delta threshold
Cached :Output feature map / Linear operation results C
Result: Output this convolution layer Of. Updated linear operation results C
if = 0 then
/* First iteration, initialize the cache with convolution on the original input
C 1
3 else
/* Apply delta rule elementwise
Al + [ I I 11-1-112
00 otherwise
/* Update the linear operator results to the cache
C-C+K* AT;
end
/+ Apply the non-linearity function o O (C)
/* Move on to the next convolution layer, or if the DeltaDEQ block is finished processing, move on to the next iteration i+1.
14

---

A.3 Applying delta rule with other fixed-point solving techniques
Not only can the delta rule be applied in fixed-point iteration such as in formulation? 2 it can also be utilized in other solvers commonly used in other DEQ works. We take the Broyden method example. As described in the work (2]. the Broyden's 's method 7 updates the hidden states as:
++1 = (Bi
(19)
where (0(2 = [o(=i, 22) zi. the Jacobian inverse and a is the step size. This update step requires computing forward pass fo(=), ze) which is same the vanilla fixed-point iteration in Fg. 2
Then
the computing of this forward can he approximated and accelerated same in Eq. 6 or Eq. 17 Morcover,
the other computatinally heavy part Broyden's 's method, which the calculation of the inverse Jacobian
could also potentailly be accelerated with delta rule. Considering the inverse Jacobian approximation method Sherman-Morrison formula 47 (omitting the timestep for simplicity):
A B¹ Aggtt
B B¹
A A (A=
(20)
where Az 2++1 2 and Agg+t (= .I) (0 .))
(fa( 2 (fa(2' 2²)
(fa(z't1 fa(z') A So theoretically we could apply the delta rule on A+ and 9 gà in order to create sparse vectors in order have sparse matrix vector multiplications in the Therman-Morrisse formula. We
leave the experimental verification future works.
A.4 Feasibility of applying the delta rule in transformer-based DEQ networks
In this section, we discuss the compatibility of the delta rule with transformer-hasee DEQ networks, the transformer architecture is widely used in various applications 446] their DEQ version also has great potential. Here we provide conceptual delta formulation architecture namely the fully connected layers and the self-attention module.
two computationally costly components a transformer
Computing the fully connected layers consists mainly of matrix-vecto multiplications followed by nonlinearity G, FFN(z') = o'''). ). Multiplication of matrix vectors Wx can be formulated with the delta rule just like in the ENN-based DeltaDEQ in [
The self-attention module Self-Attenton((Q. K,V) V) = softma(QK"/
where Q' = Z' K"
Z' "'Wx, k, V Z'Wv, and Zz Rxxx is the state of the DEQ-transforme, architecture, T is the sequence length (equal to the number of tokens) and d is the dimension size. Taking the construction of Q as an example, using the delta rule on the computation Q', K" V gives us:
Q' (Z' z + +2 Wo
(21)
(7' - - 7' T
(22)
Az' + Co
(23)
where the elements in AZ are set to corresponding elements in (Z' 2111 if the absolute value of individual dimensions is greater than the delta threshold т, otherwise it is set to 0. Thus Az'Wo could be sparse matrixx multiplication, depending by Co + Co + A activation function and layer normalization remain the same as a regular transformer model. The computation savings for the self-attention module come from the sparse matrix-matrix multiplication. Assuming the sparsity percentage of AZ is o% then the FL.OPs for computing Q' is o% T d² in comparison to Td?
how Z' evolves. Co is the cached results Co which will be updated Wq. Computing K' and V follows the same pattemn. The following operations such as
15

---

A.5 DeltaDEQ for INR: methods and computing platform details and additional results
A.5.1 Illustration for RNN type of DeltaDEQ
We show two steps of fixed-point iteration for the RNN type of DeltaDEQ in Fig. An additional cache needs be maintained to store the results of the linear components inside the activation function. Here omit the delta rule input dt and focus on the part for hidden state z.
A
Azf
Az
Az
E+2
2F
2
W
+
DeLDD
10 only dimensions participate in
CoLID
10
add cached results
while z converging Az gets sparser
sparse
cached
(W2
AZ +
W2
zt
+ W Xx.
Figure 5: Illustration of saving mechanisms of DeltaDEQ. Fig. Fully connected W 2 type of computation skip. Entire columns of MACs at zero entries of Azi can be skipped and the sparsity of zi grows with iteration
A.5.2 Additional results
Ground Truth
Recon. DEQ-Fourier
Recon. DeltaDEQ-Fourier (ours)
PSNR-30.55 60.62, FL0Ps(Tera)=2.9
PSNR- 29
FLOPs(Tera)-OA 0.4
Figure 6: Original image of the astronaut and reconstructions with INR network. For DeltaDEQ- Fourier, 0.05.
100
FLOPS Reduction (%)
PSNR
25
DeltaDEQ-Fourier DeltaDEQ Siren
15
1
10
10
Inference Delta Threshold
Figure 7: FLOPs reduction and task accuracy (PSNR) at different inference delta threshold on the astronaut image.
16

---

Training Method
Phantom Gradient
Implicit Function Theorem
Tr. FLOPs*
PSNR1
Tr. FLOPs*
PSNR
DEQ-Fourier [32
140.8
29.90
278.4
32.04
DeltaDEQ-Fourde
112.6
29.60
215.5
31.66
DEQ-Siren B2
140.8
25.67
378.5
30.01
-Sirn 122.3
25.59
219.6
30.06
Table 3: Results on Astraunaut. Comparison of PSNR and training FLOPs w/o vs. w the delta rule during training. For reference, with same hyperparameters, the original Fourier and Siren (non-DEQ) networks recorded PSNRs of 30.06 and 33.31, respectively. All FLOPs values are presented in Tera- FLOPs (le12). *This table only includes FLOPs for the forward pass of training; the computation cost of backward pass is independent of the forward pass.
A.5.3 Delta rule formulation for DeltaDEQ-Siren and DeltaDEQ-Fourier
The formulations of DeltaDEQ-Sirent and DeltaDEQ-Fourier are given as follows:
DeltaDEQ-Sire
(24)
21
Sin W - z/- 2 + W + WSin(Vx) + Ux+
(25)
~ Sin
WA
W 1-1 WSin(Vz) Ur cached C22
(26)
cached in
update cache C²,
DelraDEQ-Fourier:
(27)
2
= (W - 21 - W + Wo(x; 00(;; U'
(28)
WA
Wz
W Wg(w;V
+b
ب(ا:U)
(29)
coched
cached C₁
update zache C2
(30)
A.6 DeltaDEQ for optical flow
A.6.1
Heterogeneous convergence for CNN-based DEQ
In Fig. 9 we present a convergence analysis for CNN-based DEQ model 4 The model uses the RAFT architecture 1.6.2]which is the state-of-the-an for optical flow estimation. We randomly selected 60 fcature map pixel locations from the separable ConvGRU and plotted their evolution during the processing of 5 pairs of input images. The activation values are normalized and sorted according their average. The basic fixed-point iteration namely the Picard iteration is applied. From the figure, we can see that certain activation values converge faster than others and some activations oscillate. Fig. sissows the sparsity level of the delta feature maps (original DEQ-flow applying the delta rule with threshold 0.005). We can see that different components tend to reach high delta sparsity different speeds average. This also reflects the heterogeneous convergence phenomenon exists in the CNN- based implicit models.
17

---

100
100
Motion Encoder
Motion Encoder
Head
Average Sparsity (1))
60
(a) input pair
(b) input pair
100
Head
50
80-
Noseape Spanaity
20-
25
Number
Namber (larations
(c) input pair
(d) input pair
Figure 8: Averaged sparsity level of the delta feature map in different components of the update block in Fig. 10 obtained on 4 exemplary pairs of inputs. The flow head has substantially lower delta sparsity than the other two components and they all reached a very high level before the iteration converged globally.
18

---

0.75
Feature Index
0.25
Feature Values
00
0.00
0.25
-0.50
10
-0.75
50
100
150 Iteration Index
200
250
300
(a) with fixed-point reuse flow reuse
0.75
50
0.50
Featire index
Feature Valles
00
0.00
0.25
0.50
10
-0.75
50
100
150
200
250
300
Iteration Index
(b) with fixed-point reuse and flow reuse
Figure 9: Illustration of the evolution of activation values the original DEQ-flow network along the fixed-point iterations when processing 5 consecutive pairs of input frames in the Sintel g dataset. The startings of a new pair are marked with vertical dashed lines. Each pair processed with 60 iterations. Different activation sites converge different speeds and the fourth pair takes more iterations to reach convergence, due to the large motion in the input space.
19

---

A.6.2 DeltaDEQ conversion details for the RAFT architecture
The original RAFT architecture [53], also as adopted in the DEQ optical flow work [4], contains mainly two parts: a))) the feature encoder and the context encoder. Fig. [Q]b) the update block. In hoth RAFT and DEQ-RAFT, the update block is used to iteratively refine the flow prediction thus is the key to obtaining high-quality flow but is also the computationally expensive part. The general architecture illustration is given Fig. [T0 and our DeltaDEQ for optical flow tasks are also instantiated with this architecture. We did not apply the delta rule on the feature and context encoders since these two encoders are only inference once per input pair and xt-1. But it is possible to also convert these blocks with the delta rule to further increase the computation savings. In this work, focus the computationally heavy part which is the update block that will be inference many times for processing every pair of inputs. The update block mainly consists of three parts as marked in orange dashed boxes: (1) motion encoder (2) flow head and (3) separable ConvGRU, The flow head outputs the delta flow A F? which will be used to update the flow prediction F F + A F?- We apply the delta rule to all these three components to accelerate their convolution layers with the delta rule in the DEQ fixed-point iterations.
In Fig. [8 we show the averaged sparsity level of the delta feature maps in different components of the update block when inferencing on four different input pairs. During the evolution the iixed-point iteration, all three components' sparsity levels went up. It is worth noticing that the flow head's sparsity level started with a low level. potentially due to the fact that it is predicting a first-order difference value A/"
Context
Motion Encoder
Flow Head
Corr
Convix1
Comv3x3
Conv3x3
Conv3x3
Convixt
APE
Conv7x7
Res.Unit
Rei.
Res. Unit
Res.Unit
Res Uni
Conv3x3
Res:
SepConvGRU
Flow
Conv7x7
Conv3x3
A
(a) Feature Encoder and Context Encoder
(b) Update Block with hidden dimension 256
Figure 10: Architecture illustration for RAFT 53
DEQ flow and our DeltaDEQ.
In our experiments, we use DEQ-RAFT-H where H represents model class huge in the work 4 with approx. 13.4 M parameters, that the hidden channels for the separable ConvGRU are 256, the output channels for the feature encoder and context encoder both 256. We use the DEQ-RAFT-H checkpoint provided in 4 which trained with first 120k iterations on the FlyingChairs [16] dataset and with phantom gradient [26] and 120k steps on the Things3D [40] dataset with phantom gradient. The fine-tuned DeltaDEQ-ft in Tab, 2 was further fine-tuned only on the Thinggghng datasct with 20k iterations and batch size 12 and learning rate 1 and phantom gradient. During fine-tuning set the weight decay zero and did not use the FlyingChair dataset since either will degrade the accuracy. Fixed-point reuse and flow reuse are also applied.
A.7 Comparison of fixed-point iteration and root-solving techniques in the forward pass
The DEQ series works first proposed the use of root-finding techniques to solve the fixed point in the forward pass. We argue that using fixed-point iterations including the simple Picard's iteration or the Krasnoselskii-Mann (KM) iteration suffices obtaining a good estimation of the fixed-point =j. Tab, shows the comparison among using the solvers (Broyden, Anderson) and using fixed point iterations (Picard, KM). We care mostly about the best possible solving quality each method could get thus we did not use carly stopping nor fixed- point reuse and we increased the number of iterations until the performance increase reached plateau. We found that fixed-point iterations actually outperformed root-solvers in indicating a faster global convergence speed. Moreover, the fixed-point iterations also have an advantage in terms of the best performance reached. Thus we argue that using fixed-point iteration is better choice for the inference of DEQ models since they not only achieve better task accuracy and have faster convergence speed, but they also require fewer computes and simpler computing patterns.
fixed-point
of performance controlling the number of iterations,
20

---

Sintel
(train)
KITTI (train)
Forward
model
iterations
clean
final
AEPE!
Fl-all|
DEQ-RAFTH
10
3.97
4.87
10.93
28.83
Anderson
DEQ-RAFT-H
20
1.90
3.16
6.02
17.72
DEQ-RAFT-H
30
1.54
2.79
5.15
15.23
DEQ-RAFT-H
40
1.47
2.72
4.62
14.11
DEQ-RAFT-H
50
1.44
2.68
4.62
13.68
DEQ-RAFT-H
60
1.43
2.64
4.37
13.46
DEQ-RAFT-H
10
1.87
3.32
6,26
19.27
Broyden
DEQ-RAFT-H
20
1.45
2.70
4.58
14.26
DEQ-RAFT-H
30
1.39
2.63
3.98
13.29
DEQ-RAFT-H
40
1.36
2.60
3.85
13.13
DEQ-RAFTH
50
1.31
2.61
3,86
13.10
DEQ-RAFT-H
60
1.30
2.59
3.83
13.07
DEQ-RAFTHH
10
2.03
3.94
6.02
18.55
DEQ-RAFTHH
20
1.48
2.70
4.49
14.17
Picard
DEQ-RAFTHI
30
1.35
2.63
3.91
13.22
DEQ-RAFTH
40
1.32
2.60
3.80
13.04
DEQ-RAFT-
50
1.32
2.58
3.78
12.98
DEQ-RAFT-
60
1.27
2.61
3.78
12.97
DEQ-RAFT-H
20
1.42
2.71
4.56
14.32
0.9
KM
DEQ-RAFT-H
40
1.29
2.60
3,85
13.04
DEQ-RAFT-
60
1.28
2.59
3.81
12.97
a
DEQ-RAFT-
80
1.29
2.60
3.77
12.94
Table 4: Comparison among different solvers fixed fxeedpooint iteration methods for the forward pass.
A.8
Computing platform
All experiments were conducted on an Nvidia RTX 3090 GPU with 24GB of RAM and Intel(R) Xcon(R) W-2195 CPU @ 2.30GHz.
21

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reffect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed known limitations in the conclusion section. We also descrihed the assumptions and scope of our method in the methods description and results sections.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, those are not discussed the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting.
speech-to- seech-to-tex- text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There is discussion about theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and refsereffeeeced.
22

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper discloses the information needed to reproduce experimental results to the best we can. The code for the methods will be made publically available upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution is a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct
model (e.g.. large language model), then there should either be
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The datasets used in this work are publicly available. The code will be made publically available upon formal publication.
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpe://nips... htttpe://niis.cccpull..t cc/public/
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
23

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines
Buzdee//DodeSubmiissiooooPP
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We specified the training and test details for all the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the mean and standard deviations for experiments that permit. However, part our experiments are based on the conversion of model checkpoints from other works, and the authors did not relcase multiple random seeded checkpoints for the same setting. We compensate for that by showing the effectiveness of our method with various conversion settings to show the robustness of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the computing resources required for the main experimental results.
24

---

Guidelines:
answer means the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics
Answer:
[Yes]
Justification:
The paper conforms with NeurIPS Code Ethics to the hest our knowledge.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Potential societal impacts the work is discussed conclusion section.
Guidelines:
The answer means that there is no societal impact of work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples negative socictal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is no mentioned risk for this work.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make hest faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g.. code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected"?
Answer: [Yes]
Justification:
We have cited necessary code package and datasets in the work.
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
27


Reviews:
[{"review_id": "VBhnATFgak", "reviewer": "Reviewer_dpig", "summary": "The paper proposes the DeltaDEQ framework to accelerate the forward pass of the Deep Equilibrium Model. Initially, the paper identifies the heterogeneous convergence phenomenon, which shows that different dimensions of the state converge at uneven speeds in the forward pass of DEQ framework. Inspired by the phenomenon, the DeltaDEQ framework stores the past linear operation results and only propagates the activation when its change exceeds a threshold. Experiments on implicit neural representation and optical flow estimation tasks show that DeltaDEQ improves the efficiency of DEQ framework without hurting performance.", "questions": "See the sections above.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Very clear writing.\n2. The method is simple and easy to understand.", "weaknesses": "1. Lack of experiments. The paper only shows the result of implicit neural representation and optical flow estimation tasks. However, DEQ has a wide variety of applications, including language modeling [1] and generative modeling[2]. I think the authors should provide more experimental results to show that the method can be applied to different tasks. \n2. The method is limited to CNN and RNN architecture. In the paper, only CNN and RNN-based architectures are discussed. Can this method be applied to popular transformer-based architectures？\n\n[1] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. \"Deep equilibrium models.\" Advances in neural information processing systems 32 (2019).\n\n[2] Pokle, Ashwini, Zhengyang Geng, and J. Zico Kolter. \"Deep equilibrium approaches to diffusion models.\" Advances in Neural Information Processing Systems 35 (2022): 37975-37990.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and the precise summarization of our work! For points 1 and 2 in Weaknesses, the corresponding answers are under $\\textbf{W1}$ and $\\textbf{W2}$ respectively.\n\n$\\textbf{W1: }$ We thank the reviewer for the suggestion! We conducted new experiments exploiting the delta activation sparsity on a new language modeling task using the Wikitext-103-v1 dataset and the DEQ-Transformer architecture. The formulation is given under the next point. We give some results for this task in the following table (R-Tab 2). $\\textbf{For the complete results table (R-Tab 1)}$ and more experimental details and other benchmarks, please refer to the second point $\\textbf{G2}$ of the general rebuttal reply. The conclusion on the language modeling task with DeltaDEQ-Transformer is that the results do generalize to the text dataset. By applying the delta rule, the computation costs for the FFN and QKV construction dropped by  52% and 72% respectively, and with a slightly better test perplexity.  \n\n| Model              |             mem_len=150, | seq_len=150.           |            | |                    mem_len=480, |seq_len=150.  |        |           |\n|--------------------|------------|-------------------------------------|------------|--------|-----------|------------|-------------------------------------|-----------|\n|                    | Test PPL   | FLOPs FFN                           | FLOPs QKV  | | Test PPL  | FLOPs FFN  | FLOPs QKV                           |           |\n| DEQ-Transformer    | 22.34      | 242.4                               | 34.4       | | 21.85     | 242.4      | 72.2                                |           |\n| $\\textbf{DeltaDEQ τ=0.01}$    | $\\textbf{22.24}$      | 115.3 (-52%)                        | 15.6 (-55%)| | $\\textbf{21.75}$     | 115.5 (-52%)| 20.5 (-72%)                         |           |\n\nR-Tab 2: Results on the Wikitext-103-v1 dataset. DeltaDEQ refers to DeltaDEQ-Transformer and $\\tau$ is the inference delta threshold. Complete results are in $\\textbf{G2}$ of general author rebuttal.\n\n$\\textbf{W2: }$ We show that our method can be applied to transformer-based architectures such as DEQ-Transformer [1]. Here we provide the conceptual delta formulation (experimental results are in the reply to $\\textbf{W1}$ and the general review rebuttal) for the two computationally costly components in a transformer architecture (a) self-attention mechanism and (b) the feedforward layer (FFN), or fully connected layers. \n\n$\\text{SelfAttention}(Q,K,V) = \\text{softmax}(QK^{T}/\\sqrt{d})V$,\n\nwhere $Q^i = Z^i W_Q, K^i = Z^i W_K, V^i = Z^iW_V$, and $Z^i \\in \\mathcal{R}^{T \\times d}$ is the state of the DEQ-transformer architecture, T is the sequence length and d is the dimension size.\n\nApplying the delta rule on the computation of $Q^i, K^i, V^i$: \n\n$Q^i = (Z^i-Z^{i-1} + Z^{i-1}) W_Q = (Z^i-Z^{i-1}) W_Q + Z^{i-1} W_Q\n= \\Delta Z^i W_Q + C_Q$,\n\nwhere the elements in $\\Delta Z^i$ are set to corresponding elements in $(Z^i-Z^{i-1})$ if each is  larger than the delta threshold $\\tau$, otherwise it is set to 0. Thus $\\Delta Z^i W_Q$ is a sparse matrix-matrix multiplication. And $C_Q$ is the cached results, which will be updated by $C_Q  \\gets C_Q + \\Delta Z^i W_Q$. Computing $K^i$ and $V^i$ follows the same pattern. The following non-linearities such as layer normalization remain the same. The computation savings come from the sparse matrix-matrix multiplication. Assuming the sparsity percentage of  $\\Delta Z$  is $sp%$ then the FLOPs for computing $Q^i$ is $sp% \\cdot T \\cdot d^2$ in comparison to $Td^2$.\n\nComputing the feedforward (FFN) layers consists mainly of matrix-vector multiplications and applying the activation function $\\sigma$, $\\text{FFN}(x^i) = \\sigma(Wx^i)$, where $Wx^i$ can be formulated with the delta rule like the fully connected layer of DeltaDEQ in Eqs. 5-7 as in our work. So we omit the formulation here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TsPoVAlbrF", "reviewer": "Reviewer_71es", "summary": "The authors begin by presenting the observation that in fixed-depth networks, only three dimensions are sufficient to explain the trajectory of a dimension-20 hidden state. With this motivation in mind, they introduce the notion of heterogeneous convergence, which can be summarized as the concept that different dimensions in the hidden state converge at different rates. This is demonstrated for example in Figure 1d, where it is clear that some dimensions fluctuate significantly more than others which seem to converge rapidly. Hence, the authors re-formulate the fixed point iteration as in Equation (4), and apply what they call a “delta rule” whose goal is to save on computation by creating a sparse vector to multiply with a matrix. In experiments, they show that the training number of FLOPs greatly decreases.", "questions": "* Some of my questions are in the above so I will not repeat them here. Overall, it would be very helpful to clarify the answers to these questions in order to make the results of the method more clear.\n* It would be nice to maybe include some ablations. For instance, what is the impact of the other methods you included to speed up inference? \n* I am a bit confused by what is being said in lines 253-259 due to the wording. Can you please clarify what you mean? \n* Table 1 is unclear for a few reasons. First, what is “inf.”? If one is to assume that “inf.” refers to inference FLOPs, then I am confused– do the FLOPs increase despite decreasing for training? I assume this has to do with the fact that tau is higher, but what is the reasoning for choosing different tau in training and inference? This is actually a nice result, and maybe it should be emphasized in the paper that perhaps an even lower value for tau would be sufficient to obtain a nice result.\n* Line 65 is incorrect. It is not the case that iteratively applying a function guarantees convergence to a fixed point. Can you please fix this? \n* Did you try your method for any language modeling tasks? I would be interested in seeing if the result holds for different tasks.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper aims to decrease the number of FLOPs required for DEQs. This is a relevant problem given that DEQs are parameter-efficient models in terms of performance, and their main drawback is their computational cost. Hence the paper is well-motivated.\n* The experiments section (5) is strong and provides a careful analysis of the benefits of the method. In particular, there is a clear reduction in training FLOPs. Figure 4 makes it clear that increasing the threshold for Delta decreases the number of inference FLOPs required, which is to be expected, while also decreasing the PSNR. It is nice to see how increasing tau affects the performance of the model in terms of*  FLOPs and PSNR.\n* The method is novel and appears to be a very principled approach to reducing the FLOPs required by DEQs. It is clever to re-formulate Equation (4) and Equation (5) and to use caching along with the Delta method to speed up inference.. \n* The authors appear to be well-aware of the related literature and include relevant references. They include other methods as well for accelerating DEQs in order to maximize the acceleration.", "weaknesses": "* Section A.5 is very interesting but in a sense seems to discount the use of a DEQ. Classically a DEQ should find the fixed point, and should not halt until some threshold of being close enough to a fixed point is reached. It seems to me as though the DeltaDEQ method is only compatible with fixed-point point iterations due to the update method in Equation (3). Consequently, line 156 is lacking in precision. What exactly is defined as “good … convergence”? Furthermore, how many fixed-point iterations are there, for example, in the models in Table 1? I am curious to what extent convergence to a fixed point is necessary for your method to work, or if your method is more closely related to a weight-tied network as in “End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking” [Bansel et al 2022]. If it is the case that convergence to a fixed point is not necessary to obtain a good result with your method, I would truly hesitate in considering this a method for speeding up a DEQ.\n* In Section 3, authors do not actually train a DEQ, they trains a recurrent neural network, or what some may call a weight-tied network. Furthermore, there is no noise in this setting and f(x) = sin(x) is quite a simple function. Consequently, it would be more interesting to (1) train a DEQ with fixed-point solving (2) either choose a more complicated ground-truth function or show many examples of this observation with other functions. Furthermore, although this is a toy example, it must be noted that this latent space of dimension 20 is far less than one used for a larger-scale DEQ. Does this observation of heterogeneous convergence hold in a much larger dimension? \n* Some details that are incorrect/ confusing: The caption states that the backward pass takes a constant amount of FLOPs, although this should not be the case when IFT is the backward method, since IFT also has a fixed-point iteration. Also, the authors write that the training method is Delta-DEQ in the first column, despite having also non- Delta-DEQ methods included as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the in-depth reviews! Our reply is arranged as $\\textbf{W1}$ for the first point in Weakness etc.\n\n$\\textbf{W1.1: }$ The delta rule is not limited to the vanilla fixed-point iteration (FPI) form as in Eq.3. Methods like Broyden’s solver as used in the original DEQ work can also be accelerated with delta rule: as in [C1, Eq.6] Broyden’s method gives $z^{i+1} = z^{i} - \\alpha B (f_{\\theta}(z^{i}, x) - z^{i})$, where the computationally heavy part is the calculation of the inverse jacobian matrix, $B$ and the forward pass $f_{\\theta}(z^{i}, x)$. We can still apply the delta rule on this forward pass to generate sparse delta activations by exploiting the heterogeneous convergence nature of DEQ. We applied the delta rule with FPI because of the findings in A.5 which shows that task accuracy-wise FPI is better than solver-based methods when using the same number of iterations/solver steps during inference.  FPI also requires fewer computes and memory storage than other methods including Broyden and Anderson root-solvers[C1]. \n\n$\\textbf{W1.2 ‘Consquently,..’: }$ Line 156 ‘Good … convergence speed’ describes the findings in Sec. A.5 where global convergence speed refers to the iterations needed for the task accuracy to saturate. Moreover, with the same number of fixed-point iteration (FPI, including Picard and KM) vs. solver steps (Broyden, Anderson), the task accuracy of the former is on-par or better than the latter. \n\n$\\textbf{W1.3 ‘I am curious…’: }$ Firstly, if DeltaDEQ is unable to converge to fixed points, then when using early stopping as shown in Tab. 2, the FPI should not converge or will always use the maximum allowed number of steps, since with early stopping the inference will only halt when the distance of the activations between two iterations is smaller than a tolerance. Secondly, global convergence to fix points is also not guaranteed in practical use cases for the original DEQs [C2]. One of the major differences between DeltaDEQ/DEQ and weight-tied network lies in the training method: the former uses Implicit Function Theorem or Phantom Gradient, while the latter often uses (truncated) BPTT type of training. The BPTT training has a large memory footprint for storing past activation values. We will add the differences in the related work section.\n\n$\\textbf{W2: }$ We thank the reviewer for the valuable suggestion! For the optical flow task, Fig 9. Sec A.4.1. shows exemplary convergence speed for a subset of feature map locations. Fig. 8 presents the average delta sparsity levels (\\%) in DeltaDEQ-RAFT model on 4 pairs of inputs. The figure shows that the delta activations at certain pixel locations converge earlier than other locations during the fixed-point iterations.\n\n$\\textbf{W3: }$ We are very sorry for the confusion. Indeed the FLOPs of the backward method vary depending on the IFT’s implementation. This will be corrected to ‘The computation cost of backward pass is independent of the forward pass’.  Since the main focus of this work is to accelerate inference using the heterogeneous convergence phenomenon, it should not affect our conclusions. We will also fix the naming of the first column of Tab.1 to distinguish between the two cases.\n\n$\\textbf{Q1: }$ Please see the replies for the weaknesses.\n\n$\\textbf{Q2: }$ Thanks for the suggestion. One ablation is contained in Tab 2. i.e. the effect of (global) early stopping. Results show that with 60 fixed-point iterations, our method reduced the FLOPs by a lot, and the delta rule also works effectively when early stopping is applied.\n\n$\\textbf{Q3: }$ Line 253-259 describes a DEQ training technique introduced in [C3]. We will rephrase the sentences for better clarity. Basically, between two $\\textbf{training}$ epochs, the fixed point obtained from a previous epoch is used as the initialization value for the next epoch. The authors [C3] found when the reuse of these hidden state values is applied, it is in general sufficient to only use 1 forward fixed-point iteration for each subsequent epoch, which greatly speeds up the training. If we do not apply this training trick, we will see an even bigger reduction in training FLOPs. On other tasks such as OF, one forward iteration is not sufficient for convergence even when reusing the hidden states, so the delta variant will benefit from more training epochs. \n\n$\\textbf{Q4}$ Inf. stands for inference. The inference TFLOPs (0.64 for instance) of delta trained DeltaDEQ-INR variants should be compared with the inference TFLOP numbers (4.14) without using delta rule in inference, which means that there is over 84\\% decrease in FLOPs. The inference $\\tau$ is chosen according to the best trade-off spot found in Fig. 4 when only using delta in inference. We found that for the INR task, using a large  $\\tau$ when training from scratch makes the training process unstable and often diverges. The smaller training $\\tau$ in the table led to stable training. We will study how $\\tau$ affects DeltaDEQ training in future work. \n\n$\\textbf{Q5: }$ Thanks for pointing this out! We will emphasize in the text, the assumption of “assuming $f_{\\theta}$ is a contraction mapping.” or under the assumption of the existence of equilibrium as in [C1].\n\n$\\textbf{Q6: }$ We have added new experimental results for a language modeling task! Due to the character limit of rebuttal reply, we kindly ask the reviewer to refer to the second point of the general rebuttal reply G2. \n\n[C1] Deep Equilibrium Models, Bai et al.\n\n[C2] Global Convergence of Over-parameterized Deep Equilibrium Models, Ling et al.\n\n[C3] (Implicit)2 : Implicit Layers for Implicit Representations, Huang et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfCdRtfHe5", "reviewer": "Reviewer_zWM7", "summary": "This work provides a finer-grained analysis of the dynamics involved in updating hidden states and identifies a phenomenon of heterogeneous convergence in implicit models, where certain dimensions of hidden states converge much faster than others. Based on this observation, the forward pass update of DEQ models was modified using a delta updating rule that stores intermediate results of computationally intensive linear operations and only recalculates dimensions when changes between updates exceed a threshold. This approach is complementary to other acceleration techniques and does not rely on temporal input correlation, making it effective even for static inputs. The method, primarily applied to DEQ-based models but adaptable to other iterative methods, was tested on implicit neural representation (INR) for images and optical flow (OF) estimation.", "questions": "- Lines 89-91: According to the description in Appendix A.1, the number of PCA components was measured on a toy dataset, specifically the approximation of $sin(x)$. I am curious whether the same phenomenon can be observed on a real dataset. Additionally, although the major components account for most of the variance, this does not necessarily mean the residual values are negligible, particularly in the context of image restoration.\n- Could the authors provide specific information on the sparsity ratio achieved after element-wise delta threshold pruning?\n- How is the optimal threshold for each layer determined? It seems that an ablation study is necessary for different tasks and networks.\n- The authors mention that \"due to the fixed-point nature of DeltaDEQ, it is guaranteed to have a certain degree of sparsity from the delta activations between two FP iterations.\" Is there any proof supporting the desirable degree of sparsity between two FP iterations? Typically, the feature map is considered as a whole to derive the solution to the deep equilibrium model. How can a pixel-wise analysis be conducted on the fixed-point nature of DeltaDEQ? Additionally, theoretically, the proposed method reuses different fixed points $z_{t-k}^*$ as the initialization for processing the next timestep input $x_t$. Could this approach lead to convergence to the same $z_t^*$ as the original method, or is the convergence not guaranteed?", "limitations": "I found no potential negative societal impact of this work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I find the topic of pixel-level faster convergence in deep equilibrium models to be quite intriguing. Based on the experimental results presented in Section 5, this approach appears to have practical value in mitigating the issue of extensive computations during inference, thereby improving the approximation of the fixed point.\n- The technical aspects of this paper are straightforward and easy to follow.", "weaknesses": "- While the proposed method appears unique in the context of accelerating deep equilibrium iterations, its underlying concept is akin to standard dynamic network pruning methods, such as dynamic gating or controllable layer skipping. Unfortunately, these related works are not discussed in Section 6. The key differences between the proposed method and existing dynamic pruning approaches seem to be the pruning pattern (element-wise in this paper versus the more common block-wise pruning along the spatial dimension) and the threshold selection strategy (learnable predictor networks versus handcrafted thresholds like $\\tau$). Additionally, the current submission lacks information on real runtime speedup. As the authors noted, \"employing the delta rule might result in more overhead, offering no computational advantages,\" leaving me curious about whether the proposed method can achieve actual runtime speedup instead of merely theoretical reductions in computing cost.\n- The \"cache update\" operation described in Equation (7) may introduce extra approximation error, in addition to the pruning error from Equation (5), since $C_z:=W_z\\cdot\\Delta z_t^i+C_z$ relies on $\\Delta z_t^i=z_t^i-z_t^{i-1}$, which may not hold under the delta rule.\n- The additional description of the \"Convolutional DeltaDEQ\" might be unnecessary, as the derivation from Equation (6) to Equation (8) is fairly straightforward. The main difference is that the sparse pattern should be compatible with the computational operations.\n- Although the backward pass is independent of the forward pass, the value of $z^*$ determines the optimization process during backward propagation. I wonder if the learned parameters can still be used for model inference without applying the \"Delta rule.\"\n- The results in Table 1 show a noticeable performance drop with only a slight reduction in computing cost. Furthermore, the reported inference speedup seems inconsistent with the results under \"Inf. $\\tau=0.05$.\"\n- The authors note that \"the backward pass FLOPs remain constant and are not influenced by the forward pass.\" I would expect to see an overall reduction in training cost based on DeltaDEQ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments! \n\n**W1.1:** In comparison to 'layer skipping' or 'dynamic gating' methods including the Adaptive Computation Time (ACT) work[C1], our method  exploits much finer-grained delta activation sparsity, because ACT can only decide to halt the entire global recurrent update. In comparison to other non-RNN-based works like SkipNet [C2] or [C3], DeltaDEQ uses a single block of layer(s) iteratively which could achieve similar accuracy level with fewer parameters.\n\n\n‘Dynamic network pruning’ could refer to selecting input channels for CNNs [C4] or patch for transformers[C5]. These works usually set certain dimensions/channel/tokens of **activations** $z^i$ to 0s or discard them, while DeltaDEQ allows the activations to still maintain non-zero values but their **delta activation values under threshold** ($|z^i - z^{i-1}| < \\tau $) are set to 0s. Thus, with the same activation/feature map size, in theory, DeltaDEQ should contain more information and thus more expressive.\n\nOur method is in principle **orthogonal** to these methods and can be used in combination. \n\nCould the reviewer provide specific literatures so we can discuss the differences in the related work section. We are also happy to continue the discussion in rebuttal. \n\n**W1.2: Additionally’:** It is feasible to realize actual runtime speed-ups. The computation saving in DeltaDEQ are brought by element-level delta activation sparsity. Although current mainstream hardware platforms may not natively optimize for this type of sparsity, several works have offered promising solutions. That include custom hardware designs (FPGA/ASICS) [C6-7] or tailored implementation on CPU/GPU [C8]. Specifically, the authors of [C8] estimated a 1.86x runtime speed-up for a CNN with 67\\% activation sparsity. Given that the primary focus of our work is on leveraging heterogeneous convergence to reduce the computation cost of DEQ networks, we will leave the exploration of hardware implementation in future work.\n\n\n**W2:** Indeed Eq.7 should be an operation rather than a mathematical equivalence. We will modify Eq.7 to $C_z \\gets W_z \\Delta z^{i}+C_z$. Regarding the approx. error in C_z and z*, we reply under Q4. \n\n**W3:** The main difference is indeed delta sparsity patterns. We included Eq.8 for ease of readability. We can consider moving them to the Appendix.\n\n**W4:** The task accuracy remained similar level. We provide several results (due to space limit, please compare to our paper) of evaluating delta-trained/finetuned model as regular DEQs: For INR: Delta-DEQ-Fourier, tr.$\\tau=1e-4$, IFT, PSNR=34.21.  For OF: DeltaDEQ-ft, Sintel Clean=1.21, Final=2.56, KITTI AEPE=3.78, F1-all=13.33.\n\n\n**W5:** Sorry for the confusion. The reported FLOPS reduction in Fig 3 and Fig. 4. are according to the text under the subsection of  **Inference acceleration with DeltaDEQ** i.e. without delta in training. Only using delta in inference already lead to a reduction in inference FLOPs as shown in Fig 3. The inference FLOPs in Tab.1 should be compared to the **inference** FLOPs without using delta, namely the inference FLOPs under $inf. \\tau = 0$ in the numbers above for W4. We will definitely fix the confusions and add complete results for W4 on the paper.\n\n**W5:** A brief comparison of the computation costs of the backward pass (BP) and the forward pass (FP): Assuming in the DEQ formulation as in Eq.3, the FP of obtaining $z^*$ costs $O(k\\cdot d^2)$ where k is the number of forward iterations. The backward pass, using phantom gradient [C9], costs $O(c \\cdot d^2)$ where c is the number of steps in PG and usually $k > c$. The computational cost for FP and BP are on par thus accelerating the FP could benefit the overall training speed.  \n\n**Q1:** Similar low-dimensional dynamics was also observed in other fixed-point modeling works [C10, Fig 3,4,7 etc]. A direct implication from PCA to the activity space indeed can not be given, thus we also provided element-wise states evolution in Fig.1.d and Fig. 9.\n\n**Q2:** OF task: the sparsity ratio is in Tab. 2 under ‘$\\Delta Sp.$’ and sparsity ratio for examples are in Fig. 8. INR task: FLOPs reduction reflects the delta activation sparsity ratio because most of the FLOPs come from computing $W\\cdot h$ vs. $W\\cdot \\Delta h$, which means the sparsity of $\\Delta h$ is roughly the same as FLOPs reduction.\n\n**Q3:** OF task: we did a global inference delta threshold sweep to obtain results in Table 2. Further increasing delta threshold will decrease accuracy. INR task: results from the delta threshold sweep can be seen in Fig 4. and 7. \n\n**Q4.1, Q4.2:** Theoretically, when the $i \\to \\infty$, the $|z^{i+1} -z^i|\\to 0$. So when the distance goes to zero it is guaranteed to have a fully sparse delta since $z^{i+1} = z^i$. Empirically we can’t do infinite iterations, but Figs.8 and 9 show that  a high degree of delta sparsity already appears even after a few iterations.\n\n**Q4.3 ‘Additionally’:** It is indeed not guaranteed to converge to the exact same $z_t^∗$. But as being empirically verified in our work and other DEQ works cited in Sec. 4.3, converging to the exact same $z_t^∗$ is not required for obtaining similarly good task accuracy.\n\n[C1] Adaptive Computation Time for Recurrent Neural Networks, Graves et al. \n\n[C2] SkipNet: Learning Dynamic Routing in Convolutional Networks, Wang et al.\n\n[C3] Adaptive Neural Networks for Efficient Inference, Bolukbasi et al. \n\n[C4] Dynamic Channel Pruning: Feature Boosting and Suppression, Gao et al. \n\n[C5] Patch Slimming for Efficient Vision Transformers, Tang et al.\n\n[C6] NullHop: A Flexible Convolutional Neural Network ..., Aimar et al. \n\n[C7] EIE: Efficient Inference Engine on Compressed Deep Neural Network, Han et al. \n\n[C8] Inducing and Exploiting Activation Sparsity..., Kurtz et al.\n\n[C9] On Training Implicit Models. Geng et al.\n\n[C10] Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks, Sussillo et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMt6x0JOpI", "reviewer": "Reviewer_WcEE", "summary": "This proposed a DeltaDEQ method, which is designed to enhance computational efficiency for implicit models represented by deep equilibrium models. This method is inspired by the authors' observation of the heterogeneous convergence phenomenon prevalent in implicit neural networks, where differen dimensions of DEQ hidden states converge at varying speeds. The authors have tested DeltaDEQ across tasks involving implicit neural representation and optical flow, employing both RNN and CNN architectures. It seems that the proposed DeltaDEQ maintains accuracy while reducing computational demands across these network types.", "questions": "1. What is the generalization ability of DeltaDEQ on different types of tasks and data sets? Can the author provide more data on their performance on diverse tasks?\n2. What is the basis for the selection of key parameters (such as threshold τ) in DeltaDEQ ? Do these parameters have optimal values for different data sets or tasks?\n3. How to deal with sparsity in the actual implementation of DeltaDEQ? Is there a special data structure or algorithm optimization?\n4. How is the statistical significance of the experimental results evaluated? Is there an appropriate error analysis?", "limitations": "1. DeltaDEQ may perform well on specific types of tasks, such as implicit neural network representation and optical flow estimation, but the paper may not provide sufficient evidence to demonstrate its applicability in other types of tasks or different fields.\n2. Although the amount of calculation is reduced during reasoning, the computational complexity and convergence speed of DeltaDEQ in the training phase have not been fully evaluated.\n3. If the paper does not provide or promise to provide code and data, this may limit the transparency and reproducibility of the study.\n4. The experimental design of the paper may have limitations, such as the data sets used may not be diversified enough, or the experimental settings may not fully test all key aspects of the model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 It is interesing to propose heterogeneous convergence, which may exist in deep equilibrium models and other iterative methods and is found by analyzing the element-wise trajectory and dimensionality of hidden states update.\n2 The proposed  method does not assume temporal correlation of inputs i.e. effective even for static, which is different from fixed-point reuse acceleration.\n3  The proposed approach achieved an 84% reduction in FLOPs for the INR task and a 73% and 76% reduction for the OF task using the Sintel and KITTI datasets, respectively, without significant loss in task accuracy.", "weaknesses": "1. Although the paper provides some implementation details, more details about the implementation of the algorithm may be needed, especially for the sparse processing and threshold selection of DeltaDEQ.\n2. The paper is not compared with the current state-of-the-art methods or enough other methods, it may be unconvincing and unable to fully demonstrate the superiority of DeltaDEQ.\n3. Although the paper mentions the reduction of FLOPs, it does not elaborate on the actual running time or resource consumption, including memory usage.\n4. The paper does not fully explore the sensitivity of DeltaDEQ to threshold parameters or other hyperparameters, this may affect the stability and reliability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. In the reply below, we use for example $\\textbf{W1}$ for the reply to point 1 in Weaknesses, etc. \n\n$\\textbf{W1 and W4: }$ We thank the reviewer for the reminder. For the realization of the delta rule and sparse processing in RNN and CNN, we provided illustrations in Fig. 5 and Fig. 2 respectively, and a pseudo-code for the DeltaDEQ-CNN in Sec. A.2. We will release the code under paper publication. \n\nRegarding the delta threshold selection, we already conducted an inference delta threshold sweep for the results shown in Fig. 4, Tab 2, and in the new results of R-Tab. 1 (under general author rebuttal $\\textbf{G2}$). We would be happy to provide more details for further clarification.\n\n$\\textbf{W2: }$ We have now conducted new experiments on a language modeling task with a DeltaDEQ-Transformer architecture. We provide some results in the following partial table R-Tab. 3. $\\textbf{For the complete results}$ and descriptions, please refer to the second point of the general rebuttal reply $\\textbf{G2}$ and R-Tab 1. The results show that our method generalizes to the language modeling dataset and transformer architecture. By applying our method, the computation costs for the FFN inference and QKV construction dropped by  52\\% and 72\\% respectively, and with a slightly better test perplexity (PPL).  \n\n\n| Model              |             mem_len=150, | seq_len=150.           |            | |                    mem_len=480, |seq_len=150.  |        |           |\n|--------------------|------------|-------------------------------------|------------|--------|-----------|------------|-------------------------------------|-----------|\n|                    | Test PPL   | FLOPs FFN                           | FLOPs QKV  | | Test PPL  | FLOPs FFN  | FLOPs QKV                           |           |\n| DEQ-Transformer    | 22.34      | 242.4                               | 34.4       | | 21.85     | 242.4      | 72.2                                |           |\n| $\\textbf{DeltaDEQ τ=0.01}$    | $\\textbf{22.24}$      | 115.3 (-52%)                        | 15.6 (-55%)| | $\\textbf{21.75}$     | 115.5 (-52%)| 20.5 (-72%)                         |           |\n| DeltaDEQ τ=0.05    | 23.28      | 80.3 (-67%)                         | 13.7 (-60%)| | 22.66     | 81.6 (-66%) | 16.8 (-77%)                         |           |\n\nR-Tab 3: Results on the Wikitext-103-v1 dataset. DeltaDEQ refers to DeltaDEQ-Transformer and $\\tau$ is the inference delta threshold.\n\nFor the optical flow task in Tab 2 of our work, we provided a comparison with several impactful works. Because the work focuses on studying an acceleration method for DEQ and potentially other iterative methods, our main comparisons are with the original DEQ models.\n\n$\\textbf{W3: }$ Indeed in this work we mainly analyse the theoretical maximum FLOPs reduction for the forward pass of DeltaDEQ. We leave the hardware implementation and actual runtime of our method to future works and focus on studying the potential of this method at the algorithmic level. We would like to mention that for the RNN-based DeltaDEQ, the sparsity in the delta activation is structured and therefore,  can be implemented with the sparse-matrix-vector multiplication libraries on GPU/CPU platforms. For the CNN-based DeltaDEQ, there also exists tailored implementations [C3] on general computing platforms. Another direction of exploiting the delta activation sparsity is by using special hardware platforms (FPGA/ASICS) [C1-2]\n\nOne thing worth pointing out is although the delta sparsity differs from activation sparsity algorithmically (the former can have non-zero activation values even with zero values for the delta sparsity, and the latter has zero activation values for sparsity), our method can still be implemented on these accelerators designed for activation sparsity. \n\n$\\textbf{W4: }$ Replied in W1\n\n$\\textbf{Q1: }$ We now provide new results on a language modeling task with our DeltaDEQ-Transformer architecture. Please refer to $\\textbf{W2}$ and $\\textbf{G2}$ in the general rebuttal reply. \nApart from the new results, Tab. 2 in our paper demonstrates the generalization capability of DeltaDEQ to a new task. The models were pretrained on the FlyingChair and FlyingThings3D datasets and tested on the Sintel and KITTI, that also tests their zero-shot generalization ability. \n\n$\\textbf{Q2: }$For both implicit neural representation and optical flow tasks we performed an inference delta threshold sweep (Fig. 4 and Tab. 2). Indeed different tasks can have different optimal delta thresholds. We leave automatic learning of the delta threshold $\\tau$ to future works.\n\n$\\textbf{Q3: }$ Please refer to the reply for W3.\n\n$\\textbf{Q4: }$ In Fig 4. and Fig. 7, we provided the standard deviation band in the plot. \n\n$\\textbf{L1: }$ Replied in W2 with new results on language modeling with a new architecture type which was not in the paper. In the paper, we have also provided results on 2 tasks with 3 datasets and 2 very different base architectures (DEQ-CNN and DEQ-RNN).\n\n$\\textbf{L2: }$ Our work mainly focuses on accelerating the inference/reasoning of DEQ to address one of the major disadvantages of DEQs and other iterative methods, that is, the expensive computation cost of the DEQ algorithm. \n\n$\\textbf{L3: }$ We are glad to release the code upon acceptance. In the submission, we forgot to include a footnote to mention this code release. For the moment, please refer to the formulations and Pseudo-code in Algorithm 1. All datasets used in the work are publicly available. \n\n$\\textbf{L4: }$ Answered under reply to W2 and G2.  \n\n\n[C1]NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps, Aimar et al.\n\n[C2]EIE: Efficient Inference Engine on Compressed Deep Neural Network, Han et al.\n\n[C3]Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference, Kurtz et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their time in reviewing our work!

We first clarify the common concerns among some of the reviewers:

$\textbf{G1: }$In this work, we present the heterogeneous convergence phenomenon, which is a finding that shows different dimensions of the hidden states converging at different speeds during fixed-point iterations. Following that finding, we exploit the $\textbf{delta activation sparsity}$ to accelerate the $\textbf{forward pass}$ of DEQs and other iterative methods that have high inference computation costs. Our method is effective on both static (e.g. images, texts) or temporally correlated (e.g. videos) inputs.

$\textbf{G2: }$We now provide results from a new $\textbf{language modeling task}$ on the WikiText-103-v1 dataset using a new architecture, the DEQ-Transformer architecture. We applied the delta rule on the QKV matrices construction and the feed-forward layers after the self-attention module. The DEQ block has 3 transformer layers and 12 fixed-point iterations are carried out during inference on each example. The token dimension is 700. Our architecture design follows the settings of the deq-lm model in the public library[1]. For the formulation of the delta rule as applied to the Transformer architecture, please refer to Reply W2 for the first Reviewer ‘dpig’. 

The results are presented in the following table (R-Tab. 1). By applying the delta rule on the DEQ-Transformer using a delta threshold $\tau=0.01$, the test perplexity (PPL, lower the better) improved slightly, while the FLOPs on both parts are greatly reduced (52% and 72% reduction on memory length (mem_len)=480 settings). Increasing $\tau=0.05$ led to a further reduction in FLOPs while maintaining a similar test PPL. We will add these results to a future version of this work along with a more detailed analysis.

| Model              |             mem_len=150, | seq_len=150.           |            | |                    mem_len=480, |seq_len=150.  |        |           |
|--------------------|------------|-------------------------------------|------------|--------|-----------|------------|-------------------------------------|-----------|
|                    | Test PPL   | FLOPs FFN                           | FLOPs QKV  | | Test PPL  | FLOPs FFN  | FLOPs QKV                           |           |
| DEQ-Transformer    | 22.34      | 242.4                               | 34.4       | | 21.85     | 242.4      | 72.2                                |           |
| $\textbf{DeltaDEQ τ=0.01}$    | $\textbf{22.24}$      | 115.3 (-52%)                        | 15.6 (-55%)| | $\textbf{21.75}$     | 115.5 (-52%)| 20.5 (-72%)                         |           |
| DeltaDEQ τ=0.05    | 23.28      | 80.3 (-67%)                         | 13.7 (-60%)| | 22.66     | 81.6 (-66%) | 16.8 (-77%)                         |           |
| DeltaDEQ τ=0.1     | 30.54      | 49.3 (-80%)                         | 9.3 (-73%) | | 29.15     | 50.9 (-79%) | 12.5 (-83%)                         |           |

R-Tab.1.: DeltaDEQ in the table stands for DeltaDEQ-Transformer and $\tau$ is the inference delta threshold. The FLOPs are measured in Giga-FLOPs. For comparison, we provide task performance from others models with a similar number  of parameters (DEQ-Transformer and DeltaDEQ-Transformer have 98M parameters) : 
(a)Transformer-XL Standard (151M) [2],  Test PPL = 24.0  (b) Feedback Transformer 8 layers (139M) [3],  Test PPL = 18.2 (c) Feedback Transformer 4 layer (44M) [3], Test PPL = 22.4 (d) GPT-2-Large (774M) [4], Test PPL = 22.05. 

$\textbf{G3: }$We have conducted a hyperparameter sweep for the $\textbf{inference}$ delta threshold, $\tau$  for both implicit neural representation (INR) and optical flow (OF) tasks, and the results are shown in Fig 4 and Tab 2. Additionally, we include such results in the new R. Tab. 1 above.

[1]TorchDEQ: A Library for Deep Equilibrium Models, Geng et al.

[2]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, Dai et al.

[3]Addressing Some Limitations of Transformers with Feedback Memory, Fan et al.

[4]Language Models are Unsupervised Multitask Learners, Radford et al.

Meta-review: The paper develops a method to save computation within Deep Equilibrium Models (DEQs) by dynamically detecting elements of the state representation that converge faster and caching their values to avoid recomputation. After the rebuttal and discussion, three of four reviewers favor accept. The AC agrees with these three reviewers and believes the paper offers interesting insight into the behavior of DEQs.

In discussion, Reviewer zWM7, who gave a reject rating, indicated this was based on: (1) that "pixel-level dynamic sparsity is challenging to translate into theoretical speedup" and (2) not being "convinced that the current manuscript offers any novel insights into pruning." The AC does not believe either of these are appropriate criteria for rejecting the work: (1) This paper is not about hardware design, and unstructured sparsity is a valid research topic; (2) The rebuttal in response to Reviewer zWM7 addresses the question of distinction with respect to related work on dynamic pruning.
Decision: Accept (poster)