Flow Equivariant Recurrent Neural Networks
Authors: T. Anderson Keller
Venue: NeurIPS 2025 spotlight ()
Keywords: Equivariance
Recurrence
Sequence Model
Flow
Motion
Primary area: deep_learning
TL;DR: We introduce sequence models that are equivariant to time-parameterized symmetries such as motion.

Abstract: Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.

Flow Equivariant Recurrent Neural Networks
T. Anderson Keller The Kempner Instutite for the Study of Natural and Artificial Intelligence Harvard University, Cambridge, MA 15213 andersoo. keller@gmail. com
Abstract
Data arrives at senses continuous stream, smoothly transforming from instant to the next. These smooth transformations can be viewed as continuous sym- metries of the environment that inhabit, defining equivalence relations between stimuli over time. machine learning. neural network architectures that respect symmetries of their data called equivariant and have provable benefits in terms generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding mamamearamttriized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' one- ane-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail transform stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-ece aoneeiivvriant counterparts terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. towards building sequence models that respect the time time-parameterized symmetries which govern the world around us.
geometrically structured manner for moving
present this work as first step
Introduction
Every moment, the world around us shifts continuously. As you walk down the street, rooftops glide past, trees sway in the breeze, and cars drift in and out of view. These smooth, structured transformations flows not arbitrary, but instead underpinned by precise of rules: an algebra which ensures self-consistency across space and time. Mathematically. these transformations which leave the essence of the world around you unchanged are called symmetries, and, in sense, these symmetries over time can be seen to weave together the instantaneous into the continuous.
In machine learning. symmetry has emerged as powerful handle from which we can grapple with the challenges generalization and data efficiency. Models which encode symmetries are called 'equivariant', and their steady evolution has delivered significant performance improvements for data with known structure. Simultaneously, with the success of large language models, sequence modeling has become prominent learning paradigm throughout machine learning sub-fields. We therefore ask: can these everyday smooth, continuous motion symmetries be encoded in our modern sequence models and leveraged for improved performance? Surprisingly, to date, the study of equivariance has largely been limited to temporally static transformations, narrowly missing the opportunity to capture the continuous time-parane pamamearamtterized symmetry transformations that dominate our natural experience (see Figure T). In this work, we consider a carefully structured subset of natural motion transformations called ñlww: formally, one-parameter Lie subgroups. We introduce formal notion of what means for sequence model to be 'flow equivariant' and proceed to demonstrate that existing sequence models are indeed not flow equivariant, even when all sub-components of these models are individually equivariant with respect to the full 'static' Lie group symmetry. 39th Conference on Ncural Information Processing Systems (NeurIPS 2025).

---

Static Symmetry
Motivated by this gap in the literature, and the potential advantages of integrating a ubiquitous natural symmetry into sequence models, in this work, we study how to build Recurrent Neural Networks (RNNs) that are explicitly equivariant with respect to pime-parameterized symmetry transformations. At the highest level, we demonstrate that we are able to do so in a manner very similar to Group equivariant Convolutional Neural Networks (G-CNNs): by lifting the hidden state of our RNN to a dimension of flows, indexed by the generating vector fields v. In slices, our model resembles a bank of RNNs (one for each flow), but weight sharing across V turns that bank into a single group -ouiieqnuiariintt dynamical system which guarantees generalization to unseen flows.
f 0
2
9F
9t
g+
Flow Symmetry
t 0 t
2
In the following, we will review the principle of equivariance in neural network architectures ($2), introduce the notion of 'Flow Equivariance" ($3). and demonstrate that existing RNNs are indeed not 'flow equivariant' as we would desire. We will then introduce Flow Equivariant Recurrent Neural Networks (FERNNs) in $ and demonstrate empirically that FERNNS achieve zero-shot generalization to new flows at test time, improved length generalization, and improved performance on datasets which possess strong flow symmetries in $5 In our discussion, ($ we briefly outline related work on equivariance with respect to motion, and highlight the differences with proposed approach, but leave thorough review of related work to $D We conclude with the limitations of our proposed framework and promising future directions. We provide an accompanying blog pos with additional visualizations.
1/
M/2
Figure 1: Static vs. Flow Symmetries. Static sym- metries are constant while flows progress over time.
2 Equivariant Neural Networks
Informally, a network is said to be equivariant with respect to set of structured transformations every transformation of the input has a corresponding structured transformation of the output. The fact that this set of transformations is 'structured' is important. Typically, this structure is the abstract algebraic structure known as a 'group' G. meaning that any two transformations 9i- 9j E G will combine through binary operation (-) to form another known transformation gi 9j - 9k E G (closure), and the 'chunking' of transformations is arbitrary, i.e. (g: : (i) 9h = 9i [g) gs) (associativity). As we will see later, we will exploit both these properties when designing equivariant models. At an intuitive level. an equivariant neural network 4 is network whose activations 'live on the group'. This means that when processes a transformed input, by associativity, this is equivalent to transforming the activations, and by closure, this transformation just yields another set of activations on the group- - i.e., a structured transformation of the output.
Formally, a neural network layer Tb is equivariant with respect to symmetry group G if the action of the group commutes with the application of the layer. Following prior work (Cohen and Welling) 2016a)), we will define this layer as a map between function spaces $ o (X) + FK.(Y), where Fx(X)= [ʃ X > R&] denotes the set of signals defined on the abstract domain X with K channels. We will use the notation {[[) to denote the output of the model (in Fxx(Y) space) for an
input signal f. and (v) R to similarly denote the value of that output function at location y. For f E Fx(X). we can then define the action of the group element g E G to be the left action:
(g. (x)) = : x),,
G, x x X.
Concretely, if is 2D image, x translation group, then the translation action is: (g f))) = f(x - g). a leftward shift of pixel coordinates. In the output domain, the action of g depends on how construct the equivariant map; but as we will show next, for group equivariant convolutional layers, this ends up being the same as it as for input, i.e. (g +f(1)(v) ( v)) Finally, the equivariance condition is then written as:
2 the set of pixel coordinates, and G= = (Z?.+)
is the 2D
g f 2gg fl VgeG, fe (X)
We can see this is the formalization of our intuitive definition: the action of the group on the input f can instead be pulled through to act on the lifted group indices of the network's output $ff].

---

Group Convolution. A simple example of such an equivariant map O is the group convolution [Cohen and Welling 201 a generalization of the standard convolutional neural network (CNN) layer from the translation group to other arbitrary discrete groups. To accomplish this, we define kernels to be functions on the group G, i.e. W G -> RK and perform "lifting" with kernels U'' : X + T&K at the first layer of the network to 'lift' from the input space to the group space. Mathematically, we write the lifting convolution (*c) and subsequent group convolutions (tc) as:
K If & (()) - Σ Σ 1 xE22
x)) & IG (ooll
Σ E (o NEG
' .h) (3)
Where and k are the output and input channel indices of the convolution respectively. We see that the output of the layer is now defined over the group G, and we can verify that these layers are indeed equivariant with respect to Equation 2 (see $A.J). The important property being that [[g f) * w*] = g- [f c W"]. that is, the convolution commutes with the group action.
Group Equivariant Recurrent Neural Networks. In this work we define recurrent neural network h as a map between space-time function spaces h : Fx(X, Z) - FY, Z), where Fx == {f: : X x Z RK denotes the set of time-varying K -ddimensional signals defined over space X and (discretized) time t E Z. In the remainder of this paper we will use discrete time domains for our space-time signals, but we note that in order to accommodate continuous time sequence models, time may be made continuous (f E IR) by changing domain of h, with only minor changes to the analysis. We can then define a group-equivarian RNN (G-RNN) as simple RNN with group convolutions in place of both the usual linear recurrence and input maps. Explicitly:
h4 If = o (helfeel ko W + fe &g 4²),
where W : :G G RKK and 4i : - R are the group convolutional kernels for the hidden- to-hidden mapping and input-to-hidden mapping respectively (for output channel i), and ho is initialized to be invariant to the group action, i.e. ho(g) = ho(g') Vg.g' E G. The notation [e] implies that ht is a causal function of the input signal f prior to time f. The non-linearity $A.2 we prove by induction that this network is indeed equivariant with respect to action of the group G on the input signal f. meaning hilg fzi] = g h] Vg E G,t E Z. E Fx Z). Intuitively, this is because both the input- to-hidder mapping and the recurrent mapping are equivariant, meaning any constant transformation of the input can be pulled through both mappings (and the nonlinearity) yielding a corresponding transformation of the output (hidden state). Concretely, this implies that a recurrent neural network with convolutional recurrent connections, and convolutional encoders is indeed still equivariant with respect static translation of the entire input sequence. However, as we will show next, this does not imply that the RNN will be equivariant with respect to time-parameterizer translation, i.e. motion.
a is assumed group-equivariant as well (pointwise).
3 Flow Equivariance
Like the pathline traced out by droplet of dye carried along by moving fluid, the flow (v)
generated by vector field y gives the position of any point after time / E R. More formally, a flow
a one-parameter subgroup of Lie group G. generated by an element v of the corresponding Lie algebra of G Hall 2015). We denote a generator by V E g. and its time-t flow (v):: R x g + G by (v) = exp(tv) E G. where exp is the exponential map. By construction, the flow admits an identity at t = O, (v) = e, and a composition property: (b())) - (v( = (e+)(v), such that flowing for time s and then t is the same as flowing for time + t. Intuitively, the flow th (v) is the group element that transports a point from the identity to where it will be after time when moving with instantaneous velocity v. Colloquially, we therefore call e transformation. For given f E Fx (X, Z) and generator // E g, we define a flow acting on f as
mane-parameterized symmetry
(b(v)) (G())-)ec Me(x)) = fe(0e(v)-1.1)) 2))
(5)
and identically for a flow acting on function in the output space FK,(Y, (V)V) +(f)(())
=
(.[]]( y). Throughout this work we will use the notation (1)()) (e(v).f) f] and (18(2) )) leaving time implicit, to denote the flow acting on the entire time domain of the function simultaneously.
We can then define 'flow equivariance" as the property of a sequence model such that its sequential output commutes with such time- paramettttee transformations of the input:
3

---

Definition 3.1 (Flow Equivariance). For set of generators V S a sequence model +b Fx((, Fr (Y, is equivariant with respect to the flows generated by V iff
(b()) + [ -V)(()) f) VVEV, f E Fx(
(6)
Flow Equivariance of Sequence Models. Our proposed Definition 3.1 has noteworthy special case which captures much of the literature to date on equivariance in sequence models. Specifically: Remark Ftrame-wise feed-forward equivariant models are trivially flow equivariant.). Let $ (o(fo), (ff).... (fT)] be a frame-wise applied equivariant map, with é Fx (X) + Fxx such that: (V) fi) ve(v) o(f)) Vt E Z, E V. Then & is flow equivariant by Definition 3.1
See A.3 for the short proof. Intuitively, one sees this very clearly for CNNs applied to video sequences frame-wise: since the CNN commutes with the transformation of each frame, and the transformation on the sequence distributes amongst the frames (by linearity of the action), the act of transforming the input frame-by- -rammeee-rreeeeee is equivalent to transforming the output frame-by-frame. Since this is achieved through complete lack of sequence modeling in D we call it 'trivial flow equivariance'. Equivalently, one can see that the G-RNN defined in Equation [ with W = 0 reduces to such feed-forward equivariant map (since recurrence is removed), and is thus also trivially flow equivariant. To achieve more sophisticated flow equivariance in the context of sequence processing, must use sequence models with non-zero recurrence. Notably, while in 2 we show that group- equivariant RNNs do commute with static transformations, we show below that such models generally do not commute with time- parameeeeeete transformations, meaning they not flow equivariant.
Theorem 3.1 (G-RNNs are not flow equivariant. A G-RNN as defined in Equation 4 with non-zero W. not flow equivariant according to Definition 3.1 except in the degenerate flow-invariant case.
Proof Sketch. (Theorem 3.1 As visualized in Figure 2 let a = Id, and [* &g U1, [* *G W] be the identity, such that the hidden state evolves simply as sum of the past state and the new input: h++1 [ = he]fce] + Jr. If the input is a static bump centered at O, het1[se]] will be an ever- growing bump, while 14 [bb()) Sse] will be 'train' of bumps, since, each time step, the hidden state is 'lagging behind' the input which has shifted by th1 (v). Thus a corresponding shift of the
growing bump will never equal the train of bumps, i.c. (v(v)- h[f=el)e+1 & het1(b(v) feel- O
Remark. (Degenerate flow invariance) If kernels W & U/ are constant on G, then the outputs of Equation are spatially uniform. Since the flow action permutes only the G-index, these constants fixed points, making the G-RNN flow invariant.
One can verify that this counterexam- ple applies to most other existing se- quence models, including state space models, gated recurrent neural net- works, neural ODEs, and even trans- formers. In the following section, show how to build recurrent neural network which is equivariant with respect to these transformations, and argue that this same construction can likely be carried over other se- quence models as well.
AVI
Pir AM
m
M
Figure 2: G-RNNs are not generally flow equivariant. We show this by simple counterexample with: hef1 h4 + ft- See AA.4 for the full proof.
4 Flow Equivariant Recurrent Neural Networks
In this section we introduce our Flow Equivariant Recurrent Neural Network (FERNN) and prove the equivariance condition (Definition 3.1 holds. Inspired by the above analysis of the failure of G-RNNs to be flow equivariant, we note that a simple condition by which equivariance can be achieved is if computation is performed 'in the reference frame' of the signal. For the case of static symmetry groups, we can interpret the group convolution as performing the same operation (dot product of filter and signal) in all possible statically transformed reference frames (group elements). In the case of time-pare mamapparaeterizzed symmetry groups, we can then assume that we desire to be performing the same recurrent update in all possible dynamically transformed (moving) reference frames. To achieve this,

---

we see that, analogous to the g-lift in the G-CNN, we must now lift our hidden state and input to the flow elements y E V. To do so, we define the action of the instantaneous group elements (v) E G on the lifted V x G space to be the same left action as before, only on the G dimension. Explicitly, for (v, V G + RK", we define (v) (, g) == he(u, (e(v)-1.g)) 9))
Input Flow-Lifting Convolution. To lift the input the product group V G, will re-use the original lifting group convolution from Equation exactly, and simply perform a 'trivial lift' to the additional V dimension by copying the output of the convolution for each value of Explicitly;
K
If Xv u (v,9)
ΣΣ fe(z))((g0
mEX k=1
In |A.5 we prove that this is equivariant with respect to the group action, yielding the following:
(v) [ft &v (v) = (())(v) fi) xv (v. := [fa k (V) (1))))1- g) (8) We note that it is also possible perform non-trivial lift, where 24 is defined on the lifted space V x G, as in G-CNNs. This changes the later recurrence relation (Fqn. [1] necessary to achieve flow equivariance, but is largely a design choice. In the experiments and proofs we stick with the trivial lift, but describe the model with non-trivial lift in C
Flow Convolution. The group convolution can then be extended to our lifted hidden state as:
K' hu m) wi(T - vg v..- -1. -m).
[A X w (v,9)
ΣΣ
TEV m=Gk=1
We see that for the convolution over V, we have assumed that V C is an additive (abelian) discrete group, such that the combination (-v) = 7 - v. If V is not abelian, the general Baker- -ampbel- Hausdorff product ((-v) 77) should be used instead (Hall 2015). Similarly, if V is not discrete, one should perform an integral with respect to the Haar measure. In all experiments we use abelian V, and discrete grid for V, so this is exact. In $A.6 we prove this is equivariant with respect to the action of individual flow elements. Explicitly:
[())(()) h) *vx w (v,g)= (v))) [4 xxx (")
(10)
Intuitively, this is simply because the individual group elements along the flow only act on the G coordinate of the signal, and thereby commute with the standard group convolution.
Flow Equivariant Recurrence Relation. We define the recurrence relation for the FERNN as:
1 g) = (v) [he xvx W] (v, + [ft kv U] (v,
(1I)
We see that there are two primary differences between this model and the G-RNN: the extra dimension v, and the action of the instantaneous one-step flow element th (v) at each position of the lifted hidden state. Intuitively, this can be understood as a bank of G-RNNs, each flowing autonomously according to their corresponding vector fields v simultaneously. Similar to G-CNNs then, when an input arrives undergoing a given flow transformation with generator 2, this construction allows us to 'pull out' this transformation into shift of the y dimension. Formally:
Theorem 4.1. (FERNNS are flow equivariant) Let h[[]] E FY, Z) be a FERNN as defined in Equations 7 9 II] with hidden-state initialization invariant to the group action and constant in the flow dimension, i.e. hn g) ho .g) V v E V and in (v) ho g) hn g) V u E
V, g E G. Then, h [/]] is flow equivariant according to Definition 3.1 with the following re-defined representation of the action of the flow in the output space:
(w(D) - -O0)) g) h. b, th-n (D)-1
(12)
In words, the equivariance condition hr (b)())) (((0))]((,))) h. ( b, s/. (0) g) means that the hidden state of the FERNN processing flowing input sequence has a corresponding flow along the dimension, and a shift along the V dimension, both determined by the input flow generator In $A.7 we prove Theorem 4. by induction, and give brief sketch below.
5

---

Proof Sketch. (Theorem 4.1 We wish to prove hy (v)) (v, = hylfl D, 8 (0) .9) Vt. Letting H- V G, we write the recurrence relation for the transformed imput as:
h441 (0) [[(,,g) = a (v) (M)(v)0) fea] *H W] (u,g) + [(db(()) ft) RH 2] (v. g)) (13)
Assume we initialize the hidden state fixed point of the flow-element action 1 (v) hg (o(vg)
ho(u,g), constant in v. We can see that the base case is satisfied trivially since Φ (v) (v,g) =
"-1(1) hu(P 0, g) by constancy. We then assert the inductive hypothesis, allowing us to sub- stitute he(xb())- of (v.g) + ha[f](v- D. ( g) = = helf(( -  g) into equation Similarly, due the trivial lift and the equivariance of lifting convolution, we can substitute (v) fi) xv UI (v,g) 9) + (D) Iff &vxc U] (v 1, g. Ultimately, this gives us:
13
13
o (vh(1)) (2) [he]fed KH W| - 2. + (2) [fe RH 24] (v p.g)).
(14)
By the properties of flows, we know that 1/1 (V). (0) = 1/1 (v) (D))s) (B) = the (e)))) (v- and thus we can pull out a factor of (D) from both terms, leaving us with the original recurrence defined over a new V index
8. exactly the shift in V.
Intuitively, the idea is that we would like to build a sequence model which fixes the lack of equivariance demonstrated by the counterexample in Figure 2 To accomplish this, we note that the hidden state of the model in Figure 2 appears to be lagging behind the input at each time step. To fix this, we therefore propose to augment our hidden state with bank of flows, each generated by a separate vector field // such that when the input arrives at a specific velocity, it will exactly match one of these flow parameters, effectively shifting the 'zero' point of the moving coordinates of our RNN to this new flow. Since these flows are relative quantities (obeying the group law), we see that all the other v channels in the bank will similarly shift according to their relative position in V space (e.g. relative velocity). These effects combined lead to the output representation we observe analytically. In Figure 9 we visualize the model and the same example of a moving bump from Figure 2 providing a clear demonstration of what the action of the flow on the hidden state looks like in practice.
We finally note that since the hidden state lives on the product group V G. and the input similarly lifted to the product group, one can simply stack these layers, replacing the lifting convolution with another flow convolution and maintain flow equivariance. Since a composition of equivariant maps is itself an equivariant map [Kondor and Trivedi 2018). a deep FERNN is also flow equivariant (provided all non- non-linearities, normalization layers, and other additional architectural components are additionally flow equivariant- - we refer readers to [Bronstein et al.) 2021 for an extensive review).
Practical Implementation. We note that exact equivariance requires that the group V be closed under the action of the flow on the output. In theory, this can be satisfied by many constructions, but in practice, we cannot lift our hidden state to an infinite dimensional group. Therefore, similar to prior work [Worrall and Welling 2019). we implement a truncated version of the group V, and incur equivariance error at the boundaries. Furthermore, in the majority of experiments in this work, we define the flow convolution to not mix the V channels at all, explicitly: Wi(,.9) 9) = 5x=eW(() We find that this beneficial to not propagate errors between the v channels induced by the truncation, similar to prior work on scale equivariance (Sosnovik et al. 2020b). In Appendix $E.5 we demonstrate cases where including V V-mixing helps modeling inter-velocity interactions (e.g. object collisions).
Experiments
In the following, we introduce datasets with known flow symmetries and study how adding flow equivariance impacts performance compared with non-equivariant baselines. We investigate two sequence datasets: (i) next-step prediction on modified 'Flowing MNIST dataset with 2 simultane- ous digits undergoing imposed translation and rotation flows [LeCun et al. 1998]. and (ii) sequence classification on the KTH human action recognition dataset [Shhllt al.) 2004]. augmented with additional translation flows to simulate camera motion. In SB we include the full details of the dataset creation, model architectures, training, and evaluation procedures; and in we include extended results. Code is available at:
com/ akandykeller//Faa
Flowing MNIST Datasets. We construct number of variants of the Flowing MNIST dataset to test generalization, each defined by the set of generators we use for training and evaluation (Verain- Viest). In words, sequences are composed of two digits moving with separate random translation rotation

---

Translating MNIST (V.)
Rotating MNIST (VC)
0.08
0.06
Model
Test MSE
0.07
G-RNN
F
0.05
FERNN FERNN-V2
G-RNN
Verain
Viest
V27
Valdenten Loss
FERNN-VI
G-RNN
8.1e Ge-4
Validzzion
FERNN-V7
5.3e + 85-5
0.04
0.02
FERNN-V
1.5e 4 + 20
0.03
0.02
Verain = Viest = V2
G-RNN
4.0c 3 + 5e-4
FERNN-V2
1.3e 3 1 56-5
Taining Step (Batches)
Training Steps (Ratches)
FERNN-V?
6.1e-4 4 Se
Figure 3: Increased flow equivariance increases training speed on data with flow symmetry. Validation loss vs. train steps.
Table 1: Flowing MNIST Test
MSE. (mean = std. 5 seeds)
velocities, For rotation flows, we discretize the rotation algebra s2 at A0 = 10 intervals. yielding: V [k:0JJk = -N.... N]. where J = (9 o E s2 For 2D translation flows, we always use the integer lattice up to velocities N Niizeess VX = {u E z²| || S N.. See Figures 4 & 5 for examples. A train/test example from the dataset is sequence St constructed by drawing two random static samples f² from the original MNIST dataset and two corresponding random vector fields f², D22 uniformly from the corresponding Va, applying the action of the corresponding flows to each sample, and summing the results to generate the timeseries: ft = (B*) f + the (D²) f2.
Next-Step Prediction Task. Models are trained to solve an autoregressive forward prediction task: given the first 7 = 10 time-steps of the timeseries ʃ. predict the next 10 time-steps. Model parameters are optimized minimize the mean- grean-suarre error (MSE) between predictions and the ground truth for 50 epochs. In the length generalization experiments of Figure 4 we evaluate the model's MSE on forward prediction up to 70 steps, significantly longer than the IO seen in training.
Next-Step Prediction Models. We use G-RNNs and FERNNs exactly as defined in Equations F & I1 respectively where G is defined as S0(2) with translations (S (2)) for the rotation flow datasets, and G == (Z²,, , +) is the standard 2D translation group for the translation flow datasets. We use the escnn library to implement the (2) convolutions [Cesa et al. 2022]. We set a = ReLU, use a single recurrent layer, hidden state size K' of 128 channels, and (3 3) convolutional kernels in the spatial dimensions. We note that the spatial dimensions of the input are preserved throughout the entire forward pass, i.e. there is spatial pooling, stride, or linear layers. We augment these models with small 4-layer CNN decoder go (1++1) with 128 channels, and ReLU activations. which maps from each hidden state to the corresponding output. Explicitly: go go(h+1)) = fe+1. where the training loss is the MSE between f(+1 and fi+1 averaged over time t 20). For the FERNN models, we also crucially 'max-pool' over the V dimension of the hidden state before decoding, e.g. ge (max, h (v, g)) = fr+1. This means that all G-RNN and FERNN models have exactly the same number of parameters. since the FERNN can be seen to share its parameters over the different flow elements. For each FERNN, we denote its corresponding set of generators with FERNN-V N.
Next-Step Performance. In Figure g we show the validation loss curves for these models on Flowing MNIST, and in Table we show the final MSE on the test sets (mean = std. over 5 random initializationss. We see that on both datasets, FERNNS dramatically outperform G-RNNs, reaching nearly an order of magnitude lower error, and doing so in significantly fewer iterations. We also see that even partial equivariance (e.g. FERNN-V," on VgT) ) is highly beneficial to model performance.
Length Generalization In Figure 4 we plot generated forward predictions on Translation-Fin MNIST with Virain = Viest = Vi2 for both the G-RNN and FERNN-V;" models compared with the ground truth sequence. The length of sequences seen in training indicated by the shaded gray area, and at test time we autoregressively generate sequences from the models which are significantly longer than this. We see that the G-RNN predictions begin to degrade after the training length, while the FERNN remains highly consistent. We quantify this error for each forward prediction step the adjacent plot, demonstrating that indeed FERNNS dramatically outperform G-RNNs in length generalization. We note that the performance improvement of FERNNs over G-RNNs is more significant for translation flows compared with rotation flows in our experiments. Particularly, we do not see quite as strong length generalization on Rotating MNIST (see EE; however, we speculate
7

---

Noat- Step Prodiction Longth Goneroli Dontroliatioon
Emoo vs Forword Prodiction Horior
FERNN-9
G-RNN
Timestep
14
16
22
26
30
34
30
42
45 50
54
66 70 74 70
Timestep!
capabilities far surpassing Figure 4: FERNNS exhibit next-step prediction l
G-RNNs simple flows. We plot samples for the forward prediction trajectories of a G-RNN and FERNN-V; V/2" trained on Translating MNIST V22 to predict 10-steps into the future (down-sampled by factor of 4 in time for visualization). We the G-RNN performs well on this training regime but diverges rapidly with lengths longer than training. The FERNN generalizes nearly perfectly. On the right, we plot this forward prediction error vs. the forward prediction timestep for both models.
that this is due to the interpolation procedure necessary to implement small rotations on a finite grid, not due to any difference in the underlying theory. Interpolation of rotating pixel grids is known to break exact equivariance, and we suggest prior methods for addressing this issue in G-CNNs may be equally applicable to FERNNS DDaconu and Worrall 2019].
Velocity Generalization Similar to standard group equivariant neural networks, weight sharing induces automatic generalization to previously unseen group elements. In the case of flow equivariance, we then expect zero-shot generalization to flow generators that are part of the lifted hidden state's V but not seen during training. In Figure [S] we show example sequences which demonstrate this to be the case. Specifically, the models are trained on low velocity examples (V²" and Vidt), and evaluated on the full set of higher velocity generators (from Vi and V respectively). In the adjacent plots, we show the test MSE for sequences flowing according to each generator. We see that the FERNNS (middle) generalize nearly perfectly, while the G-RNNs (bottom) only learn to forward predict in-domain flows. and fail to generalize outside. Furthermore, in Figure 6 evaluate the performance of FERNNs trained and tested on velocities outside of their integrated V set. We see that while eπrог increases slightly, FERNNs can generally interpolate between integrated velocity values to yield significantly improved performance on a wider range of data.
Translating MNIST Velacity Generalization (-2, 2)
2D-Trandarion Test Varecity FFBNey-12
Trasnop Surgo
G-RNN
Cround Truth
FERNN-
10
1.5
0.3
0.09
G-RNN
0,00
Timestep
Rotating MNIST Angular Velocity Generalizetton (-40 20
x10
Error vs. Ang for velecity
Ground Truth
A
9
9
6
6
6
Formard Preditinn
FE
9
9
6
6
6
Training Range
G-RNN
Timestep
11
13
14
15
18
17
18
19
20
55
-40
300
Rotation Angular Velocity
220
-20
0
10
20
30
40 50
Figure 5: FERNNS exhibit next-step prediction generalization to previously unseen translation and rotation flow velocities where G-RNNs fail. Samples of forward predictions for FERNNS (V- & V2( trained on Translation & Rotating MNIST with limited flow velocities (VT"' Vi2t and tested on -orward-prediction prediction performance within their V set. On the right we plot the full distribution of errors across Viest-
a wider range of velocities (V. & V2 We see the FERNNs achieve near perfect forward
(st MSE
1.0
x10
G-RNN
FERNN V(T
FERNN 17 4AA4-1
FERNN V/T.
0.5
0.0
Test Velocity (pixels/step)
Figure 6: FERNNS can interpolate between V E V. Test MSE for FERNNS trained and tested on gr-translation velocity flows from -5 to +5, with varying V-set coverage (see legend). Hatched bars indicate the test velocity is outside the model's equivariant set.
8

---

VT
A V
Figure Moving KTH Dataset Samples. Translation flow augmentation simulate camera motion.
Moving KTH Action Recognition. Finally, we test our FERNN model's ability to classify natural video se- quences with induced motion using the simple task of
Model
Test Acc. f
0
3D-CNN
0.626 + 0.02
209K
Action Recognition from the KTH dataset. The dataset composed of videos of 25 people performing differ- actions. We use the traditional train-vall val-test split, by person, requiring strong generalization to new individuals. We down-sample each clip by factor of 2 in space and time for computational convenience, leading to grayscale sequences f of size 32x32, with 16 time-steps each. In our experiments, we apply flows from V," and V2" to the dataset an attempt to emulate slow camera motion (see Figure 7) On this dataset, differently from MNIST, we do not re-sample velocities per example, leading realistic restricted dataset. In Appendix 6 we include results with this velocity data augmentation.
G-RNN+
0.639 + 0.02
242K
G-RNN
0.665 0.03
242K
FERNN-V
0.698 + 0.03
242K
FERNN-V
0.716 0.04
242K
Table 2: FERNNS outperform non- equivariant models on sequence clas- sification in the presence of strong tion. Test accuracy (mean = std) for models trained and tested the Moving KTH dataset (V²) )
Vinst x."
1/ (Nf Minie)))
16'
30-CNN
Action Recognition Models. In Figure g and Table 2 compare the accuracy of FERNN models with non-flow-equivariint counterparts (G-RNN, G-RNN+, 3D-CNN), each with comparable parameter counts. The G-RNN and FERNN-V models are identical to those used on the Translating MNIST dataset with the addition of a 3-Layer CNN encoder (in place of the input convolution 24), and class-readout module composed of over the spatial dimensions followed
G-NMP BRANKi1
Accuracy
PSRAN-22
0.4
0.2
Verain
an average-poolle by linear layer. The G-RNN+ model an ablation of the FERNN-V; which has the same lifted hidden state and max-ppolling over the lifted dimensions, but instead of applying the transformation eh (v) to each hidden index v. the G-RNN+ learns its own transformation through small convolutional kernel (see !B. The inclusion of this model ensures that the benefits that we see from the FERNN not simply due having increased hidden state size and an extra max- max-pooling non-linearity instead they must be coming from the flow the hidden state itself. The 3D-CNN is a 5-layer network composed of 3D convolutions that performs striding and pooling eventually yield a 64-dimensional feature vector which fed linear classification head.
Accuracy
0.6
Figure 8: FERNNS outperform non- equivariant models action recog- nition undergoing out-of-distribution flows (orange). Test accuracy of models trained on KTH with no motion V/8" (top row) and with motion V (bottom row); tested on ViT. VíT, V." (columns).
We see in Figure & that the FERNN models again outperform non- flow-equivariant models on generalization classifying actions undergoing new flows at time (plots highlighted in orange). We note that the drop in performance is likely due to the V truncation discussed in o breaking exact equivariance and causing overfitting. In Table 2 we that when trained and tested on strongly moving data, Virain = Viess = V.", the FERNN significantly outperforms the non- won-ffow-equivariin counterparts with no train-test disparity.
Action Recognition Results
Discussion
In this work, we have introduced flow equivariance, general notion of equivariance with respect to parameaaaatterzzed group transformations for sequence models, and provided an initial framework for constructing sequence models which satisfy this definition.

---

Limitations & Future Work. An immediately apparent limitation of FERNNs is the formal restriction to constant 'velocity' transformations. In Appendix $E.55 we show that in practice, FERNNS are still able to model variable velocities, and even inter-velocity interactions, through the use of inter-V elements of the flow convolution kernels. A second significant limitation of the proposed FERNN construction is that it relies on the shift representation of the flow action on the dimension. This implies that the number of activations, compute, and memory must linearly increase with the size of V (see {B.9). and additionally induces the truncation errors described in $ Future work on an analog to Steerable CNNs [Cohen and Welling 2016b] for flow equivariance would be highly valuable in alleviating this limitation. In summary, many of the FERNN's s limitations are nearly identical to those of the original Group Equivariant Neural Networks - namely, scalability and discrete group equivariance. With time, however, researchers have developed methods to ameliorate these limitations, such as through steerability and gauge equivariance [Cohen et al. 2019]. In doing so, group equivariance has become practically relevant, including being used in highly popularized models [Jumper et al. 2021]. We therefore suggest that our paper may be seen as a first step in the introduction of a new theoretical foundation for dynamic equivariance in sequence models.
Related Work. While our definition of flow equivariance is novel, prior work has developed equivariance with respect to space-time transformations such as Lorentz transformations (Bogatskiy) al. 2020 Gong et al.] 2022]; ; essentially processing the input as a 4D spacetime block such that the transformation is therein self-contained. The novelty of our work compared to this literature is the fact that our approach applies to causal sequence processing, and is defined for arbitrary sequence models rather than spatio-temporal convolutions only. A number of works consider equivariance in recurrent frameworks, [Azari and Erdogmus 2022| Nguyen et al. 2023 but only consider static coordinate transformations, not those which change over time, as our flows (e.g. Figure II. Closest to our work, 'Equivariant Observer Theory' [Mahony et al., [022] has studied how build hand-crafted models and filters which are indeed equivariant with respect to sime-araaeterizza work lacks a notion of training familiar to the geometric deep learning community. Notably, several biologically inspired sequence models, including the Topographic VAE [Keller and Welling [022] and Neural Wave Machine [Keller and Welling [023], explicitly 'roll' latent representations so that a single-velocig flow is baked into the network and subsequently learned from data. Our formulation suggests that these heuristics may be special cases of Flow Equivariant RNNs and generalizes the principle to any set of continuous flows V. We refer readers to DD for extended discussion.
symmetries, however, this
Relation to Traveling Waves in Biological Neural Networks. Spatiotemporal flows of neural
activity, such as traveling waves, have been observed throughout the brain since the earliest recordings
[Adrian and Matthews 1934 Muller et al., 2018). Related work in the domain of neuroscience has
studied how such dynamics can be used to facilitate encoding visual motion (Jancke et al. 2004
Chemla et al., [0018] Townsend et al. 2017 Heitmann and Ermentrout 2020
Benigno et al. 2023
May and Gjorgjieva 2024). physical motion Kaske and Bertschinger 2005 Rubino et al. 2006].
or motion through abstract spaces (Keller et al.,) 2024b). Our work can be seen to formalize this previously observed connection between motion and flowing neural dynamics in the language of equivariant neural network theory. Specifically, the study of flow equivariance shows that autonomous traveling-waye waaeing--wv-llkke like dynamics (flows) are not only useful for encoding motion, but rather they are necessary for the accurate and stable representation of any stimulus undergoing a corresponding flow. In 3 we have shown that for any non-trivia recurrent neural network to process a sequence undergoing a flow transformation (such as visual motion) in structured equivariant manner, the hidden state dynamics must have hidden-state dynamics that autonomously realize a homomorphic representation of the same flow, rather than being input-driven emulate it. In addition this theoretical argument, we highlight the relationship between lie-group structured flows (our (v)) and traveling waves in the neuroscience literature. Specifically, it has been found that the lateral recurrent connections in cortex, along which spontancous traveling waves in visual cortex are believed to propagate (Davis et al. 2024). preferentially connect neurons along smooth integral curves of lie group transformations with respect to individual neural selectivity [Ben-Shahar and Zucker 2004 Seriès et al., [202 [Hoffman| 1989]. In other words, the connections which are believed support latent flows are known to connect neurons with receptive fields which are precisely related by the representation of the flow on the input space. Altogether, we believe this combined evidence supports serious consideration of the hypothesis that some forms of observed spatiotemopral neural dynamics may be functioning as a form of ame-pparmeterried symmetry equivariance, and we hope this encourages further empirical investigation in the future.
10

---

Acknowledgments
We would like to thank Max Welling for continued guidance and advice, which ultimately led the author to the development this work. We would also like to thank Maurice Weiler, John Vastola, and Gyu Heo for discussions which improved this work. We would like to thank members of the CRISP Lab at Harvard, and the Kempner Institute for the opportunity to present and improve early versions of the work. Finally, would like to thank the Kempner Institute for funding, and for the opportumity to independently pursue the development of this work through the author's s research fellowship. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University.
References
E. Adrian and B. H. C. Matthews. The interpretation of potential waves the cortex. of Physiology, 81(4)440-471, 1934, doi: http:://doi.or/10..11 1934 //physoc. onl ine ibrrrr.rbrr... wiley. http:////////////////e//e 11w.:1...111111 11tp://h///.p..l...mmmm 1934. sp003147
The Journal URL https
Bahar Azari and Deniz Erdogmuş. Equivariant deep dynamical model for motion prediction, 2022. URL
https://arra /aapsi////xxvi.o/// wwt.:or//abbb.....././/
Sourya Basu, Prasanna Sattigeri, Karthikeyan Natesan Ramamurthy, Vijil Chenthamarakshan, Kush R. Varshney, Lav R. Varshney, and Payel Das. Equi-tuning: group equivariant fine-tuning of pretrained models. In Proceedings of the Thirty--eventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications Artificial Intelligence and Thirteenth Symposium
Educational Advances in Artificial Intelligence, MAAI23AAAT23AATT2 AAAI Press, 2023, ISBN 778-1-57735-880-00 doi: 10. (309//aai..3776625555. URL https://doi wttg/...1////aaa...... 1/609//aai.. v37i6. 25832
Ohad Ben-Shahar and Steven Zucker: Geometrical computations explain projection patterns of long-range horizontal connections in visual cortex. Neural Computation, 16(3):445-476, 2004. doi: 10.1162/
089976604777744666.
Gabriel Benigno. Roberto C. Budzinski, Zachary W. Davis, John H. Reynolds, and Lyle Muller. Waves traveling over a map of visual space can ignite short-term predictions of sensory input. Nature Commu- nications, 14(1):3409 2023. doi: 10. 46000337777 URL https://doi.orz ceroi.oogg/000 htg/10..r 1038/
841467-023 39076-2
Alexander Bogatskiy, Brandon Anderson, Jan T. Offermann, Marwah Roussi, David W. Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics, 2020. URL/attps://arxi org/ aba/20b/5
04780
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: Grids, groups, graphs, geodesics. and gauges. arXiv preprint arXiv:2104.13478 2021.
Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build equiiiauaan steerable CNNs. In International Conference on Learning Representations, 2022. URL httpps//opporrrri....eee///oooot 1d- Id-WE4ge9xInQu
S Chemla, A Reynaud, M di Volo, Y Zerlaut, I. Perrinet, A Destexhe, and F Chavane. Cortical suppressive waves shape the representation of long-range apparent motion. bioRxiv. 2018. doi: 10.1101/372763. URL
lattps://www bwwrrii...vv....................
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional 08755
neural networks, 2019.
Trinity Chung, Yuchen Shen, Nathan C. L. Kong, and Aran Nayebi. Task fisk-optimized convolutional recurrent networks align with tactile processing in the rodent brain, 2025. URL https://arxiv, httpaaawwiv.a..aaaaaeoeeee abs/ 2505
18361
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990-2999, New York, New York, USA, 20-22 Jun 2016a. PMLR. URL http:ss:/ppoceeediggee mlr. prress/448coheeccte html
Taco S. Coben and Max Welling. Steerable cnns, 2016b. URL https:t//axxiii www.aarwwww.oorrraaaaee www.://r//aabbbbeeeeeo 08498

---

Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral can, 2019. /tt.ttts:////////t https://azziv wwwpziii...ggaaae wttg/ab//////// 04615
Zachary W. Davis, Alexandra Busch, Christopher Steward, Lyle Muller, and John Reynolds. Horizontal cortical
connections shape intrinsic traveling waves into feature- seleccive motif's that regulate perceptual sensitivity.
Cell Reports, 43(9):147777, 2024. ISSN 2211 2211-1247, doi: https:///oii hrgp:///ooooooooo////oo ww.
wttp://.eerre..222.1444... htttps:/www. www.ecciieeccrt...cc.... www http://cecccecceeeeeeeeeeecceees htt.://wwwa.aa...a./////////////
URL
Nichita Diacomu and Daniel E Worrall. Learning to convolve: generalized weight-tying approach, 2019. URL
04663
Carlos Esteves, Christine Allen--aBhae--tteeettttetee Xiaowei Zhou, and Kostas Danilidis, Polar transformer networks, 2018. URL hRtpphtts://aaiiv..oo http:////aaxiiv..rgabbbaa wrtp:////ggaabb///0// .18889
Fynn Fromme, Christine benard convection with equivariant autoencoders, 2025. URL httpp://axxxii wttp://ww..rgga/////oo//// 13569
Hans Harder, and Sebastian Peitz. Surrogate modeling of 3d rayleigh-
Mathias Gehrig and Davide Scaramuzza. Recurrent vision transformers for object detection with event cameras, 2023. URL WtRthtts:://arret org/ Sargg//bb//an 2212. 2212805598288
Shiqi Gong, Qi Meng, Jue Zhang, Huilin Qu, Congqiao Li, Sitian Qian, Weitao Du, Zhi-Ming Ma. and Tic- Yan Liu. An efficient lorentz equivariant graph neural network for jet tagging. Journal High Energy Physics. 2022(7), July 2022. ISSN 1029-8479. doi: 00:10077jhep07(2222)03. URL lhtt:://daa httpp://xxx.oo.... www.dooooo...oooo 1007/JHEP07 (2022)000
Jonathan Gordon, David Lopez-Paze Paz. Marco Baroni, and Diane Bouchacourt. Permutation equivariant models for compositional generalization in language. In International Conference Learning Representations, 2020.
Deepak K. Gupta, Devanshu Arya, and Efstratios Gavves, Rotation equivariant siamese networks for tracking, 2020. https://://aax org/ abl/aaa/../////.///////aaaeea 13078
Brian C. Hall, Lie Groups, Lie Algebras, and Representations: An Elementary Introduction. Springer In- URL http
ternational Publishing, 2015. ISBN 9783319134673. doi: 10.1007/998--3311--146---
//dx. doi..doi.coo .or. 1007 /97--3/98-3--11 --37-----1-----1 13467-3
Stewart Heitmann and G. Bard Ermentrout. Direction-selectivee motion discrimination by traveling waves in visual cortex. PLOS Computational Biology. 16(9):1-20. 09 2020. doi: 10. 10.1377/joumall.pc..0000 1occbi.0081166 164. URL
https://doi.org: 10. 1371 1/7r/jo ppbi. 1008164
Geoffrey Hinton, Sara Sabour, and Nicholas Frosst Matrix capsules with em routing. 2018. URL https paapenrrviee.net/pooe www.eeeppddddddddittee HHdHHJLLGHHWB
William C. Hoffman. The visual cortex is a contact bundle. Appl. Math. Comput., 32(2-3):137-167, aug 1989. ISSN 0096-3003.
doi: 10. 001001600666300388999000000 URL.
(003 (B00000001 90091-X
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. 2016. URL http::::://aiiv..rr//////// wwtp://..ggbbbbbbbbbbbbbabb 1506. 02025
Dirk Jancke, Frédéric Chavane, Shmuel Naaman, and Amiram Grinvald. Imaging cortical correlates of illusion in early visual cortex. Nature, 428(6981):423 -244244426, 2004. doi: 10. 10.00/auuurrr0396 URL httpps://do
org/ 10.0/..13//aaaa 0000///eee
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneherger, Kathryn Tunyasuvunakol,, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera-Parede Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873)583-559, 2021. doi: 10-100888441586--2-0011922 URL htttp::/doi..rgg http:///.......0///11100000000-0
Alexander Kaske and Nils Bertschinger. Travelling wave patterns in a model of the spinal pattern generator using spiking neurons. Biological Cybermetics, 92(3):206 218, 2005. doi: 10.1007/s00422-000 -05-02000000-----
httpstss::/ddo www.liorrro...m 0wep/110.007//00422-00000-000000
12

---

T Anderson Keller. Nu-wave state space models: Traveling waves biologically plausible context. Science
Communications Worldwide. 2025. doi: 10.57736/b300--eee. URL Inttps://ww..wot worldw org/
cosyne-25/mu 25veu-wave-eeeeeee statt- satetee--paa-emmmmeeeee -ww.ddess-aaeaelaa----000000 3803886
Anderson Keller and Max Welling. Topographic vaes Iearn equivariant capsules, 2022. URL https
//arxiv oragziiv.rraaaa abs/ /t0b9////9../ 2109. .01394
T. Anderson Keller and Max Welling, Neural machines:
Learning spatiotemporally structured represen- tations with locally coupled oscillatory recurrent neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Rexearch, pages 16168-16189. PMLR, 23-29 Jul 2023.
mlr press/ grees/vv022 v202/
keller23a. html
Anderson Keller, Lyle Muller, Terrence Sejnowski, and Max Welling, Traveling waves encode the recent past and enhance sequence learning. 2024a. URL http:://aaxxxi.. wwtp:/iiioor//bbbea httg//bb//3009 08044
T. Anderson Keller, Lyle Muller, Terrence J. Sejnowski. and Max Welling. A spacetime perspective on dynamical computation in neural information processing systems, 2024b. https:////// htttps://arxi..orr/bbs.. org/abs/ 2409. 13669
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization, 2017. URL https
//arxiv wwwxiivoog///o aaaaabssioo 1412.69828
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference Machine Learning (ICML.), 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pages 2278-2324, November 1998. doi: 10.1109/5.
726791.
Tony Lindeberg. Invariance of visual operations the level of receptive fields. PLOS ONE, 8(7):1333, 07 2013. doi: 10. how.i/ourall..oo..0006..
URL
1371/journal. pone. 0066990
Tony Lindeberg. Unified theory for joint covariance properties under geometric image transformations for teatro-temporal receptive fields according to the generalized gaussian derivative model for visual receptive fields. Jourmal Mathematical Imaging and Vision. 67(4):44 2025, doi: 10.1007//108511-25--02244
StRtaatta:://doo htt::/dooo.oogg////// orgg/00 10011000/001-005-000-0000000 0124799
Alejandro López-Cifuentes Cifuentes, Marcos Escudero- Viñolo, and Jesús Bescós. A prospective study on sequence-driven temporal sampling and moo-on compensation for action recognition in the epic-kitchens datasct, 2020.
URL http:htts///axiii.rrrrrgeeee abs/////////////// 2008. 2008 11588
Robert Mahony, Pieter van Goor, and Tarek Hamel. Observer design for nonlinear systems with equivariance. Annual Review of Control, Robotics, and Autonomous Systems, 5(Volume 5, 2022):221-252, 2022. ISSN 2573- 5144. doi: org.:/00000.00010000//10 www../r///....///mmmmeo. 1166//nnuevvvvvve//e control-on1100000 (11---00112210001 010324. URL htttp::/wwwwt wnw.alleeeeeiii.eeea.....aa..... org/ coottent/journalst /00. 1146/ annurnn-
0000000060000000 010324
Lilly May and Julijana Gjorgieeva. Pre-training artificial neural networks with spontaneous retinal activity
improves motion prediction in natural scenes. bioRxiv, 2024. doi: 10.1 10.1100/2024.06 12..22002..0.11.59944.. 15.599143. URL
httpp::/www biorxiv. www..//riwi...ggo/ttte///ee////e www.aaeenn/arll////////////////0 06. 15. 599143
J. Mochlis and E. Knobloch. Equivariant dynamical systems. Scholarpedia, 2(10):2510, 2007. doi: 10.4249/
scholarpedia.2511 2510. revision #91240.
Lyle Muller, Frédéric Chavane, John Reynolds, and Terrence J. Sejnowski. Cortical travelling waves: mechanisms and computational principles. Nature Reviews Neuroscience, 1955:2255- March 2018. doi: 10. 10.1038/mm
2018.20. URL hattps:///oo http://doi..o//////////o 10. 1038/arn. nrn. 2018. 20
Hai Huu Nguyen, Andrea Baisero, David Klee, Dian Wang, Robert Platt, and Christopher Amato, Equivariant reinfoncement leamning under partial observability. In 7th Annual Conference Robot Learning, 2023. URL
J. E. Pedersen, J. Conradt, and T. Lindeberg. Covariant networks. Nature Communications, 16(1):2231, 2025. doi: 10.103//44446-000 63493-0. URL https //doi. org/ 10.
receptive fields for spiking neural
13

---

Doug Rubino, Kay A Robbins, and Nicholas G Hatsopoulos. Propagating waves mediate information transfer in the motor cortex. Nature Neuroscience, 912):1549-1557, 2006. doi: 10.1038/nn 1802. URL https
1038/nn1802
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing hetween capsules, 2017. URI.
htttp::/arxiv.orggg wttp://.ogrrabbbbaaaaaeeeea 09829
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: A local svm approach. In
Pattern Recognition, 2004. ICPR 2004, Proceedings of the 17th International Conference on, volume 3, pages 32-36. IEEE Computer Society, 2004. ISBN 0-7695-2128- doi: 0..1199/CCR.22004.77
Peggy Seriès, Sébastien Georges, Jean Lorenceau, and Yves Frégnac. Orientation dependent modulation of apparent speed: a model based the dynamics of feed-forward and horizontal connectivity in v1 cortex. Vision Research, 42(25):2781 2797, 2002. ISSN 0042 6989, doi: http:///bl.orr//10000066/6 6989(02) 00302-4,
http:://wwwwwwwccccccceeeee. com/ http:///www.aacoeeeeee/errreraee http:///saaall//rrr.///////////0
Ivan Sosnovik, Artem Moskalev, and Amold Smculders. Scale cquivariance improves siamese tracking, 2020a.
09115
Ivan Sosnovik, Michal Szmaja. and Amold Smeulders. Scale-equivariant steerable networks, 2020b. URL
https://arxiv. wrgg abs/ 1910 11093
Courtney J. Spoerer, Patrick McClure, and Nikolaus Kriegeskorte. Recurrent convolutional neural networks: A better model of biological object recognition. Frontiers in Psychology. Volume 8 2017, 2017. ISSN 1664- 1078. doi: wwt./////wg....11111.. 1..555 URL WRRLhttp:://ww/ www.:wwww.oniiein.nnmmmmm http://rrrooooooaa//////oooooooo articles/100 3tt.://0...099.g..ggg//
Spren Kaae Sønderby, Casper Kaae Sonderby, Lars Maaløe, and Ole Winther. Recurrent spatial transformer networks, 2015. URL htttttttt///aaiiv. org/ ha////aaa///////////// 1509 05329
Rory G. Townsend, Selina S. Solomon, Paul R. Martin, Samuel G. Solomon, and Pulin Gong. Visual motion discrimination by propagating patterns primate cerebral cortex. Journal of Neuroscience, 37(42):10074 10084, 2017. ISSN 0270-6474, doi: 10. 100152//NEUROSCI..... 1538-1..538-1.20117 Utttttss:://wax haattp::/wwwwjneurooss jneurosci org/ co/c/ontnnn/33 37/ 02/////1077//
Eric Wang, Paul G. Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder. Marissa A. Weis, An- dersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding. Dat Tran, Jiakun Fu, Casey M. Schneider- Mizell, Nuno Maçarico da Costa, R. Clay Reid. Forrest Collman, Katrin Franke, Alexander S. Ecker. Jacob Reimer, Xaq Pitkow, Fabian H. Sinz, Andreas S. Tolias, and MICrONS Consortium. Foundation model of neural activity predicts response to new stimulus types. Nature, 640(8058):47-477) 2025. doi: 10. 1038/ s41886-1186668 025 08829-
10.1008//41188 586-025 025-08829-90
y. URL htttpp//////
Rui Wang. Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. 2021. URL /ttpss:/:///r http://wwwwv..rr//a//////./.....
Daniel E. Worrall and Max Welling. Deep scale-spaces: Equivariance over scale, 2019. URL wttth:tt///aatt/ org/ aba/ 1905 11697
Haixu Wu, Zhiyu Yao, Jianmin Wang, and Mingsheng Long. Motionmn: A flexible model for video prediction with saacetime-varyine motions, 2021. htt//ttps://aaxi httpt/tpp/s//a....ooo/aaaaa///// 02203
Wenhao Zhang, Ying Nian Wu, and Si Wu. Translation-cquuivarinnt representation in recurrent networks with continuous manifold of attractors. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15770-15783. Curran Associates, Inc., 2022. URL https: httt:::s//proceeddnngs. neereeeeeedddnneeeeeeeeeeeeeeeee www.:es..eeeeeepeeeeeeeeeeeeeeee cctpppper.fille/ppprre///// Paper- Conference. pdf
Maksim Zhdanov, David Ruhe, Maurice Weiler, Ana Lucic, Johannes Brandstetter, and Patrick Forré. Clifford- steerable convolutional neural networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41xt International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61203- 61228. PMLR, 21-27 Jul 2024. URL
www.m.r.p...... www.mmp......sv/.m wtwppprrwss//////// 2hdanov24.. httmmllm22...mm html
Alex Zihao Zhu, Ziyun Wang, and Kostas Daniilidis. Motion equivariant networks for event cameras with the temporal normalization transform, 2019. URL
Yuliang Zhu, Jing Cheng, Zhuo-Xu Xu Cui, Jianfeng Ren, Chengbo Wang, and Dong Liang. Sre-cnn: A spatiotem- poral rotation equivariant cnn for cardiac cine mr imaging, 2024. URL / org/ abs/2409 2409 08537
14

---

Appendix
Extended Proofs
17
Equivariance of Group-Litting Convolution & Croup-Convolution
17
A.2
Equivariance of Group Equivariant RNN
18
A.3
Frame-wise Flow Equivariance
18
A.4
Group Equivariant RNNs are not Flow Equivariant
19
A.5
Equivariance Lifting Flow Convolution
A.6
Equivariance of Flow Convolution
21
A.7
Equivariance of FERNN
21
B
Experiment Details
24
Flowing MNIST: Dataset Creation
24
B.2
Flowing MNIST: Models
25
B.3
Flowing MNIST: Next- Step Prediction Training & Evaluation
26
B.4
Moving KTH: Datasets
27
B.5
Moving KTH: Models
27
B.6
Moving KTH: Action Recognition Training & Evaluation
................................
28
B.7
Moving KTH Samples
28
B.8
Compute
28
B.9
Runtime and Memory Usage
29
Variations of FERNNs
30
FERNN with Non- Trivial Lift
30
D
Related Work
31
D.1
Flow Equivariance without a Hidden State
D.2
'Statically Equivariant' Sequence Models
31
................................
D.3
Neuroscience and Biologically Inspired Neural Networks
......s.........................
................................
31
D.4
Reference Frames in Neural Networks
32
D.5
Broadly Related Work
32
................................
D.6
Equivariant Dynamical Systems
33
Extended Results
34
In-Distribution Next-Step Prediction (Table & Figure 3)
34
Length Generalization MSE vs. Forward Prediction Plot (Rotation)
35
Length Generalization Visualizations (Translation)
36
Velocity Generalization Visualizations
37
Flow Equivariant RNNs and Variable Velocity Flows
38
15

---

E.5.1
Variable Velocity Moving KIH Action Recognition
38
E.5.2
Mixing Across the V Dimension: Bouncing MNIST
38
E.6
Moving KTH with Data Augmentation
39
Larger Basclines
39
16

---

A Extended Proofs
A.1 Equivariance of Group-Lifting Convolution Group-Convolution
We can verify that group convolutional layers given by Equation 3 are equivariant according to Equation 2 While these are well-known facts, we re-derive them here in our own notation for completeness and to lay the groundwork for our later proofs. First, for the lifting convolution, wish to prove:
Kg f) & ()) = 9 If te Wi Vg.9 FEF(X))
(15)
Proof. (Group-Lifting Conv. is Group Equivariant)
K [6 x)Wi .x) (by def. action, Eqn.
[00 *c)")(()== = Σ
(16)
ZE
(. -Q-2)) (where 8 = =g-t.a0 x:)
(17)
Σ
- -2) (by associativity)
(18)
2E22 k=
MC)WW(( (0))))((0900)---- -))++a)) (by defn. inverse)
(19)
BE k=1
[f WoW'' 9) 9 (f (f (by Eqn. 3 & defn. action) (20)
In line 17 we use the fact that because the group G acts on X by a bijection (formally, the input space representation of the group is xx G - Aut(X)), )), the substitution è = g 1_ x is just relabeling of the index set, and hence the sum over x E is the same as the sum over E X.
Next for the group convolution, the proof is virtually identical. We wish to prove:
[0 f) * W و - [f * W'(()) Vg.9 G. Fx()).
(21)
Proof. (Group-Conv. is Group Equivariant)
K
[0 f) *G (') = Σ fulō -WWWo h) (by def. action, Eqn. 1
(22)
bEG k=1
Σ
- (0 (Q-h)) (where à = 1-- h))
(23)
hEG
ΣΣ - -g)-)))))))))))))))))))))) (by associativity)
(24)
hEG
ΣΣ Σ (((h)WW((G (.))) -1)-- (by defn. inverse)
(25)
bEG 1=1
= = . g) = 9. ( (by Eqn. 3 defn. action) (26)
Again, we use the fact that the group G is closed under the group action, so the substitution â = 9- 1-g-- h is just relabeling of the index set, and hence the sum over h G is the same as the sum over h G.
17

---

A.2 Equivariance of Group Equivariant RNN
We wish prove that recurrent neural network built with croup-qquivarian convolutional layers (Equation 4 is equivariant according to Equation 2 If we write the hidden state as a function of the input signal h441lfg((9)) E FY) where Y G. then we can write the the equivariance condition as:
h++ 9 fg(l(g) = 9- hot1lfs(l9)) V f Fx(()), BgEG, G. where 9- f&t := {g- fi Z, S t), denotes the input signal generated by applying the same group element to each timestep. We prove this by induction for all t Z4:
(27)
Proof. (G-RNN is Group Equivariant)
We assume (i) & and #G are the group-lifting convolution and group convolution defined in Equation (ii) is G-equivariant non- non-linearity (e.g. pointwise), and (iii) g acts linearly h-space. Since is defined by the lifting and recurrent convolutions, we see that Y = G. We further assume (iv) ho is initialized to be invariant the group action, i.e. 9 ho(g) ho g) ha(g), and (v) the input signal is zero before time zero, i.e. fco = 0.
Base Case: We can see that the base case is trivially true from the initial condition, since the initial condition is not a function of the imput sequence, so:
holo [c)](g) ho) (since ho is indep. of input fco)
(28)
holfcoll@ g) (by initializaatin)
(29)
= 9 holf<al(o) (by defn. action)
(30)
Inductive Step: Assuming h f<e(g)) = 9 hl)f< we wish to prove this holds also for
hetilö lg (g) o (1velo Sea] ka W] (g) + [] Ai) (g))
(by defn. G-RNN)
(31)
or (1l0 h.[f<dl * W| (g) + Il9 fi] (g)) (by inductive hyp.)
(32)
(g [helfeel kg W](g) + 9- (f(Aa((((g))
(by equivar. of G-conv.)
(33)
ğ ([he]fet) *o (g) + [SA (g)) (by equivar. non-lin.)
(34)
9 h4tlffe)(g) (by defn. G-RNN, Equ
(35)
We note that the step on the fourth line assumes that the group action is linear, in that it distributes over the addition operation between the previous hidden state and the input. Traditionally, group equivariant neural network architectures have been constructed to exhibit linear representations in the latent space, and therefore this is natural assumption. However, if one allowed y to act by non-linear maps (i.e. dropped the mneaarrreresentttion assumption), then neither distributivity nor o-equivariance would hold in general.
A.3 Frame-wise Flow Equivariance
We wish prove that any G-equivariant map : Fx(X) + (Y),), that is applied 'frame- wise' to space-time function f E (x( is also flow equivariant according to Definition 3.1 Specifically, let L[ (o(f)), (fff)...offr.. ((T)] be a sequence model built by concatenating the output of G-equivariant map ø applied to each timestep t of the input signal f.. Furthermore, let ø be equivariant to the individual group elements generated by vector fields V, i.e. ((vr(v) f() ()(()) o(f)) Vt E Z, E V, (X)... Then:
Proof. (Frame-wise Equivariant Maps are Flow Equivariant) ((((v) f) (Ø(Vo(v) fo), o(1 (v) fi) o(vT(v) fr)]
(by defn. flow action)
(36)
[vol(v) dDfo), (v) (f{1) ... (r(()) (or(v))(1))) ((r)) (by G-equivariance of ()
(37)
('()) [d(fo),
{(T)] (by defn. flow action, eqn. 5
(38)
= (w()) (()) (by defn. ))
(39)
18

---

A.4 Group Equivariant RNNs are not Flow Equivariant
In this subsection we prove Theorem B.I that the group- croup-equivariant RNN as defined in Equation 4
with oonzerro W, is not flow equivanant according to Definition 3.1 except in the degenerate flow invariant case.
We will prove this in two parts: (i) First we will prove that a G-RNN with constant kernels is indeed flow invariant through induction (the degenerate case). Then (ii), will proceed with proof by contradiction to show that this is the only such flow equivariant G-RNN possible.
To begin, we recall that the degenerate flow invariant case is defined as a G-RNN where both W and are constant over G. We will denote such kernels U and W, such that:
W(g) W(g),, Vg.g EG & U(g- x) = =ggg' x)
Vg.g' E GG,
(40)
We wish prove that such network satisfies Definition 3.1 but more specifically that it is actually invariant to the flow action, meaning:
hel(()) fce) = 1.fed A E v VEV, tE
(41)
We can see that this a special case of Definition 3.1 where the group action on the output space trivial, i.e. given by the identity: (v) ho[S<c]
[<r]. We can prove this by induction as before.
First we introduce the following lemma for convenience:
Lemma A.I. (Lifting convolution with U is group invariant) The result of applying the lifting convolution (Rc) defined in 3 with constant kernel U to a signal f Fx(X) is invariant to flow element action on f.i.e.
[(bb(V) f) Aa ( = 8 (g)
YUEV,
(42)
Proof. (Lemma A.I
K
((V((V) f) (o))"(()) = Σ [x( x x) (by def. action, Eqn.
(43)
EE2 k=1
Σ
f (We(()) -2)) (where & = =b
(44)
EES
K
Σ
(.) .)) (by associativity and inv.)
$E2 k=1
(45)
K
Σ Σ 2E22
fA(O) -..
+)) (by closure, 9 = 4/)))))-1 g E G
(46)
K
ΣΣ
fa +) (by defn 11 .2) = =ū(g-1 2)) (47)
ZE2
= [f (ol
(by Eqn. 3
(48)
We have again used the fact that the group acts X by bijection as in $A.I We can then proceed with the proof by induction to prove that the G-RNN with constant kernels is flow invariant:
Proof. (G-RNN with Constant Kernels is Flow Invariant)
Let he fel be defined as in Equation with 24 = Li and W = W. We again assume (i) A and *G are the group-lifting convolution and group convolution defined in Equation 3 (ii) a G-equivariant non-linearity (e.g. pointwise), (iii) acts linearly on /t-pace, (iv) hn is initialized to be invariant to the group action of the individual flow elements, i.e. (v)) ho(g) ho(*) g) ho(g), and (v) the input signal is zero before time zero, i.c. feo 0.
19

---

Base Case: We can see that the base case is again trivially true from the initial condition, since the initial condition is not function of the imput sequence, so:
ha (v) (<) = holf<ol(g) (since he is indep. of input f<o)
(49)
Inductive Step: Assuming the inductive hypothesis that (x(((p) feil = he[fee], we wish to prove this holds for [ 1:
Me+1lb(1) fcel(e) o ((hi/((()) fed to w/ (0) + [[((()) ft. f|]o (g)) (by defn. G-RNN)
(50)
= < *G W] (0) [(b Mi (o))
(by inductive hyp.)
(51)
[[hiffd to W(g + (f(A)(((9))) (by Lemma A.1
(52)
h+tl//s](gg) (by eqn.
(53)
Finally then, we use a proof by contradiction to show that this degenerate flow invariant case is the only possible case for the G-RNN to be flow equivariant.
Proof. (Theorem 3.1
G-RNNs are not Generally Flow Equivariant)
Assert the converse, that the G-RNN defined in Equation is flow equivariant, and that the kernels W and 24, are not constant, i.e. W * W + 17 (as defined in Equation 40). Then, it should be the case that:
het1 (vb))) fg))() = (v(()) (oo(v))M$s()eeeH))) M(sc))es1l)) (att://)=nlooo0//// -= g)
(54)
For / = O have the following:
hi(b(v) - (g) = o ([holv(v)) fcol to (x) + [[bo(v) fol (g))
(55)
= a (HholSeo) to () + [oAaul (0)) (by defn. ho & 1b))
(56)
= hg)
(57)
For the next step, we get:
h2(bb()) f52)(g) = a (1ba(bb())) fso] *o W) + (b)()) A1) (o)
(58)
[ W| (g) + (1() [fikal]] (g)) (by above & g-conv.) (59)
If we look at the action of the flow on the output space, according to flow eguivariance, we should
have:
(1b(1) M[Ssa))2(o) a (v) [/] * w| (g) + (())()) S1)oal)] (0))
(60)
(vb(ivv) [hilfso] t W| (g) (1))) [f (g)) (by g-conv.) (61)
Setting h2((b()) (g) (v(v) M[f=zl)2(g), as per our assumption, we see the input terms match exactly, but the 1 terms imply the following equivalence:
[hil/5o] *o ()) v (v) [h (Szo] *o W((g)
(62)
[f *o W] (v) g) (by action defn.)
(63)
We see this is precisely the 'lagging' hidden state that we visualized in Figure In order for this equality hold for all V. the output of the convolution must be constant along the flows generated by all u. We can see that this is only satisfied by W = W, a contradiction.
20

---

A.5 Equivariance of Lifting Flow Convolution
In this section we verify that the flow-lifting convolution given by Equation 7 is equivariant action of the individual flow elements, with the following representation of the action:
(v)) [Je X (v, 9) (())()) ft) xv (²) g) [Se kv (" ( 9) (64)
Since the flow lifting convolution is a trivial lift, equivalent to the group-lifting convolution with an extra duplicated index, this proof is trivial replication of the group-lifting convolution proof:
Proof. (Flow-Lifting Conv. is Flow Equivariant)
K
(v)(()) - ft) &vx (v
Σ -(0)(1)---221(1( .x) (by defn. action)
(65)
xEX
K fu (1)(()) (where & = (b(()))
xcX k=1
(66)
K Σ Σ
( .g)- (by associativity & inv.)
(67)
= Use tvxo CxvxuX((u (v)(v)))
(68)
A.6 Equivariance of Flow Convolution
In this section, prove that the flow convolution in Equation is equivariant to the action of the individual flow elements yielding:
((b(()) h) xvx w (v, (/())) +vx (v.g)
(69)
Proof. (Flow Conv. is Flow Equivariant)
[(We()) h) xv (w, = ΣΣ
h hx(7, m)) Willr - v.-- m)
(70)
NEV mEG k=1
K'
(where in - (b(v)) (m)
= ΣΣΣ
hx(7, W - - (We(v) -m)
(71)
TVV mEG k=1
K¹
(by associativity & inverse)
= Σ
h4(7, m) W - v. (1)(21) - -)- m)
(72)
VEV MEG k=1
(by Eqn.
9
= [h x (v))) .))
(73)
(by action)
= (x()) [h Avxo wo (v))))))
(74)
A.7 Equivariance of FERNN
In this section, we prove Theorem 4.1 by induction. We restate the theorem below:
Theorem. (FERNNs are flow equivariant) Let h[s] E Fr (Y, Z) be a FERNN as defined in Equations 7 and with hidden-state initialization invariant to the group action and constant in the flow dimension, I.e. ho g) = hu(v'. g) Vv', v E V and z (v) hn(U, g) = (v, g) Vv E V, E G. Then, h[f] is flow equivariant according to Definition 3.1 with the following representation of the action the flow in the output space for t 2 1:
(0(D) (1fl)((v.)) helf((v D, (D)
(75)
21

---

We note for the sake of completeness, that this then implies the follow equivariance relations:
he (D)) g) = [4] - 2, (e-1 g) = - hall(( D.
(76)
Proof. (Theorem 4.1 FERNNS are flow equivariant)
Identical to the G-RNN, we assume that a is a G-equivariant non-linearity, y acts linearly on h-space, (v 9) is defined constant as above, and the input signal is zero before time zero, i.c. f<o 0.
Base Case: The base case is trivially true from the initial condition:
ho (o) fcul(e,)) g) hulfcol(u,g) (by initial cond.)
(77)
hols<ol(u D2 1/ (D)- - g) (by constant init.)
(78)
Inductive Step: Assuming he
(0)) f[ bulfl(u -, (.) g) E V,g G, for some T2 O, we wish to prove this also holds for + 1:
Using the FERNN recurrence (Eqn.
the transformed input, and letting H V x G, we get:
he-1lv(0)-f((u.)) = ((s())))11((000)))))) * (( + [(0(00)-A)&
(79)
(by inductive hyp.)
of
*n Ww (((v-v,9)) (,9) + [(v)(D) fi)&H
(v.
(80)
(by trivial inp. lift)
[( *n W - (,9)
2.0))
(81)
(by equiv. flow-conv)
(00) *H
Wi( - -2,9) + (x) [f00[[(AnH@ - v, v.9)
(82)
(by flow properties)
= (() *H wi(v 0,9) + (.(1)) (fi g)
(83)
(by eqiv. non-lin.)
vo(()--((111 (of *n WI(U 0.g) + [sAnU]]u - (,
(84)
(by FERNN Eqn.
11
= -bi(0)- [5[ i,g)
(85)
-h+1lS]( D, (e((0)11.))
(86)
Thus, assuming the inductive hypothesis for time implies the desired relation time + 1; together with the base case this completes the induction and proves Theorem
Similar to the main text, we additionally provide a visual example which demonstrates that the counterexample to equivariance of the G-RNN now no longer holds for the FERNN. As we see in Figure 9 the additional V dimension results in the moving input being picked up as if it were stationary the corresponding channel. The V channels can then be seen permute result of the action of the flow in the latent space.
22

---

m
vit)
P-
Blu)
m
vip) f
Figure 9: A visualization of the same counter example of Figure 2 from the main text, but for the Flow Equivariant RNN. We see that the model built with a set extra V channels, depicted as the three separate rows of hidden states, where each row flows independently per timestep according to its " parameter (e.g. the bottom row flows with velocity -I, to the left, while the top row flows with velocity 1 to the right). When the model is then processing an input with a corresponding flow symmetry (for example motion velocity to the right), the corresponding flowing channel hidden state processes this input in the same reference frame, if it were stationary (the top row in this example). The remaining rows then process the input with a difference corresponding to the difference in velocity between the hidden state channel and the input velocity. We that this results the y & channel permutation in the latent space (the vertical shift in the V dimension by 1 row).
23

---

B Experiment Details
In this section we describe the datasets, models, training and evaluation procedures used in $ of the main text. We additionally include samples from each dataset for visualization. The full code to reproduce these experiments is available at: hhttps://gitieei com akandykeller/FERnI
B.1 Flowing MNIST: Dataset Creation
As described in the main text, we construct sequences from the Flowing MNIST dataset by applying a flow generators y randomly picked from an admissible set Virain. Vial. & Viest to samples from the corresponding train / validation test split of the original MNIST dataset (LeCun et al., 1998.. The training sequences are always composed of T = 20 time-steps, and identically for the test sequences (except in the length-generralizztion experiments where the test sequence length increased to 40 time-steps). We similarly generally set Virain Veat Viest: except in the velocity generalization experiments, where we specify the training and test flows explicitly. Note that the set V which defines the set of flows to which the FERNNs are equivariant can be anything, and must not match the Verain of the dataset. Explicitly, denoting the value of the ith MNIST training image at pixel coordinate (x, as m mraii ((x, 8)), a sample from the training set can be written as: 1. ((x, v)) = 1/r (on())-mmm (3)
V Ex, LE ST (87) where v², 2² ~ Virain and j. h and random indices sampled with replacement from the MNIST dataset. We note that above, in the main paper, have written the signals and the kernels on the infinite domain Z In practice, however, both are only non-zero on a small portion of this domain (i.e. between (0, 0) & (28, 28) for MNIST digits). We do this to simplify the analysis, analogous to prior work [Cohen and Welling 2016a].
((I,y)) (+())))+(((((mmmmm) (R)
Translating MNIST.
For the 2D-translation variant of Flowing MNIST, we consider the group
G T(2) (z², (22,+), with binary operation (z. + (x',3/) 3/) (a x', y²). (88) We use flow generators which are elements of the Lie algebra of this group. V/ E ((2), which we similarly denote vectors in Z2, with the Lie bracket [,, 0 Vn; v E t(2), meaning the translations are commutative. We note that in a matrix representation, we embed the elements of T(2) into the affine group with homogeneous 3 3 matrices:
1
x
(x. v)
0
1
(89)
0
0 1
where the corresponding Lie algebra elements are
0 0
X(v)=
0000 :
0
== (21, E ZZ.
(90)
0
The exponential map is then simply: exp(X (v)) = I + (()). We see that the flow is then given as: (v) = exp(tX (v)) 7 tx((v) and the action of the flow on given pixel coordinate g (1r, y) is given as:
(0
0
0
tu y + tu2
(v) g
1
1/2
0
1
y
=
991)
0
0
0
0
1
0
i.e. a shift the pixel coordinates by velocity v for / time-steps. practice, order to be able to generate long sequences without excessively large images, we perform all translations with cyclic boundary conditions on our input and hidden states, i.e. (v) v) = ((x+lo)) mod W, (y++
mod H), for image size (H, W). Since all of our convolutions are also performed with cyclic boundary conditions, this does not impact performance.
In our experiments, to define the sets of generators that we are interested in, we always use the integer lattice up to velocities N pizels We denote these sets with the notation: step
V {v E z² |||| N).
(92)
For example VoJ the set of all 2D translation vectors with maximal velocity component +2 in either dimension, i.e. Vi ((-2, -2), (-2, -1), (-2,0), ... (,,2)
24

---

Rotating MNIST. For the planar rotation variant of Flowing MNIST, we consider the group
G S) (R. +), with binary operation
0 0' (e 0') mod 2x. (93)
Elements of the Lie algebra so(2) are one-parameter generators of in-plane rotations, each of the form w. J, where J 0 and E denotes the (integer-scaled) angular velocity.
Because so(2) is abelian, the Lie bracket [7, v/ 0 V7, so
We embed S0(2) into the affine group with 2 2 matrices:
cos 0 sin
(94)
sin
cos
whose corresponding Lie-algebra elements are ((v) (0)
wJ,
(95)
Since X(v)² -2212 the exponential map is the usual matrix exponential for planar rotations:
exp(X(u)) = + sin(w) J + (1 - -coosww y².
Hence the flow generated by (/)(v) exp( (())). and its action on a pixel coordinate g = (2; y) (in homogeneous form) is
cos((w)) sin(tw)
sin(tw) cos(tw))
(os() siin(tw) x sin(()) + y cos(tw))
(v( =
(96)
4/
i.e. rotation about the image center by angle tw.
Following experimental protocol, discretize the angular velocity at A = 100 intervals and collect the set of generators
V (u= (u=
|k| S N).
(97)
For instance, {-40%, ,00 -00, 10% 09 10", - 30" 40%) deg In practice, implement the spatial step rotation we use the Pytorch function F gridd sample with zero-padding and bilinear interpolation. We additionally zero-pad all images with 6-pixels on each side (resulting in images of size (40 x 40)) allow for the rotation to within the full image frame.
VR would
consist of the set of angular velocities
B.2 Flowing MNIST: Models
For the Flowing MNIST datasets we compare three types of models: standard group-qquivariant RNNs (G-RNNs) defined in Equation 4 FERNNs with equivariance to subset of the training flows (L.e. Vinodet = V/" and Virain = V ). and FERNNs with full equivariance to the training flows (Vmodel V.rain). For all models each dataset
the same model architecture, and since there are no extra parameters introduced by the FERNN model, all models have the same number of trainable parameters.
For the translation group, the corresponding group-convolution is the standard 2D convolution. We therefore build G-RNN exactly as written in Equation 4 with regular convolutional layers in place of the group-convolution. We use kernel sizes of 3 x 3 for both 21 and W with no bias terms, strides of 1 and circular padding of 1, resulting in hidden state with spatial dimensions equal to the input spatial dimensions: 28 x 28. We use 128 output channels for convolutional 'encoder' U, and similarly 128 input and output channels for recurrent kernel W. This results in a hidden state h E IR 1812822822 We use a ReLU activation function: a (h) ma h). We initialize all hidden states zero: hig O, satisfying our equivariance proof assumptions. At each timestep, we decode the updated hidden state to predict the next input through a 4-layer CNN decoder: go (het1) ) = fe+1- The CNN decoder is composed of three primary convolutional layers each with 128 input and output channels, kernel size stride and eircular padding 1, followed by ReLU activations. A final identical convolutional layer, but with 1 output
Translating MNIST G-RNN.
channel. is used to predict fi+1.
25

---

Translating MNIST FERNNS. For the FERNNS, we use the exact same RNN and decoder architecture, with the only difference being that we extend the hidden state with an extra V dimension for each of the flows in V monee: (v. Explicitly then, through the trivial lift, the input is copied identically to each of these channels, and the flow convolution on the hidden state also operates identically on each channel. As stated in the main text, we define the flow convolution to not mix the V channels at all, explicitly: (i((,,) = Syme Wi(g). thus giving us constant parameter count.
Our lifted hidden state is thus h E R [mmoter x 22 28x 28, The flow action 4 (v)- in the recurrence is implemented practically as Roll of the hidden state tensor by (21,12) steps along the (z, y) spatial dimensions. To achieve invariance of the reconstruction to the input flows, and thereby achieve the generalization we report, we max- over the V dimensions before decoding. Explicitly, the output of the model for each time step is computed as: go (mnxxx he+ 1 (v, g)) se+1-
For the rotating MNIST models, we use nearly identical setup to the translation experiments, with the only difference being that we use (2) group convolutions in place of the standard convolutional layers to achieve the necessary rotation equivariance. In practice, we use the escnn library to implement the (2) convolutions (Cesa et al. 2022]. We discretize the rotation group into AO- = 10" rotations, yielding a cyclic group with 36 elements: Cas- We lift the input to this space and assert a regular representation of the group action on the output. Due to the increased dimensionality of the hidden state from the lift to the discrete rotation group, we decrease the number of hidden state channels to 32 due to hardware limitations. This then yields a hidden state: h E IR 2336x3xx4xx4
Rotating MNIST G-RNN.
Rotating MNIST FERNNs. For the FERNN models, we follow the same procedure as for the translating MNIST FERNNS, except with the action of the group th (v). in the recurrence now taking the form of the regular representation of rotation in the SE()) equivariant CNN output space. Explicitly, this means that in addition to rotating the inputs, lifted rotation channel for each angular velocity v. Again this yields a hidden state of size: h E Rileodde xx36x32x4xxxx
also permute along the
B.3 Flowing MNIST: Next-Step Prediction Training & Evaluation
For the in-distribution next step prediction experiments, (Table Figure of the main text, we use Virgin Veal Viest Vi for the translating MNIST experiments, and Virain Veal Viest Vi for the rotating MNIST experiments. We set the training sequence length to 20 steps, providing the models with 10 time-steps as input, and computing the next-step prediction reconstruction loss (MSE) of the model output on the remaining 10 time-steps. Explicitly:
20
L = 10
Σ llfe 90 (h.))l 1=11
(98)
All models are trained for 50 epochs, with a learning rate of 10-4 using the Adam optimizer Kingma and Ba [017]. For translation flows, we use a batch size of 128, and clip gradient magnitudes at in all models for additional stability. For rotation flows we use a batch size of 32 due to memory constraints, and find gradient clipping not necessary. For evaluation, we save the model with the best performance on the validation set (over epochs), and report its corresponding performance on the held-out test set. For each model we train with 5 random initializations (5 seeds) and report the mean and standard deviation of the test set performance from the best saved models.
Length Generalization. For the we again use Virain = Vea! = Viest = ViF for the translating MNIST experiments, and Virain Vuat = Viest = V for the rotating MNIST experiments. Where the length of training and validation sequences is again set to T = 20 as before. At test time, we increase the length of the sequences to T = 70, but continue to only feed the models the first 10 time-steps as input. In Figure we show the loss of the model for each of these remaining 60 time steps ahead.
length generalization experiments (Figure
Velocity Generalization. For the cuttof-diistribution velocity generalization experiments (Figure 5 we leave the train and test sequence length at 20, but we instead set Virain = Voat C Viest- Explicitly, in Figure 5 for rotating MNIST, we set Virain
Veat V and Vieat =V.R. For translating
26

---

MNIST, we set Virain = Voll = VI and Viest V/2". This tests the ability of models to generalize to new flows not seen during training, and we see that the FERNNs perform significantly better on this test set when they are made equivariant to these velocities.
B.4 Moving KTH: Datasets
To test the benefits of flow equivariance on a sequence classification task with real image se- quences, we opted to use the KTH action recognition dataset (Schuldt et al. 22004, obtained from http://www. csc wsc.kth.c........... wtth.s/eecaaaaaeeeeeeeeeeeeeeeee attaap/ccioss////// The dataset is composed of 2391 videos of 25 people performing 6 different actions: running. jogging, walking, boxing, hand clapping, and hand waving. The original videos are provided at a resolution of 160 120 with 25 frames per second. and an av- erage clip length of 4 seconds. The training. validation, and test sets are constructed from this dataset by taking the videos from the first 16 people as training, the next 4 people as validation, and the last 5 people test. We split the videos into clips of 32-frames each, downsample to a spatial resolution of 32 x 32, and subsample them by half in time, yielding a final set of clips which are 16 steps long.
Since the videos from this dataset are taken entirely from stationary camera viewpoint, there are no global flows of the input space which our model might benefit from. We call this original dataset KTH with VO" (no motion). To test the benefits of flow equivariance in an action recognition setting with a moving viewpoint, we construct two additional variants of the KTH dataset, augmented by translation flows from the sets V/" and V22" These sets are identical to those described in the Flowing MNIST examples: translation with eircular boundary conditions. Again, since we use convolution with circular boundary conditions in all our models, this does not impact model performance.
B.5 Moving KTH: Models
We compare five models on the KTH dataset: 3D-CNN, G-RNN, two FERNN variants, and an ablation of the FERNN (denoted G-RNN+). We describe these in detail below:
Moving KTH 3D-CNN. The spatio-temporal 3D-CNN baseline is built as sequence of five 3D convolution layers, interleaved with 3D batchnorm and ReLU activations. Each layer has kernels of shape (3 x 3 3), no bias, and padding 1. The first layer has 16 output channels, temporal stride of 2, and a spatial stride of 1. Layer 2 has 32 output channels, temporal stride of 1, and spatial stride of 2. Layer 3 has 32 output channels, temporal stride of 1, and spatial stride of 1. Layer 4 has 64 output channels, temporal stride of 1, and spatial stride of 2. Layer has 64 output channels, temporal stride of 1, and spatial stride of 1. This final layer is followed by global average pooling over the remaining (8 x x 8) space-time feature map dimensions, yeilding a single vector of dimensionality 64 which is passed through a linear layer to predict the logits for the 6 classes.
Moving KTH G-RNN. For the baseline G-RNN, each grayscale input frame fr E X1 is passed through three-layer convolutional encoder that preserves spatial resolution:
Conv
1-82 BN+ReLU Conv
32-454 BN+ReLU 3x3
Conv 3x3
64-1128 BN-ReLU
5x5
all with stride and circular padding chosen that the hidden state has the same spatial dimensions as the input (32 x 32). The output defines the encoder feature map 4 * ft- The hidden state he R222x32x32 is updated with single recurrent layer using circularly padded 3 x 3 convolution where W E R 128 x128x12 contains no bias terms. The recurrent convolution is also followed by batch-norm and ReLU non-linearity for added expressivity. The non-linearity or is tanh. Initial states are zero, fn 0, satisfying the assumptions of our equivariance proof. At the final timestep of the sequence (t 16) we take the hidden state of RNN, perform global average pooling to a x 1 spatial dimensions, and feed the resulting 128 dimenssinio vector through fully-connected layer:
(br) FC - (SpatialAvgPool((r)) E R
producing logits for the six KTT action classes.
the exact same architecture, but use the trivial lift to lift the corresponding sets of flows VT for FERNN-V7 and V.T for FERNN-V; (denoted FERNN- 1 and FERNN-2 in Figure 8. Concretely, for every translation velocity E Vmodel allocate
Moving KTH FERNNs. For the FERNNs,
an additional velocity channel, that the hidden state becomes he(v, g) E R Vasall x128xx2xxx33
27

---

where (z, y) indexes spatial position. Input frames are trivially lifted by copying the same encoder features into each v-channel. Following Equation [T] the flow action the recurrence is implemented again Roll of the spatial dimensions of the hidden tensor by (Vn, My) pixels, and the flow convolution uses weight sharing across velocities, W (v: g) = Syme W (g), so the total number of trainable parameters is identical to the G-RNN. As the Flowing-MNN -NNIST experiments. we consider two settings: (i) partial equivariance with Vmodel = V C Vuain V22 and (ii) full equivariance where Vmndel Vinin- To achieve fow-invariant action classification we take the maximum over the velocity dimension after the final FERNN layer, max, (v, (),g), followed by the same global-average-poolig and linear classifier used for the G-RNN. This design ensures that any translaton-iidduce shifts present at test time are pooled over, yielding the generalization results reported in Figure
Moving KTH G-RNN+. To ensure that the observed performance improvement of the FERNN was not simply due to the increased number of hidden state activations and the associated max-pooling, but instead could be attributed the precise flow equivariant form of the recurrence introduced in Equation MT we built a third baseline which is as close as possible to the best performing FERNN (FERNN-D ). while removing precise flow equivariance. Specifically, while keeping all other architectural components of the FERNN-VA V/2 identical, we replaced the single-step action of the flow in the recurrence (v)-) with convolution by separate learned 5 x 5 convolutional kernel for each y channel (randomly inititalized). Since the action e (v) is a simple translation (a local linear operation), this indeed be represented by such kernel. However,
in practice (Figure [), the model fails to learn such kernels and instead overfits to the training data distribution.
B.6 Moving KTH: Action Recognition Training & Evaluation
Models are trained to minimize the cross entropy loss between the predicted class and the ground truth label using the Adam optimizer. Due to the small dataset size, all models are trained for 500 epochs, with batch size of 32. We search over learning rates in the set {3 10 1 10 3 10 10-4), for each model, running three random initialization seeds for each. For each random seed of each tyyperparameete setting, performing model based on the mean value of the best validation loss across all three seeds. We then report in Table the mean test loss of the models (3 seeds) saved at the best validation loss epoch for the best identified learning rate. We generally find the lower learning rates (3 x 10 4) work better for the RNN models, while the higher learning rates (1 x 10 3) work better for the CNNs.
store the model with the best validation loss. We then pick the best
B.7 Moving KTH Samples
0
Original KTH dataset V/="
(b) KTH with VTT
(c) KTH with V V
Figure 10: Samples of the original KTH dataset and its two motion-augmented variants.
B.8 Compute
All experiments in this paper were performed on a private cluster containing a mixture of NVIDIA A100 and H100 GPUs, each having 40GB and 80GB of VRAM respectively. No parallelization of
28

---

individual models across GPUs was required, i.c. most models and training paradigms were able to fit on a single A100 GPU, with the larger models on a single H100 GPU. The cluster nodes allocated up to 24 CPU cores and 375GB of RAM per job, although only a small fraction of this was required for training and evaluation. The majority of models were able to train fully in less than 24 hours. For example, the FERNN-V; models on KTTH trained in 7 hours, and the FERNN-V2 models on MNIST trained in hours, with all other models training faster. The significant exception to this were the FERNN-V models Rotating MNIST which took roughly 67 hours to complete 50 epochs (although they converged much more quickly than this, see Figure 3 we ran them to the same number of epochs the G-RNN for consistency). The reason for this increased computational time was an inefficient implementation of the rotation operation and our custom recurrence, which could both be accelerated in future work. Specifically, we used a naive vanilla Pytorch implementation of our custom FERNN recurrence (Equation 11] using "for loops', which dramatically slowed down training for all models. In future work, implementation of the model with scan operation in JAX, or a custom CUDA kernel would dramatically improve runtime performance. Overall, we estimate the computational requirements necessary to develop the models and run all experiments for this paper totaled approximately 30 days of H100 compute time.
B.9 Runtime and Memory Usage vs. [V]
The flow convolution in Equation | performs convolution over standard group elements (g G, which can be thought of as spatial positions for the simple translation case), and also over flow generators (v V, which be thought of movement velocities). This can be implemented efficiently as a 3D convolution for the case of 2D images, and a ID generator group. For sets with higher dimensional generators, in frameworks like Jax.
use N-D convolutions, which are also implemented efficiently
We predict the computational complexity of the model and the memory usage should scale linearly with the size of the set of generators (which we denote [VD. In the figure below we validate this by plotting the memory requirements and runtime per epoch
function of the size of this set. Note
the G-RNN is equivalent to a FERNN with [V = 1, i.. V = {0].
Runtime os M
Mamory Allaatted vo M
Miemory Reserved vs M
1000
Purtima EEpoc of Bansh (196)
Utservad B1 0.9772
Ubserver Mereryy Alocaled
Ubsarvvr Virerry Hoserved
000
50
E2 doE0
80
600
400
-5
200
5
10
15 20
10
15
20
M
Figure 11: Runtime and Memory usage scale linearly with size of flow generator set VV. Results generated with the FERNN-VT on Translating MNIST, restricted to translation in only the x-direction.
29

---

Variations of FERNNS
C.1 FERNN with Non-Trivial Lift
As noted in $4 is is possible to build a FERNN with non-trivial lift, such that the lifting convolution itself incorporates the flow transformation for each dimension.
Explicitly, we can define such lift as:
Vxx (v, = Σ Σ FEX 1=1
(99)
Another way to think of this, that there is lime-parameterized input kernel defined on the full space V x G, i.e. (u, 9, () = U(vx(v)-1 g); however, we find this viewpoint less elegant given that the kernel then depends time.
We see then when the flow is incorporated into the lifting convolution, the output of the convolution is no longer constant along the index (as it was in the trivial lift). Instead, it is now flowing according to the y'th vector field. Therefore, when processing an input undergoing a given flow (D)), this input flow will combine with the flows of the lifting convolution yield shift along the V dimensions, similar to what we previously observed in the hidden state, i.e.:
K
(V) fi) Xvx (v,
ΣV
2 (2(()) fa(1(())
(100)
TEX 1=1
K f
Σ Σ
(v 0)-1 1 .8) (where 2 == cb((D)
(101)
lfa xvx [v - 0,g)
(102)
Notably then, keeping the flow convolution from Equation 9 unchanged, remove the additional v/n (v) shift from the original FERNN in order to maintain flow equivariance. Specifically, the non-trivial-lift recurrence relation is then given simply
that must
( g) a (lh, *V W] (v, + [fe tvxo U| (v,
(103)
In this setting, the action of the flow on the output space changes to just permutation the V dimension, with corresponding flow g:
(V(D (VD)(v.g) = hIff(v - v.g)
(104)
In sense, this model can be seen as "undoing' the action of each flow on the input when lifting. We find this to be somewhat analogous the traditional group-equivariant CNN design choice where the transformation can either be applied to the filter or the input. In the FERNN setting, the 'filter' is now defined by the full recurrence relation, input sequence, or to the hidden state sequence.
either apply the flow transformation the
Overall, we find this to be slightly less elegant construction since the indexing of the kernel in the convolution then dependent on the time index explicitly. the trivial-lift setting introduced main text, this timeddepeneence is rather implicitly imposed by the recurrence of the hidden state itself, therefore allowing us to only require the instantancous one-step flows during each recurrent update. Regardless,
interested in future work which may explore this non-trivia lift setting more fully, and other interpretations of the FERNN model as described.
30

---

D Related Work
In this section we provide an overview of work which is related to flow equivariance and equivariance with respect to panamearaettri symmetries generally.
D.1 Flow Equivariance without a Hidden State
As mentioned in the main text, it is possible to achieve flow equivariance without an explicitly flow- equivariant sequence model. The two primary methods for accomplishing this are through frame-wise application of an equivariant model (as described in } and through group-convolution over the entire space-time block (as described in (6 Both of these methods are verifiably flow-equivariant, however they are fundamentally different class of model than what have described in this work. They are not recurrent sequence models, and therefore intrinsically have finite temporal context or 'receptive field' which can be used when computing any output. By contrast, recurrent networks can theoretically have an infinite temporal context, if need be, through the maintenance of a hidden-state. Given this fundamental distinction between recurrent and non-recurrent networks which the subject of research beyond the domains of equivariance, we find it to be beyond the scope of this work to compare with these models explicitly. Instead, we propose flow equivariant RNNs as an extension of existing equivariant network theory to this class of models which maintain a hidden state and can operate in the online recurrent setting.
The list of prior work which can be included this category of 'flow equivariance without a hidden state" quite broad, since it encompasses most of the equivariant deep learning literature to date, however we list few notable examples here. The Lorentz equivariant work of Bogatskiy et al. [2020). Gong al. [022] is the most relevant, while other applied work has developed 3-D convolutional networks which are equivariant with respect to Galilean shifts (our translation flows) in the context of event-based cameras (Zhu al.) 2 or rotations over time in the context of medical imaging [Zhu et al.] 2024]. Early work developed Minkowski CNNs [Choy et al.] 2019]. which are equivariant with respect to 4D translations, thus making them equivariant to axis- axisneligned motions. Further, Clifford Steerable CNNs [Zhdanov et al. 2024) have also been developed to achieve Poincaré-equivariance on Minkowski spacetime. Related work on equivariance for PDE solving / forecasting has built dynamics models which are equivariant with respect to galilean transformations in the sense that the model is equivariant if the input vector field has a global additive constant [Wang et al. 2021]. While this is valid for neural networks applied to vector field data input, it is clearly not the same as method in more general settings. The method of Wang et al. [021] can be interpreted as viewing a dynamical system which has unknown global current introduced, while ours is better interpreted as viewing the dynamical system from a moving reference frame the two concepts are compatible and may even be combined.
D.2 'Statically Equivariant" Sequence Models
The second broad category of related work includes sequence models which are equivariant with respect to instantaneous static group transformations, but are not equivariant with respect to time- parameterized group transformations, as FERNN is. Examples in this category include [Azari] and Erdogmus 2022, Nguyen et al.] 2023 Basu et al., 2023]. which introduce equivariance to transformations such as statie rotations in sequence models including RNNs. For example, these models train on one frame of reference and then test on 'rotated' frame of reference. Our work can be seen to generalize these models to instead allow them be tested on 'rotating' frames of reference. This class of prior work can most readily be compared to the group equivariant RNN (G-RNN) we describe in $2] Other researchers have developed sequence to sequence models which are equivariant with respect to fixed permutations and demonstrated that this is beneficial in the context of language modeling [Gordon et al. 22020). Further, recent work has developed an equivariant sequence autoencoder which uses group convolutions in an LSTM to achieve static equivariance for the purpose of PDE modeling [Fromme al. 2025).
D.3 Neuroscience and Biologically Inspired Neural Networks
The study of symmetry has also grown increasingly relevant in the computational neuroscience literature. Zhang et al. (2022) have studied equivariant representations in biological systems in
31

---

terms of continuous attractor networks. Interestingly, these models also use convolutional recurrent dynamics; however, again, this work only considers static translations and not motion over time. Others such as Lindeberg (2033) have developed general theory for motion equivariant receptive fields in the visual system [Lindeberg 2025], J, and even built spiking recurrent networks which integrate such filters [Pedersen et al. 2025], yet these models are equivariant only in the encoder portion of the network, not in the recurrence as we propose for FERNNS.
One class of models which is highly related to the FERNN in both theory and implementation comes from line of biologically inspired work aiming to learn symmetries from data. One of the first models in this category. the Topographic VAE [Keller and Welling) [022] can be seen as similar to our FERNN but with an explicitly imposed translation flow of a single velocity in the latent space. This makes these models equivariant to input transformations which are isomorphic to the translation group on the integers modulo the capsule length. Interestingly, the authors find that by simply imposing this translation flow in the latent space, the model learns to encode dataset symmetries into these flows in order to better model the dataset. This seems to imply that simply imposing flow symmetries in sequence models without priori knowledge of the structure of the flow symmetries in the input may still be beneficial. Similar results were shown with the Neural Wave Machine [Keller and Welling 2023], where flows in the latent space were implemented implicitly through bias towards traveling wave dynamics. Finally, perhaps most interestingly. related work on traveling waves in simple recurrent neural networks (the wave- wave-RNN) [Keller et al. [024a] and SSMs [Keller 2025) has demonstrated that waves implemented through similar 'roll" operations have significant benefits for long-term memory in recurrent neural network architectures. Incredibly, these models are identical to translation flow-equivariant RNNS in implementation, but without any mention of equivariance, and applied to entirely different set of tasks. It is therefore of great interest study there is something unique to translation flows which benefit memory performance, or if similar performance benefits may be gained from any latent flow symmetry.
Finally, convolutional RNNs are becoming increasingly of interest in the computational neuroscience and 'NeuroAl' domains (Spoerer et al. [017]] with notable examples being trained as 'foundation models' for mouse visual cortex [Wang et al., 2025], J. and recently demonstrating state-of-ther of-the-art performance in modeling rodent somatosensory cortex [Chung et al., 2025]. Our work provides
new theoretical lens through which to study and build such models, offering potential novel insight
into biological neural systems.
D.4 Reference Frames in Neural Networks
As mentioned in $ one way to interpret the flow-equivariant RNN is that its hidden state lives in a number of moving reference frames simultaneously (one for each y E V). Thus, for moving inputs, the corresponding co-moving hidden state reference frame will see the input as stationary, and process it as normal. This idea of reference frames in neural networks is not new, and significant interesting related work should be noted.
Specifically, Spatial Transformer Networks [Jaderberg et al. 2016). and Recurrent Spatial Transformer Networks [Sønderby et al. 2015) can be seen to predict a frame of reference for given input and then switch to that reference frame to gain invariance properties. However, these models do not discuss moving reference frames. Other models have been built in this vein with respect to other symmetries, such as polar coordinate networks which are inherently equivariant with respect to scale and rotation [Esteves et al. 2018). Related in theory is the idea of Capsule Networks (Sabour et al.) 2017 (iinton et al.) 2018). These models do have an explicit notion of reference frames, similar to and use this to gain structured equivariant representation. but again this is defined only in the spatial context.
D.5 Broadly Related Work
More broadly, prior work has looked at the integration of recurrence and motion modeling specifically for vision [Wu et al. [021] Gehrig and Scaramuzza 2023]. These models do not have any mention of motion equivariance, and therefore are highly unlikely to provide the strong generalization benefits such as those that we present in this paper. Other work has studied ego motion for action recognition LL6pez-Cifuentes et al. [2020]; and relevant work has looked at equivariance in the context of object
tracking [Gupta et al. 2020 Sosnovik et al. 2020a
32

---

D.6 Equivariant Dynamical Systems
In the dynamical systems literature, equivariance is typically defined for autonomous (or homo- geneous) dynamical systems which have no 'input' or "driving force'. Abstractly, for system dir = f(x), we say the dynamical system equivariant if f(g x))
[x) for all E G. Then for any stt) that solves the differential equation, we also know that g .x()) solves the differential equation for the full group orbit 9 E G [Mochlis and Knobloch, 2007]. Similar to the equivariant neural network setting, we that the "output" of an equivariant dynamical system (interpreted the particular solution) transforms in predictable well-behaved manner for given transformation of the dz input. As the simplest example, a = x + x², has a sign flip symmetry in æ, meaning that if take the sign flipped version of the trajectory st((), the solution to the equation also inherits sign flip. is
straightforward to see from the above definition that if the function f(x) defining the time derivative is equivariant with respect G, then the system is considered equivariant, since the definitions equivalent.
In this work. are interested in recurrent neural networks, which generally operate in the "forced" or non-autonomous setting. The study of symmetries in non-autonomous dynamical systems has been previously explored in the control theory literature, and has proven highly valuable for the design of robust and high performing equivariant filters [Mahony et al. 2022]. In this setting, the dynamical system is defined as dt = f(xx, u) for some driving force u, and the equivariance property is then defined as f(r, v. = [[ 9 u). We see that our FERNNs indeed are equivariant non-autonomous dynamical systems by this definition.
The difference of our current work with this prior control-obsere work, is that Mahony et al. [2022] treats time-paramen paramettereeeedd symmetries known biases inside hand-written dynamical model and uses lifts/adjoint operators to keep the estimation eтror autonomous. The FERNN instead learns the dynamics, lifts the hidden state group-indest indexed field, and enforces equivariance by a simple group-com convoloovoo weight-sharing rule. This removes the need for handwritten dynamical models and allows learning flexible non-linear dynamics capabilities not covered by existing observer theory.
33

---

E Extended Results
E.1 In-Distribution Next-Step Prediction (Table Figure
In Figure we show the sequence predictions of the models presented Figure 3 Table trained on Translating MNIST V/2 and Rotating MNIST VAR, and evaluated the same flows. We that when tested in-distribution, all models appear to perform well from visual inspection.
In buttonbution Next-Step Prediction Rotating MNIST (22)
In Distribuiton Next-Step Prediction Translating MNIST (Vg')
G-RNN
Input
Input
Pradiction
Prediction
Error
Input
%
K
Input
Prediction
%
0
Prodiction
Error
Input
Input
Prediction
0
Prediction
Error
FERNN-16
FERNN- V
Input
8
8
8
%
Input
Prediction
8
8
8
%
Prediction
Error
3
Ab
23
23
Prediction
3
3
Ab
23
23
Error
Error
Input
Input
Prediction
Prediction
Error
FERNN-13
FERNN- V
Input
Input
Prediction
9
Prediction
Error
Error
Input
Input
Prediction
Prediction
Error
Input
Input
0
Prediction
Prediction
Figure 12: In-distribution sequence predictions for the models from Table 1 & Figure 3 trained on Rotating MNIST V2 (left) and Translating MNIST V. (right), evaluated on the same flows.
34

---

E.2 Length Generalization MSE vs. Forward Prediction Plot (Rotation)
we show the length generalization plot, analogous plot Figure (right), but for Rotating MNIST. We see that the length generalization performance gap is not as significant on Rotating MNIST compared with Translating MNIST. We suspect that this is due the accumulation of errors induced by repeated interpolation when performing rotation by small angles discrete grid. Despite this, we see that the FERNN-V still achieves strong generalization up 30-time steps forward, significantly outperforming the non-equivariant model.
In Figure 13
Rotation Flow MNIST Length Generalization
0.20
G-RNN
FERNN-V
0.16
MSE with Ground Truth
0.12
Training Length
0.04
0.00
10
20
30
40
Timestept t
Figure 13: MSE vs. Forward prediction horizon for models Rotating MNIST. Analogous plot to Figure{(right (right) but for Rotating MNIST.
35

---

E.3 Length Generalization Visualizations (Translation)
In Figure 14 we show the sequence predictions of the same models presented Table I & Figure 3 trained on Translating MNIST V/2 and Rotaiing MNIST V4 but tested in the length generalization setting. We plot the 70-forward prediction steps here, subsampled by half in time (giving 35 elements). We plot the ground truth sequences on top, and the forward predictions below, with the error (in blue-red color scheme) on bottom. We see the FERNNs significantly outperform the G-RNNs in length generalization on Translating MNIST, mirroring the quantitative results in Figures 13 &
Length Generalization Trans. MNIST (v?)
G-RNN
Input
22
Pred.
Error
Input
Pred.
Error
Input
R
A
A
A
Pred.
R
A
A
Error
FERNN- Y
9
5
SA
3 3
42
2h
d
Prad.
R
3
3
AA
Error
Input
RR
Pred.
Error
Input
Pred.
Eroi
FERNN- VIT
Input
Pred.
Error
Pred.
Erior
Pred.
Erior
Figure 14: Samples from models trained on Translating MNIST VI with training sequence lengths of 20, tested on sequence lengths 80. We plot the 70-forward prediction steps here, subsampled by half in time (giving 35 elements).
36

---

E.4 Velocity Generalization Visualizations
In Figure 15 we show the sequence predictions of the models presented in Figure S trained on Translating MNIST V and Rotating MNIST V/", but evaluated on 1/2 and V respectively. These figures show that FERNNs which are equivariant to flows beyond their training distribution are able automatically generalize these transformations at test time, achieving near perfect next-step prediction performance where non-flow equivariant RNNs fail.
Velocity Generalization Rotating MNIST
Velocity Generalization Translating MNIST
(Vtrain VIR Viest = Vift)
(Vtram VT* Viest = V221
G RNN
G-RNN
Input
+
X
X
+
Input
8 1 8
18
19 1Q 10
111
Prod.
X
3 8 SIRS
Error
Input
Pred.
6
Error
Error
Input
X
1X
L
X
7
X
Pred.
A
K K
Pred.
Error
Error
Input
4
Input
Pred.
Error
3
FERNN-V/
FERNN- V
Input
X
+
8
8
8
88
8
1
18 1 19
Pred.
+
X
X
X
Y
X
+
Pred.
8 8
8
18
18
18
18
19
Input
6
6 6
Input
Pred.
6
6
Error
Input
X
7
X
Input
Pred.
1
X
K
Error
6
Input
Input
Pred.
Error
Figure 15: Samples from models trained on Rotating MNIST Vi (left) and Translating MNIST VT
(right), and tested on sequences with significantly higher velocity flows (V/Rt and V/2" respectively). We see that the FERNN (bottom rows) has no problem generalizing to these new velocities despite never having seen them in training, while the G-RNN (top) fails. The specific flows plotted in this example are +40 step
deg for Rotating MNIST (left), and (#2, +2) for Translating MNIST.
37

---

E.5 Flow Equivariant RNNs and Variable Velocity Flows
By definition, flows formally maintain constant velocity over time: they exponentiate a single constant generator element of the Lie algebra. This property is in fact necessary to get the flow composition property described as (e's (v) ve(()) (v) = v and this flow composition property is essential to the equivariance proof of the FERNN (see $A.7
For real world sequences, this can be quite limiting, since most natural data contains flows of varying velocities throughout the sequence length. However, even though the theory only extends to constant velocity flows, in practice, we observe that sequences with variable velocity flows are still better modeled by FERNNs than non-flow-equivarte two additional non-constant-fior datasets below, and further provide intuition for why convolutional weights between lifted velocity dimensions (we call V V-mixing) may allow for better modeling interactions between velocity channels, such as those resulting from elastic object collisions.
counterparts. We provide empirical results of this on
E.5.1 Variable Velocity Moving KTH Action Recognition
First, we test our model on the same moving KTH action recognition dataset as in the main text, but with a random velocity switch halfway through the sequence. Since it is entirely random, is ultimately unpredictable to the model. Despite this, we find that the FERNN models still outperform the non-flow equivariant baselines by significant margin, indicating the practical benefits of flow equivariance hold even when the theoretical assumptions may be violated.
Model
Test Acc. 1
3D-CNN
0.59 + 0.02
G-RNN+
0.62 + 0.04
G-RNN
0.65 = 0.03
FERNN-VT
0.69 + 0.04
FERNN-V
0.69 = 0.02
Table 3:
FERNNs model variable velocity flows better than baselines. Test accuracy (mean + std) for models trained and tested on the Moving KTH dataset (V22") with random velocity changes halfway through each sequence.
E.5.2 Mixing Across the V Dimension: Bouncing MNIST
We additionally experiment with a 'bouncing' variant of the Translating MNIST dataset. Specifically, construct this dataset identically to the Translating MNIST dataset from the manuscript, but with the 2 digits now having clastic collisions with the image boundary and with each other. This causes digits to (predictably) change velocity at multiple points during the sequence own velocity, and the velocity of the other objects that they collide with. Thus, to forward predict these sequences, models must be able to model interactions between velocities. In our framework, this is precisely the role of inter- V terms of the flow convolution kernel of Equation 9
function of their
In order to have space for both digits and the boundary, we increase the canvas size to 28 x 48. We additionally draw tight bounding boxes around the digits (in white, value 1.0) and draw a one-pixel image boundary on the canvas (in gray, value 0.5). Such boundaries make it easier for the model to identify collisions. In Table below we report the results of the FERNN and the G-RNN baseline on this dataset, as well as the results of FERNN with convolutional kernels that are non-zero for the cross-V terms (denoted V V-Mixing).
Ultimately, we see that the FERNN model nearly halves the error compared with the G-RNN baseline, and the addition of velocity mixing parameters further improves the performance, despite having no guarantee of equivariance with respect to these highly complex flow transformations. Intuitively, we believe V-mixing most beneficial when there are dynamical computations that require the interaction of multiple different velocity features simultaneously to predict the next frame. In essence, one may think of the flow convolution as identifying collisions space, and then transferring activation between opposing velocity channels model the interaction. The constant velocity global
38

---

Model
Bouncing MNIST Test MSE + Std.
G-RNN
2.5c 2 2 3e
FERNN-VA VT
1.8c 2 3c-4
FERNN-V2 + V--Mixing
8.9e 3 9
Table 4: FERNNS can model elastic collisions, and V-mixing improves modeling flow interac- tions. Mean std over random seeds on variant of the translating MNIST dataset where digits bounce off each other and walls elastically.
flows in the Translating MNIST task of the main text require such inter-velocity features, and therefore we found no benefit practice to adding V-mixing to the main text results.
E.6 Moving KTH with Data Augmentation
As noted in the main text, the flowing MNIST dataset can be seen as already 'data augmented' since the velocities are randomly re-sampled for each training iteration. For the Moving KTH dataset however, this is not the case by default. In Table [ below, we show the corresponding results for the Moving KTH dataset when re-sampling the velocities each iteration, analogous 'data augmentation' by camera motion.
Model
Test Acc. f
w/ Data Augmentation
3D-CNN
0,626 = 0.02
0.742 = 0.01
G-RNN+
0.639 0.02
0.662 + 0.04
G-RNN
0.665 + 0.03
0.684 + 0.04
FERNN-V7
0.698 + 0.03
0.694 + 0.05
FERNN-V
0.716 0.04
0.751 + 0.01
Table 5: FERNNS still outperform non-equivariant baselines on Moving KTH dataset with data augmentation. Test accuracy (mean std) for models trained and tested the Moving KTH dataset (V22) with velocities re-sampled at each training iteration, analogous to data augmentation. We see that all models improve performance with data augmentation expected, and the 3D-CNN improves the most, however the FERNN still performs the best and significatly better than the comparable G-RNN non-low-equivarian baselines.
E.7 Larger Baselines
In our exploration we have found that increasing the number of channels of the non-equivariant baselines has led to only marginal increases in performance at best (MNIST), and significantly worse performance in other cases (KTH).
Concretely, on the KTH dataset, when multiplying the number of G-RNN channels by 25 (the number of flow generator elements, IV1, for FERNN-V;" we found that we needed to reduce the learning rate from 3e-4 to le-4 to stabilize training, and even then, the model reached only 54.6 + 1% test accuracy, auder-performing the original G-RNN baseline value 66.5% test accuracy. We believe this drop in performance is due to the significantly increased number of trainable parameters resulting in increased training difficulty and more overfitting. On Translating MNIST, the amount of memory usage required to fit these additional parameters and activations prevents from multiplying the number of channels by When multiplying the number of channels by (the maximum we could fit our GPUs), find that performance is only marginally improved from 8.1e-3 (the original 128 channels) to 2.0e-3 (1024 channels), however this is still significantly below the FERNN-V;" performance of 1.5e-4 4 (128 channels).
39

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we define equivariance with respect to flows $
Definition 3.1 and
introduce our flow equivariant RNN in $ We include proofs for all of this. We show benefits to training speed Figure length generalization in Figure 4 and velocity generalization
in Fig 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We candidly state limitations of the framework in S and openly discuss the poor performance of the model with respect implementation details in the text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
40

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We include proof sketches after each theorem in the main text, and a full appendix section ς dedicated to the formal proofs of each theorem and remark.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In $5 we outline the models and dataset precisely. In $B we outline the training details and exact architectural details. We provide the code in anonymous github repo
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
41

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code to reproduce all results is available at: https://gits com/ Neur Sourips-FERN/FFEEE FERNN/FERNN
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublic//guides/CddSSuuiiiis ins.ssabiissoooocoooopoooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ips.cc/
guwdes/ PwoddSSumiissiooooolo...ooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See experiment details $B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All figures and tables in the main text display results mean + standard deviation of performance over three random initializations of the model parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
42

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See $B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttpps:///euureeeeee newrips........ weeuurps.ccppublicee htc//pbblic/tthhccccceeeee.....
Answer: [Yes]
Justification:
There are no ethical concerns, the paper is anonymized properly.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our contribution is foundational research with only indirect applications. We mention briefly in the appendix the potential societal applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
43

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper releases no models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All packages and datasets are cited. See experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
44

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
45

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
46


Reviews:
[{"review_id": "zb2AOMmSCO", "reviewer": "Reviewer_vWda", "summary": "This paper presents recurrent neural networks that are equivariant w.r.t. given time-parameterized symmetry transformations on the input sequence, referred to as flows. For example, for a sequence of images, a symmetry transformation $\\psi: \\N, \\X \\to \\X$ could be such that $\\psi(t, x_t)$ translates input image $x_t$ by $t$ pixels. To achieve this, they use the same idea as in group convolutional networks, which is to lift the input space from $\\X$ to $\\X \\times G$, and in the case of recurrent networks, also lifting the hidden state from $\\Z$ to $\\Z \\times G$, where $G$ is the symmetry group. Experimentally, they show good performance in next step prediction for MNIST digits with constant motion and also for action classification in videos with constant camera motion.\n\n$$\n\\renewcommand{\\N}{\\mathbb{N}}\n\\newcommand{\\X}{\\mathcal{X}}\n\\newcommand{\\Z}{\\mathcal{Z}}\n$$", "questions": "None.", "limitations": "Some of the weakness that I've described could be added to the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* I like the topic of the paper and the problem that it studies (obtaining equivariance w.r.t. time-parameterized symmetries). I think it addresses an interesting under-explored topic. In this regard it is original as far as I am aware.\n* The paper seems technically sound. I did not read all the proofs in detail but I did not notice any issues in the equations, claims, or proof sketches.\n* The proposed solution makes sense and is connected to the idea from group convolutional layers [Cohen & Welling, 2016] for which there are a lot of research papers and positive empirical results.\n\n## Weaknesses\n\n* The authors mention that some of the experiments do not perform mixing across $G$ channels. Without mixing we will have a independent processing path for each $g$ and the model becomes equivalent to a kind of group averaging method, which is not the method proposed in this paper. In other words, mixing is a *core* component of the group convolution approach. Therefore, it is crucial to have some mixing. Group averaging methods should be included as a baseline (i.e., zero mixing).\n* G-equivariance is an orthogonal idea and it overcomplicates the notation and exposition. Saying that G-equivariance is different from time-parameterized equivariance is necessary and useful for the paper but, afterward, it should not be included in the equations for simplicity. If the authors like, they can move the more general setup to the appendix. This is a suggestion to make the paper digestible for a wider audience and not a major issue in my opinion.\n* There should be some pictures of the action recognition dataset after camera motion has been applied.\n* The authors motivate their method with camera motion but, in reality, camera motion is not a constant motion; it changes direction sometimes. If direction of motion is allowed to change at every step then the size of the symmetry group will be exponentially large and the proposed method won't work. This issue can perhaps be left for future works. \n* Data augmentation should also be included as a natural baseline and it can actually more closely match natural camera motion with changing directions. It is likely that most videos are centered and data augmentation would fix that. For the experiments in this paper it suffices to do augmentation with constant camera motion.\n\nThe most important issue I think is the mixing which invalidates the experiments. If one weakness is to be addressed it should be that one. I hope the authors spend some time thinking about this issue and address it and that they don't dismiss it. I'm also quite sure the numbers will improve. If the authors worry about computational difficulties, they can only add mixing to a fraction of *convolution* channels (not $G$ channels).\n\nSummary of things to do to improve paper:\n- Data augmentation baseline.\n- Group averaging baseline (no mixing).\n- Add some mixing across $G$.\n- Expand on limitations.", "was_revised": true, "final_justification": "The authors have implemented all of the changes that I suggested for improving the paper. I think the paper is currently fit for publication at neurips so I recommend accept.", "rebuttal": "### **Summary**\nWe thank Reviewer vWda for their comments, suggestions, and critical analysis of our work. We appreciate that they find our work *original*, *technically sound*, and that it *addresses an under explored topic*. We also appreciate that they acknowledge the connection of our work to Group Convolution [Cohen & Welling, 2016], supporting our belief that FERNNs lay the theoretical groundwork for exploiting a new type of time-dependent symmetric structure in data, just as Group Convolution did for 'static' symmetric structure. \n\nBelow, we reply to a few of the reviewer's comments and include results that directly address their suggested improvements regarding:\n- *Mixing across velocity ($V$) channels*\n- *Data augmentation baselines* \n- *Realistic non-constant camera motion* \n\nAs a high level summary of our response:\n1. We provide new results with mixing across $V$ channels for both the original MNIST data (where we see no improvement) and a new non-constant velocity 'bouncing MNIST' dataset (where we see improvement), and explain these results with a mechanistic hypothesis for the functional role of $V$-mixing. \n2. We clarify that our baselines on MNIST can already be understood as 'data augmented' from our dataset construction procedure. We provide new results on the KTH action recognition dataset with proper data augmentation and still find the FERNN performs best.\n3. We provide new results on both MNIST and KTH with non-constant velocity flows. We see that the FERNN still outperforms non-flow-equivariant baselines, despite the mismatch with theory, supporting the practical relevance of our approach. \n\n\n### **Mixing Across Velocity ($V$) Channels**\n\nWe thank the reviewer for highlighting the importance of mixing across $V$ channels, and encouraging us to add these experiments; we find the results quite illuminating. \n\nIn the table below we show the performance (Test MSE $\\pm$ std. over 3 seeds) of the G-RNN, original FERNN, and FERNN with $V$-mixing (as described in Eqn. 9, with a convolutional kernel of size 3 in the $V$ dimension). We find that on the original Translating MNIST dataset (first column), the results are not significantly different with $V$-Mixing added, and the model actually performs measurably worse. As mentioned in the main text (lines 248-255), the reason for not including $V$-mixing in our initial results is not due to computational limitations, but rather a conscious design choice to limit the propagation of equivariance error arising from the truncation of the infinite velocity group to a finite range. This is similar to what was found in prior work on scale-equivariance [Worall and Welling, 2019, Sosnovik et al., 2020b], and is likely the reason for the drop in performance. \n\nHowever, we believe we can also understand why $V$-mixing *does not improve* performance significantly here -- namely, we believe $V$-mixing is most beneficial when there are dynamical computations that require the *interaction of multiple different velocity features simultaneously* to predict the next frame. The constant velocity global flows in Translating MNIST require no such features. \n\nTo test this mechanistic hypothesis for the role of $V$-mixing, we provide new results on a dataset we call 'bouncing MNIST', identical to translating MNIST, but where MNIST digits now have elastic collisions with the image boundary and with each-other (we have now added two digits per image, and increased the image size to 28x48 correspondingly). These collisions require exactly the types of muli-velocity computations we believe $V$-mixing would help with. Experimentally, we validate this hypothesis in the second column of the table below, where the FERNN with $V$-Mixing achieves the lowest test error on this bouncing MNIST dataset. \n\n| Model                    | Translating MNIST (MSE) | Bouncing MNIST (MSE)          |\n| ------------------------ | ----------------------- | ----------------------- |\n| G-RNN                    | 1.4e-3 $\\pm$ 8.9e-6     | 2.5e-2 $\\pm$ 3.2e-4     |\n| FERNN-$V_2^T$            | **1.8e-4 $\\pm$ 2.6e-6** | 1.8e-2 $\\pm$ 3.1e-4     |\n| FERNN-$V_2^T$ + V-Mixing | 3.0e-4 $\\pm$ 2.8e-5     | **8.9e-3 $\\pm$ 9.3e-4** |\n\n### **Data Augmentation**\nWith respect to data augmentation, we appreciate the reviewer's suggestion; we include a general clarification of our data generation procedure below, followed by new results for the KTH data (both of which we will add to the main text). \n\n**Clarification:**\nSince the datasets we use in our paper have no natural motion, they are already 'augmented' in some way in order to generate the training sequences. The main design choice is then whether the velocities ('motions') are sampled once for each image and fixed throughout training, or if velocities are re-sampled at each epoch (each time the image is presented). This second option, we call 're-sampling', is more akin to standard data augmentation.\n\n**MNIST results are already 'data augmented' according to the 're-sampling' definition:**\nWe highlight that the MNIST results in our paper are already 'data augmented' according to the 'resampling' definition above. This is because the velocities are randomly sampled for each training example in the data loader by default, meaning each digit is observed with a different velocity at each epoch, equivalent to data augmentation. This is described briefly on lines 270-272, and more thoroughly in Supplementary Material Section B.2. The only setting where this augmentation is not applied is on the velocity generalization tasks, where it would not make sense by definition of zero-shot generalization. \n\n**KTH results are not 're-sampling' data augmented, new results below:**\nThe KTH (action recognition) results in Table 2 are not with 're-sampled' augmentations for each epoch, instead they are sampled once (with a fixed seed for all models) at the beginning of training. We appreciate the reviewer bringing attention to this distinction. In the table below we show the new results for Table 2 replicated with 're-sampled' data augmentation for each model. We see that the performance of all models improves, and the performance gap between models reduces, although the FERNN still outperforms the baselines by a narrow margin. \n\n| Model         | Test Acc (No Aug) | Test Acc + Data Aug   |\n| ------------- | ----------------- | --------------------- |\n| 3D-CNN        | 0.626 ± 0.02      | 0.742 $\\pm$ 0.015     |\n| G-RNN+        | 0.639 ± 0.02      | 0.662 $\\pm$ 0.037     |\n| G-RNN         | 0.665 ± 0.03      | 0.684 $\\pm$ 0.041     |\n| FERNN-$V_1^T$ | 0.698 ± 0.03      | 0.694 $\\pm$ 0.049     |\n| FERNN-$V_2^T$ | **0.716 ± 0.04**  | **0.751 $\\pm$ 0.009** |\n\nIn the camera ready version, we propose to include 're-sampled' data augmentation baselines in Figure 5 as well. We also propose to include non-augmented baselines on MNIST for completeness.  \n\n### **'Realistic' Non-Constant Camera Motion**\n\nThe reviewer comments that natural camera motion is not a constant velocity, and suggests that this may make the method less applicable to real-world data. We agree, and further believe that the study of the performance of FERNNs on non-constant velocity flows is certainly of interest to us as well. As a preliminary step in this direction, we first appeal to the results above on 'Bouncing MNIST' where we see that the FERNN still achieves a significant performance improvement over non-flow-equivariant counterparts, despite the variable velocity flows. \n\nSecondly, we provide new results on the KTH dataset where we simulate camera motion that changes its velocity vector randomly half-way through the sequence. We see in the table below that the FERNN still significantly outperforms the baselines. \n\n| Model         | KTH Variable Velocity, Test Acc $\\pm$ std. |\n| ------------- | ------------------------------------------ |\n| 3D-CNN        | 0.59 $\\pm$ 0.016                           |\n| G-RNN+        | 0.62 $\\pm$ 0.043                           |\n| G-RNN         | 0.65 $\\pm$ 0.034                           |\n| FERNN-$V_1^T$ | **0.69 $\\pm$ 0.036**                       |\n| FERNN-$V_2^T$ | **0.69 $\\pm$ 0.017**                       |\n\nIn future work, we believe that it may be possible to train a 'flow predictor' to estimate the current flow between any two time-steps, and apply only this flow to this hidden state. This would avoid the 'exponential growth' problem that the reviewer mentions, while still maintaining equivariance. We leave this as a potential future direction which is motivated and supported by the theory of our proposed work.\n\n### **Response to minor points:**\n- The reviewer requested pictures of the action recognition dataset with camera motion applied. We bring to the reviewer's attention that these are indeed already in Section B.7, Figure 6, of the supplementary material. We will move these to the main paper for the camera ready. \n- In our camera ready draft we have split Figure 4 into two separate figures and removed empty space, significantly improving readability, and addressing the Reviewer's minor formatting concerns. \n- In our camera ready we have expanded on the limitations of our model with respect to variable velocity flows, data augmentation, and computational scaling. \n- We sincerely appreciate the authors suggestion to help simplify our exposition by reducing background on G-Equivariance. We agree that the paper could be simplified to improve the exposition, and reducing background mathematical clutter is a good option to do so. We do however believe we need to include a minimal example of G-equivariance in the text since it is explicitly required as a subset of a flow-equivariant network, i.e. it is not possible to build a flow-equivariant model which is not also G-equivariant. \n\nWe thank the reviewer again for their suggestions and we welcome further discussion if we may have misunderstood any of their comments."}, {"review_id": "YiSe1WZi8j", "reviewer": "Reviewer_ngxM", "summary": "This paper proposes, analyzes, and proposes a model architecture for flow equivariance. Flow equivariance is symmetry that exists in the context of time series prediction where a transformation on the input of a function produce the same result as the same transformation to the output of the function. Examples of flow equivariant processes are next step prediction models and generalization over flow velocities. This paper states and analyzes flow equivariance and proposes a flow equivariant convolution and recurrence relation. The resulting FERNN (flow equivariant recurrent neural networks) are shown to have the desired equivariance properties and to improve performance on simple one-step and sequence prediction tasks performed on a \"flowing MNIST\" dataset.", "questions": "none", "limitations": "See my comments in strengths and weaknesses. It would be a major step to demonstrate this in the context of more challenging flow prediction problems.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n-- Flow equivariance seems to be a timely and important problem. The applications are significant and the approach formalizes a fundamental intuitive idea.\n\n-- The analysis is terrific. The g-rnn counter example is great. The mathematical development is right on target.\n\n-- The experiments bear out the claims of the paper. Indeed flow equivariance seems to help in sequence prediction tasks, at least in the simple \"flowing MNIST\" case considered.\n\n\nWeaknesses:\n\n-- The experimental domains (flowing MNIST) are very simple. It would be a big step to demonstrate this on slightly more complex flow prediction problems.\n\n-- It is unclear what are the computational requirements of the FERNN layers. Apparently, eq 9 does a convolution in time as well as space. This strikes me as potentially expensive. It would be great to understand this better and discuss.", "was_revised": true, "final_justification": "I think this is a strong paper. I think the main strengths are: 1) flow equivariance seems to be a timely and important problem with a range of applications. I also like that this is an intuitive idea at its core. 2) The mathematical grounding and analysis here is terrific. \n\nThe biggest weakness in my view is that the various MNIST experiments still feel like toy examples. However, I nevertheless think the strengths outweigh this weakness.", "rebuttal": "### **Summary**\nWe sincerely thank Reviewer ngxM for their strong support of our work. We appreciate that they acknowledge the *timeliness* and *importance* of introducing equivariance for sequence models at this point in time. We are very grateful that they find our analysis and exposition exceptional, and that they find our claims to be well supported by our experiments. \n\nIn our response, we will address the two weaknesses listed by the reviewer: namely, *simplicity of the modeled data flows*, and *the computational complexity of the FERNN model*.\n\nSpecifically, we first highlight a number of more complex flow prediction problems that we have tested FERNNs on since our submission of the original manuscript. Secondly, we clarify the computational requirements of FERNNs both theoretically and empirically.\n\n### **More Complex Flow Prediction**\nBelow we present results for three new sequence modeling tasks with more complex flow symmetries as part of the data generating process. We see in each case that the flow equivariant models maintain their advantage over non-flow-equivariant counterparts. \n\n#### **2-Digit Independently Flowing MNIST**\nThe first dataset we explore is a 2-digit variant of the flowing MNIST task. The task is identical to that in the main paper, except that 2 random digits are simultaneously placed on the canvas, and each digit is 'flowed' according to its own independent randomly sampled flow generator. Visually, this looks like two digits moving in two separate random directions (for translation), or two overlapping digits rotating at different angular velocities (for rotation). Below we report the forward prediction test set MSE in the same setting as in the main paper (10 steps of input, forward predict the next 10 steps).\n\n| Model                 | 2-Digit Translating MNIST (Test MSE) | 2-Digit Rotating MNIST (Test MSE) |\n| --------------------- | ------------------------------------ | --------------------------------- |\n| G-RNN                 | 8.1e−3 ± 6e−4                        | 4.0e−3 ± 5e−4                     |\n| FERNN-$V_1^T$/$V_2^R$ | 5.3e−4 ± 8e−5                        | 1.3e−3 ± 5e−5                     |\n| FERNN-$V_2^T$/$V_4^R$ | **1.5e−4 ± 2e−5**                    | **6.1e−4 ± 3e−5**                 |\n\n Although it is not apparent from this table, we have also explored the length generalization and velocity generalization of the FERNN on this 2-digit MNIST task and we find that it performs equally as well as for the 1-digit task, demonstrating that all of the benefits of flow equivariance appear to function as expected in this more complex flow setting. \n\nWhile still a toy dataset, this modification represents a significant new ability of the model -- namely, the ability to model multiple input flows simultaneously. One can imagine this as a highly simplified version of the visual transformations that would occur when one simultaneously combines camera motion with the independent motion of the video subject: while the camera motion may induce a global flow in one direction, the subject is free to move in an entirely independent direction (e.g. a camera pans to the left while a basketball player runs up on the screen). We are pleased to show that FERNNs can handle this type of scenario.\n\n\n#### **KTH with a random velocity switch halfway through the sequence**\nThe second more complex flow type we explore is non-constant-velocity flows. Specifically we explore the ability of FERNNs to model sequences where the flow velocity switches to a new random velocity mid-way through the sequence. We test this on the KTH action recognition dataset by modifying our simulated 'camera motion' with such a switch. We again see that FERNNs outperform their non-flow-equivariant counterparts significantly. \n\n| Model         | KTH Test Acc $\\pm$ std. |\n| ------------- | ----------------------- |\n| 3D-CNN        | 0.59 $\\pm$ 0.016        |\n| G-RNN+        | 0.62 $\\pm$ 0.043        |\n| G-RNN         | 0.65 $\\pm$ 0.034        |\n| FERNN-$V_1^T$ | **0.69 $\\pm$ 0.036**    |\n| FERNN-$V_2^T$ | **0.69 $\\pm$ 0.017**    |\n\n\n#### **'Bouncing MNIST', where 2-digits bounce off each other and the image boundary**\nThe third and perhaps most complex dataset we experiment with is a 'bouncing' variant of the flowing MNIST dataset. Specifically, we construct this dataset identically to the Translating MNIST dataset from the manuscript, but with 2 digits which now have elastic collisions with the image boundary and with each-other. This causes digits to change velocity at multiple points during the sequence as a function of their own velocity, and the velocity of the other objects that they collide with. Thus to forward predict these sequences, models must be good at modeling interactions between velocities. \n\nIn order to have space for both digits and the boundary, we increase the canvas size to 28x48. We additionally draw tight bounding boxes around the digits (in white, value 1.0) and draw a one-pixel image boundary on the canvas (in gray, value 0.5). In the table below we report the results of the FERNN and the G-RNN baseline on this dataset. We additionally report the results of a FERNN with convolutional kernels that are non-zero for the cross-$V$ terms (denoted $V$-Mixing). We refer the Reviewer to our discussion with Reviewer vWda for a more in-depth discussion of the role of $V$-mixing, and why it helps here. \n\n| Model                    | MNIST Test MSE $\\pm$ std. |\n| ------------------------ | ------------------------- |\n| G-RNN                    | 2.5e-2 $\\pm$ 3.2e-4       |\n| FERNN-$V_2^T$            | 1.8e-2 $\\pm$ 3.1e-4       |\n| FERNN-$V_2^T$ + V-Mixing | **8.9e-3 $\\pm$ 9.3e-4**   |\n\nUltimately, we see that  FERNNs again outperform the non-flow-equivariant counterparts on this dataset, despite having no guarantee of equivariance with respect to these highly complex flow transformations. We find the fact that $V$-mixing improves *modeling of object interactions* to be very exciting, and a direction certainly worth exploring further in future work. \n\n### **Computational Requirements of FERNN layers**\nFinally, we address the computational complexity of the FERNN layer. The flow convolution in Equation 9 performs a convolution over standard group elements ($g \\in G$, which can be thought of as spatial positions for the simple translation case), and also over flow generators ($\\nu \\in V$, which can be thought of as movement velocities). This can be implemented efficiently as a 3D convolution for the case of 2D images, and a 1D generator group -- (as we do in the above model denoted $V$-mixing). For sets with higher dimensional generators, one can use N-D convolutions, which are also implemented efficiently in frameworks like Jax. \n\nThe computational complexity of the model scales linearly with the size of the set of generators $V$ (which we denote $|V|$). Below we report the memory requirements and runtime per epoch as a function of the size of this set.  Note the G-RNN is equivalent to a FERNN with $|V| = 1$, i.e. $V = $ {$0$}.\n\n| Model | Memory Allocated (GB) | Runtime / Epoch (s) |\n| ------------------- | --------------------- | ------------------- |\n| G-RNN ($\\|V\\| = 1$) | 7.9 | 96.9 |\n| FERNN $\\|V\\| = 5$   | 16.3 | 186.8 |\n| FERNN $\\|V\\| = 9$   | 25.1 | 304.1 |\n| FERNN $\\|V\\| = 15$  | 38.3 | 537.9 |\n| FERNN $\\|V\\| = 19$  | 47.2 | 731.6 |\n\nIf we fit a linear model to these results, we obtain extremely good fits, and the following parameters:\n\n| Metric  | Intercept | Slope (per $V$ element) | R² (goodness‑of‑fit) |\n| ------- | --------- | ----------------------- | -------------------- |\n| Memory  | 5.4 GB    | + 2.2 GB                | 0.99                 |\n| Runtime | ‑12.0 s   | + 39.3 s                | 0.98                 |\n\nWe note that our current FERNN implementation is extremely computationally inefficient (looping over each flow generator), while a scan-based optimized implementation of this model is certainly possible and would dramatically reduce runtime and improve its scalability.  In our limitations and future work section, we describe how something like this may be possible within the State Space Model (SSM) framework. \n\n### **Conclusion**\nWe thank the reviewer for their engagement with our work and strong support. We welcome further discussion of any of the above points."}, {"review_id": "VdUdgAQycj", "reviewer": "Reviewer_scMk", "summary": "This paper introduced the concept of flow equivariance for sequence models where sequence data undergo flow actions defined by a set of generators and a time variable t. The paper discussed why recurrent neural networks equivariant to single-frame transformations are not flow-equivariant and derived flow-equivariant RNNs based on lifting and truncated group convolutions. Experiments are conducted on a flowing MNIST dataset and an action recognition dataset with artificial translation flows, and they show the effectiveness of flow-equivariance.", "questions": "1. Experiments on data with less restrictive assumptions. For example, with a time-varying flow or a flow that lands in between the discretized generators. \n\n2. Computational overhead from the introduced flow equivariance.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \nThe paper proposed a new type of equivariance that addresses the symmetry of sequential data with flows from specific generators. Good originality here. The notations, definitions, and derivations are overall clear. The visualization, especially figure 1 with a simplified example, helps convey the key idea. \n\nWeaknesses:\nThe designed network FERNN only works if the flow is generated from a predefined discrete set of generators, which is limiting, and not the case in most practical scenarios. The experiments are restricted with the following unrealistic assumptions: 1. the flow is spatially global; 2. the flow is of constant speed; 3. the flow generators are a discrete set small enough to be covered in the lifted group dimension. I think that at least the second condition can be relaxed for the proposed model. The authors could do experiments on data with non-constant flow. Overall, the flow model used in this paper is overly simplified, and it is hard for me to see its practical value. \n\nThe paper did not disclose the memory usage and computational resources needed for the models. Given the group convolution nature, I would anticipate significant overhead. Could you compare the FERNN, G-RNN, and regular RNNs?", "was_revised": true, "final_justification": "My concerns in the review are resolved. I recommend accepting this paper.", "rebuttal": "### **Summary**\nWe sincerely thank the Reviewer scMK for their time spent with our work, and for greatly helping us to improve the manuscript. We appreciate that they commend the *originality*, *clarity* and *exposition* of our work. \n\nIn our response, we directly address the Reviewer's primary concern about the practical applicability of our method. This concern appears to be due to the perception that the benefits of our proposed model are limited to *spatially global flows*, *constant speed flows*, and *flows which exactly match the model's small discrete set of generators*. \n\nThrough a suite of new results, we demonstrate that the *practical benefits* of FERNNs are not so limited, even if the theory does not yet extend this far. Explicitly, we present positive empirical results on:\n1. Non-global flows,\n2. Variable-velocity flows, \n3. Data flows landing between the model's generator set.\n\nIn each of these cases, we demonstrate FERNNs still excel over non-flow-equivariant baselines. \n\nWe subsequently include a quantitative analysis of the computational overhead of flow-equivariance, both in terms of memory and compute time, and how this scales with the size of the generator set $V$ (equivalent to a notion of the 'amount of flow equivariance'). \n\nFinally, we conclude with a comparison of the limitations raised by the Reviewer and those present in the original Group Equivariant Convolutional Neural Networks (G-CNNs) of Cohen & Welling (2016). We highlight that G-CNNs have delivered remarkable practical value since their introduction (as noted by Reviewer vWda) despite very similar limitations, and thus suggest that our work may be interpreted analogously as the introduction of a new foundational framework for sequence-model equivariance, opening the door for future work to address the theoretical limitations. \n\n### **Non-Global Flows**\nTo demonstrate that the benefits of FERNNs extend to non-spatially global flows, we provide new results on a 2-Digit Flowing MNIST task. The task is identical to that in the main paper, except that 2 random digits are simultaneously placed on the canvas, and each digit is 'flowed' independently. Visually, this looks like two digits moving in two separate random directions (for translation), or two overlapping digits rotating at different angular velocities (for rotation). The full image sequence is thus not describable by a single global flow (in the formal mathematical sense).\n\n| Model | 2-Digit Translating MNIST (Test MSE) | 2-Digit Rotating MNIST (Test MSE) |\n| ------- | ----------- | -------- |\n| G-RNN | 8.1e−3 $\\pm$ 6e−4 | 4.0e−3 $\\pm$ 5e−4 |\n| FERNN-$V_1^T$/$V_2^R$ | 5.3e−4 $\\pm$ 8e−5 | 1.3e−3 $\\pm$ 5e−5 |\n| FERNN-$V_2^T$/$V_4^R$ | **1.5e−4 $\\pm$ 2e−5** | **6.1e−4 $\\pm$ 3e−5** |\n\nAs can be seen, FERNNs still achieve roughly an order of magnitude lower forward prediction error than their non-flow-equivariant counterparts. This strong performance is explained by the fact that while each $\\nu$ channel flows globally, each pixel is simultaneously represented in the full lifted space of all $\\nu \\in V$. Therefore, the readout from the hidden state (in our case, max-pooling) can select an independent velocity channel to represent each spatial location.\n\n### **Variable-Velocity Flows**\nThe reviewer is correct to note that *constant speed is inherent to the mathematical definition of a flow*: flows exponentiate a single constant generator element of the Lie algebra. This property is in fact necessary to get the flow composition property described on line 125 ($\\psi_s(\\nu) \\cdot \\psi_t(\\nu) = \\psi_{s+t}(\\nu)$), and this flow composition property is essential to the equivariance proof of the FERNN (see line 230, & Eqn. 83 of Supplement).  \n\nHowever, even though the theory only extends to constant velocity flows, *in practice*, we observe that sequences with variable velocity flows are still better modeled by FERNNs than non-flow-equivariant counterparts. We provide tables below demonstrating this for two non-constant-flow datasets:\n\n**'Bouncing MNIST', where 2-digits bounce off each other and the image boundary.**\n\n| Model| MNIST Test MSE $\\pm$ std. |\n| -------- | ---------- |\n| G-RNN | 2.5e-2 $\\pm$ 3.2e-4       |\n| FERNN-$V_2^T$ | 1.8e-2 $\\pm$ 3.1e-4       |\n| FERNN-$V_2^T$ + V-Mixing | **8.9e-3 $\\pm$ 9.3e-4**   |\n\n\n**KTH with a random velocity switch halfway through the sequence.**\n\n| Model | KTH Test Acc $\\pm$ std. |\n| ------------- | ----------------------- |\n| 3D-CNN        | 0.59 $\\pm$ 0.016        |\n| G-RNN+        | 0.62 $\\pm$ 0.043        |\n| G-RNN         | 0.65 $\\pm$ 0.034        |\n| FERNN-$V_1^T$ | **0.69 $\\pm$ 0.036**    |\n| FERNN-$V_2^T$ | **0.69 $\\pm$ 0.017**    |\n\nWe see that on Bouncing MNIST, the FERNN model nearly halves the error, and the addition of velocity mixing features further improves the performance. We refer the Reviewer to our reply to Reviewer vWda for an extended discussion $V$-mixing and the dataset details. We similarly see on the variable velocity KTH dataset, FERNNs again significantly outperform their non-flow-equivariant counterparts. \n\n### **Handling Data Flows Outside the Model's Generator Set $V$**\nFinally, to demonstrate that FERNNs yield improved performance even when data 'lands between' the elements of the generator set, we generate a Translating MNIST dataset with x-velocity flows from -5 to 5 pix/step (0 y-velocity), and train models with equivariance to subsets of these data-generating flows. \n\nIn the table below, we report the test set MSE (mean over 3 seeds) computed for each x-velocity separately (Avg. at end). We see that *any* flow equivariance improves the average loss over non-flow-equivariant models, and further, when the data lands between the elements of the generator set (bolded), the FERNNs always achieve lower error than the G-RNN, sometimes by up to an order of magnitude, despite not being analytically equivariant to these flows. In the camera ready draft, we plan to include a figure of this data to improve readability. \n\n\n| Model | -5       | -4       | -3       | -2       | -1       | 0    | 1        | 2        | 3        | 4        | 5        | Avg  |\n| ----------- | -------- | -------- | -------- | -------- | -------- | ---- | -------- | -------- | -------- | -------- | -------- | ---- |\n| G-RNN | **1e-3** | **9e-4** | **1e-3** | **1e-3** | **7e-4** | 5e-4 | **6e-4** | **1e-3** | **1e-3** | **1e-3** | **1e-3** | 9e-4 |\n| FERNN-$V_{-1, 0, 1}^T$ | **7e-4** | **7e-4** | **7e-4** | **5e-4** | 3e-4     | 3e-4 | 4e-4     | **6e-4** | **7e-4** | **7e-4** | **7e-4** | 6e-4 |\n| FERNN-$V_{-5, 0, 5}^T$ | 2e-4     | **4e-4** | **9e-4** | **8e-4** | **4e-4** | 2e-4 | **4e-4** | **8e-4** | **1e-3** | **5e-4** | 2e-4     | 5e-4 |\n| FERNN-$V_{-3, 0, 3}^T$ | **8e-4** | **2e-4** | 3e-4     | **2e-4** | **2e-4** | 2e-4 | **2e-4** | **2e-4** | 3e-4     | **2e-4** | **8e-4** | 3e-4 |\n| FERNN-$V_{-5, -3, -1, 0, 1, 3, 5}^T$ | 9e-5     | **1e-4** | 8e-5     | **1e-4** | 6e-5     | 6e-5 | 8e-5     | **1e-4** | 8e-5     | **1e-4** | 8e-5     | 9e-5 |\n| FERNN-$V_{[-5, 5]}^T$ | 5e-5     | 5e-5     | 5e-5     | 5e-5     | 5e-5     | 4e-5 | 5e-5     | 5e-5     | 5e-5     | 5e-5     | 5e-5     | 5e-5 |\n\n\n### **Computational Overhead**\nFinally, we study the computational overhead of flow equivariance. Theoretically, both compute and memory should scale approximately linearly with the number of flow generators (denoted $\\|V\\|$). In the tables below, we validate this for FERNNs with different sized generator sets. Note the G-RNN is equivalent to a FERNN with $\\|V\\| = 1$, i.e. $V = ${$0$}.\n\n| Model | Memory Allocated (GB) | Runtime per Epoch (s) |\n| -------- | -------- | ----- |\n| G-RNN ($\\|V\\| = 1$) | 7.9 | 96.9 |\n| FERNN $\\|V\\| = 5$   | 16.3 | 186.8 |\n| FERNN $\\|V\\| = 9$   | 25.1 | 304.1 |\n| FERNN $\\|V\\| = 15$  | 38.3 | 537.9 |\n| FERNN $\\|V\\| = 19$  | 47.2 | 731.6 |\n\nIf we fit a linear model to these results, we obtain extremely good fits, and the following parameters:\n\n| Metric  | Intercept | Slope (per $V$ element) | R² (goodness‑of‑fit) |\n| ------- | --------- | --------- | --------- |\n| Memory  | 5.4 GB    | + 2.2 GB | 0.99 |\n| Runtime | ‑12.0 s   | + 39.3 s | 0.98 |\n\nWe note that our current FERNN implementation is extremely computationally inefficient (looping over each flow generator), while a scan-based optimized implementation of this model is certainly possible and would dramatically reduce runtime and improve its scalability. \n\n### **Conclusion**\nIn conclusion, we thank the reviewer for encouraging these new experiments. We believe many readers are likely to share the same concerns, so we are grateful for the opportunity to demonstrate empirically that these limitations are likely not as severe as theory alone might suggest. \n\nInterestingly, the limitations brought up by the reviewer are nearly identical to those of the original Group Equivariant Neural Networks of Cohen & Welling (2016) -- namely, *scalability,* *discrete group equivariance* and *the restriction to global transformations*. With time, however, researchers have developed methods to ameliorate these limitations: such as steerable equivariance (Weiler et al. 2019), addressing the discrete approximation limitation, and simultaneously scalability; and gauge equivariance (Cohen et al. 2019), addressing the local vs. global transformation limitation. In doing so, as Reviewer vWda has pointed out, group-equivariance has become incredibly practically relevant, including being used in highly popularized models such as AlphaFold 2. \n\nWe therefore suggest that our paper may be seen as a the first step in the introduction of a new theoretical foundation -- namely a foundation for dynamic equivariance in sequence models that opens the door for future work. We welcome further discussion."}, {"review_id": "leD8umtKgd", "reviewer": "Reviewer_oXJm", "summary": "This paper introduces a novel approach to integrate continuous, structured transformations, referred to as \"flows\" (like motion), into neural networks designed for sequential data. The authors define these flows as one-parameter Lie subgroups. They define \"flow equivariance\" and demonstrate that existing sequence models, even those built with components already equivariant to static symmetries, do not inherently possess this flow equivariance. The paper then presents their \"Flow Equivariant Recurrent Neural Network (FERNN)\" model , which adapts the RNN structure by incorporating principles from Group Equivariant Convolutional Neural Networks (G-CNNs) to achieve flow equivariance. Empirically, the authors show that FERNNs offer significant advantages in terms of training speed, generalization capabilities, and performance.", "questions": "* Line 84 - “..$(g\\cdot \\Phi)(y)$..” f is missing?\n* Equation 3 - $\\mathcal{W}^i_k$ appears in both convolution equations in Equation (3). However, its domain seems to change. in the lifting convolution as a function of $X$ and in the group convolution as a function of $G$. While this might be consistent if $G=\\mathbb{Z}^2\\subset \\mathbb{R}^2$ it's not generally well-defined for arbitrary domains $X$ and $G$.  Also, The superscript i is not defined in $\\mathcal{W}^i_k$  and same goes for the subscript $f_k$.\n* Regarding the model in Equation (3), a question arises: shouldn't there be an additional operation to project the features from the group space to some output space? This operation would likely resemble the lifting convolution.\n* Line 130-131 $f$ is missing from the equivariance definition?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths - \n* This paper presents a novel solution to a new problem: handling data with flow symmetry, a challenge that has not been extensively studied. The authors propose a unique approach and demonstrate why existing methods are inadequate for this specific problem.\n\nWeaknesses - \n* The paper's heavy reliance on notations and complex formulations, while understandable given its theoretical depth, makes it challenging to follow. Although simplifying intricate theoretical concepts can be difficult, the inclusion of a running example would significantly aid in motivating the proposed method and clarifying the distinction between static and flow transformations, a point that Figure 1 does not adequately address.", "was_revised": true, "final_justification": "The authors have successfully addressed my concerns about the paper's readability. Additionally, they conducted extensive new experiments to further validate their claims. For these reasons, I will keep my current rating of accept.", "rebuttal": "### **Summary**\n\nWe thank the reviewer for their time spent with our manuscript. We appreciate that they find our work to be *novel* in both goals and methods, agreeing with sentiments of the other reviewers. \n\nThe reviewer has suggested numerous areas for improvement of the clarity of our work, which we have taken to heart. Below we propose how we aim implement these clarifying modifications to our manuscript.  \n\n### **Clarity**\n\nWe sincerely thank the reviewer for their suggestions about how to improve the clarity of our submission. We readily admit that in pursuit of technical precision, and to match the conventions of prior equivariance literature, the manuscript has become less accessible to those outside the field. We believe this complication is not strictly necessary, and therefore appreciate the Reviewer encouraging us strive for greater clarity.\n\nTo facilitate communication of the main ideas and findings of our work, we plan to change the following:\n- We plan to add a visualization of the difference between static equivariance and flow equivariance to the main text.\n- We plan to add an additional visualization of the representation of the flow action in the FERNN latent space for the simple bump example (visualizing the result of the main Theorem, 4.1).\n- We plan to add visualizations of data flows to the main text (on KTH), for a more immediate conceptual picture of what a flow symmetry is. \n- Finally, as reviewer vWda has suggested, our detailed review of G-equivariance can be shortened and moved to the appendix to reduce extended mathematical complication of the main text. \n\nWe additionally plan to release a blog post accompanying the paper which will have the following clarifying properties:\n- a motivational introduction to the 'flow equivariance problem'.\n- a direct visual and motivational comparison between static equivariance and flow equivariance.\n- an extension of the 'bump' example which runs throughout the blog post, showing the difference between the G-RNN and FERNN, and the visualization of the flow action. \n\nWe propose to include a link to this blog post in the paper to assist those who stumble on the paper but prefer a gentler introduction. \n\n\n### **Questions**\n- The reviewer is correct that $f$ is missing in line 84. It should be:  $(g \\cdot \\Phi [f])(y) := \\rho_Y(g) \\Phi [f] (\\pi_Y(g^{-1}) y)$. We appreciate them mentioning this error. \n- We appreciate the reviewer bringing up where we missed defining indices $i$ and $k$ in Equation 3. These are the output and input channels for the convolution respectively, following the original group convolution notation of Cohen & Welling (2016). We have made this explicit in the camera ready draft, and have further made explicit the domains of the lifting and group convolutions, as the reviewer noted this may have been ill defined. \n- With respect to output mappings, this choice is typically left up to the modeler, and is not part of traditional equivariance frameworks. Again we defer to the formulation of Cohen & Welling (2016), of which our Equation 3 is merely a restatement. \n- In lines 130-131 the reviewer is correct that $f$ is again missing. It should be: $(\\psi (\\nu) \\cdot \\Phi  [f] )_t(y) := \\Phi_t [f] ( \\psi_t ( \\nu )^{-1} \\cdot y)$. We greatly appreciate the reviewer catching these notational inconsistencies and we have amended them in the camera ready draft. \n\nWe warmly welcome discussion on any further points the reviewer may have in regards to how we may improve the clarity of the text."}]

Author rebuttal:

Meta-review: This paper introduces Flow Equivariant Recurrent Neural Networks (FERNNs), extending equivariant network theory to handle time-parameterized transformations like motion in sequential data. Four reviewers provided consistent positive evaluations, acknowledging the novel theoretical contribution while raising concerns about experimental simplicity, practical applicability given restrictive assumptions about discrete generator sets and constant flows, computational overhead, and mathematical presentation clarity. The authors effectively addressed these concerns through comprehensive rebuttals including new experiments on variable velocity flows and bouncing scenarios that demonstrate benefits extend beyond theoretical assumptions, detailed computational scaling analysis, and commitments to improve presentation with visualizations and simplified exposition. While the work has limitations—requiring predefined discrete generators and theoretical guarantees only for constant velocity flows—it represents a solid foundational contribution that opens new research directions in dynamic equivariance, drawing appropriate parallels to early group equivariant CNNs that evolved from initially restrictive frameworks into practically valuable tools. The mathematical framework is rigorous, experimental validation adequately establishes core claims within scope, and the potential for inspiring follow-up research justifies acceptance, contingent on implementing the promised clarity improvements in the camera-ready version.
Decision: Accept (spotlight)