Title: Flow Equivariant Recurrent Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Equivariance

Recurrence

Sequence Model

Flow

Motion
Abstract: Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to integrate continuous, structured transformations, referred to as \"flows\" (like motion), into neural networks designed for sequential data. The authors define these flows as one-parameter Lie subgroups. They define \"flow equivariance\" and demonstrate that existing sequence models, even those built with components already equivariant to static symmetries, do not inherently possess this flow equivariance. The paper then presents their \"Flow Equivariant Recurrent Neural Network (FERNN)\" model , which adapts the RNN structure by incorporating principles from Group Equivariant Convolutional Neural Networks (G-CNNs) to achieve flow equivariance. Empirically, the authors show that FERNNs offer significant advantages in terms of training speed, generalization capabilities, and performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths - \n* This paper presents a novel solution to a new problem: handling data with flow symmetry, a challenge that has not been extensively studied. The authors propose a unique approach and demonstrate why existing methods are inadequate for this specific problem.\n\nWeaknesses - \n* The paper's heavy reliance on notations and complex formulations, while understandable given its theoretical depth, makes it challenging to follow. Although simplifying intricate theoretical concepts can be difficult, the inclusion of a running example would significantly aid in motivating the proposed method and clarifying the distinction between static and flow transformations, a point that Figure 1 does not adequately address.\n\nQUESTIONS:\n* Line 84 - “..$(g\\cdot \\Phi)(y)$..” f is missing?\n* Equation 3 - $\\mathcal{W}^i_k$ appears in both convolution equations in Equation (3). However, its domain seems to change. in the lifting convolution as a function of $X$ and in the group convolution as a function of $G$. While this might be consistent if $G=\\mathbb{Z}^2\\subset \\mathbb{R}^2$ it's not generally well-defined for arbitrary domains $X$ and $G$.  Also, The superscript i is not defined in $\\mathcal{W}^i_k$  and same goes for the subscript $f_k$.\n* Regarding the model in Equation (3), a question arises: shouldn't there be an additional operation to project the features from the group space to some output space? This operation would likely resemble the lifting convolution.\n* Line 130-131 $f$ is missing from the equivariance definition?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced the concept of flow equivariance for sequence models where sequence data undergo flow actions defined by a set of generators and a time variable t. The paper discussed why recurrent neural networks equivariant to single-frame transformations are not flow-equivariant and derived flow-equivariant RNNs based on lifting and truncated group convolutions. Experiments are conducted on a flowing MNIST dataset and an action recognition dataset with artificial translation flows, and they show the effectiveness of flow-equivariance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe paper proposed a new type of equivariance that addresses the symmetry of sequential data with flows from specific generators. Good originality here. The notations, definitions, and derivations are overall clear. The visualization, especially figure 1 with a simplified example, helps convey the key idea. \n\nWeaknesses:\nThe designed network FERNN only works if the flow is generated from a predefined discrete set of generators, which is limiting, and not the case in most practical scenarios. The experiments are restricted with the following unrealistic assumptions: 1. the flow is spatially global; 2. the flow is of constant speed; 3. the flow generators are a discrete set small enough to be covered in the lifted group dimension. I think that at least the second condition can be relaxed for the proposed model. The authors could do experiments on data with non-constant flow. Overall, the flow model used in this paper is overly simplified, and it is hard for me to see its practical value. \n\nThe paper did not disclose the memory usage and computational resources needed for the models. Given the group convolution nature, I would anticipate significant overhead. Could you compare the FERNN, G-RNN, and regular RNNs?\n\nQUESTIONS:\n1. Experiments on data with less restrictive assumptions. For example, with a time-varying flow or a flow that lands in between the discretized generators. \n\n2. Computational overhead from the introduced flow equivariance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes, analyzes, and proposes a model architecture for flow equivariance. Flow equivariance is symmetry that exists in the context of time series prediction where a transformation on the input of a function produce the same result as the same transformation to the output of the function. Examples of flow equivariant processes are next step prediction models and generalization over flow velocities. This paper states and analyzes flow equivariance and proposes a flow equivariant convolution and recurrence relation. The resulting FERNN (flow equivariant recurrent neural networks) are shown to have the desired equivariance properties and to improve performance on simple one-step and sequence prediction tasks performed on a \"flowing MNIST\" dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-- Flow equivariance seems to be a timely and important problem. The applications are significant and the approach formalizes a fundamental intuitive idea.\n\n-- The analysis is terrific. The g-rnn counter example is great. The mathematical development is right on target.\n\n-- The experiments bear out the claims of the paper. Indeed flow equivariance seems to help in sequence prediction tasks, at least in the simple \"flowing MNIST\" case considered.\n\n\nWeaknesses:\n\n-- The experimental domains (flowing MNIST) are very simple. It would be a big step to demonstrate this on slightly more complex flow prediction problems.\n\n-- It is unclear what are the computational requirements of the FERNN layers. Apparently, eq 9 does a convolution in time as well as space. This strikes me as potentially expensive. It would be great to understand this better and discuss.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nSee my comments in strengths and weaknesses. It would be a major step to demonstrate this in the context of more challenging flow prediction problems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents recurrent neural networks that are equivariant w.r.t. given time-parameterized symmetry transformations on the input sequence, referred to as flows. For example, for a sequence of images, a symmetry transformation $\\psi: \\N, \\X \\to \\X$ could be such that $\\psi(t, x_t)$ translates input image $x_t$ by $t$ pixels. To achieve this, they use the same idea as in group convolutional networks, which is to lift the input space from $\\X$ to $\\X \\times G$, and in the case of recurrent networks, also lifting the hidden state from $\\Z$ to $\\Z \\times G$, where $G$ is the symmetry group. Experimentally, they show good performance in next step prediction for MNIST digits with constant motion and also for action classification in videos with constant camera motion.\n\n$$\n\\renewcommand{\\N}{\\mathbb{N}}\n\\newcommand{\\X}{\\mathcal{X}}\n\\newcommand{\\Z}{\\mathcal{Z}}\n$$\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* I like the topic of the paper and the problem that it studies (obtaining equivariance w.r.t. time-parameterized symmetries). I think it addresses an interesting under-explored topic. In this regard it is original as far as I am aware.\n* The paper seems technically sound. I did not read all the proofs in detail but I did not notice any issues in the equations, claims, or proof sketches.\n* The proposed solution makes sense and is connected to the idea from group convolutional layers [Cohen & Welling, 2016] for which there are a lot of research papers and positive empirical results.\n\n## Weaknesses\n\n* The authors mention that some of the experiments do not perform mixing across $G$ channels. Without mixing we will have a independent processing path for each $g$ and the model becomes equivalent to a kind of group averaging method, which is not the method proposed in this paper. In other words, mixing is a *core* component of the group convolution approach. Therefore, it is crucial to have some mixing. Group averaging methods should be included as a baseline (i.e., zero mixing).\n* G-equivariance is an orthogonal idea and it overcomplicates the notation and exposition. Saying that G-equivariance is different from time-parameterized equivariance is necessary and useful for the paper but, afterward, it should not be included in the equations for simplicity. If the authors like, they can move the more general setup to the appendix. This is a suggestion to make the paper digestible for a wider audience and not a major issue in my opinion.\n* There should be some pictures of the action recognition dataset after camera motion has been applied.\n* The authors motivate their method with camera motion but, in reality, camera motion is not a constant motion; it changes direction sometimes. If direction of motion is allowed to change at every step then the size of the symmetry group will be exponentially large and the proposed method won't work. This issue can perhaps be left for future works. \n* Data augmentation should also be included as a natural baseline and it can actually more closely match natural camera motion with changing directions. It is likely that most videos are centered and data augmentation would fix that. For the experiments in this paper it suffices to do augmentation with constant camera motion.\n\nThe most important issue I think is the mixing which invalidates the experiments. If one weakness is to be addressed it should be that one. I hope the authors spend some time thinking about this issue and address it and that they don't dismiss it. I'm also quite sure the numbers will improve. If the authors worry about computational difficulties, they can only add mixing to a fraction of *convolution* channels (not $G$ channels).\n\nSummary of things to do to improve paper:\n- Data augmentation baseline.\n- Group averaging baseline (no mixing).\n- Add some mixing across $G$.\n- Expand on limitations.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nSome of the weakness that I've described could be added to the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}