Incorporating Symmetry into Deep Dynamics Models for Improved Generalization
Authors: Rui Wang
Robin Walters
Rose Yu
Venue: ICLR 2021 Poster (2021)
Keywords: deep sequence model
equivariant neural network
physics-guided deep learning
AI for earth science

Abstract: Recent work has shown deep learning can accelerate the prediction of physical dynamics relative to numerical solvers. However, limited physical accuracy and an inability to generalize under distributional shift limit its applicability to the real world. We propose to improve accuracy and generalization by incorporating symmetries into convolutional neural networks. Specifically, we employ a variety of methods each tailored to enforce a different symmetry. Our models are both theoretically and experimentally robust to distributional shift by symmetry group transformations and enjoy favorable sample complexity. We demonstrate the advantage of our approach on a variety of physical dynamics including Rayleigh–Bénard convection and real-world ocean currents and temperatures. Compare with image or text applications, our work is a significant step towards applying equivariant neural networks to high-dimensional systems with complex dynamics.

Published as a conference paper at ICLR 2021
INCORPORATING SYMMETRY INTO DEEP DYNAMICS MODELS FOR IMPROVED GENERALIZATION
Rui Wang"
Robin Walters
Computer Science and Engineering University of California San Diego, CA 92093 ruw020@ucsd. edu
Khoury College of Computer Science
Northeastern University
Boston, MA 02115
walterrshhorthhstemm edu
Rose Yu
Computer Science and Engineering University of California San Diego, CA 92093 roseyalucsd.edi
ABSTRACT
Recent work has shown deep learning can accelerate the prediction of physi- cal dynamics relative numerical solvers. However, limited physical accu- racy and an inability to generalize under distributional shift limits its applica- t by incorporating symmetries into convolutional neural networks. Specifically, we employ variety of methods each tailored enforce different symmetry. Our models are both theoretically and experimentally robust to distributional shift by symmetry group transformations and enjoy favorable sample complex- ity. We demonstrate the advantage of our approach on variety of physical dynamics including Bayleigh-Bénard Bénard convection and real-world ocean currents and temperatures. Compared with image or text applications, our work is sig- nificant step towards applying equívariant neural networks high-dimensional systems with complex dynamics. We open-source simulation, data and code at wwco////osee......oooo.ooocccooe STL Lab/Equivariant-
bility the real world. We propose to improve accuracy and generalization
INTRODUCTION
Modeling dynamical systems in order to forecast the future is of critical importance in wide range of fields including, e.g., fluid dynamics, epidemiology, economics, and neuroscience 2 45 22]14] Many dynamical systems described by systems of non-linear differential equations that difficult to simulate numerically. Accurate numerical computation thus requires long run times and manual engineering in each application.
Recently, there has been much work applying deep learning to accelerate solving differential equations 466 6] However, current approaches struggle with generalization. The underlying problem is that physical data has no canonical frame of reference to use for data normalization. For example, it not clear how rotate samples of fluid flow such that they share common orientation. Thus real-world -ntor--distribution test data is difficult to align with training data. Another limitation of current approaches low physical accuracy. Even when mean error is low, errors are often spatially correlated, producing different energy distribution from the ground truth.
We propose to improve the generalization and physical accuracy of deep learning models for physical dynamics by incorporating symmetries into the forecasting model. In physics, Noether's Law gives a correspondence between conserved quantities and groups of symmetries. By building neural network which inherently respects given symmetry, thus make conservation of the associated quantity more likely and consequently the model's prediction more physically accurate.
"qqual contribution

---

Published as a conference paper at ICLR 2021
A function ff is equivariant if when its input r is transformed by symmetry group g. the output is transformed by the same symmetry,
See Figure T for illustration. In the setting of forecasting, / approximates the underlying dynamical system. The set of valid transformations is called the symmetry group of the system.
By designing model that is inherently equivariant transformations of its imput, we can guarantee that our model generalizes automatically across these trans- formations, making it robust to distributional shift. The symmetries we consider, translation, rotation, uniform motion, and scale, have different properties, and thus we tailor our methods for incorporating each symmetry.
ro:(45)
rox(45)
Specifically, for scale equivariance, we replace the convolution operation with group correlation over the group generated by translations and rescalings. Our method builds on that of Worrall and Welling [51], with significant novel adaptations to the physics domain: scaling affecting time, space, and magni- tude; both up and down scaling; and scaling by any leverage real number. For rotational symmetries, the key insight of Cohen and Welling [9] that the in- put, output, and hidden layers of the network all acted upon by the symmetry group and thus should be treated as representations of the symmetry group. Our onation-equivarian model built using the flexible E (2) -NNN framework developed by Weiler and Cesa [49]. In the case of uniform motion, or Galilean transformation, we show the above methods are too constrained. We use the simple but effective technique of convolutions conjugated by averaging operations.
Figure 1: Illustration of equivariance of e.g.
f(x) = 2r with respect to = rro((T/4)
Research into equivariant neural networks has mostly been applied to tasks such as image classification and segmentation [27 50) 49). In contrast, design equivariant networks in a completely different context, that of a time series representing a physical process. Forecasting high-dimensional turbulence is significant step for equivariant neural networks compared the low-dimensional physics examples and computer vision problems treated in other works.
We test on simulated turbulent convection dataset and real-world ocean current and temperature data. Ocean currents are difficult to predict using numerical methods due to unknown external forces and complex dynamics not fully captured by simplified mathematical models. These domains are chosen as examples, but since the symmetries we focus on are pervasive in almost all physics problems, we expect our techniques will be widely applicable. Our contributions include:
We study the problem of improving the generalization capability and physical accuracy of deep learning models for learning complex physical dynamics such as turbulence and ocean currents.
We design tailored methods with theoretical guarantees to incorporate various symmetries, includ- ing uniform motion, rotation, and scaling, into convolutional neural networks.
When evaluated turbulent convection and ocean current prediction, models achieve signifi- cant improvement on generalization of both predictions and physical consistency.
For different symmetries, our methods have an average 31% and maximum 78% reduction in energy error when evaluated on turbulent convection with no distributional shift.
2
MATHEMATICAL PRELIMINARIES
2.1 SYMMETRY GROUPS AND EQUIVARIANT FUNCTIONS
Formal discussion of symmetry relies on the concept of an abstract symmetry group. We give a brief overview, for more formal treatment see Appendix A Lang 28
2

---

Published as a conference paper at ICLR 2021
A group of symmetries simply group consists of set G together with composition map o: G G - G. The composition map is required to be associative and have an identity 1 E G. Most importantly, composition with any element of G' is required to be invertible.
Groups are abstract objects, but they become concrete when we let them act. A group has an action set $ if there is action map : G S - S which is compatible with the composition law. We say further that S is G-representation the set 5 is a vector space and the group acts on by linear transformations.
Definition (invariant, equivariant). Let f: X + Y be a function and G be a group. Assume G acts on X and Y. The function / is G-equivariant if [gx) (2r) for all X and g E G. The function f is G-invariant if [gr) = f(x) for all x E and g E G.
2.2 PHYSICAL DYNAMICAL SYSTEMS
We investigate dynamical systems: Rayleigh-Bénard convection and real-world ocean current and temperature. These systems are governed by Navier-Stokes equations.
2D Navier-Stokes (NS) Equations. Let w (2x t) be the velocity vector field of a flow. The field w has two components (u, v), velocities along the x and directions. The governing equations for this physical system are the momentum equation, continuity equation, and temperature equation,
dw (w V)w 0t
1 - Vp+ -p++V>
OH KAH (w V,,, (DNs) Ōt
w
Po
where (ax, f) is temperature, p is pressure, is the heat conductivity, Pn is initial density, is the coefficient of thermal expansion, is the kinematic viscosity, and f is the buoyant force.
2.3 SYMMETRIES OF DIFFERENTIAL EQUATIONS
By classifying the symmetries of a system of differential equations, the task of finding solutions is made far simpler, since the space of solutions will exhibit those same symmetries. Let G be group
equipped with an action on 2-dimensional space X = I and 3-dimensional spacetime X = R².
Let V = R be G-represennationn Denote the set of all V-fields on X as Fy = (w: X - V smooth). Define Fy similarly to be V-fields on X. Then G has induced action on Fy by (zw)(c, 1) = g( ",r (() and on Fv analogously.
Consider a system of differential operators D acting on Fy. Denote the set of solutions Sol(D) C Fv. We say G is symmetry group D if G preserves Sol(D). That is, if yp is solution of then for all E G, ((p)) is also. In order to forecast the evolution of system D, we model the forward prediction function f. Let E Sol(D). The input
a collection of i snapshots at times ......,t denoted w; E Fa. The prediction function f: F - Fa is defined f(.,-k,.... w;-1) )- w, It predicts the solution at time based on the solution in the past. Let be symmetry group of D. Then for g E G, ((w) is also solution of D. Thus f(gw,,k.... ..... -W:=-1 gw,, Consequently, G-equivariant.
2.4 SYMMETRIES OF NAVIER-STOKER EQUATIONS
The Navier-Stoker equations invariant under the following five different transformations, Individ- ually each of these types of transformations generates group of symmetries of the system. The full list of symmetry groups of NS equations and Heat equations are shown in Appendix B.6
(B66
Space translation:
TeP wo 1) = w(x c, e R2,
Time translation: T
w(x, 1-11-)), 7), TER. Uniform motion: Tum w(x t) = t) +c, CE R?,
Rotation/Reflcction: Th' wwx t) = Rw It - R E 0(2),
Scaling: w 1) Awv 221, X E Ry
3

---

Published as a conference paper at ICLR 2021
3 METHODOLOGY
We prescribe equivariance by training within function classes containing only equivariant functions. Our models can thus theoretically guaranteed to be equivariant up to discretization error. We incorporate equivariance into two cate-of-thee of-the-art architectures for dynamics prediction, ResNet and U-net [48]. Below, we describe how we modify the convolution operation in these models for different symmetries G form four Equg-Res and four Equa-Unet models.
3.1 EQUIVARTANT NETWORKS
The key to building equivariant networks is that the composition of equivariant functions is equivariant. Hence, if the maps between layers of neural network equivariant, then the whole network will be equivariant. Note that both the linear maps and activation functions must be equivariant. An important consequence of this principle is that the hidden layers must also carry G-action. Thus, the hidden layers are not collections of scalar channels, but vector-valued G-representations.
Equivariant Convolutions. Consider convolutional layer Fzdm FReont with kernel K from fiell to a Redes -feeld. Let Ri and RRdout be G-representationss with action maps and Pout. respectively. Cohen et al. [ Theorem 3.3] prove the network is G-equivariant if and only
K (ggv) = Poul (g) (v))pin())
for all y E G.
A network composed of such equivariant convolutions is called steerable CNN.
Equivariant ResNet and U-net. Equivariant ResNet architectures appear in ariant transposed convolution, a feature of U-net, is implemented in 49 We prove in general that adding skip connections to a network does not affect its equivariance with respect to linear actions and also give a condition for ResNet or Unet to be equivariant in Appendix B.2
and equiv-
Relation Data Augmentation. To improve generalization, equivariant networks offer better performing alternative to the popular technique of data augmentation [13]. Large symmetry groups
normally require augmentation with many transformed examples. In contrast, for equivariant models, we have following proposition. (See Appendix B. for proof.)
Proposition 1. G-equivariant models with equivariant loss learn equally (up to sample weight) from any transformation g(8) of sample 8. Thus data augmentation does not help during training.
3.2 TIME AND SPACE TRANSLATION EQUIVARIANCE
CNNs are time translation- translation-equivarian equivariant as long as we predict in autoregressive manner. Convolu- tional layers are also naturally space manslation-eqqivariaa (if cropping is ignored). Any activation function which acts identically pixel-by-pixel is equivariant.
3.3 ROTATIONAL EQUIVARIANCE
To incorporate rotational symmetry, we model ff using (O())-equivarian convolutions and activations within the E (2) -CNN framework of Weiler and Cesa 49 In practice, we use the cyclic group G Cn instead G SO(2) as for large enough n the difference is practically indistinguishable due to space discretization. We use powers of the regular representation layers. The representation [R[,,, has basis given by elements of C, and C, -action by permutation matrices. It has good descriptivity since it contains all irreducible representations of Cn, and it is compatible with any activation function applied channel-wise.
R[C,,]" for hidden
3.4 UNIFORM MOTION EQUIVARIANCE
Uniform motion is part of Galilean invariance and is relevant to all non-relativistic physics modeling. For a vector field X: R² -> R and vector c E R², uniform motion transformation is adding a constant vector field to the vector field (()), Tum (T)m(X((u) (X)(u) X(u) c, R². By the following corollary, proved in Appendix (..3] enforcing uniform motion equivariance as above by requiring all layers of the CNN to be equivariant severely limits the model.
4

---

Published as a conference paper at ICLR 2021
Corollary 2. If is CNN alternating between convolutions fs and channel-wist activations o; and the combined layers 02 o fr are uniform motion equivariant, then f is affine.
To overcome this limitation, we relax the requirement by conjugating the model with shifted input distribution. For each sliding local block in each convolutional layer, we shift the mean of input tensor to zero and shift the output back after convolution and activation function per sample. In other words, if the input is P Xxxd and the output is @bxdout ( K) for one sliding local block, where b is batch size, is number of channels, is the kernel size, and K' the kernel, then
μ. = Meanjji (P Pijht); P
Paja ps Di Qij + p4:
(2)
This will allow the convolution layer be equivariant with respect uniform motion. If the input is a vector field, we apply this operation to each element.
Proposition 3. A residual block f(x))
uniform motion equivariant if the residual connection
is uniform motion invariant.
By the proposition 3 above that is proved in Appendix B.3 within ResNet, residual mappings should be invariant, not equívariant, to uniform motion. That is, the skip connection f(i++2) equivariant and the residual function f(i.it1) should be invariant. Hence, for the first layer in each residual block, we omit adding the mean back to the output Qij. In the case of Unet, when upscaling, we pad with the mean preserve the overall mean.
3.5 SCALE EQUIVARIANCE
Scale equivariance in dynamics is unique the physical law dictates the scaling of magnitude, space and time simultaneously. This is very different from scaling in images regarding resolutions [51]. For example, the Navier Stokke equations preserved under specific scaling ratio of time, space, and velocity given by the transformation
Tx: w(x t) + AwaAx, A?t),
where E Ry.0 We implement two different approaches for scale equivariance, depending on whether we tie the physical scale with the resolution of the data.
Resolution Independent Scaling. We fix the resolution and scale the magnitude of the input by varying the discretization step size. An input E Fk with step size (z(w) and (,ww) can be scaled w' = T (w) = Aw by scaling the magnitude of vector alone, provided the discretization constants now assumed be A, 1/AA,(w) and A,(w') 1/>2A, (wr). We refer to this as magnitude equvariance hereafter.
To obtain magnitude equivariance, divide the input tensor by the MinMax scaler (the maximum of the tensor minus the minimum) and scale the output back after convolution and activation per sliding block. We found that the standard deviation and mean L2 norm may work well but not as stable as the MinMax scaler. Specifically, using the same notation as in Section 3.4
a: MinMaxjat (P Piykt); P Pijki Pijka/o
Qi
O..
(4)
Resolution Dependent Scaling, the physical scale of the data is fixed, then scaling corresponds to change in resolution and time step size. To achieve this, replace the convolution layers with group correlation layers over the group G = (R :) x (R22 +) of scaling and translations. In convolution, translate kernel K across input as such v(p) Eyez w(p 9) (q)). The G-correlation upgrades this operation by both translating and scaling the kernel relative to the input,
v(p. s, s.
Σ
Aw(Ap + q, Xt, X s,t,
(5)
AER
where s and t denote the indices of output and input channels respectively. We add an axis to the tensors corresponding the scale factor μ. Note that we treat the channel as time dimension both with respective to our input and scaling action. As a consequence, as the number of channels increases in the lower layers of Unet and ResNet, the temporal resolution increases, which is analogous to temporal refinement in numerical methods (24 3T). For the input i of first layer where vü has no levels originally, w(p, 8, A) Xux X2s).
5

---

Published as a conference paper at ICLR 2021
Our model builds on the methods of Worrall and Welling [51], but with important adaptations for the physical domain. Our implementation of group correlation equation direecly incorporates the physical scaling law equation 3 of the system equation Pxs] This affects time, space, and magnitude. (For heat, we drop the magnitude scaling.) The physical scaling law dictates our model should be equivariant to both up and down scaling and by any À E Ryo. Practically, the sum is truncated to 7 different 1/3 S À < 3 and discrete data is continuously indexed using interpolation. Note equation demands we scale anisotropically, i.c. differently across time and space.
4 RELATED WORK
Equivariance and Invariance. Developing neural nets that preserve symmetries has been funda- mental task in image recognition [12
49
9
7 29;
27 3 52
10 19 50:
16 42) But these models
have never been applied to forecasting physical dynamies. Jaiswal et al. 23 Moyer al. (37
proposed approaches to find representations of data that are invariant to changes in specified factors, which is different from our physical symmetries. Ling al. [30] and Fang al. [T7] studied tensor invariant neural networks to learn the Reynolds stress tensor while preserving Galilean invariance, and Mattheakis M4 embedded even/odd symmetry a function and energy conservation into neural networks to solve differential equations. But these two papers are limited to fully connected neural networks. Sosnovik et al. [44] extend Worrall and Welling [51] to group correlation convolution. But these two papers are limited to 2D images and are not magnitude equivariant, which is still inadequate for fluid dynamics. Bekkers [4] describes principles for endowing a neural architecture with invariance with respect Lie group.
Physics-informed Deep Learning. Deep learning models have been used often to model physical dynamics. For example, Wang et al. [48] unified the CFD technique and U-net to generate predictions with higher accuracy and better physical consistency. Kim and Lee [25] studied unsupervised generative modeling of turbulent flows but the model is not able to make real time future predictions given the historic data. Anderson et al. 1 designed rotationally covariant neural network for learning molecular systems. Raissi et al. [40 41 applied deep neural networks to solve PDEs automatically but these approaches require explicit input of boundary conditions during inference, which are generally not available in real-time. Mohan et al. 335 proposed a purely data-driven DL model for turbulence, but the model lacks physical constraints and interpretability. Wu et al. (53 and Beucler al. [5] introduced statistical and physical constraints in the loss function to regularize the predictions of the model. However, their studies only focused on spatial modeling without temporal dynamics. Morton et al. (6 incorporated Koopman theory into encoder-decoder architecture but did not study the symmetry of fluid dynamics.
Video Prediction. Our work is related to future video prediction. Conditioning on the observed
frames, video prediction models are trained to predict future frames, e.g., [33 18 54
47
39
18]
Many of these models are trained on natural videos with complex noisy data from unknown physical processes. Therefore, it is difficult to explicitly incorporate physical principles into these models. Our work is substantially different because we do not attempt to predict object or camera motions.
5 EXPERIMENTS
We test models on Eayleigh-Bénard convection and real-world ocean currents. We also evaluated on the heat diffusion systems, see Appendix [ for more results. The implementation details and a detailed description of energy spectrum error can be found in Appendices D B.7
Evaluation Metrics. Our goal is to show that adding symmetry improves both the accuracy and Root Mean Square Error (RMSE)
the physical consistency of predictions. For accuracy, between the forward predictions and the ground truth over all pixels. For physical consistency, we calculate the Energy Spectrum Error (ESE) which is the RMSE of the log of energy spectrum. ESE can indicate whether the predictions preserve the correct statistical distributions of the fluids and obey the energy conservation law, which critical metric for physical consistency.
Experimental Setup. ResNet 20 and U-net 43 are the best-performing models for our tasks [48] and are well-suited for our tasks. Thus, we implemented these two convolutional architectures equipped with four different symmetries, which we name Equ-ResNes (U-net). ). We use rolling window approach to generate sequences with step size 1 for the RBC data and step size 3 for the

---

Published as a conference paper at ICLR 2021
Table 2:
The RMSE and ESE of the ResNet (Ünet) and four Equ-ResNets (Unets) pre- dictions on the original and four transformed test sets of Rayleigh-Bénard Convection. Augm is ResNet (Unet) trained on the augmented training set with additional samples applied with random transformations from the relevant symmetry group, Each column contains all models' prediction errors on the original test set and four different transformed test sets.
Root Moan Square (10or(10")
Energy Spectrum Errors
Orig
UM
Mag
Rot
Scale
Orig
UM
Mag
Rot
Scale
0.67:0024
2.94=0.84
4.30+:1.2
3.46:1-.39
199:20.16
0.46:00199
0.56::0.29
0.26::0114
159=0042 4.32+22.33
Augm
1.10
1,54:10.12 0.92:10.09
1.01 10.11
1.37::0.02
1.1.10.32
1.92:0.21 1.55
Rquux
0.71:1:.26
0.31.0.111
0.3310.11
Bquxaa
0.69=00244
0.67=20.14
0,34=0.09
0.19::0.02
Equaat
0.65110.2
0.76
0.31
1.23 0.04
BqUscal
0.70:10.02
0,85
0.44:0.22
0.68:10.26
U-net
0.64:1:.24
2.27
3.59.11.04
2.78.10833
166110.17
0.501:0.04
0.34
0.55.10.05
0.91
4.25
Augm
0.75=0.28
1.33
0.86+20.04
1.11
0,96+0.23
0.44::0.11
1.24=0.04
1.47=0.11
Bquuxx
0.68+0.26
0.71=0024
0.23+0006
0.14+0005
Bquing
0.67::0.11
0.68-10.14
0.42=00.44
0.34:10.06
Equnitt
0.68.10.25
0.74
0.11:10.02
1.16 0.05
RqUgcca
0.69:10.13
0.90
0.45
0.89:1..29
ocean data. All models predict velocity and temperature fields up to 10 steps ahead auto- regressively. We use the MSE loss function that accumulates the forecasting errors. We split the data 60%-20%-20% for raining-validation-eee across time and report mean errors over five random
5.1 EQUIVARIANCE ERRORS
The equivariance errors can be defined as F.F((=)) - S(T(x))), where x is input, is a neural net, is a transformation from a symmetry group. We empirically measure the equivariance errors of all equivariant models we have designed. T of ResNet and the transformed Rayleigh-Bénard Convection test sets. See more details in Appendix B.5
the equivariance errors The transformation I' is sampled in the same way as we generated
5.2 EXPERIMENTS ON SIMULATED RAYLEIGH- BÉNARR CONVECTION DYNAMICS
Data Description. Rayleigh-Bénard Convection occurs horizontal layer of fluid heated from below and is a major feature of the EI Niño dynamics. The dataset comes from wwoddimensional turbulent flow simulated using the Lattice Boltzmann Method [8 with Rayleigh number 2.5 10 We divide each 1792 x 256 image into 7 square subregions of size 256 x 256, then downsample to 64 x pixels. To test the models' generalization ability, we generate additional four test sets I) UM: added random vectors drawn from U); 2) Mag: multiplied by random values sampled from (0, 2); 3) Rot: randomly rotated by the multiples of т T/2; 4) Scale: scaled by À sampled from (1155, 2). Due to lack of a fixed reference frame, real-world data would be transformed relative to training data. We use transformed data to mimic this scenario.
Prediction Performance. Table shows the prediction RMSE and ESE on the original and four transformed test sets by the non-equivariant ResNet (Unet) and four Nests (Unets). Augm is ResNet (Unet:) trained on the augmented training set with additional samples with random transformations applied from the relevant sym- metry group. The augmented training set contains additional transformed samples and is three times the size of the original training set. Each column contains the prediction errors by the non-equivariant and equivariant models each test set. On the original test set, models have similar RMSE, yet the cquivariant models have lower ESE. This demonstrates that incorporating symmetries preserves the representation powers of CNNs and even improves models' physical consistency.
Table 1: Equivariance rors of ResNet (Unets)
Er-
and
(Unets).
EEr(10")
UM
Mag
Rot
Scale
ResNets
2010
1.885
5.895
1.658
Equiseoters
0.0
0.0
1.190
0.579
Uneta
1,070
0.200
1.548
1.809
Equinees
0.0
0.0
0.794
0.481
On the transformed test sets, performs even much better than Augm-ResNett (Unets). . This demonstrates the value of equiv- ariant models over data augmentation for improving generalization. Figure 2 shows the ground truth
that ResNet (Unet) fails, while Equ-ResNets (Unets)
7

---

Published as a conference paper at ICLR 2021
Target
ResNet
Fa
Target
ResNet
Equater
Target
ResNet
Equaee
Target
ResNet
Equigga
t
t=
Figure 2: The ground truth and the predicted velocity norm fields || at time step 1, 5 and 10 by the ResNet and four Equ-ResNets the four transformed test samples. The first column is the target, the second is ResNet predictions, and the third is predictions by Seuu-ResNets.
and the predicted velocity fields at time step 1, 5 and 10 by the ResNet and four Equ-ResNets. on the four transformed test samples.
Generalization. In order to evaluate models' generalization ability with respect the extent distributional shift, we created additional test sets with different scale factors from _ to 1. Figure 3 shows ResNet and Equisaa ResNet prediction RMSEs (left) and ESEs (right) on the test sets upscaled by different factors. We observed that E is very robust across various scaling factors while ResNet does not generalize.
Table 3: Performance compar- ison on transformed train and sets.
RMSE
ESB
ReeNet
1.03:0.05
0.96:t0.10
2quax
0.69:10.01
0.35:10.13
also compare ResNet and Equ-ResNet when both train and test sets have random transformations from the relevant symmetry group applied to each sample. This mimics real-world data in which each sample has unknown reference frame. As shown in Table 3 shows Equ -ee--Reeeetttt outperforms ResNet on average by 34% RMSE and 40% ESE.
ReaNet
1.00+0.22
0.55:t0.11
Equiag
0.75:10.04
0.39:10.02
ResNat
118810.00
1.21:10.04
Equaee
0.77:10.01
0.68:1001
ResNet
0.92.10.01
1.34
0.74:10.03
1.02:10.2
Target
ResNet
EqUIM
Equiwa
Fou
Equ
Fastlet
Modlet
3.0
3.5
10
Upecal Factor 1300
Usscaling (1,5)
Figure 3: Left: Prediction RMSE and ESE over five runs of ResNet and Equsea1 RResNet on the Rayleigh-Bénne Bénard Convection test set upscaled by different factors. Right: The ground truth and predicted ocean currents |/l|z by ResNet and four Equ-ResNets on the test set of future time.
5.3 EXPERIMENTS ON REAL WORLD OCEAN DYNAMICS
Data Description. We the reanalysis We selected an area from each of the Atlantic, Indian and North Pacific Oceans from 01/01/2016 to 08/18/2017 and extracted 64 64 sub-regions for experiments. The corresponding latitude and longitude ranges for the selected regions are (-44~-233, 25--6),, (55~76, -39~--18 18) and (-174--113, 5~266 respectively. We not only test all models the future data but also on a different domain (-180~-159) -40~-59) in South Pacific Ocean from 01/01/2016 to 12/15/2016.
current velocity data generated by the NEMO ocean
engine 332
Prediction Performance. Table ssoow the RMSE and ESE of ResNets s (Unets), and equiv- ariant Equ-ResNets (Unets) on the test sets with different time range and spatial domain from
'The data available at https://resourcess mari coperni su/?opt onncom wwwvvieeeeettilsssppoooo
8

---

Published as a conference paper at ICLR 2021
the training set. All the equivariant models outperform the non-equivariant baseline RMSE, and Equseei -Reset achieves the lowest RMSE. For ESE, only the E -ResNet (Unet) is worse than the baseline. Also, it is remarkable that the Equao: models have significantly lower ESE than others, suggesting that they correctly learn the statistical distribution of ocean currents.
Comparison with Data Augmentation. We also compare Equ-ResNets (Unets) ResNets (Unets) that are trained with data-augmentation (Augm) in Table f In all cases, equivariant models outperforms the baselines trained with data augmentation. We find that data augmentation sometimes improves slightly on RMSE but not as much as the equivariant models. And, in fact, ESE is uniformly worse for models trained with data augmentation than even the baselines. In contrast. the equivariant models have much better ESE than the baselines with or without augmentation. We believe data augmentation presents a trade-off in learning. Though the model may be less sensitive to the various transformations we con- sider, we need to train bigger models longer on many more samples. The models may not have enough capacity to learn the symmetry from the augmented data and the dynamics of the fluids at the same time. By comparison, equivariant architectures do not have this issue.
Figure 3 shows the ground truth and the predicted ocean currents at time step 1,5, 10 by different mod- els. We can see that equivariant models' predictions more accurate and contain more details than the baselines. Thus, incorporating symmetry into deep learning models can improve the prediction accuracy of ocean currents. The most recent work on this dataset de Bezenac et al. [15) which combines a warping scheme and a U-net to predict temperature. Since our models can also be applied to advection-diffusion systems, we also investigated the task of ocean temper- ature field predictions. We observe that Equg Unet performs slightly better than de Bezenac et al. 15 For additional results, see Appendix
Table 4: Prediction RMSE and ESE com- parison on the two ocean currents test sets.
RMSE
ESE
Testasn
Testaie
Testaosain
ResNet
0.71+:0077
0.7244004
0.83++0066
0.75+0011
Augman
0.701:0.01
0.70:10:07
1.06:1:0066
1.06:1001
Augmiue
0.76++0.02
0.71 0.71
1.08
1.05+088
Augmauu
0.73::01 0.69::001
0.94::0011
0,86:2011
Augmssas
0.97:20.06 Augmesa:0099.10000
0.92:10.04
0.85:20.03
0.95
Equex
0.68110.06
0.68:10.16
0.75:10.06
0.73:1008
0.66+t-014
0.68--0.11
0.84-+0.04
0,85++0.14
Equaut
0.69::0011
0.70+:0.0
0.43-:..15
0.28
Equacul
0.63-1-002
0.68::0.21
0.44-10.05
0.42:40.12
U-net
0.701:0.13
0.73110.0
0.77:10.12
0.7310007
Augmar
0.68++0.02
0.68 +:001
0.85-+0004
0,83+0004
Augmaug
0.69::002
0.6740030
0.78:10.03
0.86+:002
Augmaui
0.79::001
0.70-2001
0.79:0001
0.78
Augittzaa
0.71:1001 0.77::0.02
0.84:0001
0,77:002
CONCLUSION AND FUTURE WORK
2que
0.66:10.10 0.67
0.73:10.03
0.82:1013
0.63:::05
0.66
0.74:t0.05
0,79+0004
Bquau. 0.68
0.69
0.42::002
0.47:0007
We develop methods improve the generalization of deep sequence models for learning physical dynamics. We incorporate various symmetries by designing equiv- ariant neural networks and demonstrate their superior performance on 2D time series prediction both theoreti-
ZqUseell 0.65+:0.99 0.69
0.45++0.13 0.43++0005
cally and experimentally. Our designs obtain improved physical consistency for predictions. In the case of transformed test data, our models generalize significantly better than their non-equivariant counterparts. Importantly, all of our equivariant models can be combined and can be extended to 3D cases. The group also acts on the boundary conditions and external forces a system D. If these are G-invariant, then the system D is strictly invariant as in Section 2.3 If not, one must consider a family of solutions (g)))) to retain equivariance. To the best of our best knowledge, there does not exist single model with equivariance to the full symmetry group of the Navier-Stokes equations. It is possible but non-trivial, and we continue to work on combining different equivariances. Future work also includes speeding up the the scale-cquivariant models and incorporating other symmetries into DL models.
ACKNOWLEDGMENTS
This work was supported in part by Google Faculty Research Award, NSF Grant #2037745, and the U. S. Army Research Office under Grant W91 INF-20-1-0333 The Titan Xp used for this research was donated by the NVIDIA Corporation. This research used resources of the National Energy Research Scientific Computing Center, DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-ACO2-OSCHI 1231. We also thank Dragos Bogdan Chirila for providing the turbulent flow data.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
[1] Brandon Anderson, Truong-Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in neural information processing systems (NeurIPS), 2019.
[2] John David Anderson and Wendt. Computational fluid dynamics, volume 206. Springer, 1995.
[3] Erkao Bao and Linqi Song. Equivariant neural networks and equivarification. arXiv preprint arXiv:/906.071 07722, 2019.
[4] Erik Bekkers. B-spline enns on lie groups. In International Conference on Learning Repre- sentations, 2020. URL http:htttttsseeeeeeeeeeeeee
[5] Tom Beucler, Michael Pritchard, Stephan Rasp, Pierre Gentine, Jordan Ott, and Pierre Baldi. Enforcing analytic constraints in neural-networks emulating physical systems. arXiv preprint arXiv:1909.00912, 2019.
[6] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pages 6571-6583, 2018.
[7] Benjamin Chidester, Minh N. Do, and Jian Ma. Rotation equivariance and invariance in convolutional neural networks. arXiv preprint arXiv: 12301, 2018.
[8] Dragos Bogdan Chirila. Towards lattice Bolrzmann models for climate sciences: The GeLB programming language with applications. PhD thesis, University of Bremen, 2018.
[9] Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning (ICML), pages 2990-2999, 2016.
[10] Taco S. Cohen and Max Welling. Steerable CNNs. arXiv preprint arXiv:1612.08490. 2016.
[11] Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. In Advances in Neural Information Processing Systems, pages 9142-9153, 2019.
[12] Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral CNN. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pages 1321-1330, 2019.
[13] Tri Dao, Albert Gu, Alexander J Ratner, Virginia Smith, Christopher De Sa, and Christopher Ré. A kernel theory of modern data augmentation. Proceedings of machine learning research, 97:1528, 2019.
[14] Richard H. Day. Complex economic dynamics-vol. 1: An introduction to dynamical systems and market mechanisms. MIT' Press Books, 1, 1994.
[15] Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical pro- cesses: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018. URL U itttpss//ppeneeeieew net meet//orummidddyABS EWAZ
[16] Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural networks. In International Conference on Machine Learning (ICML), 2016.
[17] Rui Fang, David Sondak, Pavlos Protopapas, and Sauro Succi. Deep learning for turbulent channel flow. arXiv preprint arXiv:1812.002244 2018.
[18] Chelsea Finn, Ian Goodfellow, and Sergey Leine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pages 64-72, 2016.
[19] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002. /2880, 2020.
10

---

Published as a conference paper at ICLR 2021
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. SXXiv:1505.04597, 2015.
[21] Herbert W Hethcote. The mathematics of infectious diseases. SIAM review, 42(4):599-6533 2000.
[22] Eugene M. Izhikevich. Dynamical systems neuroscience. MIT press, 2007.
[23] Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premkumar Natarajan. Invariant representations through adversarial forgetting. arXiv preprint arXiv: 1911: 1.00600, 2019.
[24] Ihn Kim and Wolfgang JR Hoefer. A local mesh refinement algorithm for the time domain- finite difference method using maxwell's curl equations. IEEE Transactions on Microwave Theory and Techniques, 38(6):812-81)) 1990.
[25] Junhyuk Kim and Changhoon Lee. Deep unsupervised learning of turbulence for inflow generation at various Reynolds numbers. Journal of Computational Physics, page 109216, 2020.
[26] Anthony W. Knapp. Lie Groups Beyond an Introduction, volume 140 of Progress in Mathematics. Birkhäuser, Boston, 2nd edition, 2002.
[27] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pages 2747-2755, 2018
[28] Serge Lang. Algebra. Springer, Berlin, 3rd edition, 2002.
[29] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their cquivariance and equivalence. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
[30] Julia Ling, Andrew Kurzawskim, and Jeremy Templeton. Reynolds averaged turbulence modeling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 2017.
[31] Vadim Lisitsa, Galina Reshetova, and Vladimir Tcheverda. Finite-difference algorithm with local time-space grid refinement for simulation of waves. Computational geosciences, 16(1): 39-54, 2012.
[32] Gurvan Madec et al. Simon Laplace (IPSL), France. hhtps://epic. c. wtt.:/aww....eeeeeee.. wtw..ee/////pprmm//3399999
NEMO ocean engine, 2015. Technical Note. Institut Pierre-
NEMM__book_vdoe pdl
[33] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXh::1511.05444 2015.
[34] Marios Mattheakis, Pavlos Protopapas, D. Sondak, Marco Di Giovanni, and Efthimios Kaxiras. Physical symmetries embedded in neural networks. arXiv preprint arXiv: #rXiv:/904.0899) 2019.
[35] Arvind Mohan, Don Daniel, Michael Chertkov, and Daniel Livescu. Compressed convolutional LSTM: An efficient deep learning framework to model high fidelity 3D turbulence. arXiv preprint 1rXiv:
2019.
[36] Jeremy Morton, Antony Jameson, Mykel J. Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[37] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant representations without adversarial training. In Advances in Neural Information Processing Systems (NeurIPS), pages 9084-9093, 2018.
[38] Peter J. Olver. Applications of Lie groups to differential equations, volume 107. Springer Science & Business Media, 2000.
11

---

Published as a conference paper at ICLR 2021
[39] Sergiu Oprea, P. Martinez-Gonzalezz A. Garcia-Garcia, John Alejandro Castro- Vargas, S. Orts- Escolano, J. Garcia Rodriigue, and Antonis A. Argyros. review on deep learning techniques for video prediction. ArXiv, abs/2004.05214, 2020.
Castro-Vargas,,
[40] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (part D): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 1r1iv: 10561, 2017.
[41] Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics- informed neural networks:
A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Compulational Physics, 378:686-707, 2019.
[42] Anupam K. Gupta Rohan Ghosh. Scale steerable filters for locally scale-invariant convolutional neural networks. arXiv preprint arXiv: 1906.03811. 2019.
[43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 241. Springer, 2015.
[44] Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. equierqqui steerable networks. In International Conference Learning Representations, 2020. URL |RL httpp:://ppeneeiiie.e//
net/ fttt/forummidd =igg
[45] Steven H. Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
[46] Jonathan Tompson. Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating Eulerian fluid simulation with convolutional networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pages 3424-3433, 2017.
[47] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference Learning Representations (ICLR), 2017.
[48] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. arXiv preprint
2019.
[49] Maurice Weiler and Gabriele Cesa. General E())-equivariant steerable CNNs. Advances in 2019.
Neural Information Processing Systems (NeurIPS), pages
[50] Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning stecrable filters for rotation cquivariant CNNs. Computer Vision and Pattern Recognition (CVPR), 2018.
[51] Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In Advances in Neural Information Processing Systems (NeurlPS), pages 7364-7376, 2019.
[52] Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Har- monic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[53] Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Prabhat, and Heng Xiao. En- forcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems. Journal of Computational Physics, page 109209, 2019.
[54] Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances neural information processing systems (NeurIPS), pages 91-99, 2016.
12

---

Published as a conference paper at ICLR 2021
A ADDITIONAL BACKGROUND ON GROUP THEORY
We give a brief overview of group theory and representation theory. For a more complete introduction to the topic see Lang [28 We start with the definition of an abstract symmetry group.
Definition 2 (group). A group of symmetries or simply group is a set G together with a binary operation G x G G called composition satisfying three properties:
1. (identity) There is an element E G such that o g g for all g E G,
2. (associativit)) (g) o g2) o ga = 91 (g2 g3) for all 91,92. 93 E G.
3. (inverses) if g E G, then there is an element g E G such that g -
Definition 3 (Lie group). A group G is Lie group if also smooth manifold over R and the composition and inversion maps are smooth, i.e. infinitely differentiable.
Example Let
(R) the set of 2 invertible real matrices. The set is closed under inversion and matrix multiplication gives well-defined composition. This 4-dimensional real Lie group.
Example 2. Let G = D3 {1,r, rs. r} where is rotation by 2nT / and is reflection over the y-axis. This is the group of symmetries of an equilateral triangle pointing along the y-axis, see Figure
R
R
T3
Figure 4: Illustration D3 acting on a triangle with the letter "R".
Groups are abstract objects, but they become concrete when we let them act.
Definition 4 (action). A group G acts S if there is action map G S - satisfying
1. -11==r for all x S.
2. 91 002 ( (6) 92) x for all x E S1 E
Definition (representation). We say S a G-representation if an RRveector space and G acts on S by linear transformations, that is,
= g g) = for all x. v. S.
2. g (cz) =lg c(g-2) for all x E S. G, C.
Example 3. The group D3 acts on S, the set of points in an equilateral triangle, as in Figure The vector space B2 is both a Dgrrepresentation and a
The language of group theory allows us to formally define equivariance and invariance. Definition (invariant, equivariant). Let f: X - be a function and G group.
1. Assume G acts on X. The function f is G-invariant if f(gr) = x for all x E X and g E G.
2. Assume G acts on X and Y. The function f G-equivariant if f(gx) gf(x) for all x E X and e G.
See Figure for illustration. Note that often omit the different action maps of G on X and on in our notion when they are clear from context.
We combine and decompose representations in differrnt ways.
Definition (direct sum, tensor product). Let V and W be G-representations.
1. The direct sum V W has underlying set V W. As vector space it has scalars c(v, w) = (cv, cw) and addition (0), wi) + (V2, W2) = (v) + 02 Wi + W2). G- representation with action g (v, w) ggv.
13

---

Published as a conference paper at ICLR 2021
2. The tensor product
v@w W- Σ 0 w e E W
is G-representation with action g v w = (gv) @ (gw).
Definition 8 (inreducible). Let V be a G-representation.
1. If W is subspace of V and is closed under the action of G, i.c. gu W for all w e W. g E G. then we say it is subrepresentation.
2. If O and V itself are the only subrepresentations of V, then it is irreducible.
Irreducible representations are the "prime" building blocks of representations. compact Lie group one which is closed and bounded. The rotation group SO IR) is compact, but the group (TR, +) not. All finite groups are also compact Lie groups. The following theorem vastly simplifies our understanding of possible representations of compact Lie groups (see e.g. Knapp (26)).
Theorem (Weyl's Complete Reducibility Theorem). Let be compact real Lie group. Every Emite-dimensione representation of V is direct sum of irreducible representations V = V,V,
Thus to classify the possible fmite-dimensional representations of G, one need only to find all possible irreducible representations of G.
B ADDITIONAL THEORY
B. EQUIVARIANT NETWORKS AND DATA AUGMENTATION
A classic strategy for dealing with distributional shift by transformations in a group G is to augment the training set S by adding samples transformed under G. That is, using the new training set S' = U (S)). We show that data augmentation has no advantage for perfectly equivariant parameterized function fa( since training samples (z, y) and (gz, gy) equivalent. That is, fe learns the same from (x, y) as from (gz, gy) but with only possibly different sample weight. The following is more formal statement of Proposition I
Proposition 5. Let G act X and Y. Let fo: X +Y Y be parameterized class of G-equivariant functions differentiable with respect to Let L: X Y + R G-equivariant loss function where G acts IK by X, we have,
Vo [o
Proof. Equality of the gradients follows equality of the functions C(fo(gx), (gg) fo(gx),n) ((o())))(
In the case of RMSE and rotation uniform motion, the loss function is invariant. That is, equivariant with xg)) 1. Thus the gradient for sample (xx, y) and (gr, gy) equal. In the case of scale, the loss function is equivariant with G (R>o,)) and x() = X. In that case, the sample (gx.gy) is the same as the sample (z, y) but with sample weight x(g).
B.2 ADDING SKIP CONNECTIONS PRESERVES EQUIVARIANCE
We prove in general that adding skip connections to a network does not affect its equivariance with respect to linear actions in the following proposition 6 Define f(i)) as the functional mapping between layer and layer j-
Proposition 6. Let the layer (ii be G-representations for 0 SiiS n. n. Let ((0); v(i) - vi) be G-equivarian for Define recursively x)) = (x)(1))))) Then (n) (x) is G-equivariant
14

---

Published as a conference paper at ICLR 2021
Proof. Assume (+)) is equivariant function of (0)) for < Then by equivariance of f(() and by linearity of the G-action,
Σ (100 (gx!
E
gaol))
USi<J
USiC)
for E G. By induction, (n) f(x!0)) is equivariant with respect to G.
Both ResNet and U-net may modeled as Proposition witht some convolutional and activation components f(i,i+1) and some skip connections f( I with j - 2. Since I is equivariant for any G, we thus have:
Corollary 7. If the layers of ResNet or U-net are G-representations and the convolutional mappings and activation functtion G-equivariant, then the entire network is G-equivariant. O
Corollary 7 allows us to build equivariant convolutional networks for rotational and scaling transfor- mations, which are linear actions.
B.3 RESULTS ON UNIFORM MOTION EQUIVARIANCE
In this section, we prove that for the combined convolution-activatien layers of a CNN to be uniform motion equivariant, the CNN must be an affine function. We assume that the activation function is applied pointwise. That is, the same activation function is applied to every one-dimensional channel independently.
Proposition Let X be a tensor of shape w e and K be convolutional kernel of shape k x k x. e. Let f(X) uniform motion all i, k for some fixed e E R. Then the sum of the weights of K is 1.
X * K be convolutional layer which is equivariant with respect to arbitrary X + for C a constant tensor of the same shape as X. That Cisk f
Proof. Since f is equivariant * (+CC)= K. By linearity, C* K C. Then because C is constant vector field, C K = C(E, K())). As C arbitrary, Σ. K(u) =1
For an activation function to be uniform motion equivariant, it must be a translation.
Proposition Let a: - be a function satisfying (fr + c) = (xx) (r)) + c. Then a is a translation.
Proof. Let a (0). Then (xx) a(x + c) Choosing x gives (())
I.
Proposition 10. Let X and K be as in Prop 8 Let I be a convolutional layer with kernel K and a an activation function. Assume a: I + IR is piecewise differentiable. Then if the composition is equivariant with respect arbitrary uniform motions, it is an affine map of the form (X) K' & X + b, where is a real number and Σ. K'vv)=
Proof. If f is non-zero, then we can choose a tensor X, and constant tensor C full of R, and 2² such that and = (f()), are any two real numbers. Let A = E, K(v). As before f(C) XC. Equivariance thus implies
o + cA) = ((3) +
Note # 0, since if 0, then (5) )) ( + implies c = 0. However is arbitrary. Let cd. Then
(B + h) (
h
X
This holds for arbitrary 3 and h, and thus find is everywhere differentiable with slope X-1 So or (z) = z/A+ e/>+D for some b R. We can then rescale the convolution kernel K' = K/A to get pp ( K' X + b.
Corollary 11 (Corollary 2) If f is CNN alternating between convolutions fs and pointwise activations a; and the combined layers as o fo are uniform motion equivariant, then f is affine.
Proof. This follows from Proposition 9 and the fact that composition of affine functions is affine. 1
15

---

Published as a conference paper at ICLR 2021
Since our treatment is only for pointwise activation functions, it remains possibility that more descriptive networks can be constructed using activation functions which span multiple channels.
Proposition 12 (Proposition 3 A residual block f(x) + a is uniform motion equivariant if the residual connection uniform motion invariant.
Proof. We denote the uniform motion transformation by by TTm + Let f be an invariant residual connection which is a composition of convolution layers and activation functions. Then we compute
f(Te (w)) + Te (w) = f(w) w + c (f(w) w + Te (f (w) + w)
as desired.
B.4 RESULTS ON SCALE EQUIVARIANCE
We show that a scale-invariant CNN in the sense of equation would be extremely limited. Let G = (R>a, -) be the rescaling group. It is isomorphic to (R,+). For e a real number, Pe X gives an action of G I.. There is also, e.g., two-dimensional representation
p(A) = ( log(2) ) (ox))
Proposition 13. Let K be G-equivariant kernel for convolutional layer. Assume G acts on the input layer by Pin and ouiput layer by Pout- Assume that the input layer is padded with Then K 1xl.
Proof. If * 0 then there exists À R>o such that Au is outside the radius of the kernel. So K(Au) = 0. Thus by equivariance, for some n,
((v) X" X" K (Ae) 0.
B.5 EQUIVARIANCE ERROR.
In practice it is difficult to implement a model which is perfectly equivariant. This results in equivariance error EEy(x)
(=)())) f(T())) Given an input with true output and transformed data Txx), the transformed test error TTE = ((T())(((()))) can be bounded using untransformed test error TE = 1 [x)) and EE.
Proposition 14. The transformed test error is bounded
TTE < TE EE
(6)
Proof. By the triangle inequality
(T()) f(T(x)) (T()) - +T((()) f(T()))
[ f(x)) + EE.
For uniform motion TTE < EE + TE since [T() - T(f(x)) - f(x) cl = TE. Consider x and y as flattened into vector. |T| sup =1 [(x)) denotes the operator norm. For g SO(2), acting by T, on vector fields, Tg = 1. For scaling (x, t) = Aw(Ar, X²t).
[T>| = =
16

---

Published as a conference paper at ICLR 2021
B.6 FULL LISTS OF SYMMETRIES OF HEAT AND NS EQUATIONS.
Symmetries of NS Equations.
The Navier-Stokes equations are invariant under five different
transformations (see e-g. (8))
Space translation: T2 x. () = - c,,, R²,
Time translation: Trime f) w(z,t-7), ER,
Uniform motion: w(x
w(x, c RR?
Reflect/rotation: w 1) = Rw 1), R E ((2), Scaling: wx, Aw (Az A2t), Rzo.
Individually each of these types of transformations generates group of symmetries of the system. Collectively, they form 7-dimensional symmetry group.
Symmetries of Heat Equation. The heat equation has even larger symmetry group than the NS Let (x,t)) 1) be solution to equation
equations 38
heatt Then the following are also solutions:
Space translation: H(x v. L), ER?
Time translation: H(x, -c.cttcc... R,
Galilean: e- H|
2ul, (), IRZ
Reflect/Rooation: (RRx, t), R O(2),
Scaling: H(xx, A²t), R>0
Linearity: (H t), and H (x, + (2, t), Hi E Soal(Deat)
Inversion: a(t)e-a(t)er ((((t) a(t)t), where a(t) (1 + 4ct) -1, R.
B.7 TURBULENCE KINETIC ENERGY SPECTRUM
The turbulence kinetic energy spectrum E(k) is related to the mean turbulence kinetic energy as
Integral Scale Scale
E(k)dk = (11'22 + (1')
(u')²
Σ (u(t) - u)² T
1
T
log E(k)
Large Scale
Inertial
Subrange
1=
3
where the & is the wavenumber and t is the time step. Figure 5 shows a theoretical turbulence kinetic energy spectrum plot. The spectrum describe the transfer of energy from large scales motion to the small scales and provides a representation of the dependence of energy on frequency. Thus, the Energy Spectrum Error can indicate whether the predictions preserve the correct statistical dis- tribution and obey the energy conservation law. A trivial example that can illustrate why we need ESE is that if model simply outputs moving averages of input frames, the accumulated RMSE predictions might not be high but the ESE would be really big because all the small or even medium eddies are smoothed out.
Dissipation Scale
log
Figure 5: Theoretical turbulence energy spectrum plot
C HEAT DIFFUSION
2D Heat Equation. Let (1, be scalar field representing temperature. Then H satisfies
0H
aAH.
(Dheat)
Øt
Here A 02 06 is the tvo-dimensional Laplacian and a E R is the diffusivity.
17

---

Published as a conference paper at ICLR 2021
The Heat Equation plays major role in studying heat transfer, Brownian motion and particle diffusion. We simulate the heat equation at various initial conditions and thermal diffusivity using the finite difference method and generate 66 scalar temperature fields. Figure Souure/@@hhue shows heat diffusion process where the temperature inside the circle is higher than the outside and the thermal diffusivity 4. Since the heat equation is much simpler than the NS equations, shallow CNN suffices forecast the heat diffusion process.
=
25.0
t= 50.0
t= 75.0
t 100.0
10
Figure 6: Five snapshots in heat diffusion dynamics, The spatial resolution 50 pixels.
For heat diffusion, due to the law of energy conservation, the sum of each temperature field should be consistent over the entire heat diffusion process. We evaluate the physical characteristics of the predictions using the L1 loss of the thermal energy. shows shows the prediction RMSE and thermal energy loss of the CNNs and three Equ-CNNs three transformed test sets, We can that Equ CNNs consistently outperform CNNs over three test sets.
Table 5: The prediction RMSE and thermal energy L1 loss of the CNNs and three CNNs on three transformed test sets. Equ CNNs outperform the CNNs over all three test sets.
Testset$
RMSE (Thermal Energy Loss)
Models
Mag
Ror
Scale
CNNs
0.103 (4696.3)
0.308 (1125.6)
0.357 (1447.6)
Equ-C CNNs
0.028 (107.7)
0.153 (127.3)
0.045 (396.6)
D IMPLEMENTATION DETAILS
D.I DATASETS DESCRIPTION
Bayleigh-Bénard convection Rayleigh-Bénne Bénard convection results from horizontal layer fluid heated from below, which is major feature of the El Nino dynamics. The dataset comes from two dimensional turbulent flow simulated using the Lattice Boltzmann Method with Rayleigh number = 2.5 x 108 We divided each 1792 x 256 image into 7 square sub-regions of size 256 x 256, then downsample them into 64 pixels sized images. Figure in appendix shows a snapshot in our RBC flow dataset. We generate the following test sets to test the models' generalization ability.
Uniform motion (UM): transformed test sets by adding random vectors drawn from U--1, 1).
Magnitude (Mag): transformed test sets by multiplying random values sampled from U U(0 2).
Rotation (Rot): transformed test sets by randomly rotated by the multiples of T/12.
Scale: transformed test sets by scaling each sample sampled from (1/55, 2).
Figure 7: snapshot of the Rayleigh-Bénard convection flow, the velocity fields along æ direction (left) and y direction (right) g The spatial resolution is 1792x256 pixels.
18

---

Published as a conference paper at ICLR 2021
Ocean Currents We used the reanalysis ocean currents velocity data generated by the NEMO (Nucleus for European Modeling of the Ocean) simulation engine H We selected an area from each of the Atlantic, Indian and North Pacific Oceans from 01/01/2016 to 08/18/2017 and extracted 64x64 sub-regions for our experiments. The corresponding latitude and longitude ranges for the selected regions (-44~-23, 25-466, (55~76, 19) and (-174~-153, 5~~6) respectively. We not only test all models on the future data but also on a different domain (-180~-155, -40~--99 -5000-159-00-00599) in South Pacific Ocean from 01/01/2016 to 12/15/2016. Also, the most recent work on this dataset is [S] which unified warping scheme and an U-net to predict temperature. So to compare our equivariant models with sfate-of-arts, we also investigate our models on the task of temperature field predictions. Since the data back to year 2006 that [15] used is no longer available, we collect more recent temperature data from a square region (-50~-20, 20~50) Atlantic Ocean from 01/01/2016 to 12/31/2017.
D.2 EXPERIMENTS SETUP
We tested our convolutional equivariant layers in architecture, 18-layer ResNet and 13-layer U-net. One of our goals is to show that adding equivariance improves the physical accuracy of state-of-the-ar dynamics prediction. ResNet and U-net are the popular state-of-the-art methods at the moment and our equivariance techniques are well-suited for their architecture. The reason we did not use recurrent models, such as Convolutional LSTM, is that they are slow to train especially for our case where the input length is large. This does not fit our long-term goal of accelerating computation.
The input to each model is I 64 x 64 x 2-size tensor representing the past / timesteps of the velocity field. The output is a single velocity field. The value of / is hyper-parameter we tuned. We found the optimal value of to be around = 25. To predict more timesteps, we apply the model autoregressively, dropping the oldest timestep and concatenating the prediction to the input.
To make this a fair comparison, we adjust the hidden dimensions for different equivariant models to make sure that the number of parameters in all models are about the same for either architecture, which can be found in Table Table 7 gives the rrper-parameter tuning ranges for our models. Note that the hidden dimension and the number of layers of the shallow CNNs for the heat diffusion task are also well-tuned.
The loss function used is the MSE between the predicted frames and the ground truth for next k steps, where is a parameter we tuned. We found = 3 or 4 give the best performance. We use 60%-20%-20% rraining-validation-te split in time and use the validation set for haper-parameters tuning based on the average error of predictions. The training set corresponds to the first 60% of the entire dataset in time and the validation/test sets contains the following 40%. For fluid flows, we standardize the data by the average of velocity vectors and the standard deviation of the L2 norm of velocity vectors. For sea surface temperature, we did the exact same data preprocessing described in de Bezenac et al. [5]
Table 6: The number of parameters in each model and time costs for training epoch on 8 V100 GPUs.
ResNet
Reg
UM
Mag
Rot
Scale
U-net
Reg
UM
Mag
Rot
Scale
Params (10%)
11.0
11.0
11.0
10.2
10.7
6.2
6.2
6.2
7.1
5.9
Time(min)
3.04
5.21
5.50
14.31
160.32
2.15
4.32
4.81
11.32
135.72
E ADDITIONAL RESULTS
Table shows the RMSES of temperature predictions. Figure g shows the ground truth and the predicted velocity norm fields (Vu² u²) time step 1, and by the U-net and four Equ-Unet.
TThe data available at
com on-com
w paw.rreerrraaiaieeoooooeeeeeeeer
19

---

Published as a conference paper at ICLR 2021
Table 7: The Hypeer-aramettrr tuning range: Learning rate, the number of accumulated errors for backpropogation. the number of input frames, batch size, and the hidden dimension and the number of layers of the shallow CNNs for heat diffusion
Learning rate
#Accum Errors
#lnput frames
Batch Size
Hidden dim (CNNs) #Layers (CNNs)
le-1 le-6
1~10
I~30
4~66
8~~128
1~110
on the four transformed test samples. Siowr the ground truth and the predicted ocean currents (vu2 2²) time step and 10 by the regular ResNet and four Equ-ResNet s on the set of future time.
Table 8: The RMSEs of temperature predictions test data. For equivariant models, the left number in the cell is ResNet and the right number in the cell is U-net
CLSTM
Bézenac
ResNet U-net
EqUax
Equasa
EqUeet
Equseai
RMSE
0,46
0.38
0.41
0.391
0.38 0.37
0.39 10.37
0.38 10.40
0.42 00.1
Target
U-net
Equtew
Target
U-net
Equasy
Target
U-net
Equisca
Target
U-net
Equica
10
Figure 8: The ground truth and the predicted velocity norm fields (v + n²) at time step and 10 by the U-net and four Equ Unet the four transformed test samples. From left to right, the transformed test samples are the original test samples uniform-motion-ssifee by (1, -.55), magnitude-scales by 1.5, rotated by 90 degrees and upscaled by 3 respectively. The first row is the target, the second row is Equ Eqqu-nnets predictions, and the third row predictions by U-net.
Target
ResNet
Equax
Equasg
Equaet
Equical
U-net
Equaas
Equaug
Equas
Equiccll
t=s
10
t=
Figure 9: The ground truth and the predicted ocean currents (v + v at time step and 10 by the regular ResNet and four Equ-ResNet, the test set of future time.
20


Reviews:
[{"review_id": "q0QgqK9DRKz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Physical dynamics have symmetry properties, which can be leveraged by neural networks for better accuracy and generalization. This paper takes 2D Navier-Stokes (NS) equation as an example and re-design the convolutions in networks. Simulations and experiments on real data are conducted to verify that the new models are effective. \n\nThe common logic of the work is reasonable. However, the novelty and technical contribution is limited. 1) the symmetries of NS equations are well-studied and borrowed from [37]. 2) The design of Equivariant networks, i.e., Equivariant ResNet and U-Net are incremental comparing to [9,10] and [47]. \n\nSome further questions: 1. Are the proposed re-design for uniform motion equivariance in eqn (2) and that for scale equivariance in eqn (5) compatible? Can we design networks that simultaneously have multiple kinds of symmetry? Even potentially to automatedly learn the symmetry of the unknown systems? It is more attractive to me than the current approach. 2. What is the detailed implementation of Augm in Table 1? How many augmentation for each instance on average? Will the performance increase with the number of augmentation? Is it possible to do Augm for Ocean dynamic? More experiments will firmly validate the superiority of the proposed method over the data augmentation approach. 3. Shall we keep convolution, which are original designed for image, in physical dynamics? \n\nSec 2.1-2.3 are hard to follow. Fortunately, I can get the main idea with almost skip this part. Maybe this part is not necessarily in that rigorous. Otherwise, please move some theoretical results into the body to highlight the technical contribution. \n\nOverall, this paper proposed methods to make CNN to be symmetric as (NS equation) dynamics. The technical contribution need be highlighted.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lO8ZHqy_NFp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies improving the modeling of physical dynamics with equivariant neural networks. In particular, this paper focuses on a new type of data governed by physical models. Several special symmetry groups are considered to better characterize the system, including uniform motion equivariance, resolution-independent scaling, and resolution-dependent scaling, etc. Simulation results show that the proposed equivariant model yields better accuracy and physical consistency than the non-equivariant models even with data augmentation, given the type of distributional shift is known. Results on the real-world data show some of the equivariant models can generalize better than the non-equivariant models.\n\nPros\n* The idea of using equivariant networks in physical dynamics seems well-motivated. In cases global alignment is difficult and the distributional shift is unknown, improving generalization by  incorporating known symmetries seems to be a natural idea. \n\n* Although the idea of equivariant networks has been proposed before, the proposed treatments tailored to the modeling physical dynamics are new. \n\nCons\n\n* It is claimed the data is governed by the differential equation, which has several symmetry properties. However, how the \"ResNet and U-net\" networks are used to solve the dynamics prediction problem is missing from the main text. Maybe due to the same reason, the connections to the differential equations are unclear. This paper is not quite self-contained.  \n\n* The content is targeted to a narrow audience. \n \nQuestions: \n\n- Is data augmentation available as a baseline for experiments in Table 3? \n\n- It seems different kinds of symmetries are incorporated separately - not sure if this is a limitation. If a system is known to satisfy multiple symmetries, is it possible to incorporate all of them together in a network? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ThYQvD25BG-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work incorporates symmetries into a convolutional neural network to improve generalization performance and prediction accuracy. The work incorporates various symmetries by designing equivariant neural networks and demonstrate their superior performance on 2D time series prediction both theoretically and experimentally. \n\nThis work studies an interesting and important question in deep learning. However, the reviewer feels that the paper in the current form is difficult to follow with many places unclear. The overall result is based on a combination of previous work on equivariant convolutional neural network, the reviewer finds it hard to parse (the t and obtain a general methodology (or systematic approach) for dealing with symmetry from the work, and there is no high-level message conveyed in this work.\n\nBelow are some more detailed comments:\n\nIn Section 3.1, Equivariant Convolutions.: the result in (1) is quite unclear without much background provided. Equivariant ResNet and U-net: what is the implication on skip connection with no effects for network equivariance? Not clear.\nFor Section 3.2 - 3.5, the reviewer finds it very hard to follow the proposed approaches for dealing with each symmetry. (Perhaps this is because the reviewer has limited domain background in equivariant CNN) In general, the reviewer has it very hard to follow and interpret the proposed methods for dealing with each symmetry, and there is very little background provided.\n3. It would be better if the author can provide more numerical results on real data other than simulated data. In Table 3, it seems that some of the methods with symmetry consideration are even worse than vanilla ResNet and U-net. This is a bit confusing.\n\nOverall, the reviewer feels that the study is interesting with improved ways of dealing with symmetry for learning complex physical dynamics, but the work needs significant improvement in presentations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KYWZwtyqlF5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper demonstrates that incorporating equivariance (i.e. symmetries) into model for predicting fluid dynamics improves its performance, especially when the test distribution is transformed by those symmetry groups. Leveraging the recent literature on equivariant CNNs, the paper proposes a CNN model that is equivariant with respect to known symmetries of the Navier-Stokes equations (time/space translation, rotation, uniform motion, and scaling). This approached is validated on 2 datasets on fluid dynamics: a synthetic dataset on Rayleigh-Benard convection and a real-world ocean dynamics dataset. On the synthetic dataset, the proposed models demonstrate better performance under distribution shift. On the real-world dataset, the models yields predictions that are more accurate and physically consistent.\n\n======\n\nStrengths:\n1. The problem setting and datasets are realistic and could have high impact. Given the growing importance of machine learning in physical dynamics and climate science, better understanding on how to incorporate prior knowledge (symmetries) could lead to better modeling.\n2. The claim of generalization under distribution shift is well-validated. The paper identifies 5 types of symmetries of Navier-Stokes equations, and in the synthetic dataset perturbs the test set by applying transforms from these symmetry groups. The experiment results show that models that incorporate those symmetries perform better on this perturbed dataset (though it's unclear how a model that incorporates one symmetry, e.g. scale symmetry, performs on test set that was transformed by another symmetry, e.g., rotation.)\n\nWeaknesses:\n1. The claim of model equivariance is not directly validated. Even though the main claim that incorporating equivariance leads to better generalization, model equivariance is never measured directly. There could be components of the model that are not equivariant, such as pooling or convolution with stride 2. At least the U-net architecture uses stride 2, so it might not be equivariance. It would be interesting if the degree of equivariance could be measured, to see how that relates to the generalization ability.\n2. Sample complexity claim is not validated. In the abstract, the proposed models are claim to \"enjoy favorable sample complexity\". However, there is no experiment validating this claim. For example, with varying amount of data, how do the models perform?\n3. The need for separate models for each of the different symmetries. For each of rotation, uniform motion, and scale symmetries, there is a model that incorporates such symmetry. However, it's unclear if these could be combined. In the conclusion, it is claimed that \"all of our equivariant models can be combined\". But the conclusion also states that \"there does not exist a single model with equivariance to the full symmetry group of the Navier-Stokes equation\".\n4. [Minor] Limited novelty. It's not clear how the proposed models different from equivariant CNNs in the literature [9, 10, 11, 47]. For symmetries not covered in existing work (e.g. uniform motion), the proposed equivariant model seems ad-hoc. However, I do recognize that making all existing architectures work for this realistic problem is non-trivial, so this is only a minor weakness.\n\n======\n\nOverall, I'm on the fence about accepting/rejecting. The proposed model is shown to generalize well under specific distribution shift, but the main claim about the role of equivariance is not yet validated directly. Addressing this concern would make the paper stronger.\n\n======\n\nAdditional feedback and questions:\n- Visualization in Figure 1 is great.\n- The intro mentions Noether's theorem about the correspondence between symmetry and conserved quantity. Is there any evidence that the proposed models conserve some quantity?\n\n====\nAfter rebuttal: Thank you for addressing my concerns. \n- I believe the claim of equivariance leading to improved generalization (weakness 1 above) has now been validated more directly. \n- Regarding the validation of sample complexity claim: a more direct investigation of performance against training set size, for equivariant and non-equivariant models, would be more convincing. I understand that augmenting data does increase the training set size by a factor of 3, but the training set is then not iid so it's not clear what the \"effective size\" of training set is. In any case I do believe that equivariance reduces sample complexity.\n\nOverall, I vote for accepting.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Symmetries play an important role in physics, and more and more papers show that they also play an important role in statistical machine learning. In particular, employing symmetries might be the key to improve training and predictive performance of machine learning models.  In this context, the present paper shows how previous physical knowledge can be leveraged to improve neural network performance, in particular within Deep dynamic models. To this end, they show how to incorporate equivariance into resnets and u-nets for dynamical systems. On a technical level, as pointed out by the reviews and also clearly mentioned by the authors, the basic building blocks are well known in the literature. However, dynamical systems also raises their own challenges resp. laws when it comes to modelling symmetries, as the authors argue in the paper and also clarified in the rebuttal. For instance, it pays off to adapt the techniques known from the literature deal better with scale, magnitude and uniform motion equivariance. This is a solid contributions and will help many other who want to apply DNNs to dynamic and physical models. 
Decision: Accept (Poster)