SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Formally, define our embedding matrices as  $Z$ ,  $Z' \in \mathbb{R}^{N \times D}$ . Next, we define the similarity criterion,  $\mathcal{L}_{\mathrm{sim}}$ , as
$$
\mathcal {L} _ {\text {s i m}} (\boldsymbol {u}, \boldsymbol {v}) = \| \boldsymbol {u} - \boldsymbol {v} \| _ {2} ^ {2},
$$
which we use to match our embeddings, and to make them invariant to the transformations. To avoid a collapse of the representations, we use the original variance and covariance criteria to define our regularisation loss,  $\mathcal{L}_{\mathrm{reg}}$ , as
$$
\mathcal {L} _ {\text {r e g}} (\boldsymbol {Z}) = \lambda_ {c o v} C (\boldsymbol {Z}) + \lambda_ {v a r} V (\boldsymbol {Z}), \quad \text {w i t h}
$$
$$
C (\boldsymbol {Z}) = \frac {1}{D} \sum_ {i \neq j} \operatorname {C o v} (\boldsymbol {Z}) _ {i, j} ^ {2} \quad \text {a n d}
$$
$$
V (\boldsymbol {Z}) = \frac {1}{D} \sum_ {j = 1} ^ {D} \max  \left(0, 1 - \sqrt {\operatorname {V a r} (\boldsymbol {Z} _ {: , j})}\right).
$$
Algorithm 1 VICReg Loss Evaluation
Hyperparameters:  $\lambda_{var},\lambda_{cov},\lambda_{inv},\gamma \in \mathbb{R}$
Input:  $N$  inputs in a batch  $\{\pmb {x}_i\in \mathbb{R}^{D_{in}},i = 1,\dots ,N\}$
VICRegLoss(N,  $x_{i},\lambda_{var},\lambda_{cov},\lambda_{inv},\gamma)$
1: Apply augmentations  $t, t' \sim \mathcal{T}$  to form embedding matrices  $\mathbf{Z}, \mathbf{Z}' \in \mathbb{R}^{N \times D}$ :
$$
\boldsymbol {Z} _ {i,:} = h _ {\theta} \left(f _ {\theta} \left(t \cdot \boldsymbol {x} _ {i}\right)\right) \text {a n d} \boldsymbol {Z} _ {i,:} ^ {\prime} = h _ {\theta} \left(f _ {\theta} \left(t ^ {\prime} \cdot \boldsymbol {x} _ {i}\right)\right)
$$
2: Form covariance matrices  $\operatorname{Cov}(\mathbf{Z})$ ,  $\operatorname{Cov}(\mathbf{Z}') \in \mathbb{R}^{D \times D}$ :
$$
\operatorname {C o v} (\boldsymbol {Z}) = \frac {1}{N - 1} \sum_ {i = 1} ^ {N} \left(\boldsymbol {Z} _ {i,:} - \overline {{\boldsymbol {Z}}} _ {i,:}\right) \left(\boldsymbol {Z} _ {i,:} - \overline {{\boldsymbol {Z}}} _ {i,:}\right) ^ {\top}, \quad \overline {{\boldsymbol {Z}}} _ {i,:} = \frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {Z} _ {i,:}
$$
3: Evaluate loss:  $\mathcal{L}(\pmb {Z},\pmb{Z}^{\prime}) = \lambda_{var}\mathcal{L}_{var}(\pmb {Z},\pmb{Z}^{\prime}) + \lambda_{cov}\mathcal{L}_{cov}(\pmb {Z},\pmb{Z}^{\prime}) + \lambda_{inv}\mathcal{L}_{inv}(\pmb {Z},\pmb{Z}^{\prime})$
$$
\mathcal {L} _ {v a r} (\boldsymbol {Z}, \boldsymbol {Z} ^ {\prime}) = \frac {1}{D} \sum_ {i = 1} ^ {N} \max  (0, \gamma - \sqrt {\operatorname {C o v} (\boldsymbol {Z}) _ {i i}}) + \max  (0, \gamma - \sqrt {\operatorname {C o v} (\boldsymbol {Z} ^ {\prime}) _ {i i}}),
$$
$$
\mathcal {L} _ {c o v} (\boldsymbol {Z}, \boldsymbol {Z} ^ {\prime}) = \frac {1}{D} \sum_ {i, j = 1, i \neq j} ^ {N} [ \operatorname {C o v} (\boldsymbol {Z}) _ {i j} ] ^ {2} + [ \operatorname {C o v} (\boldsymbol {Z} ^ {\prime}) _ {i j} ] ^ {2},
$$
$$
\mathcal {L} _ {i n v} (\boldsymbol {Z}, \boldsymbol {Z} ^ {\prime}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \| \boldsymbol {Z} _ {i,:} - \boldsymbol {Z} _ {i ^ {\prime},:} \| ^ {2}
$$
4: Return:  $\mathcal{L}(Z, Z')$
The variance criterion,  $V(Z)$ , ensures that all dimensions in the representations are used, while also serving as a normalization of the dimensions. The goal of the covariance criterion is to decorrelate the different dimensions, and thus, spread out information across the embeddings.
The final criterion is
$$
\mathcal {L} _ {\mathrm {V I C R e g}} (\boldsymbol {Z}, \boldsymbol {Z} ^ {\prime}) = \lambda_ {\mathrm {i n v}} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} _ {\mathrm {s i m}} (\boldsymbol {Z} _ {i, \mathrm {i n v}}, \boldsymbol {Z} _ {i, \mathrm {i n v}} ^ {\prime}) + \mathcal {L} _ {\mathrm {r e g}} (\boldsymbol {Z} ^ {\prime}) + \mathcal {L} _ {\mathrm {r e g}} (\boldsymbol {Z}).
$$
Hyperparameters  $\lambda_{var},\lambda_{cov},\lambda_{inv},\gamma \in \mathbb{R}$  weight the contributions of different terms in the loss. For all studies conducted in this work, we use the default values of  $\lambda_{var} = \lambda_{inv} = 25$  and  $\lambda_{cov} = 1$  unless specified. In our experience, these default settings perform generally well.
D Expanded related work
Machine Learning for PDEs Recent work on machine learning for PDEs has considered both invariant prediction tasks [52] and time-series modelling [53, 54]. In the fluid mechanics setting, models learn dynamic viscosities, fluid densities, and/or pressure fields from both simulation and real-world experimental data [55, 56, 57]. For time-dependent PDEs, prior work has investigated the efficacy of convolutional neural networks (CNNs), recurrent neural networks (RNNs), graph neural networks (GNNs), and transformers in learning to evolve the PDE forward in time [34, 58, 59, 60]. This has invoked interest in the development of reduced order models and learned representations for time integration that decrease computational expense, while attempting to maintain solution accuracy. Learning representations of the governing PDE can enable time-stepping in a latent space, where the computational expense is substantially reduced [61]. Recently, for example, Lusch et al. have studied learning the infinite-dimensional Koopman operator to globally linearize latent space dynamics [62]. Kim et al. have employed the Sparse Identification of Nonlinear Dynamics (SINDy) framework to parameterize latent space trajectories and combine them with classical ODE solvers to integrate latent space coordinates to arbitrary points in time [53]. Nguyen et al. have looked at the development of foundation models for climate sciences using transformers pre-trained on well-established climate
datasets [7]. Other methods like dynamic mode decomposition (DMD) are entirely data-driven, and find the best operator to estimate temporal dynamics [63]. Recent extensions of this work have also considered learning equivalent operators, where physical constraints like energy conservation or the periodicity of the boundary conditions are enforced [29].
Self-supervised learning All joint embedding self-supervised learning methods have a similar objective: forming representations across a given domain of inputs that are invariant to a certain set of transformations. Contrastive and non-contrastive methods are both used. Contrastive methods [21, 64, 65, 66, 67] push away unrelated pairs of augmented datapoints, and frequently rely on the InfoNCE criterion [68], although in some cases, squared similarities between the embeddings have been employed [69]. Clustering-based methods have also recently emerged [70, 71, 6], where instead of contrasting pairs of samples, samples are contrasted with cluster centroids. Non-contrastive methods [10, 40, 9, 72, 73, 74, 39] aim to bring together embeddings of positive samples. However, the primary difference between contrastive and non-contrastive methods lies in how they prevent representational collapse. In the former, contrasting pairs of examples are explicitly pushed away to avoid collapse. In the latter, the criterion considers the set of embeddings as a whole, encouraging information content maximization to avoid collapse. For example, this can be achieved by regularizing the empirical covariance matrix of the embeddings. While there can be differences in practice, both families have been shown to lead to very similar representations [16, 75]. An intriguing feature in many SSL frameworks is the use of a projector neural network after the encoder, on top of which the SSL loss is applied. The projector was introduced in [21]. Whereas the projector is not necessary for these methods to learn a satisfactory representation, it is responsible for an important performance increase. Its exact role is an object of study [76, 15].
We should note that there exists a myriad of techniques, including metric learning, kernel design, autoencoders, and others [77, 78, 79, 80, 81] to build feature spaces and perform unsupervised learning. Many of these works share a similar goal to ours, and we opted for SSL due to its proven efficacy in fields like computer vision and the direct analogy offered by data augmentations. One particular methodology that deserves mention is that of multi-fidelity modeling, which can reduce dependency on extensive training data for learning physical tasks [82, 83, 84]. The goals of multifidelity modeling include training with data of different fidelity [82] or enhancing the accuracy of models by incorporating high quality data into models [85]. In contrast, SSL aims to harness salient features from diverse data sources without being tailored to specific applications. The techniques we employ capitalize on the inherent structure in a dataset, especially through augmentations and invariances.
Equivariant networks and geometric deep learning In the past several years, an extensive set of literature has explored questions in the so-called realm of geometric deep learning tying together aspects of group theory, geometry, and deep learning [86]. In one line of work, networks have been designed to explicitly encode symmetries into the network via equivariant layers or explicitly symmetric parameterizations [87, 88, 89, 90]. These techniques have notably found particular application in chemistry and biology related problems [91, 92, 93] as well as learning on graphs [94]. Another line of work considers optimization over layers or networks that are parameterized over a Lie group [95, 96, 97, 98, 99]. Our work does not explicitly encode invariances or structurally parameterize Lie groups into architectures as in many of these works, but instead tries to learn representations that are approximately symmetric and invariant to these group structures via the SSL. As mentioned in the main text, perhaps more relevant for future work are techniques for learning equivariant features and maps [41, 42].
E Details on Augmentations
The generators of the Lie point symmetries of the various equations we study are listed below. For symmetry augmentations which distort the periodic grid in space and time, we provide inputs  $x$  and  $t$  to the network which contain the new spatial and time coordinates after augmentation.
E.1 Burgers' equation
As a reminder, the Burgers' equation takes the form
$$
u _ {t} + u u _ {x} - \nu u _ {x x} = 0. \tag {49}
$$
Lie point symmetries of the Burgers' equation are listed in Table 4. There are five generators. As we will see, the first three generators corresponding to translations and Galilean boosts are consistent with the other equations we study (KS, KdV, and Navier Stokes) as these are all flow equations.
Table 4: Generators of the Lie point symmetry group of the Burgers' equation in its standard form [44, 100].
Lie algebra generator	Group operation (x,t,u) ↔
g1 (space translation)	εδx	(x+ε,t,u)
g2 (time translation)	εδt	(x,t+ε,u)
g3 (Galilean boost)	ε(tδx+δu)	(x+εt,t,u+ε)
g4 (scaling)	ε(xδx+2tδt-uδu)	(eεx,e2εt,e-εu)
g5 (projective)	ε(xtδx+t2δt+(x-tu)δu)	(x/1-εt, t/1-εt, u+ε(x-tu))
Comments regarding error in [12] As a cautionary note, the symmetry group given in Table 1 of [12] for Burgers' equation is incorrectly labeled for Burgers' equation in its standard form. Instead, these augmentations are those for Burgers' equation in its potential form, which is given as:
$$
u _ {t} + \frac {1}{2} u _ {x} ^ {2} - \nu u _ {x x} = 0. \tag {50}
$$
Burgers' equation in its standard form is  $v_{t} + vv_{x} - \nu v_{xx} = 0$ , which can be obtained from the transformation  $v = u_{x}$ . The Lie point symmetry group of the equation in its potential form contains more generators than that of the standard form. To apply these generators to the standard form of Burgers' equation, one can convert them via the Cole-Hopf transformation, but this conversion loses the smoothness and locality of some of these transformations (i.e., some are no longer Lie point transformations, although they do still describe valid transformations between solutions of the equation's corresponding form).
Note that this discrepancy does not carry through in their experiments: [12] only consider input data as solutions to Heat equation, which they subsequently transform into solutions of Burgers' equation via a Cole-Hopf transform. Therefore, in their code, they apply augmentations using the Heat equation, for which they have the correct symmetry group. We opted only to work with solutions to Burgers' equations itself for a slightly fairer comparison to real-world settings, where a convenient transform to a linear PDE such as the Cole-Hopf transform is generally not available.
E.2 KdV
Lie point symmetries of the KdV equation are listed in Table 5. Though all the operations listed are valid generators of the symmetry group, only  $g_{1}$  and  $g_{3}$  are invariant to the downstream task of the inverse problem. (Notably, these parameters are independent of any spatial shift). Consequently, during SSL pre-training for the inverse problem, only  $g_{1}$  and  $g_{3}$  were used for learning representations. In contrast, for time-stepping, all listed symmetry groups were used.
Table 5: Generators of the Lie point symmetry group of the KdV equation. The only symmetries used in the inverse task of predicting initial conditions are  $g_{1}$  and  $g_{3}$  since the other two are not invariant to the downstream task.
Lie algebra generator	Group operation (x,t,u) ↠
g1 (space translation)	εδx	(x+ε,t,u)
g2 (time translation)	εδt	(x,t+ε,u)
g3 (Galilean boost)	ε(tδx+δu)	(x+εt,t,u+ε)
g4 (scaling)	ε(xδx+3tδt-2uδu)	(eεx,e3εt,e-2εu)
E.3 KS
Lie point symmetries of the KS equation are listed in Table 6. All of these symmetry generators are shared with the KdV equation listed in Table 4. Similar to KdV, only  $g_{1}$  and  $g_{3}$  are invariant to the downstream regression task of predicting the initial conditions. In addition, for time-stepping, all symmetry groups were used in learning meaningful representations.
Table 6: Generators of the Lie point symmetry group of the KS equation. The only symmetries used in the inverse task of predicting initial conditions are  $g_{1}$  and  $g_{3}$  since  $g_{2}$  is not invariant to the downstream task.
Lie algebra generator	Group operation (x,t,u) ↔
g1(space translation)	εδx	(x+ε,t,u)
g2(time translation)	εδt	(x,t+ε,u)
g3(Galilean boost)	ε(tδx+δu)	(x+εt,t,u+ε)
E.4 Navier Stokes
Lie point symmetries of the incompressible Navier Stokes equation are listed in Table 7 [101]. As pressure is not given as an input to any of our networks, the symmetry  $g_{q}$  was not included in our implementations. For augmentations  $g_{E_x}$  and  $g_{E_y}$ , we restricted attention only to linear  $E_{x}(t) = E_{y}(t) = t$  or quadratic  $E_{x}(t) = E_{y}(t) = t^{2}$  functions. This restriction was made to maintain invariance to the downstream task of buoyancy force prediction in the linear case or easily calculable perturbations to the buoyancy by an amount  $2\epsilon$  to the magnitude in the quadratic case. Finally, we fix both order and steps parameters in our Lie-Trotter approximation implementation to 2 for computational efficiency.
Whereas we implemented our own pretraining and evaluation (kinematic viscosity, initial conditions and buoyancy) pipelines, we used the data generation and time-stepping code provided on Github by [12] for Burgers', KS and KdV, and in [18] for Navier-Stokes (MIT License), with slight modification to condition the neural operators on our representation. All our code relies relies on Pytorch. Note that the time-stepping code for Navier-Stokes uses Pytorch Lightning. We report the details of the training cost and hyperparameters for pretraining and timestepping in Table 9 and Table 10 respectively.
F.1 Experiments on Burgers' Equation
Solutions realizations of Burgers' equation were generated using the analytical solution [32] obtained from the Heat equation and the Cole-Hopf transform. During generation, kinematic viscosities,  $\nu$ , and initial conditions were varied.
Representation pretraining We pretrain a representation on subsets of our full dataset containing 10,000 1D time evolutions from Burgers equation with various kinematic viscosities,  $\nu$ , sampled uniformly in the range [0.001, 0.007], and initial conditions using a similar procedure to [12]. We generate solutions of size  $224 \times 448$  in the spatial and temporal dimensions respectively, using the default parameters from [12]. We train a ResNet18 [17] encoder using the VICReg [9] approach to joint embedding SSL, with a smaller projector (width 512) since we use a smaller ResNet than in the original paper. We keep the same variance, invariance and covariance parameters as in [9]. We use the following augmentations and strengths:
- Crop of size (128, 256), respectively, in the spatial and temporal dimension.
Uniform sampling in  $[-2, 2]$  for the coefficient associated to  $g_{1}$ .
Uniform sampling in [0, 2] for the coefficient associated to  $g_{2}$ .
Uniform sampling in  $[-0.2, 0.2]$  for the coefficient associated to  $g_{3}$ .
Table 7: Generators of the Lie point symmetry group of the incompressible Navier Stokes equation. Here,  $u, v$  correspond to the velocity of the fluid in the  $x, y$  direction respectively and  $p$  corresponds to the pressure. The last three augmentations correspond to infinite dimensional Lie subgroups with choice of functions  $E_x(t), E_y(t), q(t)$  that depend on  $t$  only. For invariant tasks, we only used settings where  $E_x(t), E_y(t) = t$  (linear) or  $E_x(t), E_y(t) = t^2$  (quadratic) to ensure invariance to the downstream task or predictable changes in the outputs of the downstream task. These augmentations are listed as numbers 6 to 9.
Lie algebra generator	Group operation
(x,y,t,u,v,p) ↔
g1 (time translation)	εθt	(x,y,t+ε,u,v,p)
g2 (x translation)	εθx	(x+ε,y,t,u,v,p)
g3 (y translation)	εθy	(x,y+ε,t,u,v,p)
g4 (scaling)	ε(2tθt+xθx+yθy-uθu-vθv-2pθp)	(eεx,eεy,e2εt,e-εu,e-εv,e-2εp)
g5 (rotation)	ε(xθy-yθx+uθv-vθu)	(xcosε-ysinε,xsinε+ycosε,t,u cosε-v sin ε,u sin ε+v cos ε,p)
g6 (x linear boost)1	ε(tθx+θu)	(x+εt,y,t,u+ε,v,p)
g7 (y linear boost)1	ε(tθy+θv)	(x,y+εt,t,u,v+ε,p)
g8 (x quadratic boost)2	ε(t2θx+2tθu-2xθp)	(x+εt2,y,t,u+2εt,v,p-2x)
g9 (y quadratic boost)2	ε(t2θy+2tθv-2yθp)	(x,y+εt2,t,u,v+2εt,p-2y)
gEx (x general boost)3	ε(Ex(t)θx+E′x(t)θu-xEx''(t)θp)	(x+εEx(t),y,t,u+εEx'(t),v,p-E''x(t)x)
gExy (y general boost)3	ε(Ey(t)θy+E′y(t)θv-yE''y(t)θp)	(x,y+εEx(t),t,u,v+εEy'(t),p-E''y(t)y)
gq (additive pressure)3	εq(t)θp	(x,y,t,u,v,p+q(t))
1 case of  $g_{E_x}$  or  $g_{E_y}$  where  $E_{x}(t) = E_{y}(t) = t$  (linear function of  $t$ )
2 case of  $g_{E_x}$  or  $g_{E_y}$  where  $E_{x}(t) = E_{y}(t) = t^{2}$  (quadratic function of  $t$ )
${}^{3}{E}_{x}\left( t\right) ,{E}_{y}\left( t\right) ,q\left( t\right)$  can be any given smooth function that only depends on  $t$
Uniform sampling in  $[-1, 1]$  for the coefficient associated to  $g_4$ .
We pretrain for 100 epochs using AdamW [33] and a batch size of 32. Crucially, we assess the quality of the learned representation via linear probing for kinematic viscosity regression, which we detail below.
Kinematic viscosity regression We evaluate the learned representation as follows: the ResNet18 is frozen and used as an encoder to produce features from the training dataset. The features are passed through a linear layer, followed by a sigmoid to constrain the output within  $[\nu_{\mathrm{min}},\nu_{\mathrm{max}}]$ . The learned model is evaluated against our validation dataset, which is comprised of 2,000 samples.
Time-stepping We use a 1D CNN solver from [12] as our baseline. This neural solver takes  $T_{p}$  previous time steps as input, to predict the next  $T_{f}$  future ones. Each channel (or spatial axis, if we view the input as a 2D image with one channel) is composed of the realization values,  $u$ , at  $T_{p}$  times, with spatial step size  $dx$ , and time step size  $dt$ . The dimension of the input is therefore  $(T_{p} + 2, 224)$ , where the extra two dimensions are simply to capture the scalars  $dx$  and  $dt$ . We augment this input with our representation. More precisely, we select the encoder that allows for the most accurate linear regression of  $\nu$  with our validation dataset, feed it with the CNN operator input and reduce the resulting representation dimension to  $d$  with a learned projection before adding it as supplementary channels to the input, which is now  $(T_{p} + 2 + d, 224)$ .
We set  $T_{p} = 20$ ,  $T_{f} = 20$ , and  $n_{\mathrm{samples}} = 2,000$ . We train both models for 20 epochs fol
Table 8: One-step validation NMSE for time-stepping on Burgers for different architectures.
Architecture	ResNet1d	FNO1d
Baseline (no conditioning)	0.110 ± 0.008	0.184 ± 0.002
Representation conditioning	0.108 ± 0.011	0.173 ± 0.002
lowing the setup from [12]. In addition, we use AdamW with a decaying learning rate and different configurations of 3 runs each:
- Batch size  $\in$  {16, 64}.
- Learning rate  $\in$  {0.0001, 0.00005}.
F.2 Experiments on KdV and KS
To obtain realizations of both the KdV and KS PDEs, we apply the method of lines, and compute spatial derivatives using a pseudo-spectral method, in line with the approach taken by [12].
Representation pretraining To train on realizations of KdV, we use the following VICReg parameters:  $\lambda_{var} = 25$ ,  $\lambda_{inv} = 25$ , and  $\lambda_{cov} = 4$ . For the KS PDE, the  $\lambda_{var}$  and  $\lambda_{inv}$  remain unchanged, with  $\lambda_{cov} = 6$ . The pre-training is performed on a dataset comprised of 10,000 1D time evolutions of each PDE, each generated from initial conditions described in the main text. Generated solutions were of size  $128 \times 256$  in the spatial and temporal dimensions, respectively. Similar to Burgers' equation, a ResNet18 encoder in conjunction with a projector of width 512 was used for SSL pre-training. The following augmentations and strengths were applied:
- Crop of size (32, 256), respectively, in the spatial and temporal dimension.
Uniform sampling in  $[-0.2, 0.2]$  for the coefficient associated to  $g_{3}$ .
Initial condition regression The quality of the learned representations is evaluated by freezing the ResNet18 encoder, training a separate regression head to predict values of  $A_{k}$  and  $\omega_{k}$ , and comparing the NMSE to a supervised baseline. The regression head was a fully-connected network, where the output dimension is commensurate with the number of initial conditions used. In addition, a range-constrained sigmoid was added to bound the output between  $[-0.5, 2\pi]$ , where the bounds were informed by the minimum and maximum range of the sampled initial conditions. Lastly, similar to Burgers' equation, the validation dataset is comprised of 2,000 labeled samples.
Time-stepping The same 1D CNN solver used for Burgers' equation serves as the baseline for time-stepping the KdV and KS PDEs. We select the ResNet18 encoder based on the one that provides the most accurate predictions of the initial conditions with our validation set. Here, the input dimension is now  $(T_{p} + 2,128)$  to agree with the size of the generated input data. Similarly to Burgers' equation,  $T_{p} = 20$ ,  $T_{f} = 20$ , and  $n_{\mathrm{samples}} = 2,000$ . Lastly, AdamW with the same learning rate and batch size configurations as those seen for Burgers' equation were used across 3 time-stepping runs each.
A sample visualization with predicted instances of the KdV PDE is provided in Fig. 7 below:
Predicted (SSL pre-training)
Figure 7: Illustration of the 20 predicted time steps for the KdV PDE. (Left) Ground truth data from PDE solver; (Middle) Predicted  $u(x, t)$  using learned representations; (Right) Predicted output from using the CNN baseline.
Predicted (CNN baseline)
Table 9: List of model hyperparameters and training details for the invariant tasks. Training time includes periodic evaluations during the pretraining.
Equation	Burgers'	KdV	KS	Navier Stokes
Network:
Model	ResNet18	ResNet18	ResNet18	ResNet18
Embedding Dim.	512	512	512	512
Optimization:
Optimizer	LARS [102]	AdamW	AdamW	AdamW
Learning Rate	0.6	0.3	0.3	3e-4
Batch Size	32	64	64	64
Epochs	100	100	100	100
Nb of exps	~ 300	~ 30	~ 30	~ 300
Hardware:
GPU used	Nvidia V100	Nvidia M4000	Nvidia M4000	Nvidia V100
Training time	~ 5h	~ 11h	~ 12h	~ 48h
F.3 Experiments on Navier-Stokes