Your task now is to draft a high-quality review for a top-tier Machine Learning (ML) conference for a submission:\n\nTitle:\n```\nFactorized Fourier Neural Operators\n```\n\nKeywords:\n```\nfourier transform,fourier operators,pde,navier stokes\n```\n\nAbstract:\n```\nWe propose the Factorized Fourier Neural Operator (F-FNO), a learning-based approach for simulating partial differential equations (PDEs). Starting from a recently proposed Fourier representation of flow fields, the F-FNO bridges the performance gap between pure machine learning approaches to that of the best numerical or hybrid solvers. This is achieved with new representations \u2013 separable spectral layers and improved residual connections \u2013 and a combination of training strategies such as the Markov assumption, Gaussian noise, and cosine learning rate decay. On several challenging benchmark PDEs on regular grids, structured meshes, and point clouds, the F-FNO can scale to deeper networks and outperform both the FNO and the geo-FNO, reducing the error by 83% on the Navier-Stokes problem, 31% on the elasticity problem, 57% on the airfoil flow problem, and 60% on the plastic forging problem. Compared to the state-of-the-art pseudo-spectral method, the F-FNO can take a step size that is an order of magnitude larger in time and achieve an order of magnitude speedup to produce the same solution quality.\n```\n\nFigures/Tables Captions:\n```\nFigure 1: An illustration of the input and output of different PDE problems. See the accompanying Table 1 for details. On the torus datasets (a), the operator learns to evolve the vorticity over time. On Elasticity (b), the operator learns to predict the stress value on each point on a point cloud. On Airfoil (c), the operator learns to predict the flow velocity on each mesh point. On Plasticity (d), the operator learns the displacement of each mesh point given an initial boundary condition.\nFigure 2: The architecture of the Factorized Fourier Neural Operator (F-FNO) for a 2D problem. The iterative process (Eq. (4)) is shown at the top, in which the input function a(i, j) is first deformed from an irregular space into a uniform space a(x, y), and is then fed through a series of operator layers L in order to produce the output function u(i, j). A zoomed-in operator layer (Eq. (7)) is shown at the bottom which shows how we process each spatial dimension independently in the Fourier space, before merging them together again in the physical space.\nFigure 3: Performance (lower is better) on TorusLi, with error bars showing the min and max values over three trials. We show the original FNO (Li et al., 2021a), along with variants that use: teacher forcing, Markov assumption, improved residuals, a bag of tricks, Fourier factorization, and weight sharing. Note that F-FNO and F-FNO-WS are presented on a separate plot (b) to make visualizing the improvement easier (if shown in (a), F-FNO and F-FNO-WS would just be a straight line).\nFigure 4: Performance of F-FNO on TorusKochkov. In (a), we plot the time until the correlation with the ground truths in the test set drops below 95% on the y-axis, against the time it takes to run one second of simulation on the x-axis. In (b), we show how, on the validation set of TorusKochkov, given a fixed spatial resolution of 64x64, changing the step size has no effect on the numerical solver; however there is an optimal step size for the F-FNO at around 0.2.\nFigure 5: Performance of F-FNO on different contexts and input representations.\nTable 1: An overview of the datasets and the corresponding task.\nTable 2: Performance (N-MSE, expressed as percentage, where lower is better) on point clouds (Elasticity) and structured meshes (Airfoil and Plasticity) between our F-FNO and the previous state-of-the-art geo-FNO (Li et al., 2022). Cells with a dash correspond to models which do not converge. The N-MSE is accompanied by the standard deviation from three trials. More detailed results are shown in Tables A.4 to A.6.\nTable A.1: An overview of the four fluid dynamics datasets on regular grids. Our newly generated datasets, TorusVis and TorusVisForce, contain simulation data with a more variety of viscosities and forces than TorusLi (Li et al., 2021a) and TorusKochkov (Kochkov et al., 2021). Note that Li et al. (2021a) did not generate a validation set.\nTable A.2: An overview of the three PDE datasets on irregular geometries. These datasets were generated by Li et al. (2022).\nTable A.3: Detailed performance on TorusLi. These results are used to generate Fig. 3 in the main paper. We run three trials for each experiment, each with a different random seed. We report the mean N-MSE from the three trials, along with the min and max value. A dash indicates that the data is not available.\nTable A.4: Detailed performance on Airfoil. These results are more detailed version of Table 2 in the main paper. We run three trials for each experiment, each with a different random seed. We report the mean N-MSE from the three trials, along with the min and max value.\nTable A.5: Detailed performance on Elasticity. These results are more detailed version of Table 2 in the main paper. We run three trials for each experiment, each with a different random seed. We report the mean N-MSE from the three trials, along with the min and max value. Note that for a given layer, our F-FNO (whether with weight sharing or without) has slightly more parameters than the geo-FNO. This is due to the F-FNO using a bigger hidden size H. We find that on the geo-FNO, increasing its hidden size does not necessarily translate to a better performance.\nTable A.6: Detailed performance on Plasticity. These results are more detailed version of Table 2 in the main paper. We run three trials for each experiment, each with a different random seed. We report the mean N-MSE from the three trials, along with the min and max value.\nTable A.7: The F-FNO is flexible in its input representation. We find that different datasets benefit from having different features. Shown here is the optimal input combination for each dataset on the torus.\n```\n\nMain Content:\n```\n\nSection Title: INTRODUCTION\n  INTRODUCTION From modeling population dynamics to understanding the formation of stars, partial differential equations (PDEs) permeate the world of science and engineering. For most real-world problems, the lack of a closed-form solution requires using computationally expensive numerical solvers, sometimes consuming millions of core hours and terabytes of storage ( Hosseini et al., 2016 ). Recently, machine learning methods have been proposed to replace part ( Kochkov et al., 2021 ) or all ( Li et al., 2021a ) of a numerical solver. Of particular interest are Fourier Neural Operators (FNOs) ( Li et al., 2021a ), which are neural networks that can be trained end-to-end to learn a mapping between infinite-dimensional function spaces. The FNO can take a step size much bigger than is allowed in numerical methods, can perform super-resolution, and can be trained on many PDEs with the same underlying architecture. A more recent variant, dubbed geo-FNO ( Li et al., 2022 ), can handle irregular geometries such as structured meshes and point clouds. However, this first generation of neural operators suffers from stability issues.  Lu et al. (2022)  find that the performance of the FNO deteriorates significantly on complex geometries and noisy data. In our own experiments, we observe that both the FNO and the geo-FNO perform worse as we increase the network depth, eventually failing to converge at 24 layers. Even at 4 layers, the error between the FNO and a numerical solver remains large (14% error on the Kolmogorov flow). In this paper, we propose the Factorized Fourier Neural Operator (F-FNO) which contains an improved representation layer for the operator, and a better set of training approaches. By learning features in the Fourier space in each dimension independently, a process called Fourier factorization, we are able to reduce the model complexity by an order of magnitude and learn higher-dimensional problems such as the 3D plastic forging problem. The F-FNO places residual connections after activation, enabling our neural operator to benefit from a deeply stacked network. Coupled with training techniques such as teacher forcing, enforcing the Markov constraints, adding Gaussian noise to inputs, and using a cosine learning rate scheduler, we are able to outperform the state of the art by a large margin on three different PDE systems and four different geometries. On the Navier-Stokes (Kolmogorov flow) simulations on the torus, the F-FNO reduces the error by 83% compared to the FNO, while still achieving an order of magnitude speedup over the state-of-the-art pseudo-spectral method ( Figs. 3  and 4). On point clouds and structured meshes, the F-FNO outperforms the geo-FNO on both structural mechanics and fluid dynamics PDEs, reducing the error by up to 60% ( Table 2 ). Overall, we make the following three key contributions: 1. We propose a new representation, the F-FNO, which consists of separable Fourier represen- tation and improved residual connections, reducing the model complexity and allowing it to scale to deeper networks ( Fig. 2  and Eqs. (7) and (8)). 2. We show the importance of incorporating training techniques from the existing literature, such as Markov assumption, Gaussian noise, and cosine learning rate decay ( Fig. 3 ); and investigate how well the operator can handle different input representations ( Fig. 5 ). 3. We demonstrate F-FNO's strong performance in a variety of geometries and PDEs ( Fig. 3  and  Table 2 ). Code, datasets, and pre-trained models are available 1 .\n\nSection Title: RELATED WORK\n  RELATED WORK Classical methods to solve PDE systems include finite element methods, finite difference methods, fi- nite volume methods, and pseudo-spectral methods such as Crank-Nicholson and Carpenter-Kennedy. In these methods, space is discretized, and a more accurate simulation requires a finer discretization which increases the computational cost. Traditionally, we would use simplified models for specific PDEs, such as Reynolds averaged Navier-Stokes ( Alfonsi, 2009 ) and large eddy simulation ( Lesieur & M\u00e9tais, 1996 ), to reduce this cost. More recently, machine learning offers an alternative approach to accelerate the simulations. There are two main clusters of work: hybrid approaches and pure machine learning approaches. Hybrid approaches replace parts of traditional numerical solvers with learned alternatives but keep the components that impose physical constraints such as conservation laws; while pure machine learning approaches learn the time evolution of PDEs from data only. Hybrid methods typically aim to speed up traditional numerical solvers by using lower resolution grids ( Bar-Sinai et al., 2019 ;  Um et al., 2020 ;  Kochkov et al., 2021 ), or by replacing computationally expensive parts of the solver with learned alternatives  Tompson et al. (2017) ;  Obiols-Sales et al. (2020) .  Bar-Sinai et al. (2019)  develop a data driven method for discretizing PDE solutions, allowing coarser grids to be used without sacrificing detail.  Kochkov et al. (2021)  design a technique specifically for the Navier-Stokes equations that uses neural network-based interpolation to calculate velocities between grid points rather than using the more traditional polynomial interpolation. Their method leads to more accurate simulations while at the same time achieving an 86-fold speed improvement over Direct Numerical Simulation (DNS). Similarly,  Tompson et al. (2017)  employ a numerical solver and a decomposition specific to the Navier-Stokes equations, but introduce a convolutional neural network to infer the pressure map at each time step. While these hybrid methods are effective when designed for specific equations, they are not easily adaptable to other PDE tasks. An alternative approach, less specialized than most hybrid methods but also less general than pure machine learning methods, is learned correction ( Um et al., 2020 ;  Kochkov et al., 2021 ) which involves learning a residual term to the output of a numerical step. That is, the time derivative is now u t = u * t + LC(u * t ), where u * t is the velocity field provided by a standard numerical solver on a coarse grid, and LC(u * t ) is a neural network that plays the role of super-resolution of missing details. Pure machine learning approaches eschew the numerical solver altogether and learn the field directly, i.e., u t = G(u t\u22121 ), where G is dubbed a neural operator. The operator can include graph neural networks  Li et al. (2020a ;b), low-rank decomposition  Kovachki et al. (2021b) , or Fourier transforms ( Li et al., 2021a ;b). Pure machine learning models can also incorporate physical constraints, for example, by carefully designing loss functions based on conservation laws ( Wandel et al., 2020 ). They can even be based on existing simulation methods such as the operator designed by  Wang et al. (2020)  that uses learned filters in both Reynolds-averaged Navier-Stokes and Large Eddy Simulation before combining the predictions using U-Net. However, machine learning methods need not incorporate such constraints - for example,  Kim et al. (2019)  use a generative CNN model to represent velocity fields in a low-dimensional latent space and a feedforward neural network to advance the latent space to the next time point. Similarly,  Bhattacharya et al. (2020)  use PCA to map from an infinite dimensional input space into a latent space, on which a neural network operates before being transformed to the output space. Our work is most closely related to the Fourier evolve vorticity compute stress compute flow velocity compute displacement over time (a) Torus (b) Elasticity (c) Airfoil (d) Plasticity transform-based approaches ( Li et al., 2021a ; 2022) which can efficiently model PDEs with zero-shot super-resolution but is not specific to the Navier-Stokes equations. Fourier representations are popular in deep-learning due to the efficiency of convolution operators in the frequency space, the O(n log n) time complexity of the fast Fourier transform (FFT), and the ability to capture long-range dependencies. Two notable examples of deep learning models that employ Fourier representations are FNet ( Lee-Thorp et al., 2021 ) for encoding semantic relationships in text classification and FNO ( Li et al., 2021a ) for flow simulation. In learning mappings between function spaces, the FNO outperforms graph-based neural operators and other finite-dimensional operators such as U-Net. In modeling chaotic systems, the FNO has been shown to capture invariant properties of chaotic systems ( Li et al., 2021b ). More generally,  Kovachki et al. (2021a)  prove that the FNO can approximate any continuous operator.\n\nSection Title: THE FACTORIZED FOURIER NEURAL OPERATOR\n  THE FACTORIZED FOURIER NEURAL OPERATOR Solving PDEs with neural operators An operator G : A \u2192 U is a mapping between two infinite- dimensional function spaces A and U. Exactly what these function spaces represent depends on the problem. In general, solving a PDE involves finding a solution u \u2208 U given some input parameter a \u2208 A, and we would train a neural operator to learn the mapping a \u2192 u. Consider the vorticity formulation of the 2D Navier-Stokes equations, \u2202\u03c9 \u2202t + u \u00b7 \u2207\u03c9 = \u03bd\u2207 2 \u03c9 + f \u2207 \u00b7 u = 0 (1) where u is the velocity field, \u03c9 is the vorticity, and f is the external forcing function. These are the governing equations for the torus datasets (Fig. 1a). The neural operator would learn to evolve this field from one time step to the next: \u03c9 t \u2192 \u03c9 t+1 . Or consider the equation for a solid body in structural mechanics, \u03c1 \u2202 2 u \u2202t 2 + \u2207 \u00b7 \u03c3 = 0, (2) where \u03c1 is the mass density, u is the displacement vector and \u03c3 is the stress tensor. Elasticity (Fig. 1b) and Plasticity (Fig. 1d) are both governed by this equation. In Plasticity, we would learn to map the initial boundary condition s d : [0, L] \u2192 R to the grid position x and displacement of each grid point over time: s d \u2192 (x, u, t). In Elasticity, we are instead interested in predicting the stress value for each point: x \u2192 \u03c3. Finally consider the Euler equations to model the airflow around an aircraft wing (Fig. 1c): where \u03c1 is the fluid mass density, p is the pressure, u is the velocity vector, and E is the energy. Here the operator would learn to map each grid point to the velocity field at equilibrium: x \u2192 u.\n\nSection Title: Original FNO and geo-FNO architectures\n  Original FNO and geo-FNO architectures Motivated by the kernel formulation of the solution to linear PDEs using Green's functions,  Li et al. (2020b ; 2022) propose an iterative approach to map input function a to output function u, where \u2022 indicates function composition, L is the number of layers/iterations, P is the lifting operator that maps the input to the first latent representation z (0) , L (\u2113) is the \u2113'th non-linear operator layer, and Q is the projection operator that maps the last latent representation z (L) to the output. On irregular geometries such as point clouds, we additionally define a coordinate map \u03d5, parameterized by a small neural network and learned end-to-end, that deforms the physical space of irregular geometry into a regular computational space. The architecture without this coordinate map is called FNO, while the one with the coordinate map is called geo-FNO.  Fig. 2  (top) contains a schematic diagram of this iterative process. Originally,  Li et al. (2021a)  formulate each operator layer as L (\u2113) z (\u2113) = \u03c3 W (\u2113) z (\u2113) + b (\u2113) + K (\u2113) (z (\u2113) ) , (5) where \u03c3 : R \u2192 R is a point-wise non-linear activation function, W (\u2113) z (\u2113) +b (\u2113) is an affine point-wise map in the physical space, and K (\u2113) is a kernel integral operator using the Fourier transform, The Fourier-domain weight matrices {R (\u2113) | \u2113 \u2208 {1, 2, . . . , L}} take up most of the model size, requiring O(LH 2 M D ) parameters, where H is the hidden size, M is the number of top Fourier modes being kept, and D is the problem dimension. Furthermore, the constant value for M and the affine point-wise map allow the FNO to be resolution-independent.\n\nSection Title: Our improved F-FNO architecture\n  Our improved F-FNO architecture We propose changing the operator layer in Eq. (5) to: Note that we apply the residual connection (z (\u2113) term) after the non-linearity to preserve more of the layer input. We also use a two-layer feedforward, inspired by the feedforward design used in transformers ( Vaswani et al., 2017 ). More importantly, we factorize the Fourier transforms over the problem dimensions, modifying Eq. (6) to The seemingly small change from R (\u2113) to R (\u2113) d in the Fourier operator reduces the number of parame- ters to O(LH 2 M D). This is particularly useful when solving higher-dimensional problems such as 3D plastic forging (Fig. 1d). The combination of the factorized transforms and residual connections allows the operator to converge in deep networks while continuing to improve performance ( Fig. 3 ). It is also possible to share the weight matrices R d between the layers, which further reduces the parameters to O(H 2 M D).  Fig. 2  (bottom) provides an overview of an F-FNO operator layer. Furthermore, the F-FNO is highly flexible in its input representation, which means anything that is relevant to the evolution of the field can be an input, such as viscosity or external forcing functions for the torus. This flexibility also allows the F-FNO to be easily generalized to different PDEs.\n\nSection Title: Training techniques to learn neural operators\n  Training techniques to learn neural operators We find that a combination of deep learning techniques are very important for the FNO to perform well, most of which were overlooked in  Li et al. (2021a) 's original implementation. The first is enforcing the first-order Markov property. We find  Li et al. (2021a) 's use of the last 10 time steps as inputs to the neural operator to be unnecessary. Instead, it is sufficient to feed information only from the current step, just like a numerical solver. Unlike prior works ( Li et al., 2021a ;  Kochkov et al., 2021 ), we do not unroll the model during training but instead use the teacher forcing technique which is often seen in time series and language modeling. In teacher forcing, we use the ground truth as the input to the neural operator. Finally during training, we find it useful to normalize the inputs and add a small amount of Gaussian noise, similar to how  Sanchez-Gonzalez et al. (2020)  train their graph networks. Coupled with cosine learning rate decay, we are able to make the training process of neural operators more stable. Ablation studies for the new representation and training techniques can be found in  Fig. 3 .\n\nSection Title: DATASETS AND EVALUATION SETTINGS\n  DATASETS AND EVALUATION SETTINGS\n\nSection Title: PDEs on regular grids\n  PDEs on regular grids The four Torus datasets on regular grids (TorusLi, TorusKochkov, TorusVis, and TorusVisForce, summarized in  Table 1 ) are simulations based on Kolmogorov flows which have been extensively studied in the literature ( Chandler & Kerswell, 2013 ). In particular, they model turbulent flows on the surface of a 3D torus (i.e., a 2D grid with periodic boundary conditions). TorusLi is publicly released by  Li et al. (2021a)  and is used to benchmark our model against the original FNO. The ground truths are assumed to be simulations generated by the pseudo-spectral Crank- Nicholson second-order method on 64x64 grids. All trajectories have a constant viscosity \u03bd = 10 \u22125 (Re = 2000), use the same constant forcing function, f (x, y) = 0.1[sin(2\u03c0(x+y))+cos(2\u03c0(x+y))], and differ only in the initial field. Using the same Crank-Nicolson numerical solver, we generate two further datasets, called TorusVis and TorusVisForce, to test the generalization of the F-FNO across Navier-Stokes tasks with different viscosities and forcing functions. In particular, for each trajectory, we vary the viscosity between 10 \u22124 and 10 \u22125 , and set the forcing function to where the amplitudes \u03b1 pij and \u03b2 pij are sampled from the standard uniform distribution. Furthermore, \u03b4 is set to 0 in TorusVis, making the forcing function constant across time; while it is set to 0.2 in TorusVisForce, giving us a time-varying force. Finally, we regenerate TorusKochkov (Fig. 1a) using the same settings provided by  Kochkov et al. (2021)  but with different initial conditions from the original paper (since the authors did not release the full dataset). Here the ground truths are obtained from simulations on 2048x2048 grids using the pseudo-spectral Carpenter-Kennedy fourth-order method. The full-scale simulations are then downsampled to smaller grid sizes, allowing us to study the Pareto frontier of the speed vs accuracy space (see Fig. 4a). TorusKochkov uses a fixed viscosity of 0.001 and a constant forcing function f = 4 cos(4y)x \u2212 0.1u, but on the bigger domain of [0, 2\u03c0]. Furthermore, we generate only 32 training trajectories to test how well the F-FNO can learn on a low-data regime.\n\nSection Title: PDEs on irregular geometries\n  PDEs on irregular geometries The Elasticity, Airfoil, and Plasticity datasets (final three rows in  Table 1 ) are taken from  Li et al. (2022) . Elasticity is a point cloud dataset modeling the incompressible Rivlin-Saunders material ( Pascon, 2019 ). Each sample is a unit cell with a void in the center of arbitrary shape (Fig. 1b). The task is to map each cloud point to its stress value. Airfoil models the transonic flow over an airfoil, shown as the white center in Fig. 1c. The neural operator would then learn to map each mesh location to its Mach number. Finally, Plasticity models the plastic forging problem, in which a die, parameterized by an arbitrary function and traveling at a constant speed, hits a block material from above (Fig. 1d). Here the task is to map the shape of the die to the 101 \u00d7 31 structured mesh over 20 time steps. Note that Plasticity expects a 3D output, with two spatial dimensions and one time dimension.\n\nSection Title: Training details\n  Training details For experiments involving the original FNO, FNO-TF (with teaching forcing), FNO-M (with the Markov assumption), and FNO-N (with improved residuals), we use the same training procedure as  Li et al. (2021a) . For our own models, we train for 100,000 steps on the regular grid datasets and for 200 epochs for the irregular geometry datasets, warming up the learning rate to 2.5 \u00d7 10 \u22123 for the first 500 steps and then decaying it using the cosine function ( Loshchilov & Hutter, 2017 ). We use ReLU as our non-linear activation function, clip the gradient value at 0.1, and use the Adam optimizer ( Kingma & Ba, 2015 ) with \u03b2 1 = 0.9, \u03b2 2 = 0.999, \u03f5 = 10 \u22128 . The weight decay factor is set to 10 \u22124 and is decoupled from the learning rate ( Loshchilov & Hutter, 2019 ). In each operator layer on the torus datasets, we always throw away half of the Fourier modes (e.g., on a 64x64 grid, we keep only the top 16 modes). Models are implemented in PyTorch ( Paszke et al., 2017 ) and trained on a single Titan V GPU.\n\nSection Title: Evaluation metrics\n  Evaluation metrics We use the normalized mean squared error as the loss function, defined as N-MSE = 1 B B i=1 \u2225\u03c9 i \u2212 \u03c9\u2225 2 \u2225\u03c9\u2225 2 , where \u2225\u00b7\u2225 2 is the 2-norm, B is the batch size, and\u03c9 is the prediction of the ground truth \u03c9. In addition to comparing the N-MSE directly, for TorusKochkov, we also compute the vorticity correlation, defined as \u03c1(\u03c9,\u03c9) = i j \u03c9 ij \u2225\u03c9\u2225 2\u03c9 ij \u2225\u03c9\u2225 2 , and from which we measure the time until this correlation drops below 95%. To be consistent with prior work, we use the N-MSE to compare the F-FNO against the FNO and geo-FNO  Li et al. (2021a ; 2022), and the vorticity correlation to compare against  Kochkov et al. (2021) 's work.\n\nSection Title: RESULTS FOR NAIVER-STOKES ON A TORUS\n  RESULTS FOR NAIVER-STOKES ON A TORUS Comparison against FNO The performance on TorusLi is plotted in  Fig. 3 , with the raw numbers shown in  Table A.3 . We note that our method F-FNO is substantially more accurate than the FNO regardless of network depth, when judged by N-MSE. The F-FNO uses fewer parameters than the FNO, has a similar training time, but generally has a longer inference time. Even so, the inference time for the F-FNO is still up to two orders of magnitude shorter than for the Crank-Nicolson numerical solver. In contrast to our method,  Li et al. (2021a)  do not use teacher forcing during training. Instead they use the previous 10 steps as input to predict the next 10 steps incrementally (by using each predicted value as the input to the next step). We find that the teacher forcing strategy (FNO-TF, orange line), in which we always use the ground truth from the previous time step as input during training, leads to a smaller N-MSE when the number of layers is less than 24. Furthermore, enforcing the first-order Markov property (FNO-M, dotted green line), where only one step of history is used, further improves the performance over FNO and FNO-TF. Including two or more steps of history does not improve the results. The models FNO, FNO-TF, and FNO-M do not scale with network depth, as seen by the increase in the N-MSE with network depth. These models even diverge during training when 24 layers are used. FNO-R, with the residual connections placed after the non-linearity, does not suffer from this problem and can finally converge at 24 layers. FNO++ further improves the performance, as a result of a careful combination of: normalizing the inputs, adding Gaussian noise to the training inputs, and using cosine\n```\n\n======\nYour task:\nCompose a high-quality peer review of an ML paper submitted to a top-tier ML conference on OpenReview.\n\nThe review form is shown below:\n\n1. Summary and contributions: Briefly summarize the paper and its contributions\n\n2. Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the community.\n\n3. Weaknesses: Explain the limitations of this work along the same axes as above.\n\n4. Correctness: Are the claims and method correct? Is the empirical methodology correct?\n\n5. Clarity: Is the paper well written?\n\n6. Relation to prior work: Is it clearly discussed how this work differs from previous contributions?\n\n7. Reproducibility: Are there enough details to reproduce the major results of this work?\n\n8. Additional feedback, comments, suggestions for improvement and questions for the authors\n\n9.  Overall score:\nYou should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The \u201cOverall Score\u201d for each submission should reflect your assessment of the submission\u2019s contributions.\n10): Top 5% of accepted papers. Truly groundbreaking work.\n9): Top 15% of accepted NeurIPS papers. An excellent submission; a strong accept.\n8): Top 50% of accepted NeurIPS papers. A very good submission; a clear accept.\n7): A good submission; accept.nI vote for accepting this submission, although I would not be upset if it were rejected.\n6): Marginally above the acceptance threshold. I tend to vote for accepting this submission, but rejecting it would not be that bad.\n5): Marginally below the acceptance threshold. I tend to vote for rejecting this submission, but accepting it would not be that bad.\n4): An okay submission, but not good enough; a reject. I vote for rejecting this submission, although I would not be upset if it were accepted.\n3): A clear reject. I vote and argue for rejecting this submission.\n2): I'm surprised this work was submitted to NeurIPS; a strong reject.\n1): Trivial or wrong or already known.\n\n10.  Confidence score:\n5): You are absolutely certain about your assessment. You are very familiar with the related work.\n4): You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.\n3): You are fairly confident in your assessment.  It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.\n2): You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.\n1): Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.\n\n11. Have the authors adequately addressed the broader impact of their work, including potential negative ethical and societal implications of their work?\n\n12. Does the submission raise potential ethical concerns? This includes methods, applications, or data that create or reinforce unfair bias or that have a primary purpose of harm or injury. If so, please explain briefly.\n\n