Frequency-aware Interface Dynamics with Generative Adversarial Networks
Authors: Lukas Prantl
Tassilo Kugelstadt
Jan Bender
Nils Thuerey
Venue:  (2021)
Keywords: physical simulations
spatio-temporal dynamics
generative adversarial networks
fluids
elasto-plasticity

Abstract: We present a new method for reconstructing and refining complex surfaces based on physical simulations.  Taking a roughly approximated simulation as input, our method infers corresponding spatial details while taking into account how they evolve over time.  We consider this problem in terms of spatial and temporal frequencies, and leverage generative adversarial networks to learn the desired spatio-temporal signal for the surface dynamics.  Furthermore, we investigate the possibility to train our network in an unsupervised manner, i.e. without predefined training pairs. We highlight the capabilities of our method with a set of synthetic wave function tests and complex 3D dynamics of elasto-plastic materials.

Under review as a conference paper at ICLR 2021
FREQUENCY-AWARY INTERFACE DYNAMICS WITH GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We present a new method for reconstructing and refining complex surfaces based on physical simulations. Taking a roughly approximated simulation as input, our method infers corresponding spatial details while taking into account how they evolve over time. We consider this problem in terms of spatial and temporal fre- quencies, and leverage generative adversarial networks learn the desired spatio- temporal signal for the surface dynamics. Furthermore, we investigate the pos- sibility train our network in an unsupervised manner, i.e. without predefined training pairs. We highlight the capabilities of our method with set of synthetic wave function tests and complex 3D dynamies of elasto-plastic materials.
INTRODUCTION
Complex and chaotic physical phenomena such as liquids, gels and goo are still very challenging when it comes to representing them as detailed and realistically as possible. A variety of numerical methods have been proposed to simulate such materials, from purely Eulerian methods (Harlow & Welch 1965 Stam 1999). over particle based methods (Gingold & Monaghan 1977] Thmsen et al.) also been targeted with deep learning methods (Tompson et al.) 2017] Mrowca et al. 2018 Li et al.) 2019), but despite significant advances, they remain very consumin and highly challenging solve.
2014). to hybrids (Zhu & Bridson 2005; Stomakhin et al. 2013)) Such simulations
One approach to speed up the necessary calculations and to allow for more control is to employ super-sampling. This can be seen as a form of post-processing where one simulates only low- resolution simulation and uses an up-sampling technique to approximate the behavior of high- resolution simulation. Neural networks are of special interest here because of their capability to efficiently approximate the strongly nonlinear behavior of physical simulations. Applying neural networks to space-time data sets of physical simulations has seen strongly growing interest re- cent years (Ladicky et al., 2015 Kim et al. 2020), and is particularly interesting in this context to incorporate additional constraints, e.g., for temporal coherence (Xie et al. 2018), or for physical plausibility (Tompson al. 2017 Kim et al. 2019).
An important aspect here is that methods based on simple distance losses, such as mean square errors, quickly reach their limits. The generated data tends be smooth without the necessary small-scale features. Generative adversarial networks (GANs) have been proposed to overcome this issue (Goodfellow) (0066). They are characterized by the fact that, apart from generative network, they also make use of discriminator that classifies the results of the generator with respect to the ground-truth data. Via joint training, the distribution of solutions of the generator is guided to approximate the ground-truth data distribution. As the quality of the results is primarily determined by the discriminator network, it remains an open problem to accurately evaluate the quality of the inferred results. In our work we propose to evaluate the problem in the Fourier space. In this way, we are able to evaluate the given methods reliably, and it allows us to design improved learning algorithms that more faithfully recover the small scale details of the reference data.
For the core our method, we build on an existing GAN-based architecture that employs two discriminator networks, one for the spatial and one for the temporal behaviour (Xie et al. 2018). In terms of ground truth data, we focus on multi-phase (solid-fluid-air) interactions with sharp fluid-air interface. Unlike single-phase flow whose details are visible and relevant solely due to transparency throughout the volume, the details of our data are in most cases only visible on the surface. Of course, the internal dynamics in the volume also play a role, but they mostly hidden

---

Under review as a conference paper at ICLR 2021
from the viewer, only the effects the surface visible. Furthermore, we consider phenomena that build up and take place over the course of several frames. Thus, as we will outline below, we employ recurrent approach that is conditioned previous output in order to produce the solution for a subsequent timestep.
In order to represent and process fine details, we treat such detail high-frequency displacements of a low-frequency surface, and correspondingly formulate the problem in Fourier space. The trans- formation into Fourier space yields an isolated view of the individual frequencies, and thus allows for a much improved analysis of the results achieved by different methods. E.g., it robustly identifies the strong smoothing behavior of L2 metries, and can detect mode collapse problems of adversarial training runs. We also demonstrate how frequency information can be incorporated into the learning objective in order to improve results.
To summarize, the central contributions of our work are: (1) A method for frequency evaluation with a consideration of spatial properties, (2) A novel frequency aware loss formulation, (3) A simple, yet intuitive evaluation of different generative methods, (4) A time consistent spatio-temporal up- sampling of complex physical surfaces.
Deep learning methods in conjunction with physical models were employed in variety of contexts. ranging from learning models for physical intuition (Battaglia et al. 2016 Sanehez-Gonzalez et al. 2018). over robotic control (Schenck Fox 2018 Hu et al. 2019) to engineering applications (Ling et al. 2016; Morton et al.) 2018). In the following, we focus on fluid-like materials with continuous descriptions, which encompass wide range of behavior and pose challenging tasks for learning methods (Mrowca et al. 2018: Li et al. 2019). For fluid flows in particular, a variety of learning methods were proposed ( Tompson et al. 2017 Prantil al. 2017] Um et al.] 2018). A common approach to reduce the high computational cost of simulation is to employ super-resolution techniques (Dong et al.) [2016] Chu & Thuerey 2017) Bai et al. 2019). In this context, our work targets the up-sampling for physics- based animations, for which we leverage the approach proposed by Xie et al. (2018). However, in contrast to this work, we target phenomena with clear interfaces, which motivates the frequency-based viewpoint of our work.
Related Work
For sharp interfaces, Lagrangian models are a very popular discretization of continuum mechani- cal systems. E.g., smoothed particle hydrodynamics (SPH) (Gingold & Monaghan 1977, Koschier lt al. 2019) is a widely-used particle-hased simulation method. While points and particles are like- wise frequently used representations for physical deep learning (Li et al. 2019 Ummenhofer al.) 2019 Sannhezz-onzalez et al. 2020), Eulerian, i.c., grid-based representations offer advantages in terms of efficient and robust kernel evaluations.
We employ generative adversarial networks (Goodfellow 2016), as a powerful and established method for learning generative models. Here, "unconditional" GANs typically rely on a synthetic input vector from Gaussian noise to produce the desired output distribution, e.g., the DC-GAN ap- proach (Radford et al.] 2016). Conditional GANs (Mirza & Osindero (014) were introduced to provide the network with input that allows the neural network to steer the generation of the out- put. Hence super-resolution tasks for natural images (Ledig et al.. 2016), or image translation tasks IIsola et al. 2017) employ conditional GANs. The time dimension was also taken into account in natural imaging works, e.g., by Saito et al. in the form of a temporal generator (Saito et al., 2017). or via stochastic sequence generator (Yu et al. (017)) Other works have included direct Le loss terms as temporal regularizers (Bhattacharjee & Das 2017: Chen et al., 2017), which, however, typically strongly restricts the changes over time. Similar to flow advection, video networks also often use warping information to align data over time (Liu et al., 2017); de Bezenac et al. (017). We will demonstrate that recurrent architectures similar those used for video super-resolution (Saijadi et al. 2018) are likewise very amenable for physical problems over time.
2
METHOD
The input for our method is a coarsely approximated source simulation, with the learning objective to infer the surface of a target simulation over space and time. This target is typically computed via a potentially very costly, finely resolved simulation for the same physical setup. When it comes to the possibilities of simulation representations, there is great variance. In our case we have chosen an implicit representation of the data, by a signed-distance field (SDF) denoted by g R33 IR.
2

---

Under review as a conference paper at ICLR 2021
Hon (..))
(0.0)
X
(ouN)
re (1.1)11))
(a) Generator.
(b) Spatial & temporal discriminator.
The generator (a) takes low-resolution data X₁ as input, together with a high- high-resolutiin version, from the previous time step Y,-1. The low-resolution data is tri-linearly up-scaled (orange layer), while the data with the previous state is advected with the velocity of the input (green layer). The processed inputs are concatenated and processed via four residual blocks (in blue). A high- resolution frame Y is generated as output. (b) The structure of both discriminators (spatial and temporal) is similar: both use high- data as input, either from the generator (Y) or ground- truth data (Y)). The temporal discriminator additionally takes the previous and following frames and advects them with the velocity of the middle frame. Both discriminators process the data with four strided convolution layers (red layers) and two fully-connected layers (brown layers), combined with a spectral normalization. The output is the classification O (fake) to 1 (real).
Figure 1:
An SDF returns, for a given point, the signed distance to the surface, with negative being inside the medium. Such a function is realized in practice by a grid X E IRM, x MyMMM, storing the pre-computed signed distance values, where M., * E {x, y, 2 specifies the size of the grid in the respective dimension v, or We have chosen this representation because most neural network layers are designed for array-like representations, and the loss functions on grid-based data are very efficient to evaluate. Additionally, an implicit representation via grid can leverage tools from the field level-set processing (Adalstcinsson & Sethian, 1999), and facilitates the frequency viewpoint via a Fourier transformation. Additional values, like the velocity, are also mapped on grid V e R. x,y x M, x3 Our goal is to let a generative network G : RM xM, x M, + N infer a grid Y which approximates a desired high-resolution simulation Y E NNN,N,,N,, with N, = K.M,, N, E N(a.y.=)) and sa-sample factor E N, i.c. G(X) = Y x Y. As our method only requires position and velocity data from simulation, it is largely agnostic to the type of solver or physical model for generating the source and target particle data.
2.1 NEURAL NETWORK FORMULATION
Our method based on a generative, neural network with 3D fully-convolutionaal ResNet architec- ture (He et al.) 2016) that produces an output field at single instance in time. The low-resolution input data first up-sampled with tri-linear up-sampling and then processed with several convo- lutional layers, as shown in Figure Ia We use leaky ReLU as activation function after each layer, except for the last layer, where we use a tanh activation. In our case, the input data consists of the implicitly represented geometry data X,, the velocity VÚ of the simulation well as the results of previous pass Y,. The previously generated data is advected with the low-resoluse resoluttttion velocity before further processing. Through this feedback loop we train our network recurrently by iterating over sequence of T 10 frames. This yields stability over longer periods of time and gives better in- sights about temporal behaviour. Furthermore, the recurrent training is important to enable persistent behavior over time, such as the progression of fine surface waves. Unlike the process for generating the input data, the network training cannot resort to a physical simulation with full resolution, and hence cannot uniquely determine the evolution of future states. Therefore, its main learning objec- tive is to capture the dynamics of the target simulations beyond that basic motion computed with an advection step. For initialization of the undefined first frame Y-1 we use a tri-linear up-sampled version of the input. To train our network we have define first loss function that allows us to evaluate the differences between generated and ground-trute data. The most basic loss function is a simple mean squared error (MSE):
-I=1I-- 812
This has the big disadvantage that it is ill-suited to measure the similarity differences of solutions. For example, considering function with multiple solutions for a given input, i.c., a multi-modal setting, method that trains with an MSE loss will learn the expected value of the output distribution,
3

---

Under review as a conference paper at ICLR 2021
Le, the average of the different solutions. However, the average is typically not a part of the solution set. Thus, the MSE loss often does not correspond to the correct distance in solution space, based on significant factors corresponding to the distribution of the solutions. Our super-sampling setup is such a problem: Due to the low resolution input, the high resolution details cannot be determined uniquely, resulting in variety of possible solutions when up-sampling. Via physical properties of the material and its temporal sequence, some solutions can be eliminated, but nonetheless the space of solutions typically remains infinitely large. If an MSE loss used, all such samples from the training data set are simply averaged to obtain a mean value, so that the result does no longer reflect the level of detail of the ground-trud -oound-truth data.
The MSE loss nevertheless gives rough direction, and provides a stable learning target. Hence, we still use it as a component the final loss formulation, in combination with an adversarial loss. In contrast to direct distance metric, the adversarial loss approximates the ground-truth distribution. Hence, the network no longer learns one mean value, but chooses one valid solution out of the possible ones. We define discriminator D. that takes as input high-resolution version of simulation frame and classifies it, distinguishing between ground-truth and generated frames. It does this through binary output, where 0 is "fake" and 1 is "real". Its task is provide the generator with feedback on the correctness of the given data. The special feature is that the discriminators trained together with the generator, thus creating competitive interaction where both parties improve each other. As loss for the discriminator we use a binary cross-entropy:
Loce y log(ū) + (1 1---) v) log(1 - 5)
where y is the ground-trun truth and ÿ the generated value from the discriminator. For complex tasks, GANs can be unstable and difficult to control. For this reason we additionally use the recent Spectral Normalization (Miyato et al. 2018), which we found to provide more stable adversarial training.
While we have primarily focused on spatial content, i.e., the surface of the material so far, the tem- poral behavior likewise plays a crucial role, and poses similar difficulties in our multi-modal setting. On the one hand, the generation of details can quickly lead to temporally incoherent results, which is characterized by unappealing flickering. On the other hand, our network also should be able to match and recreate spatial solutions over time that reflect the physical behavior. Following previous work (Xie et al. 2018). we use an additional discriminator D, to classify the temporal behavior of data. This is done by passing three corresponding frames, which are aligned with each other using advection A : RRN.XN, xN,xx x3 x Rr XMy M,, x3 - R.N, N2 XN, x3 Apart from this, the temporal discriminator closely follows the structure of the spatial discriminator. Both discrimina- tors (Figure [b) use a typical funnel structure, where the dimension is increasingly reduced using strided convolutional layers, with last fully connected layer computing the classification result. We likewise use leaky ReLU activations, with a sigmoid function for the last layer.
The classification of the discriminators included in the loss formulation of the generator:
T CDs = + Σ Xx)
(3)
LD. = D. ())()) VX),
which gives the final loss function:
Lo L. aLDs + BC
(4)
where a and ( indicate the weighting of the individual loss terms.
An additional benefit of the adversarial loss is that it allows for learning from unpaired data. A common problem for up-sampling methods is the generation of paired ground truth data for training. Due to different numerical approximations, and hence potentially differing physical behavior, the easiest solution is to simulate at high resolution, and down-sampl the data. While at training time the down-sampled data is used, at test time, the model needs to be applied to data from a low- resolution simulation instead. This typically leads to large distribution shifts, and correspondingly impaired inference quality. Therefore, we take an unpaired training approach into account that decouples the low and high resolution data. The feedback from the discriminators is still based on the ground-truth data, which makes the output conditionally dependent on the input, but also approximates the behavior of the reference data. However, there is direct supervision in the
4

---

Under review as a conference paper at ICLR 2021
FFT
FFT
Figure 2: The process of our blockwise frequency evaluation. The input data is divided into blocks of a given size. 2D/3D FFT performed on the blocks, thus obtaining the individual Fourier coefficients, again in the form of grids. These grids are then arranged so that each grid corresponds to Fourier coefficient.
generator anymore: the output is no longer compared with a matching ground-truth truth in the loss, but only related to the input. This is done by down-sampling the output and comparing it with the input:
C
||x 1
(5)
where p: RNNxXN,NNXX3 + MM x3 is down-sampling function based average pool- ing. This effectively removes the need for paired low- and high-resolution samples at training time, and fully relies on the discriminator match both distributions.
To indicate the focus on surface structures, we refer to the final version of our generative network surfGAN. For more detailed description of the training and the network architecture refer to the appendix A.1 A
and A.2
2.2 FREQUENCY EVALUATION
Given the formulation so far, an inherent difficulty that remains is a robust and reliable evaluation of the generated outputs. In a GAN setting, the discriminator determines the content, but it is typically not possible to evaluate whether it has correctly learned the shape of the output distribution. While obvious cases such
mode collapse towards constant signal are easy to detect, it remains challenging to reliably detect shifts in the data distributions, especially so for GANs (Arjovsky eet al. [2017] Jolicoeur-Marttncaal Martineau] 2018). In our setting, the outputs should the one hand represent a high-resolution version of the input, i.e., the down-scaled output should correspond to the input. This can be measured with simple MSF. In addition,
also expect the generation of details that match the ground-truth as closely as possible, MSE is no longer usable for this, as even if the right details are generated, a slight translation would already lead to substantial errors. In addition, small- scale features have little effect on the MSE, despite being crucial for a realistic simulation result. Finally, the temporal behaviour should also correspond that of the ground ground-truth.
Considering the problem in frequency space, the sought after detail consists of high-freet frequency fea- tures that cannot be represented by the low-resolution simulation. In addition, we have distinguish between spatial and temporal frequencies. The frequency-based view has the advantage that it yields simple but powerful performance evaluation that complements the discriminator of the GAN train- ing. For the frequency evaluation consider the SDF of our implicit grid G, E N. N.N for t E [O, which can be equated with the frequency behavior of the surface. In order to retain spatial properties in addition to the local frequency behaviour of the surface, age into blocks with size 33 whose Fourier coefficients ,k, we evaluate, where k. E [0, b). divide the domain
im- divide
The method is similar to the used for JPEG compressions. Therefore, into N' x N x N regions for the spatial frequency evaluation, where N = N_/b. Using three-dimensional fast Fourier transformation (FFT), then transform the extracted blocks into the Fourier components:
9h Ry. R2
b
Σ
(6)
where W = é 12x
An important point here is that we take the absolute value from the coefficients in order to eliminate the phase, and focus only the amplitude of the frequency component. This
5

---

Under review as a conference paper at ICLR 2021
ampliusl
frequemay
frequency [B1c]
(a) Frequency histograms (spatial and temp.).
(b) Synthetic data sample.
Figure 3: The averaged frequency spectrum of our data set (a) for the surface, spatial (left), and for the temporal behavior (right). The spectrum the low frequency input data is shown in blue, while in orange the spectrum of the ground-truth data is shown. In the spatial spectrum, it's clearly visible that the spectrum of the input data only covers a quarter of the frequencies, while a large peak of the ground-trun -runt is visible for higher frequencies. This highlights what the generator needs to reproduce. A similar shape can be seen in the time spectrum as the time response strongly coupled to the surface frequency. Here, the spectrum of the input data (blue) continues the time discretization the same for both. Figure (b) shows an example of a surface from our synthetic data set: the input wave in blue, and the corresponding ground-trus target in purple.
b)
c)
Data.
Spatial frequency blocks.
Temporal frequency blocks.
Figure 4: From left to right the image data and the corresponding spatial and temporal frequency images are shown. The respective columns indicate the frequency (from left, low to right, high frequency). We consider the ground-truth (a), result based on MSE (b) and the surfGAN result (c).
makes the problem more robust for translated small-scale structures. Equation 6 yields x b x b Fourier components per box. The individual components can now be grouped with the same com- ponents from other blocks that get one N2/b Ny / x N2/b grid per Fourier component (Figure 2 with each grid corresponding to a certain frequency. These grids can further pro- cessed, e.g., via computational kernels, and in addition For the temporal frequencies we consider the changes per pixel over time via FFT. For longer se- quences, block-wise evaluation would be conceivable for time as well. The separation of spatial and temporal frequencies gives us the possibility to compensate for differences by adjusting the weighting. Based on this setup can now evaluate results. On the one hand, we can directly compare the frequency range of generated data and identify missing details for single samples or E.g., this amenable to loss formulations for learning objectives. On the other hand can also create histogram for the frequencies of the whole solution space and thus compare the frequency distribution of the generated and ground-truth data. This especially helpful for GANs and works also for unsupervised setups.
be inspected by humans for verification.
RESULTS
For evaluation we consider two different data sets: synthetic 2D case and 3D particle-based simulation. The 2D data is fully controllable such that the frequency spectrum of the surface can be evaluated reliably. The basis is wavy surface, based sine wave with varying frequency (Figure 3) This forms a wide range of analysis to isolate problems the generative process and illustrate the aspects of the proposed method. We then evaluate the established methodology in 3D for a more complex scenario with data generated from a highly viscous SPH simulation (Weiler al. 2018). For more information both data sets, refer the appendix and and A.4

---

Under review as a conference paper at ICLR 2021
Enequency [ld]
frequency [ll]]
frequency
Erequeccy [la]]
(a) MSE.
(b) Frequency loss.
(c) SurfGAN.
(d) surfGAN unpair.
Figure 5: Frequency spectrum comparison of different versions of our network for the synthetic 2D data set. The top row always shows the spatial frequency, while the bottom shows the temporal frequency. The spectrum of the input is shown in blue, the ground-truth in yellow, and the predictions in green. Versions (a) was trained with simple MSE loss, while (b) was extended with a frequency loss; (c) and (d) were trained with an adversarial loss, whereby the latter evaluation was trained with unpaired data.
MSE
Frequency
surfGAN
surfGAN unpaired
MSE
0.036
0.0361
0.0556
0.0389
Spatial freq. MAE (x 100)
0.127
0.0711
0.0742
0.178
Temp. freq. MAE (x 100)
0.302
0.128
0.0945
0.228
Table 1: Mean errors for 2D variants. The spatial and temporal MAE values represent the averaged difference of the respective frequency blocks.
3.1 FREQUENCY EVALUATION
We first consider the controlled 2D data. Training is performed with the full data set of 10000 samples with 30 frames, and we evaluate the resulting models 10 simulations that are part of the training set with 30 frames each. Examples of the block-wise analysis from our frequency evaluation (Figure are shown in Figure From top to bottom, it compares ground-truth, network with MSE loss, and the surfGAN result. While the MSE-based variant is not able to reconstruct fine details, the surfGAN can recover these, as highlighted by the frequency evaluation. The high- frequency blocks clearly illustrate the differences into terms of reconstruction accuracy. In Table and Figure [ we also compare the mean error values of the different blocks quantitatively and via histograms. The mean errors likewise illustrate that the MSE version is not able reconstruct the high frequencies of the target function. The spectrum of the generated data hardly shows any deflection in the high frequency range (Figure 5(a)). We additionally extended this fully supervised MSE loss with a term that takes into account differences in the frequency spectrum via an L2 norm between the Fourier components of the data. Figure ()) shows that this improves the situation, but does not suffice to reconstruct the full high frequency range. This is caused by the inherent surfGAN setup
averaging of the MSE, which is still suboptimal in Fourier space. In contrast, (Figure [])) achieves the best results. The spectrum of the prediction very closely matches that of the ground truth. The GAN based method seems to be able to implicitly learn the underlying frequency distribution. Finally, we repeat the evaluation with an unpaired surfGAN setup. Despite the fundamentally more challenging learning setup, the network manages to recover the missing frequencies, as shown in Figure dj.). Interestingly, the discriminator feedback causes the generator to slightly overshoot in terms of high frequency content.
MSE
tempoGAN
surfGAN
MSE
0.0198
0.129
0.0404
Spatial freq. MAE (x100)
0.02599
0.039
0.0119
Temp. freq. MAE
0.206
0.477
0.205
Table 2: Mean error measurements for 3D test scenarios.
7

---

Under review as a conference paper at ICLR 2021
For 3D results have collected the values in Table 2 We compare method with tem- poGAN, whose performance suffers mainly from the missing recursion loop. is clearly visible that surfGAN has the smallest error in the spatial frequencies, which correlates to the spatial detail level. The temporal frequency, on the other hand, almost equal to the value of the NN with the MSE loss. This is probably due the fact that the temporal behavior is relatively smooth.
3.2 QUALITATIVE RESULTS
Source
Ground-truth
surfGAN
MSE
Frequency loss
surfGAN unpaired
Figure 6: Comparison of a frame from a test run with the synthetic 2D data set. The input is shown in blue, ground-truth in red, and predictions in green.
qualitative evaluation we executed our network with test data and visualize the results. Figure 6 shows visual examples of the 2D frequency evaluation. While version based on MSE generates very smooth images in the random data set, the surfGAN versions are able to reconstruct the jagged edges. In direct comparison with the ground-truth data there differences, but this is due to the randomness of the data. Therefore, the exact solution cannot be reconstructed, but the surfGAN is still able generate very plausible solution.
(b) MSE
(a) Frame 190.
(a) Frame 200.
Frame 220 suriGAN.
(c) tempoGAN
Figure 7: Comparison of 3D frames with a surGGAN model (a), with MSE-based generator (b) and tempoGAN (c). The input is shown in blue, ground-truth in red, and predictions in green.
For the 3D case we focus on the paired surfGAN setup. Figure aa) shows that our prediction is able reconstruct most of the details, even they are not present in the input. We have deliberately cho- sen frames after long run-time (190-220 time-steps) to show that details can persist and that they are the result of complex, physically-based behavior over multiple frames. Again, we compared our method with simpler MSE-based generator and tempoGAN, With MSE, the results become com- paratively smooth, as can be seen in Figure 7b). In Figure (c) however, the results of tempoGAN shown, where details are generated quite randomly and display rather chaotic behavior time. Due to the lack of a recurrent processing, the network cannot build an accurate internal state, which has the effect that the details generated unnaturally.
CONCLUSION
We have presented a learning- based method to infer spatio-temporal detail to complex physical sim- ulations. Our method puts special emphasis high frequency content, and we present approach
8

---

Under review as a conference paper at ICLR 2021
for assessing GAN-based outputs in terms of the generated spatial and temporal frequencies. In- terestingly, our proposed surfGAN performs better than a direct supervision in terms in frequency space. Our method provides first step towards evaluation and synthesis of physical space-time processes, and could be employed for other phenomena such as turbulence Ling et al. (2016) or weather /Zaytrr & El Amrani (2016). Purthermore, it will be interesting employ it in conjunction with other frequency-based representations Sitzmann et al. (2020).
REFERENCES
David Adalsteinsson and James A Sethian. The fast construction of extension velocities in level set methods. Journal of Computational Physics, 148(1):2-22, 1999.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv:1701.07873 2017.
Kai Bai, Wei Li, Mathieu Desbrun, and Xiaopei Liu. Dynamic upsampling of smoke through dictionary-basee learning. arXiv:1910.09166. 2019.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Inter- action networks for learning about objects, relations and physics. In Advances in Neural Infor- mation Processing Systems, 2016.
Jan Bender. Splishsplash library, 2017.
Prateep Bhattacharjee and Sukhendu Das. Temporal coherency based criteria for predicting video frames using deep multi-stage generative adversarial networks. In Advances in Neural Informa- fion Processing Systems, pp. 4271-4280, 2017.
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Mengyu Chu and Nils Thuerey. Data-driven synthesis of smoke flows with CNN-based feature descriptors. ACM Trans. Graph., 36(4)(69), 2017.
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. arXiv preprint 17Xiv:1711.0799 2017.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2): 295-307, 2016.
Robert A Gingold and Joseph J Monaghan. Smoothed particle hydrodynamics: Theory and appli- cation to non-spherical stars. Monthly Notices of the Royal Astronomical Society, 181(3), 1977.
Ian Goodfellow. Nips 2016 tutorial:
Generative adversarial networks.
arXiv preprint
17X1v:1001.00160. 2016.
Francis Harlow and Eddie Welch. Numerical Calculation of 'imeedependent Viscous Incompress- ible Flow of Fluid with Free Surface. Physics of Fluids, 8(12):112-2118) 1965.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proc. of IEEE Comp. Vision and Pattern Rec., pp. 770-778, 2016.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan- Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint arXiv: (511:199000095: 2019.
Markus Ihmsen, Jens Orthmann, Barbara Solenthaler, Andreas Kolb, and Matthias Teschner. SPH fluids in computer graphics. In Eurographics State of the Art Reports, pp. 21-42, 2014.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. Proc. of IEEE Comp. Vision and Pattern Rec., 2017.
Alexia Jolicoeur Dolicoeur-Marticauu The relativistic discriminator: a key element missing from standard 2018. URL htt.tttt//// http::ttptt/ttee httt:://aaxxv..orr//b////111 wwt.oo/g/bbbs///// 00734
GAN. CoRR,

---

Under review as a conference paper at ICLR 2021
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. Computer Graphics Forum, 2019. ISSN 1467-8659. doi: 10.11 I/egf. 13619.
Byungsoo Kim, Vinicius C Azevedo, Markus Gross, and Barbara Solenthaler. Lagrangian neural style transfer for fluids. ACM Trans. Graph., 39(4), 2020.
Dan Koschier, Jan Bender, Barbara Solenthaler, and Matthias Teschner. Smoothed particle hydro- dynamies techniques for the physics based simulation of fluids and solids. In EUROGRAPHICS 20/9 Tutorials. Eurographics Association, 2019.
Lubor Ladicky, SoHyeon Jeong, Barbara Solenthaler, Mare Pollefeys, and Markus Gross. Data- driven fluid simulations using regression forests. ACM Trans. Graph., 34(6):199, 2015.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using generative adversarial network. arXiv: 04802, 2016.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019.
Julia Ling, Andrew Kurzawski, and Jeremy Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807, 2016.
Ding Liu, Zhaowen Wang, Yuchen Fan, Xianming Liu, Zhangyang Wang, Shiyu Chang, and Thomas Huang. Robust video super-resolution with learned temporal dynamics. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.178 2014.
arXiv preprint
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems, 2018.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physies prediction. In Advances in Neural Information Processing Systems, 2018.
Lukas Prantl, Boris Bonev, and Nils Thuerey. Pre-computed liquid spaces with generative neural networks and optical flow. arXiv: 7704.78844 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. Proc. ICLR, 2016.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singu- lar value clipping. In IEEE International Conference on Computer Vision (ICCV), pp. 2830-2839, 2017.
Mchdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recureen video super- resolution. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Alvaro Sanchez-Gonzaler, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A. Ried- miller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference Machine Learning. 2018
Alvaro Sanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying. Jure Leskovec, and Peter W Battaglia. Learning to simulate complex physics with graph networks. In International Confer- ence on Machine Learning, 2020.
10

---

Under review as a conference paper at ICLR 2021
Connor Schenck and Dieter Fox. SPNets: Differentiable fluid dynamics for deep neural networks. In Conference on Robot Learning. 2018.
Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint arXiv:2006.09661 2020.
Jos Stam. Stable Fluids. In Proc. ACM SIGGRAPH, pp. 121-128. ACM, 1999.
Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A Material Point Method for Snow Simulation. ACM Trans. Graph., 32(4):article 102, 2013.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. ICMI., 2017.
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Liquid splash modeling with neural networks. In Computer Graphics Forum, volume 37(8), pp. 171-182. Wiley Online Library, 2018.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In International Conference on Learming Representations, 2019.
Marcel Weiler, Dan Koschier, Magnus Brand, and Jan Bender. physically consistent implicit viscosity solver for sph fluids. Computer Graphics Forum, 37(2), 2018.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow. ACM Trans. Graph., 37(4):95:1 1-95115, 2018. doi: 10.1145/ 3197517.3201304
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In AAAI, pp. 2852-2858, 2017.
Mohamed Akram Zaytar and Chaker El Amrani. Sequence to sequence weather forecasting with long short-term memory recurrent neural networks. International Journal Computer Applica- tions, 143(11):7-11, 2016.
Yongning Zhu and Robert Bridson. Animating Sand as a Fluid. ACM Trans. Graph., 24(3):965-99 972, 2005.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 IMPLEMENTATION DETAILS
The main part of the generator consists of four blocks containing two convolution layers and a residual connection. The feature count of the convolution layers per block is as follows: [[8, 32], [64, 64], [32,8] [1, 1]]. The kernel size we use only convolution layers and no residual blocks. Both discriminators, spatial and temporal, have the same number of features per layer: [8, 16, 32, 32). The fully-connected layer at the end of the layer consists of 64 and neurons. The kernel size is 4 x 4 x 4 for all layers. It is important to mention that for the first convolution of every network we do not use zero padding as usual but mirror padding. This is because we work with tiles from the training data and not with the complete frame and zero padding falsifies the values at the transitions between the tiles.
for all layers. For the discriminators
A.2 TRAINING DETAILS
For the training have implemented network with the Tensorflow Framework. We use an Adam optimizer with learning rate of 0.00001 and a batch size 16 for the 2D tests and 4 for the 3D tests for 50k iterations. All other weights initialized with the respective standard initializers of Tensorflow version 2.1. The weighting factors a and / of Equation 4 are set to 1.0 and 10.0 correspondingly.
mmm M
Figure 8: In blue a wave with randomly varying low frequency ft, which can still be represented by the low-resolution sampling and serves as input data set for our synthetic data set. In red, on the other hand, we see a wave with high frequency fn that can only be correctly represented with much higher resolution. Combined with the low-frequency version, we get the ground-truti data (violet) for the synthetic data set.
A.3 SYNTHETIC DATA SET
We use synthetic data set to test and evaluate different aspects of our approach. The data set is designed to have a simple, clearly defined behavior, and such that the frequency spectrum of surface can be evaluated reliably. Therefore, we use a horizontal wavy surface based on ID sine function (z) with randomly varying frequency fi E (0, M
so(x) sin( xf(x)²
) r MO,
M
where M is the resolution of the low-resolution data.
To make a time sequence out of this, we use a simple wave equation:
82s
52s
(8)
812
Discretizing this we can calculate the vertical velocity of our surface as follows:
(x) 0t-1 + sz(x) - sx(x - Ar) + sc(x + Ar)
(9)
Ar
where (z(x)) 0. Given the velocity can now calculate the next frame follows:
(x) st (=)r(()) + Atv(
(10)
For the high resolution data set, i.c, the targets to be learned, we use the same low resolution wave as base, and modulate with a high frequency component (Figure 8):
to tx( = sin( 1 f((x) + sin( n fn(x)) ) E [0, kM),
(11)
kM
kM
12

---

Under review as a conference paper at ICLR 2021
where l is the chosen up-sampling factor and the frequency fa(()) is chosen that it cannot be represented by the low-resolution version. According to the Nyquist-Shamnan sampling theorem, &M the frequency should be higher than f and below 2 (Figure aa)).. The generation of a sequence is done in the same way like for the low-resolution data.
Based on this setup, we generate two different data sets of high resolution data: one where modulate with a fixed high frequency fn(z) = const, whereas in the second version we vary the this high frequency component (Figure B)). Thus the first version represents deterministic up- sampling, which generator should be able to reconstruct perfectly, whereas the second version is ill-posed, i.e. several solutions are possible, and hence poses a much more difficult learning target. Both data sets consist of 10000 sequences with 30 frames each at the end. While the deterministic version only serves as a sanity check which we only discuss here the appendix (A.5). the second, randomized version shows how well the method can approximate the ground-truth distribution for ill-posed tasks like the actual super-resolution problem for physical simulation data.
A.4 SIMULATION DATA
For the generation of simulation, data we use an SPH solver of the SPlisHSPlasH framework (Ben- der 2017). There are different materials to choose from. With materials that exhibit high-frequency physical behavior, chaotic behavior occurs in some cases, such as splashes in water, which are typ- ically very difficult to reconstruct. With more viscous materials, such as gel, details are mainly distinguished by folds and fine waves on the surface. 'he chaotic behaviour is very difficult to re- construct and it is sometimes very difficult to understand how correct the behaviour is. For these reasons we focus more materials like gel. With gel. fine wrinkles can form on the surface which allows good evaluation of the method. Another special feature is that details are persistent over time. Methods such as Xie et al. (2018) cannot represent such details because they do not provide feedback in the network. This means there is no memory. Therefore, we use an plastic material with high viscosity, simulated with an advanced viscosity solver (Weiler et al. 2018).
For the training we generate 60 different scenes with 300 frames per simulation. The time step corresponds to 100 frames per second. The scenes consist of randomly generated shapes that fall into a pool from different heights at random times. This creates interesting waves and folds on the surface. The ground-truth resolution is 160 For the generation of the low-resolution data we distinguish between the way the training is done. For the supervised setup the ground-truth data is scaled down by the desired up- sampling factor i and then smoothed with a Gaussian blur. This results in synchronous data pairs that can be used in supervised setup. In the unsupervised setup, however, the low-resolution data is generated in the same way as the high-resolution data, with the correspondingly lower resolution. From the position and velocity data of the particles, we then generate our SDF and velocity grid which we use for training. Before the training, the data is normalized over the whole data set, that the data is in the value range between -1 and 1. For sharper edge in the SDF grid, which prevents unwanted noise in the generated data, we additionally modify the normalized values with tangent hyperbolic function:
f(z) tanh(cx)
(12)
where e can be freely selected, depending on the strength of the transition. We found 5 to be good value. Finally, we take advantage of the locality of our problem and only use excerpts from the training data frames in training. On the one hand, this saves memory, because the data is sometimes the other hand it allows us extract
very large and can cause problems with the GPU memory, only relevant parts of the data. So in our example we can only consider the data in places where there is surface. Finally we have the advantage to augment the data by overlapping the tiles extract, so we can get a lot of information from only a few frames.
A.5 ADDITIONAL RESULTS
As a sanity check we used deterministic setup as described in A.3 With this setup we tested if the network is able to modulate a static high frequency with a simple MSE loss. In Figure Pa this is shown to be the case. This serves as a baseline for what the network can achieve. In Table 99 we compare the frequency deviation with the MSE network trained with the non-deteriiiistic setup. As expected, the error is much smaller with the deterministic setup.
13

---

Under review as a conference paper at ICLR 2021
In Figure 10 and Figure 1
compare the frequency evaluation of two more data samples, the
same as in Figure
Finally there some more 3D results in Figure 12 and Figure 13
MSE
Det. MSE
MSE
0.036
0.0377
Spatial freq. MAE (x (00)
0.127
0.0226
Temp. freq. MAE (x100)
0.302
0.106
(a) MSE with deterministic setup.
(b) Comparison det. vs not det.
Figure 9: Evaluation of deterministic setup.
b)
c)
Data.
Spatial frequency blocks.
Temporal frequency blocks.
Figure 10: F.l.tr the image data and the corresponding spatial and temporal frequency images
shown. We consider the ground-truth (1), result based on MSE (2) and the surfGAN result (3).
b)
Data.
Spatial frequency blocks.
Temporal frequency blocks.
Figure 11: Flltr the image data and the corresponding spatial and temporal frequency images are shown. We consider the ground-truth (a), a result based MSE (b) and the surfGAN result (c).
14

---

Under review as a conference paper at ICLR 2021
(c) MSE.
(a) Prior frames.
(b) suriGAN.
(d) tempoGAN.
Figure 12: Comparison of a 3D test run wit surfGAN (b), with MSE-based generator (c) and tempoGAN (d). The input is shown in blue, ground-truth in red, and predictions in green.
ICLR ICLR ICLR
Figure 13: An additional output generated with the surGGAN network. The input is shown in blue, ground-truth in red, and the prediction in green.
15


Reviews:
[{"review_id": "kskV924ERdU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper presents a GAN framework to learn spatial and temporal representation on complex physical surfaces to apply to simulation. The method represents data in a SDF-like way so it is agnostic to the properties of material and simulation model. The network is built upon conventional GAN, with two discriminator for temporal and spatial. A loss function is proposed to evaluates on a grid-based Fourier transform of the output and ground truth to better preserve high frequency details (temporally and spatially). Results show performance on a physical simulation with opaque, elasto-plastic materials.\n\n**Strength**\nUsing generative model for physical simulation is an interesting task. Physical simulation contains high temporal frequency / variance that would benefit from GAN that is known to preserve high frequency features.\n\n**Weakness**\n\n    - Method on Fourier domain supervision lacks more analysis and intuition. It's unclear how the size of the grid is defined to perform FFT, from my understanding, the size is critical as local frequency will be changed using different grid size. Is it fixed throughout training? What is the effect of having different sizes?\n    - The generator has a recurrent structure that supports 10 frame generation, but the discriminator looks at three frames (from figure 1) at a time, which seems to limit the power of temporal consistency. \n    - In figure 7 result and supplemental video result, SurfGAN produces smoother results (MSE seems closer to the red ground truth in figure 7). This seems contradicts the use of Fourier components for supervision -- what causes this discrepancy?\n    - Figure 4 is confusing. It's not clear what the columns mean -- it is not explained in the text or caption.\n    - Notation is confusing. M and N are used without definition.\n\n**Suggestion**\n\n    - Spell out F.L.T.R in figure 4\n    - Figure 1 text is too small to see\n    - It is recommended to have notation and figure cross-referenced (e.g. M and N are not shown in the figure)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bw8n5gtUoPS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use the same formulation of adversarial losses over space and time as done by Xie et. al. The block-wise frequency evaluation is a new contribution of the paper. \n\nThe paper reads more like a report as opposed to showing unique insights for the niche problem the authors have tackled. There is little to no novelty in the MSE and adversarial loss formulation over space and time. The use of a generator and two discriminators is exactly the same as done in tempogan. I can see two differences:\n1. Use of MSE as additional loss\n2. Use of generator output from previous time step as a recursive input. \n\nBoth these additions are incremental and not novel. The evaluation is contrived in my opinion. It is natural that tempogan will underperform wrt to MSE metric. The frequency based metrics are derived by the authors and the sparse qualitative comparison to tempogan on a single example make it hard to be confident about the improvement over tempogan. It needs to be evaluated both qualitatively and quantitatively on a wider range of applications like done in tempoGan to validate its usefulness. I would like evalaution with more neutral metric which exists in literature beyond the ones in the paper. As the evaluation metric is derived by the authors, it is hard to validate the benefit of the approach and the evaluation protocol. \n\nThere should be ablation studies that discuss the suggested improvements in architecture over tempogan. Also the authors should compare with simple baselines like nearest neighbor and encoder-decoder networks. As the MSE loss is evaluated over paired samples,  the usefullness of GAN should be validated in the paired case. Some discussion on directly optimzing the frequency metric proposed in the paper using a GAN will be useful to the readers. Overall, the originality of the work is very weak and experiments are insufficient. \n\nPost rebuttal comments: Thank you for the rebuttal. The broad concerns of insufficient novelty remain and I am sticking to my initial paper rating. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9hiF4UAdyJr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the adversarial generative network into a physical simulation task, i.e., reconstruction and refining of complex surfaces. Several losses are applied in the proposed framework, such as MSE, adversarial losses, low-resolution reconstruction loss. Experimental results seem good.\n\nPros:\n+ As far as I know, this is the first work that uses GANs to refine complex surfaces. \n+ Some specific changes have been made in the proposed framework. Specifically, MSE loss is introduced to help the low-frequency reconstruction, a spatial discriminator and also a temporal discriminator are used during the training progress, the adversarial losses also provide the ability of learning from unpaired samples, and to overcome the shortage of unpaired samples that cannot learn an exactly map function, a down-sampling reconstruction loss is applied as a cycle-consistence loss. All these loss functions look reasonable.\n+ This paper is well-written and easy to read.\n\nCons:\n- The main shortage of this paper is the experimental part. First, the comparison method is too weak, several good GAN frameworks, e.g. WGAN-GP, PU-GAN, and SN-GAN, are proposed and none of them is compared. Second, many loss functions are introduced in the proposed method. However, how these losses reflect the experimental results is not well studied. An ablation study is desired.\n- The frequency part is interesting. But I am not sure if I miss something, I do not understand how the method leverage the frequency? Do you use the frequency block for reconstruction loss, or only use it to guide the training progress? I am confused about this. \n- Although many loss functions are introduced, the proposed method is not novel. Most of the losses have been proposed previously. I think the contribution of this paper is not sufficient. However, I would like to read the response from the author about this, especially the frequency part.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of super-resolution of coarse physical simulations into fine-grained video by satisfying some physical properties. The method uses generative models for sequences of images (conditional GANs with spatial and temporal discriminators), that take into account both the multiplicity of realisations given the same conditioning (via adversarial losses) and the spatial frequencies of the modeled surfaces using an evaluation in the Fourier domain, and that also allow training without paired low- and high-resolution samples by relying on a generative losses applied to the downsampled reconstruction. As demonstrated on synthetic 2D data or on 3D frames showing particle simulations, the proposed method can generate images with high-frequency content.

Reviewers have praised the intuition of using GANs for modeling high-frequency features in physical simulation reconstruction. Weaknesses included:
* insufficient analysis and explanation of the Fourier-domain supervision
* limited novelty w.r.t. TempoGAN (with the exception of frequency-based evaluation) and lack of evaluation on more general tasks
* unconvincing results with marginal improvement over plain MSE loss
* missing ablation studies
* toy dataset evaluation only, which makes this work seem preliminary

This paper comes out as 6/7 in my AC stack and I will unfortunately recommend to reject it, hoping that the authors will resubmit an improved version, taking into account the reviewers’ suggestions, for an upcoming conference or for a journal venue.

Decision: Reject