Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces the adversarial generative network into a physical simulation task, i.e., reconstruction and refining of complex surfaces. Several losses are applied in the proposed framework, such as MSE, adversarial losses, low-resolution reconstruction loss. Experimental results seem good.

Pros:
+ As far as I know, this is the first work that uses GANs to refine complex surfaces. 
+ Some specific changes have been made in the proposed framework. Specifically, MSE loss is introduced to help the low-frequency reconstruction, a spatial discriminator and also a temporal discriminator are used during the training progress, the adversarial losses also provide the ability of learning from unpaired samples, and to overcome the shortage of unpaired samples that cannot learn an exactly map function, a down-sampling reconstruction loss is applied as a cycle-consistence loss. All these loss functions look reasonable.
+ This paper is well-written and easy to read.

Cons:
- The main shortage of this paper is the experimental part. First, the comparison method is too weak, several good GAN frameworks, e.g. WGAN-GP, PU-GAN, and SN-GAN, are proposed and none of them is compared. Second, many loss functions are introduced in the proposed method. However, how these losses reflect the experimental results is not well studied. An ablation study is desired.
- The frequency part is interesting. But I am not sure if I miss something, I do not understand how the method leverage the frequency? Do you use the frequency block for reconstruction loss, or only use it to guide the training progress? I am confused about this. 
- Although many loss functions are introduced, the proposed method is not novel. Most of the losses have been proposed previously. I think the contribution of this paper is not sufficient. However, I would like to read the response from the author about this, especially the frequency part.
review_title: Generation of complex surfaces with spatial and temporal discriminators under cycle-consistance supervision of down-sampling reconstruction loss

final_score: rating: 3: Clear rejection
review_content: The authors propose to use the same formulation of adversarial losses over space and time as done by Xie et. al. The block-wise frequency evaluation is a new contribution of the paper. 

The paper reads more like a report as opposed to showing unique insights for the niche problem the authors have tackled. There is little to no novelty in the MSE and adversarial loss formulation over space and time. The use of a generator and two discriminators is exactly the same as done in tempogan. I can see two differences:
1. Use of MSE as additional loss
2. Use of generator output from previous time step as a recursive input. 

Both these additions are incremental and not novel. The evaluation is contrived in my opinion. It is natural that tempogan will underperform wrt to MSE metric. The frequency based metrics are derived by the authors and the sparse qualitative comparison to tempogan on a single example make it hard to be confident about the improvement over tempogan. It needs to be evaluated both qualitatively and quantitatively on a wider range of applications like done in tempoGan to validate its usefulness. I would like evalaution with more neutral metric which exists in literature beyond the ones in the paper. As the evaluation metric is derived by the authors, it is hard to validate the benefit of the approach and the evaluation protocol. 

There should be ablation studies that discuss the suggested improvements in architecture over tempogan. Also the authors should compare with simple baselines like nearest neighbor and encoder-decoder networks. As the MSE loss is evaluated over paired samples,  the usefullness of GAN should be validated in the paired case. Some discussion on directly optimzing the frequency metric proposed in the paper using a GAN will be useful to the readers. Overall, the originality of the work is very weak and experiments are insufficient. 

Post rebuttal comments: Thank you for the rebuttal. The broad concerns of insufficient novelty remain and I am sticking to my initial paper rating. 
review_title: The authors use adversarial losses over space and time for simulation refinement. Incremental novelty in approach. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary**
This paper presents a GAN framework to learn spatial and temporal representation on complex physical surfaces to apply to simulation. The method represents data in a SDF-like way so it is agnostic to the properties of material and simulation model. The network is built upon conventional GAN, with two discriminator for temporal and spatial. A loss function is proposed to evaluates on a grid-based Fourier transform of the output and ground truth to better preserve high frequency details (temporally and spatially). Results show performance on a physical simulation with opaque, elasto-plastic materials.

**Strength**
Using generative model for physical simulation is an interesting task. Physical simulation contains high temporal frequency / variance that would benefit from GAN that is known to preserve high frequency features.

**Weakness**

    - Method on Fourier domain supervision lacks more analysis and intuition. It's unclear how the size of the grid is defined to perform FFT, from my understanding, the size is critical as local frequency will be changed using different grid size. Is it fixed throughout training? What is the effect of having different sizes?
    - The generator has a recurrent structure that supports 10 frame generation, but the discriminator looks at three frames (from figure 1) at a time, which seems to limit the power of temporal consistency. 
    - In figure 7 result and supplemental video result, SurfGAN produces smoother results (MSE seems closer to the red ground truth in figure 7). This seems contradicts the use of Fourier components for supervision -- what causes this discrepancy?
    - Figure 4 is confusing. It's not clear what the columns mean -- it is not explained in the text or caption.
    - Notation is confusing. M and N are used without definition.

**Suggestion**

    - Spell out F.L.T.R in figure 4
    - Figure 1 text is too small to see
    - It is recommended to have notation and figure cross-referenced (e.g. M and N are not shown in the figure)

review_title: interesting approach using GAN for physical simulation, while method lacks novelty and results are quite preliminary

Meta-review: This paper addresses the problem of super-resolution of coarse physical simulations into fine-grained video by satisfying some physical properties. The method uses generative models for sequences of images (conditional GANs with spatial and temporal discriminators), that take into account both the multiplicity of realisations given the same conditioning (via adversarial losses) and the spatial frequencies of the modeled surfaces using an evaluation in the Fourier domain, and that also allow training without paired low- and high-resolution samples by relying on a generative losses applied to the downsampled reconstruction. As demonstrated on synthetic 2D data or on 3D frames showing particle simulations, the proposed method can generate images with high-frequency content.

Reviewers have praised the intuition of using GANs for modeling high-frequency features in physical simulation reconstruction. Weaknesses included:
* insufficient analysis and explanation of the Fourier-domain supervision
* limited novelty w.r.t. TempoGAN (with the exception of frequency-based evaluation) and lack of evaluation on more general tasks
* unconvincing results with marginal improvement over plain MSE loss
* missing ablation studies
* toy dataset evaluation only, which makes this work seem preliminary

This paper comes out as 6/7 in my AC stack and I will unfortunately recommend to reject it, hoping that the authors will resubmit an improved version, taking into account the reviewers’ suggestions, for an upcoming conference or for a journal venue.

Decision: Reject