Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes a learned similarity metric for spatial scalar and vector fields obtained from physics simulations. The metric is like a perceptual loss but for physical fields. The training is self-supervised by comparing one-parameter families of fields obtained by modifying initial conditions or restricting or resampling.
main_review: The core idea of using one-parameter families of states as ground-truth is nice, but this idea appears to be already present in Kohl et al. 2020. I do not follow the motivation for hypothesizing that similarity should behave like entropy. Classical statistical physics posits entropy as a state function for states *at equilibrium,* where generally no macroscopic physical variation is discernable. When comparing, e.g., velocity fields of a fluid that have complicated macroscopic dynamics, why is a statistical physics perspective relevant? What are the "microstates," and how are they related to the macroscopic dynamical variables over which the metric is being defined? Why is it reasonable or necessary to impose a logarithmic prior on similarity between states in one-parameter families? Is there any advantage to this choice over any other monotonic function of the parameter?

The logarithmic hypothesis requires the fitting of a parameter $c$, which is then deduced by using a different similarity metric, PCC, as a proxy. If the authors propose their metric as superior, why does the method rely on PCC as part of its construction of the ground truth?

Perhaps most importantly, the empirical results are underwhelming, with simple metrics like MSE and PSNR often coming out on top of the learned ones. Overall, it is not clear to me what technical or empirical novelty the authors demonstrate over past works such as Kohl et al. 2020.

### Miscellaneous ###
- Algorithm 1 could be expressed more clearly and concisely with formulae rather than an algorithm.
- "our iteration schemes only calibrate $\Delta$ to a suitable magnitude, compared to using a full rejection sampling approach." What do you mean by full rejection sampling? What is being sampled? And if this approach would be better, why not use it?
- It seems from section 5 that the authors use "equivariance" to mean "invariance," but then do not observe scale-invariance. The justification for invariance is that "physical systems exhibit Galilean invariance." But the Galilean group does not include scalings, and many systems are not scale-invariant. It seems reasonable to allow scale-equivariance rather than invariance. Section 5 could be cleaned up to clarify this story.
summary_of_the_review: The core idea of using one-parameter families of states as ground-truth is nice, but this idea appears to be already present in Kohl et al. 2020. Overall, it is not clear to me what technical or empirical novelty the authors demonstrate over past works such as Kohl et al. 2020.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a learning-based method for evaluating the similarity between 3D volumetric data. A new similarity model based on the entropy of a physical system is proposed, which in my opinion is the paper's major contribution. The method combines the new entropy-based model with a Siamese network that is commonly used for similarity measurement on 2D imagery data. The proposed method further modifies the Siamese network in order to process 3D data. Experimental results are demonstrated on turbulence flow data, and are compared with other 2D similarity measurements. 
main_review: Strengths:
The paper is very well written and easy to follow. The proposed entropy-based similarity model is new and appears to be effective in evaluating the similarity between volumetric data in presence of rotation and scaling. The proposed method seems to be the first learning-based similarity measurement methods that compares 3D data generated by a physical system.

Weaknesses:
Overall I'm positive about this paper. But there're still some aspects of the method that I hope the authors could clarify in the rebuttal.

1. Simulated sequence for distance learning. It appears to me that the validity of the proposed similarity measurement relies highly on the simulated sequence. The paper presents an iterative method for determining the simulation step \Delta. However, little is mentioned on how to determine the length n of the simulated sequence. Will different sequence length affect the effectiveness of the similarity model? (which appears to be so based on my understanding.) If so, then how to decide the optimal sequence length for different input data?
Also, in order to simulate the data sequence, the underlying physical mechanism needs to be known. This limits the method to be only applicable to comparing data of a known physical process. In addition, if the simulation model does not match the actual process well, I suspect that the validity of the similarity model will be downgraded.

2. It's not clear how to determine the parameter v (number of elements in vector slices). The current determination method seems be to quite empirical (test on several combinations of b and v). It's not clear whether such combination is universally good for all kinds of volumetric data.

3. Some ablations on the network are missing. In order to prove the effectiveness of the multiscale structure, performance the proposed network should be compared with the baseline network that only uses the native resolution (without the multiscale components). Also some ablations should be done on different sequence lengths and steps.
summary_of_the_review: The paper has good quality. The proposed entropy-based similarity model is novel and seems to be useful. Although some technical details need to be further clarified, overall the work is nice and worth to be presented in ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a similarity model for volumetric simulations derived from Boltzmann’s equation. A formulation of a similarity measure between a reference state and any other microstate in a dissipative physical system is presented. This similarity cannot be used as a metric as is, but can be used to train a Siamese CNN (VolSiM) that is introduced in this paper. The authors formulate a learning task by generating a sequence of physical systems. They use Pearson’s distance to estimate how quickly the microstates in a system are changing, and compute a ground truth for the CNN training using the proposed similarity measure. The network is trained to predict the distances. The CNN is designed to preserve a certain degree of translational, rotational and scale equivariances. Translational equivariance, apart from pooling operations, is given by the nature of CNNs; scale equivariance is directly encoded in the architecture of VolSiM by using four blocks processing the input data in multiple scales; and rotational equivariance is encouraged by data augmentation. The network is given ground truth distances based on the proposed similarity model, predicts distances and uses a loss which is a weighted sum of an MSE term and a correlation term. The authors propose two approaches to generate data, one which is based on varying the initial conditions of PDEs and using PDE solvers for a number of physical simulations, which is used as training data, and one which is based on spatial and temporal variations, which is used to generate test data. Furthermore, a publicly available database is used for testing. The datasets in the experiments cover a wide range of different physical models.
main_review: The paper is very well written and structured, and it is easy to read. The goal of the paper and its contributions are clearly formulated. The literature review nicely places the proposed work within its field. The argumentations in the paper are well funded and design choices motivated. 

The authors included an ethics and reproducibility statement, and announced to publish the complete data and implementation upon acceptance of this paper.

Regarding the equivariance of the proposed CNN. Instead of encouraging the network to learn rotational equivariant features by data augmentation, have you considered to use group convolutions or steerable filters to use a network which is rotational equivariant in every layer? 
The plot to explore the rotational equivariance of the metric in Fig. 7 could be interesting to visualize as a polar plot.

Minor comments:
Typo in “[...] against a know ground truth” in the last paragraph of sec. 1. 
The abbreviation DNS (Direct Numerical Simulation) is not introduced in the main paper, only in the appendix.




summary_of_the_review: I believe the contribution is relevant to a large audience, is well and clearly written, and method aspects are clearly argued for. The datasets include a lot of variation, and the results are thoroughly discussed. I think the semi-supervised approach to learn a metric for comparing simulation or measurement data based on the proposed similarity model is useful for a wide range of applications and believe the authors also present a nice line of argumentation for design choices which are applicable to other tasks, hence I recommend accepting the submission.
review_title: null

Meta-review: This paper studies a data-driven similarity metric for physical simulation data, based on entropy rate of a physical system. The authors consider a one-parameter family of spatial fields obtained by varying certain parameter, and use those in a self-supervised setup. 
Reviewers were split in this submission. While some reviewers highlighted the novelty in the problem setup and the idea of considering one-parameter families, they also expressed concern about the lack of proper justification of the entropy analogy, as well as doubts on the empirical evaluation. Ultimately, and taking all these considerations into account, the AC believes this work would greatly benefit from another review cycle, by addressing the concerns expressed here. Therefore, the AC recommends rejection at this time.
Decision: Reject