Learning Similarity Metrics for Volumetric Simulations with Multiscale CNNs
Authors: Georg Kohl
Liwei Chen
Nils Thuerey
Venue: ICLR 2022 Submitted (2022)
Keywords: metric learning
PDEs
numerical simulation
physical modeling

Abstract: Simulations that produce three-dimensional data are ubiquitous in science, ranging from fluid flows to plasma physics. We propose a similarity model based on entropy, which allows for the creation of physically meaningful ground truth distances for the similarity assessment of scalar and vectorial data, produced from transport and motion-based simulations. Utilizing two data acquisition methods derived from this model, we create collections of fields from numerical PDE solvers and existing simulation data repositories, and highlight the importance of an appropriate data distribution for an effective training process. Furthermore, a multiscale CNN architecture that computes a volumetric similarity metric (VolSiM) is proposed. To the best of our knowledge this is the first learning method inherently designed to address the challenges arising for the similarity assessment of high-dimensional simulation data. Additionally, the tradeoff between a large batch size and an accurate correlation computation for correlation-based loss functions is investigated, and the metric's equivariance with respect to rotation and scale operations is analyzed. Finally, the robustness and generalization of VolSiM is evaluated on a large range of test data, as well as a particularly challenging turbulence case study, that is close to potential real-world applications.

Under review as a conference paper at ICLR 2022
LEARNING SIMILARITY METRICS FOR VOLUMETRIC SIMULATIONS WITH MULTISCALE CNNS
Anonymous authors Paper under double-blind review
ABSTRACT
Simulations that produce three-dimensional data are ubiquitous in science, rang- ing from fluid flows to plasma physics. We propose similarity model based on entropy, which allows for the creation of physically meaningful ground truth dis- tances for the similarity assessment of scalar and vectorial data, produced from transport and motion-based simulations. Utilizing two data acquisition methods derived from this model, we create collections of fields from numerical PDE solvers and existing simulation data repositories, and highlight the importance of an appropriate data distribution for effective training process. Further- more, a multiscale CNN architecture that computes a volumetric similarity met- ric (VolSiM) is proposed. To the best of our knowledge this is the first learning method inherently designed to address the challenges arising for the similarity as- sessment of high-dimensional simulation data. Additionally, the tradeoff between a large batch size and an accurate correlation computation for correlation parelation-based loss functions is investigated, and the metric's 's invariance with respect to rotation and scale operations is analyzed. Finally, the robustness and generalization of VolSiM evaluated on a large range of test data, as well as a particularly chal- lenging turbulence case study, that is close to potential real-word world applications.
INTRODUCTION
Making comparisons is a fundamental operation that is essential for any kind of computation. This especially true for the simulation of physical phenomena, as we are often interested in comparing simulations against other types of models or measurements from the physical system. Mathemati- cally, such comparisons require metric functions that determine scalar distance values as a similarity assessment. A fundamental problem is that traditional comparisons are typically based on simple, element-wise metrics like the L or I distances, due to their computational simplicity and lack of alternatives. Such metrics can work reliably for systems with few elements of interest, e.g. if we want to analyze the position of a moving object at different points in time, matching our intuitive un- derstanding of distances. However, more complex physical problems often exhibit large numbers of degrees of freedom, and strong dependencies between elements in their solutions. Those coherences should be considered when comparing physical data, but element-wise operations by definition ig- nore such interactions between elements. With the curse of dimensionality this situation becomes significantly worse for systems that are modeled with dense grid data, as the number of interac- tions grows exponentially with a linearly increasing number elements. Such data representations widely used, e.g. for medical blood flow simulations (Olufsen et al., 2000), over climate and weather predictions (Stocker et al., 2014), to the famous unsolved problem of turbulence (Holmes al., 2012). Another downside of element-wiss metries that each element weighted equally, which is typically suboptimal: E.g. in the context of fluids, smoke plumes behave differently along the vertical dimension due to gravity or buoyancy, and small key features like vortices are more indicative of the fluid's general behavior than large areas of near constant flow (Pope. 2000).
In the domain of natural images, neural networks have been successfully employed for similarity models that can consider larger structures, typically via learning from the semantic meaning of the images through class labels or data that encodes our human perception. Similarly, physical systems exhibit spatial and temporal coherence due to the underlying laws of physics, from which we can derive a similarity model. To robustly learn similarity assessments of sealar and vectorial volumetric data from such model, work makes the following contributions:

---

Under review as a conference paper at ICLR 2022
We propose novel similarity model based on the entropy of physical systems. It is em- ployed to synthesize sequences of volumetric physical fields suitable for metric learning.
We show that our Siamese, multiscale feature network results in a stable metric that suc- cessfully generalizes to new physical phenomena. To the best of our knowledge this is the first learned metric inherently designed for the similarity assessment of volumetric fields.
The metric is employed to analyze turbulence in a case study, and its invariance to rota- tion and scale are evaluated. In addition, we analyze correlation-base baated loss functions with respect to their tradcoff' between batch size and accuracy of correlation computation.
The central application of the proposed VolSIM metric is the similarity assessment of new physical simulation methods, numerical or learning-based, against known ground truth. This ground truth can take the form of measurements, higher resolution simulations, or existing models. Furthermore, the trained metric be used differentiable similarity loss for physical learning problems, similar to perceptual losses for computer vision tasks.
RELATED WORK
Apart from simple L distances, the two metrics PSNR and SSIM (Wang al.. 2004) are commonly used across disciplines for the similarity assessment of data. Similar to the underlying L distance, PSNR shares the issues of element- wise metries (Huynh- The & Ghanbari, 2008; 2012). SSIM com- putes more intricate function, but it was shown to be closely related to PSNR (Horé & Ziou, 2010) and thus has similar problems (Nilsson & Akenine-Möller, 2020). Furthermore, statistical tools like the Pearson correlation coefficient PCC (Pearson, 1920) and Spearman's 's rank correlation co- efficient SRCC (Spearman, 1904) can be employed as a simple similarity measurement. There are several learning-based metries specialized for different domains such rendered (Andersson et al., 2020) and natural images (Bosse et al., 2016), interior object design (Bell & Bala, 2015), audio (Av- goustinakis et al., 2020), and haptic signals (Kumari et al., 2019). Especially for images, similarity measurements have been studied in multiple settings, mainly by combining deep embeddings as per- ceptually more accurate metrics (Prashnani et al., 2018 8; Talebi & Milanfar, 2018). Similarly, such metries can be employed for various image related tasks like super-resolution (Johnson et al., 2016) or image generation (Dosovitskiy & Brox, 2016). We additionally study the behavior of invari- ance and equivariance to transformations, which was targeted previously for rotational symmetries (Weiler et al., 2018; Chidester et al., 2019) and improved generalization (Wang et al., 2021).
Traditional metric learning for natural image domains typically works in one of two ways: Either, the training is directly supervised by learning from manually created labels, e.g. via two-alternative forced choice where humans pick the most similar option to a reference (Zhang et al., 2018). Or, the training is indirectly semi-supervised through images with class labels and contrastive loss (Chopra et al., 2005; Hadsell et al., 2006). In that case, triplets of reference, same class image, and other class images are sampled, and the corresponding latent space representations are pulled together pushed apart. We refer to Roth al. (2020) for overview of different image metric training strategies. Similarity metrics for simulation data have not been studied extensively yet. For smoke flow synthesis, Siamese networks for finding similar fluid descriptors have been applied (Chu & Thuerey, 2017). Um et al. (2017; 2021) used crowd-sourced user studies for the similarity assess- ment of liquid simulations. Simulation data was compared via learned metric (Kohl et al., 2020) using a feature extractor with a structure similar to classification CNN, however only considering scalar 2D data and a simple, linear distance model.
Deep learning techniques also receive attention for improving simulations, e.g. for predicting states over time (Morton et al., 2018; Kim et al., 2019), for learning via physical laws (Watters al., 2017; Greydanus et al., 2019), and for the dynamics of complex materials (Ummenhofer et al., 2020; Sanchez-Gonzale et al., 2020). While these approaches have potential to be combined with our metric, we focus on traditional numerical methods to acquire data in the following.
3 MODELING SIMILARITY OF SIMULATIONS
To formulate our methodology for learning similarity metrics that target dissipative physical sys- tems, turn to the fundamental quantity of entropy. The second law of thermodynamics states
2

---

Under review as a conference paper at ICLR 2022
that the entropy S of physical system never decreases, thus AS 2 0. In the following,
make the reasonable assumption that the behavior of the system is continuous and non-oscillating
and that AS > 0. On the right, Eq. is the Boltzmann equation from statistical mechanics (Boltzmann, 1866). that describes 5 in terms of the Boltzmann constant Ky and the number of microstates W of a system (here an ideal gas).' Since entropy only depends on single system state, it can be reformulated to take the relative change between two states into account: given a sequence of states so: .1...., Sn, the relative entropy S(s) is defined in Eq. 2, where w, is the monotonically increasing, relative number of microstates defined as 0 for and as 1 for Sn- >> 0 is a system-dependdet factor that determines how quickly the number of microstates increases. Next, we shift Eq. such that the lower bound is O to derive first similarity model D. This offset is necessary, since the properties of similarity metries dictate that distances are always non-negative and only zero for identical states. Finally, constant & appropriately, through linear transformations, and thus k does not influence the general behavior of D. Choosing k 1/olo + 1) leads to the full similarity model ((s) in Eq. 4, that is bound to the range [0, 1/, and directly predicts the level of similarity between the reference 8o and any state 8 from a sequence.
S k log(W)
(1)
(() k log(cw.)
(2)
D(s) = k log(e wis + 1)
(3)
D(8) = log(e W. +
(4)
log(c +
choose the gystem-dependee
relative similarities do not change
Fig. illustrates this connection between the logarithmic increase of entropy within a system and the proposed sim- ilarity model. In addition, trajectories for different A through this space are shown. This A between states corresponds e in the similarity model in Eq. 4, and is directly proportional to it: For large changes Ainf to the system, the entropy quickly increases and the states $1...., 8n contain little information, hence are quickly very dissimilar to so (red dotted curve). For the small- est possible changes A., entropy slowly increases while similarity linearly decreases, due to the continuous nature of the system (green dashed line). Ideally, a sequence of states employed for learning tasks should evenly exhibit both regimes as well intermediate ones, as indicated by the black curve. For practicality, this should not rely on an extremely large number of states, as would be necessary for A, In the fol- lowing, we will refer to this property of sequence being informative with respect to similarity analysis as difficulty of a sequence with a fixed length n. The red area in Fig. I generally corresponds to sequences that are too difficult, i.c. that do not bare any similarity to sg very quickly, while the green area generally corresponds to sequences that are too easy, i.e. that barely change.
high
low
Smillaiit to So
Awt
high
States Si...,, of physical system
Figure I: Idealized model of the behav- ior of entropy and similarity for phys- ical system for different changes A.
The resulting formulation for represents an idealized sim- ilarity model and does not directly work as metric due to two reasons: 1) a metric only takes two states as an input, so estimating would be difficult, and 2) both inputs do not necessarily have to come from the same system, for example when comparing velocity observation to simulated ground truth. To overcome this, we formulate a semi-supervised learn- ing problem by creating sequences 8o, $1,..., Sn of different physical systems, and training a neural network m to predict the distance d given the reference distances g as computed via D. This technique incorporates the underlying physical be- havior. compared to adding variations in a post-process (as is often done in the domain of images, c.g. by Ponomarenko et al. (2015)). Alg. shows how d and g are computed: First, we estimate by evaluating Pearson's 's distance that indicates how quickly the states become uncorrelated, normalizing it to [0, 1]. and optimizing Eq. 4 for via a standard least-squares
Algorithm 1 Computation of dis- tance prediction d of metric and ground truth distances on an or- dered sequence so, S1,...,, Sp.
for i = 1 to n do q gappend (1 PCC((s, 8:))
q + normalize q to [0, 1]
c + fit log(10" g-1) for 1og(10++1)
for i = t do
for j= 1 to n do
d.append(m((s.. 8,) w..append((( 1)/n) log(10' +1)
1og(10++1]
We do not have any a priori information about the distribution of the likelihood of each microstate in a general physical system. Thus, more generic entropy models such as Gibb's entropy or Shannon's 's entropy, hoth of which assume non-uniform distribution of microstates, not immediately applicable.
3

---

Under review as a conference paper at ICLR 2022
optimization. Then, 9 directly follows from Eq. 4 through the fit of and the linearly increasing number of microstates Wg. while is computed by a pairwise evaluation of m.
4 SEQUENCE CREATION
To create sequences 80: 81, 8n within a controlled environment, we make use of the knowledge about the underlying physical processes: We either employ direct changes, based on spatial or tem- poral coherences to so, or use changes to the initial conditions of the process that lead to so- The central challenge now becomes how to find suitable sequences via the magnitude of A, without knowing a priori which degree of non-linearity in the similarity model provides the most informa- tion for each setup. However, given value of A and corresponding sequence, we can compute how strongly it differs from a sequence with A., as depicted in Fig. 1. This is achieved by a proxy similarity function, via computing the PCC between this proxy and a simple linearly decreasing sim- ilarity as predicted by A. If the resulting correlation is high, the sequence is close to a similarity behavior as predicted by A.. while lower correlations indicate stronger deviations. Finally, we sam- ple sequences based on simple correlation thresholds to acquire a suitable difficulty level. By using the mean squared error (MSH) as a proxy, we empirically determined that correlations between 0.65 and 0.85 work well for all cases we considered. Hence, we propose two semi-automatic iterative methods (see Fig. 2) to create data, depending on the method to introduce variations to given state. Both methods iteratively sample small set of sequences to calibrate A to a suitable magnitude and use that for the full data set. Compared to a stricter sampling of every single sequence, this method avoids potential biases in the data and is computationally more efficient as less sampling is needed.
p.F. p.v sends
V sond
Noise added to simulation field strength noiselt (t. seed]]
Random
Spatio-temporal data repository Dir.x.y.2)
Data sequence output
spatio-temporal starting position = rand (seed)
Data sequence output
Varied inttial condition
Increasing change
P. p,+1-,, p. p,
solve
5g
Spatial cubical time P, around center
PDE soliver with time steps fosher
St
g+ +1 +.
p q. +2:4, A
Pa.y.e
p+q. fq.+t. A.
Random spatial jitee q strength-rund (seed)
Other initial conditions p'pp.
Adjust difficulty via updating strenghh
harder if > 0.85 Jeasier
Adjust difficulty via updating A Au.. Strengthh
harder if > 0.85
MSE correlation
(easier r
MSE correlation
Figure 2: Iteration schemes to create data sequences of decreasing similarity. Variation from the reference state be introduced via the initial conditions of numerical PDE simulation (method [A], left), or via spatio-temporal data changes on data from a repository (method [B], right).
[A] Variations based on Initial Conditions of Simulations Given a numerical PDE solver and set of initial conditions or parameters p. the solver computes solution to the PDE over the time steps tg, .......ff 4t. To create a larger number of different sequences, we make the systems non- deterministic by adding noise simulation field and randomly generating the initial conditions from a given range. Adjusting one of the parameters p; in steps with a small perturbation Ai, allows for the creation of sequence sp, 81, 8n with decreasing similarity the unperturbed simulation so. This is repeated for every suitable parameter in p. and the corresponding A is updated individually until the targeted MSE correlation range is reached. The global noise strength factor also influences the difficulty to some degree and can be updated.
[B] Variations based on Spatio-temporal Coherences For a source of volumetric spatial- temporal data without access to solver, we rely on larger spatial and/or temporal dimension than the one required for a sequence. We start at a random spatio-temporal position to extract a cubical spatial area 8o around it. p can be repeatedly translated in space and/or time by Ar.r,, to create sequence 8p, 81...., 8n of decreasing similarity. Using a different position leads to new sequences, as long as the repository features enough diverse data. It possible to add global amount of random perturbations q with chosen strength to the positions, to further increase the difficulty.
Data Sets To create training data with method [A], we utilize solvers for a basic Advection- Diffusion model (Adv), Burger's equation (Bur) with additional viscosity term, and the full Navier-Stokes equations via a Eulerian smoke simulation (Smo) and a hybrid Eulerian- Lagrrngina liquid simulation (I.i q). The corresponding validation sets are generated with separate set of
4

---

Under review as a conference paper at ICLR 2022
random seeds. Furthermore, use adjusted versions of the noise integration for two test sets, by adding noise to the density instead of the velocity field in the Advection- Diffusion model (AdvD), and overlayed noise to the background area of the liquid simulation (LicN).
We create seven test sets via method [B]. Four come from the Johns Hopkins Turbulence Database JHTDB (Perlman et al., 2007) that con- tains a large amount of direct numerical simula- tion (DNS) data, where each is based on a sub- set of the JHTDB and features different char- acteristies: isotropic turbulence (Iso), a chan- turbu-
flow (Cha), maagneto-hydodyyee lence (Mhd), and transitional boundary layer (Tra). Since turbulence contains structures of interest across all length scales, we additionally randomly stride or interpolate the query points for scale factors in [0.25, 4] to create sequences of different physical size. One additional test set (SF) via temporal translations is based ScalarFlow (Eckert et al., 2019), consisting of 3D reconstructions of real smoke plumes. Fur- thermore, method [B] is slightly modified for two synthetic test sets: Instead of using data repository, we procedurally synthesize spatial fields. For that, we employ linearly moving random- ized shapes (Sha), and randomized damped waves (Wav) of the general form f(z) = cos(#) *e=.. All data was gathered in sequences with = 10 at resolution 1283, and downsampled to 64 for computational efficiency during training and evaluations. Fig. 3 shows parts of multiple example sequences via mean projection to 2D, and App. B and C contain further details each data set.
Figure 3: Example sequences from training sets (top, column-wise Adv, 3 Smo, Bur, 3x and test sets (bottom, column-wise. 2xIso, Cha, Mhd, Tra, AdvD, Li qN, SF, Sha, Wav).'
Data Distribution There are many independent factors that influence the difficulty of the created data sequences, and measuring the similarity in the generation process with proxy function in- troduces additional uncertainties. In addition, our iteration schemes only calibrate A to suitable magnitude, instead of enforcing stricter bounds on every single sequence. As a result, the computed PCC values from the MSE (see Fig. 2) on the full data sets exhibit a more natural, smooth distri- bution with controllable difficulty, instead of introducing an artificial distribution with hard cutoffs. Due to the central limit theorem, expect normal distribution of the PCC values that indicate the sequence difficulty. However, correlations have a fixed upper bound of 1, meaning a truncated normal distribution is more accurate prediction. Intuitively, this corresponds to trajectories with different curvature in the similarity model in Fig. 1. In fact, we empirically determined that only training data distributions which reasonably closely follow this prediction, result in successful training of our model. We assume, that significantly different distributions indicate unwanted biases in the data, which prevented effective learning in our experiments. Fig. shows the correlation his- tograms of our training and validation data sets, all of which roughly follow this desired truncated normal distribution.
Smoke
Liquid
Dnsection-Diffuston
Burger's Equation
20
0
0.0
Figure 4: MSE correlation histograms of training data (orange) and validation data (blue) with a bin size of 0.05. Each histogram roughly follows a truncated normal distribution.
5 LEARNING A DISTANCE FUNCTION
For our method, we generally follow the established Siamese structure, that was originally proposed for 2D domains (Zhang et al., 2018; Kohl et al., 2020): First, two inputs are embedded in latent
Here the predominant x-component in Cha and separately normalized for more clear visualization.
5

---

Under review as a conference paper at ICLR 2022
space using a CNN feature extractor. The Siamese network structure means that the weights shared, which ensures the mathematical requirements for a pseudo-metric. Next, the features are normalized and compared with an element-wise comparison like absolute squared difference. Finally, the feature difference is aggregated with sum, mean, and learned weighted average func- tions. To compute the proposed VolSiM metric, that operates inherently more complex 31) data, the following changes to this framework are proposed.
ScaleBlock
Input
5x5x5 Conv. sm2
3x3x3 Cony, 9
2x2x2 AvgPool
4x4x4 AvgPool
8x8k8 AvgPool
5x5x5 Conv.
ReL
Relu
ReLU
3x3K3 Conv,
ReLu
Concalenation
Concatenation
Concalenation
ScaleBlock 4
ScaleBlock
ScaleBlock 16
ScaleBlock 32
each Conv:c c channels, "samne" paddiing
native resolution
1/2resolution
1/4 resolution
r/soluttio
Figure 5: Standard Conv+ReLU blocks (left) are interwoven with input and resolution connections (blue dotted and red dashed), to form the combined network architecture (right) with about 350k: weights. The output of each scale block is concatenated with the downsampled input for lower scales, leading to features that are spread across multiple resolutions for a stable metric computation.
Multiscale Network Scale is important for a reliable similarity assessment, since physical systems often exhibit self-similar behavior that does not significantly change across scales, as indicated by the large number of dimensionless quantities in physics. Generally, scaling a data pair should not alter its similarity, and networks learn such invariance scale most effectively by processing data at different scales. One example where this is crucial is the energy cascade in turbulence (Pope, 2000), which is also analyzed in our case study below. For learned image metrics, this invariance is also useful (but less crucial), and often introduced with large strides and kernels in the convolutions, e.g. via a feature extractor based on AlexNet (Zhang al., 2018). In fact, our experiments with similar architectures showed, that models with large strides and kernels generally perform better than models that modify the scale over the course of the network lesser extent. However, we propose to directly encode this scale structure in multiscale architecture for more accurate similarity assessment, and a network with a smaller resource footprint. Fig. 5 shows the proposed fully convolutional network: Four scale blocks individually process the input on increasingly smaller scales, where each block follows the same layer structure, but deeper blocks effectively cover a significantly larger volume due to the reduced input resolutions. Furthermore, deeper architectures can model complex functions more easily, so we additionally include resolution connections from each scale block the next resolution level via concatenation. Effectively, the network learns mixture of connected deep features and similar representations across scales as result.
Training and Evaluation To increase the model's robustness during training, we used the follow- ing data augmentations for each sequence: the data is normalized to [-1, 1]. and together randomly flipped and rotated in increments of 90º around a random axis. The velocity channels are randomly swapped to prevent directional biases from some simulations, while scalar data is extended to the three input channels via repetition. For inference, only the normalization operation and the repeti- tion of scalar data is performed. The final metric model was trained with the Adam optimizer with a learning rate of 10-4 for 30 epochs via early stopping, further details can be found in App. A. To train the metric end-to-end, Alg. is employed compute the predicted distances d and the ground truth g according to the similarity model. Both are passed to the correlation loss function below, that compares them and provides gradients. To determine the accuracy of a metric during inference, we compute the SRCC between d and w,, where a value closer to 1 indicates a better reconstruction."
Loss Function Given predicted distances d and a ground truth of size n, we train our metric networks with the loss in Eq. 5. It consists of weighted combination of MSE and inverted correlation term r, where d and ğ denote the mean. While the formulation follows existing work (Kohl et al., 2020), it is important to note that g is computed by our similarity model from See. 3, and below we introduce a slicing technique to apply this loss formulation to high-dimensional data sets. For the training process via stochastic gradient descent, Eq. 5 requires a trade-off: large batch size b is useful to improve training stability via less random gradients for optimization. Similarly, a
"This is equivalent to SRCC(d. g), since the SRCC measures monotonic relationships and is not affect by monotonic transformations, SRCC(d, w. ) is just computationally more efficient and has numerical benefits.

---

Under review as a conference paper at ICLR 2022
sufficiently large value of n is required to keep the correlation values accurate and stable.
Ei= (d, - d) (9i g)
g)² 20 + X2 1 2i=i (d. - d)² 2i
(5)
(9: g)²
However, with finite amounts of memory, choosing large values for n and is not possible in prac- tice. Especially for 3D cases, where single sample can already be memory intensive. In general, n is implicitly determined by the length the created sequences, via the number of possible pairs within sequence. Thus, provide an analysis how the correlation ple steps for a fixed n, to allow for increasing not explicitly shown, but all expressions can be extended with vectorized first dimension.
be approximated in multi- return. In the following, the batch dimension is
The full distance vectors d and are split in slices with elements, where should be a proper divisor of n. For
0.74
b=1.v
Validetion Currelation
1.-8.v
any slice k, we can compute a partial correlation r& with
0.70
b..32.mm
E (d: d) (g: - 9) (d. - d)² EiEk
0.68
Tk =
(6)
0.66
RM, A5 no
k (g. g)²
0.64
0.62
ne
Note that this is only approximation, and choosing larger values of for a given is always beneficial, if suf- ficient memory is available. For all slices, the gradients are accumulated during backpropagation since other ag- gregations would required a computational graph of the original, impractical size. Eq. 6 still requires the com- putation of the means d and 9 as pre-process over all samples. Both can be approximated with the running means and for efficiency (RM). For small values of v, the slicing results in very coarse, unsta- ble correlation results. To alleviate that, it is possible to use a rumning mean over all previous values Fk = (1/k)(Tı + Σk=1 71). This aggregation (AG) can stabilize the gradients of individual Tı as they converge the true correlation value during training, App. D shows in more detail.
0.74
0.78
0.78
0.80
0.82
0.84
Test Correlation
Figure 6: Combined validation and test performance for different batch sizes and slicing values (markers), and the usage of a running sample mean R.M and correlation aggregation AG (colors).
Fig. 6 displays the resulting performance on our data, when training with different combinations of RMM and AG. All models exhibit similar levels of memory consumption and were trained with the same random training seed. When comparing models with and without RM (blue vs red and purple vs orange) both
par in most cases, even though computation times for a running mean are about 20% lower. The results of models with and without AG' (blue vs purple and red vs orange) confirm the expected stabilization effect, where reduced fluctuations in the training translate to better generalizing metrics. Overall, this experiment demonstrates that choosing larger consistently leads to better results (marker shape), so more accurate correlations are beneficial over large batch size b in memory limited scenarios. Thus, we use b = and v 55 for the final model.
6 RESULTS
We compare the proposed VolSiM metric to a variety of existing methods in the upper section of Tab. 1. All metrics were evaluated the volumetric data from Sec. which contain wide range of test sets that differ strongly from the training data. VolSiM consistently reconstructs the ground truth distances from the entropy-based similarity model more reliably than other approaches on most data sets. As expected, this effect is most apparent on the validation sets since their distribution is closest the training data. But even on the majority of test sets with very different distribution, VolSiM is the best performing or close to the best performing metric. Metrics without deep learning often fall short, indicating that they were initially designed for different use cases, like SSIM (Wang et al., 2004) for images, or variation of information VI (Meilä, 2007) for clustering. The strictly element-wise metrics MSE and PSNR exhibit almost identical performance, and both work poorly on variety of data sets. The learning-based methods LPIPS (Zhang et al., 2018) for images, and LSiM (Kohl et al., 2020) are only available for vio-dimensional data. Hence, they can not correlate structures along all dimensions. To evaluate them in setting, the 3D data was sliced along all three dimensions to avoid biases against one direction, and the resulting distances are averaged over all slices. As shown in Tab. 1, this results in reasonable performance overall.
7

---

Under review as a conference paper at ICLR 2022
Table 1: Top: performance comparison of different metrics for 3D data via the SRCC, where values closer to 1 indicate a better reconstruction of the ground truth distances. Bold-underlined values show the best method for each data set, bold values are within a 0.01 margin of the best performing. Bottom:
ablation study of the proposed method. Gray models make use of a different training setup.
Validation data sets
Test data
Simulated
Simulated
Generated
HHTDB
Metric
Adv
Bux
Siq
Smo
AdvD
LigN
She
Wav
Isu
Chis
Mhd
Tze
SP
A11
MSE
0.61
0.70
0.51
0.68
0.77
0.76
0.75
0.65
0.76
0.86
0.80
0.79
0.79
0.70
PSNRR
0.61
0.68
0.52
0.68
0.78
0.76
0.75
0.65
0.78
0.86
0.81
0.83
0.79
0.73
SSIM
0.75
0.68
0.49
0.64
0.81
0.80
0.76
0.88
0.49
0.55
0.62
0.60
0.44
0.61
VI
0.57
0.69
0.43
0.60
0.69
0.82
0.67
0.87
0.59
0.76
0.68
0.67
0.41
0.62
LPIPS (21))
0.63
0.62
0.35
0.56
0.76
0.62
0.87
0.92
0.71
0.79
0.76
0.87
LSM (2D)
0.57
0.55
0.48
0.71
0.79
0.75
0.93
0.97
0.69
0.86
0.79
0.81
0.98
0.81
VolSIM rowrs)
0.73
0.66
0.84
0.88
0.96
0.77
0.81
0.88
0.71
0.63
0.81
0.77
0.92
0.93
0.78
0.85
MSnad
0.57
0.66
0.45
0.69
0.76
0.75
0.80
0.78
0.74
01.8
0,80
0.82
0.84
0.74
0.66
0.49
0.77
0.70
0.93
0.96
0.74
0.79
0.83
0.95
MS
0.80
0.70
0.78
0.75
0.86
0.88
0.80
0.95
0.76
0.86
0.78
0.86
0.91
0.82
0.70
0.69
0.70
0.83
0.82
0.95
0.94
0.76
0.80
0.88
0.93
MS+ water
0.78
0.72
0.78
0.78
0.81
0.90
0.94
0.93
0.75
0.85
0.77
0.88
0.93
0.82
MSatost too
0.72
0.77
0.84
0.84
0.92
0.97
0.80
0.86
MSent Zap
0.58
0.62
0.32
0.63
0.78
0.65
0.72
0.92
0.82
0.77
0.86
0.79
0.65
0.75
" Johns Hopkins Turalence Database (Perlman et al., 2007)
"SallaFFow data set (Eckert et 2019)
Combined test data sets
The bottom half of Tab. 1 contains an ablation study of the proposed architecture MS, and simple CNN model. This model is similar to an extension of the convolution layers of AlexNet (Krizhevsky et al., 2017) to three dimension, and does not utilize multiscale structure. Even though VolSiM has more than 80% fewer weights compared to CNN it can fit the training data more easily and works better or equally well on every single dataset in Tab. 1, indicating the strengths of the proposed multiscale architecture. The performance of untrained models CNN and MS confirm the findings from Zhang et al. (2018), who also report surprisingly strong performance of random networks. Using no resolution skip connections for MS stip (see Fig. 5) slightly lowers the metrics generalization across most data sets. Removing the last resolution scale block for MS3 reales overly reduces the capacity of the model, while adding another block for woles only leads to minor differences. In addition, we also investigate two slightly different training setups: for MS Iso we integrate extra sequences created like the Iso data in the training, while MS Iso is exclusively trained on such sequences. MS 1sa only slightly improves upon the baseline, and even the ceoullnce-specifii MS iso model does not consistently improve the results on the JHTDB data sets. Both cases indicate high level of generalization for VolSiM, as it was not trained on any turbulence data. App. E contains more details for each model and App. F features further ablations.
Transformation Invariance Physical systems are often characterized by Galilean invariance (Mc- Comb, 1999), i.e. identical laws of motion across inertial frames. Likewise, a metric should be invariant to transformations of the input, meaning a constant distance output when translating, scal- ing, or rotating both inputs. Element-wise metrics fulfill these properties by construction, but our Siamese network structure requires an equivariant feature representation that changes along with input transformations to achieve them. As CNN features are translation equivariant by design, we only empirically examine rotation and scale invariance for our multiscale metric and a standard Conv+ReLU model on fixed set of 8 random data pairs from each data set. For the rotation exper- iment, we rotate the pairs in steps of 5 around a random coordinate axis. The emply volume inside the original frame is filled with value of O, and data outside the frame is cut to prevent any scaling.
CNN
ValSiM
CNNNeratned
VolSiM
Distenee Devaiion
Deviation
0.1
0.2
0.0
0.0
-0.2
-0.4
30
80
90
120
150
180
0
30
60
90
120
150
180
0.5 0.75
1.0
1.25
1.75
2.0
0.5 0.75
1.25
1.5
1.75
2.0
Rotation Angle
Rotation Angie
Scale Factor
Scale
Sma
U1
Bur
LigN AdVD
Is0
Mhd
Way
Figure 7: Distance deviation from the mean prediction over different rotation angles (left) and scal- ing factors of the inputs (right) for a simple CNN and the proposed multiscale model.
8

---

Under review as a conference paper at ICLR 2022
For scaling, the data is bilinearly up- or downsampled according the scale factor, and processed fully convolutionally by each network. In Fig. 7, the resulting distance deviation from the mean of the predicted distances is plotted for rotation and scaling operations. The optimal result would be a perfectly equal distance with zero deviation across all transformations. Compared to the model mainedd it can be observed that VolSiM produces less deviations overall, and leads significantly smoother distance curves. Note that we observe scale equivariance rather than invariance for VolSiM, i... a mostly linear scaling of the distances according to the input size. This is not as problematic as the unstable behavior of mNNimined as the general order is preserved and data is typically not compared across resolutions without resampling. The main reason that the fully convolutional features are larger and thus lead to scale equivariant distances when compared. With a normalization for the physical size of the system, this effect could be further reduced in future work.
Turbulence Analysis As particularly challenging test for generalization, we further perform a detailed case study on forced isotropic turbulence. This study resembles a potential real- world scenario for our metric. For this purpose, fully resolved raw data over long temporal interval from the isotropic turbulence data set from JHTDB is utilized (see bottom of Fig. 8). The 1024* domain is filtered and reduced to a size of 1283 with strides of 8. meaning VolSiM is applied in a fully convolutional manner, and has to generalize beyond the training resolution of 64 Three different time spans of the simulation are investigated, where the long span also uses a temporal stride of 20. Traditionally, turbulence research makes use of two-point correlations to study such cases (Pope, 2000). Since we are interested in comprchensive spatial analysis instead of two single points, can make use of Pearson's 's distance the full fields to obtain reference values for an evaluation. App. G contains further details on the implementation of this case study.
Case Study:
(nermalzee)
Short time span
SPOC Medium time span
Long time span
1.00
SROC, 1.00
SPCC in
0.75
0.50
PCC
Similarity model VoisiMiC, C)
0.25
0.00
20
200
400
1000
2000
3000
4000
Time
Time
Time
1-1
2-20
A- 1-100
T- 200
7-300
[-400
t-1000
T-
7-3000 T- 4000
Figure 8: Top: Analysis of forced isotropic turbulence across three time spans. The high SRCC values indicate strong agreement between Pearson's 's distance and the similarity model (SRCC,). and between the similarity model and VolSiM (SRCC,). Bottom: Examples from the overall sequence, visualized via a mean projection along the x-axis and color-coded channels.
At the top of Fig. 8, we find that our similarity model from Sec. 3 (gray dotted) closely follows this correlation- based trajectory (red dashed) over time, as also indicated by the value SRCC, between both cases. Even though there are smaller fluctuations, the proposed VolSiM metric (blue) results very similar distance trajectory to the similarity model (see SRCC,) across all time spans. Our similarity metric faithfully recovers the correlation-basee reference, despite not having seen any turbulence data at training time. Hence, this experiment indicates the accuracy of the similarity model, and the capabilities for generalization of the multiscale metric network to new cases.
7 CONCLUSION
We presented the multiscale CNN architecture VoISiM, and demonstrated its capabilities as a simi- larity metric for volumetric simulation data. A similarity model based on the behavior of entropy in physical systems was proposed and utilized to learn a robust, physical similarity assessment. Dif- ferent methods to compute correlations inside a loss function were analyzed, and the invariance to scale and rotation transformations investigated. The proposed metric potentially has an impact on a broad range of disciplines where volumetric simulation data arises. An interesting area for future work is designing metric specifically for turbulence simulations, first steps towards which were taken with our case study. Additionally, investigating saaming-based methods with features that are by construction equivariant rotation and scaling may lead to further improvements in the future.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Since target the fundamental problem of the similarity assessment of numerical simulations, we do not see any direct negative ethical implications of our work. However, there could be indirect negative effects since this work can act as a tool for more accurate and/or robust numerical simula- tion methods in the future, for which military relevance exists. A further indirect issue could be explainability, e.g. when simulations an engineering process yield unexpected inaccuracies.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of work, took the following steps: An extensive appendix with details and hyperparameters regarding the implementation, data sets, and sequence creation is provided below. Furthermore, the trained VolSiM model and source code evaluate are provided alongside this submission.
The complete data and implementation of this work will be published upon acceptance. This in- cludes the source code to create the data sets as well as the full data sets themselves, the source code recreate all figures, and the source code train the metric end-to-end and evaluate it.
REFERENCES
Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. Flip: A difference evaluator for alternating images. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 3(2):15:111::23, 2020. doi: 145/34406883..
Pavlos Avgoustinakis, Giorgos Kordopatis- Symeon Papadopoulos, Andreas L. Symeonidis, and Ioannis Kompatsiaris. Audio-baseed near-duplicate video retrieval with audio similarity learn- ing. arXiv:2010.08737 [cs, eess], 2020. URL http: /wwpiv..ivvv/..////////aa//aaaia org/ abs 2010. 08737.
Sean Bell and Kavita Bala. Learning visual similarity for product design with convolutional neural networks. ACM Transactions on Graphics, 34(4):98:1-98:10 2015. doi: 10. 145/2766/96
Ludwig Boltzmann. Über Die Mechanische Bedeutung Des Zweiten Hauptsatzes Der Wärmetheo- rie: (Vorgelegt Der Sitzung Am 8. Februar 1866). Staatsdruckerei, 1866.
Sebastian Bosse, Dominique Maniry, Klaus-Robert Mueller, Thomas Wiegand, and Wojeiech Samek. Neural network-based full-reference image quality assessment. In 2016 Picture Cod- ing Symposium (PCS), 2016. doi: 10. 109/PCS.201 2012670906776
Benjamin Chidester, Tianming Zhou, Minh N. Do, and Jian Ma. Rotation equivariant and invariant neural networks for microscopy image analysis. Bioinformatics, 35(14):1500isg -000--57777000 2019. ISSN 1367-4803. doi: 10. 1093/hoinfommaiic///2333...
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively. with application to face verification. In 2005 IEEE Conference Computer Vision and Pattern Recog- nition (CVPR), pp. 539-546. IEEE Computer Society, 2005. doi: 10900/9R..0000
Mengyu Chu and Nils Thuerey.
Data-driven synthesis of smoke flows with cnn-
based feature descriptors. ACM Transactions Graphics, 36(4):69:1-69:1, 2017.
doi::0.1145337229933776666
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based deep networks. In Advances in Neural Information Processing Systems 29, volume 29, 2016. URL http:://aaaiii org/ abs/ / /602..6.00 026022666 04444
Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: large-scale volumetrie data set of real-world scalar transport flows for computer animation and machine learning. ACM Transac- tions on Graphics, 38(6), 2019. doi: St..:114//335589.35355555
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural net- works. In Advances in Neural Information Processing Systems 32, pp. 15353- 15363, 2019. URL https://proceedingg... iew.eeeeeddiinn...e....ee...e..e wttpppppppppppp.eeeeeeeeeeeeeeee 2t/////h/hhhh//// ciccleeeaceeddefffcccea575555. f8 ratt html.
10

---

Under review as a conference paper at ICLR 2022
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Conference on Computer Vision and Patter Recognition (CVPR), volume 2, pp- 2006, doi:10.1109//00 101.1001111//VPP.2000...
Philipp Holl, Nils Thuerey, and Vladlen Koltun. Learning control pdes with differentiable physies. In 8th International Conference on Learning Representations (ICLR 2020). OpenReview.net, 2020. URL https::////.mm wwwpreweeeeew..eeoo/pp / net www.oromwm.ccyiiinnn....
Philip Holmes, John L Lumley, Gahl Berkooz, and Clarence W Rowley. Turbulence, Coherent Structures, Dynamical Systems and Symmetry, Cambridge University Press, 2012. ISBN 978-0-
511-91970-1, 00t:101017//809881119999999
A. Horé and D. Ziou. Image quality metries: Psnr vs. ssim. 20th International Conference on Pattern Recognition (ICPR 2010), pp. 2366-2369, 2010. 100.1.11109///P.2000.000
Q. Huynh-Thu and M. Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics Letters, 44(13):8008811, 2008. ISSN 0013-5194. doi:10. 1099/22200000022..
Quan Huynh-Thu and Mohammed Ghanbari. The accuracy psnr predicting video quality for different video scenes and frame rates. Telecommunication Systems, 49(1):35 2012. ISSN 1572-9451. doi: 1007/ss 1235-010-9351-x
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision ECCV 2016, volume 9906, pp. 694-711, 2016. doi: 10. 1000100779978333194444773
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus H. Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. Computer Graphics Forum, 38(2):59-70, 2019. doi: 10c1f
Georg Kohl, Kiwon Um, and Nils Thuerey. Learning similarity metrics for numerical simulations. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), volume 119, pp. 5349-5360, 2020. URL http:: http::://pccceddeeee html.
presspppesss.. /v / kohl20a.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017. doi: 10.11.5655366
Priyadarshini Kumari, Siddhartha Chaudhuri, and Subhasis Chaudhuri. Perceptnet: Learning per- ceptual similarity of haptic textures in presence of unorderable triplets. In 2019 IEEE World Haptics Conference (WHC), pp. 163-168. IEEE, 2019. doi:
William David McComb. Dynamies and Relativity. Oxford University Press, 1999. ISBN 0- 19-950112-9. URL https:// adsabs harvard. edu/ /tct///bs/9999// htw.b//..o.rr..o......... ..Maabst
Marina Meilä. Comparing clusterings-ant information based distance. Journal of Multivariate Analysis, 98(5)8873-895, 2007. ISSN 0047-259X. 10i:
1..0066.1..1... 1.013.
Jeremy Morton, Antony Jameson, Mykel J, Kochenderfer, and Freddie D. Witherden. Deep dynam- ical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems 31, pp. 2778-2288, 2018. URL https:ht/tocceeeeeee 2018 htt./aashhhaaaaaaaaaa http:////...000000000033331eeea/
neur www.euriss...papppppaeeeeee html.
Jim Nilsson and Tomas Akenine-Möller. Understanding ssim. arXiv:2006. 13846 [cs, eess), 2020. 13846.
URL http::/t/heee http:://rxxivoogg//aaaa.
Mette S Olufsen, Charles S Peskin, Won Yong Kim, Erik M Pedersen, Ali Nadim, and Jes- per Larsen. Numerical simulation and experimental validation of blood flow arteries with structured-tree outflow conditions. Annals of Biomedical Engineering, 128119918,,1 2000.
Karl Pearson. Notes on the history of correlation.
Biometrika, 13(1):25-45 1920.
doi: 10. 3...9/
11

---

2018 IEEE International Conference on Computational Photography (ICCP), 2018. doi:10. 11//1//CHHT.2008..666666 Under review as a conference paper at ICLR 2022
Eric Perlman, Randal Burns, Yi Li, and Charles Meneveau. Data exploration of turbulence simula- tions using a database cluster. In Proceedings of the ACM/IEEE Conference on High Performance Nenworking and Computing, pp. 1-11, 2007. doi: 145.1111114//1322222333332
Nikolay Ponomarenko, Lina Jin, Oleg leremciev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C. C. Jay Kuo. Image database tid2013: Peculiarities, results and perspectives. Signal Processing-Imase Communicationn, 30: 57-77, 2015. doi:10. 10i:10.1016/j wma...aaaa...1.11......
Stephen Pope. Turbulent Flows. Cambridge University Press, 2000. ISBN 978-0-511-84053-1. 1oi:10.1017/CBBB 1111100017//0007881118188 1840531..
Ekta Prashnani, Hong Cai. Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image- eтror assessment through pairwise preference. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1808-1817. IEEE Computer Society, 2018.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjöm Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In Proceed- ings of the 37th International Conference on Machine Learning (ICML 2020), volume 119, pp. 8242-8252. PMLR, 2020. URL html.
wwwpeeediinnmm........mmmmmmm..m mml. www..p.....//...m.........e wttpe////11/rrthh00000e
Alvaro Sanchez- ter W. Battaglia. Learning to simulate complex physics with graph networks. In Proceed- ings of the 37th International Conference on Machine Learning (ICMI. 2020), volume 119, pp- 8459 8468. PMLR, 2020. URL http: httt:://proceedinng mlr. press patrees//vii//aa/a sanchez- gonzalez20a. html.
Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Pe-
Charles Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):22-101, 1904. doi: 000100..07/141225599
Thomas Stocker, Dahe Qin, Gian-Kasper Plattner, Melinda Tignor. Simon Allen, Judith Borschung, Alexander Nauels, Yu Xia, Vincent Bex, and Pauline Midgley. Climate Change 2013: The Physical Science Basis. Cambridge University Press, 2014. ISBN 978-1-107-41532-4, doi: 10. 01177//00781100441555
Hossein Talebi and Peyman Milanfar. Learned perceptual image enhancement. In
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Perceptual evaluation of liquid simulation methods.
ACM Transactions on Graphics, 36(4), 2017.
Kiwon Um, Xiangyu Hu, Bing Wang, and Nils Thuerey. Spot the difference: Accuracy of numerical simulations via the human visual system. ACM Transactions on Applied Perception, 18(2):6:1- 6:15, 2021. doi: 10. 145/3449064.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In 8th International Conference Learning Representa- tions (ICLR 2020). OpenReview.net, 2020. URL https: bttps:://penzevieet net wtw.:/rrumteereeee// forum?iniii B11DoJSYDH.
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for im- proved generalization. In 9th International Conference on Learning Representattions (ICLR 2021).
OpenReview.net, 2021. URL httph:ttt///eetc http:////ppeeeeieeeee net www.ttertrmmtteee www.mmidddwwwaaae...mmmmmmm _x2xX..
Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero Simoncelli. Image quality assess- ment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13 (4):600-612, 2004. 10. 109/T1PP2003.819922
12

---

Under review as a conference paper at ICLR 2022
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter W. Battaglia, Razvan Pas- canu, and Andrea Tacchetti. Visual interaction networks: Learning a physics sim- ulator from video. In Advances in Neural Information Processing Systems 30, pp- 4539-4547, 2017. URL https: httt:sttt:///pp//e http://///eoooeeeee....eeeee www.is..cee............ wttp.ccpppp///1///eeeeee
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems 31, pp. 10402-10413, 2018. URL https: wtt.ttsss/ppopeeeeeee ngs. 2et.eeeee004420.ceaeeaaaa3a3a330
petrips.cc/ http:/pppp///////ss//a/ Abstract. html.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In 2018 IEEE Conference Computer 8.00068.
Vision and Pattern Recognition (CVPR), pp. 586-595, 2018.
Yongning Zhu and Robert Bridson. Animating sand as a fluid. In ACM Transactions on Graphics, pp. 965-972, 2005. doi:10. 155.5/1188222.1332299
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
In the following, additional details for the proposed VolSiM metric are provided: App. A contains implementation details regarding the training and the metric model setup, and App. B features gen- eration details for all our data sets. Next, additional examples of the created data sequences are App. D.
displayed in App. C, and the training stability for the loss experiment investigated Details for the experimental setup of the ablation study models are given in App. E, and App. F con- tains additional ablation studies. Finally, details regarding the turbulence case study can be found in App. G.
A IMPLEMENTATION DETAILS
The training and evaluation process of the metric was implemented in PyTorch, while the data was simulated and collected with specialized solvers and data interfaces as described in App B. The data acquisition, training, and metric evaluation was performed on a server with an Intel i7-6850 (3.60Ghz) CPU and an NVIDIA GeForce GTX 1080 Ti GPU. It took about 38 hours of training to fully optimize the final VolSiM model for the data sequences with a spatial resolution of 64
In addition to the multiscale feature extractor network, the following operations were used for the Siamese architecture of the metric: Each feature map is normalized via a mean and standard de- viation normalization to standard normal distribution. The mean and standard deviation of each feature map is computed in a pre-processing step for the initialization of the network over all data samples. Both values fixed for training the metric afterwards. To compare both sets of fea- ture maps in the latent space, a simple element-wise, squared difference is employed. To keep the mathematical metric properties, this also requires square root operation before the final distance output. The spatial squared feature map differences are then aggregated along all dimensions into scalar distance output. Here, we used single learned weight with dropout for every feature map, to combine them to a weighted average per network layer. The activations of the average feature maps are spatially combined with a simple mean, and summed over all network layers afterwards. This process of normalizing, comparing, and aggregating the feature maps computed by the feature extractor follows previous work (Kohl et al., 2020; Zhang et al., 2018).
The weights to adjust the influence of each feature map are initialized to 0.1, all other weights of the multiscale feature extractor are initialized with the default PyTorch initialization. For the final loss, the MSE term was weighted with A 1.0, while the correlation term weighted with X2 0.7.
B DATA SET DETAILS
In the following sections, the details underlying each data set are described. Tab. 3 contains summary of simulator, simulation setup, varied parameters, noise integration, and used fields for the simulated and generated data sets. Tab. features summary of the collected data sets, with repository details, jitter and cutout settings, and spatial and temporal A values. Both tables also contain the number of sequences created for training, validation, and testing for every data source. These values only apply for the general metric setup, and changes for the ablation study models can be found in App. E below.
B. ADVECTION-DIFFUR DIFFUSION AND BURGER'S 'S EQUATION
In its simplest form, the transport of matter in a flow can be described by the two phenomena of advection and diffusion. Advection describes the movement of passive quantity inside velocity field over time, and diffusion describes the process of dissipation of this quantity due to the second law of thermodynamics.
3d πt = - - u Vd
Eq. 7 is the simplified Adveetion-Diffusion equation with constant diffusivity and no sources or sinks, where u denotes the velocity, is a scalar passive quantity that is transported, and v is the diffusion coefficient kinematic viscosity.
14

---

Under review as a conference paper at ICLR 2022
Burger's Equation in Eq. 8 is similar to the Advection-Diffusion equation, but it describes how the velocity field itself changes over time with advection and diffusion. The diffusion term can also be interpreted viscosity, that models the resistance of the material to deformations. Furthermore, this variation can develop discontinuities (also called shock waves). Here, u also denotes the velocity and the kinematic viscosity diffusion coefficient.
Ou Ot
レV u-
(8)
To solve both PDEs, the differentiable fluid framework PhiFlow (Holl et al., 2020) was used. The solver utilizes Semi-Lagrangian advection scheme, and we chose periodic domain boundary condi- tions to allow for the usage of a Fourier space diffusion solver. We introduced additional continuous foreing to the simulations by adding force term f to the velocity after every simulation step. Thus, depends on the time steps that is normalized by division of the simulation domain size be- forehand. For Adv, Bur, and AdvD, we initialized the fields for velocity, density, and force with multiple layered parameterized sine functions. This leads to large range of patterns across multiple scales and frequencies when varying the sine parameters.
u² n²(p)= = sum (FF
Σ fit * (2n(2'mp + cs ( mod
(9)
where c = (1, 1, 0.4, 0.3)
(p. sum (fō (1 fe 20)
Σ fL sin(2' cs of modd))))))
(10)
where p + f 0.5 f sin(3t) and (0, 1, 1, 0.7)
{..z.=)
d(p) sum Σ
sin * 24Tp: o't)
(11)
Eq. 9, 10, and 11 show the layered sine functions in u² (p). f") (p). and d(p) for a spatial grid position E IRS, The sum operation denotes sum of all vector elements here, and all binary operations on vectors and scalars use broadcasting of the scalar value to match the dimensions. Eq. 12 shows the definition of the function parameters used above, all of which on the simulation seed for more diverse simulations. Note that v was multiplied by 0.1 for Bur. The remaining velocity and force components n (p)), u² (p), (p), and (p) and corresponding parameters are omitted for brevity here, since they follow the same initialization pattern as u (p) and f (p). Tab. 3 shows the function parameters that were varied, by using the random initializations and adjusting one of them in linear steps to create a sequence.
randomly sampled based
- -
f U(-022,0.2)
~ 0.15)
f5 -==((-001,0011'
f ((0.0.00000000 0.1)3
ff U(-00_00,)
(0.00002, 0.1002)
(12)
of 100)
o (0 100)3
1 33
fa ~ (1, 3'5''
o Z.V.= u(0, 100)3
The main difference between the Advection -iveetion-Diffusion training data and the test set is the method of noise integration: For Adv is integrated into the simulation velocity, while for AdvD added to the density field instead. The amount of noise added to the velocity for Bur and Adv and to the density for AdvD was varied isolation as well.
15

---

Under review as a conference paper at ICLR 2022
B.2 NAVIER-STORE STOKES EQUATIONS
The Navier-Stokes Equations fully describe the behavior of fluids like gases and liquids, via mod- elling advection, viscosity, and pressure effects, as well mass conservation. Pressure can be interpreted as the force exerted by surrounding fluid mass at a given point, and the conservation of mass means that the fluid resists compression. du Øt + (u V.-VV=
VP
+ Vv
(13)
P
V
0.
(14)
Eq. describes the conservation of momentum, and Eq. 14 describes mass conservation. Again, denotes the velocity. P is the pressure, is the fluids density, v is the kinematic viscosity, and denotes external forces like gravity.
Smoke To create the smoke data set Smo, the fluid framework MantaFlow that provides grid- based Eulerian smoke solver for the Navier-Stokes Equations was used. is based on a Semi- Lagrangian advection scheme and on the conjugate gradient method as pressure solver. The sim- ulation setup consists cylindrical smoke source at the bottom of the domain with a fixed noise pattern initialization create more diverse smoke plumes. Furthermore, constant spherical force field is positioned over the source. This setup allows for variation of multiple simulation param- eters, like the smoke buoyancy, the source position and different force field settings. They include position, rotation, radius and strength. In addition, the amount of added noise to the velocity can also be varied in isolation.
Liquid Both liquid data sets, Liq and LiqN, were created with a liquid solver in MantaFlow. utilizes the hybrid Fulerian-Lagrangiaa fluid implicit particle method (Zhu Bridson, 2005), that combines the advantages of particle and grid-based liquid simulations for reduced numerical dissipation. The simulation setup consists of two liquid cuboids of different shapes, similar to the common breaking dam setup. After 25 simulation time steps a liquid drop is added near the top of the simulation domain, and it falls down on the water surface that is still moving. Here, the external gravity force well the drops position and radius varied create similarity sequences. As for the smoke data, a modification of the amount of noise added to the velocity was also employed varied parameter. The main difference between the liquid training data and the test set is the method of noise integration: For L q it is integrated into the simulation velocity, while for L qN it overlayed on the simulation background.
B.3 GENERATED DATA
To create the shape data set Sha and the wave data set Wav, a random number of straight paths are created by randomly generating start and end point inside the domain. It is ensured that both not too close to the boundaries and that the path has a sufficient length. The intermediary positions for the sequence are result of linearly interpolating on these paths. The positions on the path determine the center for the generated objects that are added to a occupancy marker grid. For both data sets, overlapping shapes and waves are combined additively, and variations with and without overlayed noise to the marker grid were created.
Shapes For Sha, random shapes (box or sphere) are added to the positions, where the shape's size is a random fraction of the path length, with minimum and maximum constraint. The created shapes are then applied to the marker grid either with or without smoothed borders.
Waves For Mav, randomized volumetric damped cosine waves added around the positions instead. The marker grid value m at point p for a single wave around a center e is defined as
m(p) cos( w p) (3.78/r) where
1pp
c||2-
(15)
Here, r is the radius given by the randomized size that is computed as for Sha, and w ~ ((011, is randomized waviness value, that determines the frequency of the damped wave.
B.4 COLLECTED DATA
The collected data sets Iso, Cha. Mhd, and Tra are based on different subsets from the Johns Hopkins Turbulence Database JHTDB (Perlman et al., 2007), that contain different types of data
16

---

Under review as a conference paper at ICLR 2022
from direct numerical simulations (DNS). In these simulations, all spatial scales of turbulence up to the dissipative regime are resolved. The data set SF is based on the ScalarFlow data (Eckert et al., 2019), that contains dense 3D velocity fields of real buoyant smoke plumes, created via multi-view reconstruction technique.
ScalarFlow Since 100 reconstructions of different smoke plumes are provided in ScalarFlow, there is no need to add additional randomization to create multiple test sequences. Instead, we directly use each reconstruction sequence to create one similarity sequence in equal temporal steps. The only necessary pre-processing step is cutting off the bottom part of domain that contains the smoke inflow, since it is frequently not fully reconstructed. Afterwards, the data is interpolated to the full spatial size of 128 to match the other data sets.
The JHTDB subsets typically contain a single simulation with a very high spatial and temporal resolution and variety of fields. We focus on the velocity fields, since turbulent flow data is especially complex and potentially benefits most from better similarity assessment. We can mainly rely on using temporal sequences, and only need to add spatial jitters in some cases to increase the difficulty. As turbulence generally features structures of interest across all length scales, we create sequences of different spatial scales for each subset. To achieve this, we randomly pick cutout scale factor s. If 8 1, we directly use the native spatial discretization provided by the database. For 8 > 1 we stride the spatial query points of the normal cubical cutout by s after filtering the data. For the cutout is interpolated to the full spatial size of 1283 afterwards. Among other details, Tab. 4 shows the cutout scale factors, as well the corresponding random weights.
JHTDB
the size of the cubical cutout is reduced by a factor of s in each dimension, and
C ADDITIONAL EXAMPLE SEQUENCES
Fig. 10, 11, and 12 show multiple full example sequences from all our data sets. In every sequence, the leftmost image is the baseline field. Moving further to the right, the change of one initial param- eter increases for simulated data sets, and the spatio-temporal position offset increases for generated and collected data. To plot the sequences, the 3D data is projected along the z-axis to 2D via simple mean operation. This means, noise that was added to the data or the simulation is typically signifi- cantly less obvious due to statistical averaging in the projection. Velocity data is directly mapped to RGB color channels, and scalar data is shown via different shades of gray. Unless note otherwise, the data is jointly normalized to [o, 1] for all channels at the same time, via the overall minimum and maximum of the data field.
D TRAINING STABILITY FOR LOSS EXPERIMENT
b-8, 1= RM,
y-11, RM,
b-16,
b-16, y-5, RM,
0.4
row
raw
raw
Training Lo5a
0.3
0.05
0.05
0.05
0.05
0.3
0.1
0.0
2000
1500
herations
Itorations
Iteratons
Itoration
Figure 9: Loss curves from the loss function analysis in Fig. for different models. Shown are the raw losses per batch over all training iterations (lighter line), and an exponential weighted moving average with a 0.05 (darker line) for better visual clarity.
To further analyze the resulting stability of different variants of the loss investigated in Sec. 5, Fig. shows training trajectories of multiple models. Displayed are the direct training loss over the training iterations, i.c. one loss value per batch, and a smoothed version via an exponential weighted moving average computed with a 0.05, corresponding to window size of 40. The models are color- coded according to Fig. 6, and were trained with different batch sizes b, slicing values v, usage of the running sample mean R.M, and usage of the correlation aggregation AG. Note that the models were generally trained with the same training seed, meaning the order of samples is should be very similar across all runs (only with potential deviations due race conditions and GPU processing).
17

---

Under review as a conference paper at ICLR 2022
can be observed that models trained with lower batch sizes generally achieve lower training losses, leading to the better generalization on test and validation sets in Fig. 6. Using no AG makes the training procedure less stable and occasionally large loss spikes occur, leading worse generalization as a result.
E ABLATION STUDY DETAILS
In the following, details for the ablation study models in Sec. 6 are provided. The proposed VolSiM metric uses the multiscale architecture MS described in Sec. 5 as feature extractor. For all models, the general training setup mentioned in App. A stays identical, apart from: 1) changes to the feature extractor, 2) different feature extractor, 3) the training amount, or 4) the training data.
Changes to MS Architecture For MS skip the resolution skip connections (red dashed arrows in Fig. 5) are entirely removed. This means each resolution is isolated, and sharing information across features of different levels more difficult. For wuiis senies the last scale block is removed, meaning the architecture from Fig. 5 ends with the 1/4 resolution level, while the 1/8 resolution level is omitted. For the MS$ reales model, one additional scale block is added. In Fig. 5, this corresponds to 1/16 resolution level with a 16 X 16 X 16 AvgPool and a scale block with 64 channels.
Simple CNN Feature Extractor The CNN trained model employs an entirely different feature tractor network, similar to the classical AlexNet architecture (Krizhevsky et al., 2017). It consists of 5 layers, each with 3D convolution followed by a ReLU activation. The kernel sizes (12 5
3 3) decrease along with the strides (4 1 1- 1- 1). while the number of features first increase, then decreases again (32 96 192 128 128). To create some spatial reductions, two x 4 x MaxPools with stride 2 are included before the second and third convolution. The normalization and aggregation of the resulting feature maps is performed as described App. A, in the same way as for the proposed feature extractor network. As result, CNN can also be applied in fully convolutional manner for the scaling invariance experiment in Fig. 7.
Random Models For the random models CNN yand and MS, yand- the corresponding feature extractor along with the aggregation weights are only initialized and not trained. However, to normalize every feature a standard normal distribution, the training data is processed once to determine the feature mean and standard deviation before evaluating the models, as detailed in App. A.
Different Training Data For the MSadded Iso model, we created 400 additional sequences from the JHTDB according to the Iso column in Tab. 4, that are added to the other training data. Note that they utilize different random seeds than the 60 test sequences, so the Iso data set becomes an additional validation set in Tab. 1. Similarly for the MS Ise model, 1000 sequences according to the Isc column in Tab, 4 were collected. They replace all other training data from the original model, meaning the Isc data set becomes validation set, and Adv, Bur, Lig, and Smo become further test sets in Tab. 1. For consistency, the All column in Tab. 1 still reports the combined values of the original test sets for both cases.
F ADDITIONAL ABLATIONS
To further investigate our method, perform three additional ablations in the following. The first ablation focuses on the benefits of our entropy-based similarity model, by replacing it with a simplified identity transformation. The other two replace the multiscale aspect in different ways, to demonstrate the robustness and memory efficiency of the proposed architecture. Tab. 2 shows the resulting SRCC values for these models and the corresponding baselines on our data sets, which are computed as for Tab. 1.
Simplified Similarity Model
We compare the VolSiM model as proposed in the main paper with MS that does not make use of the similarity model based on entropy during training. For that, the logarithmic transformation of w, is replaced with an identity transformation, corresponding to linear ground truth distances, i.e. g w, in Alg. 1. The results in the first block in Tab. indicate the benefits of the entropy-base based ground truth, with improvements across most data sets. Note that the simplified network version is slightly more efficient as the computation of the parameter e in Alg. I can be omitted. However, pre-computing these values for all sequences is computationally very light compared training the full metric network.
18

---

Under review as a conference paper at ICLR 2022
To investigate the multiscale aspect of our architecture, we removed all scale blocks at lower resolutions, such that only the component at the native input resolution remains (see Fig. 5). To compensate for the lower number of network parameters, we scale the number channels in each layer by a factor of 11, leading to the rather shallow but wide network MS; scale with around 360k weights. The original network structure MS₃ scales with the four scale blocks reconstructs the ground truth more accurate for almost all data sets as displayed in the middle block of Tab. 2. Furthermore, the proposed structure requires about five times less memory during training.
Singlescale Model
No Pooling Layers A different approach to analyze the multiscale aspect, is to eliminate all Avg- Pool layers and set all convolution strides to (also see Fig.5). Here, no further adjustments to the network are required neither pooling layers nor strides alter the number of network weights. The resulting MS,n pool model is only slightly worse compared to the baseline, as indicated by bottom block in Tab. 2. However, the MSwith pool model needs about ten times less memory compared the adjust variant during training due to the significantly smaller feature sizes.
Note that for both ablations on the multiscale structure, the training and test data resolution was reduced to 323. Furthermore, the number of channels in each layer for the MSwith poot and MS pool models was reduced by factor of 0.5. Both choices are motivated by memory limitations and are the reason for the slightly different results across the three baseline models in Tab. 2.
Table 2:
Performance comparison of further ablation study models via the SRCC. Shown are model trained with ground truth distances where the proposed logarithmic similarity model was re- placed with identity transformation (top block), and two models without multiscale architecture (middle and bottom block). The upper row in each block is the corresponding baseline.
Validation data sets
Test data sets
Metric
Adv
Baz
Liq
Smo
Advb
1ig8
Sha
Nav
Esc
Cha
Mho.
Ura
8F
All
VolSIM
0.75
0.73
0.66
0.77
0.84
0.88
0.95
0.96
0.77
0.86
0.81
0.88
0.95
0.85
MSstostt
0.75
0.71
0.68
0.73
0.83
0.85
0.87
0.96
0.74
0.87
0.77
0.87
0.94
0.82
MS
0.74
0.74
0.86
0.77
0.82
0.83
0.93
0.97
0.77
0.88
0.82
0.89
0.95
0.84
MS, muse
0.73
0.71
0.78
0.75
0.83
0.76
0.91
0.96
0.73
0.87
0.78
0.88
0.94
0.82
MS
0.70
0.70
0.76
0.72
0.82
0.79
0.94
0.96
0.76
0.83
0.79
0.88
0.97
0.83
MS paol
0.73
0.70
0.72
0.71
0.83
0.77
0.94
0.95
0.76
0.87
0.78
0.86
0.90
0.82
G TURBULENCE CASE STUDY DETAILS
For the case study in Sec. 6, the velocity field from the (Perlman et al., 2007) utilized. Instead of creating adjusted sequences according to the similarity model, the data is directly converted to three sequences of different time spans without any random- ization in this case. A spatial stride of is employed is employed for all cases, and reduce memory consumption an additional temporal stride of 20 is used for the long time span only. The resulting sequences exhibit spatial resolution of 1283, with 40 frames for the short span, 400 frames for medium span, and 200 frames for the long span. Before further processing, each frame is individu- ally normalized to [-1, 1]. To create the results in Fig. 8, the first simulation frame 4 is compared to all following simulation frames &; via Pearson's 's distance 1 PCC((1, t4) (red dashed trajectory) and (1 tx) (blue solid trajectory). Note that Vo/SM is applied fully convolutionally here, as it trained on 64 data. For both cases, the resulting distances normalized to [0. 1] to visually compare them more easily. The gray dotted curve, is the direct application of the proposed similar- ity model via Alg. 1 by fitting e for each time span, described in Sec. 3. The examples from the sequences are visualized as described in App. C, while projecting along the x- x-axis here.
data set from the JHTDB
19

---

Under review as a conference paper at ICLR 2022
Table 3: Data set detail summary for the simulated and generated data sets.
Adv
Smo
Advu
Lägh
Sha
Sequences Irain val test
57
108-51-0
45-0
Equation
Eg.
Eq. 13, 14
Eq.
Simulator
Phillow
PhiFlow
ManiaFl.
Maniali.
PhiFlow
MantaFi,
Mantali,
MantaFi
Simulation
layered sines
lnyered
breaking dam drop
rising plume
layered
breaking dam drop
random shapes
damped waves
sctup
with force field
Time steps
120
120
buogz
buoyy
drops
frot
drope
fa. f2. far for fTi o1 o2 04,
dropy
1 Frots
fi, f2.
dropy
ft fax fa- fas Ja- Jr- o1 o2s noise
drops
fatr
fa.
drops
Varied aspocts
dropeud
fatra
fos fts
dropeuu
shape position
wave position
gravt
01 02
graat
gravy
gravy
graws
frod
gravs noise
noise
sourcEs
sourcey
noise
Noise integration
added to velocity
adiedd to velocity
addedd to velocity
added to
added to density
overlay on non-liquid
overlay on marker
overlay on marker
velocity
velocity
density
Used fields
donsity
velocity
flags levelset
pressure velocity
donsity
velocity
marker
marker
PhiFlow from Holl (2020)
MantaFlow: http: htttpt//mnttaalleee
Table 4: Data set detail summary for collected data sets.
Eso
Cha
Mhd
Tro
SF
Sequences train-val-test
0-0-60
0-0-60
0-0-60
0-0-60
0-0-106 100
JHTDB
Repository
isotropic
JHTDB cbannel
JHTDB
JHTDB transition
ScalarFlow
mhd1024
1024cnarse
Repository size" h
5028
150
1024 x
2048 x 512
1024
10240
100
178
1024 1021
1336
1024
1024
1536 x 2018
Temporal offset A
180
37
95
25
13
Spatial
Ax.2.z
0
0
0
Spatial jitter
0
0.25, 0.5,
0.25, 0.5,
0.25,0.5,
0.25,
Cutout scales
0.75, 1,
0,75, 1,
0.75.
0.75, 1,2
2,3.4
2,3,4
2,3,4
0.14,
0.14.0.14. 0.14,
0.14, 0.14,
0.14,
Cutout scale
0.14,0016,
0.
0.14, 0.16
0.14, 0.30,
random weights
0.11,0.14,
0.14,
0.14,
0.28
0.14
Used fields
velocity
velocity
velocity
velocity
velocity
JJHTDB from Perlman (2007)
ScalarFlow from Eoken al. (2019)
simulations x time steps é x spatial dimensions x. v. cut to 100 x 150 x 100 x 160 x 100 (removing
from y), smoke
fully
20

---

Under review as a conference paper at ICLR 2022
Adv: Silvection-Diffusionn (2xdensity)
Bur: Burger's Equation (2xvelocity)
Smo: Smoke (velocity, density, and pressure)
Liq: Liquid (velocity, leveset, and flags)
Figure 10: Example sequences of simulated training data, where each row features full sequence from different random seed.
21

---

Under review as a conference paper at ICLR 2022
AdvD: Advection-Diffusion with density noise (2x (exditstyy)
LigN: Liquid with background noise (2x velocity
SF: ScalarFlow (2x velocity)
Sha: Shapes (2xmarker, without and with noise)
Wav: Waves (2xmarker, without and with noise)
8
8
Figure 11: Example sequences of simulated (top two data sets), collected (middle data set), and generated (bottom two data sets) test data. Each row contains full sequence from different random seed.
22

---

Under review as a conference paper at ICLR 2022
Iso: Isotropic turbulence (3x
Cha: Channel flow (3xvelocity)
Mhd: turbulence (3 x velocityy
Tra: Transitional boundary layer (3xvelocity)
Example sequences of collected test data from JHTDB, where each row shows a full sequence from different random seed. Notice the smaller cutout scale factor for the middle example in each case. The predominant x-component in Cha and is separately normalized for more clear visualization.
Figure 12:
23


Reviews:
[{"review_id": "fumu1zMSU1I", "reviewer": "Reviewer_nzbf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is very well written and easy to follow. The proposed entropy-based similarity model is new and appears to be effective in evaluating the similarity between volumetric data in presence of rotation and scaling. The proposed method seems to be the first learning-based similarity measurement methods that compares 3D data generated by a physical system.\n\nWeaknesses:\nOverall I'm positive about this paper. But there're still some aspects of the method that I hope the authors could clarify in the rebuttal.\n\n1. Simulated sequence for distance learning. It appears to me that the validity of the proposed similarity measurement relies highly on the simulated sequence. The paper presents an iterative method for determining the simulation step \\Delta. However, little is mentioned on how to determine the length n of the simulated sequence. Will different sequence length affect the effectiveness of the similarity model? (which appears to be so based on my understanding.) If so, then how to decide the optimal sequence length for different input data?\nAlso, in order to simulate the data sequence, the underlying physical mechanism needs to be known. This limits the method to be only applicable to comparing data of a known physical process. In addition, if the simulation model does not match the actual process well, I suspect that the validity of the similarity model will be downgraded.\n\n2. It's not clear how to determine the parameter v (number of elements in vector slices). The current determination method seems be to quite empirical (test on several combinations of b and v). It's not clear whether such combination is universally good for all kinds of volumetric data.\n\n3. Some ablations on the network are missing. In order to prove the effectiveness of the multiscale structure, performance the proposed network should be compared with the baseline network that only uses the native resolution (without the multiscale components). Also some ablations should be done on different sequence lengths and steps.", "summary_of_the_paper": "The paper presents a learning-based method for evaluating the similarity between 3D volumetric data. A new similarity model based on the entropy of a physical system is proposed, which in my opinion is the paper's major contribution. The method combines the new entropy-based model with a Siamese network that is commonly used for similarity measurement on 2D imagery data. The proposed method further modifies the Siamese network in order to process 3D data. Experimental results are demonstrated on turbulence flow data, and are compared with other 2D similarity measurements. ", "summary_of_the_review": "The paper has good quality. The proposed entropy-based similarity model is novel and seems to be useful. Although some technical details need to be further clarified, overall the work is nice and worth to be presented in ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "anJTgYwWOvM", "reviewer": "Reviewer_zceM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and structured, and it is easy to read. The goal of the paper and its contributions are clearly formulated. The literature review nicely places the proposed work within its field. The argumentations in the paper are well funded and design choices motivated. \n\nThe authors included an ethics and reproducibility statement, and announced to publish the complete data and implementation upon acceptance of this paper.\n\nRegarding the equivariance of the proposed CNN. Instead of encouraging the network to learn rotational equivariant features by data augmentation, have you considered to use group convolutions or steerable filters to use a network which is rotational equivariant in every layer? \nThe plot to explore the rotational equivariance of the metric in Fig. 7 could be interesting to visualize as a polar plot.\n\nMinor comments:\nTypo in “[...] against a know ground truth” in the last paragraph of sec. 1. \nThe abbreviation DNS (Direct Numerical Simulation) is not introduced in the main paper, only in the appendix.\n\n\n\n", "summary_of_the_paper": "The paper proposes a similarity model for volumetric simulations derived from Boltzmann’s equation. A formulation of a similarity measure between a reference state and any other microstate in a dissipative physical system is presented. This similarity cannot be used as a metric as is, but can be used to train a Siamese CNN (VolSiM) that is introduced in this paper. The authors formulate a learning task by generating a sequence of physical systems. They use Pearson’s distance to estimate how quickly the microstates in a system are changing, and compute a ground truth for the CNN training using the proposed similarity measure. The network is trained to predict the distances. The CNN is designed to preserve a certain degree of translational, rotational and scale equivariances. Translational equivariance, apart from pooling operations, is given by the nature of CNNs; scale equivariance is directly encoded in the architecture of VolSiM by using four blocks processing the input data in multiple scales; and rotational equivariance is encouraged by data augmentation. The network is given ground truth distances based on the proposed similarity model, predicts distances and uses a loss which is a weighted sum of an MSE term and a correlation term. The authors propose two approaches to generate data, one which is based on varying the initial conditions of PDEs and using PDE solvers for a number of physical simulations, which is used as training data, and one which is based on spatial and temporal variations, which is used to generate test data. Furthermore, a publicly available database is used for testing. The datasets in the experiments cover a wide range of different physical models.", "summary_of_the_review": "I believe the contribution is relevant to a large audience, is well and clearly written, and method aspects are clearly argued for. The datasets include a lot of variation, and the results are thoroughly discussed. I think the semi-supervised approach to learn a metric for comparing simulation or measurement data based on the proposed similarity model is useful for a wide range of applications and believe the authors also present a nice line of argumentation for design choices which are applicable to other tasks, hence I recommend accepting the submission.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Jhk9A-_-xUA", "reviewer": "Reviewer_f7VS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core idea of using one-parameter families of states as ground-truth is nice, but this idea appears to be already present in Kohl et al. 2020. I do not follow the motivation for hypothesizing that similarity should behave like entropy. Classical statistical physics posits entropy as a state function for states *at equilibrium,* where generally no macroscopic physical variation is discernable. When comparing, e.g., velocity fields of a fluid that have complicated macroscopic dynamics, why is a statistical physics perspective relevant? What are the \"microstates,\" and how are they related to the macroscopic dynamical variables over which the metric is being defined? Why is it reasonable or necessary to impose a logarithmic prior on similarity between states in one-parameter families? Is there any advantage to this choice over any other monotonic function of the parameter?\n\nThe logarithmic hypothesis requires the fitting of a parameter $c$, which is then deduced by using a different similarity metric, PCC, as a proxy. If the authors propose their metric as superior, why does the method rely on PCC as part of its construction of the ground truth?\n\nPerhaps most importantly, the empirical results are underwhelming, with simple metrics like MSE and PSNR often coming out on top of the learned ones. Overall, it is not clear to me what technical or empirical novelty the authors demonstrate over past works such as Kohl et al. 2020.\n\n### Miscellaneous ###\n- Algorithm 1 could be expressed more clearly and concisely with formulae rather than an algorithm.\n- \"our iteration schemes only calibrate $\\Delta$ to a suitable magnitude, compared to using a full rejection sampling approach.\" What do you mean by full rejection sampling? What is being sampled? And if this approach would be better, why not use it?\n- It seems from section 5 that the authors use \"equivariance\" to mean \"invariance,\" but then do not observe scale-invariance. The justification for invariance is that \"physical systems exhibit Galilean invariance.\" But the Galilean group does not include scalings, and many systems are not scale-invariant. It seems reasonable to allow scale-equivariance rather than invariance. Section 5 could be cleaned up to clarify this story.", "summary_of_the_paper": "This paper describes a learned similarity metric for spatial scalar and vector fields obtained from physics simulations. The metric is like a perceptual loss but for physical fields. The training is self-supervised by comparing one-parameter families of fields obtained by modifying initial conditions or restricting or resampling.", "summary_of_the_review": "The core idea of using one-parameter families of states as ground-truth is nice, but this idea appears to be already present in Kohl et al. 2020. Overall, it is not clear to me what technical or empirical novelty the authors demonstrate over past works such as Kohl et al. 2020.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies a data-driven similarity metric for physical simulation data, based on entropy rate of a physical system. The authors consider a one-parameter family of spatial fields obtained by varying certain parameter, and use those in a self-supervised setup. 
Reviewers were split in this submission. While some reviewers highlighted the novelty in the problem setup and the idea of considering one-parameter families, they also expressed concern about the lack of proper justification of the entropy analogy, as well as doubts on the empirical evaluation. Ultimately, and taking all these considerations into account, the AC believes this work would greatly benefit from another review cycle, by addressing the concerns expressed here. Therefore, the AC recommends rejection at this time.
Decision: Reject