Learning Similarity Metrics for Numerical Simulations
Authors: Georg Kohl
Kiwon Um
Nils Thuerey
Venue:  (2020)
Keywords: metric learning
CNNs
PDEs
numerical simulation
perceptual evaluation
physics simulation

Abstract: We propose a novel approach to compute a stable and generalizing metric (LNSM) with convolutional neural networks (CNN) to compare field data from a variety of numerical simulation sources. Our method employs a Siamese network architecture that is motivated by the mathematical properties of a metric and is known to work well for finding similarities of other data modalities. We leverage a controllable data generation setup with partial differential equation (PDE) solvers to create increasingly different outputs from a reference simulation. In addition, the data generation allows for adjusting the difficulty of the resulting learning task. A central component of our learned metric is a specialized loss function, that introduces knowledge about the correlation between single data samples into the training process. To demonstrate that the proposed approach outperforms existing simple metrics for vector spaces and other learned, image based metrics we evaluate the different methods on a large range of test data. Additionally, we analyze generalization benefits of using the proposed correlation loss and the impact of an adjustable training data difficulty.

Under review as a conference paper at ICLR 2020
LEARNING SIMILARITY METRICS FOR NUMERICAL SIMULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel approach compute stable and generalizing metric (LNSM)) with convolutional neural networks (CNN) to compare field data from a variety of numerical simulation sources. Our method employs a Siamese network architec- ture that is motivated by the mathematical properties of metric and is known to work well for finding similarities of other data modalities. We leverage con- trollable data generation setup with partial differential equation (PDE) solvers to create increasingly different outputs from a reference simulation. data generation allows for adjusting the difficulty of the resulting learning task. A central component our learned metric a specialized loss function, that in- troduces knowledge about the correlation between single data samples into the training process. To demonstrate that the proposed approach outperforms existing simple metrics for vector spaces and other learned, image based metrics eval- uate the different methods on a large range of test data. Additionally, we analyze generalization benefits of using the proposed correlation loss and the impact of adjustable training data difficulty.
addition, the
INTRODUCTION
Evaluating computational tasks for complex data sets a fundamental problem in all computational disciplines. Regular vector space metrics, such as the 2 distance were shown to be very unreliable (Wang et al., 2004; Zhang et al., 2018), and the advent of deep learning techniques with convolu- tional neural networks (CNNs) made it possible to more reliably evaluate complex data domains such as natural images, texts (Benajiba et al., 2018), or speech (Wang et al., 2018). Our central aim is to demonstrate the usefulness of CNN-based evaluations in the context of numerical simulations. These simulations are the basis for a wide range of applications ranging from blood flow simulations to aireraft design. Specifically, we propose novel learned numerical simulation metric (LNSM) that allows for a reliable similarity evaluation simulation data.
Potential areas of application for such metric are fundamental problems that arise in all areas where such numerical simulations are performed: among others, accuracy evaluations of existing and new simulation methods with respect a known ground truth solution (Oberkampf al., 2004) could be performed more reliably than with regular vector norm. In addition, the energy landscape for parameter inference via optimization through a differentiable solver could be improved in order to match experimental data or observations, and generative networks for physical phenomena such as GANs, could employ a specialized metric to replace commonly used perceptual loss terms.
In this work, we focus on field data, i.e. dense grids of scalars, similar to images, which were generated with known partial differential equations (PDEs) in order to ensure the availability of ground truth solutions. While we focus on 2D data in the following to make comparisons with existing techniques from imaging applications possible, our approach naturally extends to higher dimensions. Every sample of this 2D data be regarded high dimensional vector, metrics the corresponding vector space are applicable to evaluate similarities. These metrics are typically simple, elementwise functions such L L distances. We denote these shallow metrics, and their inherent problem is that they can not capture structures of any scale or contextual information.
This problem is not confined to the field of natural images: it represents fundamental challenge for many areas of simulation, most prominently for the field turbulence simulations (Moin & Mahesh, 1998; Lin al., 1998). Many practical problems require solutions over time, and need

---

Under review as a conference paper at ICLR 2020
Plume (a)
Reference
Plume (b)
Distance to Reference
(a)
(b)
LNSM
Figure 1: Example of field data from fluid simulation of hot smoke with normalized distances for different metrics. Our method (LNSM, in green) approximates the ground truth distances (GT, gray) determined by the data generation method best, i.e., version (a) is closer the ground truth data than (b). An L metric (red) erroneously yields a reversed ordering.
vast number of non-linear operations that often result in substantial changes of the solutions even for small changes of the inputs. Hence, despite being based on known, continuous formulations, these systems can be seen chaotic. We illustrate this behavior in Fig. 1, where two smoke flows compared to a reference simulation. A single simulation parameter was varied for these examples, and visual inspection shows that smoke plume (a) more similar to the reference. This matches the data generation process: version (a) has a significantly smaller parameter change than (b), as shown in the inset graph on the right. Our LNSM robustly predicts the ground truth distances, while the I metric labels plume (b) more similar. In our work, we focus on retrieving the relative distances of simulated data sets. Hence, we do not aim for retrieving the absolute parameter change, but a relative distance that preserves ordering with respect to this parameter.
Using existing image metries based on CNNs for this problem is not optimal either: Natural images only cover a small fraction of the space of possible 2D data, and numerical simulation outputs are located in a fundamentally different data manifold within this space. Hence, there are crucial aspects that can not be captured by purely learning from photographs. Furthermore, we have full control over the data generation process for simulation data. As a result, we can create arbitrary amounts of training data with gradual changes and known ground truth order. With this data, learn metric that is not only able to directly extract and use features, but additionally knows the fundamental interactions between them. The central contributions of our work are:
A Siamese network architecture with feature map normalization which is able to learn metric that generalizes well to unseen simulations methods.
We also propose an improved loss function that combines a batchwise correlation loss term with mean squared error improve the accuracy of the learned metric.
In addition, we show how data generation approach for numerical simulations can be em- ployed to train networks with general and robust feature extractors for metric calculations.
2 RELATED WORK
One of the earliest methods go beyond using simple metrics based on [P-norms for natural im- ages was the structural similarity index (Wang et al., 2004). Despite improvements, this method can still be considered shallow metric. Over the years multiple large databases for human evaluations of natural images were presented, for instance CSIQ (Larson & Chandler, 2010), TID2013 (Pono- marenko et al., 2015), and CID:IQ (Liu al., 2014). With this data and the discovery that CNNs can create very powerful feature extractors that are able to recognize patterns and structures. deep feature maps quickly became established as means for evaluation (Amirshahi et al., 2016; Berardino et al., 2017; Bosse et al., 2016; Kang et al., 2014; Kim & Lee, 2017). Recently, these methods were improved by predicting the distribution of human evaluations instead of directly learning distance values (Prashnani et al., 2018; Talebi & Milanfar, 2018b). Finally, Zhang et al. (2018) compared dif- ferent architecture and levels of supervision, and showed that metrics can be interpreted as a transfer learning approach, by applying linear weighting to the feature maps of any network architecture to form the image metric LPIPS vO.I. Typical use cases of these maage-based CNN metrics are computer vision tasks like detail enhancement (Talebi & Milanfar, 2018a), style transfer, and super-
2

---

Under review as a conference paper at ICLR 2020
resolution (Johnson et al., 2016). Generative adversarial networks also leverage CNN-baser losses by training a discriminator network in parallel to the generation task (Dosovitskiy & Brox, 2016).
Siamese network architectures are known to work well for variety of comparison tasks such as audio (Zhang & Duan, 2017), satellite images (He et al., 2019) or finding similar interior product designs (Bell & Bala, 2015). They were additionally demonstrated to yield robust object trackers (Bertinetto et al., 2016), algorithms for image patch matching (Hanif, 2019), and for descriptors for fluid flow synthesis (Chu & Thuerey, 2017). Inspired by these works we use similar Siamese neural network architecture for our metric learning task. In contrast to other work on self-supervised learning that utilizes spatial or temporal changes to learn meaningful representations (Agrawal al., 2015; Wang & Gupta, 2015), our method does not rely on tracked keypoints in the data.
While correlation terms have been used for learning joint representations by maximizing correlation of projected views (Chandar et al., 2016), and are popular for style transfer applications via the Gram matrix (Ruder et al., 2016), they were not used for learning distance metrics. As we demonstrate below, they can yield significant improvements in terms of the inferred distances.
Similarity metries for numerical simulations are a topic of ongoing investigation. A variety of spe- cialized metrics have been proposed to overcome the limitations of I,P-norms, such as the displace- ment and amplitude score from the area of weather forecasting (Keil & Craig. 2009), and permu- tation based metries for energy consumption forecasting (Haben et al., 2014). Turbulent flows, on the other hand, are often evaluated in terms of aggregated frequency spectra (Pitsch, 2006). Crowd- also proposed to evaluate simulation
sourced evaluations based on the human visual system animation (Um et al., 2017), and for comparing non-oscillatee non-oscillatory dis- cretization schemes (Um et al., 2019). These results indicate that visual evaluation methods in the context of field data are possible and robust, but they require extensive (and potentially expensive) user studies. Additionally, our method naturally extends to higher dimensions, while human evalu- ations inherently rely on projections with at most two spatial and one time dimension.
methods for physics-
3 CONSTRUCTING A CNN-BASED EVALUATION METRIC
In the following, we explain our considerations when employing CNNs as evaluation metrics. For comparison that corresponds to our intuitive understanding of how distances work, an underlying
metric has to obey certain criteria. More precisely, a function m I x II + [0, oc) is a metric with respect to its input space I, if satisfies the following properties Vaz, y. E
m(x y) 2 0
non-negativity symmetry
m (w.,80 y) = m m( x)
(2)
m (ax y) I m(x 2) + m(z,)) v)
triangle inequality
(3)
m( y) =0
x y
identity of indiscernibles
(4)
The properties (1) and (2) are crucial as distances should be symmetric and have a clear lower bound. Eq. (3) ensures that direct distances can not be longer than a detour. Property (4), on the other hand, is not really useful for discrete operations as approximation errors and floating point operations can easily lead to distance of zero for slightly different imputs. Hence, we focus relaxed, more meaningful definition m( æ) = 0 which leads to a so called pseudometric. It allows for a distance of zero for different inputs, but has be able to spot identical inputs.
We realize these requirements for pseudometric with an architecture that follows popular percep- tual metries such as LPIPS: The activations of CNN are compared in latent space, and accumulated with a set of weights. Afterwards, the resulting per-feature distances are aggregated to produce a final distance value. Fig. 2 gives visual overview of this process.
Base Network The sole purpose of the base network is to extract feature maps from both inputs. The Siamese architecture implies that the weights of the base network are shared for both inputs, meaning all feature maps are comparable. In addition, this ensures the identity of indiscernibles because for identical feature maps a distance of zero is guaranteed by the following operations. In the last years, variety of powerful CNN-based feature extraction architectures were proposed. We experimented with various networks, such as AlexNet (Krizhevsky et al., 2017), VGG (Simonyan & Zisserman, 2015), SqueezeNet (landola et al., 2016), and fluid flow prediction network (Thuerey
3

---

Under review as a conference paper at ICLR 2020
7 Lesemed weight per favtuura map
Inpul
Base network
Festure normolication
Elementaise Istent space diference
Channel aggr- weighted avg.
Spatial aaor everage
Layer agor: summation
output
Input 2
Base network
Feature pormalication
RGB imputs
Feature maps: sets of 3 order tensors
Difference maps:
Average maps: fensors
distances: scallars
Result: scalar
tensors
Figure 2: Overview of the proposed distance computation for a simplified base network that con- tains three layers with four feature maps each in this example. The output shape for every operation is illustrated below the transitions in orange and white; bold operations are learned by the CNN.
et al., 2018). In all cases, only the feature extracting layers are used, and the remaining layers which are responsible for the original task, e.g., classification, are discarded. Building on this previous work, consider three variants of the networks below: using the original pre-trained weights, fine- tuning them, or re-training the full networks from scratch. In contrast to typical CNN tasks where only the result of the final output layer is further processed, features across the layers of a CNN (see Fig. 2). This implies a slightly different goal: while early features should be general enough allow for extracting more complex features in deeper layers, this is not their sole purpose. Rather, features in earlier layers of the network can directly participate in the final distance calculation, and can yield important cues. As we will demonstrate below, we achieved the best performance for our data sets using custom architecture with five layers, similar to reduced AlexNet, that was trained from scratch (see App. B.
make use of the full range of extracted
Feature Map Normalization The goal of normalizing the feature maps is to transform the ex- tracted features of each layer, that typically have very different orders of magnitude, into comparable ranges. While this task could potentially be performed by the learned weights, we found the nor- malization to yield improved performance in general (see App. B.2). Zhang et al. (2018) proposed length unit normalization using division by the Euclidean norm in channel dimension, measure the angle between the latent space vectors with a cosine distance. Instead, we suggest to in- terpret all possible feature maps as a normal distribution and normalize them standard normal distribution. This is achieved via preprocessing step using the full training data set: we subtract the mean of the feature maps and then divide by their standard deviation in channel dimension for each layer. As a result, we can measure angles for the latent space vectors and compare their magnitude in the global length distribution.
only
Latent Space Differences Combining the latent spaces representations ã. ÿ that consist of all extracted features from the two inputs æ, y lies at the core of the metric computation. Here, the most obvious approach to employ an elementwise Ø; y; difference is not advisable, as this would directly violate the metric properties above. Instead, possible options to ensure non-negativity and symmetry are |/= ÿ or (æ- 2) We found that both work equally well in practice. Considering the importance of comparing the extracted features, the simple operations used for comparing the features do not seem optimal. Rather, one imagine that improvements in terms of comparing one set of feature activations could lead to overall improvements for derived metrics. Hence, we experimented with replacing these operations with a pre-trained CNN-based metric for each feature map. This creates a recursive process, and a "meta-metric" that reformulates the initial problem of the similarity between inputs terms of the similarity series of deep representations of the inputs. However, as detailed in App. B.3, we have not found this recursive approach to yield any substantial improvements. This implies that once a large enough number of expressive features is available for comparison, the in-place difference of each feature is sufficient to compare two inputs. In the following, we compute the feature difference maps (Fig. 2, in yellow) via (œ - g)
Aggregations The subsequent aggregation operations (Fig. 2, in green) are applied to the differ- ence maps to compress the contained per feature differences along the different dimensions into a single distance value. The aggregation operations only need preserve the metric properties already established via the latent space difference. To aggregate the difference maps along the channel di- mension, we found the weighted average proposed by Zhang et al. (2018) work very well. Thus,
4

---

Under review as a conference paper at ICLR 2020
one learnable weight to control the importance of feature. The weight is multiplier for the corresponding difference map before summation along the channel dimension. To preserve non- negativity and the triangle inequality, the weights are clamped to be non-negative. A negative weight would mean that a larger difference in this feature produces a smaller overall distance, which is not helpful. For spatial and layer aggregation, functions like a summation and generally interchangeable. We tested more intricate aggregation functions, such as a learned spatial average or determining layer importance weights dynamically from the inputs. When the base network is fixed and the metric only has very few trainable weights, this did improve the over- all performance. But with a fully trained base network the feature extraction seems to automatically adopt these aspects, making a more complicated aggregation unnecessary.
averaging sufficient
Additional details for the steps above are given App. A, where we also show that the proposed Siamese architecture by construction qualifies as pseudometric.
4 DATA GENERATION AND TRAINING
Similarity data sets for natural images typically rely on changing already existing images with dis- tortions, noise or other operations, and assigning ground truth distances according to the strength of the operation. Since we can control the data creation process for numerical simulations directly by altering the simulation, ities. In this way, the data contains more information about the nature of the problem, i.e., which changes of the data distribution should lead to increased distances, than by applying modifications as a post-process.
can generate large amounts of simulation data with growing dissimilar-
Data Generation Given a set of model equations, c.g. a PDE from fluid dynamics, typical so- lution methods consist of solver that, given set of boundary conditions, computes discrete approximations of the necessary differential operators. The discretized operators and the bound- ary conditions typically contain problem dependent parameters which collectively denote with Po, Pl..... ...... in the following. We only consider time dependent problems, and our solvers start with initial conditions at to compute a series of time steps L1, l,,... until target point in time (t,) is reached. At that point we obtain a reference output field on from one of the PDE variables, e.g., velocity.
Initial conditions
Finite difference solver with time discretization
Output
Increasing parameter change
P P. ... P. ...]
Og
Decreasing output similarity
(s)
noise, (8)
noise,,
[Po Pi P.+.A [.]
O
(s)
noise,_(s) (s)
[po P ... A. ..]]
0,
noise,..
noise,,
noise,
Figure 3: General data generation method from a PDE solver for a time dependent problem. With increasing changes of the initial conditions for a parameter pe in A; increments, the outputs decrease in similarity. Controlled Gaussian noise is injected in a simulation field of the solver. The difficulty of the learning task can be controlled by scaling A, as well the noise strength
For data set generation, we now incrementally change single parameter p; in steps A,, 2 We consider a series obtained this way to be increasingly different from Op. To create natural variations of the resulting data distri- butions, we add Gaussian noise fields with zero mean and adjustable variance to an appropriate simulation field such as a velocity. This noise allows to generate large number of varied data samples for a single simulation parameter p;. In addition, it is similar in nature to numerical errors introduced by discretization schemes. Thus, these perturbations enlarge the space covered by the
A....., n A; to create a series of outputs o2, On-
5

---

Under review as a conference paper at ICLR 2020
training data, and we found that training networks with suitable noise levels improves robustness, as we will demonstrate below. The process for data generation is summarized in Fig. 3.
As PDEs can model extremely complex and chaotic behaviour, there is no guarantee that the outputs always get increasingly dissimilar with the increasing parameter change. This behaviour is what makes the task of similarity assessment so challenging. Even if the solutions are essentially chaotic, their behaviour is not arbitrary but rather governed by the rules of the underlying PDE. For our data set, we choose range of representative PDEs: We include pure Advection-Diffusii model (AD) and Burger's 's equation (BE) which introduces a viscosity term. Furthermore, we use the full Navier- Stokes equations (NSE) which introduce a conservation of mass constraint. When combined with a deterministic solver and a suitable parameter step size, all these PDEs exhibit chaotic behaviour at small scales, so that medium and large scale characteristies of the solutions shift smoothly with increasing changes of the parameters pi- The noise n amplifies the chaotic behaviour to larger scales to create an environment with controlled amount of perturbations. This lets the network learn about the nature of the chaotic behaviour of PDEs, without overwhelming it with data where patterns are not observable anymore. The latter can easily happen when A or n grow too large and produce essentially random outputs. Instead, in terms of a shallow metric. We choose the smallest A and n such that the ordering of several random output samples with respect to their 2,2 difference drops below correlation value of 0.8.
specifically target solutions which are difficult to evaluate
Training For training, the scalar 2D fields from the simulations were individually normalized to the [0, 255] range. To produce additional variation, the data is augmented with random color maps, flips, rotations, and a random crop obtain an input size of 224 224. Afterwards, each input is normalized to a standard normal distribution by subtracting the mean and dividing by the standard deviation (pre-computed from all available training data). Unless noted otherwise, networks were trained for 40 epochs with an Adam optimizer using a learning rate of 10-5 that was reduced to 5 10 after 15 epochs. To evaluate the networks, only bilinear resizing and the normalization step is applied to validation or test inputs.
5 CORRELATION LOSS FUNCTION
The central goal of our networks to identify relative differences of input pairs produced via nu- merical simulations. Thus, instead of employing loss that forces the network to only infer given labels or distance values, we train our networks to infer the ordering of a given sequence of varying inputs .I...., On. We propose to use the Pearson correlation coefficient (see Pearson, 1920) which yields a value in [-1, 1] that measures the linear relationship between two distributions. A value of 1 implies that a linear equation describes their relationship perfectly. We compute this coefficient for a full series of outputs, such that the network can learn to extract features that arrange this data series in the correct ordering.
We train our networks with minibatches consisting of n outputs, and provide a linearly increasing distance distribution [0, 1J" representing the parameter change. For a distance prediction d [0, ) of our network for one minibatch, we compute the loss with
(c-c). (d
M(c, d) A1 - d)² + 21 || Ccc-cl2 d 2
(5)
Here, the mean of a minibatch distance vector is denoted by ē or d respectively. The first part of the loss is a regular MSE term, which minimizes the difference between predicted and actual distances. The second part is the Pearson correlation coefficient, which is inverted such that the optimization results in maximization of the correlation. As this formulation depends on the batch size, the terms in Eq. (5) can be scaled to adjust their relative influence with a and X2 For our training, we have ensured a constant batch size due to the fixed number of 10 variations for each reference simulation comprising one batch. If the batch size should be allowed to vary, the influence of both terms needs to be adjusted accordingly. We found that scaling both terms to similar order of magnitude worked best in our experiments.
In Fig. 4 we investigate how the proposed loss function compares to other commonly used loss for- mulations for our full network and pre-trained network similar to Zhang et al. (2018). In addition to our full loss function, we consider a loss function that replaces the Pearson correlation with a simpler cross-correlation (c-d)/ / [[]cc] 2 [|d]|)) We also include networks trained with only the MSE only

---

Under review as a conference paper at ICLR 2020
the correlation terms for each of the two vari- ants. As shown in Fig. 4, a simple MSE loss yields low accuracy of less than 0.6. Us- ing any correlation based loss function for the AlexNefgozen metric improves the results, but there is no major difference due to the limited number of only 1152 trainable weights. For LNSM, the proposed combination of MSE loss with the Pearson correlation performs signifi- cantly better than using cross-correlation or variant without the MSE loss. Interestingly, combining cross correlation with MSE yields worse results than cross correlation alone. This happens because the cross correlation not only affects the ordering but also impacts the abso- lute distance values. In combination with MSE, this can lead to influences that cancel each other. For our loss, the Pearson correlation only handles the ordering while MSE deals with the absolute distances.
Correlation on all test data
Proposed
MSE cross cor.
Pearson cor.
Cross cor.
LNSM (ours)
MSE
AlexNetrozen
0.55
0.60
0.65
0.70
Figure 4: Performance on our test data for the proposed approach (LNSM) and smaller model (AlexNet(jzze) using different loss functions.
6 RESULTS
Data Sets Using the data generation approach described in Section 4, created four training (Smo, Lig, Adv and Bur) and two test data sets (LiqN and AdvD) with ten parameter steps for each reference simulation. Based on two 2D NSE solvers, the smoke and liquid simulation training sets (Smo and Liq) add noise to the velocity field and feature varied initial conditions such as fluid position or obstacle properties, in addition to variations of buoyancy and gravity forces. The two other training sets (Adv and Bur) are based on ID solvers for AD and BE, concatenated over time to form a 2D result. In both cases, noise was injected into the velocity field, and the varied parameters are changes the field initialization and forcing functions.
For the test data set, we substantially change the data distribution by injecting noise into the den- sity instead of the velocity field for AD simulations to obtain the AdvD data set, and by including background noise for the velocity field of a liquid simulation (LiqN).
Figure 5: Samples from data sets. For each subset the reference is on the left, and three varia- tions in equal parameter steps follow. From left to right and top to bottom: Smo (density, velocity, and pressure), Adv (density), Lig (flags, velocity, and levelset), Bur (velocity), Li (velocity), AdvD (density), Sha and Vid.
In addition, we employed three more test sets (Sha, Vi d, and TID) created without PDE models to explore the generalization for data far from our training data setup. We include shape data set that features multiple randomized moving rigid shapes, video data set consisting of frames from random video footage, and the perceptual image data set TID2013 from Ponomarenko et al. compaoaaeee structure, combined into one data
(2015). All test sets, excluding TID due set (A11) as well. Simulation examples for each data set are shown in Fig. and generation details with additional samples can be found in App. D.
Performance Evaluation To evaluate the performance a metric on a data set, we first com- pute the distances from each reference simulation to all corresponding variations. Then, the pre- dicted and the ground truth distance distributions over all samples are combined and compared using Spearman's rank correlation coefficient (see Spearman, 1904). Like the Pearson correlation it is

---

Under review as a conference paper at ICLR 2020
value in [-1.1] compare distributions, but it measures the correlation between ranking variables, ie. monotonic relationships.
The top part of Tab. shows the performance of the shallow metries L and SSIM, as well as the LPIPS metric (Zhang et al., 2018) for all our data sets. The results clearly show that shallow metrics not suitable to compare the samples in data set, and only achieve good correlation values on the TID2013 data set which contains a large number of pixel-based image variations without contextual structures. The perceptual LPIPS metric performs better in general and outperforms our method on the image data sets Vid and TID. This is not surprising, as LPIPS is specifically trained for such images. For the simulation data sets LiqN and Sha, however, it performs significantly worse than for the image content. The last row of Tab. I shows the results of our LNSM network, with a very good performance across all data sets. Note that even though was not trained with any natural images it still performs well for the image test data sets.
Table Performance comparison of existing metries (top block), experimental designs (middle block), and variants of the proposed method (bottom block) on validation and test data sets measured in terms of Spearman's 's rank correlation coefficient. Bold values show the best performing metric for each data set and bold+italic values are within 0.01 error margin of the best performing. On the right a visualization of the combined test data results is shown for selected models.
Validation data sets
Test data sets
Shallow
Metric
Image-based
Smo
Lig
Adv
Bur
TID
Li Li
Advi
Sha
Vid
A11
Experimental
1.2
0.67
0.80
0.72
0.59
0.83
0.72
0.58
0.50
0.77
0.56
Proposed
SSIM
0.67
0.75
0.75
0.68
0,80
0.25
0.70
0.35
0.70
0.48
0.7
LPIPS vO.1.
0.72
0.75
0.75
0.72
0.80
0.63
0.60
0.81
0.82
0.66
Conclation (Al)
AlexNet,
0.64
0.75
0.67
0.64
0.84
0.64
0.67
0.61
0.78
0.62
AlexNelfrccm
0.67
0.70
0.68
0.70
0.79
0.40
0.64
0,84
0.81
0.62
Optical flow
0.62
0.57
0.36
0.37
0.55
0.49
0.28
0.61
0.75
0.48
0.5
Non-Siamese
0.72
0.82
0,76
0.69
0.29
0.73
0.62
0.60
0,72
0.63
Skipfeom scranch
0.65
0.84
0.74
0.67
0.80
0.79
0.59
0.83
0.79
0.70
LNSM
0.64
0.83
0.74
0.60
0,80
0.81
0.58
0.84
0.76
0.69
SSIM
LPIPS
Optiion
LNSM strongg aurine
0.63
0.82
0.71
0.61
0.80
0.78
0.50
0,80
0.77
0.65
LNSM (ours) 0.68
0.82
0.76
0.70
0.78
0.80
0.61
0.85
0.76
0.71
The middle block of Tab, contains several interesting variants: AlexNetrandom and AlexNet are small models, similar to Zhang et al. (2018) where the base network is the original AlexNet with pre-trained weights. AlexNetranion contains purely random aggregation weights without training, while AlexNelfrzen only has trained weights are for the channel aggregation. The random model per- forms surprisingly well, pointing to powers of the underlying CNN architecture, while AlexNetfivzen lacks enough trainable weights to fully adjust to the data distribution of the numerical simulations.
Recognizing that many PDEs include transport phenomena, we investigated optical flow (Horn & Schunck, 1981) as means to compute motion from field data. For the Optical flow metric used FlowNet2 (Ilg et al., 2016) to bidirectionally compute the optical flow field between two inputs. The flow fields are aggregate to a single distance value by adding the magnitude of all flow vectors. On the data sets Sha and Vid d that are similar to the training data of FlowNet2 it performs relatively well, but on most other data performs poorly. This shows that computing a simple warping from one input to the other is not enough for a stable metric, although it seems like an intuitive solution. A more robust metric needs the knowledge of the underlying features and their changes to generalize better to new data.
For the Non- Sia-eee metric, we used non- -on-nee architecture that directly predicts the distance from both inputs, to evaluate whether a Siamese architecture is really beneficial. For this purpose, we employed a modified version of AlexNet that reduces the weights of the feature extractor by 50% and of the remaining layers by 90%. As expected, this metric works great on the validation data, but has huge problems with generalization. In addition, even simple metric properties like symmetry are no longer guaranteed because this architecture does not have the inherent constraints of the Siamese setup. Finally, we experimented with multiple fully trained base networks. As re-training existing feature extractors only provided small improvements, connections for the Skip scratch metric. Its results already come close to the proposed approach on most data sets. App. B contains additional details for the experimental models above.
used custom base network with skip
8

---

Under review as a conference paper at ICLR 2020
The last block in Tab. shows variants of the proposed approach, trained with varied noise levels. This inherently changes the difficulty of the data sets. Hence INSMeoiseees was trained with relatively simple data without perturbations, while LNSM arooy noise Both cases decrease the generalizing capabilities of the trained model, resulting in a deteriorated performance for the test data. This indicates that the network needs see a certain amount of variation at training time in order to become robust, but overly large changes hinder the learning of useful features. We provide a more detailed analysis of varying noise levels in App. and additional evaluations in App. E.
trained with strongly varying data.
7
CONCLUSION
We have presented the LNSM metric to reliably and robustly compare outputs from numerical sim- ulation methods. Our method significantly outperforms existing shallow metric functions and pro- vides better results than other learned metrics on our test data. We demonstrated the usefulness of the correlation loss and the robustness to natural data errors provided via controlled data generation.
Our trained LNSM network has the potential to impact wide range of fields; from the fast and robust aceuracy assessment of new simulation methods, over robust optimizations of PDE parameters for real-world data sets, to guiding generative models of physical systems. Furthermore, it will be highly interesting to evaluate other loss functions for learning specialized metrics, e.g. mutual information (Bachman et al., 2019) contrastive predictive coding (Hénaff et al., 2019). We also plan to evaluate our approach for an even larger collection of PDEs, and to train LNSM models for 3D and 4D data sets. Especially unsteady, turbulent flow fields are highly relevant and interesting area for future work on learned evaluation metrics.
REFERENCES
Pulkit Agrawal, João Carreira, and Jitendra Malik. Learning to see by moving. In 2015 IEEE Inter- national Conference Computer Vision (ICCV), pp. 37-45, 2015, 1tt:00.119//CCC2201..10
Seyed Ali Amirshahi, Marius Pedersen, and Stella X. Yu. Image Quality Assessment by Comparing CNN Features between Images. Journal of Imaging Sience and Technology, 60(6), 2016. ISSN 1062-3701. doi: 10.2352/ Imagin/...
ing mutual information across views. CoRR, www.gingecc...ccco.220.6..6666
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- 2019. URL http: arxiv.
org/ /abs//////// / 19066.4 00910.
Sean Bell and Kavita Bala. Learning visual similarity for product design with convolutional neural networks. ACM Transactions Graphics, 34(4):98:1-98:10 2015. ISSN 0730-0301.
doi:10.1145/276699
Yassine Benajiba, Jin Sun, Yong Zhang, Longquan Jiang, Zhiliang Weng, and Or Biran. Siamese networks for semantic pattern similarity. CoRR, abs/1812. 2018. URL WtRLhttp:///axt org/abs/ 181211111111 06604.
Alexander Berardino, Johannes Balle, Valero Laparra, and Eero Simoncelli. Figen-Distortions of Hierarchical Representations. In Advances in Neural Information Processing Systems 30 (NIPS 2017), volume 30, 2017. URL http: /tap:://aaxiiii org/abs/////oooooooooo...ooo.o
02266.
Luca Bertinetto, Jack Valmadre, Joao F. Henriques, Andrea Vedaldi. and Philip H. S. Torr. Fully- Convolutional Siamese Networks for Object Tracking. In Computer Vision ECCV 2016 Work- shops, PT II, volume 9914, pp. 850-865, 2016. ISBN 978-3-319-48880-6 1o1:10010077978-3
119-48881-3_56
Sebastian Bosse, Dominique Maniry, Klaus-Robert Mueller, Thomas Wiegand, and Wojciech Samek. Neural Network-Based Full-Referrenee Image Quality Assessment. In 2016 Picture Cod- ing Symposium (PCS), 2016. ISBN 978- -0000-99669
Sarath Chandar, Mitesh M. Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlational neural networks. Neural Computation, 28(2):257-285, 2016. doi: 1111/.111///COO 00801.

---

Under review as a conference paper at ICLR 2020
Mengyu Chu and Nils Thuerey. Data-Driven Synthesis of Smoke Flows with CNN-based Fea- ture Descriptors. ACM Transactions on Graphics, 36(4):69:1-69) 69:14, 2017. ISSN 0730-0301. dot:100.145/302295930776666
Alexey Dosovitskiy and Thomas Brox. Generating Images with Perceptual Similarity Metrics based on Deep Networks. In Advances in Neural Information Processing Systems 29 (NIPS 2016), volume 29, 2016. URL http: a arxiv wwwg/ abs / 602. 02644.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. URL http: http:///wwwwdeeleeaeeeiieee
org.
Stephen Haben, Jonathan Ward, Danica Vukadinovic Greetham, Colin Singleton, and Peter Grindrod. new error measure for forecasts of houschold-level, high resolution electrical en- ergy consumption. International Journal of Forecasting, 30(2):246-256, 2014. ISSN 0169-2070. doi:10. wff.oorraatt..13....... 100
Muhammad Shehzad Hanif. Patch match networks: Improved two-channel and Siamese networks for image patch matching. Pattern Recognition Letters, 120:54-61, 2019. ISSN 0167-8655.
doi: 10::10.11 wttp::/.paarre...1.1....
Haiqing He, Min Chen, Ting Chen, Dajun Li, and Penggen Cheng. Learning to match multitemporal optical satellite images using muulti-support-pathhie Siamese networks. Remote Sensing Letters, 10(6):516-525 2019. ISSN 2150-704X. d0.:10. 00000000000025557000022001115555
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. 101110.109//CPP220166..9
Olivier Hénaif, Ali Razavi, Carl Doersch, M. Ali Eslami, and Aäron van den Oord. Data- efficient image recognition with contrastive predictive coding. CoRR, 0bss/1905099250 2019. URL http:h org/ abs 09272.
Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3): 185-203, 1981. ISSN 0004-3702. doi: 100.10016000043702666 199024444.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2261-2269, 2017. doi:10. 109.1199...P...........
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <Imb model size. CoRR, abs/1602.07360, 2016. URL http:////// /wtp/v/rivvv////a///////a////iia org/
07360.
Eddy lgg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. CoRR, abs/1612.01925,
/aapaa////////aao 2016. URL http:///// /awxiv.vvvv. org/ abs / 16612..0 C1925.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual Losses for Real-Time Style Transfer and Super-Resollutinn In Computer Vision - ECCV 2016, PT II, volume 9906, pp. 694-711, 2016. ISBN 978-3-319-46474-- 000110..0079978-3339946667---
Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional Neural Networks for No-Reference Image Quality Assessment. In 2014 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 1733-1740, 2014, ISBN 978-1-4499-5517--8 doi:
Christian Keil and George C Craig. A displacement and amplitude score employing an optical flow technique. Weather and Forecasting, 1297-1388 2009. 10 11..111177/00099/2222244
Jongyoo Kim and Sanghoon Lee. Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework. In 30TH IEÉE Conference on Computer Vision and Pattern Recognition (CVPR 2017), pp. 1969-1977, 2017. ISBN 978-1-5386-0457-11 doi: 10.1 1099CV
10

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep volutional neural networks. Communications of the ACM, 60(6):84 2017. ISSN 0001-0782. doi:10.1 11..1.1133/6668666
Eric C. Larson and Damon M. Chandler. Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 19(1), 2010. ISSN 1017- 9909. doi: 0.1117/1.3267005
Zhihong Lin, Taik Soo Hahm, W.W.Lee, William M. Tang, and Roscoe B White. Turbulent transport reduction by zonal flows: Massively parallel simulations. Science, (88584)::8335-18777 1998. ISSN 1095-9203. doi:10.1 1tt.111///ccccece222 5384.1833
Xinwei Liu, Marius Pedersen, and Jon Yngve Hardeberg. CID:IQ A New Image Quality Database. In Image and Signal Processigg ICISP 2014, volume 8509, pp. 193-202, 2014. ISBN 978-3-319- 07997-4. 40011011077977-4339999999---21
Parviz Moin and Krishnan Mahesh. Direct numerical simulation: tool in turbulence research. Annual review of fluid mechanics, 30(1):539-578, 1998. doi: 14t////rrre..tu......1....
William L Oberkampf, Timothy G Trucano, and Charles Hirsch. Verification, validation, and pre- dictive capability in computational engineering and physics. Applied Mechanics Reviews, 57: 345-384, 2004. ISSN 0003-6900, doi: 10. 10.1111//1..767477.
Karl Pearson. Notes on the History of Correlation. Biometrika, 13(1):25-45, 1920. ISSN 0006- 3444. 200:10..003//imme/13......
Heinz Pitsch. Large-eddy simulation of turbulent combustion. Annu. Rev. Fluid Mech., 38:453-482,
2006. doi: 1466/nnnree....mmm
Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C. Jay Kuo. Image database TID2013: Peculiarities, results and perspectives. Signal srrressinn--mmaaee Communication. 30: 57-77, 2015. ISSN 0923-5965. doi: 101.1..0161..mmaee2011111
Ekta Prashnani. Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. CoRR, 8060000666 2018. URL httt:://aaxiii org/abs 1806. 02067.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. CoRR, abs/1505.04597, 2015, URL http::/aaxx...ooo/////ooo/ 15C5.04597
Manuel Ruder, Alexey Dosovitskiy. and Thomas Brox. Artistic style transfer for videos. In Pattern Recognition, pp. 26-36, 2016. ISBN 978-3-319-45886-11 401:10..107777888-3994488--
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015. URL http:
1.56.
Charles Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. ISSN 0002-9556. 4oi:10..300714111559
Hossein Talebi and Peyman Milanfar. Leamndd Perceptual Image Enhancement. In 2018 IEEE International Conference on Computational Photography (ICCP), 2018a. ISBN 978-1-5386 2526-2. 10i:10.1109/1CC 100.0000000// 88366474.
Hossein Talebi and Peyman Milanfar. NIMA: Neural Image Assessment. IEEE Transactions on Image Processing, 27(8):3998 4011, 2018b. ISSN (057-7149,
Nils Thuerey, Konstantin Weissenow, Harshit Mehrotra, Nischal Mainali, Lukas Prantl, and Xiangyu Hu. Well, how accurate is it? A study of deep learning methods for reynolds-averaged navier- stokes simulations. CoRR, abs/1810.082/11 2018. URL http: / axxiv. abs//bbs//11 08217.
11

---

Under review as a conference paper at ICLR 2020
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Perceptual Evaluation of Liquid Simulation Methods.
ACM Transactions on Graphics, 36(4), 2017. ISSN 0730-0301. doi:10. 145/3072959933733333
Kiwon Um, Xiangyu Hu, Bing Wang, and Nils Thuerey. Spot the Difference: Accuracy of
Numerical Simulations via the Human Visual System. CoRR, 1os//990.00479, 2019. URL http:/ /ttp///aavvv.v/aaa//aa/eaaaaiwia org/ abs /oss110 .4179.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2794 2802, 2015. doi: 10. 00.11099/CC.2201.3399.0
Zhenyu Wang, Jinsong Zhang, and Yanlu Xie. 1.2 Mispronunciation Verification Based Acous- tic Phone Embedding and Siamese Networks. 2018 1ITH International Symposium on Chi- Spoken Language Processing (ISCSLP), pp. 444 448, 2018. ISBN 978-1-5386-562733.
doi: 10i:10.11000 10110///SSSL...1888866666
Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero Simoncelli. Image quality assess- ment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13 (4):600-612, 2004. ISSN 1057-7149. doi:10. 1199T1122003.819860
B. P. Welford. Note method for calculating corrected sums of squares and products. Techno- metrics, 4(3))419-427 420, 1962. doi: 10. 10000000000//000...6... 199.1660992...49002..
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreason- able Effectiveness of Deep Features Perceptual Metrie. In 2018 IEFE Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 586-595, 2018. ISBN 978-1-5386-6420-9 101:10..109/CPPR.2220 800066.
Yichi Zhang and Zhiyao Duan. IMINET: Convolutional Semi-Siamese Networks for Sound Search by Vocal Imitation. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 304-308, 2017. ISBN 978- 1-538--166666 1632-1. 2000:0011007AASLL222888888
Yongning Zhu and Robert Bridson. Animating sand as a fluid. In ACM SIGGRAPH 2005 Papers, pp. 965-972, New York, NY, USA, 2005. do 145/1 86822, 1073298.
12

---

Under review as a conference paper at ICLR 2020
Appendix: Learning Similarity Metrics for Numerical Simulations
This supplemental document contains an analysis of the proposed metric design with respect to properties of metrics in general (App. A), and details to the used network architectures (App. B). Afterwards, material that deals with the data sets is provided. It contains examples and failure cases for each of the data domains and analyzes the impact of the data difficulty (App. C and D). Finally, we explore additional evaluations (App. E) to strengthen the results from Tab. 1, give an overview on the used notation (App. F), and provide download link for further material (App. G).
A DISCUSSION OF METRIC PROPERTIES
To analyze if the proposed method qualifies as a metric, it is split in two functions my : L. and m2 : L x L + [0, xc) which operate on the input space Il and the latent space L. Through flattening elements from the input or latent space into vectors, I = Ra and IL = R where and b are the dimensions of the input data or all feature maps respectively, and both values have a similar order of magnitude. my describes the non-linear function computed by the base network combined with the following normalization and returns a point in the latent space. My uses two points in the latent space to compute a final distance value, so it includes the latent space difference and the aggregation in the spatial, layer, and channel dimensions. With the Siamese network architecture the resulting function for the entire approach is
m y) m2(m1(x)), mi (y)).
Property (4) mainly depends my because even if my itself guarantees this property, m. could still be non-injective, which means it can map different inputs to the same point in latent space ă = ý for x * y. Due the complicated nature of mi it is difficult make accurate predictions about the injectivity of my. Each base network layer of my recursively processes the result of the preceding layer with various feature extracting operations so intuitively, significant changes in the input should produce different feature map results in some layer. But very small changes in the input do lead to zero valued distances predicted by the CNN (i.e. an identical latent space for different inputs), meaning m, is in practice not injective. In an additional experiment, the proposed architecture was evaluated on about 3500 random inputs from all our data sets, where the CNN received one unchanged and one slightly modified input. The modification consisted of multiple pixel adjustments by one bit (on 8-bit color images) in random positions and channels. When adjusting only a single pixel in the 224 x 224 input, the CNN predicts zero valued distance about 23% of the inputs, but we never observed an input where seven or more changed pixels resulted in a distance zero in all experiments.
In this context, the problem of numerical errors is important, because even two slightly different latent space representations could lead to result that seems be zero if the difference vanishes in the aggregation operations or is smaller than the floating point precision. On the other hand, an automated analysis to find points that have different input but identical latent space image is challenging problem and left as future work.
The evaluation of the base network and the normalization is deterministic, and hence Vo : mz(x) = m11 (x) holds. Further, we know that m(æ, Φ) = 0 if my guarantees that Vm (x) m²(m; (z), m2 m1())
O. Thus, the remaining properties (1), (2), and (3) only depend on m2, since for them the original inputs are not relevant, only their respective images in the latent space. The resulting structure with a relaxed identity of indiscernibles is called a pseudometric, where Va, ú, غ E I.:
m2 ( 9)
m2 x) m2 (ē, 9) < m2( 2) + m2
m2
[ 7)
(7)
(8)
my( ã. σ) 0
(9)
Notice, that m2 has to fulfill these properties with respect to the latent space and not the input space. If m2 is carefully constructed the metric properties still apply, independently of the actual design of the base network the feature map normalization.
13

---

Under review as a conference paper at ICLR 2020
A first observation concerning my is that if all aggregations were operations and the element- wise latent space difference was the absolute value of difference operation, m2 would be equivalent to computing the L' -norm of the difference vector in latent space.
m2
(i) y)
Similarly, adding square operation to the elementwise distance in the latent space and computing the square root at the very end leads to the 1--norr of the latent space difference vector. In the same way, it is possible to use any [P-morm with the corresponding operations.
Fum (z. m2
" ÿ)
Vr
In both cases, this forms the metric induced by the corresponding norm which by definition has all desired properties (6), (7), (8), and (9). we change all aggregation methods weighted average operation, each summand is multiplied by weight w/. This is even possible with learned weights, as they are constant at evaluation time, they are clamped to be positive as described above. Now, wi can be attributed to both inputs by distributivity, meaning each input is elementwise multiplied with a constant vector before applying the metric, which leaves the metric properties untouched. The reason is that it is possible to define new vectors in the same space, equal to the scaled inputs. This renaming trivially provides the correct properties.
b Σ
weighted m2
(.
wi i
m wiYi
Accordingly, doing the same with the L"-norm idea possible, each wa just needs suitable adjust- ment before distributivity can be applied, keeping the metric properties once again.
m12
weighted
(Ø.
i
"ill"
Wil 33
Del [ ūrl ... |ac 97al
Wis a Dil wo lac, W7s
w/ [
my00
Σ
w क. wi ÿ
With these weighted terms for m2, it is possible to describe all used aggregations and latent space difference methods. The proposed method deals with multiple higher order tensors instead of a sin- gle vector, so the weights w; additionally depend on constants like the direction of the aggregations and their position in the latent space tensors. But it easy to see that mapping higher order tensor to vector and keeping track of additional constants still retains all properties in the same way. As a result, the described architecture by design yields pseudometric that is suitable for comparing simulation data in a way that corresponds to intuitive understanding of distances.
B ARCHITECTURES
The following sections provide details regarding the architecture of the base network and some experimental designs mentioned in Tab.
14

---

Under review as a conference paper at ICLR 2020
B. BASE NETWORK DESIGN
Fig. shows the architecture of the base network for the LNSM metric. Its purpose is extracting features from both inputs of the Siamese architecture that useful for the further processing steps. To maximise the usefulness and avoid feature maps that show too similar features, the chosen kernel size and stride of the convolutions important. Starting with very larger kernels and strides means the network has a big receptive field and can extract large scale features. For the two following layers, the large strides are replaced by additional MaxPool operations that serve a similar purpose and reduce the spatial size of the feature maps.
11x11 Convolution stride 4 Rel.U
224
27
MaxPool with sirde
224
55
27
13
13
13
Bx5 Convelution with strida Ret.U
Convelution stiile
Layer f
Layer
Layer 3
Layer
Layer 5
Figure 6: Proposed base network architecture consisting of five layers with up to 192 feature maps that decreasing in spatial size. It is similar the feature extractor from AlexNet identical spatial dimensions for the feature maps are used, but reduces the number of feature maps for each layer by 50% have fewer weights.
For the three final layers only small convolution kernels and strides are used, but the number of channels is significantly larger than before. The reason is that the very deep features, that typically contain only small scale structures, are most important to distinguish small changes the inputs. Keeping the number of trainable weights low possible design, to prevent overfitting to certain simulations types and increase generality. We explored a weight range by using the same architecture and only scaling the number of feature maps in each layer. The final design shown in Fig. 6 with about 0.62 million weights worked best.
important consideration for this
INSM(ours)
AlexNetrrozen
0.25
25 8
0.25
25
of feature map weights
of map weights
Mean and std. dev.
0.20
Unused fealurn mapsill
Mean and std. dev.
Unused feature maps
0.15
15
0.15
15
0.10
10
0.10
10
0.05
0.00
0.00
Layer
Layer
Figure 7: Analysis of the distributions of learned feature map aggregation weights across the base network layers. Displayed is method for fully training the base network (left) in comparison to using pre-trained weights (right).
In the following, we analyze the contributions of the per-layer features of two different metric net- works highlight differences in terms how the features are utilized for the distance estimation task. In Fig. 7 the learned feature map aggregation weights of our LNSM network show very sim- ilar mean and a small standard deviation throughout the five layers. This means, all feature maps similarly contribute to establishing the distances, and the aggregation just fine-tunes the relative im- portance of each feature. In addition, all features receive a weight greater than zero, and as a result no feature is excluded from contributing to the final distance value.
Employing a fixed pre- praitre feature extractor on the other hand shows a very different picture: Al- though the mean across the different network layers is similar, the contributions of different features vary strongly, which is visible the standard deviation being significantly larger. Furthermore, 2- 10% of the feature maps in each layer receive weight of and hence deemed not useful
15

---

Under review as a conference paper at ICLR 2020
at all for establishing the distances. This illustrates the usefulness of targeted network in which all features contribute to the distance inference.
B.2 FEATURE MAP NORMALIZATION
Here analyze how feature maps best normalized for the "normalization" step of algo- rithm, i.e., the red ellipses in Fig. Assume we have 3 order feature tensor G with dimensions (ger 92 9x: from one layer of the base network (see Fig. 2). We can form series Go GI.... for all possible forms of this tensor in our training samples (computed as a preprocessing step). Below, evaluate three different normaliza- tion methods and also consider not normalizing all, denoted by
Feature Map Normalization
Standard normal distr.
norm
G G.
Global length
The normalization only happens in the channel dimension, so following operations accumu- late values along (=. Ias 9y) while keeping Oz and gy constant, ie., are applied independently of the spatial dimensions. The unit length nor- malization proposed by Zhang et al. (2018)
Unit length
LNSM (ours)
None
Alex/Vetrozen
0.4
0.5
0.6
0.7
G |G|||2
Correlation on all test data
norm G))
Figure 8: Performance on our test data for proposed approach (LNSM) and a smaller model (AlexNetnozen) using different normalizations.
only considers the current sample. In this case ||G||2 is 2nd order tensor with the Euclidean norms G along the channel dimension. Com- bined with summation the aggregation operation in channel direction, this results in a cosine dis- tance which only measures angles of the latent space vectors. Using a learned average instead means the angles are longer uniform, but warped according to the importance of each feature (i.e. the re- sulting angle changes differently for the same amount of change in two separate features). Extending this idea consider other training samples as well, leads global unit length normalization
norm
(G)
max Go l| |
where the maximum Euclidean norm of all available samples is employed. As result, not only the angle of the latent space vectors, but also their magnitude compared to the largest feature vector is available in the aggregation. This formulation is not really robust yet, because the largest feature vector could be an outlier w.r.t. the typical content. Instead, we can consider the full feature vector standard normal distribution with the proposed
normal distribution and transform
mean (| Gollo || G1ll2....
nor
(G)
std ( | G.ll......)
In addition to the angle, this formulation allows for robust comparison of the magnitude of each feature vector in the global magnitude distribution. Fig. 8 shows a comparison of these normal- ization methods the combined test data. Using no normalization is detrimental in both cases as succeeding operations can not reliably compare the features. Interestingly. the unit length normal- ization works best for the AlexNetnozen metric (similar to LPIPS from Zhang al. (2018)) that only uses learned aggregation weights with fixed AlexNet feature extractor. This observation allows for a conclusion about the features extracted by AlexNet. For the original task of image elassification, the magnitude of feature vector does not seem to carry information about the feature. Interpret- ing the length as part the feature for our task in normgiobel and normasst.. obviously harms the performance of the metric. Therefore, training the feature extractor such that the magnitude of the feature vectors bears some meaning should improve the results for the complex normalizations. The performance of our approach with fully trained feature extractor in Fig. 8 shows exactly this behaviour: A more complex normalization directly yields better results since the features can be adapted utilize it.
16

---

Under review as a conference paper at ICLR 2020
B.3 RECURSIVE
Since comparing the feature maps, i.e., the yellow ellipse in Fig. 2, is a central operation of the pro- posed metric calculations, experimented with replacing it with existing CNN-based metric. In theory, this would allow for a recursive, arbitrarily deep network that repeatedly invokes itself::
I Learned per map
Input
Base network
Feelure map normalizaiio
Spatial extensiio
Distance
LPIPS latent space difference
Aggregation: weighted avg.
output
Input
Base network
Feelure map nomaaizztion
Spatial extension
d
inputs
Feature maps:
Extended maps: sets of 3" order tensors
differences: sets of scalars
Result: scalar
sets of 3 order tensors
Figure 9: Adjusted distance computation for LPPPS-basee latent space difference. To provide sufficiently large inputs for LPIPS, small feature maps are spatially enlarged with nearest neighbor interpolation. In addition, LPIPS creates scalar instead of spatial differences leading to simplified aggregation.
first, the deep representations of inputs are used, then the deep representations of the deep represen- tations, etc. In practice, however, using more than one recursion step is currently not feasible due to increasing computational requirements in addition to vanishing gradients.
Fig. 9 shows how our computation method can be modified for CNN-based latent space difference, instead of an elementwise operation. Here we employ LPIPS (Zhang al., 2018). There two main differences compared to Fig. 2: First, the LPIPS latent space difference creates single distance values for a pair of feature maps instead of spatial feature difference. As result, the following aggregation is a single learned average operation and spatial or layer aggregations are no longer necessary. We also performed experiments with spatial LPIPS version here, but due memory limitations these were not successful, Second, the convolution operations in LPIPS have lower limit for spatial resolution, and some feature maps of our base network are quite small (see Fig. 6). Hence, we up-scale the feature maps below the required spatial size of 32 32 using nearest neighbor interpolation.
On our combined test data, such metric with fully trained base network results in a correlation value of 0.62 (compare to Tab. 1). Overall, achieves a performance comparable to AlexNet nandon or AlexNel
B.4 OPTICAL FLOW METRIC
In the following, we describe our approach to compute a metric via optical flow (OF). For an efficient
OF evaluation we employed pre-trained network (Ilg al., 2016). From an OF network f IxI Riman x.maz X2 with two inputs data fields x. E I, we get the flow vector field (i.j). = (f (i,j), f2y (i.j))T where and denote the location and fi and fz denote the components of the flow vectors. In addition, we have a second flow field (1, computed from the reversed II o0):
input ordering. We can define function m I x
ámaz Jman
m(x y)
ΣΣ (O'"( (i,)) + (UT (1.5))² (f2 (i. j=0
Σ
Intuitively, this function computes the sum over the magnitudes of flow vectors in both vector fields. With this definition, it is obvious that m( y) fulfills the metric properties of non-negatyvy and symmetry (see Eq. (1) and (2)). Under the assumption that identical inputs create a zero flow field, relaxed identity of indiscernibles holds well (see Eq. (9)). Compared to the proposed approach there is no guarantee for the triangle inequality though, so m(x, y) only qualifies as a pseudo-semimetric.
Fig. 10 shows flow visualizations on data examples produced by FlowNet2. The metric works relatively well for inputs that are similar the training data from FlowNet2, like the shape data
17

---

Under review as a conference paper at ICLR 2020
Reference
Varialion
Flow streamlines
Flow direction
Flow magritude
Figure 10: Outputs from FlowNet2 data examples. The flow streamlines sparse visualization of the resulting flow field and indicate the direction the flow by their orientation and its magnitude by their color (darker being larger). The two visualizations the right show the dense flow field and are colorcoded to show the flow direction (blue/yellow: vertical, green/red: horizontal) and the flow magnitude (brighter being larger).
example in the top row. For data that provides some outline, for instance the smoke simulation example in the middle row or also liquid data, the metric does not work as well but still provides reasonable flow field. But for full spatial examples like from the Burger's 's Advection-Diffusioo equation (see bottom row) the network is no longer able produce meaningful flow fields. The results are often very uniform flow with similar magnitude and direction. These observations are reflected the performance of the Optical flow metric Tab. I.
B.5 NON-SIAMESE ARCHITECTURE
To compute a metric without the Siamese architecture outlined above, we use a network structure with single output, shown in Fig. 11. Thus, instead of having identically feature extractors and combining the feature maps, here the distance directly predicted from the stacked inputs with a single network with about 1.24 million weights. After using the same feature extractor as described in Section B.I, the final set of feature maps is spatially reduced with an adaptive MaxPool operation. Next, the result is flattened and three consecutive fully connected layers process the data to form the final prediction. Here, the last activation function is sigmoid instead of ReLU. The reason is that a ReLU would clamp every negative intermediate value to a zero distance, while sigmoid compresses the intermediate value to small distance that is more meaningful than directly clamping
Adaplive MaxPool
Flatlen
Feature extractor identical to base network
13
Dropout
224
13
128
128
connected
128
layer
Predicted
Stacked Inputs
distance
Sigmold
Figure 11: Non- Non-Siamese network architecture with the same feature extractor used in Fig. 6. It uses both stacked inputs and directly predicts the final distance value from the last set of feature maps with several fully connected layers.
18

---

Under review as a conference paper at ICLR 2020
In terms of metric properties, this architecture only provides non-negativity (see Eq. (1)) due the final sigmoid function. All other properties can not be guaranteed without further constraints. This is the main disadvantage of non-Siamese network. These issues could be alleviated with specialized training data or by manually adding constraints to the model, e.g., to have some amount of symmetry (see Eq. (2)) and at least weakened identity of indiscernibles (see Eq. (9)). However, compared to Siamese network that guarantees them by design, these extensions are clearly sub-optimal. As result of the missing properties, this network has significant problems with generalization. While it performs well on the training data, the performance noticeably deteriorates for several of the test data sets, as shown in Tab.
B.6 SKIP CONNECTIONS IN BASE NETWORK
As explained above, our base network primarily serves as feature extractor to produce activations that employed to evaluate learned metric. In many state-of-the-art methods, networks with
224
35
55
Skip connection via channel concatenation
11x11 Convolutinn with strice
56
56
224
MaxPool with
32
32 + 32
32
27
RRLUU
64
ReLUU
13
13
Transpoaed convalution stride 1 ReLU
13
13
convilution with stride 2 ReLu
Network architecture with skip connections for better information transport between feature maps. Transposed convolutions are used to upscale the feature maps in the second half of the network to match the spatial size of earlier layers for the skip connections.
Figure 12:
skip connections are employed (Ronneberger et al., 2015; He et al., 2016; Huang et al., 2017), as experiments have shown that these connections help to preserve information from the inputs. In case, the classification "output" of network such as the AlexNet plays no actual role. Rather, the features extracted along the way are crucial. Hence, skip connections should not improve the inference task for our metrics. To verify that this is the case, have included tests with base net- work similar to the popular UNet architecture (Ronneberger et al., 2015). For our experiments, we kept the early layers closely in line with the feature extractors that worked well for the base network (see Section B. Only the layers in the decoder part have an increased spatial feature map size accommodate the skip connections. As expected, this network can be used compute reliable metrics for the input data without negatively affecting the performance. However, as expected, the improvements of skip connections for regular inference tasks do not translate into improvements for the metric calculations, shown in Tab. 1.
C IMPACT OF DATA DIFFICULTY
We shed more light on the aspect of noise levels and data difficulty via six reduced data sets, that consist of smaller amount of Smoke and Advection-Diffusion data with differently scaled noise strength values. Results are shown in Fig. 13. Increasing the noise level creates more difficult data as shown by the dotted and dashed plots representing the performance of the L and the LPIPS metric on each data set. Both roughly follow an exponentially decreasing function. Each point on the solid line plot is the test result of reduced LNSM model trained the data set with the corresponding
19

---

Under review as a conference paper at ICLR 2020
1.0
LNSM on all lest data
L2 training data
0.8
X LPIPS on training data
Correlation
0.6
0.4
0.2
K
0.0
Ox
5x
10x
15x
20x
25x
Difficulty of training data (scaled noise strength)
Impact increasing data difficulty for a reduced training data set. Evaluations on training data for L and LPIPS, and the test performance of models trained with the different reduced data sets (LNSMmdurod are shown.
Figure 13:
noise level. Apart from the data, the entire training setup was identical. This shows that the training process is very robust to the noise, as the result on the test data only slowly decreases for very high noise levels. Furthermore, small amounts of noise improve the generalization compared model that was trained without any noise. This is somewhat expected, as a model that never saw noisy data during training not learn to extract features which robust w.r.t. noise.
the
D DATA SET DETAILS
In the following sections the generation of each used data set is described. For each figure showing data samples (consisting of a reference simulation and several variants with single changing initial parameter), the lefimost image is the reference and the images to the right show the variants in order of increasing parameter change. For the figures 14, 15, 16, and 17 the first subfigure (a) demonstrates that medium and large scale characteristies behave very non-chaotic for simulations without any added noise. They are only included for illustrative purposes and are not used for training. The second and third subfigure (b) and (c) in each show the training data of LNSM, where the large majority of data falls into the category (b) of normal samples that follow the generation ordering, even with more varying behaviour. Category (c) small fraction of the training data and the shown examples are specifically picked, worst case examples to show how the chaotic behaviour can sometimes override the ordering intended by the data generation. In some cases, category (d) is included to show how normal data samples from the test set differ from the training data.
D.1 NAVIER- STOEES EQUATIONS
These equations describe the general behaviour fluids with respect to advection, viscosity, pres- sure, and mass conservation. Eq. (10) defines the conservation of momentum and Eq. (11) the conservation of mass inside the fluid.
Ou
VP +
+ (u V)u
(10)
0t
P
v
(11)
In this context, u is the velocity, P is the pressure the fluid exerts, p is the density of the fluid (usually assumed to constant), is the kinematic viscosity coefficient that indicates the thickness of the fluid, and y denotes the acceleration due gravity. With this PDE three data sets created using
20

---

Under review as a conference paper at ICLR 2020
smoke and liquid solver. For all data, 2D simulations until certain step and useful data fields were exported afterwards.
Smoke For the smoke data, standard Fulerian fluid solver using preconditioned pressure solver based on conjugate gradient and a Semi-Lagrangian advection scheme was employed. The gen-
(a) Data samples generated without noise: tiny output changes following generation ordering
(b) Normal training data samples with noise: larger output changes but ordering still applies
(c) Outlier data samples: noise can override the generation ordering by chance
Figure 14: Various smoke simulation examples using one component of the velocity (top rows), the density (middle rows), and the pressure field (bottom rows).
eral setup for every smoke simulation consists of rectangular smoke source at the bottom with a fixed additive noise pattern to provide smoke plumes with more details. Additionally, there is downwards directed, spherical force field area above the source which divides the smoke in two major streams along it. We chose this solution over an actual obstacle in the simulation, in order to avoid overfitting clearly defined black obstacle area inside the smoke data. Once the simulation reaches predefined time step, the density, pressure, and velocity field (separated by dimension) is exported and stored. Some examples can found Fig. 14. With this setup the following initial conditions were varied in isolation:
Smoke buoyancy in and y-direction
Strength of noise added the velocity field
Amount of force x- and y-direction provided by the force field
Orientation and size of the force field
Position of the force field in x- and y-direction
Position of the smoke source in x- and y-direction
21

---

Under review as a conference paper at ICLR 2020
Liquid For the liquid data, solver based on the fluid implicit particle (FLIP) method proposed by Zhu & Bridson (2005) was employed. is a Eulerian- Eulrrian-Lagrangian hybrid approach that replaces the Semi-Lagrangian advection scheme with particle based advection to achieve higher accuracy and prevent the loss of mass. Still, this method is not optimal as we experienced problems with mass loss, especially for larger noise values.
(a) Data samples generated without noise: tiny output changes following generation ordering
(b) Normal training data samples with noise: larger output changes ordering still applies
(c) Outlier data samples: noise can override the generation ordering by chance
(d) Data samples from test set with additional background noise
Figure 15: Several liquid simulation examples using the binary indicator flags (top rows), the ex- trapolated levelset values (middle rows), and one component of the velocity field (bottom rows) for the training data and only the velocity field for the test data.
The simulation setup consists of large breaking dam and several smaller liquid areas for more detailed splashes. After the dam hits the simulation boundary a large, single drop of liquid is created in the middle of the domain that hits the already moving liquid surface. Then, the extrapolated level set values, binary indicator flags, and the velocity field (separated by dimension) are saved, with some examples shown in Fig. 15. The list of varied parameters include:
22

---

Under review as a conference paper at ICLR 2020
Radius of the liquid drop
Position of the drop in x- and y-direction
Amount of additional gravity force in and y-direction
Strength of noise added the velocity field
For the liquid test additional background noise was added to the velocity field of the simulations (see Fig. 15d). Because this only alters the velocity field, the extrapolated level set values and binary indicator flags used for this data set.
D.2 ADVECTION-DIPFUSIE AND BURGER'S EQUATION
For these PDEs our solvers only discretized and solve the corresponding equation in 1D. Afterwards, the different time steps of the solution process are concatenated along a new dimension form 2D) data with one spatial and one time dimension.
(a) Data samples generated without noise: tiny output changes following generation ordering
(b) Normal training data samples with noise: larger output changes but ordering still applics
Outlier data samples: noise override the generation ordering chance
(d) Data samples from test set noise directly added to the density field
Figure 16: Various examples from the Advection-Diffuusee equation using the density field.
evvectio-Diffusiin Equation This equation describes how passive quantity is transported in- side a velocity field due to the processes of advection and diffusion. Eq. (12) is the simplified Advection-Diffusion equation with constant diffusivity and no sources sinks.
dd 0t
2
Vd
(12)
Here, d denotes the density, u is the velocity, and // is the kinematic viscosity (also known as diffu- sion coefficient) that determines the strength of the diffusion. Our solver employed simple implicit
23

---

Under review as a conference paper at ICLR 2020
time integration and diffusion solver based conjugate gradient without preconditioning. The initialization for the ID fields of the simulations was created by overlaying multiple parameterized sine curves with random frequencies and magnitudes.
In addition, continuous forcing controlled by further parameterized sine curves was included in the simulations time. In this case, the only initial conditions vary the forcing and initialization parameters of the sine curves and the strength density field used shown Fig. 16. For the Advection-Diffusiin
the added noise. From this PDE only the passive
set the noise was instead added directly to the passive density field of the simulations. This creates results with more small scale details as shown in Fig. 16d.
Burger's Equation This equation is very similar the Divection-Diffusionn equation and de- scribes how the velocity field itself changes due to diffusion and advection.
du =ル u Vu Ot
(13)
Eq. (13) is known as the viscous form of the Burger's equation that develop shock waves, and again u is the velocity and / denotes the kinematic viscosity. Our solver for this PDE used a slightly different implicit time integration scheme, but the same diffusion solver used for the Advection- Diffusion equation. The simulation setup and parameters were also the same; the only difference is
(a) Data samples generated without noise: tiny output changes following generation ordering
bb) Normal training data samples with noise: larger output changes but ordering applies
(c) Outlier data samples: noise override the generation ordering by chance
Figure 17: Different simulation examples from the Burger's ' equation using the velocity field.
that the velocity field instead of the density is exported. As consequence, the data in Fig. 17 looks relatively similar to the results from the Advection-Diffusii equation.
D.3
OTHER
The remaining data sets are based on PDEs and thus not generated with the proposed method. The data is only used to test the generalization of the discussed metrics and not for training or validation.
Shapes This data set tests if the metrics able to track simple, moving geometric shapes. To create it, straight path between two random points inside the domain is generated and a random shape is moved along this path in steps of equal distance. The size of the used shape depends on the
24

---

Under review as a conference paper at ICLR 2020
distance between start and end point, such that significant fraction of the shape overlaps between two consecutive steps. is also ensured that no part of the shape leaves the domain at any step. by using a sufficiently big boundary area when generating the path.
es
Examples from the shapes data set using field with only binary shape values (first row), shape values with additional noise (second row), smoothed shape values (third row), and smoothed values with additional noise (fourth row).
Figure 18:
With this method, multiple random shapes for single data sample are produced and their paths overlap, such that they occlude each other to provide an additional challenge. All shapes are moved in their parametric representation and only when exporting the data, they are discretized onto fixed binary grid. To add more variations to this simple approach, we also apply them in a non-binary way with smoothed edges and include additive gaussian noise over the entire domain. Examples for the different exports can be seen in Fig. 18.
Video For this data different publicly available video recordings acquired and processed in three steps. First, videos with abrupt cuts, scene transitions or camera movements were discarded,
Figure 19: Multiple examples from the video data set.
and afterwards the footage was broken down into single frames. Then, each frame was resized to match the spatial size of our other data by linear interpolation. Since directly using consecutive frames is no challenge for any analyzed metric and all of them recovered the ordering almost per- fectly, achieved more meaningful data set by skipping several intermediate frames. For the final data set, we defined the first frame of every video as the reference and subsequent frames in an interval step of ten frames the increasingly different variations. Some data examples can be found in Fig. 19.
TID2013 This data set created by Ponomarenko al. (2015) and used without any further modifications. It consists of 25 reference images with 24 distortion types in five levels. As a result not directly comparable data sets, it is excluded from the test set aggregations in Tab. 1.
25

---

Under review as a conference paper at ICLR 2020
The distortions focus various types of noise, image compression and color changes. Fig. 20 contains examples from the data set.
Figure 20: Examples from the TID2013 data proposed by Ponomarenko al. (2015). Displayed are a change of contrast, three types of noise, denoising, jpg2000 compression, and two color quan- tizations (from left right and top bottom).
D.4 DATA AUGMENTATION
To add more variation to our data, we include different data augmentation techniques. These aug- mentations help the network become the augmented data can be found in Fig. 21. In order to compare to existing natural image feature extractors trained for RGB images, we focus on three channel inputs. As a first step, the scalar, i.c. grey-scale, data is converted to three channel RGB input with a random color map (out of five fixed variations) no color map all by copying the data directly each channel.
invariant typical data transformations. Examples for
Figure 21: Augmented samples from the training sets in groups with Smoke, diveecion-Diifusiin Equation, Burger's Equation, and Liquid data. The upper row in each group shows the same ref- erence simulation, and the lower row contains variations with different ground truth distances (in- creasing from left to right).
Next, the data is randomly flipped on either axis and rotated in increments of 90" to provide robust- ness to rotations. The rotated data fields are then cropped from their simulation size 256 x 256 to size of 224 x 224 which is the typical input size for existing feature extractors. Finally, each input is normalized to a standard normal distribution. The mean and standard deviation are computed from all available training data without augmentations in pre-processing step with online algorithm from Welford (1962). Note that each data sample gets a new augmentation every time it used, and that the corresponding reference receives the identical transformations the keep comparability (see Fig. 21). For the validation and test data, only a bilinear interpolation to the correct input size and the final normalization performed.
26

---

Under review as a conference paper at ICLR 2020
E ADDITIONAL EVALUATIONS
In the following, we demonstrate other ways to compare the performance of the analyzed metrics on our data sets. In Tab. 2 the Pearson correlation coefficient used instead of Spearman's 's rank correlation coefficient. While Spearman's correlation measures monotonic relationships by using ranking variables, it directly measures linear relationships as discussed in Section 5.
The results in Tab. 2 match very closely the numbers provided in Tab. 1. The best performing metrics in both tables are identical, only the numbers slightly vary. Since a linear and a mono- tonic relation describes the results of the metries similarly well, there are no apparent non-linear dependencies that can not be captured using the Pearson correlation.
Table 2: Performance comparison on validation and test data sets measured in terms of the Pearson correlation coefficient. Bold values show the best performing metric for each data set and bold+italic values are within a 0.01 error margin of the best performing. On the right visualization of the combined test data results is shown for selected models.
Validation data sets
Test data sets
Shallow
Metric
Image-based
Smo
T.qq Adv
Bur
TTD
T gN
AdvL
Sha
Vid
A11
Experimental
L,2
0.66
0.81
0.71
0.58
0.85
0.72
0.57
0.55
0.76
0,56
Proposed
SSIM
0.67
0.74
0.75
0.68
0.83
0.25
0.69
0.34
0.66
0.47
LPIPS vO.1.
0.71
0.75
0.76
0.72
0.79
0.63
0.61
0.82
0.80
0.65
Comcistion (AI)
AlexNelranoo
0.63
0.75
0.66
0.64
0.80
0.63
0.65
0.68
0.77
0.60
0.6
AlexNefpasen
0.67
0.70
0.68
0.70
0.79
0.40
0.63
0.84
0.80
0.61
Optical flow
0.63
0.56
0.37
0.39
0.49
0.45
0.28
0.61
0.74
0.48
0.5
Non- Sia-ese
0.71
0.82
0.75
0.69
0.26
0.72
0.62
0.65
0.68
0.63
Skiptoon scrotch
0.65
0.83
0.74
0.66
0.72
0.78
0.59
0.83
0.78
0.70
LNSM nosteless
0.64
0.82
0.74
0.60
0.69
0.80
0.58
0.83
0.75
0.68
SSI
UPIPS
Optioo NonSiam JWS
LNSM strong surive
0.63
0.81
0.71
0.61
0.70
0.78
0.50
0.80
0.76
0.64
LNSM (ours) 0.68
0.82
0.76
0.70
0.71
0.79
0.61
0.86
0.76
0.71
In the Tables 3 and we employ a different, more intuitive approach to determine combined cor- relation values for each data set using the Pearson correlation. We no longer analyzing the entire predicted distance distribution and the ground truth distribution at once as done for Tab. 1. Instead,
Table 3: Performance comparison on test data sets measured by computing mean and std. dev. (in brackets) of Pearson correlation coefficients from individual data samples. Bold values show the best performing metric for each data set and bold+italic values are within a 0.01 error margin of the best performing.
Test data sets
Metric
TID
LiqN
AdvD
Sha
Vid
A11
0.98 (0.19)
0.76 (0.29)
0.59 (0.45)
0.61 (0.29)
0.82 (0.33)
0.68 (0.37)
SSIM
0.92 (0.40)
0.26 (0.49)
0.73 (0.36)
0.45 (0.45)
0.77 (0.37)
0.57 (0.46)
LPIPS vO.1.
0.94 (0.33)
0.66 (0.38)
0.65 (0.41)
0.83 (0.24)
0.85 (0.30)
0.74 (0.36)
AlexNetnudom
0.96 (0.27)
0.68 (0.34)
0.69 (0.38)
0.68 (0.26)
0.82 (0.33)
0.71 (0.34)
AlexNel
0.94 (0.33)
0.41 (0.49)
0.67 (0.39)
0.85 (0.21)
0.85 (0.29)
0.70 (0.40)
Optical flow
0.74 (0.67)
0.50 (0.34)
0.32 (0.53)
0.63 (0.45)
0.78 (0.45)
0.53 (0.49)
Non-Siamese
0.47 (0.88)
0.76 (0.24)
0.66 (0.41)
0.67 (0.28)
0.76 (0.42)
0.71 (0.35)
Skip scratch
0.99 (0.14)
0.85 (0.15)
0.61 (0.42)
0.84 (0.23)
0.82 (0.33)
0.76 (0.33)
nnSMneiselesr
0.98 (0.19)
0.86 (0.15)
0.61 (0.41)
0.84 (0.26)
0.79 (0.38)
0.76 (0.34)
LNSM strong noise
0.99 (4.
0.83 (0.19)
0.52 (0.45)
0.81 (0.23)
0.82 (0.35)
0.73 (0.36)
LNSM (ours)
0.97 (0.23)
0.83 (0.22)
0.64 (0.42)
0.86 (0.23)
0.80 (0.37)
0.77 (0.34)
individually compute the correlation between the ground truth and the predicted distances for the single data samples of the data set. From the single correlation values, we compute the mean and standard deviations shown in the tables. Note that this approach potentially produces less accurate
27

---

Under review as a conference paper at ICLR 2020
comparison results, as small errors in the individual computations accumulate to larger devia- tions in mean and standard deviation. Still, the values in both tables lead to very similar conclusions as Tab. I: The best performing metrics are almost the same and low combined correlation values match with results that have a high standard deviation and a low mean.
Table 4: Performance comparison on validation data sets measured by computing mean and stan- dard deviation (in brackets) of Pearson correlation coefficients from individual data samples. Bold values show the best performing metric for each data set and bold+italic values are within a 0.01 error margin of the best performing. On the right a visualization of the combined test data results is shown for selected models.
Validation data sets
Shallow
Metric
Image-based
Smo
Liq
Adv
Bur
Experimental Proposed
I,2 SSIM
0.71 (0.34)
0.84 (0.23)
0.76 (0.28)
0.63 (0.41)
0.73 (0.30)
0.78 (0.29)
0.80 (0.26)
0.72 (0.38)
LPIPS vO.1.
0.77 (0.28)
0.79 (0.24)
0.81 (0.26)
0.77 (0.32)
Correlation (All)
1.0
AlexNet
0.68 (0.36)
0.79 (0.28)
0.71 (0.36)
0.69 (0.36)
0.8
AlexNelfrzee
0.72 (0.31)
0.74 (0.29)
0.73 (0.35)
0.75 (0.33)
0.6
Optical flow
0.66 (0.38)
0.59 (0.47)
0.38 (0.52)
0.41 (0.49)
0.4
Non-Sion Siamese
0.76 (0.27)
0.87 (0.19)
0.80 (0.24)
0.75 (0.33)
0.2
Skip from scratch
0.69 (0.34)
0.87 (0.19)
0.79 (0.26)
0.72 (0.34)
0.0
LNSM moiseless
0.68 (0.33)
0.85 (0.24)
0.78 (0.30)
0.66 (0.37)
L2
SSIM LPIPS
OptFlow Sprns Skip
LNSM
LNSM strong noise
0.67 (0.36)
0.85 (0.22)
0.76 (0.33)
0.67 (0.39)
LNSM (ours)
0.72 (0.31)
0.85 (0.22)
0.81 (0.23)
0.75 (0.33)
Fig. 22 shows a visualization of predicted distances e against ground truth distances d for different metries on every sample from the test sets. Each plot contains over 6700 individual data points to illustrate the global distance distributions created by the metrics, without focusing on single cases. theoretical optimal metric would recover a perfectly narrow distribution along the line = d, while worse metrics recover broader, more curved distributions. Overall, the sample distribution of an L metric is very wide. LPIPS manages to follow the optimal diagonal a lot better, but our approach approximates it with the smallest deviations, as also shown in the tables above. The L?2 metric performs very poorly on the shape data indicated by the too steeply increasing blue lines that flatten after ground truth distance of 0.3. LPIPS already significantly reduces this problem, but LNSM still works slightly better. A similar issue is visible for the Cnvvection-Diffsiiin data, where
22
LPIPS
LNSM
Predicted diecance (ntrmalized
2.0
Predicted distance (narmalize)
1.0
Predicted distance (nommaized)
1.0
0.8
0.8
0.8
Ligh
0.6
0.6
Sha
0.4
0.4
0.2
0.0
0.0
Ground truth distance
Ground truth distance
Ground truth distance
Figure 22: Distribution evaluation of ground truth distances against normalized predicted distances for 2 LPIPS and LNSM on all test data (color coded).
for L a larger number of red samples is below the optimal c = d line, than for the other metrics. LPIPS has the worst overall performance for liquid test set, indicated by the large number of fairly chaotic green lines in the plot. On the video data, all three metrics perform similarly well.
A fine tneigrained distance evaluation in 200 steps of 2 and our LNSM metric via the mean and stan- dard deviation of different data samples is shown in Fig. 23. Similar to Fig. 22, the mean of an optimal metric would follow the ground truth line with standard deviation of zero, while the mean
28

---

Under review as a conference paper at ICLR 2020
of worse metrics deviates around the line with high standard deviation. The plot the left com- bines eight samples with different seeds from the Sha data set, where only a single shape is used. Similarly, the center plot aggregates eight samples from Sha with more than plot shows six data samples from the I. Li test set that vary by the amount of noise that was injected into the simulation (see Fig. 3).
shape. The right
The task of only tracking single shape in the example the left
single-Sha 8 samples [seed]
multi-Sha samples [seed]
LigN
samples [noise amount]
Predicted distance (rommalize)
Prodicted distacce (nommaizdd)
Predkcctdd distrnce (rommltbed)
1.0
1.0
1.0
Ground truth
L mean
0.6
0.6
0.6
LNSM mean
0.4
LNSM dev.
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.9
1.0
Position
GravityX variation
Figure 23: Mean and standard deviation of normalized distances over multiple data samples for L and LNSM. The samples differ by the quantity displayed in brackets. Each data sample uses 200 parameter variation steps instead of 10 like the others in our data sets. For the shape data the position of the shape varies and for the liquid data the gravity in x-direction is adjusted.
is the easiest of the three shown cases. Both metrics have no problem to recover the position change until variation of 0.4, where L can no longer distinguish between the different samples. Our metric recovers distances with continuously rising mean and very low standard deviation. The task in the middle is already harder, as multiple shapes can occlude each other during the position changes. Starting at position variation of 0.4 both metrics have quite high standard deviation, but the proposed method stays closer to the ground truth line. L shows a similar issue as before, because it flattens relatively fast. The plot on the right features the hardest task. Here, both metrics perform similar as each has a different problem in addition to an unstable mean. Our metric stays close to the ground truth, but has quite high standard deviation starting at about variation of 0.4. The standard deviation of I lower, but instead starts off with a big jump from the first few data points. To some degree this is caused by the normalization of the plots, but it still overestimates the relative distances for small variations in the simulation parameter.
These findings also match with the distance distribution evaluations in Fig. 22 and the tables above: Our method has significant ad advantage over shallow metrics on shape data, while the differences of both metries become much smaller for the liquid test
F NOTATION
In this work, we follow the notation suggested by Goodfellow et al. (2016). Vector quantities displayed in bold and tensors use a sans-serif font. Double-barred letters indicate sets or vector spaces. The following symbols are used:
R
Real numbers
i,j
Indexing in different contexts
I
Input space the metric, i.e., color images field data of size 224 224 Dimension of the input space when flattened to a single vector Elements in the input space
a
v.
Latent space of the metric, i.e., of 3rd order feature map tensors
b
Dimension of the latent space when flattened to a single vector
á, ÿ.
Elements in the latent space IL, corresponding to æ,
Weights for the learned average aggregation (1 per feature map)
29

---

Under review as a conference paper at ICLR 2020
Po: .
Initial conditions parameters of a numerical simulation
Number of steps when varying simulation parameter, thus size of a minibatch
00; 01,
On
Series of outputs of simulation with increasing ground truth distance to og
Amount of change single simulation parameter
1, to
t
Time steps of numerical simulation
Strength of the noise added to simulation
Ground truth distance distribution, determined by the data generation via Predicted distance distribution (supposed to match the corresponding c) Mean of the distributions and d Euclidean norm of a vector Entire function computed by our metric
e,d
Il....l
m(x,y) y)
mi (2,
First part of m(x, y). i.e., the base network and the feature map normalization Second part of m(x y). i.c., the latent space difference and the aggregations 3rd order feature tensor from one layer of the base network
m² 37)
Jcs Ix.
Channel dimension (ge) and spatial dimensions (gxz Py. of G Optical flow network
fmy, fum
Flow fields computed by an optical flow network f from two inputs Components of the flow field fay
fTu .f2
V,
2
Gradient (V) and Laplace operator (V2)
Partial derivative operator
t
Time in our PDEs
Velocity in PDEs
V
Kinematic viscosity diffusion coefficient in our PDEs
d.p
Density our PDEs
P
Pressure in the Navier-Stoke Stokes Equations
g Gravity in the Navier-Stokes Equations
G
MATERIAL DOWNLOAD
Anonymized and time- stamped supplemental material for our submission can be downloaded here.
30


Reviews:
[{"review_id": "rJeuRmuJ9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a well-written paper which looks into options for learning similarity metrics on data derived from PDE models common in the sciences. In comparison to other metric learning settings, here a type of ground-truth distance information is available (rather than, say, triplets), and it is possible to attempt to directly target an objective function which aims to match the learned distance to the ground-truth distance. The model architecture follows a fairly standard siamese-network setup.\n\nQuite a bit of space is devoted to ensuring that the learned metric actually satisfies pseudo-metric axioms. This is all very clearly presented, with justifications for different modeling choices and how they preserve the axioms; my only criticism here is that many aspects of this are fairly obvious (i.e. an architecture which shares weights in computing the embeddings of both data points, followed by computing a squared L2 distance, will quite clearly get us in the ballpark of a pseudometric), but in my opinion \"excess\" clarity is much better than the opposite.\n\nI think the more important contribution of the paper is in sections 4 and 5, which outlines a specific data generation process, including means of injecting noise, and compares options for loss functions. I would have expected pearson correlation to work quite well in this context, and it is interesting to note that performance notably improved by also adding an MSE term. I am curious about the \"distance\" prediction, as described just before equation 5, where it is stated that d \\in R^n — is this really R^n? The target distance c is in [0,1]^n, and it seems like a simple modification to the distance prediction network would be capable of ensuring that the predicted values d also fall in this range. Such normalization could reduce the need for the MSE loss term, which presumably helps keep the overall relative scales of the two distances in check.\n\nThe empirical testing is also thorough, and I particularly appreciate the use of the random-weight networks as a baseline — I think it is good to note that these are actually fairly competitive on many of the test data sets (in fact, I believe it should be in bold for \"TID\" in table 1). \n\nI think the main weakness of this paper is that it falls slightly short of actually presenting the real use cases and needs for a similarity metric on PDE outputs — in my opinion, this comes to play when matching the output of a PDE-based model with real data. It would be nice to see a discussion of how this could be useful for parameter inference in PDE models. If there are other important applications of a distance learned in this way, I think the paper could benefit *greatly* by pointing them out. Otherwise, this risks being perceived as adding little value, since for individual PDE runs with known parameters, there is a ground-truth distance available — in which case, why bother using deep learning to estimate the distance, if the parameters are known? I think relevance to applications should to be clearly addressed.\n\nThe supplemental material is long, but complete and clearly presented.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syx5mi3lqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is very well written and easy to understand. They focus no domains of data where there exists some controllable parameter(s) for data generation, using this parameter in a way that resembles self-supervised learning losses. The main contribution I think is in the use of correlations of changes in the scoring space (d) with changes in the generative parameters.\n\nThe main weakness of the paper though is the novelty / proper connection to self-supervised learning work. There is a large body of research that uses techniques similar to those outlined in this work, particularly in the use of a siamese-like network to ensure that representations follow some characteristic. Besides being very close to [1] (cited in this work), there are a number of works that try to make sure that the output (loosely, and not directly) corresponds to a binary variable indicating whether the two inputs come from the same image / sequence / graph / etc [2, 3]. Closer to this work, there are self-supervised methods that learn by \"ego motion\" [4] and video representation comparisons [5], to name a few.\n\nFull disclosure: I didn't read the appendix, and there seems to be interesting / useful results there. Perhaps some of the architecture description can be scaled down in the main text and some of the useful stuff from the appendix could be added.\n\n[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric\n[2] Data-Efficient Image Recognition with Contrastive Predictive Coding\n[3] Learning Representations by Maximizing Mutual Information Across Views\n[4] Learning to See by Moving\n[5] Unsupervised Learning of Visual Representations using Videos\n\nUpdate:\nI have read the responses and generally I am happy with them. I'd like to wait on R2 and see how their concerns over the metric play out, but I'm still leaning towards acceptance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeG8N4atS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use siamese convolutional network to generate a metric for images. The authors use both PDE generated simulated data and some real-world set to evaluate the proposed metric. \n\nWhile it seems to be a legit metric I have some concerns listed below:\n\n1. Need a proof why the cnn-based evaluation metric is a metric. In particular, the triangle inequality.\n\n2. Instead of using traditional metrics, I can first do a feature transform and then try a metric on top of the transformed feature space. Intuitively I do not quite see how the proposed metric is better than this naive feature+distance design.\n\n3. The paper spend one and a half page describing the data generation process. The authors claim using PDEs to generate data can have some special control of the data de-similarity, on the other hand, the performance on the simulated data does not indicate how practical the designed “metric” is for real-world data which is possibly the main interest of most of the audience. \n\n4. The evaluation on the real-world data set is very limited. How does the proposed metric perform on image datasets such as CIFAR, ImageNet, MNIST? For example we may design experiment comparing inter/intra-class metric comparisons. And how that metric can be used to improve the sota results on the data sets?\n\n5. The metric construction section is not quite self-contained. For example, could the authors states in details how the feature map normalization and aggregations are actually done in the algorithm instead of just citing some related works?\n\n6. The loss function used in the training seems weird. I am worried how those two terms balance. Also it is a batch-dependent loss given the \\bar{c} and \\bar{d}. As the batch size gets changed the estimation accuracy of the Pearson coefficient may change in a different way as your first squared loss.\n\n7. Pearson coefficient only captures the linear correlation. I would suggest looking into something like mutual information instead.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a Siamese neural net architecture for learning similarities among field data generated by numerical simulations of partial differential equations. The goal would be to find which two field data are more similar to each. One use case mentioned is the debugging of new numerical simulators, by comparing them with existing ones. 

The reviewers had mixed opinions on the paper. I agree with a negative comment of all three reviewers that the paper lacks a bit on the originality of the technique and the justification of the new loss proposed, as well as the fact that no strong explicit real world use case was given. I find this problematic especially given that similarities of solutions to PDEs is not a mainstream topic of the conference. Hence a good real world example use of the method would be more convincing.
Decision: Reject