Learning Similarity Metrics for Numerical Simulations

1 INTRODUCTION . Evaluating computational tasks for complex data sets is a fundamental problem in all computational disciplines . Regular vector space metrics , such as the L2 distance were shown to be very unreliable ( Wang et al. , 2004 ; Zhang et al. , 2018 ) , and the advent of deep learning techniques with convolutional neural networks ( CNNs ) made it possible to more reliably evaluate complex data domains such as natural images , texts ( Benajiba et al. , 2018 ) , or speech ( Wang et al. , 2018 ) . Our central aim is to demonstrate the usefulness of CNN-based evaluations in the context of numerical simulations . These simulations are the basis for a wide range of applications ranging from blood flow simulations to aircraft design . Specifically , we propose a novel learned numerical simulation metric ( LNSM ) that allows for a reliable similarity evaluation of simulation data . Potential areas of application for a such a metric are fundamental problems that arise in all areas where such numerical simulations are performed : among others , accuracy evaluations of existing and new simulation methods with respect to a known ground truth solution ( Oberkampf et al. , 2004 ) could be performed more reliably than with a regular vector norm . In addition , the energy landscape for parameter inference via optimization through a differentiable solver could be improved in order to match experimental data or observations , and generative networks for physical phenomena such as GANs , could employ a specialized metric to replace commonly used perceptual loss terms . In this work , we focus on field data , i.e . dense grids of scalars , similar to images , which were generated with known partial differential equations ( PDEs ) in order to ensure the availability of ground truth solutions . While we focus on 2D data in the following to make comparisons with existing techniques from imaging applications possible , our approach naturally extends to higher dimensions . Every sample of this 2D data can be regarded a high dimensional vector , so metrics on the corresponding vector space are applicable to evaluate similarities . These metrics are typically simple , elementwise functions such as L1 or L2 distances . We denote these as shallow metrics , and their inherent problem is that they can not capture structures of any scale or contextual information . This problem is not confined to the field of natural images : it represents a fundamental challenge for many areas of simulation , most prominently for the field of turbulence simulations ( Moin & Mahesh , 1998 ; Lin et al. , 1998 ) . Many practical problems require solutions over time , and need a vast number of non-linear operations that often result in substantial changes of the solutions even for small changes of the inputs . Hence , despite being based on known , continuous formulations , these systems can be seen as chaotic . We illustrate this behavior in Fig . 1 , where two smoke flows are compared to a reference simulation . A single simulation parameter was varied for these examples , and a visual inspection shows that smoke plume ( a ) is more similar to the reference . This matches the data generation process : version ( a ) has a significantly smaller parameter change than ( b ) , as shown in the inset graph on the right . Our LNSM robustly predicts the ground truth distances , while the L2 metric labels plume ( b ) as more similar . In our work , we focus on retrieving the relative distances of simulated data sets . Hence , we do not aim for retrieving the absolute parameter change , but a relative distance that preserves ordering with respect to this parameter . Using existing image metrics based on CNNs for this problem is not optimal either : Natural images only cover a small fraction of the space of possible 2D data , and numerical simulation outputs are located in a fundamentally different data manifold within this space . Hence , there are crucial aspects that can not be captured by purely learning from photographs . Furthermore , we have full control over the data generation process for simulation data . As a result , we can create arbitrary amounts of training data with gradual changes and a known ground truth order . With this data , we can learn a metric that is not only able to directly extract and use features , but additionally knows the fundamental interactions between them . The central contributions of our work are : • A Siamese network architecture with feature map normalization which is able to learn a metric that generalizes well to unseen simulations methods . • We also propose an improved loss function that combines a batchwise correlation loss term with a mean squared error to improve the accuracy of the learned metric . • In addition , we show how a data generation approach for numerical simulations can be employed to train networks with general and robust feature extractors for metric calculations . 2 RELATED WORK . One of the earliest methods to go beyond using simple metrics based on Lp-norms for natural images was the structural similarity index ( Wang et al. , 2004 ) . Despite improvements , this method can still be considered a shallow metric . Over the years multiple large databases for human evaluations of natural images were presented , for instance CSIQ ( Larson & Chandler , 2010 ) , TID2013 ( Ponomarenko et al. , 2015 ) , and CID : IQ ( Liu et al. , 2014 ) . With this data and the discovery that CNNs can create very powerful feature extractors that are able to recognize patterns and structures , deep feature maps quickly became established as means for evaluation ( Amirshahi et al. , 2016 ; Berardino et al. , 2017 ; Bosse et al. , 2016 ; Kang et al. , 2014 ; Kim & Lee , 2017 ) . Recently , these methods were improved by predicting the distribution of human evaluations instead of directly learning distance values ( Prashnani et al. , 2018 ; Talebi & Milanfar , 2018b ) . Finally , Zhang et al . ( 2018 ) compared different architecture and levels of supervision , and showed that metrics can be interpreted as a transfer learning approach , by applying a linear weighting to the feature maps of any network architecture to form the image metric LPIPS v0.1 . Typical use cases of these image-based CNN metrics are computer vision tasks like detail enhancement ( Talebi & Milanfar , 2018a ) , style transfer , and super- resolution ( Johnson et al. , 2016 ) . Generative adversarial networks also leverage CNN-based losses by training a discriminator network in parallel to the generation task ( Dosovitskiy & Brox , 2016 ) . Siamese network architectures are known to work well for a variety of comparison tasks such as audio ( Zhang & Duan , 2017 ) , satellite images ( He et al. , 2019 ) or finding similar interior product designs ( Bell & Bala , 2015 ) . They were additionally demonstrated to yield robust object trackers ( Bertinetto et al. , 2016 ) , algorithms for image patch matching ( Hanif , 2019 ) , and for descriptors for fluid flow synthesis ( Chu & Thuerey , 2017 ) . Inspired by these works we use a similar Siamese neural network architecture for our metric learning task . In contrast to other work on self-supervised learning that utilizes spatial or temporal changes to learn meaningful representations ( Agrawal et al. , 2015 ; Wang & Gupta , 2015 ) , our method does not rely on tracked keypoints in the data . While correlation terms have been used for learning joint representations by maximizing correlation of projected views ( Chandar et al. , 2016 ) , and are popular for style transfer applications via the Gram matrix ( Ruder et al. , 2016 ) , they were not used for learning distance metrics . As we demonstrate below , they can yield significant improvements in terms of the inferred distances . Similarity metrics for numerical simulations are a topic of ongoing investigation . A variety of specialized metrics have been proposed to overcome the limitations of Lp-norms , such as the displacement and amplitude score from the area of weather forecasting ( Keil & Craig , 2009 ) , and permutation based metrics for energy consumption forecasting ( Haben et al. , 2014 ) . Turbulent flows , on the other hand , are often evaluated in terms of aggregated frequency spectra ( Pitsch , 2006 ) . Crowdsourced evaluations based on the human visual system were also proposed to evaluate simulation methods for physics-based animation ( Um et al. , 2017 ) , and for comparing non-oscillatory discretization schemes ( Um et al. , 2019 ) . These results indicate that visual evaluation methods in the context of field data are possible and robust , but they require extensive ( and potentially expensive ) user studies . Additionally , our method naturally extends to higher dimensions , while human evaluations inherently rely on projections with at most two spatial and one time dimension . 3 CONSTRUCTING A CNN-BASED EVALUATION METRIC . In the following , we explain our considerations when employing CNNs as evaluation metrics . For a comparison that corresponds to our intuitive understanding of how distances work , an underlying metric has to obey certain criteria . More precisely , a function m : I × I → [ 0 , ∞ ) is a metric with respect to its input space I , if it satisfies the following properties ∀x , y , z ∈ I : m ( x , y ) ≥ 0 non-negativity ( 1 ) m ( x , y ) = m ( y , x ) symmetry ( 2 ) m ( x , y ) ≤ m ( x , z ) +m ( z , y ) triangle inequality ( 3 ) m ( x , y ) = 0 ⇐⇒ x = y identity of indiscernibles ( 4 ) The properties ( 1 ) and ( 2 ) are crucial as distances should be symmetric and have a clear lower bound . Eq . ( 3 ) ensures that direct distances can not be longer than a detour . Property ( 4 ) , on the other hand , is not really useful for discrete operations as approximation errors and floating point operations can easily lead to a distance of zero for slightly different inputs . Hence , we focus on a relaxed , more meaningful definition m ( x , x ) = 0 which leads to a so called pseudometric . It allows for a distance of zero for different inputs , but has to be able to spot identical inputs . We realize these requirements for a pseudometric with an architecture that follows popular perceptual metrics such as LPIPS : The activations of a CNN are compared in latent space , and accumulated with a set of weights . Afterwards , the resulting per-feature distances are aggregated to produce a final distance value . Fig . 2 gives a visual overview of this process . Base Network The sole purpose of the base network is to extract feature maps from both inputs . The Siamese architecture implies that the weights of the base network are shared for both inputs , meaning all feature maps are comparable . In addition , this ensures the identity of indiscernibles because for identical feature maps a distance of zero is guaranteed by the following operations . In the last years , a variety of powerful CNN-based feature extraction architectures were proposed . We experimented with various networks , such as AlexNet ( Krizhevsky et al. , 2017 ) , VGG ( Simonyan & Zisserman , 2015 ) , SqueezeNet ( Iandola et al. , 2016 ) , and a fluid flow prediction network ( Thuerey et al. , 2018 ) . In all cases , only the feature extracting layers are used , and the remaining layers which are responsible for the original task , e.g. , classification , are discarded . Building on this previous work , we consider three variants of the networks below : using the original pre-trained weights , finetuning them , or re-training the full networks from scratch . In contrast to typical CNN tasks where only the result of the final output layer is further processed , we make use of the full range of extracted features across the layers of a CNN ( see Fig . 2 ) . This implies a slightly different goal : while early features should be general enough to allow for extracting more complex features in deeper layers , this is not their sole purpose . Rather , features in earlier layers of the network can directly participate in the final distance calculation , and can yield important cues . As we will demonstrate below , we achieved the best performance for our data sets using a custom architecture with five layers , similar to a reduced AlexNet , that was trained from scratch ( see App . B.1 ) . Feature Map Normalization The goal of normalizing the feature maps is to transform the extracted features of each layer , that typically have very different orders of magnitude , into comparable ranges . While this task could potentially be performed by the learned weights , we found the normalization to yield improved performance in general ( see App . B.2 ) . Zhang et al . ( 2018 ) proposed a length unit normalization using a division by the Euclidean norm in channel dimension , to only measure the angle between the latent space vectors with a cosine distance . Instead , we suggest to interpret all possible feature maps as a normal distribution and to normalize them to a standard normal distribution . This is achieved via a preprocessing step using the full training data set : we subtract the mean of the feature maps and then divide by their standard deviation in channel dimension for each layer . As a result , we can measure angles for the latent space vectors and compare their magnitude in the global length distribution . Latent Space Differences Combining the latent spaces representations x̃ , ỹ that consist of all extracted features from the two inputs x , y lies at the core of the metric computation . Here , the most obvious approach to employ an elementwise x̃i − ỹi difference is not advisable , as this would directly violate the metric properties above . Instead , possible options to ensure non-negativity and symmetry are |x̃− ỹ| or ( x̃− ỹ ) 2 . We found that both work equally well in practice . Considering the importance of comparing the extracted features , the simple operations used for comparing the features do not seem optimal . Rather , one can imagine that improvements in terms of comparing one set of feature activations could lead to overall improvements for derived metrics . Hence , we experimented with replacing these operations with a pre-trained CNN-based metric for each feature map . This creates a recursive process , and a ” meta-metric ” that reformulates the initial problem of the similarity between inputs in terms of the similarity a series of deep representations of the inputs . However , as detailed in App . B.3 , we have not found this recursive approach to yield any substantial improvements . This implies that once a large enough number of expressive features is available for comparison , the in-place difference of each feature is sufficient to compare two inputs . In the following , we compute the feature difference maps ( Fig . 2 , in yellow ) via ( x̃− ỹ ) 2 . Aggregations The subsequent aggregation operations ( Fig . 2 , in green ) are applied to the difference maps to compress the contained per feature differences along the different dimensions into a single distance value . The aggregation operations only need to preserve the metric properties already established via the latent space difference . To aggregate the difference maps along the channel dimension , we found the weighted average proposed by Zhang et al . ( 2018 ) to work very well . Thus , we use one learnable weight to control the importance of a feature . The weight is a multiplier for the corresponding difference map before summation along the channel dimension . To preserve nonnegativity and the triangle inequality , the weights are clamped to be non-negative . A negative weight would mean that a larger difference in this feature produces a smaller overall distance , which is not helpful . For spatial and layer aggregation , functions like a summation or averaging are sufficient and generally interchangeable . We tested more intricate aggregation functions , such as a learned spatial average or determining layer importance weights dynamically from the inputs . When the base network is fixed and the metric only has very few trainable weights , this did improve the overall performance . But with a fully trained base network the feature extraction seems to automatically adopt these aspects , making a more complicated aggregation unnecessary . Additional details for the steps above are given in App . A , where we also show that the proposed Siamese architecture by construction qualifies as a pseudometric .

Summary: This is a well-written paper which looks into options for learning similarity metrics on data derived from PDE models common in the sciences. In comparison to other metric learning settings, here a type of ground-truth distance information is available (rather than, say, triplets), and it is possible to attempt to directly target an objective function which aims to match the learned distance to the ground-truth distance. The model architecture follows a fairly standard siamese-network setup.