Learning Similarity Metrics for Volumetric Simulations with Multiscale CNNs

1 INTRODUCTION . Making comparisons is a fundamental operation that is essential for any kind of computation . This is especially true for the simulation of physical phenomena , as we are often interested in comparing simulations against other types of models or measurements from the physical system . Mathematically , such comparisons require metric functions that determine scalar distance values as a similarity assessment . A fundamental problem is that traditional comparisons are typically based on simple , element-wise metrics like the L1 or L2 distances , due to their computational simplicity and a lack of alternatives . Such metrics can work reliably for systems with few elements of interest , e.g . if we want to analyze the position of a moving object at different points in time , matching our intuitive understanding of distances . However , more complex physical problems often exhibit large numbers of degrees of freedom , and strong dependencies between elements in their solutions . Those coherences should be considered when comparing physical data , but element-wise operations by definition ignore such interactions between elements . With the curse of dimensionality this situation becomes significantly worse for systems that are modeled with dense grid data , as the number of interactions grows exponentially with a linearly increasing number elements . Such data representations are widely used , e.g . for medical blood flow simulations ( Olufsen et al. , 2000 ) , over climate and weather predictions ( Stocker et al. , 2014 ) , to the famous unsolved problem of turbulence ( Holmes et al. , 2012 ) . Another downside of element-wise metrics is that each element is weighted equally , which is typically suboptimal : E.g . in the context of fluids , smoke plumes behave differently along the vertical dimension due to gravity or buoyancy , and small key features like vortices are more indicative of the fluid ’ s general behavior than large areas of near constant flow ( Pope , 2000 ) . In the domain of natural images , neural networks have been successfully employed for similarity models that can consider larger structures , typically via learning from the semantic meaning of the images through class labels or data that encodes our human perception . Similarly , physical systems exhibit spatial and temporal coherence due to the underlying laws of physics , from which we can derive a similarity model . To robustly learn similarity assessments of scalar and vectorial volumetric data from such a model , our work makes the following contributions : • We propose a novel similarity model based on the entropy of physical systems . It is employed to synthesize sequences of volumetric physical fields suitable for metric learning . • We show that our Siamese , multiscale feature network results in a stable metric that successfully generalizes to new physical phenomena . To the best of our knowledge this is the first learned metric inherently designed for the similarity assessment of volumetric fields . • The metric is employed to analyze turbulence in a case study , and its invariance to rotation and scale are evaluated . In addition , we analyze correlation-based loss functions with respect to their tradeoff between batch size and accuracy of correlation computation . The central application of the proposed VolSiM metric is the similarity assessment of new physical simulation methods , numerical or learning-based , against a known ground truth . This ground truth can take the form of measurements , higher resolution simulations , or existing models . Furthermore , the trained metric can be used as a differentiable similarity loss for physical learning problems , similar to perceptual losses for computer vision tasks . 2 RELATED WORK . Apart from simple Ln distances , the two metrics PSNR and SSIM ( Wang et al. , 2004 ) are commonly used across disciplines for the similarity assessment of data . Similar to the underlying L2 distance , PSNR shares the issues of element-wise metrics ( Huynh-Thu & Ghanbari , 2008 ; 2012 ) . SSIM computes a more intricate function , but it was shown to be closely related to PSNR ( Horé & Ziou , 2010 ) and thus has similar problems ( Nilsson & Akenine-Möller , 2020 ) . Furthermore , statistical tools like the Pearson correlation coefficient PCC ( Pearson , 1920 ) and Spearman ’ s rank correlation coefficient SRCC ( Spearman , 1904 ) can be employed as a simple similarity measurement . There are several learning-based metrics specialized for different domains such as rendered ( Andersson et al. , 2020 ) and natural images ( Bosse et al. , 2016 ) , interior object design ( Bell & Bala , 2015 ) , audio ( Avgoustinakis et al. , 2020 ) , and haptic signals ( Kumari et al. , 2019 ) . Especially for images , similarity measurements have been studied in multiple settings , mainly by combining deep embeddings as perceptually more accurate metrics ( Prashnani et al. , 2018 ; Talebi & Milanfar , 2018 ) . Similarly , such metrics can be employed for various image related tasks like super-resolution ( Johnson et al. , 2016 ) or image generation ( Dosovitskiy & Brox , 2016 ) . We additionally study the behavior of invariance and equivariance to transformations , which was targeted previously for rotational symmetries ( Weiler et al. , 2018 ; Chidester et al. , 2019 ) and improved generalization ( Wang et al. , 2021 ) . Traditional metric learning for natural image domains typically works in one of two ways : Either , the training is directly supervised by learning from manually created labels , e.g . via two-alternative forced choice where humans pick the most similar option to a reference ( Zhang et al. , 2018 ) . Or , the training is indirectly semi-supervised through images with class labels and a contrastive loss ( Chopra et al. , 2005 ; Hadsell et al. , 2006 ) . In that case , triplets of reference , same class image , and other class images are sampled , and the corresponding latent space representations are pulled together or pushed apart . We refer to Roth et al . ( 2020 ) for an overview of different image metric training strategies . Similarity metrics for simulation data have not been studied extensively yet . For smoke flow synthesis , Siamese networks for finding similar fluid descriptors have been applied ( Chu & Thuerey , 2017 ) . Um et al . ( 2017 ; 2021 ) used crowd-sourced user studies for the similarity assessment of liquid simulations . Simulation data was compared via a learned metric ( Kohl et al. , 2020 ) using a feature extractor with a structure similar to a classification CNN , however only considering scalar 2D data and a simple , linear distance model . Deep learning techniques also receive attention for improving simulations , e.g . for predicting states over time ( Morton et al. , 2018 ; Kim et al. , 2019 ) , for learning via physical laws ( Watters et al. , 2017 ; Greydanus et al. , 2019 ) , and for the dynamics of complex materials ( Ummenhofer et al. , 2020 ; Sanchez-Gonzalez et al. , 2020 ) . While these approaches have potential to be combined with our metric , we focus on traditional numerical methods to acquire data in the following . 3 MODELING SIMILARITY OF SIMULATIONS . To formulate our methodology for learning similarity metrics that target dissipative physical systems , we turn to the fundamental quantity of entropy . The second law of thermodynamics states that the entropy S of a physical system never decreases , thus ∆S ≥ 0 . In the following , we make the reasonable assumption that the behavior of the system is continuous and non-oscillating , S = kB log ( W ) ( 1 ) S̃ ( s ) = k log ( cws ) ( 2 ) D ( s ) = k log ( cws + 1 ) ( 3 ) D ( s ) = log ( cws + 1 ) log ( c+ 1 ) ( 4 ) and that ∆S > 0 . On the right , Eq . 1 is the Boltzmann equation from statistical mechanics ( Boltzmann , 1866 ) , that describes S in terms of the Boltzmann constant kb and the number of microstates W of a system ( here an ideal gas ) .1 Since entropy only depends on a single system state , it can be reformulated to take the relative change between two states into account : given a sequence of states s0 , s1 , . . . , sn , the relative entropy S̃ ( s ) is defined in Eq . 2 , where ws is the monotonically increasing , relative number of microstates defined as 0 for s0 and as 1 for sn . c > 0 is a system-dependent factor that determines how quickly the number of microstates increases . Next , we shift Eq . 2 such that the lower bound is 0 to derive a first similarity model D. This offset is necessary , since the properties of similarity metrics dictate that distances are always non-negative and only zero for identical states . Finally , we choose the system-dependent constant k appropriately , as relative similarities do not change through linear transformations , and thus k does not influence the general behavior of D. Choosing k = 1/ ( log c+ 1 ) leads to the full similarity model D ( s ) in Eq . 4 , that is bound to the range [ 0 , 1 ] , and directly predicts the level of similarity between the reference s0 and any state s from a sequence . Fig . 1 illustrates this connection between the logarithmic increase of entropy within a system and the proposed similarity model . In addition , trajectories for different ∆ through this space are shown . This ∆ between states corresponds to c in the similarity model in Eq . 4 , and is directly proportional to it : For large changes ∆inf to the system , the entropy quickly increases and the states s1 , . . . , sn contain little information , hence are quickly very dissimilar to s0 ( red dotted curve ) . For the smallest possible changes ∆ , entropy slowly increases while similarity linearly decreases , due to the continuous nature of the system ( green dashed line ) . Ideally , a sequence of states employed for learning tasks should evenly exhibit both regimes as well as intermediate ones , as indicated by the black curve . For practicality , this should not rely on an extremely large number of states , as would be necessary for ∆ . In the following , we will refer to this property of a sequence being informative with respect to a similarity analysis as difficulty of a sequence with a fixed length n. The red area in Fig . 1 generally corresponds to sequences that are too difficult , i.e . that do not bare any similarity to s0 very quickly , while the green area generally corresponds to sequences that are too easy , i.e . that barely change . Algorithm 1 Computation of distance prediction d of metric m and ground truth distances g on an ordered sequence s0 , s1 , . . . , sn . for i = 1 to n do q.append ( 1− PCC ( s0 , si ) ) q ← normalize q to [ 0 , 1 ] c← fit log ( 10 cq+1 ) log ( 10c+1 ) for c for i = 0 to n− 1 do for j = i+ 1 to n do d.append ( m ( si , sj ) ) ws.append ( ( j − i ) /n ) g ← log ( 10 cws+1 ) log ( 10c+1 ) The resulting formulation for D represents an idealized similarity model and does not directly work as a metric due to two reasons : 1 ) a metric only takes two states as an input , so estimating c would be difficult , and 2 ) both inputs do not necessarily have to come from the same system , for example when comparing a velocity observation to a simulated ground truth . To overcome this , we formulate a semi-supervised learning problem by creating sequences s0 , s1 , . . . , sn of different physical systems , and training a neural network m to predict the distance d given the reference distances g as computed via D. This technique incorporates the underlying physical behavior , compared to adding variations in a post-process ( as it is often done in the domain of images , e.g . by Ponomarenko et al . ( 2015 ) ) . Alg . 1 shows how d and g are computed : First , we estimate c by evaluating Pearson ’ s distance that indicates how quickly the states become uncorrelated , normalizing it to [ 0 , 1 ] , and optimizing Eq . 4 for c via a standard least-squares 1We do not have any a priori information about the distribution of the likelihood of each microstate in a general physical system . Thus , more generic entropy models such as Gibb ’ s entropy or Shannon ’ s entropy , both of which assume a non-uniform distribution of microstates , are not immediately applicable . optimization . Then , g directly follows from Eq . 4 through the fit of c and the linearly increasing number of microstates ws , while d is computed by a pairwise evaluation of m .

Summary: The paper proposes a similarity model for volumetric simulations derived from Boltzmann’s equation. A formulation of a similarity measure between a reference state and any other microstate in a dissipative physical system is presented. This similarity cannot be used as a metric as is, but can be used to train a Siamese CNN (VolSiM) that is introduced in this paper. The authors formulate a learning task by generating a sequence of physical systems. They use Pearson’s distance to estimate how quickly the microstates in a system are changing, and compute a ground truth for the CNN training using the proposed similarity measure. The network is trained to predict the distances. The CNN is designed to preserve a certain degree of translational, rotational and scale equivariances. Translational equivariance, apart from pooling operations, is given by the nature of CNNs; scale equivariance is directly encoded in the architecture of VolSiM by using four blocks processing the input data in multiple scales; and rotational equivariance is encouraged by data augmentation. The network is given ground truth distances based on the proposed similarity model, predicts distances and uses a loss which is a weighted sum of an MSE term and a correlation term. The authors propose two approaches to generate data, one which is based on varying the initial conditions of PDEs and using PDE solvers for a number of physical simulations, which is used as training data, and one which is based on spatial and temporal variations, which is used to generate test data. Furthermore, a publicly available database is used for testing. The datasets in the experiments cover a wide range of different physical models.