Learning Similarity Metrics for Volumetric Simulations with Multiscale CNNs

1 INTRODUCTION . Making comparisons is a fundamental operation that is essential for any kind of computation . This is especially true for the simulation of physical phenomena , as we are often interested in comparing simulations against other types of models or measurements from the physical system . Mathematically , such comparisons require metric functions that determine scalar distance values as a similarity assessment . A fundamental problem is that traditional comparisons are typically based on simple , element-wise metrics like the L1 or L2 distances , due to their computational simplicity and a lack of alternatives . Such metrics can work reliably for systems with few elements of interest , e.g . if we want to analyze the position of a moving object at different points in time , matching our intuitive understanding of distances . However , more complex physical problems often exhibit large numbers of degrees of freedom , and strong dependencies between elements in their solutions . Those coherences should be considered when comparing physical data , but element-wise operations by definition ignore such interactions between elements . With the curse of dimensionality this situation becomes significantly worse for systems that are modeled with dense grid data , as the number of interactions grows exponentially with a linearly increasing number elements . Such data representations are widely used , e.g . for medical blood flow simulations ( Olufsen et al. , 2000 ) , over climate and weather predictions ( Stocker et al. , 2014 ) , to the famous unsolved problem of turbulence ( Holmes et al. , 2012 ) . Another downside of element-wise metrics is that each element is weighted equally , which is typically suboptimal : E.g . in the context of fluids , smoke plumes behave differently along the vertical dimension due to gravity or buoyancy , and small key features like vortices are more indicative of the fluid ’ s general behavior than large areas of near constant flow ( Pope , 2000 ) . In the domain of natural images , neural networks have been successfully employed for similarity models that can consider larger structures , typically via learning from the semantic meaning of the images through class labels or data that encodes our human perception . Similarly , physical systems exhibit spatial and temporal coherence due to the underlying laws of physics , from which we can derive a similarity model . To robustly learn similarity assessments of scalar and vectorial volumetric data from such a model , our work makes the following contributions : • We propose a novel similarity model based on the entropy of physical systems . It is employed to synthesize sequences of volumetric physical fields suitable for metric learning . • We show that our Siamese , multiscale feature network results in a stable metric that successfully generalizes to new physical phenomena . To the best of our knowledge this is the first learned metric inherently designed for the similarity assessment of volumetric fields . • The metric is employed to analyze turbulence in a case study , and its invariance to rotation and scale are evaluated . In addition , we analyze correlation-based loss functions with respect to their tradeoff between batch size and accuracy of correlation computation . The central application of the proposed VolSiM metric is the similarity assessment of new physical simulation methods , numerical or learning-based , against a known ground truth . This ground truth can take the form of measurements , higher resolution simulations , or existing models . Furthermore , the trained metric can be used as a differentiable similarity loss for physical learning problems , similar to perceptual losses for computer vision tasks . 2 RELATED WORK . Apart from simple Ln distances , the two metrics PSNR and SSIM ( Wang et al. , 2004 ) are commonly used across disciplines for the similarity assessment of data . Similar to the underlying L2 distance , PSNR shares the issues of element-wise metrics ( Huynh-Thu & Ghanbari , 2008 ; 2012 ) . SSIM computes a more intricate function , but it was shown to be closely related to PSNR ( Horé & Ziou , 2010 ) and thus has similar problems ( Nilsson & Akenine-Möller , 2020 ) . Furthermore , statistical tools like the Pearson correlation coefficient PCC ( Pearson , 1920 ) and Spearman ’ s rank correlation coefficient SRCC ( Spearman , 1904 ) can be employed as a simple similarity measurement . There are several learning-based metrics specialized for different domains such as rendered ( Andersson et al. , 2020 ) and natural images ( Bosse et al. , 2016 ) , interior object design ( Bell & Bala , 2015 ) , audio ( Avgoustinakis et al. , 2020 ) , and haptic signals ( Kumari et al. , 2019 ) . Especially for images , similarity measurements have been studied in multiple settings , mainly by combining deep embeddings as perceptually more accurate metrics ( Prashnani et al. , 2018 ; Talebi & Milanfar , 2018 ) . Similarly , such metrics can be employed for various image related tasks like super-resolution ( Johnson et al. , 2016 ) or image generation ( Dosovitskiy & Brox , 2016 ) . We additionally study the behavior of invariance and equivariance to transformations , which was targeted previously for rotational symmetries ( Weiler et al. , 2018 ; Chidester et al. , 2019 ) and improved generalization ( Wang et al. , 2021 ) . Traditional metric learning for natural image domains typically works in one of two ways : Either , the training is directly supervised by learning from manually created labels , e.g . via two-alternative forced choice where humans pick the most similar option to a reference ( Zhang et al. , 2018 ) . Or , the training is indirectly semi-supervised through images with class labels and a contrastive loss ( Chopra et al. , 2005 ; Hadsell et al. , 2006 ) . In that case , triplets of reference , same class image , and other class images are sampled , and the corresponding latent space representations are pulled together or pushed apart . We refer to Roth et al . ( 2020 ) for an overview of different image metric training strategies . Similarity metrics for simulation data have not been studied extensively yet . For smoke flow synthesis , Siamese networks for finding similar fluid descriptors have been applied ( Chu & Thuerey , 2017 ) . Um et al . ( 2017 ; 2021 ) used crowd-sourced user studies for the similarity assessment of liquid simulations . Simulation data was compared via a learned metric ( Kohl et al. , 2020 ) using a feature extractor with a structure similar to a classification CNN , however only considering scalar 2D data and a simple , linear distance model . Deep learning techniques also receive attention for improving simulations , e.g . for predicting states over time ( Morton et al. , 2018 ; Kim et al. , 2019 ) , for learning via physical laws ( Watters et al. , 2017 ; Greydanus et al. , 2019 ) , and for the dynamics of complex materials ( Ummenhofer et al. , 2020 ; Sanchez-Gonzalez et al. , 2020 ) . While these approaches have potential to be combined with our metric , we focus on traditional numerical methods to acquire data in the following . 3 MODELING SIMILARITY OF SIMULATIONS . To formulate our methodology for learning similarity metrics that target dissipative physical systems , we turn to the fundamental quantity of entropy . The second law of thermodynamics states that the entropy S of a physical system never decreases , thus ∆S ≥ 0 . In the following , we make the reasonable assumption that the behavior of the system is continuous and non-oscillating , S = kB log ( W ) ( 1 ) S̃ ( s ) = k log ( cws ) ( 2 ) D ( s ) = k log ( cws + 1 ) ( 3 ) D ( s ) = log ( cws + 1 ) log ( c+ 1 ) ( 4 ) and that ∆S > 0 . On the right , Eq . 1 is the Boltzmann equation from statistical mechanics ( Boltzmann , 1866 ) , that describes S in terms of the Boltzmann constant kb and the number of microstates W of a system ( here an ideal gas ) .1 Since entropy only depends on a single system state , it can be reformulated to take the relative change between two states into account : given a sequence of states s0 , s1 , . . . , sn , the relative entropy S̃ ( s ) is defined in Eq . 2 , where ws is the monotonically increasing , relative number of microstates defined as 0 for s0 and as 1 for sn . c > 0 is a system-dependent factor that determines how quickly the number of microstates increases . Next , we shift Eq . 2 such that the lower bound is 0 to derive a first similarity model D. This offset is necessary , since the properties of similarity metrics dictate that distances are always non-negative and only zero for identical states . Finally , we choose the system-dependent constant k appropriately , as relative similarities do not change through linear transformations , and thus k does not influence the general behavior of D. Choosing k = 1/ ( log c+ 1 ) leads to the full similarity model D ( s ) in Eq . 4 , that is bound to the range [ 0 , 1 ] , and directly predicts the level of similarity between the reference s0 and any state s from a sequence . Fig . 1 illustrates this connection between the logarithmic increase of entropy within a system and the proposed similarity model . In addition , trajectories for different ∆ through this space are shown . This ∆ between states corresponds to c in the similarity model in Eq . 4 , and is directly proportional to it : For large changes ∆inf to the system , the entropy quickly increases and the states s1 , . . . , sn contain little information , hence are quickly very dissimilar to s0 ( red dotted curve ) . For the smallest possible changes ∆ , entropy slowly increases while similarity linearly decreases , due to the continuous nature of the system ( green dashed line ) . Ideally , a sequence of states employed for learning tasks should evenly exhibit both regimes as well as intermediate ones , as indicated by the black curve . For practicality , this should not rely on an extremely large number of states , as would be necessary for ∆ . In the following , we will refer to this property of a sequence being informative with respect to a similarity analysis as difficulty of a sequence with a fixed length n. The red area in Fig . 1 generally corresponds to sequences that are too difficult , i.e . that do not bare any similarity to s0 very quickly , while the green area generally corresponds to sequences that are too easy , i.e . that barely change . Algorithm 1 Computation of distance prediction d of metric m and ground truth distances g on an ordered sequence s0 , s1 , . . . , sn . for i = 1 to n do q.append ( 1− PCC ( s0 , si ) ) q ← normalize q to [ 0 , 1 ] c← fit log ( 10 cq+1 ) log ( 10c+1 ) for c for i = 0 to n− 1 do for j = i+ 1 to n do d.append ( m ( si , sj ) ) ws.append ( ( j − i ) /n ) g ← log ( 10 cws+1 ) log ( 10c+1 ) The resulting formulation for D represents an idealized similarity model and does not directly work as a metric due to two reasons : 1 ) a metric only takes two states as an input , so estimating c would be difficult , and 2 ) both inputs do not necessarily have to come from the same system , for example when comparing a velocity observation to a simulated ground truth . To overcome this , we formulate a semi-supervised learning problem by creating sequences s0 , s1 , . . . , sn of different physical systems , and training a neural network m to predict the distance d given the reference distances g as computed via D. This technique incorporates the underlying physical behavior , compared to adding variations in a post-process ( as it is often done in the domain of images , e.g . by Ponomarenko et al . ( 2015 ) ) . Alg . 1 shows how d and g are computed : First , we estimate c by evaluating Pearson ’ s distance that indicates how quickly the states become uncorrelated , normalizing it to [ 0 , 1 ] , and optimizing Eq . 4 for c via a standard least-squares 1We do not have any a priori information about the distribution of the likelihood of each microstate in a general physical system . Thus , more generic entropy models such as Gibb ’ s entropy or Shannon ’ s entropy , both of which assume a non-uniform distribution of microstates , are not immediately applicable . optimization . Then , g directly follows from Eq . 4 through the fit of c and the linearly increasing number of microstates ws , while d is computed by a pairwise evaluation of m .

Summary: The paper presents a learning-based method for evaluating the similarity between 3D volumetric data. A new similarity model based on the entropy of a physical system is proposed, which in my opinion is the paper's major contribution. The method combines the new entropy-based model with a Siamese network that is commonly used for similarity measurement on 2D imagery data. The proposed method further modifies the Siamese network in order to process 3D data. Experimental results are demonstrated on turbulence flow data, and are compared with other 2D similarity measurements.