Adversarial twin neural networks: maximizing physics recovery for physical system

1 INTRODUCTION . Internet of Everything ( IoE ) expands quickly to interconnect various devices . The systematic planning , modeling , and control of IoE can bring many benefits to society ( Li et al. , 2020 ) . However , it remains an open question on how to efficiently model various grids with different levels of system information . For example , cyber-physical systems may be partially traceable by physical laws with limited sensing ( Mulani et al. , 2020 ) . Such scenarios have a significant appearance nowadays on the grid edges of various physical systems due to the availability of low-cost , low-power sensor technology . These edge areas have unequal sensors but can be used to recover some physical laws completely or partially ( Divan et al. , 2014 ; Hu & Tang , 2020 ) . Learning physical equations from data is a central topic of Artificial Intelligence ( Sahoo et al. , 2018 ; Udrescu & Tegmark , 2020 ) . While there are many related studies , e.g. , symbolic regression ( Petersen , 2019 ) and its variation with sparsity regularization ( Brunton et al. , 2016b ) , these methods are inapplicable to IoE or other complex physical systems due to the incomplete system observability . To tackle the issue , ( Li & Weng , 2021 ) builds a shallow-deep structure to learn physics in a shallow neural network and simultaneously approximate the hidden components in a deep NN . Such a method , however , doesn ’ t give a clear boundary between two NNs and suffers risks of imbalanced representation power between the physical and the virtual . Thus , there is a need to enforce restrictions for the model output within the boundary and keep the physical consistency . Further , existing methods can hardly guarantee that the extracted physical model is operationally optimal . For example , in the realistic operation , if the data of an observed node can linearly represent those of an unobserved node , these two nodes can be aggregated , leading to the so-called Network Reduction ( NR ) ( Oh , 2012 ; Zheng et al. , 2021 ) . On the other hand , some components of hidden quantities that can ’ t be represented via the observed data can be treated as noise . Thus , our target is to identify an optimal reduced grid with operationally maximal physics . Namely , the reduced grid should fully represent the input-output relationship of the observed measurements under a certain noise level . Based on the above observations , we propose Adversarial Twin NNs ( ATN ) for optimal system modeling , where a Physical Neural Network ( PNN ) represents the physical parameters of the reduced grid and a Virtual Neural Network ( VNN ) approximates the noise . To find the proper output boundary , we first restrict the PNN using sparse regularization . Simultaneously , we encourage PNN to approximate the final output with priority , indirectly restricting the output of VNN . We show that such a mechanism can be easily achieved via a skip-connection ( He et al. , 2016b ) . Secondly , to achieve physical knowledge maximization , the output of PNN should be independent of the noise output of the VNN . Thus , we propose an adversarial learning scheme with Embedding Neural Networks ( ENNs ) to extract similar features from outputs of PNN and VNN . Adversarially , training PNN and VNN leads to maximized output dissimilarity . Notably , such an idea is the reverse thinking of traditional deep learning-based adversarial learning that seeks maximal similarity between two targets ( e.g. , feature distributions ) . Namely , we aim to find the maximized feature dissimilarity of two NNs . Under this condition , classical training loss like binary loss in Generative Adversarial Networks ( GANs ) easily suffers instability . Thus , we address the problem using the similarity and dissimilarity measures of contrastive learning ( Hadsell et al. , 2006 ) . Finally , we conduct extensive experiments over various systems to demonstrate the much better performance of our model compared to other state-of-the-art methods . 2 RELATED WORK . Physical System Identification . Physical system identification is a central topic for modern physical systems , especially in the edge areas . The target is to identify system governing equations using sensor measurements ( Li & Weng , 2021 ) . While the problem is similar to learning underlying equations from data , the key challenge for physical system identification is the incomplete measurement availability due to the sensor cost in a wide area of the system . Thus , traditional methods of learning equations using symbolic regression ( Schmidt & Lipson , 2009 ; Petersen , 2019 ) can easily suffer overfitting . Then , ( Brunton et al. , 2016c ; Champion et al. , 2019 ) assume that the system is sparse and utilize Least Absolute Shrinkage and Selection Operator ( LASSO ) regularization to restrict physical equation representation and avoid overfitting . This regularization , however , causes inaccuracy to learn physical parameters due to the penalty term . Thus , ( Li & Weng , 2021 ) proposes a deep-shallow architecture to learn physics with a shallow neural network as well as approximating the hidden components with a deep neural network . However , the DNN model easily suffers from local optima , making the output boundary between the shallow neural network and the DNN deviate from the true boundary . Consequently , the recovery accuracy of the physical parameters in the shallow neural network deteriorates . To restrict the output boundary , ( Yin et al. , 2020 ) also assumes physical and augmented neural networks and propose a l2 norm to regularize the parameters of the augmented NN . However , such regularization may easily cause overfitting of the physical NN . Finally , ( Takeishi & Kalousis , 2021 ) can restricts both the physical and the virtual NNs . However , their regularization requires strict assumptions on variable distributions and the prior physical knowledge , which may not be available for general physical systems . Adversarial Learning . Adversarial learning is a popular approach for training two adversarial components in a NN . Primarily , the mini-max game training helps to achieve the Nash equilibrium for optimal policy . For example , Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ; Sauder & Sievers , 2019 ) utilizes a generator to generate fake data and , adversarially , utilize a discriminator to distinguish between the fake data and the true data . The optimal status is to enable the generator to accurately approximate the distribution of the true data . Such an idea is further utilized in various machine learning domains like domain adaptation ( Ganin et al. , 2016 ) to extract similar features between two domains and disentangled representation learning ( Tran et al. , 2017 ) . Contrastive Learning . Contrastive learning seeks representation with a minimal distance of similar samples and maximal distance of dissimilar samples ( Hadsell et al. , 2006 ) . Usually , the distance is measured in an embedding space to seek the best embedding . The elaborated contrastive loss can thus guarantee stable training due to the soft comparisons between the positive and the negative samples . Such a technique is widely utilized in image embedding ( Park et al. , 2020 ) , feature clustering ( Li et al. , 2021c ) , text recognition ( Aberdam et al. , 2021 ) , etc . 3 METHODS . 3.1 PROBLEM FORMULATION . In this paper . we study physical systems that can be modeled as a directed weighted graph G = { V , E } , where V represents the vertex ( node ) set , and E ⊆ V × V represents the edge set . All nodes in V have physical variables x and y , where x represents the system state variables , and y represents the system net outputs . Then , the system equations can be written as y = f ( x ) . For example , in electric grids , x denotes the voltage phasor , and y denotes the net power ( i.e. , the power consumption minus the power injection ) . Then , f can represent the power flow equation ( Yu et al. , 2017 ) . Parameters in f are usually ( partially ) unknown to us due to the growing system size , evolving system properties , events , and maintenance , etc . Thus , it ’ s essential to estimate parameters of f using data of x and y . However , not all nodes in V can be equipped with sensors due to the sensor cost . Thus , we denote V = O ∪ U , where O represents the observable nodes , and U represents the unobservable nodes . The physical function can then be written as [ yO , yU ] > = f ( [ xO , xU ] > ) , where > is the transpose operation . Subsequently , we denote the observed data samples as { xnO } Nn=1 and { ynO } Nn=1 , whereN is the number of samples . Based on the above measurements , we aim to find an accurate mapping gθ such that yO ≈ gθ ( xO ) . Further , a subset of the parameters θp ⊂ θ should represent the physical parameters in f as many as possible . The missing quantities of xU and yU make the problem challenging as the under-estimate or overestimate of xU and yU will easily cause the inaccurate results of θp and further hurt the generalization ability of gθ . Subsequently , this causes negative impacts on the downstream tasks like system resource optimization , reliability evaluation , and optimal expansion . To solve the above problem , we propose twin NNs to approximate both of them . As shown in Fig . 1 , the twin NNs try to map from xO to yO with physics consistency , where Physical Neural Network ( PNN ) learns physical parameters and outputs physical quantities and Virtual Neural Network ( VNN ) approximates the remaining quantities , i.e. , the uncertainties and hidden measurements . Thus , they jointly contribute to the final output yO . Before illustrating the design details , we first introduce a moderate assumption for the system prior . Specifically , we assume the physical bases z are known in the physical equation . Namely , there is a mapping z = φ ( x ) such that y = Az , where A is a constant matrix representing system parameters . Then , all the system non-linearity is incorporated in φ and A contains all physical information and needs to be estimated . If we know all nodes ’ measurements , a linear regression can help to identify A . However , we only have measurements of xO and yO . Correspondingly , we denote z = [ zO , zU ] > , where zO are physical bases purely calculated via xO and zU are the bases calculated via xU or [ xO , xU ] > . Thus , we fix the parameters from xO to zO and place the learnable parameters in the mapping from zO to yO . Namely , we build a physical library , as shown in the left part of Fig . 1 . Note that for some complex systems , the mapping φ may not be known , and we may demand symbolic regression-based methods to generate the base symbols first ( Petersen , 2019 ) . However , in this paper , we focus on the issue of incomplete system observability with the prior knowledge of φ , e.g. , the quadratic and sinusoidal functions in an electric grid .

Summary: The task of identifying a physical system on a graph is addressed. While the main part of the to-be-estimated model is assumed to be linear, the proposed model needs a nonlinear part (which is modeled by a neural net) due to the presence of unobserved nodes. The authors use a combination of a sparse linear model and a neural net, which is basically the same as the model in [Li & Weng, KDD 2021]. They consider some regularization terms to maximize the use of the physics part of the model. They examine the performance of the proposed method on several datasets.