Adversarial twin neural networks: maximizing physics recovery for physical system

1 INTRODUCTION . Internet of Everything ( IoE ) expands quickly to interconnect various devices . The systematic planning , modeling , and control of IoE can bring many benefits to society ( Li et al. , 2020 ) . However , it remains an open question on how to efficiently model various grids with different levels of system information . For example , cyber-physical systems may be partially traceable by physical laws with limited sensing ( Mulani et al. , 2020 ) . Such scenarios have a significant appearance nowadays on the grid edges of various physical systems due to the availability of low-cost , low-power sensor technology . These edge areas have unequal sensors but can be used to recover some physical laws completely or partially ( Divan et al. , 2014 ; Hu & Tang , 2020 ) . Learning physical equations from data is a central topic of Artificial Intelligence ( Sahoo et al. , 2018 ; Udrescu & Tegmark , 2020 ) . While there are many related studies , e.g. , symbolic regression ( Petersen , 2019 ) and its variation with sparsity regularization ( Brunton et al. , 2016b ) , these methods are inapplicable to IoE or other complex physical systems due to the incomplete system observability . To tackle the issue , ( Li & Weng , 2021 ) builds a shallow-deep structure to learn physics in a shallow neural network and simultaneously approximate the hidden components in a deep NN . Such a method , however , doesn ’ t give a clear boundary between two NNs and suffers risks of imbalanced representation power between the physical and the virtual . Thus , there is a need to enforce restrictions for the model output within the boundary and keep the physical consistency . Further , existing methods can hardly guarantee that the extracted physical model is operationally optimal . For example , in the realistic operation , if the data of an observed node can linearly represent those of an unobserved node , these two nodes can be aggregated , leading to the so-called Network Reduction ( NR ) ( Oh , 2012 ; Zheng et al. , 2021 ) . On the other hand , some components of hidden quantities that can ’ t be represented via the observed data can be treated as noise . Thus , our target is to identify an optimal reduced grid with operationally maximal physics . Namely , the reduced grid should fully represent the input-output relationship of the observed measurements under a certain noise level . Based on the above observations , we propose Adversarial Twin NNs ( ATN ) for optimal system modeling , where a Physical Neural Network ( PNN ) represents the physical parameters of the reduced grid and a Virtual Neural Network ( VNN ) approximates the noise . To find the proper output boundary , we first restrict the PNN using sparse regularization . Simultaneously , we encourage PNN to approximate the final output with priority , indirectly restricting the output of VNN . We show that such a mechanism can be easily achieved via a skip-connection ( He et al. , 2016b ) . Secondly , to achieve physical knowledge maximization , the output of PNN should be independent of the noise output of the VNN . Thus , we propose an adversarial learning scheme with Embedding Neural Networks ( ENNs ) to extract similar features from outputs of PNN and VNN . Adversarially , training PNN and VNN leads to maximized output dissimilarity . Notably , such an idea is the reverse thinking of traditional deep learning-based adversarial learning that seeks maximal similarity between two targets ( e.g. , feature distributions ) . Namely , we aim to find the maximized feature dissimilarity of two NNs . Under this condition , classical training loss like binary loss in Generative Adversarial Networks ( GANs ) easily suffers instability . Thus , we address the problem using the similarity and dissimilarity measures of contrastive learning ( Hadsell et al. , 2006 ) . Finally , we conduct extensive experiments over various systems to demonstrate the much better performance of our model compared to other state-of-the-art methods . 2 RELATED WORK . Physical System Identification . Physical system identification is a central topic for modern physical systems , especially in the edge areas . The target is to identify system governing equations using sensor measurements ( Li & Weng , 2021 ) . While the problem is similar to learning underlying equations from data , the key challenge for physical system identification is the incomplete measurement availability due to the sensor cost in a wide area of the system . Thus , traditional methods of learning equations using symbolic regression ( Schmidt & Lipson , 2009 ; Petersen , 2019 ) can easily suffer overfitting . Then , ( Brunton et al. , 2016c ; Champion et al. , 2019 ) assume that the system is sparse and utilize Least Absolute Shrinkage and Selection Operator ( LASSO ) regularization to restrict physical equation representation and avoid overfitting . This regularization , however , causes inaccuracy to learn physical parameters due to the penalty term . Thus , ( Li & Weng , 2021 ) proposes a deep-shallow architecture to learn physics with a shallow neural network as well as approximating the hidden components with a deep neural network . However , the DNN model easily suffers from local optima , making the output boundary between the shallow neural network and the DNN deviate from the true boundary . Consequently , the recovery accuracy of the physical parameters in the shallow neural network deteriorates . To restrict the output boundary , ( Yin et al. , 2020 ) also assumes physical and augmented neural networks and propose a l2 norm to regularize the parameters of the augmented NN . However , such regularization may easily cause overfitting of the physical NN . Finally , ( Takeishi & Kalousis , 2021 ) can restricts both the physical and the virtual NNs . However , their regularization requires strict assumptions on variable distributions and the prior physical knowledge , which may not be available for general physical systems . Adversarial Learning . Adversarial learning is a popular approach for training two adversarial components in a NN . Primarily , the mini-max game training helps to achieve the Nash equilibrium for optimal policy . For example , Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ; Sauder & Sievers , 2019 ) utilizes a generator to generate fake data and , adversarially , utilize a discriminator to distinguish between the fake data and the true data . The optimal status is to enable the generator to accurately approximate the distribution of the true data . Such an idea is further utilized in various machine learning domains like domain adaptation ( Ganin et al. , 2016 ) to extract similar features between two domains and disentangled representation learning ( Tran et al. , 2017 ) . Contrastive Learning . Contrastive learning seeks representation with a minimal distance of similar samples and maximal distance of dissimilar samples ( Hadsell et al. , 2006 ) . Usually , the distance is measured in an embedding space to seek the best embedding . The elaborated contrastive loss can thus guarantee stable training due to the soft comparisons between the positive and the negative samples . Such a technique is widely utilized in image embedding ( Park et al. , 2020 ) , feature clustering ( Li et al. , 2021c ) , text recognition ( Aberdam et al. , 2021 ) , etc . 3 METHODS . 3.1 PROBLEM FORMULATION . In this paper . we study physical systems that can be modeled as a directed weighted graph G = { V , E } , where V represents the vertex ( node ) set , and E ⊆ V × V represents the edge set . All nodes in V have physical variables x and y , where x represents the system state variables , and y represents the system net outputs . Then , the system equations can be written as y = f ( x ) . For example , in electric grids , x denotes the voltage phasor , and y denotes the net power ( i.e. , the power consumption minus the power injection ) . Then , f can represent the power flow equation ( Yu et al. , 2017 ) . Parameters in f are usually ( partially ) unknown to us due to the growing system size , evolving system properties , events , and maintenance , etc . Thus , it ’ s essential to estimate parameters of f using data of x and y . However , not all nodes in V can be equipped with sensors due to the sensor cost . Thus , we denote V = O ∪ U , where O represents the observable nodes , and U represents the unobservable nodes . The physical function can then be written as [ yO , yU ] > = f ( [ xO , xU ] > ) , where > is the transpose operation . Subsequently , we denote the observed data samples as { xnO } Nn=1 and { ynO } Nn=1 , whereN is the number of samples . Based on the above measurements , we aim to find an accurate mapping gθ such that yO ≈ gθ ( xO ) . Further , a subset of the parameters θp ⊂ θ should represent the physical parameters in f as many as possible . The missing quantities of xU and yU make the problem challenging as the under-estimate or overestimate of xU and yU will easily cause the inaccurate results of θp and further hurt the generalization ability of gθ . Subsequently , this causes negative impacts on the downstream tasks like system resource optimization , reliability evaluation , and optimal expansion . To solve the above problem , we propose twin NNs to approximate both of them . As shown in Fig . 1 , the twin NNs try to map from xO to yO with physics consistency , where Physical Neural Network ( PNN ) learns physical parameters and outputs physical quantities and Virtual Neural Network ( VNN ) approximates the remaining quantities , i.e. , the uncertainties and hidden measurements . Thus , they jointly contribute to the final output yO . Before illustrating the design details , we first introduce a moderate assumption for the system prior . Specifically , we assume the physical bases z are known in the physical equation . Namely , there is a mapping z = φ ( x ) such that y = Az , where A is a constant matrix representing system parameters . Then , all the system non-linearity is incorporated in φ and A contains all physical information and needs to be estimated . If we know all nodes ’ measurements , a linear regression can help to identify A . However , we only have measurements of xO and yO . Correspondingly , we denote z = [ zO , zU ] > , where zO are physical bases purely calculated via xO and zU are the bases calculated via xU or [ xO , xU ] > . Thus , we fix the parameters from xO to zO and place the learnable parameters in the mapping from zO to yO . Namely , we build a physical library , as shown in the left part of Fig . 1 . Note that for some complex systems , the mapping φ may not be known , and we may demand symbolic regression-based methods to generate the base symbols first ( Petersen , 2019 ) . However , in this paper , we focus on the issue of incomplete system observability with the prior knowledge of φ , e.g. , the quadratic and sinusoidal functions in an electric grid .

Summary: The paper proposes ATN to model and identify physical systems. Here, the physical systems are grids consisting of measurement sensors: because the sensor configuration is not perfect, there can be unobservable parts in the grid system. The authors assume that there are physics bases that can represent the output with a linear mapping. Thus, they firstly use a linear LASSO regressor (PNN in the paper) that transforms observable physics bases to the output. The possible remainder (due to the noise or unobservable physics) is modeled via a more complicated neural network (VNN in the paper). These two networks are integrated into a single skip-connection block. They are trained by MSE, with enforcing the PNN part predict the output mostly, to exploit the bias of physics. In addition, the authors suggest the use of siamese networks that measure the distance between PNN and VNN: then an adversarial learning loss is added to guarantee the outputs of PNN and VNN to be far from each other and play their respective roles more concisely. ATN outperforms its counterparts for some physical system datasets.