Adversarial twin neural networks: maximizing physics recovery for physical system
Authors: Haoran Li
Erik Blasch
Jingyi Yuan
Yang Weng
Venue: ICLR 2022 Submitted (2022)
Keywords: Physical Equation Learning
Incomplete Observability
Twin Neural Network
Mini-Max Game

Abstract: The exact modeling of modern physical systems is challenging due to the expanding system territory and insufficient sensors. To tackle this problem, existing methods utilize sparse regression to find physical parameters or add another virtual learning model like a Neural Network (NN) to universally approximate the unobserved physical quantities. However, the two models can't perfectly play their own roles in joint learning without proper restrictions. Thus, we propose (1) sparsity regularization for the physical model and (2) physical superiority over the virtual model. They together define output boundaries for the physical and virtual models. Further, even the two models output properly, the joint model still can't guarantee learning maximal physical knowledge. For example, if the data of an observed node can linearly represent those of an unobserved node, these two nodes can be aggregated. Therefore, we propose (3) to seek the dissimilarity of physical and virtual outputs to obtain maximal physics. To achieve goals (1)-(3), we design a twin structure of the Physical Neural Network (PNN) and Virtual Neural Network (VNN), where sparse regularization and skip-connections are utilized to guarantee (1) and (2). Then, we propose an adversarial learning scheme to maximize output dissimilarity, achieving (3). We denote the model as the Adversarial Twin Neural Network (ATN).  Finally, we conduct extensive experiments over various systems to demonstrate the best performance of ATN over other state-of-the-art methods.

Under review as a conference paper at ICLR 2022
ADVERSARIAL TWIN NEURAL NETWORKS: MAXIMIZ- ING PHYSICS RECOVERY FOR PHYSICAL SYSTEM
Anonymous authors Paper under double-blind review
ABSTRACT
The exact modeling of modern physical systems is challenging due to the ex- panding system territory and insufficient sensors. To tackle this problem, existing methods utilize sparse regression to find physical parameters or add another vir- tual learning model like Neural Network (NN) to universally approximate the unobserved physical quantities. However, the two models can't perfectly play their own roles in joint learning without proper restrictions. Thus, we propose (1) sparsity regularization for the physical model and (2) physical superiority over the virtual model. They together define output boundaries for the physical and virtual models. Further, even the two models output properly, the joint model still can't 't guarantee learning maximal physical knowledge. For example, if the data of an observed node can linearly represent those of an unobserved node, these two nodes can be aggregated. Therefore, we propose (3) to seek the dissimilar- ity of physical and virtual outputs to obtain maximal physics. To achieve goals (1) (3), we design twin structure of the Physical Neural Network (PNN) and Virtual Neural Network (VNN), where sparse regularization and skip-connections are utilized to guarantee (1) and (2). Then, we propose an adversarial learning scheme to maximize output dissimilarity, achieving (3). We denote the model as the Adversarial Twin Neural Network (ATN). Finally, we conduct extensive ex- periments various systems demonstrate the best performance of ATN over other state-of-the-ar methods.
INTRODUCTION
Internet of Everything (loF) expands quickly to interconnect various devices. The systematic plan- ning, modeling, and control of loE can bring many benefits to society (Li et al.) 2020). However, it remains an open question on how to efficiently model various grids with different levels of sys- tem information. For example, cyber-physical systems may be partially traceable by physical laws with limited sensing (Mulani et al. 2020). Such scenarios have a significant appearance nowadays on the grid edges of various physical systems due to the availability of low-cost, low-power sensor technology. These edge areas have unequal sensors but can be used to recover some physical laws completely or partially (Divan al. 2014 Hu & Tang 2020).
Learning physical equations from data is a central topic of Artificial Intelligence (Sahoo et al. 2018 Udrescu Tegmark 2020)). While there are many related studies, e.g., symbolic regression (Pe- tersen 2019) and its variation with sparsity regularization (Brunton et al. 2016b), these methods inapplicable IoE or other complex physical systems due the incomplete system observ- ability. To tackle the issue, (Li & Weng 2021) builds shallow-deep structure to learn physics in shallow neural network and simultaneously approximate the hidden components in a deep NN. Such method, however, doesn't 't give a clear boundary between two NNs and suffers risks of im- balanced representation power between the physical and the virtual. Thus, there a need to enforce restrictions for the model output within the boundary and keep the physical consistency.
Further, existing methods can hardly guarantee that the extracted physical model is operationally optimal. For example, in the realistic operation, if the data of an observed node can linearly represent those an unobserved node, these two nodes can be aggregated, leading to the so-called Network Reduction (NR) (Ohl 2012 Zheng et al., 2021). On the other hand, some components of hidden quantities that can't be represented via the observed data can be treated as noise. Thus, our target is to identify optimal reduced grid with operationally maximal physics. Namely, the reduced grid

---

Under review as a conference paper at ICLR 2022
should fully represent the input-output relationship of the observed measurements under certain noise level.
Based on the above observations, we propose Adversarial Twin NNs (ATN) for optimal system mod- cling, where a Physical Neural Network (PNN) represents the physical parameters of the reduced grid and Virtual Neural Network (VNN) approximates the noise. To find the proper output bound- ary, we first restrict the PNN using sparse regularization. Simultaneously, we encourage PNN to approximate the final output with priority, indirectly restricting the output of VNN. We show that such a mechanism can be easily achieved via a skip-connection (He et al.) 2016b). Secondly, to achieve physical knowledge maximization, the output of PNN should be independent of the noise output of the VNN. Thus, we propose an adversarial learning scheme with Embedding Neural Net- works (ENNs) to extract similar features from outputs of PNN and VNN. Adversarially, training PNN and VNN leads to maximized output dissimilarity.
Notably, such an idea is the reverse thinking of traditional deep sarming-basedd adversarial learning that seeks maximal similarity between two targets (e.g., feature distributions). Namely, we aim to find the maximized feature dissimilarity of two NNs. Under this condition, classical training loss like binary loss in Generative Adversarial Networks (GANs) easily suffers instability. Thus, we address the problem using the similarity and dissimilarity measures of contrastive learning (Hadsell et al. 2006). Finally, we conduct extensive experiments over various systems to demonstrate the much better performance of our model compared to other state-of-the-ar methods.
RELATED WORK
Physical System Identification. Physical system identification a central topic for modern phys- ical systems, especially in the edge areas. The target is to identify system governing equations using sensor measurements (Li & Weng 202T). While the problem is similar to learning underlying equations from data, the key challenge for physical system identification is the incomplete measure- ment availability due to the sensor cost in a wide area of the system. Thus, traditional methods of
learning equations using symbolic regression (Schmidt & Lipson 2009 Petersen 2019) can easily suffer overfitting. Then, (Brunton et al. 2016c) Champion et al., 2019) assume that the system is sparse and utilize Least Absolute Shrinkage and Selection Operator (LASSO) regularization to re- strict physical equation representation and avoid overfitting. This regularization, however, causes inaccuracy to learn physical parameters due to the penalty term.
Thus, Li & Weng 2021) proposes deep-shallow architecture to learn physics with shallow neu- ral network as well as approximating the hidden components with a deep neural network. However, the DNN model easily suffers from local optima, making the output boundary between the shallow neural network and the DNN deviate from the true boundary. Consequently, the recovery accuracy of the physical parameters in the shallow neural network deteriorates. To restrict the output bound- ary, (Yin et al. 2020) also assumes physical and augmented neural networks and propose a lz norm to regularize the parameters of the augmented NN. However, such regularization may easily cause overfitting of the physical NN. Finally, (Takeishi & Kalousis] 202T) can restricts both the physical and the virtual NNs. However, their regularization requires strict assumptions on variable distribu- tions and the prior physical knowledge, which may not be available for general physical systems.
Adversarial Learning. Adversarial learning is a popular approach for training two adversarial components in a NN. Primarily, the mini-max game training helps to achieve the Nash equilibrium for optimal policy. For example, Generative Adversarial Network (GAN) (Goodfellow al. 2014] Sauder & Sievers 2019) utilizes generator to generate fake data and, adversarially, utilize a dis- criminator to distinguish between the fake data and the true data. The optimal status is to enable the generator to accurately approximate the distribution of the true data. Such an idea is further utilized in various machine learning domains like domain adaptation (Ganin et al.) 2016) to extract similar features between two domains and disentangled representation learning (Tran et al. 2017).
Contrastive Learning. Contrastive learning seeks representation with a minimal distance of similar samples and maximal distance of dissimilar samples (Hadsell et al. 2006). Usually, the distance is measured in an embedding space to seek the best embedding. The elaborated contrastive loss can thus guarantee stable training due to the soft comparisons between the positive and the negative sam- plcs. Such a technique is widely utilized in image embedding (Park et al. 2020). feature clustering (Li al. 2021c), text recognition (Aberdam et al. 2021). etc.
2

---

Under review as a conference paper at ICLR 2022
3 METHODS
3.1 PROBLEM FORMULATION
In this paper. we study physical systems that can modeled as a directed weighted graph g {V.E), where V represents the vertex (node) set. and & S V V represents the edge set. All nodes in V have physical variables a and y. where a represents the system state variables, and y represents the system net outputs. Then, the system equations can be written as y = f()). For example, in electric grids, a denotes the voltage phasor, and denotes the net power (i.e., the power consumption minus the power injection). Then, f can represent the power flow equation (Yu et al.) 2017). Parameters in f are usually (partially) unknown to us due to the growing system size, evolving system properties, events, and maintenance, etc. Thus, it's essential to estimate parameters of f using data of x and y.
However, not all nodes in V can be equipped with sensors due the sensor cost. Thus, we denote V = O UU, where O represents the observable nodes, and U represents the unobservable nodes. The physical function can then be written as [yo, yu] = f [[ au]] where T the transpose operation. Subsequently, we denote the observed data samples as { J and {yb) > where N is the number of samples. Based on the above measurements, we aim to find accurate mapping ge such that yo z go(xo). Further, a subset of the parameters 0p C O should represent the physical parameters in f as many as possible.
The missing quantities of Ty and yuu make the problem challenging as the under-estimate or over- estimate of zu and yu will easily cause the inaccurate results of 0, and further hurt the generaliza- tion ability of ge. Subsequently, this causes negative impacts on the downstream tasks like system resource optimization, reliability evaluation, and optimal expansion.
To solve the above problem, we propose twin NNs to approximate both of them. As shown in Fig. [ the twin NNs try to map from æo to yo with physics consistency, where Physical Neu- ral Network (PNN) learns physical parameters and outputs physical quantities and Virtual Neural Network (VNN) approximates the remaining quantities, i.e., the uncertainties and hidden measure- ments. Thus, they jointly contribute to the final output yo.
Before illustrating the design details, we first introduce a moderate assumption for the system prior. Specifically, we assume the physical bases = are known in the physical equation. Namely, there mapping z = Φ(x) such that y Az, where is a constant matrix representing system parameters. Then, all the system non-linyyyity is incorporated in & and A contains all physical information and needs to be estimated. If we know all nodes' measurements, a linear regression can help to identify A. However, we only have measurements of æo and yo. Correspondingly, we denote z = [=o, =u]] where zo are physical bases purely calculated via ao and zy are the bases calculated via Tu or [xO æzus" Thus, we fix the parameters from zo zo and place the learnable parameters in the mapping from zo to yo. Namely, we build a physical library, as shown in the left part of Fig. [ Note that for some complex systems, the mapping á may not be known, and we may demand symbolic regression-basee
methods to generate the base symbols first (Petersen) 2019). However, in this paper, we focus on the issue of incomplete system observability with the prior knowledge of o, c.g., the quadratic and sinusoidal functions in an electric grid.
3.2 PHYSICAL NEURAL NETWORK (PNN): LEARN PHYSICS WITH SPARSE REGULARIZATION
Due to the linear relationship between zo and yo. we introduce linear mapping An such that h = (20) = Wizo in PNN. Usually, the system structure is sparse (Brunton et al. 2016c Champion et al. 2019). To maintain the physical consistency, we enforce a sparse regularization, i.e., the LASSO term, to guarantee sparse connections of the system. Specifically, the LASSO
regularization is created as ||Will1 Eis [i] where Wi [ is the (1,j)th element in W1.
3.3 VIRTUAL NEURAL NETWORK (VNN): UNIVERSAL. APPROXIMATION UNDER PHYSICAL SUPERIORITY
After showing the regularized PNN, this section will explain the modeling and benefits of VNN. The goal for VNN is to (1) approximate the remaining quantities, and (2) avoid deteriorating the output of PNN. Namely, we want to find a boundary to let PNN and VNN output properly. For the first goal, we build VNN a fully connected neural network with multiple layers, which has high capacity to approximate the remaining quantities. Specifically, let the mapping of the VNN be go. and we have ha4+1 gu (h1), where M is the number of layers for VNN.
3

---

Under review as a conference paper at ICLR 2022
PNN: Learn Physical Knowledge
VNN: Effective Complementation
Physical Library
LASSO Regularization
Skip for Physics
X2
2/2
X3
ENNs: Extract Common Features for Similarity
Similarity Loss Design for Comparison
Similarity Loss
Prediction Loss
Figure The structure of the proposed twin NNs.
For the second goal of finding the boundary, we need to restrict the output VNN and encourage PNN output much as possible. On the other hand, we do not worry about the overfitting of the PNN as PNN has been strictly regularized via LASSO regularization to guarantee physical consistency. Thus, we employ skip-connection from the output of PNN directly to the final output (He et al. 2a). Mathematically, we have: yo h1 + ge = 9p(zo) + Io (gp(zz)).
The skip-connection further ensures the possibility to create physical superiority, i.e. to encourage PNN to output much as possible. Namely, we encourage 9p (=o) to directly approximate yo. In general, we have the following objective for the twin NNs.
min (yo 100)pp(20)>> - Do Sn (o(((()))) - Ti (yo 3 + Al[Wi|h,
(1)
where 7 and À are positive constants for the penalty terms.
3.4 MINI-MAX GAME TO ADVERSARIALLY TRAIN TWIN NNS FOR MAXIMIZED PHYSICAL KNOWLEDGE
In this subsection, we propose the Adversarial Twin NNs (ATN) based on the above twin NNs. Primarily, we show that ATN helps recover maximized physics, which is much better than previous work for physical system modeling.
As we mentioned in the Introduction Section, even if the boundary between the observed and unob- served measurements is successfully found, the physics may not be maximized. Essentially, we aim to find reduced grid to govern the model and operation of observed measurements. Mathemati- cally, we can treat the hidden components as zμ Bzo + e, where another constant matrix for the linear representation of the unobservable parts using the observable parts, and e is the noise term
Based on the above formulation, the maximized physics can only be achieved when (=o) rep- resents the linear relationships from both zo and zu ze Bzo, and ge(zo) represents the output transformed from the noise term This inspires clear probabilistic distance between gp (20) and ge (zo). Specifically, we aim to maximize the distribution dissimilarity between gp (x() and (x())), thus achieving maximized physics. As distribution dissimilarity hardly be explicitly
4

---

During the training. we denote Y 0 for the maximization process VNN to maximize the distance in Equation 2 Conversely, we denote Y = in the minimization process to minimize the defined distance. The adversarial training process finally leads to the maxi- Under review as a conference paper at ICLR 2022
written in neural network framework, we propose adversarial learning framework to achieve the goal.
Specifically, we intend to drive the training process to gradually convert physical knowledge from VNN to PNN. which needs to measure and transfer the common knowledge. Thus, we utilize a third neural network, Embedding Neural Network (ENN), transform the output of PNN and VNN into embedding vectors with large common knowledge, i.e., similarity of embedding distributions. On the other hand, our PNN and VNN should dynamically adjust the parameters achieve minimal similarity. In general, the complete framework is shown in Fig. D where two ENNs convert the original output of PNN and VNN for the similarity loss in mini-max game. Since we have the given output pair from PNN and VNN. the task of finding the similarity of two embeddings can be formalized as a contrastive learning framework (Hadsell al. 2006). Thus, we define the similarity distance as follows:
Dae, (e
Jes (go (9p( = - (10 (op(zz))) ge2 902
where ge, and Je, represent the mappings of two ENNs, respectively. Namely, we utilize the Euclidean distance in the embedding space to measure the distribution similarity. Based on Equation we subsequently define the similarity loss for training as follows:
L(O., Op, O., Y) (1- = Des (gp(zo)) Ses (gn (pp(()))))
(3)
+Y( mmxx(0, m - D D(9 (gp Joz (gr (8p(2o)))
where Or represent the set of parameters in Ge, and gesr and O and θ, represent the parameter sets for PNN and VNN, respectively. Y is scalar to indicate if the sample pair is similar (Y 0)
or dissimilar (Y = 1), and m is threshold to achieve stability in contrastive learning.
With the loss function above, derive the the value function for the mini max game between PNN and VNN:
min max V(@.,, Op. O,) L(O,,, Op. Oz Y) - Bi(o 9p(21)) @ 8..9,
(4)
-El -g((1)) Do (g) - - AI
want to train PNN and
mized dissimilarity between outputs of PNN and VNN, which implies that the physical knowledge is maximally represented in PNN. Since we do the adversarial training, denote model as Adversarial Twin NNs (ATN).
4 EXPERIMENTS
To extensively verify the performances of ATN, we elaborate our models on different datasets from different physical systems. Meanwhile, we consider testing under different system observability levels, i.e., the ratio of the number of observed nodes to the total nodes, in the set (0.2, 0.4, 0.6, 0.8} and different Aggnal-to-Noise-PP
(SNRs) in the set (60, 80, 100, 120} to mimic the realistic con- ditions. Then, we employ 8000 samples and conduct 5-fold cross-validation to obtain the average results for demonstration. For all the NN training, use Adam optimizer with a learning rate hyper- parameter set (0.001, 0.0002, 0.00005), and momentum parameters 31 = 0.5, 32 = 0.999 to train 200 epochs for each experiment. All the experiments completed with computer equipped with Intel(R) Core(T'M) 17-9700k CPU and Nvidia Geforce RTX 2080Ti GPU. The detailed experimental settings are described as follows. 4.1 DATASETS
Synthetic Data. We first utilize synthetic datasets for demonstration. Specifically, we propose 4 4-node system such that:
3/1 = x + cos(12)) y/2 = x +
(5)
= -3 ( = x² Z1.
5

---

Under review as a conference paper at ICLR 2022
The vector variable æ is simulated using uniform distribution (O, 1), and we construct the set based on the physical mappings (e.g., square or cosinusoidal.) in Equation (5)
IEEE Power Systems and PJM Load Data. There standard power system models in IEEE, including the grid nodes, edges, and attributes. Especially, we can download the model files and the simulation platform in MATPOWER (MATPOWER community 2020 In this experiment, incorporate IEEE 85-node systems for testing. To conduct the simulation, we further employ real- world power consumptions in PJM Interconnection LLC (PJM) data (PJM Interconnection LLC, 2018). The load files contain hourly electricity consumption in 2017 for the PJM RTO regions. With the above data, MATPOWER produces the system states of voltage phasor to represent the system input æ. In the meantime, the nodal net power represents the system output y for training. In the power system dataset, we have & = [ sin zr].
Mass-damper system data. We can utilize the following equation to represent the governing law of the nass-r system: q DRD M- -qq, where q is the vector of momenta of the masses, D is the incidence matrix of the graph, R is the diagonal matrix of the damping coefficients of the damper attached to the edges, and M the diagonal mass matrix (van der Schaft, 2017). Using MATLAB, we simulate the dynamic process of the mass-damper system with 10 nodes for testing. In the mass damper system dataset, we have ø = dt
de x].
UF sparse matrix- system. We can find large set of sparse baared networks in the UF sparse matrix systems (Texas A&M University 201T). In this experiment, we utilize the 274-node system test. For the convenience of later presentation, simply utilize 4-, 10-, 85-, and 27denode systems to denote the above datasets. the UF sparse matrix dataset, we have e = [x]].
4.2 BENCIMARRK MODELS
To validate the performance of learning physical system representation, the following benchmark methods from related literature are used.
Deep Residual network (Resnet). Similar to ATN, Resnet (He et al., 2016b) also utilizes the shortcut design to pass the representation of shallow layers to deeper layers. Such a design helps Resnet achieve excellent performances various machine learning tasks (Guo & Du 2019
Ma et al. 2020).
SINDY method conducts sparse regression to recover the physical parameters. Especially, in our paper, we assume the physical bases are known. Thus, the essence of SINDy is to utilize a LASSO regularization to select correct bases from the built-in library.
Sparse Identification of Nonlinear Dynamics (SINDy) (Brunton et al. 2016a).
Deep Symbolic Regression (DSR) (Petersen et al. 202T). DSR employs a risk risk-seeking policy gradient to train a reinforcement learning agent for symbolic regression.
Equation Learner (EQL*) (Sahoo et al. 2018). EQL* treat the physical bases as activation functions a neural network. Also, the operators like summation, multiplication, and division are included in the neural network. Finally, the target neural network includes sparsity constraints for symbol selections.
Physics-Consien Neural Network (PCNN) Li Weng 2021 PCNN has a deep- shallow architecture represent the physical model, which is similar ATN. However, PCNN doesn't include a skip-connection for physical superiority, and mini-max game for training.
In this paper, the hyper-pann parameerrreeters of NNs (i.e., ATN, Resnet, DSR, E and PCNN) are selected via cross-validation for the best performance.
4.3 MODEL EVALUATION
We propose the following metrics to evaluate the generalizability and the parameter estimation re- sults of different models.
Generalizability. We utilize the Mean square error (MSE) the testing set to evaluate the model performance of predicting yo.

---

Under review as a conference paper at ICLR 2022
Parameter estimation. To evaluate the parameter estimation performance, utilize the so-called normalized Total Vector Error (nTVE) (Li et al. 2021b) to evaluate the difference between the estimated L and the true physical system parameter matrix L:
L L
RTV = 100 x
|L||
4.4
MODEL GENERALIZABILITY. ATN ACHIEVES THE LOWEST MSE IN ALL CASES
WIN
0.20
Homer USR EO0-
-4 3IMD -O-D3R
Barand SENDS
0.06
1------ 100 PONN
0.049
MSE
MBE
MSE
0.03
0.2
0
60
100
120
100
120
60
80
100
120
-00
100
120
SN
SNRs
BNRs
SNFFt
(a) 4-node result.
(b) 10-node result.
(c) 85-node result.
(d) 274-node result.
Figure 2: The results of testing MSE for different physical systems.
Firstly, evaluate the model operational performance via generalizability for different grids and under different SNR levels. Specifically, we fix the number of observability to be 0.6 and vary SNRS in the set {60, 80, 100, 120). Fig. 2 demonstrates the testing MSE results for different physical systems. Clearly, we first observe that our ATN model (red line) achieves the lowest MSE for all the result points. This implies that ATN model is the most suitable model for physical system real-time operations. The following reasons help to explain the observation.
(1) ATN vs. Resnet. In ATN model, we embed physical bases learn a physical equation. However, for Resnet, no physics is learned. Due to the high generalizability of physical models in the observed regions, ATN performs much better than Resnet. (2) ATN vs. SINDy. ATN and SINDy have the ability to estimate the parameters of physical equations. However, SNIDY only utilizes LASSO as the regularization term, but ATN utilizes LASSO PNN and employs another VNN to approximate the remaining quantities. Thus, SINDY typically suffers underfitting when there are unobserved values, which deteriorates the predicting performance.
(3) ATN vs. DSR & EQL*. DSR and EQL* utilize symbolic regression to sparsely select symbols to form an equation that fits data. The missing values make the symbolic regression incapable of finding the true governing law. However, ATN can find governing equations for the reduced grid to best fit the data. (4) ATN vs. PCNN. Though PCNN also enjoys a twin structure to use shallow NN to learn physics and deep NN approximate the remaining components, there is no design of the adversarial learning. Thus, training PCNN can't find the maximal physics of the reduced grid for optimal operations.
Then, we observe that for all methods, as the SNR increases, their MSE results decrease. This shows that the lower noise level helps all methods to obtain better accuracy. Interestingly, we find that for the synthetic system, SINDy performs much better than DSR and EQL*, For the power system, SINDy has a moderately better performance. Then, for the rest systems, SINDy, DSR, and EQL* have similar performances. This is because the synthetic data has the most complex physical bases (xr x², _,,,0 x²3 cos(x)). and power system has relative lower complexity (a², sin(x)). Then, the mass-damper system and UF sparse matrix have the lowest complexity (r). Though we prepare all the physical base pools for each method, SINDY can successfully select the right base with LASSO regularization in the observed region. However, DSR and EQL with NN structure can easily select the incorrect bases and suffer from overfitting. Finally, we find that PCNN usually performs better than SINDy, DSR, and EQL*. This shows that even without a mini-max game, the twin structure can help to identify the rough boundary of the physical and virtual outputs and obtain a more accurate approximation than methods without twin structure.
7

---

Under review as a conference paper at ICLR 2022
4.5 PHYSICAL PARAMETER ESTIMATION: ATN HAS THE MOST ACCURATE RESULTS
This subsection studies the prssical-prrmmete--reeee methods. Especially, we fix the observability level to be 0.6 and SNR = 100. To make sure the ground-truth is doable for comparison, we make the nodal inputs independent of each other for each system. Thus, it avoids the case when the obtained reduced network is different from the ground-truth network, causing the impossibility for ATN method to verify results.
ability for SINDY,, DSR, EQL+, and ATN
Table demonstrates the results of nTV (TVEE(%) (TTVE(())eeeee) (mean + standard deviation) for different datasets. First, the result shows that our proposed ATN reaches the best performance due our sophisticated design of twin architecture and adversarial learning for dynamic boundary seeking. Secondly, as system size increases, SINDy, DSR, and EQL. have the quick boosting of the (%), almost proportional to the system size growth. However, for the ATN method, nL'V (X%) remains lower with only slight growth. This is because the increasing system size largely promotes the number of unobserved nodes, given fixed observability issues.
Thus, for SINDy, DSR, and EQL*, the probability of the perfect recovery is largely decreasing for each scalar function (i.e., function to represent yoli] for the ht element in yo). Specifically, recovering yoli needs all the input variables in the function of yoli]. However, more unobservy crrors for SINDy, DSR, and EQL * increase largely. As for PCNN method, the rate of error increas- ing is not as high as SINDy, DSR, and EQL* methods, which shows that the deep neural network in the PCNN can restrict the overfitting of the physical learner with joint training.
However, for ATN method, though the chance for perfect recovery decreases, our VNN can help identify the boundary between the observed physics and the remaining components under the mini-max game mechanism. Therefore, ATN can still have relatively small estimation errors for different systems.
Table 1: The nTV ((T) (mean + standard deviation) value for different methods in different sys- tems.
ATN
SINDY
DSR
EQL
PCNN
4-NODE SYSTEM
2.21 + 0.68
8.67 + 1.69
16.49 + 3.81
14.25 + 3.62
7.66 + 2.01
10-NODE SYSTEM
4.33 2.08
14.99 4.22
16.87 1 4.35
17.73 1 4.21
10.97 4.19
85-NODE SYSTEM
5.97 + 2.32
34.17 1 8.66
39.33 1 7.46
42.21
1 5.69
26.00 6.23
274-NODE SYSTEM
6.86 + 2.56
54.70 10.02
58.29 11.25
52.17 1 10.99
33.25 1 8.19
4.6 ABLATION STUDY FOR DIFFERENT COMPONENTS OF ATN
In this subsection, we conduct the ablation study for tha ATN model to test the performance of different components. Especially, we remove the sparsity of PNN, skip-connection of VNN, and the minimax-training respectively. We denote the models as ATN-SP, ATN-SK, ATN-MI, respectively. The results are shown in Table Averagely, for the above three models, the nTV (TVE(() (()) increases by 31.85%. 26.58%. and 20.21% %. This shows that the three components have great impacts on learning the correct physics. Further, sparsity plays the essential role as it restricts the representation of the physics.
Table 2: The nTVV (())(())mma))) + standard deviation) value for the ablation study.
ATN
ATN-SP
ANT-SK
ATN-MI
4-NODE SYSTEM
2.21 + 0.68
3.02 002:0.60
0.88
2.60 = 0.72
10- JO-NODE SYSTEM
4.33 2.08
5.161
+355
4.89 4.21
85-NODE SYSTEM
5.97 2.32
7:01+3.41
7.28 3.16
7.04 + 2.19
274-NODE SYSTEM
6.86 + 2.56
9.47 +
8.77 + 2.89
8

---

Under review as a conference paper at ICLR 2022
4.7 SENSITIVITY ANALYSIS CONCERNING THE SYSTEM OBSERVABILITY AND SAMPLE NUMBERS
In this subsection, we consider the sensitivity analysis with respect the system observability and the sample numbers. Specifically, for the 85-node power system, we vary the system observability, i... the ratio of the number of observed nodes the total nodes, in the set {0.2, 0.4, 0.6, 0.8). Secondly, we vary the sample number in {6000, 8000, 10000, 12000, 14000} with observability ratio be 0.6. Finally, we always fix SNR 100 for testing.
Table illustrates the results with respect to different observability. In general, as system ob- servability increases, all methods have a better prediction accuracy, and ATN achieves the best performance for different scenarios. Further, with system observability level increasing from 0.2 to 0.8, MSEs of ATN, Resnet, SINDy, DSR, EQL*, and PCNN have a relative decrease of 80.70% 41.37%, 22.94%, 66.67%, 63.64%, 60%. It shows that ATN make the most use of the new information from newly placed meters and achieve the most relative reduction.
Table 3: The MSE value for different methods with varying system observabilities.
SYSTEM OBSERVABILITTY
ATN
RESNET
SINDY
DSR
EQL
PCNN
0.2
0.057
0.29
0.17
0.21
0.22
0.15
0.4
0.033
0.25
0.11
0.14
0.11
0.10
0.6
0.014
0.18
0.11
0.12
0.13
0.10
0.8
0.011
0.17
0.08
0.07
0.1
0.06
Table Table(]demonstren demonstrates results with respect to different sample numbers. We find that for different methods, when the sample number is larger than 10000, the MSE value will almost keep stable. This shows that 10000 a corner point for the physical model learning the 85-bus power system. Con- sidering the high resolutin (e.g.. 30 samples per second) of the Phasor Mcasurement Unit (PMU) (Li et al. 2021a). 10000 an acceptable sample number for realistic training.
Table 4: The MSE value for different methods with varying training data samples.
SAMPLE NUMBER
ATN
RESNET
SINDY
DSR
EQL*
PCNN
6000
0.022
0.23
0.15
0.16
0.14
0.12
8000
0.014
0.18
0.11
0.12
0.13
0.10
10000
0.011
0.15
0.08
0.11
0.11
0.08
12000
0.009
0.14
0.06
0.12
0.1
0.06
14000
0.010
0.14
0.05
0.11
0.1
0.06
CONCLUSION AND FUTURE WORK
To accurately model modern physical systems with incomplete system observability, we propose twin structure of the physical neural network (PNN) and virtual neural network (VNN) to simultane- ously learn physics and approximate the remaining components. Different from traditional methods, our twin structure has critical designs to find the proper output boundary between PNN and VNN: (1) we utilize sparse regularization maintain physical consistency of PNN output, (2) we employ skip-connectio ciipconnnetioon to guarantee the output of the physical superiority, thus implicitly restricting the boundary of the VNN, and (3) we propose an adversarial learning scheme to adjust the fine-grained boundary for maximized physical learning. Specifically. for (3), we introduce embedding neural networks (ENNs) to extract similar features from outputs of PNN and VNN. Adversarially, training PNN and VNN leads to maximized output dissimilarity, leading to maximized physics. To guarantee the training stability, we employ the contrastive learning loss with excellent stability properties. Fi- nally, conduct extensive experiments over various systems demonstrate the best performance of our model compared to other methods. For the future work, we will refine the design of PNN to tackle more complex physical systems with unknown physical bases.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Aviad Aherdam, Ron Litman, Shahar Tsiper, Oron Anschel, Ron Slossberg, Shai Mazor, R Man- matha, and Pietro Perona. "equence-to-sqquenee contrastive learning for text recognition. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15302- 15312, 2021.
Steven I.. Brunton, Joshua I.. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the Na- tional Academy of Sciences, 113(15):3932-39977 2016a. ISSN 0027-8424. doi: 10.1073/pnas.
1517384113, URL //www pnas org/ http//onnenttnnnnttteee
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 1 113(15):3932-339) 3937, 2016b.
Steven I. Brunton, Joshua I Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. PNAS, pp. 3932-3937, 2016c.
Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. PNAS, 116(45):22445-2244 2019.
Deepak Divan, Rohit Moghe, and Anish Prasai. Power electronics at the grid edge : The key to unlocking value from the smart grid. IEEE Power Electronics Magazine, 1(4): 22, 2014. doi: 10.1109/MPEL.2044.2230
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarte training of neural net- works. The journal of machine learning research, 17(1):2096 2030, 2016.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 2014.
Minghui Guo and Yongzhao Du. Classification of Thyroid Ultrasound Standard Plane Images Using ResNet-18 Networks. IEEE International Conference on ontt-couuterfeeiting Security, and Identification, 2019.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Puttern Recogni- tion (CVPR'06) 066), volume 2, pp. 1735-1742. IEFE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016b.
Hailin Hu and Liangrui Tang. Edge intelligence for real-time data analytics in an bot- smart metering system. IEEE Network, 34(5):68-74, 2020. doi: 10. 1091 2.000039.
Haoran Li and Yang Weng. Physical equation discovery using physics-consistet neural network (penn) under incomplete observability. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 925 933, 2021.
Haoran Li, Yang Weng, Yizheng Liao, Brian Keel, and Kenneth E Brown. Distribution grid impedance & topology estimation with limited or no micro-pmus. International Journal of Elec- trical Power & Energy Systems, 129:106794, 202
Haoran Li, Yang Weng, Yizheng Liao, Brian Keel, and Kenneth E Brown. Distribution grid impedance & topology estimation with limited or no micro-pmus. International Journal of Elec- trical Power & Energy Systems, 129:10 202
10

---

Under review as a conference paper at ICLR 2022
Xiaoxia Li, Wei Li, Qiang Yang, Wenjun Yan, and Albert Y. Zomaya. Edge- bigge-computing-enaan unmanned module defect detection and diagnosis system for large-saale photovoltaic plants. IEEE Internet of Things Journal, 7(10)96511996) 2020. doi: 10.1 10.1109/100.220222288777
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In 2021 AAAI Conference on Artificial Intelligence (AAAI), 2021c
Li Ma. Renjun Shuai, Xuming Ran, Wenjia Liu, and Chao Ye. Combining DC-GAN with ResNet for Blood Cell Image Classification. Medical & Biological Engineering & Computing, 2020.
MATPOWER community. MATPOWER. 2020. https://matpower..
Sameer B Mulani, Samit Roy, and Bodiuzzaman Jony. Uncertainty analysis of self-healed compos- ites with machine learning as part of dddas. International Conference on Dynamic Data Driven Application Systems, pp. 113-120, 2020.
HyungSeon Oh. Aggregation of buses for network reduction. IEEE Transactions Power Sys- tems, 27(2):005-712, 2012.
Taesung Park, Alexei A Efros, Richard Zhang, and Yan Zhu. Contrastive learning for unpaired image-to-image translation. European Conference on Computer Vision, pp. 319-345. Springer, 2020.
Brenden K Petersen. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv:1912.04877, 2019.
Brenden K Petersen, Mikel Landajuela Larma, Terrell N. Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference Learning
Representations, 2021. URL
PJM Interconnection LLC. Metered load data. 2018. Feed/hrl load metered/definitiio
com/
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. International Conference Machine Learning, pp. 4442-4450, 2018.
Jonathan Sauder and Bjame Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in Neural Information Processing Systems, pp. 12962-12972, 2019.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 324(5923)811885,000 2009.
Naoya Takeishi and Alexandros Kalousis. Physics-integrated variational autoencoders for robust and interpretable generative modeling. arXiv preprini arXiv:2102.13156, 2021.
Texas A&M University. Suitsparse matrix collection. 2011. https: /ttptp:://spare.... tamu, edu/
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recog- nilion, pp. 1415-1424, 2017.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16))eayy6311, 2020.
Arjan van der Schaft. Modeling of physical network systems. Systems & Control Letters, 101: 21-27, 2017.
Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. arXiv preprint 14.44.2010.04456, 2020.
Jiafan Yu, Yang Weng, and Ram Rajagopal. Robust mapping rule estimation for power flow analysis in distribution grids. pp. 1-6, 2017.
11

---

Under review as a conference paper at ICLR 2022
Weiye Zheng, Wenchuan Wu, Zhigang Li, Hongbin Sun, and Yunhe Hou. A non-iterative decou- pled solution for robust integrated electricity- scheduling based on network reduction. /EEE Transactions Sustainable Energy, 12(2):14331188. 2021.
You may include other additional sections here.
12


Reviews:
[{"review_id": "ekyY2hPiC2H", "reviewer": "Reviewer_dem1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Several things need to be clarified before a proper review can be put in place:\n\n#1. Beginning of section 3.1 The author frame the problem as a graph with node set V and edges VxV. Each node in V contains (x,y) with the relationship y = f(x). Does this mean that (x,y) are associated with each node and for different nodes there are different (x,y)?\n\n#2. Then the assumption is made for some nodes being observed and some not observed. V = O union U. The the new symbols are introduced [y_o,y_u] = f( [x_o, x_u]). Seems to me that y_o is a set instead of variables representing one specific node. Otherwise this equation [y_o,y_u] = f( [x_o, x_u]) does not make sense. If so, then the (x,y) in my point #1 are variables, not set and here y_o is a set of variables. Please clarify this point\n\n#3. There could be a typo here: “Correspondingly, we denote z = [z1,z2]⊤, where zO ..”  What are z1 and z2?\n\n#4. \\phi the last paragraph of section 3.1 represent the basis coefficients for some known basis functions? Are the values of \\phi learned end-to-end in this network? By learning PNN?\n\n#5. Eq(3) the variable Y. Y=0 if sample pair is similar and Y=1 otherwise. In Eq(3) we only pass in one zo, not clear where the ‘pair’ comes from. Perhaps it is due to the confusion I brought up in #2. If zo is a set of variables, then how to apply a function to it? Element wise application?\n", "summary_of_the_paper": "Paper presents a method to push physical quantity to a physical neural network (PNN) and push the ‘additional quantity’ such as noise to a correction term virtual neural network (VNN). Uses a pair of embedding neural network (ENN) to achieve the above.", "summary_of_the_review": "will do a more proper review after my queries have been addressed. Will change my score after the proper review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "VZ2b8xGbTIm", "reviewer": "Reviewer_FaHY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Physical system identification is a definitely interesting topic in both the machine learning and physics communities. The authors address such an important issue and validate their proposal empirically. The proposed ATN show remarkable results in terms of model accuracy and unknown parameter estimation, with robustness regarding the sample size and observability. \n\nHowever, there are some concerns that should be addressed for clear evaluation of this paper:\n\n* Q1. The authors assume that there are physics bases $z$ that yield a linear relationship $y = Az$. In addition, they assume the unobservable base is also a linear transformation of the observable one, i.e., $z_U = B z_O + \\epsilon$, where $\\epsilon$ is relatively small noise. I wonder this assumption is general for arbitrary physical systems or just for fine the examples presented in this paper.\n\n* Q2. Because the proposed architecture is complicated, I wonder all the components are truly necessary. The authors claim the usefulness of each component indirectly, by comparing other baselines (e.g., the usefulness of physics bias with comparing ResNet, that of VNN with comparing SINDy, …), However, a more systematic ablation study should be included in the paper (or Appendix if the page is not sufficient), to clearly support the necessity of the complicated ATN structure. \n\n* Q3. It seems that the following paper is highly relevant: \"Y Yin et al. Augmenting physical models with deep networks for complex dynamics forecasting. ICLR 2021.\" This paper also decomposes the unknown physical system as a known physics part (however whose parameters are unknown) and the neural network part that fills the remainder that cannot be explained by the physics-based one.\n\n* Q4. It seems that the proposed ATN already knows the appropriate physics bases $z = \\phi(x)$, referring to the explanation on page 3 (“we fix the parameters from $x_O$ to $z_O$”) and equations (1 – 4) (no $x$ and $\\phi$ in the objectives). However, on page 7, the authors state that “ATN and SINDy can both learn physical bases”, which seems that there is a searching procedure of physics bases from a set of predefined operations. Can the authors elaborate it more clearly?\n\n* Q5. There are some inconsistent or confusing notations in the paper, e.g., on page 3 “Correspondingly, we denote $z = [z_1, z_2]^T$, where $z_O$ are physical bases…”, on page 4, $y_O = g_p(z_O) + g_v(g_p(z_O))$ should be $y$ rather than $y_O$ (because $g_v$ is included in the expression, and consistency with (1 – 4))?. In addition, Figure 1 is not clear because all subscripts of variables are 1, 2, … rather than $O$ or $U$.\n\n\n", "summary_of_the_paper": "The paper proposes ATN to model and identify physical systems. Here, the physical systems are grids consisting of measurement sensors: because the sensor configuration is not perfect, there can be unobservable parts in the grid system. The authors assume that there are physics bases that can represent the output with a linear mapping. Thus, they firstly use a linear LASSO regressor (PNN in the paper) that transforms observable physics bases to the output. The possible remainder (due to the noise or unobservable physics) is modeled via a more complicated neural network (VNN in the paper). These two networks are integrated into a single skip-connection block. They are trained by MSE, with enforcing the PNN part predict the output mostly, to exploit the bias of physics. In addition, the authors suggest the use of siamese networks that measure the distance between PNN and VNN: then an adversarial learning loss is added to guarantee the outputs of PNN and VNN to be far from each other and play their respective roles more concisely. ATN outperforms its counterparts for some physical system datasets.\n\n", "summary_of_the_review": "Currently, I lean towards rejection due to the concerns I raised. I would like to update my evaluation after discussing with the authors and my fellow reviewers.\n\n***\nPost-rebuttal:  I appreciate the authors’ clarifications on my questions. I am happy to see the ablation study result, which can support the necessity of the proposed components more clearly.\n\nHowever, the essential assumptions used in this paper ($y = A \\phi(x)$ with known $\\phi$ and $z_U = B z_O + \\epsilon$) make the proposed model be applicable only for relatively restricted problems. It prevents me from recommending a clear acceptance for this paper. While I slightly tend to accept this paper (thus I updated the review score accordingly), I would not champion this paper for the acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q4uhbne5lkQ", "reviewer": "Reviewer_8dT9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation and the base problem setting are understandable, and the experiments are done on multiple datasets. However, I found not a few contradicting or unclear statements. Also, the technical novelty is not that significant. Experiments lack important ablation studies. These facts make it difficult to recommend acceptance of the current paper.\n\nBelow, I write detailed comments in an unordered way; in my intention, points (2), (3), (7), (8), (9), (10), (11), (12), (13), (14), (15), and (16) are questions, and the others are not necessarily questions (anyway, please correct if something is wrong).\n\n(1) The beginning of the problem formulation in section 3.1 is misleading. Now it says:\n\n> A physical system can be modeled as a directed weighted graph ...\n\nThis sounds a bit weird because many physical systems are not usually modeled as a directed graph. Instead, it would be more appropriate to say that you handle specific types of physical systems that are modeled as directed weighted graphs.\n\n(2) Seemingly, at the layer of the problem setting, the main difference from [Li & Weng, 2021] is the presence of a possibly nonlinear function $f$ between $x$ and $y$. Meanwhile, such a difference is actually not valid because now the prior knowledge of the nonlinear functions used in $f$ is assumed as $\\phi$. Consequently, in my understanding, there is no effective difference between the problem setting of the current paper and that of [Li & Weng, 2021]. I would like to hear if this understanding is correct or not. Please note that I understand the methods in both papers are more or less different; here I question the difference of the problem setting.\n\n(3) What are \"the remaining quantities\" that appear at the second line of section 3.3? Do they refer to $x$'s (or $z$'s) on unobserved nodes?\n\n(4) I fully agree with the need for restricting the output of VNN and encouraging PNN to output as much as possible. In fact, similar ideas have been already investigated in the context of physical system learning in literature such as [Yin et al., 2021] and [Takeishi & Kalousis, 2021].\n\n[Yin et al., 2021] Yin et al., Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, in ICLR 2021.\n\n[Takeishi & Kalousis, 2021] Takeishi & Kalousis, Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling, arXiv:2102.13156.\n\n(5) In section 3.3, the skip-connection of the two neural nets is \"proposed,\" while it is just a frequent practice. \n\n(6) The regularization in equation (1) to encourage $g_p$ to directly approximate $y$ is reasonable. I note that a very similar idea was adopted in [Yin et al. 2021], where they regularize the norm of the non-physics part (i.e., $g_v$ if the notation is aligned to the current paper). I think the two approaches, i.e., minimizing $\\Vert y - g_p(z) \\Vert$ and minimizing $\\Vert g_v(g_p(z)) \\Vert$, are almost the same thing eventually after fitting gets enough good. However, in the early phase of training, since the models are not well fitted yet, perhaps there might be some practical difference.\n\n(7) Does $y$ without subscripts in equation (1) mean $[y_\\mathcal{O}; y_\\mathcal{U}]$, similarly to $z$ and $x$? Maybe I missed something but could not find an exact definition.\n\n(8) The most puzzling thing is that in section 3.4, $g_v$ suddenly starts to take $z_\\mathcal{O}$ directly as an argument, whereas it used to take $g_p(z_\\mathcal{O})$ as an argument in section 3.3. What are the definitions of the domain and the range of the functions around here?\n\n(9) I could not see the point of introducing matrix $B$ in section 3.4. Does it appear somewhere in the model?\n\n(10) What is \"large common knowledge\", at the fourth line of the fourth paragraph of section 3.4?\n\n(11) What is the relation between the proposed adversarial loss and the well-known Wasserstein distance based on the Kantorovich-Rubinstein duality (see, e.g., the paper of Wasserstein GAN)? Do you have any observations on the advantages of the proposed loss, compared to the Wasserstein distance?\n\n(12) What is the actual use of $Y$ in equation (3)? In my understanding, the idea here is to maximize dissimilarity between $g_p(z)$ and $g_v(z)$ for each $z$, and thus $Y=1$ is always used. Then, there is no need to consider the case of $Y=0$. Is this correct?\n\n(13) What $\\phi$ exactly was given to the proposed method in each experiment?\n\n(14) In the experiment, are the baseline methods also given the knowledge of the base functions, i.e., $\\phi$? If not, then the comparison is not really fair. \n\n(15) The authors say that one of the baselines, PCNN, \"doesn't include a clear design of LASSO regularization.\" However, in my understanding, [Li & Weng 2021] use lasso for initializing the parameters for fully observed nodes. Do you also use the same lasso-based initialization for PCNN?\n\n(16) Seemingly, both the reported test results and the hyperparameter selection are based on cross-validation. Are there two different (and maybe nested) cross-validation? This is an important question. I would like to know the difference between the cross-validation for hyperparameter selection and that for generating the final test results.\n\n(17) There are no ablation studies on the effect of the proposed regularizers (in equations (1) and (3)), which makes it difficult to assess the goodness of the proposed method.\n\n(18) Twin structures of neural networks are mentioned in section 2 and some other occasions, but I could not understand why the proposed method also falls in this category, because nothing seems to be twin in the proposed method (PNN is linear, and VNN is nonlinear).\n\nBelow, minor points.\n\nUnnecessary \"and\" slips in at the third-last line of page 1.\n\n\"Target is the identify ...\" (the first line of page 2)\n\n$z=[z_1, z_2]^\\top$? (the lower half of page 3)\n\n\"lower nose level\" (the seventh line of page 8)", "summary_of_the_paper": "The task of identifying a physical system on a graph is addressed. While the main part of the to-be-estimated model is assumed to be linear, the proposed model needs a nonlinear part (which is modeled by a neural net) due to the presence of unobserved nodes. The authors use a combination of a sparse linear model and a neural net, which is basically the same as the model in [Li & Weng, KDD 2021]. They consider some regularization terms to maximize the use of the physics part of the model. They examine the performance of the proposed method on several datasets.", "summary_of_the_review": "Technical novelty is limited, and there is no ablation study on the exact proposal of the current paper. Hence, I can hardly recommend acceptance in the current form of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the identification of physical systems defined on graphs. The authors introduce the Adversarial Twin Neural Network (ATN), which consists in augmenting a simple linear model (PNN) with a virtual neural network (VNN). Some regularization terms are used to enforce maximum prediction from the PNN, and to enforce diverse outputs between PNN and VNN. 

The paper initially received tree rejection recommendations. The main limitations pointed out by reviewers relate to the limited contributions, the limiting assumption of using a linear mode for PNN, the lack of positioning with respect to related works, and clarifications on experiments. The authors' rebuttal answered to some reviewers concerned: Rdem1 increased its grade from 3 to 5, and Rdem1 from 5 to 6 - although not willing to champion the paper. R8dT9, which provided a very detailed review and feedback after rebuttal still voted for rejection, especially because he was not convinced by the positioning with recent related works and the answers on experiments.  

The AC's own readings confirmed the issues essentially raised by R8dT9 and other reviewers. Especially, the AC considers that: 
- The contributions for driving a proper cooperation between the PNN and VNN models are weak, since it reduces to using simple skip connection and adversarial training. 
- The importance of these aspects have not been analysed in depth in the revised version of the paper, neither theoretically nor experimentally: for example, the difference with respect to [Yin+ 2021] for a proper augmentation, the discussion to alternative methods for representing diversity as done in [Rame & Cord 2021], or the positioning with respect to  Wasserstein distance-based objectives.  
- There remains ambiguities in the cross-validation process, which have not been addressed in the rebuttal.

Therefore, the AC recommends rejection.
Decision: Reject