WHAT TO DO IF SPARSE REPRESENTATION LEARNING FAILS UNEXPECTEDLY?
Authors: Jingyi Yuan
Haoran Li
Erik Blasch
Yang Weng
Venue: ICLR 2022 Submitted (2022)
Keywords: Physical neural network
extrapolation

Abstract: Learning physical equations from data is essential for scientific discovery and engineering modeling. However, most of the existing methods rely on two rules: (1) learn a sparse representation to fit data and (2) check if the loss objective function satisfies error thresholds. This paper illustrates that such conditions are far from sufficient. Specifically, we show that sparse non-physical approximations exist with excellent fitting accuracy, but fail to adequately model the situation. To fundamentally resolve the data-fitting problem, we propose a physical neural network (PNN) utilizing “Range, Inertia, Symmetry, and Extrapolation” (RISE) constraints. RISE is based on a complete analysis for the generalizability of data properties for physical systems. The first three techniques focus on the definition of physics in space and time. The last technique of extrapolation is novel based on active learning without an inquiry, using cross-model validation. We validate the proposed PNN-RISE method via a synthetic dataset, power system dataset, and mass-damper system dataset. Numerical results show the universal capability of the PNN-RISE approach to quickly identify the hidden physical models without local optima, opening the door for the fast and highly accurate discovery of the physical laws or systems with external loads.

Under review as a conference paper at ICLR 2022
WHAT TO Do IF SPARSE PHYSICAL EQUATION LEARNING FAILS UNEXPECTEDLY?
Anonymous authors Paper under double-blind review
ABSTRACT
Learning physical equations from data is essential for scientific discovery and engineering modeling. However, most of the existing methods rely on two rules: (1) learn sparse representation to fit data and (2) check if the loss objective function satisfies error thresholds. This paper illustrates that such conditions are far from sufficient. Specifically, show that sparse non-physical approximations exist with excellent fitting accuracy. but fail to adequately model the situation. To fundamentally resolve the data-fitting problem, network (PNN) utilizing "Range, Inertia, Symmetry, and Extrapolation" (RISE) constraints. RISE is based complete analysis for the generalizability of data properties for physical systems. The first three techniques focus on the definition of physics in space and time. The last technique of extrapolation is novel based on active learning without an inquiry, using cross-model validation. We validate the proposed PNN-RISE method via a synthetic dataset, power system dataset, and mass- damper system dataset. Numerical results show the universal capability of the PNN-RISE approach quickly identify the hidden physical models without local optima, opening the door for the fast and highly accurate discovery of the physical laws or systems with external loads.
propose physical neural
INTRODUCTION
Internet of Everything (loE) connects new edge devices into an intelligent web at an unprecedented speed. If utilized systematically, loE can create much more efficient productivity via coordinated effort among humans, processes, data, and things (Li et al. 2020). Among various analytical frame- works, dynamic data-driven applications systems (DDDAS) naturally interweaves diversified infor- mation via data-driven analysis to assimilate the system model for a wide range of applications (Blasch et al. 2018), thus significantly increasing the performance of both the simulation and ap- plication models. The simulation model usually incorporates physical methods (Nasiakou et al.) 2018 Chowdhury & Subramani] (020)) which provides a structure for model-based and data-driven coordination. However, many of the physical laws in the web of IoE can be unknown, which remain to be discovered manually automatically to enable down-streaming applications with accurate modeling.
The most economically efficient way of physical law discovery is use machine learning ana- lyze the pattern behind the data stream, which is an important topic (Sahoo et al. 2018: Udrescu &
Tegmark 2020). In previous studies, such symbolic regression (Petersen) 2019). the sparsity rep-
resentation is used to measure the accountability of physics (Brunton et al. 2016). The underlying
assumption is that sparsity can ensure physical exactness due to parsimonious principle (Li) 2013).
While low complexity of learning models is necessary condition for enhancing the understanding of physical laws (Schölkopf et al.) 2002 Blasch et al. 202T), there not a universal sufficient condi- tion solve model recovery problem with guarantee. Therefore, conditions to ensure a physical recovery, e.g., near-zero fitting errors (Champion et al. 2019) or/and cross validation (Kim al. 2020). However, it can be observed that different models easily fit the available data with a sparse but wrong model, which can happen even with cross- validatii.
approach is to add additional
The overfitting should challenge the directions of many researchers in physical system discovery, who believe that only the true physical laws can achieve sparse recovery with a perfect fit. To resolve the problem, first eliminate non-physical local optima of the neural network (NN) by

---

Under review as a conference paper at ICLR 2022
discarding all the non-phn non-physical parameters, i... model parameters indirectly related to the physical representation. The process is novel as existing NN-baser methods require certain non non- physiccll parameters control the representation sparsity (Liu & Laili 2018: Petersen 2019). We show how to design a NN with pure physical parameters, entitled a physical neural network (PNN). The key of PNN is to enforce activation to physical weights for sparse selection, completely removing the non-physical parameters and local optima.
In training the PNN, there can still be local optimum solutions, the actual knowledge of physical system can enhance model fidelity. To obtain the ground truth, physical system examination can be used extract common and universal properties. Subsequently, the PNN is restricted to systemat- ically obey the physical system properties. Specifically, constraints based on the physical distance and symmetry are embedded in the training function to quantify the spatial relationship enforced by physical law. Furthermore, as physical systems evolve over time, inertial property for physical systems are extracted and updated. general, the regularization from physical laws focuses the leaming/optimizztionn towards more meaningful fitness understanding of the given data.
Although the two designs PNN and physics constraints aim to obtain the physical laws via fit- ting existing data, the incorporation of physical laws' biggest impact is its capability to achieve zero fitting errors even beyond the training data range. Unfortunately. it will be costly if not im- possible to conduct active learning for every Internet of Things (IoT) system due to its scale. To provide the "free lunch" of adding the extrapolation capability of PNN, we propose a new learning method for cross-model validation based on probabilistic estimates with physical kernels. We use a physically meaningful forecast to identify out-of-sample points with a confidence level. Then, the results are integrated as a weak supervision for learning the exact physies for extrapolation, in addition to the constrained PNN. Together, PNN-RISE uses the PNN container to include the RISE principles, enforeing the recovered function satisfy constraints on range (R), inertia (I), symmetry (S), and extrapolation (E). Empirically, extensive validations show that the PNN-RISE outperforms the state-of-the-an methods. Consistently, scenarios with or without each constraint in RISE vali- date the approach. Furthermore, for each constraint, we quantify its strength with different levels, highlighting that loose constraints are enough to obtain high accuracy in the proposed PNN.
Challenges
Fit Internal Data
Fit External Data
1. Choose Models
Physical Principals
3. Extrapolation
with Less Para.
RISE Did
Virtual
Active
Global
Near Global Minimum
Extragolation
Learner
Forecaat
O
Minimum
Lift
Sparse Expression
Training Data
Physical Law
Push
Low Fitting Error
Less Local Optima
Eliminate Local Optima
Cross Model Validation
Figure 1: The proposed framework.
In summary, the contributions in Fig. are 1) showing that the sparse recovery with low training and testing errors does not ensure physical law recovery, 2) reducing the possibilities of local op- tima significantly by eliminating non-physical parameters in learning model selection and design, 3) proposing four physical principles to completely characterize physical systems via constrained learning, and 4) creating a novel learning method for physical systems extrapolation as probabilistic forecasting out-of-sample cross-model validation (FOCV).
RELATED WORK
Physical System Identification. To recover system information, supervised learning (SL) can ap-
proximate input-output relationships for inferences (Huang et al. 2006 Hornik et al. 1990). They
show good numerical performance in non-physical and physical applications, including computer
vision tasks and natural language processing (Lu & Weng 2007 Wang et al. 2017a Karpathy
et al. 2014 Nadkarni et al. 2011 Hirschberg & Manning 2015). Deep neural networks (DNNs) boost classification accuracy (Hinton et al. [206]] Krizhevsky et al., 2012) by using latent layers to extract nonlinear features hidden in images. While it makes SI. useful for non-physical systems,
2

---

Under review as a conference paper at ICLR 2022
physical systems require a much higher accuracy for operation control, e.g., near 100% for Six Sigma performance (Walshe et al. 2010), so that operators can safely operate the system to avoid outages cascading failures (Chung et al., 2018 Liu et al. (017). Even worse, many SL models are black boxes that lack physical interpretability for reliable operations, as well as certifiability for
performance standards (Mestav et al. 2018 Du & Li 2019 Hu et al. 2020 Baghaee et al. 2017).
Therefore, some works find utilizing contextual information improves physical-syster identification where they assume a good knowledge of physical basis and sufficient measurements across the system Yu et al. 2017; Chen et al. 2020). Similarly, (Brunton et al., [0166; Champion et al. 2019) Li & Weng 2021) assume prior symbols are known dictionary and introduce sparse regression to select symbols. However, the assumptions are too limiting due to the difficulty of knowing all the physical basis for arbitrary complex systems (Wang et al., 2017b) TTarca et al.] 2007). One ad-hoc approach (Schmidt& & Lipson 2009 Petersen, 2019) uses genetic algorithm to search for the symbolic basis but the problem is NP-hard without any guarantee on recovering the governing physical laws from empirical data. Thus, we investigate whether the sparsity technique and a small modeling error are sufficient to claim the success of learning the physical model. Moreover, how can analytical techniques ensure exact physics learning for domain-specinn systems?
Constrained Machine Learning. There are many applications of constrained machine learning restricts the weight vectors with domain knowledge 2021) proposes constrained adversarial learning to gen-
(CML) in different systems. (Small et al. 2011 in support vector machine and (Li et al. erate adversarial examples under physical constraints. In deep learning applications, (Zhao al., 2019) employs constraints from conservation law to restrict the output variables of DNN. (Velloso) & Van Hentenryck 2020; Fioretto et al., 2020a) solve optimal power flow problems using DNN with physical constraints, where the output variables are biased to guarantee feasible solution. (Tran al., 2020) restricts the input to DNN into different groups to guarantee privacy. Among these studies, (Fioretto et al. (2220b) optimizes DNN with output constraints using Lagrangian duality. These CML methods can solve their own problems effectively; however, there is a need for general principles to embed the physical constraints systematically with efficient solutions.
Data Forecasting. For previous data forecasting methods, autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) predict the evolving variables based on their lagged values (Gilbert, 2005). To capture nonlinear correlations in the short-term and long-term, there are many deep learnin-basse methods, e.g., recurrent Neural Network (RNN), long short- term memory (LSTM), bidirectional LSTM (BiLSTM), gated recurrent units (GRUs), and varia- tional autoencoder (VAE) (Zeroual al., 2020). To forecast the result with confidence, methods like Gaussian process (GP) (Quinonero-Canee Caannero-Cander Candela & Rasmussen] 2005). Quantile Regression (Ma-) ciejowska et al.) 2016) and deep learning- based models (Salinas al. (020) afford efficient way to predict distributions for the future data. Since the uncertainty evaluation is important to integrate the forecast data into the physical learning process, probabilistic methods are presented estimate probability density functions in the proposed FOCV.
METHODS
The PNN-RISE approach is to understand the reason why a "good" sparse fitting is not physical ver- ifiable, to design physical principles guide the fitting internally based on flexible PNN structure, and to create a cost-free active learning method for extrapolation at unseen operating points.
Symbolic regression (SR) is common approach to find equation operators (1+.-.
and pressions like (x² cos (x) ). etc. Existing papers (Brunton et al. 2016 Sahoo et al., 2018 Champion al. 2019 Petersen| 2019; Kim et al. 2020) show that if the following three conditions are sat- isfied, SR is capable to find the true equation: 1) apply the sparse regularization, 2) achieve nearly zero training error, and 3) utilize cross validation to avoid overfitting. However, we observe that SR methods are prone to non-physical solutions even when all three assumptions are satisfied. For ex- ample, to learn the function y = x 21 + T223 from data, different SR methods can easily get trapped in local optima. To see if these observations are special, we test different cases, where Fig. 2 shows a table summary of four different cases, with integer coefficients or not. We also test different num- bers for multiplication and summation. From all examples, one can see that the recovered objective has sparse coefficients and negligible error (loss objective L) at non-desirable local optimums. Other than the toy examples, we conduct realistic power system case study and visualize the objective
3

---

Under review as a conference paper at ICLR 2022
surface in Appendix Similarly, for a small parameter range, there are points achieving the global optimum or local optima that are very close to the global optimum. However, only one of them is the physical ground truth. For example, one can go from the red point to non-physical solution with an error close to zero, even when the sparsity regularization is enforced. Therefore, we can see that "good" sparse fitting is insufficient avoid overfitting. Thus, we need to avoid overfitting for physical systems not only by choosing sparse learning models, but also new ways to constrain the learning process and avoid non-physical solutions.
Solations
Cisc
Casc 2
Case 3
Cisc
Physsca
0.1dt C
Truth
y 0.00242125
1.3192e4 8201 arrgg
y lef #3St anpe 1930
1.866 7008 u40 0002
Local
2. 2.0733224944 (y
+1.1106s the guiis g
8. 2.000 ro
0 of 1670 of 1eso, Lame
Optinum
4 +2.24482 1o 2g
= 0.002
0.008
0.021 10.4 19 apyd S0ur
9 1 4-112 of wis go +0.9600u him
2,618 11501 oo pam
1020 casss rass
Local
of toon hous
+1.44942 stro star golo
-9 0.8456ee,.) 2y
LOMGU
diss
Optinum
0300x 1240 2012 soos
0.001
0.00
0.003
0002
Near
O. D.O.Mi Pa
2125
Global
boco
0.3 0.3 1000 C 0.001
Optinum
C 0.020
0.056
2 0.08
Figure 2: Absolute mismatches between the recovered functions and the physical ground truth.
3.1 PROPOSED METHOD: PHYSICAL NEURAL NETWORK WITH WEIGHT ACTIVATION
For complex physical systems, the governing functions are formed by basic operations such as additive and multiplicative accumulations of variables. For example, the topology of power system can be represented as a graph, where the interaction between diffcrent nodes along the lines is represented by polynomials of the voltages for power flow (Yu et al.) 2017). To combine the benefit of universal approximation and the freedom to choose activation function, we propose to use a NN as a container to learn the representation of physical equations, and thus call it physical neural N network (PNN), denoted as f. To train f. the training data set is (zx; yı i==> where æ; and y; are the 5th input and output of f. and N is the number of training samples.
To construct the physical equation in PNN, the key is to develop sparse selection mechanism to automatically select part of the variables to join summation/subtraction and multiplication/ For sparse selection, existing work utilizes sparse regularization like LASSO for weight updates (Kim et al.] 2020). introduces extra neural network modules like gates (Liu & Laili, 2018), or con- ducts symbolic generators (Petersen 2019). In general, they demand the quantification of extra hyper-parameters (i.e., the LASSO penalty) or parameters (i.e., parameters in gates or symbolic generators), which significantly increases the possibility of local optima.
In order to address the problem towards limiting local optima, we advocate the design philosophy: enforce activation functions on weights for the sparse selection of features. Thus, no extra param- eters are added, and the searching is only for physical parameters. To elaborate on our design, we need answer (1) which parameters need to be activated and (2) how to enforce the activation. For parameter activation, we start from the basic symbolic operations in neural networks, e.g., multipli- cations and divisions of the form =j = D1." u.Pu, where zj is the jth element of the layer next to the input. 24 is the 7th element of the input vector variable x and Pij is the polynomial order number (power) of zy. Note Pij E [-K,... -1, 1, 1,... K). where the positive integer represents the multiplication, the possibility 0 corresponds to the sparsity, and the negative integer represents the division. Then, each zj is more complex symbol for the next operation and Pış is the optimization variable. To accommodate Pij into a NN, we adopt logarithm and exponential operators to achieve
N linear summation format 2j exp (EN) Piy In(zz)) (Trask et al. 2018). Thus, the linear layer with weights Ptj needs to be activated for arithmetic operations and sparsity. Considering the target integer range of Pij. it's vital to know how to control the values for parameter activation.
Di=1
We first show how to enforce Pij E (-1, 1}. Since tanh function can restrict the output value to be in 11, we can utilize a positive scaling factor e1 and tanh to activate some free and optimizable parameter wi such that tanh(c, c, why) E [-1, 1). where the scaling is to make sure the output is very close to the target- -1 1. Subsequently, the network needs to enforce 0 for sparsity. We
4

---

Under review as a conference paper at ICLR 2022
similarly create another sigmoid function to achieve ((or)))) E {0,1}. where is the sigmoid function, 02 is the positive scaling factor, and Way is another optimizable parameter. To summarize, we have Pij tanh (crwiy) ((equiy), where Púj E (-1,1). The multiplication of tanh and sigmoid functions not only generates smooth optimization surface but also naturally results in limited parameters. After that, third linear layer is used to obtain yk = Σ, mjk25. where Mik denote the layer weights for linear summation. Since the summation and abstraction are incorporated into the linear layer. we completely represent y = f(x).. Using such design as a basic unit, we can extend from { (, 1] to K, .... -,, 1,... K} for different physical systems.
Such design converts the non-convex multiplication to a convex form of linear summation for NN training. Based on this design, we show the following property of our PNN to guarantee the finding of stationary points in the optimization. The following proposition states the finding of stationary points using PNN with the design and the proof is included in Appendix.
Proposition 1 (Biconvexity of PNN). PNN is biconvex on the parameters of the logarithm and the linear summation layers. The stationary points for physical function can be found in optimization.
indicates that PNN can find the stationary points (i.e., saddle points, local optima, and global optima). Subsequently, can we find the global optima and can it represent the true physics?
Proposition D
3.2 EMBED PHYSICAL PRINCIPLES VIA CONSTRAINING PNN
After providing a flexible design to limit possibilities of local optima, we need comprehensive prin- ciples to avoid non-physion non-physical local optima. If the derived constraints can be convex, they will provide guarantees in solving or reducing the issue of non- non-physical local optima.
For physical systems, two important perspectives physical existence and systematic behaviors. As the components are physically located, there are geographical or electrical distances, etc. There- fore, one common constraint through physical systems is the symmetric (S) property, c.g., a branch connecting two components represents a common coupling. In addition, the links between different components also have their physical limitations, leading level, the physical system will continue making an impact to the environment, leading to an inertia (I) concept, which is the momentum from physical object because physical systems typically have natural inertia, whether it is strong or weak. For man-made systems, there could be additional ar- tificial inertia due factors such as guaranteeing stability for sustainable operation, e.g., in power systems. Finally, such inertia does not only live in the temporal space but also in the input/output data space. For example, the physical equations are typically continuous functions. These concepts open the door to investigate the data range beyond the training and testing data. Such generalized inertia leads to an important extrapolation (E) property across different physical systems to forecast information yet to be observed but representative of plausible scenarios. The constraints based on different properties of physical systems are called RISE concept by combining their initials, which is demonstrated via figure in Appendix. This subsection introduces the first three to constrain PNN in Subsection 3.1 while Subsection 3.3 introduces the FOCV method for physical system extrapolation.
range (R) constraint. At the system
Range (R): Shape Constraint Physical Parameters: While local optima generate many pos- sible parameters to minimize the prediction error, the true physical parameters of corresponding features usually yield specific shapes (Cotter al.) (019).. For example, the range of power line admittance (reciprocal of impedance) of power distribution systems can be specified with voltage level and line materials. In addition the bound on value range of independent parameters, there can also be shape constraints on the conditional relationships among parameters, e.g., longer lines have larger resistance than shorter lines. Such rule specified by the physical shape constraint func- tion can filter out many local optima even with low errors. In order to enforce the learned physical parameters W, within the constraint, a customized gate function is proposed to filter the weights: for g(W,)) p) < Lgpl. (WW, p) = [ap]; for 1gp] I ((W,) S [g g (W,)) for g(W,)) p) 2 [ 9(.) is a function of physical weights W, to represent the type of shape con- straint. As example, g(-) is linear for specific value constraint of independent parameters, where 19p. and [gp] are the lower and upper bounds, and g' (Wp) are the constrained weights.
g(Wp); and g' (W,)) [9
Symmetry (S): Constraints Physical Couplings: For physical systems with multiple outputs, there are variable couplings among different outputs, e.g., one term is contained in two outputs. Let's use the power system an example. A certain amount of power flowing from one place to
5

---

Under review as a conference paper at ICLR 2022
another means that the same amount of power will be subtracted and added simultaneously at the two places. Based on our observations, most of the local optima of NN do not satisfy such a rule. Thus, the property is embedded by minimizing Mutual first term is an (2 norm penalty on the symmetry of recovered physical parameters and the second term is to further ensure that the number of mutual terms N Nuuuual satisfies the system property, e.g., power system equation has N Nuuttal 2. because two nodes are connected by one power line. For practical implementation, the 1 norm is relaxed to 2 norm in optimization.
|W W ll + ||rank ) |l || The
For physical systems, the states are typically con- tinuous and cannot change suddenly, leading to physical inertia. Such property can be embedded by bounding the outputs of two adjacent time slots. To ensure the partial convexity, the Euclidean norm is modeled as penalty in loss function Iinertia [=(++)+))= f(xx)) 12-
Inertia (I): Constraints on Physical Tendency:
3.3 CROSS-MODEL VALIDATION FOR PHYSICAL EXACTNESS AT UNKNOWN DATA REGION
Subsections 3.1 and 3.2 show how avoid overfitting by considering the NN- model design and fitness of data to the physical laws. However, all these designs must be within the training data range. What will happen beyond the training data range, e.g., a new operation point of physical systems unseen in the past? Actually, determining the performance region is the most useful part for physical systems modeling after understanding the physical laws, c.g., for universal data fitting at arbitrary points. For example, Fig. B shows three globally optimal near globally optimal lines (red, blue, and green) that can fit data equally well within the training dataset. One way to avoid the selection dilemma is to conduct
active inquiry out of the historical data range for selection. For example, Fig. 3 iillusraates that a few out-of-sample data points in grey can regularize the learning to the ground truth (dotted black line). However, the costs of such an inquiry can be quite expensive as one inquiry may not be enough. Also, given the numerous networks that the internet of everything (loE) has, it infeasible and sometimes impossible conduct an inquiry for out- sa-sa estima- tion. Therefore, how can learning approach avoid any cost of active learning but still explore the generalization capability via some alternative active learning?
Out of sample
sample
Out sample
Ground Truth
1. Same behavlor
Predicted Result
Local Optima
Local Optima
Forecast data
Closé-to-boundar poonts physics learning.
Far-from-bounddrr points have negligible impacts.
Figure 3: Illustration of out-of-sample forecasting to improve extrapolation.
To conduct active learning without an inquiry cost, we implement physically meaningful forecasts via a probabilistic model for data points not existing historical measurements. The learning objective is provide another level of physical guidance for the constrained PNN in previous subsections. Specifically, an ensemble of forecasts is conducted for both the mean and variance, as the forecast ability decreases when the inquiry point is far away from the data range. Another important design is to embed the physical form into the forecast so that the model can cross-vale vols-val the learning models. We pro- pose to utilize Gaussian Process (GP), which not only has computational efficiency in probabilistic forecasts but also provides the flexibility to embed physical forms into its kernel function.
Extrapolation (E): Constraints on Physical Validation at Unknown Regions:
For training the GP, the data is represented as (æ,, Z. /i=l where æ; represents the input samples and ã; represents a sample point k-step later than æ; Based on such data, GP assumes prior func- tion values in a Bayesian framework, e.g., p(g] 1,, nN) NO K). where g is a vector of latent function values, A represents normal distribution, and K is a kernelized covariance matrix. add different kernel mappings to calculate K based
N
In particular, knowledge. For example, for power systems, the polynomial of order 2 is often utilized to represent the relationship between voltages and powers (Yu al. 2017). Using data with kernel knowl-
our priors of the physical

---

Under review as a conference paper at ICLR 2022
edge, we obtain predicting function via Bayes rule p(g") = p(a) p(ālg))((s) 9 )dg, which has a closed-form solution g" based on Gaussianity. The goal is predict the k-step later points for any input sample. Specifically, one can plug-in the input data æ and obtain the distribution of the k-step points p(g" (z)). Thus, the GP-based forecasting not only brings the forecast point Tg, e.g., the mean of p(g" (z)). but also the confidence level ar based on the standard deviation. Similarly, can obtain yı; and Bx. where yt is the forecast point for the output data y and 33 is the corre- sponding confidence level. Fig. 3 iillustrates the results of GP, where the forecast points have smaller uncertainty if they are closer to the dataset boundary.
p(ä)
With the forecast results from above, we can add Ty (ÿk) to the training data set, like active learning. However, our method is different from active learning, which is associated with cost. With the cost, active learning can provide a highly accurate label. But, our new data points are always associated with confidence bounds. The good news is that the FOCV method can provide infinite data points without a cost. Therefore, different than active learning, FOCV uses all the data points with unequal weights, e.g., the inverse of confidence level, achieve model selection. Therefore, we propose an integral-based regularization for extrapolation LEstrapot = ay 3. Ek=1 (Uk f(āx))² With extrapolation (E), the proposed methods complete the derivation of RISE. It's important to note that in RISE, the factors decoupled out of the PNN. Furthermore, the factors can be relaxed into convex form, which significantly increases the probability of finding the global optima. Specifically, Theorem [ highlights the regularization can lead strong convexity over a set of closed and convex regions of the parameter space, i.e., the piecewise strong convexity.
Theorem 1 (Piecewise strong convexity of PNN with regularization). With inertia and symmetry regularization, the loss function of the PNN is piecewise strongly convex in the parameter space.
Proof. For the derivation simplicity, we denote W the parameter set of PNN and denote I(W) as the loss function, i.e., mean square error. The range constraint (R) and the extrapolation constraint (E) are included in ((W) as they are included in the PNN representation. Subsequently, the iner- tia (I) and symmetry (S) regularizations help to define 4 (W) ((W) + X(LInertia + L as the regularized loss function for training, where À positive penalty term. To determine convexity, we write the second derivative of the regularized loss function dzz Ie + LX) = d² de lt (W++))+++))))) +XX)++++++++++++++++++ +xXX)x(hWW)))) Ah(W). where hW) is the second derivative of Linertia + L Linertie has a quadratic form to be convex. For symmetry L.Mutual, the 1 norm regularization is convex and h(W) > 0. However, due to the non-convexity of neural network, the first term ar l + LX) is generally not positive semi-definite. The following lemma in (Milne) 2018) proves lower bound for df le=ol(W + (X) with respect to the loss value of L.
df
&
Lemma 1. Suppose 1/all2 r,r > 0, S S N. The second derivative of I in direction X where is the number of hidden layers in neural network, || || is defined norm such that [|W]]. maxosisH H [[W:112-
d² satisfies azz wW +tX) > (2 11) 1)
[|
bounde by a term that depends on value. Then, for regions with small loss values, the positive value h(W) helps to restrict dia le=oh1 (W+tX) a positive number, which determines the shape of the region be convex. Mathematically, [W3 <
According to Lemma I the second derivative of
define the region O), where À 0 > 0, by U(),)) 0)
(2H(H-L)I|XX T }. It's obvious that VW E U(A, 0). loo (W + LX) > O. Namely, within the region of (() f), the regularized loss function strongly convex. general, U(}, 6) is not a convex set since many local optima points are disconnected and have small / values. However, based on measure theory, we can utilize a set of closed convex sets to completely cover U(A, 0), leading to piecewise strong convexity of PNN over these closed sets. Therefore, the piecewise strong convexity from RISE regularization guarantees that we can find the global optimal point for each sub-region, and the global optima satisfies constraints for physical parameters.
(A-0)A(W)
4 EXPERIMENTS
Data Preparation. In experiments, we first use synthetic datasets for illustration and introduce real physical system datasets of power systems and mass-damper systems as the underlying physical
7

---

Under review as a conference paper at ICLR 2022
systems for model training and comparison. The datasets summarized as follows: 1) Syn- thetic dataset is generated from quadratic functions with adding random noises. 2) Power system datasets: The full training datasets of nodal voltage states and power injections are generated from power flow simulations using real feeder models (IFEE 8-bus system and Utility feeder), one-year power consumption profiles, and distributed photovoltaies generation profiles (Yu et al.] 2017). 3) Motion dynamics dataset: We simulate the dynamic process of the mass-dampers systems with 10 nodes and obtain the measurements of velocity and momentum for the test of physical equation dis- covery. 4) Aerodynamic dataset: The aircraft sensor data for weight estimation is collected from the aircraft simulation based on the aerodynamic model in (Chen et al.) 2018).
Performance Evaluation Metries. The following metrics based on Blasch & Sung 2021 are used to evaluate the performance of learning physical system governing equations. Prediction error is to evaluate the modeling accuracy and extrapolation capacity of learning model via mean square error (MSE) of the output prediction. Physical exactness (PhyE) is to evaluate how much physies is extracted by comparing the recovered symbolic function and related parameters with true physical equation. Robustness to noise ratio (RNR) is proposed to average all results tested under different signal-to-noise ratios for practical consideration. More details are provided in Appendix.
Benchmark Methods and Testing Scenarios. The following benchmark methods are used for
comparison: (1) sparse identification of nonlinear dynamics (SINDy) (Brunton et al., 2016), (2) support vector regression (SVR) Yu et al. 2017). (3) deep residual network (Resnet) (He et al.) 2016), and (4) equation learner (EQL*) (Sahoo et al. 2018). Different scenarios generated for lesting: S₁ denotes the regular setup of dataset size and noise level (independent Gaussian noise with zero mean and 0.01 standard deviation); S2 denotes a noisy setup with noises of 0. standard devi- ation; Sy denotes a data-limited setup with 10% samples of original dataset available. The results
8' Dare
SVR
Resnet
SINDY
EQL
PNN-RISE
Sythetic
0.00
0.005
0.00
0,001
0.00
0,003
0.001
PS I
0.006 + 0.00
0.01 + 0.00
0.004 + 0.00
0.007 #: 0.00
0.001 0.00
5. PS 2
0.05 + 0.01
0,02 112-+00.00 0.01
0.03 + 0.02
0.07 - 0.15
0.01 0.01
0,002 1.002:00.000
0.003 0.00
0.00
0.001 0.00
Aero
0.003 0.00
0.005 4: 0.00
0.008 0.01
0.003
0.002 1: 0.00
Sythetic
0.001 0.00
0.007 : 0.00
0.002 - 0.00
0.00
0.00
0.01
0.007
0.00
0.001
8,
PS 2
0.06 + 0.03
0.04 + 0.02
0.09 + 0.04
0.10+0.02
0.01 0.01
Motion
0,003 0.90
0,005 : 0.00
0,009 0.00
0.001
0.001
Aero
0.00
0.00
0.004
0.003
Sythetic
0.006 1:
0.010 :: 0.00
0.002 :: 0.00
0,008 2 0.00
0.001 0.00
PS I
0,008
0.02
0.02 1.0.1.0.000 0.00
0.004 1. 0.00
0.01 1 0.00
1.000
2
0.07 0.03
0.04
0.03 0.02
0.07 0.05
0.02
Motion
0.001
0.007 : 0.00
0.006 0.00
0.001 0.00
0.001 + 0.00
Aero
0.003 0.00
0.007 1 0.00
0.01 1 0.00
0.003 1: 0.00
0.002 1 0.00
Figure 4: Performances of mapping recovery on different datasets under different testing scenarios.
are summarized in Fig. Although other methods can have low errors in some testing cases, they can not maintain the good performance as PNN-RISE does for all the datasets. Especially, EQL and SINDy's performance deteriorates when the problem size becomes larger, e.g., power system datasets with data from many nodes. While SVR is robust against noises, it can not recover the nonlinear physical function exactly to further decrease the learning error. In this case, the proposed method shows low learning error and stable performance in both noisy and data-limited scenarios.
Moreover, the performance of function recovery is compared in Fig. [5] (left). The bar plot shows the recovery rates (percentage of runs that correctly recover the ground truth symbolic expression) and the line plot with shades shows the recovery accuracy of the corresponding physical parame- ters. While EQL* + easily fails in large power system cases, PNN-RISE has high success rate in true function recovery, merely trapped into local optima. For physical parameter recovery, PNN-RISE not only has higher average accuracy but also has lower standard deviation than EQL.".
4.1 ABLATION STUDY FOR PHYSICAL CONSTRAINTS
In order to show the performance of using different physical constraints, we provide ablation study of the proposed NN. We first use the synthetic example to visualize the effect of constraints and show in Appendix. Fig. 5 (right) presents the learning results on power system datasets (IEEE 8-bus system). The performance is evaluated in different test scenarios: 1) regular data acquisition, 2) limited input range (voltages within [0.98, 1.01) for training and [0.90, 1.10) for testing), 3) out-
8

---

Under review as a conference paper at ICLR 2022
Parametts Rocevrry Acoundyy (50
WD Corstnants
100
0.2
W Params, Consraint
80
Inerta Condrain
Resovery Patr
0.15
40
A
20
Sythetic
Regular
Input range
Out-of-scale
Testing
Test Sconarios
Figure 5: Left: Comparison physics recovery. Right: Results embedding different constraints.
of-scale invariants in a governing function, and 4) limited data amount. Generally, adding physical constraints boosts the performance in different testing scenarios. The constraint mutual variables significantly reduces the learning error by pushing the features and parameters to be symmetric for multiple-output problems. When the data amount is very limited train NN efficiently, the bound on the inertia of system outputs in time sequence improves the function recovery.
4.2 ILLUSTRATION OF THE EFTECTIVENESS FORECASTING-BASED PHYSICS LEARNING
02
025
Mean square errr
ENR-100
PHNN 180
ReshoL 500-000
Foracast point number 2000
2000
2000
2500
2000
(a) Synthetic result.
(b) PS result.
(c) PS result.
(d) Motion result.
(e) Aero result.
Figure 6: The results of GP-based PNN and GP-based Resnet for different systems.
To illustrate the effectiveness of GP forecasting, we utilize 200 samples for training and propose to forecast (500, 1000, 1500, 2000} samples out of the training dataset. Also, we consider different SNRs in (50, 100, 150). The forecast samples are integrated into PNN model and other benchmark models for comparisons. Fig. [ddemonstrates the results with respect to the forecast points. Due to the space limit, we only show results of GP-based PNN and GP-based Resnet for comparisons. The patterns of results for other benchmark methods are similar to GP-based Resent. We observe that Ga-based PNN performs much better than Ga-based Resnet, which is consistent to observations in the previous subsections. As the number of forecast data points increases, the prediction MSE value generally decreases for both PNN model and Resnet model. It implies that our GP forecasting produces accurate forecast points, suitable for all prediction models. When the number is larger than 500, the results don't show a significant improvement. The reason that forecast points far from the training set boundary are with low confidence, which don't have big impact on learning results.
5 CONCLUSION
The lack of physical understanding makes it difficult for planning, monitoring, and control of IoE. To learn governing physical equations from data, the paper first presents a fundamental challenge for data-driven methods like symbolic regression and sparse regression. Data-driven only methods cannot guarantee to find the exact physics because many low-complexity expressions can perfectly fit the data. To resolve the issue, we propose complete set of constraints based on physical princi- developed.
ples to restrict search space. Thus, an efficient physical neural network (PNN-RISE) Finally, the FOCV supports the scenario when (partial) physical constraints are unknown. The fore- cast of data beyond the training dataset enhances model robustness with few extra costs. Results on physical system datasets show the superior performance of proposed methods.

---

Under review as a conference paper at ICLR 2022
REFERENCES
H. R. Baghaee, M. Mirsalim, G. B. Gharehpetian, and H. A. Talebi. Three-phase AC/DC power-flow for balanced/unbalancee microgrids including wind/solar, droop-controlled and electronically- coupled distributed energy resources using radial basis function neural networks. IET Power Electronics, 2017.
Erik Blasch and James Sung. Standards for the artificial intelligence community, Automatic Target Recognition XXXI, 2021.
Erik Blasch, Sai Ravela, and Alex Aved. Handbook of dynamic data driven applications systems. Springer, 2018.
Erik Blasch, Tien Pham, Chee-Yee Chong, Wolfgang Koch, Henry Leung, Dave Braines, and Tarek Abdelzaher. Machine caarnin/arttifiiial intelligence for sensor data fusion unsion-opportunitis and chal- lenges. IEEE Aerospace and Electronic Systems Magazine, 2021.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 2016.
Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 2019.
Sida Chen, Shigeru Imai, Wennan Zhu, and Carlos A Varela. Towards learning spatio-temporal data stream relationships for failure detection in avionics. In Handbbook of Dynamic Data Driven Applications Systems, pp. 97-121. Springer, 2018.
Zhao Chen, Yang Liu, and Hao Sun. Physics-informe learning of governing equations from scarce data. arXiv preprint, 2020.
Rohit Chowdhury and Deepak N Subramani. Physics-driven machine learning for time-optimal path planning in stochastic dynamic flows. International Conference on Dynamic Data Driven Application Systems, 2020.
Hwei-Ming Chung, Wen-Tai Li, Chau Yuen, Wei-Ho Chung, Yan Zhang, and Chao-Kai Wen. Local cyber-physical attack for masking line outage and topology attack in smart grid. IEEE Transac- tions on Smart Grid, 2018.
Andrew Cotter, Maya Gupta, Heinrich Jiang, Erez Louidor, James Muller, Tamann Narayan, Serena Wang, and Tao Zhu. Shape constraints for set functions. In International Conference Machine Learning. pp. 1388-1396. PMLR, 2019.
Yan Du and Fangxing Li. Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning. IEEE Transactions Smart Grid, 2019.
Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. Predicting AC optimal power flows: combining deep learning and lagrangian dual methods. In Proceedings of the AAAI Con- ference Artificial Intelligence, 2020a.
Ferdinando Fioretto, Pascal Van Hentenryck, Terrence WK Mak. Cuong Tran, Federico Baldo, and Michele Lombardi. Lagrangian duality for constrained deep learning. arXiv preprint, 2020h.
Breiffni Fitzgerald, Biswajit Basu, and Søren RK Nielsen. Active tuned mass dampers for control of in-plane vibrations of wind turbine blades. Structural Control and Health Monitoring, 20(12): 1377-1396, 2013.
Kenneth Gilbert. An ARIMA supply chain model. Management Science, 2005.
Jochen Gorski, Frank Pfeuffer, and Kathrin Klamroth. Biconvex sets and optimization with biconvex functions: survey and extensions. Mathematical methods of operations research, 66(3):373- 407, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
10

---

Under review as a conference paper at ICLR 2022
Geoffrey E Hinton, Simon Osindero, and Yee- Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 2006.
Julia Hirschberg and Christopher D Manning. Advances in natural language processing. Science, 2015.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks, 1990.
X. Hu, H. Hu, S. Verma, and Z. L.. Zhang. Physics-guide deep neural networks for power flow analysis. arXiv preprint, 2020.
Guang-Bin Huang, Lei Chen, Chee Kheong Siew, et al. Universal approximation using incremen- tal constructive feedforward networks with random hidden nodes. IEEE Transactions Neural Networks, 2006.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. Proceedings of the IEFF conference on Computer Vision and Pattern Recognition, 2014.
Samuel Kim, Peter Y Lu, Srijon Mukherjee, Michael Gilbert, Li Jing, Vladimir Čeperić, and Marin Soljačić. Integration of neural network-based symbolic regression deep learning for scientific discovery. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 2012.
Haoran Li and Yang Weng. Physical equation discovery using physics-consistent neural network (penn) under incomplete observability. In Proceedingy of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 923, 2021.
Jiangnan Li, Yingyuan Yang. Jinyuan Stella Sun, Kevin Tomsovic, and Hairong Qi. Conaml: Con- strained adversarial machine learning for cyber-physical systems. Proceedings of the ACM Asia Conference Computer and Communications Security, 2021.
Xiaoxia Li, Wei Li, Qiang Yang. Wenjun Yan, and Albert Y. Zomaya. Edge-compuuing-enbble unmanned module defect detection and diagnosis system for large-scale photovoltiic plants. IEEE Internet of Things Journal, 2020.
Yifeng Li. Sparse representation for machine learning. Advances in Artificial Intelligence, pp. 352-357, 2013.
Anji Liu and Yuanjun Laili. Balance gate controlled deep neural network. Neurocomputinng. 2018.
Yanli Liu, Dongxu Lu, Liangchen Deng, Tianyuan Bai, Kai Hou, and Yuan Zeng. Risk assess- ment for the cascading failure of electric cyber-physical system considering multiple information factors. IET Cyber-Physich Systems: Theory & Applications, 2017.
Dengsheng Lu and Qihao Weng. A survey of image classification methods and techniques for improving classification performance. International journal of Remote Sensing, 2007.
Katarzyna Maciejowska, Jakub Nowotarski, and Rafat Weron. Probabilistic forecasting of electricity spot prices using factor quantile regression averaging. International Journal of Forecasting, 2016.
MATPOWER. MATPOWER. 2020.
Duane T McRuer, Dunstan Graham, and Irving Ashkenas. Aircraft dynamics and automatic control, volume 740. Princeton University Press, 2014.
Kursat Rasim Mestav, Jaime Luengo-Rozas, and Lang Tong. State estimation for unobservable distribution systems via deep neural networks. In 2018 IEEE Power & Energy Society General Meeting. IEEE, 2018.
Tristan Milne. Piecewise strong convexity of neural networks. arXiv: 1810. 12805, 2018

---

Under review as a conference paper at ICLR 2022
Prakash M Nadkarni, Lucila Ohno-Machado, and Wendy W Chapman. Natural language processing: an introduction. Journal of the American Medical Informatics Association, 2011.
Antonia Nasiakou, Miltiadis Alamaniotis, Lefteri H Tsoukalas, and Manolis Vavalis. Dynamic data driven partitioning of smart grid using learning methods. Springer, 2018.
Brenden K Petersen. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint, 2019.
PJM Interconnection LLC. Metered load data. 2018.
pim. com/
Feed/hrl_load meteered/defiiitioon
Joaquin Sainonero-Candell and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. The Journal of Machine Learning Research, 2005.
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. International Conference on Machine Learning. 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 2020.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 2009.
Bernhard Schölkopf, Alexander J Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Kevin Small, Byron C Wallace, Carla E Brodley, and Thomas Trikalinos. The constrained weight space SVM: learning with ranked features. In International Conference on Machine Learning, 2011.
Adi L Tarca, Vincent J Carey, Xue-wen Chen, Roberto Romero, and Sorin Dräghici. Machine learning and its applications to biology. PLoS computational biology, 2007.
Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. Differentially private and fair deep learning: A lagrangian dual approach. arXiv preprint, 2020.
Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. arXiv preprint, 2018
Silviu-Marian Udrescu and Max Tegmark. AI feynman: A physics-inspired method for symbolic regression. Science Advances, 2020.
Arjan van der Schaft. Modeling of physical network systems. Systems & Control Letters, 2017.
Alexandre Velloso and Pascal Van Hentenryck. Combining deep learning and optimization for security-constrained optimal power flow. arXiv preprint, 2020.
Kieran Walshe, Gill Harvey, and Pauline Jas. Connecting knowledge and performance in public services: From knowing to doing. Cambridge University Press, 2010
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang. and Xiaoou Tang. Residual attention network for image classification. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017a
Jianxiao Wang, Haiwang Zhong, Ziming Ma. Qing Xia, and Chongqing Kang. Review and prospect of integrated demand response in the multi-energy system. Applied Energy, 2017b.
Jiafan Yu, Yang Weng, and Ram Rajagopal. Robust mapping rule estimation for power flow analysis in distribution grids. North American Power Symposium, 2017.
Abdelhafid Zeroual, Fouzi Harrou, Abdelkader Dairi, and Ying Sun. Deep learning methods for forecasting covid-19 time-series data: A comparative study. Chaos, Solitons & Fractals, 2020.
Wen Li Zhao, Pierre Gentine, Markus Reichstein, Yao Zhang, Sha Zhou, Yeqiang Wen, Changjie Lin, Xi Li, and Guo Yu Qiu. Physics-constraneed machine learning of evapotranspiration Geo- physical Research Letters, 2019.
12

---

Under review as a conference paper at ICLR 2022
A FIGURES
Solutions
Case
Physical
2:2g ogd3
U.lat U.dagas
dpts djit
0.00
y 0.02441123
1.31922 gaaa +1. +1.1106x) of
1.20 ag
1.80662 of ust
Local
2.079ke (940
2008 403 rard of
0.3481 dozo a
Optinum
ttug rase ro13
9 L.205Z 14225 this san
V= 2.6188 153 uur gram
10.44 2nr
1002 122 1,090 f sats fane
rass
Local
+0.9600x ham
2 hozn 300
+144884a shnn shst, a
-9 9.84562 oring Tine
Openumm 2
10000
0:001
D.00 USMUTTE
Bo0d
0,00 2 21000 9000
Global
1000 5900
0.5100u 1000 2000
Jl 5000 unw 1000
Optnnm
0.006
0.001
C 0018
Figure 7: Absolute mismatches between the recovered functions and the physical ground truth.
Sparse Fitting May not be Physical Verifiable For example, to learn the function y = 22 + 2223 from data, different SR methods can easily get trapped in local optima. To see if these observation is special, we test different cases, where Fig. shows a table summary of four different cases, with different coefficients. We also test different numbers for multiplication and summation. From all examples, one can see that the recovered objective has sparse coefficients and negligible error (loss objective L) at non-desirable local optimums. Specifically, the neural network naturally approxi- mates towards reducing the learning error. Therefore, it could ignore the consistency with physical governing functions and update in the wrong direction, leading to overfitting and the local optima solution. Differently, for the last row, the learning parameter of 0.990 (or 1.980) is close to ground truth (or 2), which means "Near Global Optimum" that has 99% accuracy of parameter recovery. However, the prediction error (highlighted in red) is larger than direct approximation of neural network (highlighted blue), where the parameters are far away from the ground truth.
Gloobal Optimum
Smooth
Near Global Optimum Soddle Point
100
tanh (wis) aw
Enforce 1 or -1
Physical Constraints:
Smooth
avoid non physical
Wy
50
global optimum
tanh(x) 2o (2x)
Borth Eriors
1.5
-0.5
0=
one to
0.5
tanh(wg)
1 or
10
0.5
1.5
-0.
Wij
V3
Wyy
Wy
Figure 8: (Left:) Illustration of the possible saddle point, local optimum, and global optimum in physical equation learning. We observe that local optima can perform numerically as well as the global optimum (ground truth equation) since there is always a set of diversified equations to per- fectly fit limited data. (Right:) Visualization of the arithmetic layer design function.
Test Sparse Fitting for Power System Identification. To make the realistic case study rather than toy examples, we use power system case study and visualize the objective surface in Fig. (left). Similarly, for a small parameter range, there are points achieving the global optimum or local optima that are very close to the global optimum. However, only one of them is the physical ground truth. For example, one can go from the red point to a non-physical solution with an error close to zero, see that "good"
when the sparsity regularization is enforced. From the observation, sparse fitting is insufficient to avoid overfitting in learning physical equations.
Visualization of Neural Arithmetic Design in PNN. We plot the surface of the neural arithmetic design in Fig. 8 (right). The visualization shows that the multiplication of tanh and sigmoid functions not only generates smooth optimization surface but also naturally results in limited parameters.
Illustration of RISE Constraints on PNN. For illustrating the completeness of these principles, Fig. 9 presents the relationship among the four RISE concepts.
13

---

Under review as a conference paper at ICLR 2022
Range (R)
Inertia (1)
Symmetry (S)
Extrapolation (E)
A
+
O
Sample
Weight Gating
+ A +
O A O
Mechanism
+
O
X
AS
1
[(xx))
Wpmine S W S Wamax
(x
f(xx)ll
11 W3'll2
agPR 8k
K=1
Figure 9: RISE principles for physical laws.
B PROOF OF THEOREM
Proof. First, we define the biconvexity as follows (Gorski et al. 2007).
function é A B IR is called biconvex if fa, is convex in for fixed b E B, and fa. b) is convex in b for fixed a E A.
Definition 1 (Biconvexity).
Then, we prove PNN is biconvex. Recall for the PNN function y= f(x),, the hthh element can be written as: yk Ej myk22 = E, mik exp E Piy In(x;)). Vj. k. if myR is known, we need to prove the convexity of f with respect to Pij, LA S.
Since the linear combination Σ Ptj (n(x;) is convex with respect to pij, and the exponential func- tion is convex, the convex function composition implies that exp (Ei=1 Pij In(rr)) is convex with respect to Pij. Therefore, Vj, k, if mys is known, the function yı = -, mijk exp (E Pij In(x;)) is convex with respect to Pij, V1 < i S N. On the other hand, Vi,j. if Pıj is known, E, mij exp (EN Pij In(x;)) is linear combination of myk- Therefore, yn is convex with re- spect to mijk given pij-
N
Therefore, PNN function y = f(x) ) is biconvex with respect to the logarithm layer and the linear summation layer. The biconvexity can lead to efficient alternate convex search (ACS) (Gorski) iet al. 2007) to seek for optimal solutions. Namely, ACS will fix one part of parameters to search for the other part iteratively. Due biconvexity, in each iteration, ACS will solve a convex problem. Then, Corollary 4.10 in (Gorski et al. 2007) guarantees that the searched points will converge to stationary points of (xx).
C DETAILS OF DATASETS AND EXPERIMENT SETUPS
Synthetic dataset: generating random samples of from Gaussian distribution (O, 1), and obtain the outputs by C + C
Power system datasets: IEEE provides standard power system models, including the grid topol- ogy. parameters, and generation models, etc., for simulations. The model files and the simulation platform, MATPOWER (MATPOWER 2020), are based on MATLAB. For simulations, the load files are needed the inputs the systems. Thus, introduce power consumption data from PJM Interconnection LLC (PJM Interconnection LLC, 20188). Such load files contain hourly power consumption in 2017 for the PJM RTO regions. With the above data, MATPOWER produces the system states of voltages and nodal power injections. Then, we utilize the voltage as input and the nodal power as the output learn the physical mapping, for which the true physical function used for evaluations Yu et al. 2017). To diversify the system types, we utilize the mass- system and aerodynamic system to further validate our results.
Motion dynamics dataset: mass dampers of in-plane wind turbine. The physical equation of the mass-damper system representative of aircraft motion for analysis of system dynamics (van der Schafi 2017: McRuer et al., 2014). Moreover, the active tuned mass dampers are implemented to control in-plane blade vibrations (Fitzgerald et al. 2013). Using MATLAB, we simulate the dy- namic process of the mass- dass- system with 10 nodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.
Aerodynamic dataset: aircraft weight estimation: The aircraft sensor data is collected from the
14

---

Under review as a conference paper at ICLR 2022
aircraft simulation in MATLAB and the corresponding aerodynamic model is illustrated in (Chen et al. 2018). In this case, the goal is to recover the aerodynamic model of gross weight and angle of attack in cambered airfoil.
For the implementation machine learning methods, the prepared data samples are split as 60% for training the model, 15% for validation of model hyperparameters, and 25% for testing. Specifically, hyperparameters of the benchmark methods are selected through k-fold cross-validation. For SVR, the kernel is chosen from the candidates that include polynomial kernel (2nd-degree and -ed-eegree), radial basis function (RBF) kernel, and hyperbolic tangent kernel. The box parameter is similarly chosen from {1 10 10 1 10 2,5x 10 x 10 5 10 100, 5 x 10 x 10
D RESULTS OF THE SYNTHETIC DATASETS
Add Symmetric Coupling Constraint
Function Recovery
Narrower
Local Ground
Narrower
Wide Optimum Truth
Smoother Surface
Add Range
Constrain
Near Zero Gap
Large Gap
Smaller Gap
Smaller Gap
Ground
Ground Truth
masane peeeeedd ao aeeee pes Add
Figure 10: Synthetic dataset: visualization of the function recovery with RISE constraints.
In order to show the performance using different physical constraints, we provide an ablation study of the proposed NN. We use the synthetic example visualize the effect of constraints. The left-hand side of Fig. compares the local optimum (transparent gray surface) learned by NN with the true function (solid gray surface). Though there some overlaps, the large gap between the two surfaces causes large errors for generalization. By adding constraints, we observe in the middle of Fig. [0 that the variable coupling constraint rotates and stretches the surface the correct shape like the maximum and minimum values of the surface. On the other hand, the parameter range constraint smoothes the surface the local optimum. Together, the constraints push the recovered function (cyan surface) towards the ground truth, where the overlap in Fig. [T0 (right) shows the physical exactness in not only feature components but also the corresponding parameters.
15


Reviews:
[{"review_id": "w6gs0HNmrWO", "reviewer": "Reviewer_8Azy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed neural network is of very specific design resulting in the weighted sums of products of original inputs (and their integer powers). Sparsity in parameters is enforced using the tanh and sigmoid transformations of weights. Additionally, model parameters are constrained by domain knowledge induced physical principles like penalties on symmetry and smoothness and limits and inequalities of parameter values. Final piece of the framework is regularization term based on outside-of-the-sample predictions. Even though the individual tricks are likely used before, to the best of my knowledge this particular combination/instance of the neural network is unique so far. The empirical evaluation is sound and shows superior fit and physics recovery. Just wondering why is 0.990 (or its integer multiple like 1.980) \"dominating\" the Near Global Optimum row of the Figure 2.?  The article is quite descriptive and detailed. Just I think the ablation study would better serve as a part of the main article, and instead of the visual - some quantitative measure of contribution of each of the piece would be appreciated. Text is clear, with just few typos like \"bus actual knowledge\" or \"different as active learning\".\n\nUPDATE\nAfter insight into other reviewers' comments and authors' responses, I tend to agree with concerns on theoretical soundness of certain aspects and need for more appropriate experimental benchmarks. Therefore I am reducing the initial score.", "summary_of_the_paper": "A framework for learning succinct and humanly interpretable descriptions of physical systems from data is outlined. The main focus of the approach is eliminating/avoiding local optima, which is achieved through both architectural design of the model and modification of learning algorithm. Empirical evaluations conducted on simulated data suggest improved error metrics as well as recovery of ground truth equations.", "summary_of_the_review": "I think that study described in this paper would be beneficial for the conference readers, and especially the complex systems modeling community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JA0EXL03qs2", "reviewer": "Reviewer_Lobz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n1. This paper shows a nice connection between machine learning models and a real-world application, i.e., estimating physical equations.\n2. This paper has been well motivated. Because directly using a deep neural network, physical equations might not be learned well, due to a number of reasons, such as too many local solutions, as shown in Figure 2.\n3. The experiments have been done pretty comprehensively, including those on synthetic data and those on real physical systems.\n\n### Weakness\n1. Some details of the experimental settings are missed. For example, in experimental section support vector regression (SVR) is used as baseline. What kernels are used in SVR? and other parameters, such as the box parameter?", "summary_of_the_paper": "This paper studies the problem to learn physical equations by neural networks. To address the limitations of traditional methods using sparsity to learn physical equations, this paper proposes to use the physical neural network as a container, with additional constraints to explore “Range, Inertia, Symmetry, and Extrapolation (RISE)”. Using the proposed method, it is expected that less local minima will be explored for better solution. Experiments have been conducted to support the presented method.", "summary_of_the_review": "This paper solves the problem of estimating physical equations by a physical neural networks (PNN), with the improvements to take into account “Range, Inertia, Symmetry, and Extrapolation (RISE)” that are important property of physical equation learning. The paper is well motivated and the method has been clearly described. Experimental results on both synthetic and real data support the presented method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FLVplgjTQq1H", "reviewer": "Reviewer_VZaq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n- The problem tackled is very important and indeed not studied as much as it should by the community.\n- The experiments showing that existing approaches fail to recover the underlying equation structure for simple systems illustrates this well.\n- The idea of relaxing the polynomial layer of PNN with tanh and sigmoid functions is very interesting, appealing and seems original to me.\n- The idea of proposing a unifying formalism for physical constraints such as RISE is interesting and should be helpful.\n- The proposed method is show-cased in a number of diverse experiments and compared to relevant SOTA methods.\n\n### Weakness\nIn my opinion, despite these positive points, the article has major weaknesses in terms of theoretical and experimental soundness and clarity. Given the multiple clarity concerns, it is possible that I misunderstood some things, so please let me know it this is the case. Here follows my more detailed concerns.\n\n**Concerning the proposed method**\n1. It is not clear to me how you go from outputs in the  {-1, 1} range to the {-K, … K} range.\n2. Furthermore, I must say that the Laurent polynomial structure of the method proposed does not seem expressive enough to model many important physical systems, which do not have a polynomial structure. This seems to me as a weakness compared to previous methods such as EQL and SINDy, which are built on top of an arbitrary dictionary of basis functions (trigonometric functions, exponentials, etc.). In this sense, the hypothesis space of PNN is included in SINDy and EQL ones.\n\n**Concerning the RISE penalty**\n\n3. Concerning the RISE constraint, while the formalism is interesting, its motivation lacks illustrative examples other than the power line example. Where else do we see the symmetry constraint and how are all these constraints implemented in the 4 experiments presented in the paper? How are constrained weights chosen? For example, I imagine you don’t symmetrize all weights, do you? Furthermore, the constraints presentation is not well connected to existing penalties from the statistical learning literature. For example, inertia constraint is usually modeled by smoothness inducing regularization (e.g. smoothing splines [1], dynamic system constraints like in system identification [2]).\n4. Also, I was a bit surprised and confused because in section 3.1 (second paragraph) it is said that existing methods such as SINDy add additional hyperparameters (LASSO penalty weight) and that the proposed method does not have this flaw. But the RISE penalty also has a penalty hyperparameter to be tuned.\n\n**Concerning the theoretical results**\n\n5. It is said just before proposition 1, on page 5, that \"such a design converts the non-convex multiplication to a convex form of linear summation for NN training\". I imagine that this refers to the exp-sum-log rewriting of the polynomial $z_j=\\prod x_i^{p_{ij}}$ at the end of page 4 as $\\exp(\\sum p_{ij} \\ln x_i)$. Am I correct? If this is the case, I cannot agree with the statement as these two expressions are exactly equal and hence, both convex in $p_{ij}$ (they are the same function). There seems to be a confusion here because $z_j=\\prod x_i^{p_{ij}}$ is non-convex in $x_i$. Please, let me know if I missed something here.\n6. Concerning proposition 1, the \"searching algorithm\" is not defined (one has to go to the appendix to understand that you mean alternate convex search).\n7. More importantly, I cannot agree with the results or at least not with its relevance here. Indeed, in addition to the linear weights $m_{jk}$, the optimization variables of the proposed PNN model are not the $p_{ij}$’s, but rather the auxiliary $w_{ij}$ and $w’_{ij}$ weights, and it happens that the problem is not biconvex wrt them (as evidenced clearly by figure 7). I do agree however that the problem is biconvex wrt $p_{ij}$’s and $m_{jk}$ and that ACS would work if one would optimize these variables instead. However this is not what the paper proposes.\n8. At the end of the proof of theorem 1 (page 7) you state that \"the piecewise strong convexity from RISE regularization guarantees that we can find the global optimal point for each sub-regions, and the global optima satisfies constraints for physical parameters\". While you proved indeed the strong convexity and while this implies that the global minimum can be achieved with gradient descent, the optimization problem solved has been relaxed and, as such, has no hard constraint imposed, but rather a penalty term. Hence, nothing guarantees that the problem solution respects the constraints (not true for low values of $\\lambda$ for example).\n9. Also inside the proof of theorem 1 you replace the L0 norm in the symmetry constraint by a L1 norm, but this is not said in the theorem statement. For clarity (and rigor), you should define the symmetry constraint directly in terms of L1 norm, saying that it is relaxing an ideal L0 norm which is not used in practice. The proof is not the right place to do the switch.\n\n**Concerning the experiments**\n\n10. As evidenced by sentences like: \"Subsequently, can we find the global optima and can it represent the true physics?\" the paper seems to imply that physical solutions correspond to global minimum (of the training loss?) and that local minima are bad solutions. This is however not really shown in experiments. Actually, figure 2 seem to indicate the opposite: physical models have higher loss.\n11. Overall, the experimental settings and protocols of all results are not sufficiently explained, which makes it impossible to interpret and assess them. For example, in figure 2, what are the hypothesis spaces considered? What method is used to obtain those scores (SINDy, EQL, …)? How are hyperparaameters selected (specially important as this is highlighted as a key pain that the proposed method should alleviate)? Same goes for figure 7 in appendix A.\n12. Also, while you say at the end of page 3 that $\\mathcal{L}$ is the loss, it is unclear in figure 2 whether the reported $\\mathcal{L}$ is the training loss, the test loss, the cross-validated loss, … It is also never stated how many data points are used and how the data is split.\n13. Experimental details are also highly insufficient in figure 4: how is the data split (could not find even in appendix)? Do the scores correspond to the training or testing error? How is optimization carried in Resnet, PNN-RISE (optimizer, number of iterations, batch size, learning rate, definitions of convergence, etc.) What dictionary of basis functions is used for SINDy and EQL? Does it include the true structure? And yours?  How are hyperparameters tuned? These details are very important to assess the fairness of the comparison here. It is indeed difficult to understand what in the proposed method allows it to beat the others given that its hypothesis space is theoretically included in the others. Maybe the RISE constraint or the training procedure? The experiments don’t really help to answer this question.\n14. Also, what is the difference between PS1/PS2 (not explained in the appendix).\n15. How many runs are used to compute the recovery rates?\n16. You say that \"EQL÷ easily fails in large power system case\" but you never state the dimension of the problems considered.\n17. In figure 5, a \"limited data\" case is considered, but it is not explained what this means exactly? How much less data is considered and how much did we have in the beginning? Also, what is the \"out-of-scale invariants\" case?\n\n**Other clarity concerns**\n\n18. It is confusing to say in the last paragraph of page 3 that \"with integer coefficients or not\" while the PNN model with integer coefficients was not introduced yet\n19. Likewise, the proof of theorem 1 seems to be the first time the reader learns that the loss considered in the paper is always the MSE.\n20. What is meant by \"we complete the function y = f(x)\" on top of page 5?\n\n\n[1] Green, P. J.; Silverman, B.W. (1994). Nonparametric Regression and Generalized Linear Models: A roughness penalty approach. Chapman and Hall.\n\n[2] L. Ljung. System Identification: Theory for the User. Prentice-hall, 1987.", "summary_of_the_paper": "This paper tackles the problem of model inference and prediction, while preserving physical correctness. This is an interesting and important problem in supervised learning, for interpratability and generalization reasons. It is also quite important in model-based reinforcement learning, in the context of nonlinear system identification.\n\nThe authors start by challenging existing sparse regression approaches (such as SINDy) and symbolic regression approaches (such as EQL):\n- Because they lead to non-physical solutions with low loss,\n- Because they depend on hyperparamters\n\nHence they propose what is coined a Physical Neural Network (PNN), where:\n-  layers have a Laurent polynomial shape with learnable coefficients and powers,\n- weights corresponding to the layer input powers are decomposed into a tanh activation multiplied by a sigmoid activation.\n\nThis last design choice allows to enforce some kind of sparsity (when sigmoid converges to 0) and constrains powers in the range {-1, 1} when active (because of tanh). Furthermore, it ensures differentiability of the integer valued layer-input powers.\n\nAdditionally, the authors also propose to penalize the model training with what they call a RISE constraint. The latter is meant to enforce physically plausible range, inertia, symmetry and extrapolation properties in the inferred model.", "summary_of_the_review": "Although the paper highlights weaknesses of existing approaches for a very important learning problem and proposes interesting novel modeling ideas, it has major weaknesses in terms of theoretical and experimental soundness and clarity. Hence, I cannot recommend it for publication at its current state.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F-z-uMB5yv1", "reviewer": "Reviewer_XyFH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Model parsimony (often synonymous with sparsity but not necessarily) has been an overarching goal in the physical sciences. Ensuring that the model is interpretable and physically consistent is also crucially important. While this has been widely recognized in the system identification community, it is my impression that this problem is not studied as much as it should in the machine learning community. The contribution by the present authors goes in this direction.\n- The design of the polynomial layer in PNN, combining sigmoïd functions for sparsity and hyperbolic tangent for integer exponents is interesting and, to the best of my knowledge, is original.\n- Although I don't find it flexible enough, the RISE framework is a step in the good direction to enforce realizability conditions in the model identification procedure.\n\n## Weakness\n\nDespite these positive points, this contribution suffers from certain limitations. The most important ones are discussed below.\n\n### Concerning the proposed method\n- The architecture of the network implicitly assumes that the unknown function $\\mathbf{f}$ is a polynomial. What if it is not (e.g. the nonlinear pendulum or the Kuramoto model for oscillator synchronization)? Does it identify the Taylor (or Laurent or Padé) series expansion of $\\mathbf{f}$?\n- What if the system is only partially observed? Does the identified model include closure terms?\n- Are $c_1$ and $c_2$ in the sigmoïd and hyperbolic tangent functions free parameters or are they actually hyperparameters?\n\n### Concerning the RISE penalty\n\nDespite what is stated in the introduction, PNNs are not restricted to systematically obeying the physical properties and actually have hyperparameters that need to be tuned! The constraints are weakly imposed by penalizing the cost function. Hence, the coefficient for this penalization is a hyperparameter and, if the latter is too small, then the identified model will not necessarily satisfy the constraints.\n\nMoreover, this framework is not as universal as what is claimed by the authors, in particular, the Symmetry part:\n- Symmetry in phase space does not necessarily imply symmetric coefficients in the equations. This is easily exemplified by the harmonic oscillator $\\dot{x} = y$ and $\\dot{y} = - x$ where the coefficient are anti-symmetric despite the trajectory in phase space being a circle.\n- Rather than focusing on symmetric coefficients, a better approach would be to enforce equivariance, i.e. dynamics of the form $\\mathbf{S} \\dot{x} = f(\\mathbf{S} x)$.\n- Although I cannot remember of any proof, I believe that symmetry in the coefficient actually implies some form of conservation, something which might be useful for the power systems considered in this contribution but is too restrictive in general.\n\n### Concerning the theoretical results\n\nMost of my concerns have already been raised by reviewer VZaq.\n\n### Concerning the experiments\n- The test cases used for the comparisons are non-standard (at least from my system identification point of view). Although it may have been done out of simplicity by the authors, using poorly documented benchmarks is a surprising choice when introducing a new method.\n- Experimental details are insufficient. How much data is used? How is it split? How are Resnet and PNN trained (optimizer, number of epochs, batch size, learning rate, etc)? Which functions are included in the library for SINDy and EQL? Which algorithm/formulation of SINDy has been considered?\n- While the method proposed by authors explicitly takes into account some constraints, only the vanilla version of SINDy has been considered (to the best of my understanding). Yet, as shown in [1, 2], SINDy can be extended to incorporate physical constraints in the model identification problem. Using SINDy with constraints would thus be a somewhat more fair comparison.\n- System identification techniques are often used in the low-data limit (either because simulating the high-dimensional system for a sufficiently long period of time would require millions of computing hours or because data is gathered from extremely large experimental facilities that cannot be operated for long). How does the method behave in this low-data limit (e.g. when you have fewer data points that free parameters in the neural network)?\n- Numerous systems have rapidly attracting manifolds in phase space which express themselves as nonlinear correlations in the time series. Consider the following model of self-sustained oscillators\n$$\\dot{x} = \\sigma x - y - xz$$\n$$\\dot{y} = x + \\sigma y - yz$$\n$$\\dot{z} = -z + x^2 + y^2$$\nwith $\\sigma > 0$. It is easy to show that, very rapidly, $z(t) \\simeq x^2 + y^2$, i.e. despite the negative term in the third equation, $z(t)$ actually grows over time because of nonlinear coupling. Given only data, simple techniques would identify the following equation for $z(t)$\n$$\\dot{z} = \\alpha z - \\beta z^3,$$\nwhich is obviously wrong (although consistent with the data). How would your technique behave in such a situation where nonlinear cross-talk between variables is what drive the dynamics at first order?\n\n### Miscellaneous\n-  The quality of the English should not be taken into account when assessing the scientific quality of a contribution. Yet, there are a lot of typos in the paper that could have been avoided using simple spellcheck.\n- It is unclear until relatively late in the paper that the loss function considered is the mean squared error. Although I do understand that a different metric could be used during the training stage, I think that making this clear early on is needed.\n\n### References\n[1] Loiseau & Brunton. Constrained sparse Galerkin regression. Journal of Fluid Mechanics, 2018\n\n[2] Kaptanoglu et al. Promoting global stability in data-driven models of quadratic nonlinear dynamics. Physical Review Fluid, 2021.", "summary_of_the_paper": "Robust methods for equation discovery respecting physics are of utmost importance for the application of data-driven methods in science en engineering. The present contribution proposes a new framework based on Physical Neural Networks (PNN) :\n- it is assumed that the function $\\mathbf{y} = \\mathbf{f}(\\mathbf{x})$ is polynomial (with possibly negative exponents),\n- the network has only one hidden layer (as far as I understood),\n- this hidden layer takes as input the state variables $x_i$ of the system and outputs $z_j = \\exp \\left( \\sum p_{ij} \\textrm{ln}(x_i) \\right)$,\n- In the above expression, $p_{ij} = \\textrm{tanh}(c_1 w_{ij}) \\cdot \\sigma(c_2 w^{\\prime}_{ij})$ with $\\mathbf{W}$ and $\\mathbf{W}^{\\prime}$ parameters to be learned.\n- The final output layer is a simple weighted sum, i.e. $y_k = \\sum_j m_{jk} z_j$.\n\nSparsity is of the model results from the use of the sigmoïd activation function in the hidden layer while the hyperbolic tangent makes sure $p_{ij}$ is an integer (either positive or negative).\n\nAlong with this simple architecture, the authors propose the so-called RISE framework to include realizability conditions (e.g. physical constraints) the model needs to satisfy. These constraints are enforced only weakly by penalizing the loss function to be minimized in the training state. The overall model is tested on a number of non-standard benchmarks to illustrate its capabilities.", "summary_of_the_review": "Robust methods for equation discovery respecting physics are of utmost importance for the application of data-driven methods in science en engineering. Although the method presented in this contribution seems interesting, its presentation lacks clarity. As discussed in the core of my review, comparisons with already existing techniques are biased (e.g. SINDy with constraints is not considered) and are not presented for representative cases or well-tested benchmarks. Other issues are also discussed in the review. Hence, I cannot recommend this work for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7H6X5QTxBJ8", "reviewer": "Reviewer_7VKA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the carried out work seems interesting, the paper does not allow to clearly highlight the contributions due to a partial presentation. The paper would have benefitted from a comprehensive presentation that provide the tangible elements to explain all the building blocks of this work. For instance, the so-called physical neural network (PNN) is presented in terms of the elements defining it. However, it is not clear its overall architecture, and most importantly the optimization method. While Proposition 1 give some elements, about the biconvexity and “the searching algorithm that can find the stationary points of PNN”, it is never explained what is the used algorithm. Moreover, in the description of the method (Section 3), we can find some incorrect information, such as “p_ij is the index number of x_i” (it is its power), “summation/abstraction”, … Finally, sentences like “After providing a flexible design to limit possibilities of local optima” are not founded with theoretical proofs.\n\nThe title is not appropriate. This paper is not about sparse representation learning, which is a more general topic than the one addressed in this work. It is on learning  physical equations with symbolic regression. Moreover, the authors never addressed the question in the title of this paper.\n\nFrom the beginning of the paper, the authors emphasize on the main motivation of this work, which is Internet of Everything (IoE) that becomes IoT in page 2. However, almost nothing in the paper is related to this motivation, as all developments are not related to IoE, as well as most experiments.\n\nThe experiments are not convincing. The authors compare the proposed method to 4 other methods, including 2 methods that are not for symbolic regression (SVR and ResNet). The only used methods that are related to this work are SINDy (sparse identification of nonlinear dynamics) and equation learner. The authors need to provide a comprehensive experimental analysis, with a comparative analysis on several recent methods from the literature, such as:\n- Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707.\n- Lusch, B., Kutz, J. N., & Brunton, S. L. (2018). Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1), 1-10.\n- Raissi, M. (2018). Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1), 932-955.\n- Udrescu, S. M., & Tegmark, M. (2020). AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16), eaay2631.\nSee also: Willard, J., Jia, X., Xu, S., Steinbach, M., & Kumar, V. (2020). Integrating physics-based modeling with machine learning: A survey. arXiv preprint arXiv:2003.04919, 1(1), 1-34.\n\nThe numbering in the body text does not correspond to the one in figure 1, making it difficult to understand.\n\nThere are some spelling and grammatical errors that can be easily identified and corrected, such as “we propose an new training “, “To see if these observation is special”, “bus actual knowledge”, “proof is inlcuded in Appendix”, “for each sub-regions”, “by combing their initials”, “leads an important extrapolation”, “the data in represented as”,  “our method is different as active learning”, “Visualization of the layer design equation ??”, “RISE Principals for Physical Laws”, “the load files are need as”\n", "summary_of_the_paper": "This paper is about learning  physical equations with symbolic regression with neural networks.", "summary_of_the_review": "We think that this paper is not of sufficient quality to be accepted in ICLR, for at least the reasons mentioned in the Main Review section.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The topic and ambition of this paper has been judged as important by all reviewers. Yet there is
a consensus that the theoretical and experimental contribution is not strong enough to effectively
argue for an important novel lead which would justify publication at ICLR. For these rejections,
this paper cannot be endorsed for publication at ICLR 2022.
Decision: Reject