WHAT TO DO IF SPARSE REPRESENTATION LEARNING FAILS UNEXPECTEDLY?

1 INTRODUCTION . Internet of Everything ( IoE ) connects new edge devices into an intelligent web at an unprecedented speed . If utilized systematically , IoE can create much more efficient productivity via coordinated effort among humans , processes , data , and things ( Li et al. , 2020 ) . Among various analytical frameworks , dynamic data-driven applications systems ( DDDAS ) naturally interweaves diversified information via data-driven analysis to assimilate the system model for a wide range of applications ( Blasch et al. , 2018 ) , thus significantly increasing the performance of both the simulation and application models . The simulation model usually incorporates physical methods ( Nasiakou et al. , 2018 ; Chowdhury & Subramani , 2020 ) which provides a structure for model-based and data-driven coordination . However , many of the physical laws in the web of IoE can be unknown , which remain to be discovered manually or automatically to enable down-streaming applications with accurate modeling . The most economically efficient way of physical law discovery is to use machine learning to analyze the pattern behind the data stream , which is an important topic ( Sahoo et al. , 2018 ; Udrescu & Tegmark , 2020 ) . In previous studies , such as symbolic regression ( Petersen , 2019 ) , the sparsity representation is used to measure the accountability of physics ( Brunton et al. , 2016 ) . The underlying assumption is that sparsity can ensure physical exactness due to parsimonious principle ( Li , 2013 ) . While low complexity of learning models is a necessary condition for enhancing the understanding of physical laws ( Schölkopf et al. , 2002 ; Blasch et al. , 2021 ) , there is not a universal sufficient condition to solve model recovery problem with a guarantee . Therefore , one approach is to add additional conditions to ensure a physical recovery , e.g. , near-zero fitting errors ( Champion et al. , 2019 ) or/and cross validation ( Kim et al. , 2020 ) . However , it can be observed that different models easily fit the available data with a sparse but wrong model , which can happen even with cross-validation . The overfitting should challenge the directions of many researchers in physical system discovery , who believe that only the true physical laws can achieve sparse recovery with a perfect fit . To resolve the problem , we first eliminate non-physical local optima of the neural network ( NN ) by discarding all the non-physical parameters , i.e. , model parameters indirectly related to the physical representation . The process is novel as existing NN-based methods require certain non-physical parameters to control the representation sparsity ( Liu & Laili , 2018 ; Petersen , 2019 ) . We show how to design a NN with pure physical parameters , entitled a physical neural network ( PNN ) . The key of PNN is to enforce activation to physical weights for sparse selection , completely removing the non-physical parameters and local optima . In training the PNN , there can still be local optimum solutions , the actual knowledge of physical system can enhance model fidelity . To obtain the ground truth , physical system examination can be used to extract common and universal properties . Subsequently , the PNN is restricted to systematically obey the physical system properties . Specifically , constraints based on the physical distance and symmetry are embedded in the training function to quantify the spatial relationship enforced by physical law . Furthermore , as physical systems evolve over time , inertial property for physical systems are extracted and updated . In general , the regularization from physical laws focuses the learning/optimization towards a more meaningful fitness understanding of the given data . Although the two designs of PNN and physics constraints aim to obtain the physical laws via fitting existing data , the incorporation of physical laws ’ biggest impact is its capability to achieve zero fitting errors even beyond the training data range . Unfortunately , it will be costly if not impossible to conduct active learning for every Internet of Things ( IoT ) system due to its scale . To provide the “ free lunch ” of adding the extrapolation capability of PNN , we propose a new learning method for cross-model validation based on probabilistic estimates with physical kernels . We use a physically meaningful forecast to identify out-of-sample points with a confidence level . Then , the results are integrated as a weak supervision for learning the exact physics for extrapolation , in addition to the constrained PNN . Together , PNN-RISE uses the PNN container to include the RISE principles , enforcing the recovered function to satisfy constraints on range ( R ) , inertia ( I ) , symmetry ( S ) , and extrapolation ( E ) . Empirically , extensive validations show that the PNN-RISE outperforms the state-of-the-art methods . Consistently , scenarios with or without each constraint in RISE validate the approach . Furthermore , for each constraint , we quantify its strength with different levels , highlighting that loose constraints are enough to obtain high accuracy in the proposed PNN . In summary , the contributions in Fig . 1 are 1 ) showing that the sparse recovery with low training and testing errors does not ensure physical law recovery , 2 ) reducing the possibilities of local optima significantly by eliminating non-physical parameters in learning model selection and design , 3 ) proposing four physical principles to completely characterize physical systems via constrained learning , and 4 ) creating a novel learning method for physical systems extrapolation as probabilistic forecasting out-of-sample cross-model validation ( FOCV ) . 2 RELATED WORK . Physical System Identification . To recover system information , supervised learning ( SL ) can approximate input-output relationships for inferences ( Huang et al. , 2006 ; Hornik et al. , 1990 ) . They show good numerical performance in non-physical and physical applications , including computer vision tasks and natural language processing ( Lu & Weng , 2007 ; Wang et al. , 2017a ; Karpathy et al. , 2014 ; Nadkarni et al. , 2011 ; Hirschberg & Manning , 2015 ) . Deep neural networks ( DNNs ) boost classification accuracy ( Hinton et al. , 2006 ; Krizhevsky et al. , 2012 ) by using latent layers to extract nonlinear features hidden in images . While it makes SL useful for non-physical systems , physical systems require a much higher accuracy for operation or control , e.g. , near 100 % for Six Sigma performance ( Walshe et al. , 2010 ) , so that operators can safely operate the system to avoid outages or cascading failures ( Chung et al. , 2018 ; Liu et al. , 2017 ) . Even worse , many SL models are black boxes that lack physical interpretability for reliable operations , as well as certifiability for performance standards ( Mestav et al. , 2018 ; Du & Li , 2019 ; Hu et al. , 2020 ; Baghaee et al. , 2017 ) . Therefore , some works find utilizing contextual information improves physical-system identification where they assume a good knowledge of physical basis and sufficient measurements across the system ( Yu et al. , 2017 ; Chen et al. , 2020 ) . Similarly , ( Brunton et al. , 2016 ; Champion et al. , 2019 ; Li & Weng , 2021 ) assume prior symbols are known in a dictionary and introduce sparse regression to select symbols . However , the assumptions are too limiting due to the difficulty of knowing all the physical basis for arbitrary complex systems ( Wang et al. , 2017b ; Tarca et al. , 2007 ) . One ad-hoc approach ( Schmidt & Lipson , 2009 ; Petersen , 2019 ) uses a genetic algorithm to search for the symbolic basis but the problem is NP-hard without any guarantee on recovering the governing physical laws from empirical data . Thus , we investigate whether the sparsity technique and a small modeling error are sufficient to claim the success of learning the physical model . Moreover , how can analytical techniques ensure exact physics learning for domain-specific systems ? Constrained Machine Learning . There are many applications of constrained machine learning ( CML ) in different systems . ( Small et al. , 2011 ) restricts the weight vectors with domain knowledge in support vector machine and ( Li et al. , 2021 ) proposes constrained adversarial learning to generate adversarial examples under physical constraints . In deep learning applications , ( Zhao et al. , 2019 ) employs constraints from conservation law to restrict the output variables of DNN . ( Velloso & Van Hentenryck , 2020 ; Fioretto et al. , 2020a ) solve optimal power flow problems using DNN with physical constraints , where the output variables are biased to guarantee feasible solution . ( Tran et al. , 2020 ) restricts the input to DNN into different groups to guarantee privacy . Among these studies , ( Fioretto et al. , 2020b ) optimizes DNN with output constraints using Lagrangian duality . These CML methods can solve their own problems effectively ; however , there is a need for general principles to embed the physical constraints systematically with efficient solutions . Data Forecasting . For previous data forecasting methods , autoregressive moving average ( ARMA ) and autoregressive integrated moving average ( ARIMA ) predict the evolving variables based on their lagged values ( Gilbert , 2005 ) . To capture nonlinear correlations in the short-term and long-term , there are many deep learning-based methods , e.g. , recurrent Neural Network ( RNN ) , long shortterm memory ( LSTM ) , bidirectional LSTM ( BiLSTM ) , gated recurrent units ( GRUs ) , and variational autoencoder ( VAE ) ( Zeroual et al. , 2020 ) . To forecast the result with confidence , methods like Gaussian process ( GP ) ( Quinonero-Candela & Rasmussen , 2005 ) , Quantile Regression ( Maciejowska et al. , 2016 ) and deep learning-based models ( Salinas et al. , 2020 ) afford an efficient way to predict distributions for the future data . Since the uncertainty evaluation is important to integrate the forecast data into the physical learning process , probabilistic methods are presented to estimate probability density functions in the proposed FOCV . 3 METHODS . The PNN-RISE approach is to understand the reason why a “ good ” sparse fitting is not physical verifiable , to design physical principles to guide the fitting internally based on a flexible PNN structure , and to create a cost-free active learning method for extrapolation at unseen operating points . Symbolic regression ( SR ) is a common approach to find equation operators ( { + , − , × , / } ) and expressions like ( x2 , cos ( x ) ) , etc . Existing papers ( Brunton et al. , 2016 ; Sahoo et al. , 2018 ; Champion et al. , 2019 ; Petersen , 2019 ; Kim et al. , 2020 ) show that if the following three conditions are satisfied , SR is capable to find the true equation : 1 ) apply the sparse regularization , 2 ) achieve nearly zero training error , and 3 ) utilize cross validation to avoid overfitting . However , we observe that SR methods are prone to non-physical solutions even when all three assumptions are satisfied . For example , to learn the function y = x1x2+x2x3 from data , different SR methods can easily get trapped in local optima . To see if these observations are special , we test different cases , where Fig . 2 shows a table summary of four different cases , with integer coefficients or not . We also test different numbers for multiplication and summation . From all examples , one can see that the recovered objective has sparse coefficients and negligible error ( loss objective L ) at non-desirable local optimums . Other than the toy examples , we conduct a realistic power system case study and visualize the objective surface in Appendix A . Similarly , for a small parameter range , there are points achieving the global optimum or local optima that are very close to the global optimum . However , only one of them is the physical ground truth . For example , one can go from the red point to a non-physical solution with an error close to zero , even when the sparsity regularization is enforced . Therefore , we can see that “ good ” sparse fitting is insufficient to avoid overfitting . Thus , we need to avoid overfitting for physical systems not only by choosing sparse learning models , but also new ways to constrain the learning process and avoid non-physical solutions .

Summary: This paper tackles the problem of model inference and prediction, while preserving physical correctness. This is an interesting and important problem in supervised learning, for interpratability and generalization reasons. It is also quite important in model-based reinforcement learning, in the context of nonlinear system identification.  The authors start by challenging existing sparse regression approaches (such as SINDy) and symbolic regression approaches (such as EQL): - Because they lead to non-physical solutions with low loss, - Because they depend on hyperparamters  Hence they propose what is coined a Physical Neural Network (PNN), where: -  layers have a Laurent polynomial shape with learnable coefficients and powers, - weights corresponding to the layer input powers are decomposed into a tanh activation multiplied by a sigmoid activation.  This last design choice allows to enforce some kind of sparsity (when sigmoid converges to 0) and constrains powers in the range {-1, 1} when active (because of tanh). Furthermore, it ensures differentiability of the integer valued layer-input powers.  Additionally, the authors also propose to penalize the model training with what they call a RISE constraint. The latter is meant to enforce physically plausible range, inertia, symmetry and extrapolation properties in the inferred model.