Title: BLENDING DIVERSE PHYSICAL PRIORS WITH NEU-RAL NETWORKS

Abstract: Rethinking physics in the era of deep learning is an increasingly important topic. This topic is special because, in addition to data, one can leverage a vast library of physical prior models (e.g. kinematics, fluid flow, etc) to perform more robust inference. The nascent sub-field of physics-based learning (PBL) studies this problem of blending neural networks with physical priors. While previous PBL algorithms have been applied successfully to specific tasks, it is hard to generalize existing PBL methods to a wide range of physics-based problems. Such generalization would require an architecture that can adapt to variations in the correctness of the physics, or in the quality of training data. No such architecture exists. In this paper, we aim to generalize PBL, by making a first attempt to bring neural architecture search (NAS) to the realm of PBL. We introduce a new method known as physics-based neural architecture search (PhysicsNAS) that is a top-performer across a diverse range of quality in the physical model and the dataset.

1. INTRODUCTION:

Advances in machine learning can transform the way physical calculations are performed. Many physical models are idealized and do not precisely match real-world data. An elementary example would be equations for projectile motion which do not account for air resistance. Using these idealized equations, a completely physics-driven approach would have large errors on real-world data. A separate approach is completely data-driven, e.g., one could repeatedly record real-world projectile tosses and use a regression model to estimate a future trajectory; unfortunately, this approach requires large datasets and lacks interpretability. To bridge this gap, the field of physics-based learning (PBL) aims to blend physical priors with data-driven inference, to combine the best of both worlds.

Previous PBL architectures have achieved competitive performance on a wide variety of tasks in microscopy (Rivenson et al., 2019; Nehme et al., 2018; Nguyen et al., 2018; Sinha et al., 2017; Goy et al., 2018) , low level and high level computer vision (Ba et al., 2019; Sun et al., 2019) , medical imaging (Jin et al., 2017; Kang et al., 2017) , and robot control (Zeng et al., 2019; Ajay et al., 2019) . These seemingly diverse problem statements share a common thread: the presence of a partially known physical prior that can be blended with a neural network.

Unfortunately, existing PBL methods are typically designed for a specific task. Generalization would (as a first step) require a PBL architecture capable of adapting to variations in the correctness of physics or the quality of training data. Our experiments show that no such architecture exists (Figure 1 and Section 4.3). Having a general recipe for blending physics and learning is an important step in adopting physics-based learning to encompass the wide range of physical problems, where priors are only approximate and training data can be sparse.

In this work, we approach the problem of PBL from a different angle. Inspired by work in neural architecture search (NAS) (Zoph & Le, 2016; Baker et al., 2016; Liu et al., 2018; Cai et al., 2018) , we propose a first attempt to automatically find the optimal PBL architecture, taking into account characteristics of not just data, but also physics. To incorporate physical models into NAS, we find that three modifications must be made to the existing NAS framework: (1) the inclusion of physical inputs; (2) the inclusion of physical operation sets; and (3) edge weights to normalize variations in the degrees of freedom introduced by the inclusion of physical operators. As these modifications are specific to the PBL problem, we refer to our algorithm as PhysicsNAS. As shown in Figure 1 , the goal of PhysicsNAS is to handle a diverse range of quality in the physical prior or data. Experiments in Section 4.3 offer support for this goal, where PhysicsNAS outperforms previous PBL methods on multiple physical tasks across a range of physical prior and dataset conditions. The performance improvement over existing PBL methods ranges from 3% to 60%.

Our contributions to physics-based learning can be summarized as follows:

• We make a first attempt at bringing neural architecture search (NAS) into the realm of physicsbased learning (PBL), introducing PhysicsNAS as a new method for PBL. • We show in our experiments that PhysicsNAS generalizes to a wider range of diversity in data and physical priors, as compared to previous PBL methods; Although our primary contributions are to PBL, it is worth noting that conventional differentiable NAS approaches (Liu et al., 2018) are not designed to incorporate physical priors; in developing this paper we found it necessary to modify such approaches to incorporate physical priors as both inputs and candidate operations.

2. CATEGORIZING PRIOR WORK IN PHYSICS-BASED LEARNING:

There has been remarkable progress in blending physical priors with neural networks, over the past few years. Here, we make a first attempt to group previous methods into the four categories as illustrated in Figure 2 :

• Physical Fusion feeds the solution from physics-based models as part of the input (Karpatne et al., 2017; Ba et al., 2019) . The solutions can be stacked with the original input, or additionalidentical network branches can be used to extract features separately; • Residual Physics is another way to improve the model-based solutions with deployments in robot control (Zeng et al., 2019; Ajay et al., 2019) and medical imaging (Jin et al., 2017; Kang et al., 2017) . By adding the physical solution onto the network output, the neural networks only need to learn the mismatch between the model-based solution and the ground truth in this case; • Physical Regularization harnesses the regularization term from a set of physical constraints to penalize the network solutions. The regularization term can be appended as part of the loss function explicitly (Karpatne et al., 2017; Stewart & Ermon, 2017; Raissi et al., 2017; Raissi, 2018; Fei et al., 2019) , or through a reconstruction process from physics (Che et al., 2018; Chen et al., 2018; Pan et al., 2018) 

𝑁𝐴𝑆:



Physical Regularization Manipulating the loss function based on physical equations:



Residual Physics:

Learning the difference between physical prediction and ground truth

Physical Fusion:

Feeding necessary physical information as part of the input to the network Embedded Physics

Incorporating physical models or physical parameters into the network design models. Unrolled networks (Gregor & LeCun, 2010; Diamond et al., 2017; Kellman et al., 2019; Monakhova et al., 2019; Sitzmann et al., 2018) , PDE-Nets (Long et al., 2017) , and variational networks (Hammernik et al., 2018; Chakrabarti, 2016) can all be classified into this category.

During training, auxiliary intermediate losses can be inserted to guarantee the learned parameters indeed carry their corresponding physical meanings as well (Hui et al., 2019; Song & Funkhouser, 2019; Li et al., 2019) .

Continuing to propose new models for PBL is a viable direction, however this may not address adaptability to diverse scenarios of physical model mismatch and sparsity in training data. Physic-sNAS is a different tack, where we design basic operation sets inspired by PBL strategies, and allow networks to customize their architectures during training.

3. PHYSICSNAS:

In what follows, we describe the PhysicsNAS algorithm. In Section 3.1, we discuss the problem setup. We then describe the search algorithm in Section 3.2 and the detailed features of PhysicsNAS in Section 3.3.

3.1. PROBLEM SETUP:

In the PBL problem, we have access to a training set D train = {(x i , y i )} N i=1 and a partially known physical operator A phy . Each sample within the training set is a data pair (x i , y i ) formed by an input instance x i ∈ X and the corresponding output (label) y i ∈ Y , and the objective is to learn a function f (•) that maps input space to output space X → Y . f (•) is approximated by a physics network from a search space H with hypotheses f (ω, α, A phy ), where ω denotes network parameters and α denotes architecture parameters. The learning algorithm searches inside H and tries to find {ω, α} that parameterizes the optimal f (ω, α, A phy ) for D train . The challenge for these problems lies in finding a suitable method to incorporate A phy into the network design under diverse regimes of physical model mismatch.

3.2. SEARCH ALGORITHM:

We develop PhysicsNAS based on differentiable NAS techniques (Liu et al., 2018; Cai et al., 2018) parameters can be updated using gradient descent. In contrast to NAS for complicated vision tasks, we do not search the cell structures and apply these searched cells to predefined meta-architectures in PhysicsNAS. As such, PhysicsNAS tries to learn an architecture that links the network input and network output directly. The search space of PhysicsNAS is illustrated in Figure 3 , where the whole architecture is represented by a directed acyclic graph with nodes {N i } N i=1 and edges {E m } M m=1 . Each edge connects two nodes (N i , N j ) through a mixed operator, and each node corresponds to a type of input or a feature vector extracted from previous nodes through the mixed operators. The output of the mixed operator between (N i , N j ) is the gated sum of all candidate operations {o k } K k=1 :

EQUATION

where m ij is the output of this mixed operator, n i is the feature vector of node N i , g o is the binarized operation mask based on the softmax probability of architecture parameters α o in (Cai et al., 2018) , and K is the number of operations inside a edge, which depends on the properties of node pair (N i , N j ). The nodes are densely connected, so that n j , the output features of node N j , is the gated sum of features from all its previous nodes:

EQUATION

where g e is the binarized edge mask based on the softmax probability of architecture parameters α e , and N j can either be an intermediate node or the output node.

During training, we retain two incoming edges for each node and one operation for each edge through the binary gate sampling in (Cai et al., 2018) . While for inference, we pick two candidate edges with the largest edge probabilities, and select the operation with maximum operation probability for each of the two edges. We choose two edges for each node to leave the potential for PhysicsNAS to learn complicated structures, like skip connection and multi-stream encoding.

In order to learn both the network weights and the associated architecture parameters, we update these two sets of parameters alternately. In the architecture step, we freeze the network weights ω and minimize the validation loss L val (ω, α) by updating α. In the network step, we update ω to minimize the training loss L train (ω, α) with frozen α.

3.3. PHYSICSNAS FEATURES:

To incorporate priors into existing differentiable NAS (Liu et al., 2018; Cai et al., 2018) , we make three unique modifications into the search process of PhysicsNAS.

Physical Inputs As a first step in blending physics into PhysicsNAS, we need to prepare unique input nodes that take into account four categories of input information: 1) the data input X; 2) the duplicated data input X dup to verify whether physical information is indeed necessary since each

𝑅𝑎𝑛𝑑𝑜𝑚 𝑊𝑖𝑛𝑑:

What is the trajectory after tossing?

𝑣 0 𝑣 1 𝑣 2 𝑣 3 𝑣 𝑎1 𝑚 𝑎 𝑚 𝑏 𝑣 𝑏1

What are the velocities after collision?

𝑚 𝑎 𝑣 𝑎1 + 𝑚 𝑏 𝑣 𝑏1 = 𝑚 𝑎 𝑣 𝑎𝑓 + 𝑚 𝑏 𝑣 𝑏𝑓 1 2 𝑚 𝑎 𝑣 𝑎1 2 + 1 2 𝑚 𝑏 𝑣 𝑏1 2 = 1 2 𝑚 𝑎 𝑣 𝑎𝑓 2 + 1 2 𝑚 𝑏 𝑣 𝑏𝑓 2

Momentum conservation:

Energy conservation: 𝐺 = 𝑚𝑔 Figure 4 : We evaluate our method on a simulator of classical tasks. The first task (Left) is predicting the trajectory of a ball being tossed, and the second task (Right) is estimating the velocities of two objects after collision. node has to pick two edges; 3) the estimated solution from physics Ŷphy = A phy (X); and 4) the concatenation of X and Ŷphy to test which stage to conduct the physical fusion.

Physical Operations:

To merge physical models inside the network, we create physics-informed operation sets O = {o N N1 , ..., o N N L , o phy }, where o N Ni denotes the neural network operations (e.g.,fully-connected layer, skip connection) and o phy denotes the physical forward operation. Specifically, for the physical forward module, we also use a light-weight network layer, such as a fully-connected (FC) layer, to make the size of its input consistent with the parameter size required by the physical module. Physical forward modules are only included in the edges that connect to the output node in our implementation.

Edge Weights In PhysicsNAS, not all edges are created with the same amount of operations, since they are used to connect different types of nodes. Consequently, if we select the edges purely based on the operation probabilities, edges with fewer operations are naturally preferred due to the softmax probability, which causes a biased architecture selection. We solve this issue by introducing the edge weight as described in Equation 2. After searching, we first pick a desired edge according to the edge weights, and then select the desired operation for that edge based on the operation weights.

4. EXPERIMENTS AND RESULTS:

To comprehensively evaluate PhysicsNAS, we simulate two representative physical tasks for which we can vary the model mismatch: 1) predicting trajectories of an object being tossed; and 2) estimating the speed of rigid objects after collision. Figure 4 illustrates these tasks; further details are provided in Section 4.1. Comparison PBL architectures are described in Section 4.2. Finally, we evaluate PhysicsNAS and provide a detailed analysis on the searched architectures in Section 4.3.

4.1. DESCRIPTION OF TASKS:

For the TOSSING TRAJECTORY PREDICTION task (see Figure 4 for visualization), the initial three locations of the object {l 1 , l 2 , l 3 } are given as input X, and our objective is to predict locations of this object in the following 15 time stamps, {l 4 , l 5 , ..., l 18 }. We only consider the displacement within a 2D plane, therefore, the coordinates of each location can be represented by two numbers, i.e. l i = (l xi , l yi ). We adopt the following elementary free-falling equations as the prior, and examine different methods under this inadequate physical prior:

EQUATION

where l xi and l yi denote the object location at time t i , l x1 and l y1 are the initial location of the object, v x and v y denote the initial velocities along horizontal and vertical directions respectively, and g is the fixed gravitational acceleration of 9.8m/s 2 . We introduce two model mismatches: the random acceleration as the winds and an additional damping factor based on F air = k × v 2 to simulate the air resistance. The future locations estimated according to mismatched prior are used as the physical input Ŷphy . As to the physical modules in Embedded Physics and PhysicsNAS, we estimate parameters { lx1 , ly1 , vx , vy }, and substitute these parameters into Equation 3 as the physical operation.

In the COLLISION SPEED ESTIMATION task (see Figure 4 for visualization), we use the speed of two objects at the initial two time stamps, object mass, and the distance between these two objects {v a1 , v a2 , v b1 , v b2 , m a , m b , D} as the input X to estimate the speed after their collision {v a f , v b f }.

We assume the objects have the different mass, and can only move along one direction. Based on energy conservation and momentum conservation for perfectly elastic collision, we adopt the following prior:

EQUATION

We add sliding friction to the system as intentional model mismatch, where conservation prior in Equation 4 does not hold. Solutions without the consideration of friction are used as Ŷphy , and { ma , mb , va1 , vb1 } are estimated for the physical modules.

4.2. MANUALLY DESIGNED PBL METHODS:

For the sake of comparison, several manually designed architectures from Section 2 are also evaluated. We use a three-layer multilayer perceptron (MLP) as the naive data-driven baseline, since it has sufficient expressiveness to fit any continuous function, especially the elementary physical tasks we have chosen (Csáji, 2001) . Network structures for the Physical Regularization model and the Residual Physics model are the same as the naive model. The output of the Residual Physics model is a summation of Ŷphy and the learned residual from the network, while there is an additional regularization term in loss function of the Physical Regularization model. Since only a partially correct physical prior is used, directly using physical solution as the regularization will in turn aggravate the error. Thus, we introduce an ReLU-based regularization similar to (Karpatne et al., 2017) . The regularization loss penalizes the network solution based on the assumption that the object moves along one direction in the horizontal axis for the trajectory prediction task, and the total kinetic energy is less than the initial kinetic energy for the speed estimation task. In the Physical Fusion approach, two separate branches are utilized to extract features from X and Ŷphy respectively, and each of them is a two-layer MLP. The extracted features will then be concatenated and fed into the output layer. The Embedded Physics model first estimates necessary parameters in Equation 3 and Equation 4 with a three-layer MLP, and then produces trajectory estimation based on the fixed physical process. All the above models are supervised by the ground-truth future locations with mean square error (MSE) loss, and the hidden dimension for FC layers are 128.

4.3. RESULTS ANALYSIS:

Training Details To evaluate the importance and success of the proposed approach, we vary the physical model mismatch and sparsity in training data in a controlled manner. When training Physic-sNAS, we split the training set into two subsets of the same size to update architecture variables and network variables respectively. We limit the number of learnable nodes in PhysicsNAS to be 5, and retrain the searched architectures with full training sets after searching. The models are implemented in PyTorch (Paszke et al., 2017) , and are trained using the Adam optimizer (Kingma & Ba, 2014).

Moreover, for all baseline approaches we compare in this paper, we fine tune their hyperparameters in order to make fair comparisons. We choose three hyperparameter sets for each scenario and run five times for each method. We finally pick out the best result for each method.

Performance Comparison We apply the proposed PhysicsNAS to learn architectures embedded in the search space. The testing results of PhysicsNAS and other existing PBL methods (as detailed in Section 4.2) are summarized in Table 1 and Table 2 . As shown in these two tables, the performance of different PBL models varies based on the disparity of mismatch levels and training data sizes, while PhysicsNAS is capable of generating architectures that outperform these manual PBL models consistently. Results in Figure 5 further demonstrate this capability along data dimension and physics dimension in a fine-grained scale. We also conduct an ablation study on PhysicsNAS by removing task-specific adaptations such as physical inputs and operations. Results in Appendix D show the task-specific adaptations improve the performance of PhysicsNAS at different mismatch levels, demonstrating that having physical inputs and physical operations in PhysicsNAS's search 

Utilization of Physical Inputs:

The physical inputs are always selected in our searched architectures for these two tasks, which verifies the importance of physical information during learning. Please refer to Appendix B for the illustration of a range of searched architectures corresponding to scenarios in Table 1 and Table 2 .

Utilization of Physical Operations:

The selection of physics-inspired operations are more nuanced, depending on the accuracy of physical information encoded in the physical operations as well as the amount of training data. Figure 6 shows two examples, one where physical operations are selected and the other where they are not selected. In particular, the inaccurate physical operation in the trajectory task is preferred at early training epochs. However, as training proceeds, the learned FC modules achieve higher accuracy and the network thus discards the physical operations. As a result, the Residual Physics strategy is adopted in the final searched architecture. For the collision task, the physical operation could model the perfectly elastic collision completely, and the estimated physical solutions are precise if the estimated physical parameters are accurate. Therefore, physical operations are selected when there exists a robust estimation of physical parameters. However, it usually requires sufficient training samples to obtain this robust estimation, which might be a reason why physical operations are only selected in the cases with 128 training samples. More details about the changes of network architectures during searching can be found in Appendix F. Failure Case In differentiable NAS, the training algorithm aims to optimize the over-parameterized network with all the edges and operations. Therefore, it is necessary to prune the redundant edges and operations after training. We adopt the pruning mechanism in (Liu et al., 2018) , where each node has to retain two incoming edges. For extreme cases where single-stream architectures are optimal, PhysicsNAS may generate sub-optimal architectures due to this arrangement. As shown in Figure 7 , we make a toy comparison between two lightweight architectures on the collision task with the friction coefficient range [0.15, 0.25] and 32 training samples. It is notable that by simply adding an additional stream to the input x, the new searchable architecture deteriorates the result. Introducing an adaptive edge selection mechanism might be a meaningful future work. This limitation could also be overcome by resorting to other NAS frameworks, such as those based on reinforcement learning (Pham et al., 2018) . In rare situations, a single-stream network could be preferred. PhysicsNAS is unable to converge to single-stream architectures due to the edge selection mechanism.

5. CONCLUSION:

In conclusion, our experiments show that PhysicsNAS can handle a wider range of input physical models and data, as compared to existing PBL methods. This is only a first attempt at increasing the diversity of PBL through architecture search. Ultimately, our hope is to apply PhysicsNAS to problems as diverse as microscopy (Barbastathis et al., 2019) , computer vision (Velten et al., 2012) , sensor fusion (Eitel et al., 2015; Xu et al., 2018) and astrophysics (Bouman et al., 2016; Akiyama et al., 2019) , where it is important to handle variations in model mismatch and dataset quality across these problem domains.

A GENERALIZABILITY FROM PHYSICAL PRIOR:

We present a VISUAL PHYSICS task to further illustrate why physical prior is significant to datadriven approaches in a qualitative manner. The network tries to estimate the future trajectory of a paper ball being tossed from the initial three frames. We deploy similar physical prior as shown in Equation 3. This simple physical prior fails to generate reliable estimation due to the physical disparity from real world, such as the existence of air resistance and the deformation of the paper during traveling. From pure data-driven approach, we train a ResNet18 (He et al., 2016) with 32 tossing samples. This network still could not generate reliable estimation due to the limited number of training samples. Most of the predicted trajectories are not physically plausible, since the ball should not change its position suddenly in Frame 4. For the PBL method, we design a physicsinformed ResNet18 with physical estimation as an additional input (Physical Fusion) and a skip connection between the physical estimation and the output (Residual Physics). The PBL network is also trained with 32 samples. With the assistance of physical prior, the trajectory could be estimated accurately. Some typical predictions of the above three methods are illustrated in Figure 8 .

B COMPLETE LIST OF SEARCHED ARCHITECTURES:

Searched architectures of PhysicsNAS in Table 1 and Table 2 are displayed in Figure 9 and Figure 10 .

C EFFECTIVENESS OF EDGE WEIGHTS:

We evaluate the effectiveness of edge weights, and exhibit the advantage of PhysicsNAS as compared to the original NAS framework. We conduct comparison between two different NAS policies: 1) the NAS framework with physical inputs and operations, yet without edge weights; 2) the proposed PhysicsNAS with both edge weights and physical modules. The initialized architectures Figure 8 : Even from limited training samples, PBL has strong generalizability as compared with conventional data-driven approaches. As exhibited in the left, estimation from physics can not match the ground-truth trajectory due to the physical disparity. As a data hungry method, deep neural networks also fail to generate physically plausible estimation with limited training samples (the middle). As depicted in the right of the figure, PBL method solves the dilemma by utilizing knowledge from both data and physical prior. The predicted trajectory (marked as green dots) from PBL highly overlaps with the ground truth in the testing stage. before neural architecture searching are illustrated in Figure 11 , and the corresponding searched architectures are illustrate in Figure 12 . We can observe that if the architectures are shallow in the initialization stage, there is a high probability that they remain shallow after searching. This phenomenon may come from the nature of differentiable NAS training process: once an operation is selected in a training update, it obtains privilege over other operations due to the backpropagation process, which in turn increases its weight and probability of being selected in the following training updates. Consequently, if the edges are selected merely based on the operation weights, NAS frameworks without edge weights will lose their potential to generate deep architectures due to the preference caused by difference in numbers of operations inside edges. PhysicsNAS addresses this issue by introducing additional edge weights, and the initial preference caused by unbalanced amount of operations within edges is thus alleviated. Comparison in Figure 11 and Figure 12 is conducted on the tossing task as detailed in Section 3.1. The range of random accelerations is [-1m/s 2 , 1m/s 2 ] and the damping factor is set to be 0.2. There are 128 training samples used for architecture searching and retraining.

D EFFECTIVENESS OF TASK-SPECIFIC/PHYSICS-BASED ADAPTATIONS:

We conduct an ablation study to demonstrate that the task-specific adaptations (i.e., adding physical inputs and operations into search space) in PhysicsNAS improve the predicted result. To this end, we conduct a comparison between: 1) the NAS framework with edge weights only; 2) the proposed PhysicsNAS with both edge weights and physics-based adaptations. The comparison is conducted The initialized architecture of NAS without edge weights is generally shallow, since the number of the candidate operations between the input nodes and the hidden nodes is less than the number of candidate operations between hidden nodes themselves.

using the tossing task. As can be seen in Fig. 13 , the introduction of physics-based adaptations enables PhysicsNAS to achieve significantly lower prediction error as compared to the NAS framework without task-specific adaptations.

Under review as a conference paper at ICLR 2020 

E DATA EFFICIENCY OF PHYSICSNAS:

We show that the introduction of physical prior knowledge reduces the data amount for learning/approximating an exact physical model by comparing PhysicsNAS with naive MLPs along the data dimension. This comparison is conducted using the aforementioned tossing task in the high mismatch level scenario. As shown in Figure 14 , PhysicsNAS can achieve significant, high prediction precision with limited data. As for this model mismatch type and level, PhysicsNAS can decrease the demand for training data by approximately 75%. This demonstrates the capability of PhysicsNAS on fewer-shot learning, where training data are burdensome to acquire due to extreme environments.

Performance gain decreases as data amount increases. As also shown in Figure 14 , the performance gap between PhysicsNAS and the naive MLP continuously decreases as the amount of training samples increases. When the size of the training data is increased to 1024 the performence of the two models is nearly identical, with the MLP method slightly outperforming PhysicsNAS. This indicates that performance gain for applying PhysicsNAS is more evident when the amount of high-fidelity training data is limited. 

F ARCHITECTURE CHANGES AT DIFFERENT EPOCHS:

We show the usage of physical operations along with the number of training epochs in this section. In Figure 15 , the experiment is conducted in the tossing task with low mismatch level and 128 training samples, while Figure 16 shows the architecture changes in the collision task with high mismatch level and 128 training samples. G UNCERTAINTY OF DIFFERENTIABLE ARCHITECTURE SEARCH DARTS (Liu et al., 2018) framework tries to find an optimal architecture by updating the architecture parameters and network weights simultaneously. Therefore, the inconsistency between the architecture optimizer and the network weight optimizer may impede this joint optimization process. The performance of searched results relies on both the hyperparameters and the initialization, and improper training process may lead to model collapse as addressed in (Liang et al., 2019) . PhysicsNAS is built based on the DARTS framework, and the model collapse issue also exists. Please refer to Figure 17 for some collapsed models of PhysicsNAS in the two demonstrative tasks. In our paper, we avoid the model collapse by tuning the learning rates of architecture and network optimizers, so that architecture parameters and network parameters can converge synchronously. 

:



Figures and Tables:

Figure 1: Generalizing PBL across a range of sparsity in training data and correctness in the physical model.

Figure2: An overview of proposed NAS-based blending approach. Our PhysicsNAS takes advantage of all the existing methods on blending physical prior, and is capable of generating new hybrid architectures for tasks under diversified physical environments. With the augmented search space and knowledge from prior information, it is possible for the proposed PhysicsNAS to generalize its performance with limited number of training samples.

Figure 3: Search space of our PhysicsNAS. In the proposed PhysicsNAS, all the nodes are densely connected by mixed operators from predefined candidate operation sets. The hidden nodes can obtain information from the original inputs or from previous hidden nodes within this search setup. The training process is supervised by both ground truth and physical constraints.

Figure6: Utilization of physical operations in PhysicsNAS. The selection of physics-inspired operation depends on its accuracy. PhysicsNAS tends to utilize the physical operations when they are more accurate (like the elastic collision model), and prefers a residual connection when they are inaccurate (like the parabola equation).

Figure7: Failure case. In rare situations, a single-stream network could be preferred. PhysicsNAS is unable to converge to single-stream architectures due to the edge selection mechanism.

Figure 9: Searched architectures of tossing task under diversified physical conditions.

Figure 10: Searched architectures of collision task under diversified physical conditions.

Figure12: Searched architectures using edge weights alleviates model collapse. The searched architecture without edge weights remains shallow due to the initial preference caused by unbalanced amount of operations, and its corresponding testing error is inferior to PhysicsNAS. The proposed PhysicsNAS has the potential to generate both deep and shallow architectures, while NAS without edge weights would generally converge to shallow architectures. The searched architecture of PhysicsNAS is thus superior to the one without edge weights in terms of the testing error.

Figure 14: Data Efficiency of PhysicsNAS. With a rough physical prior, the demand of high-fidelity training samples can be greatly reduced. PhysicsNAS only requires less than 64 training samples to reach the testing performance of a Naive MLP with 256 training samples. It is reasonable to deploy PhysicsNAS to alleviate the burden of data acquisition when there are limited training samples.

Figure 15: Changes of searched topology in tossing task. Inaccurate physical operation is used at early epochs, however, it is discarded as the training proceeds.

Figure16: Changes of searched topology in collision task. The physical operation is not adopted initially, yet the searched topology eventually adopts the physical operation.

Model collapse in the tossing task.

Model collapse in the collision task.

Figure 17: Model collapse of PhysicsNAS due to the uncertainty of training. The performance of PhysicsNAS depends on the differentiable searching process. Model collapse may appear if hyperparameters are not selected carefully.

. With learnable architecture parameters α and continuity relaxation, both network architectures and 𝒀 𝑷𝒉𝒚

Table content: <table><tr><td/><td/><td>𝑵 𝟏</td><td/><td/><td/><td/><td/><td/><td>Node Set</td><td>Operation Set</td></tr><tr><td/><td/><td>𝑿</td><td>𝑬 𝟏</td><td>𝑬 𝟓 𝑬 𝟒</td><td colspan="2">𝑬 𝟖</td><td>Knowledge from Physics and Data</td><td/><td>Input node</td><td>Mixed Fully Connected</td></tr><tr><td/><td/><td>𝑵 𝟐</td><td/><td/><td>𝑬 𝟗</td><td/><td/><td/><td>Mixed</td></tr><tr><td/><td/><td>𝑬 𝟐</td><td>𝑵 𝟒</td><td>𝑬 𝟕</td><td/><td/><td colspan="2">Supervision</td><td>Fully Connected/</td></tr><tr><td/><td>𝑵 𝟑</td><td>𝑬 𝟑</td><td>𝑬 𝟔</td><td/><td>𝑵 𝟓</td><td>𝑬 𝟏𝟏</td><td>𝑵 𝟔</td><td>𝒀</td><td>Intermediate node</td><td>Physics Forward/ Fully Connected/ Skip Connection Mixed</td></tr><tr><td>[𝑿,</td><td>𝒀 𝑷𝒉𝒚 ]</td><td/><td/><td>𝑬 𝟏𝟎</td><td/><td/><td/><td/><td>Output node</td><td>of input and output *depends on the dimensions Skip Connection*</td></tr></table>

Testing performance on tossing task. is necessary. Our experiments also show that PhysicsNAS is able to perform inference on small training datasets: the physical prior reduces the demand for high-fidelity training samples. We find that PhysicsNAS only requires less than 64 training samples to reach the same testing performance of a naive MLP with 256 training samples. Moreover, the performance gap between PhysicsNAS and naive MLP method minimizes as the number of training samples increases. This suggests that PhysicsNAS is more favorable in scenarios where training data is not rich enough on the other hand. A detailed discussion about how PhysicsNAS reduces the demand on training data is provided in Appendix E.

Table content: <table><tr><td colspan="2">Mismatch Level</td><td>Low</td><td>High</td><td colspan="2">Mismatch Level</td><td>Low</td><td>High</td></tr><tr><td>Sample Amount</td><td colspan="3">32 128 32 128</td><td colspan="2">Sample Amount</td><td>32 128</td><td>32</td><td>128</td></tr><tr><td>Naive Network</td><td colspan="3">0.684 0.232 0.696 0.267</td><td>Naive Network</td><td/><td colspan="2">1.974 0.319 8.865 4.534</td></tr><tr><td>Physical Fusion</td><td colspan="3">0.266 0.200 0.321 0.178</td><td colspan="2">Physical Fusion</td><td colspan="2">0.868 0.174 6.892 4.596</td></tr><tr><td colspan="4">Residual Physics 0.323 0.192 0.481 0.279</td><td colspan="4">Residual Physics 1.053 0.173 11.750 5.859</td></tr><tr><td colspan="4">Embedded Physics 0.617 0.169 0.617 0.305</td><td colspan="4">Embedded Physics 2.105 0.258 7.962 4.546</td></tr><tr><td>Physics Reg.</td><td colspan="3">0.459 0.272 0.674 0.315</td><td>Physics Reg.</td><td/><td colspan="2">1.916 0.271 8.631 4.451</td></tr><tr><td>PhysicsNAS</td><td colspan="3">0.183 0.097 0.264 0.152</td><td>PhysicsNAS</td><td/><td colspan="2">0.724 0.121 6.741 4.157</td></tr><tr><td/><td/><td/><td/><td colspan="4">Table 2: Testing performance on collision</td></tr><tr><td colspan="4">We adopt the average Euclidean distance be-</td><td colspan="4">task. We use similar Euclidean distance be-</td></tr><tr><td colspan="4">tween the ground truth and the predicted loca-</td><td colspan="4">tween the estimated speed and the ground-truth</td></tr><tr><td colspan="4">tions as the evaluation metric (lower distance is</td><td colspan="4">speed as the metric (lower distance is better).</td></tr><tr><td colspan="4">better). The low mismatch level corresponds</td><td colspan="4">The low mismatch level corresponds to a ran-</td></tr><tr><td colspan="4">to a small random initial acceleration range [-1m/s 2 , 1m/s 2 ] and a small damping factor</td><td colspan="4">dom initial friction coefficient in range [0.28, 0.32], and the high mismatch level corresponds</td></tr><tr><td colspan="4">0.2. The high mismatch level corresponds to a large acceleration range [-3m/s 2 , 3m/s 2 ] and</td><td colspan="4">to a friction coefficient in range [0.45, 0.55]. The best model is marked in red and the sub-</td></tr><tr><td colspan="4">a large damping factor 0.5. The best model is</td><td colspan="2">optimal is in blue.</td><td/></tr><tr><td colspan="4">marked in red and the sub-optimal is in blue.</td><td/><td/><td/></tr><tr><td>3K\VLFDO )XVLRQ</td><td/><td/><td>%HVW3%/ 3K\VLFV1$6</td><td>3K\VLFDO )XVLRQ</td><td colspan="2">3K\VLFDO )XVLRQ</td><td>%HVW3%/ 3K\VLFV1$6</td></tr><tr><td>7HVWLQJ0HDQ(UURU</td><td>3K\VLFDO )XVLRQ</td><td>5HVLGXDO 3K\VLFV</td><td>3K\VLFDO )XVLRQ</td><td>7HVWLQJ0HDQ(UURU</td><td/><td>3K\VLFDO )XVLRQ</td><td>3K\VLFDO )XVLRQ 5HVLGXDO 3K\VLFV</td></tr><tr><td>([WUHPH UN</td><td colspan="2">+LJK UN 3K\VLFDO0LVPDWFK/HYHOV 0HGLXP UN</td><td>/RZ UN</td><td/><td colspan="2">7UDLQLQJ6DPSOH$PRXQW</td></tr></table>

Peer review (iclr2020):
=== Overall comments === This paper proposes to generalize approaches to physics-based learning (PBL) by performing network architecture search (NAS) over elements from PBL models found in the literature. This entails including physical inputs to the network and the incorporation of new operations to the NAS. I think the idea has merit and rather like it. However, there are several aspects of the work that could be improved. The technical novelty is small, as the extension of the existing NAS models to handle physical inputs and a few new operators is relatively straightforward. The experiments, while well designed, only explore uninteresting toy problems. While I appreciate the necessity to explore the methods performance in a more controlled setting, a more impactful testbed would be more convincing. Another drawback of the evaluation is the lack of a proper statistical analysis of the results, given the small data and model sizes. === Relevance & Prior Work === + The related work gives a good summary and categorization of prior work in physics-based learning + The problem (physics-based learning) is interesting and relevant to the community === Novelty & Approach=== + application of NAS to physics based learning + incorporation of physics solutions as inputs into differentiable NAS + creation of physics-informed operation sets to merge physical models into network - technical steps to merge NAS and PBL are relatively straightforward === Evaluation === Two representative physical simulations were chosen for evaluation, where elements of the physics model are intentionally omitted, 1) estimating trajectory of a ball in presence of wind and air resistance, and 2) a collision speed simulation where two objects collide, where sliding friction is not accounted for in the physics model. The baselines consist of: a 3-layer MLP (data-driven), a 3-layer MLP with Physical Regularization, a 3-layer MLP with residual connection to the physics prediction, an MLP with two input branches, on for the data and one for the physics predictions (Physical Fusion), and the Embedded Physics model which estimates parameters for the physics modelu using a 3-layer MLP. PhysicsNAS can combine elements of the baseline models, but the total number of nodes is limited to 5. + Experiments testing the dependence of the model on the numbers of samples and the strength of the physical inconsistencies were conducted. In both cases, PhysicsNAS outperformed the best specialized physics models. - The chosen testbed tasks are toy problems. While these types of experiments are necessary to understand the performance of the model, it would have been interesting to see PhysicsNAS applied to a more impactful task - Given the size of the networks and the training data, there is no reason why a more sophisticated statistical analysis of the results wasn’t performed (confidence intervals, t-test, p-value). Similarly, a more complete set of experiments with more sample amounts could be provided with little effort. === Clarity & Other Comments === - “precious nodes” -> previous nodes