Blending Diverse Physical Priors with Neural Networks
Authors: Yunhao Ba
Guangyuan Zhao
Achuta Kadambi
Venue:  (2020)
Keywords: Physics-based learning
Physics-aware learning

Abstract: Rethinking physics in the era of deep learning is an increasingly important topic. This topic is special because, in addition to data, one can leverage a vast library of physical prior models (e.g. kinematics, fluid flow, etc) to perform more robust inference. The nascent sub-field of physics-based learning (PBL) studies this problem of blending neural networks with physical priors. While previous PBL algorithms have been applied successfully to specific tasks, it is hard to generalize existing PBL methods to a wide range of physics-based problems. Such generalization would require an architecture that can adapt to variations in the correctness of the physics, or in the quality of training data. No such architecture exists. In this paper, we aim to generalize PBL, by making a first attempt to bring neural architecture search (NAS) to the realm of PBL. We introduce a new method known as physics-based neural architecture search (PhysicsNAS) that is a top-performer across a diverse range of quality in the physical model and the dataset. 

Under review as a conference paper at ICLR 2020
BLENDING DIVERSE PHYSICAL PRIORS WITH NEU- RAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Rethinking physics in the era of deep learning is an increasingly important topic. This topic is special because, addition to data, one can leverage a vast library of physical prior models (e.g. kinematies, fluid flow, etc) perform more ro- bust inference. The nascent sub-field of physics-based learning (PBL) studies this problem of blending neural networks with physical priors. While previous PBL algorithms have been applied successfully to specific tasks, it is hard to generalize existing PBL methods to a wide range of physics- based problems. Such general- ization would require architecture that can adapt to variations in the correctness of the physics, or in the quality of training data. No such architecture exists. this paper, we aim to generalize PBL, by making a first attempt to bring neural ar- chitecture search (NAS) to the realm of PBL. We introduce a new method known bhsse neural architecture search (PhysiesNAS) that is per-errrreer across a diverse range of quality in the physical model and the dataset.
INTRODUCTION
Advances in machine learning can transform the way physical calculations are performed. Many physical models are idealized and do not precisely match real-world data. An elementary exam- ple would be equations for projectile motion which do not account for air resistance. Using these idealized equations, a completely physics-driven approach would have large errors on real-world data. separate approach is completely data-driven, e.g., one could repeatedly record real-word projectile tosses and use a regression model to estimate a future trajectory; unfortunately, this ap- proach requires large datasets and lacks interpretability. To bridge this gap, the field of physics-based learning (PBL) aims to blend physical priors with data-driven inference, to combine the best of both worlds.
Previous PBL architectures have achieved competitive performance on wide variety of tasks in microscopy (Rivenson et al. 2019: Nehme et al., 2018 Nguyen et al. 2018 Sinha et al. 2017 Goy et al. 2018). low level and high level computer vision (Ba et al. 2019 Sun et al. 2019). 99, medical imaging (Jin et al. 2017 Kang et 2017), and robot control (Zeng et al. 2019 Ajay et al. 2019).
These seemingly diverse problem statements share common thread: the presence of partially known physical prior that can be blended with neural network.
Unfortunately, existing PBL methods are typically designed for a specific task. Generalization would (as a first step) require a PBL architecture capable of adapting to variations in the correctness of physics or the quality of training data. Our experiments show that such architecture exists (Fig- ure I and Section 4.3) Having a general recipe for blending physics and learning is an important step in adopting physics-basee based learning to encompass the wide range of physical problems, where priors are only approximate and training data can be sparse.
In this work, we approach the problem of PBL from a different angle. Inspired by work in neural architecture search (NAS) (ooph& & Le 2016: Bakkrr et al.) 2016 Liu et al. 2018 Cai et al.) 2018), we propose a first attempt to automatically find the optimal PBL architecture, taking into account characteristics of not just data, but also physics. To incorporate physical models into NAS, we find that three modifications must be made to the existing NAS framework: (1) the inclusion of physical inputs; (2) the inclusion of physical operation sets; and (3) edge weights normalize variations in the degrees of freedom introduced by the inclusion of physical operators. As these modifications are specific the PBL. problem, we refer to algorithm as PhysicsNAS. As shown in Figure [ the

---

Under review as a conference paper at ICLR 2020
Previous Methods are Ad-hoc to Quality of Physics
Our Method Handles Range Quality of Physics
Archit. A
Archit.
PhysicsNAS
Physical Model Correctness
Physical Model Correctness
Performer Most Regions of the Plot
PBL Archit.
Archit. C
Amount of Training Data
Amount of Training Data
Figure 1: Generalizing PBL across range of sparsity in training data and correctness in the physical model.
goal of PhysicsNAS is to handle a diverse range of quality in the physical prior or data. Experiments in Section 4.3 offer support for this goal, where PhysicsNAS outperforms previous PBL methods on multiple physical tasks across range of physical prior and dataset conditions. The performance improvement over existing PBL methods ranges from 3% to 60%.
Our contributions to physics-basse based learning can be summarized follows:
We make first attempt at bringing neural architecture search (NAS) into the realm of physics- based learning (PBL), introducing PhysicsNAS as new method for PBL.
We show in our experiments that PhysicsNAS generalizes wider range of diversity in data and physical priors, as compared to previous PBL methods;
We interpret the converged architectures PhysicsNAS in context prior PBL work, to provide general evidence that: (a) Accurate physical operations be embedded into the network design if there enough training data available; (b) Residual Physics is preferred alternative to inaceu- rate physical operations; (c) Physical Fusion general strategy that can be adopted in various physical environments. Separate from our work, these insights can help lay a foundation for how to explain the choice of PBL models future work.
Although our primary contributions are to PBL, it is worth noting that conventional differentiable NAS approaches (Liu et al., 2018) are not designed to incorporate physical priors; in developing this paper we found it necessary to modify such approaches incorporate physical priors as both inputs and candidate operations.
CATEGORIZING PRIOR WORK IN PHYSICS-BASED LEARNING
There has been remarkable progress in blending physical priors with neural networks, over the past few years. Here, we make first attempt to group previous methods into the four categories as illustrated in Figure
Physical Fusion feeds the solution from physics- based models as part of the input (Karpatne et al., 2017: Ba et al. 2019h. The solutions can be stacked with the original input, additionalidentical network branches can be used to extract features separately;
Residual Physies is another way improve the model-based solutions with deployments in robot control Zeng et al. 2019 Ajay et al. 2019) and medical imaging (Jin et al. 2017 Kang et al.) 2017). By adding the physical solution onto the network output, the neural networks only need to learn the mismatch between the model-based solution and the ground truth in this case;
Physical Regularization harnesses the regularization term from a set of physical constraints to penalize the network solutions. The regularization term can be appended part of the loss func-
tion explicitly (Karpatne et al.) 20177 Stewart & Enmon 2017 Raissi et al. 2017: Raissi 2018 Fei et al. 2019). or through a reconstruction process from physics (Che al. 2018 Chen al.
2018: Pan et al. 2018);
Embedded Physics takes the physical model inside the network optimization loop, where the physical model acts skeleton, and the network is in charge of learning parameters used in these
2

---

Under review as a conference paper at ICLR 2020
R Residual hysics
Naive Network
Phys cal Regulerization
Supervise
Ypo
Plosical Commerutts
Physical Fusion
Embedded Physics
Ansorancdiote
Final
Loss
Feuture Mergo
NAS
Pippic
Physica! Jufa
Manipulating function on physical equations Physical Regularization Residuel Physics
Learning the difierence between phopical prediction and ground
Physical Fusion Embedded Physics
Feeding necessory physica Informatton as part of the input the networt Incorporating physical models or physica parameters into the network design
An overview of proposed NAS-based blending approach. Our PhysicsNAS takes ad- vantage of all the existing methods on blending physical prior, and capable of generating new hybrid architectures for tasks under diversified physical environments. With the augmented search space and knowledge from prior information, it is possible for the proposed PhysicsNAS to gener- alize its performance with limited number of training samples.
Figure 2:
models. Unrolled networks (Gregor & LeC un 2010 Diamond et al. 2017 Kellman et al. 2019 Monakhova et al.) 2019 Siizzaann al.)) 2018)) PDE-Nets (Long al. 2017). and variational networks (Hammernik et al. 2018 Chakrabarti 2016) can all be classified into this category. During training, auxiliary intermediate losses can inserted to guarantee the learned parameters indeed carry their corresponding physical meanings as well (Hui et al. 2019: Song & Funkhouser, 2019 Li al. 2019).
Continuing propose models for PBL is viable direction, however this may address adaptability to diverse scenarios of physical model mismatch and sparsity in training data. Physic- sNAS is different tack, where we design basic operation inspired by PBL strategies, and allow networks to customize their architectures during training.
3 PHYSICSNAS
In what follows, we describe the PhysicsNAS algorithm. In Section 3.1 setup. We then describe the search algorithm in Section 3.2 and the detailed features of PhysicsNAS in Section 3.3
we discuss the problem
3.1 PROBLEM SETUP
In the PBL problem, we have access training set Dtrain {1x,, y:) y:) and partially known physical operator Aphy. Each sample within the training set is a data pair (I; yi) formed by input instance F; X and the corresponding output (label) yı E Y, and the objective
learn function f(.) that maps input space to output space X + Y. f(-) is approximated by physics network from search space H. with hypotheses f(w, a, Aphyy, where denotes network parameters and denotes architecture parameters. The learning algorithm searches inside H and tries to find {w, a} that parameterizes the optimal a,,,y) for Derain. The challenge for these problems lies in finding suitable method to incorporate Aphy into the network design under diverse regimes of physical model mismatch.
3.2 SEARCH ALGORITHM
We develop PhysicsNAS based on differentiable NAS techniques (Liu et al. 2018 Cai et al. 2018). With learnable architecture parameters and continuity relaxation, both network architectures and
3

---

Under review as a conference paper at ICLR 2020
Node Set
Operation Set
Anowleige-rrm
Ea
Plynas
Fulyy
Minil
Tepul
9 pa
Misei
N1
Supervisoon
Miuni
Ns
Inserrofiede
SopCoreniie
N;
Meol
Mirsf
N
Eg
F1p
Duta
[X. (X.Y
Mineil
Figure 3: Search space of our PhysicsNAS. In the proposed PhysicsNAS, all the nodes are densely connected by mixed operators from predefined candidate operation sets. The hidden nodes can obtain information from the original inputs or from previous hidden nodes within this search setup. The training process supervised by both ground truth and physical constraints.
parameters can be updated using gradient descent. contrast to NAS for complicated vision tasks, we do not search the cell structures and apply these searched cells to predefined meta-architectures in PhysicsNAS.
such, PhysicsNAS tries to learn an architecture that links the network input and network output directly. The search space of PhysicsNAS is illustrated in Figure where the whole architecture is represented by a directed acyelic graph with nodes {Na and edges [Em) Each edge connects two nodes (NN, NN,) through a mixed operator, and each node corresponds to type of input or a feature vector extracted from previous nodes through the mixed operators. The output of the mixed operator between (N, N2) is the gated sum of all candidate operations {ox {=
K
mmy(n)) =
Σ
gop On (m:),
(1)
where maj is the output of this mixed operator, n, is the feature vector of node N;. 9o is the binarized operation mask based on the softmax probability of architecture parameters a, in (Cai et al. 2018), and K is the number of operations inside a edge, which depends on the properties of node pair (Ne, N,). The nodes are densely connected, so that ny, the output features of node Nj. is the gated sum of features from all its previous nodes:
j-1
K yo
n
ge, may (n:)
Σ i=0
ge,
ok (ni);
where Ie is the binarized edge mask based on the softmax probability of architecture parameters Oee and N2 can either be an intermediate node or the output node.
During training, we retain two incoming edges for each node and one operation for each edge through the binary gate sampling in (Cai et al. 2018). While for inference, we pick two candi- date edges with the largest edge probabilities, and select the operation with maximum operation probability for each of the two edges. We choose two edges for each node to leave the potential for PhysiesNAS to learn complicated structures, like skip connection and multi-stream encoding. In order to learn both the network weights and the associated architecture parameters, we update these two sets of parameters alternately. In the architecture step, we freeze the network weights w and minimize the validation loss (wa a) by updating a. In the network step, we update w to minimize the training loss Ltrain (w, a) with frozen
3.3 PHYSICSNAS FEATURES
To incorporate priors into existing differentiable NAS (iu et al. 2018 Cai al. 2018 make three unique modifications into the search process of PhysicsNAS.
Physical Inputs As first step in blending physics into PhysicsNAS, need prepare unique input nodes that take into account four categories of input information: 1) the data input X; 2) the duplicated data input X Xdup to verify whether physical information is indeed necessary since each
4

---

Under review as a conference paper at ICLR 2020
What is the traj crajectory after tossing?
What are telocitss after collision?
Momentum conservation:
Vo
me Pat
my Magar Myy
12
G
Energy conservation: 2 mar if
2 Vii m, =
m
Random Wind
1/
Pel
We evaluate our method on simulator of classical tasks. The first task (Left) is predicting the trajectory of a ball being tossed, and the second task (Right) is estimating the velocities of two objects after collision.
Figure 4:
node has to pick two edges; 3) the estimated solution from physics Yphy = Aphy (X); and 4) the concatenation of X and Yphy to test which stage to conduct the physical fusion.
Physical Operations To merge physical models inside the network. we create physics-informed operation sets O = {oNN)..... ONN. Ny. Ophy where ONN, denotes the neural network operations (egg..ullly-ooneccte layer, skip connection) and Ophy denotes the physical forward operation. Specifically, for the physical forward module, we also use light-weight network layer, such as fully-comnected (FC) layer, make the size of its input consistent with the parameter size required by the physical module. Physical forward modules are only included in the edges that connect to the output node in our implementation.
Edge Weights In PhysicsNAS, not all edges are created with the same amount of operations, since they are used to connect different types of nodes. Consequently, if we select the edges purely based on the operation probabilities, edges with fewer operations are naturally preferred due to the softmax probability, which causes a biased architecture selection. We solve this issue by introducing the edge weight described in Equation 2 After searching, first pick a desired edge according to the edge weights, and then select the desired operation for that edge based on the operation weights.
4 EXPERIMENTS AND RESULTS
To comprehensively evaluate PhysicsNAS, we simulate two representative physical tasks for which we can vary the model mismatch: 1) predicting trajectories of an object being tossed; and 2) esti- mating the speed of rigid objects after collision. Figure 4 illustrates these tasks; further details are provided in Section 4.1 Comparison PBL architectures described in Section Finally, evaluate PhysicsNAS and provide a detailed analysis on the searched architectures in Section 4.3
4.1 DESCRIPTION OF TASKS
For the TOSSING TRAJECTORY PREDICTION task (see Figure 4 for visualization), the initial three locations of the object {4, 12, la} are given as imput X. and our objective is to predict locations of this object in the following 15 time stamps, {44, (s. ..., 41 We only consider the displacement within a 2D plane, therefore, the coordinates of each location can be represented by two numbers, i.e. = (Za,, ). We adopt the following elementary free-falling equations as the prior, and examine different methods under this inadequate physical prior:
flz: = lus + 4244
Pphy
[4 Im Uyli 90
(3)
where 4,, and ly. denote the object location at time t; Lz, and ly, are the initial location of the object, V. and vy denote the initial velocities along horizontal and vertical directions respectively, and g is the fixed gravitational acceleration of 9.8m/s². We introduce two model mismatches: the random acceleration as the winds and an additional damping factor based on Fair
x 8² to simulate the air resistance. The future locations estimated according to mismatched prior used the physical input Y pphy- As to the physical modules in Embedded Physics and PhysicsNAS,
5

---

Under review as a conference paper at ICLR 2020
estimate parameters flu lyz Den 0y and substitute these parameters into Equation as as the physical operation.
In the COLLISION SPEED ESTIMATION task (see Figure 4 for visualization),
the speed of
two objects at the initial two time stamps, object mass, and the distance between these two objects {Va1: Vag Uby bby May Ma; D) the input X to estimate the speed after their collision {vay Pb, }- We assume the objects have the different mass, and can only move along one direction. Based on energy conservation and momentum conservation for perfectly elastic collision, we adopt the following prior:
Pphy
Vay Ub,
my + [Wa (ma mm) + 2my
(4)
ma +-mg (06, (171 ma) + 2m
We add sliding friction to the system as intentional model mismatch, where conservation prior in Equation 4 does not hold. Solutions without the consideration of friction are used as Ephy. and {ma my, Vais 06. J are estimated for the physical modules.
4.2 MANUALLY DESIGNED PBL METHODS
For the sake of comparison, several manually designed architectures from Section 2 are also eval- uated. We use three-layer multilayer perceptron (MLP) as the naive data-driven baseline, since it has sufficient expressiveness to any continuous function, especially the elementary physical tasks we have chosen (Csáji 2001). Network structures for the Physical Regularization model and the Residual Physics model are the same as the naive model. The output of the Residual Physics model is a summation of Yphy and the learned residual from the network, while there is an additional regu- larization term in loss function of the Physical Regularization model. Since only partially correct physical prior is used, directly using physical solution as the regularization will in turn aggravate the error. Thus, we introduce ReLU-based regularization similar (Karpatne al.) (017). The reg- ularization loss penalizes the network solution based on the assumption that the object moves along one direction in the horizontal axis for the trajectory prediction task, and the total kinetic energy less than the initial kinetic energy for the speed estimation task. In the Physical Fusion approach, two separate branches are utilized to extract features from X and Yphg respectively. and each of them is a two-layer MLP. The extracted features will then be concatenated and fed into the output layer. The Embedded Physics model first estimates necessary parameters in Equation 3 and Equa- tion | with a three-layer MLP, and then produces trajectory estimation based on the fixed physical process. All the above models are supervised by the ground-truth future locations with mean square error (MSE) loss, and the hidden dimension for FC layers are 128.
4.3 RESULTS ANALYSIS
Training Details To evaluate the importance and success of the proposed approach, we vary the physical model mismatch and sparsity in training data in controlled manner. When training Physic- sNAS, we split the training set into two subsets of the same size to update architecture variables and network variables respectively. We limit the number of learnable nodes in PhysicsNAS be 5, and retrain the searched architectures with full training sets after searching. The models are implemented in PyTorch (Paszke al.) 2017). and are trained using the Adam optimizer (Kingma & Ba 2014). Moreover, for all baseline approaches we compare in this paper, we fine tune their hyperparameters in order to make fair comparisons. We choose three hyperparameter sets for each scenario and run five times for each method. We finally pick out the best result for each method.
Comparison We apply the proposed PhysicsNAS learn architectures embedded in the search space. The testing results of PhysicsNAS and other existing PBL methods (as detailed in Section (.2 are summarized in Table [ and Table 2 As shown these two tables, the perfor- mance of different PBL. models varies based on the disparity of mismatch levels and training data sizes, while PhysicsNAS is capable of generating architectures that outperform these manual PBL models consistently. Results in Figure 5 and physics dimension a fine- men-grained scale. We also conduct an ablation study on PhysicsNAS by removing task-specific adaptations such physical inputs and operations. Results in Appendix [ show the spec-spe adaptations improve the performance of PhysicsNAS at different mismatch levels, demonstrating that having physical inputs and physical operations in PhysicsNAS's search
Performance
further demonstrate this capability along data dimension

---

Under review as a conference paper at ICLR 2020
Mismatch Level
Low
High
Mismatch Level
Low
High
Sample Amount
32
128
32
128
Sample Amount
32
128
32
128
Naive Network
0.684 0.232 0.696 0.267
Naive Network
1.974 0.319
8.865
4.534
Physical Fusion 0.266 0.200 0.321 0.178
Physical Fusion
0.868 0.174
6.892
4.596
Residual Physics
0,323 0, 192 0,481 0.279
Residual Physics
1.053 0.173
11.750
5.859
Embedded Physics
0.617 0.169 0.617 0.305
Embedded Physies
2.105 0.258
7.962
4.546
Physics Reg.
0.459 0.272 0.674 0.315
Physics Reg.
1.916 0.271
8.631
4.451
PhysicsNAS 0.183 0.097 0.264 0.152
PhysicsNAS 0.724 0.121 6.741
4.157
Table I: Testing performance on tossing task. We adopt the average Euclidean distance be- tween the ground truth and the predicted loca- tions as the evaluation metric (lower distance is better). The low mismatch level corresponds to small random initial acceleration range [-1m/s², 1m/s²] and a small damping factor 0.2. The high mismatch level corresponds to a large acceleration range [-3m/ s. 3m//²] and large damping factor 0.5. The best model is marked in red and the sub-optimal is in blue.
Testing performance on collision task. We use similar Euclidean distance be- tween the estimated speed and the ground-truth speed as the metric (lower distance is better). The low mismatch level corresponds to a ran- dom initial friction coefficient in range [0.28, 0.32], and the high mismatch level corresponds to a friction coefficient in range [0.45, 0.55). The best model is marked in red and the sub- optimal in blue.
Table 2:
0.45
Physiral
Best PBL
0.325
Physical
Physical
Best PBI.
PhysinNAS
0.300
Fusion
PhysicaNAS
9.40
Testing Mena Egros
0.275
Residual
Piysical
0.35
0.225
Teston
0.30
0.200
Physical Fusion
Phyyicl Filhe
Physicul
0.175
0.25
Residual
0.150
Physics
0.30
Medium ris k:8.1
Low r:ll kc0.1
32
64
128
256
512
1-44 k:
riLl K31.
Physical Mismarch Levels
Training Sumple Amount
Figure 5: PhysicsNAS has lower errors compared with the best PBL methods over range of quality conditions in physics and data. The left figure shows comparison between best PBL methods and PhyisiesNAS along different physical mismatch levels. The physical mismatch levels are from extreme to low, respectively. Here (r ビi, k j) refers to the mismatch level of a initial acceleration range [-im/s², im/s²] and a damping factor j. Analogously, the right figure shows comparison along different data amounts. Results in the left figure are all trained with 32 samples and the results in the right figure are trained low physical mismatch level.The plots show error; lower curves are preferred.
space is necessary. Our experiments also show that PhysicsNAS is able to perform inference on small training datasets: the physical prior reduces the demand for high-fidelity training samples. We find that PhysicsNAS only requires less than 64 training samples to reach the same testing performance of a naive MLP with 256 training samples. Moreover, the performance gap between PhysicsNAS and naive MLP method minimizes as the number of training samples increases. This suggests that PhysicsNAS is more favorable in scenarios where training data is not rich enough on the other hand. A detailed discussion about how PhysicsNAS reduces the demand on training data is provided in Appendix E
Utilization of Physical Inputs The physical inputs are always selected in our searched architectures for these two tasks, which verifies the importance of physical information during learning. Please fer to Appendix [ for the illustration a range of searched architectures corresponding to scenarios in Table and Table
Utilization of Physical Operations The selection of thyysic--nsspire operations are more nuanced, depending the accuracy of physical information encoded in the physical operations well the amount of training data. Figure shows two examples, one where physical operations are selected and the other where they not selected. In particular, the inaccurate physical operation in the
7

---

Under review as a conference paper at ICLR 2020
trajectory task is preferred at early training epochs. However, training proceeds, the learned FC modules achieve higher accuracy and the network thus discards the physical operations. As a result, the Residual Physics strategy is adopted in the final searched architecture. For the collision task, the physical operation could model the perfectly elastic collision completely, and the estimated physical solutions operations are selected when there exists a robust estimation of physical parameters. However, it usually requires sufficient training samples obtain this robust estimation, which might be reason why physical operations are only selected in the cases with 128 training samples. More details about the changes of network architectures during searching can be found in Appendix
precise if the estimated physical parameters are accurate. Therefore, physical
Eurela
rars
plyy Jocwud
fime
mata
(a) Tossing task, high mismatch, 128 samples
(b) Collision task, high mis- match, 128 samples
Figure 6: Utilization of physical operations in PhysicsNAS. The selection of poyssis--nssired operation depends on its accuracy. PhysicsNAS tends to utilize the physical operations when they are more accurate (like the elastic collision model), and prefers a residual connection when they are inaccurate (like the parabola equation).
Failure Case In differentiable NAS, the training algorithm aims to optimize the over- weerpparameterized network with all the edges and operations. Therefore, it is necessary to prune the redundant edges and operations after training. We adopt the pruning mechanism (Liu et al. 2018), where each node has to retain two incoming edges. For extreme cases where single-stream architectures are optimal, PhysicsNAS may generate sub-optimal architectures due to this arrangement. As shown in Figure Z make toy comparison between two lightweight architectures the collision task with the friction coefficient range [0.15, 0.25] and 32 training samples. It notable that by simply adding an additional stream to the input x, the new searchable architecture deteriorates the result. Introducing an adaptive edge selection mechanism might be a meaningful future work. This limitation could also be overcome by resorting to other NAS frameworks, such as those based reinforcement learning (Pham et al. 2018).
relu
0
fc_our f_ouut
fc relu relre
fc_out
Hip_connect
y_hat
skip_connie _o_ooneect y_hat
y_phy
Y_phy
(a) Lightweight Residual Physics (Error: 0.650)
(b) Exampled searchable architecture. 0.835)
(Error:
Figure 7: Failure case. In rare situations, single-stream network could be preferred. PhysicsNAS unable to converge single-stream architectures due to the edge selection mechanism.
CONCLUSION
In conclusion, our experiments show that PhysicsNAS can handle wider range of input physical
models and data, as compared to existing PBL methods. This is only a first attempt at increasing
the diversity of PBL through architecture search. Ultimately, our hope is to apply PhysicsNAS to
problems as diverse as microscopy BaaescooyoBarbaasaee (Barbastathis et al. 2019), computer vision (eelten et al. 2012). sensor fusion (Eitel et al., 2015 Xu et al.] (2018) and astrophysics (Bouman al. 2016 Akiyama ctt al., 2019). where it is important to handle variations model mismatch and dataset quality across these problem domains.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anurag Ajay. Maria Bauza, Jiajun Wu, Nima Fazeli, Joshua Tenenbaum, Alberto Rodriguez, and Leslie P Kaclbling. Combining physical simulators and object-based networks for control. arXiv preprint arXiv: (xXiv:990400680) 2019.
Kazunori Akiyama, Antxon Alberdi, Walter Alef, Keiichi Asada, Rebecca Azulay, Anne-Kathrin Baczko, David Ball, Mislav Baloković, John Barrett, Dan Bintley, et al. First m87 event horizon telescope results. iv. imaging the central supermassive black hole. The Astrophysical Journal Letters, 87,,1), 2019.
Yunhao Ba, Rui Chen, Yigin Wang. Lei Yan, Boxin Shi, and Achuta Kadambi, Physics-based neural networks for shape from polarization. arXiv preprint arXiv:1903.10211 2019.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint arXiv:1611.02160 2016.
George Barbastathis, Aydogan Ozcan, and Guohai Situ. On the of deep learning for computa- tional imaging. Optica, 6(8):921-943, 2019.
Katherine L. Bouman, Michael D Johnson, Daniel Zoran, Vincent L Fish, Sheperd S Doeleman, and William T Freeman. Computational imaging for vlbi image reconstruction. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 913-922, 2016
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint arXiv: 00332, 2018.
Ayan Chakrabarti. Learning sensor multiplexing design through mack-propagation. In Advances in Neural Information Processing Systems, pp. 3081-3089, 2016.
Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. Inverse transport networks. arXiv preprint vrxiv:/809) /0820, 2018.
Huaijin Chen, Jinwei Gu, Orazio Gallo, Ming- Yu Liu, Ashok Veeraraghavan, and Jan Kautz. Re- blur2deblur: Deblurring videos via self-supervised learning. In 2018 IEEE International Confer- ence on Computational Photography (ICCP), pp. 1-9. IEFE, 2018.
Balázs Csanád Csáji. Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary, 24:48, 2001.
Steven Diamond, Vincent Sitzmann, Felix Heide, and Gordon Wetzstein. Unrolled optimization with deep priors. arXiv preprint 1.XXiv:1705.00044 2017.
Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Bur- gard. Multimodal deep learning for robust rgb-d object recognition. 2015 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 681-687. IEEE, 2015.
Xiaohan Fei, Alex Wong, and Stefano Soatto. Geo-supervised visual depth prediction. IEEE
Robotics and Automation Letters, 4(2):1661-1668, 2019.
Alexandre Goy, Kwabena Arthur, Shuai Li, and George Barbastathis. Low photon count phase retrieval using deep learning. Physical review letters, 121(24):44300, 2018.
Karol Gregor and Yann LeCun. Learning fast approximations sparse coding. In Proceedings of the 27th International Conference International Conference on Machine Learning, pp. 399-406- Omnipress, 2010.
Kerstin Hammemik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning a variational network for reconstruction of accelerated mri data. Magnetic resonance in medicine, 79(6)33055-3771, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.

---

Under review as a conference paper at ICLR 2020
Zhuo Hui, Ayan Chakrabarti, Kalyan Sunkavalli, and Aswin C Sankaranarayanan, Learning to sep- arate multiple illuminants a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3780-3789, 2019.
Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9): 4509 4522, 2017.
Eunhee Kang, Junhong Min, and Jong Chul Ye. A deep convolutional neural network using direc- tional wavelets for low-dose x-ray ct reconstruction. Medical physics, (e0(10)):660-33555 2017.
Anuj Karpatne, William Watkins, Jordan Read, and Vipin Kumar. Physics-guided neural networks (pgnn): An application in lake temperature modeling. arXiv preprint 1710. 1431, 2017.
Michael Kellman, Emrah Bostan, Nicole Repina, and Laura Waller. Physics-based learned design: Optimized sndeddillmmintiin for quantitative phase imaging. IEEE Transactions on Computa- fional Imaging, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1422.694 www..1112666. 2014.
Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1633-1642, 2019.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909. 9000065,,5, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv::806. /00.:/0000.0055, 2018.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. preprint Xrvi 17X..0/710.00666 2017.
Pde-net: Learning pdes from data, arXiv
Kristina Monakhova, Joshua Yurtsever, Grace Kuo, Nick Antipa, Kyrollos Yanny, and Laura Waller. Learned reconstructions for practical mask-based lensless imaging. Optics Express, 27(20): 28075-28090, 2019.
Elias Nehme, Lucien E Weiss, Tomer Michaeli, and Yoav Shechtman, Deep-storm: super-resolution single-molecule microscopy by deep learning. Optica, 5(4):458-464, 2018.
Thanh Nguyen, Yujia Xue, Yunzhe Li, Lei Tian, and George Nehmetallah. Deep learning approach for fourier ptychography microscopy. Optics express, (12222))6640-224448 2018.
Jinshan Pan, Yang Liu, Jiangxin Dong, Jiawei Zhang, Jimmy Ren, Jinhui Tang, Yu-Wing Tai, and Ming-Hsuan Yang. Physics-based generative adversarial models for image restoration and be- yond. arXiv preprint arXiv: 808. 008.00605, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le. and Jeff Dean. Efficient neural architecture 2018.
search via parameter sharing. arXiv preprint arXiv: arXiv:/802.000
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equa- tions. The Journal of Machine Learning Research, 19(1):932-95) 955, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv: 1111111111.0111... 2017.
Yair Rivenson, Yichen Wu, and Aydogan Ozcan. Deep learning in holography and coherent imaging. Light: Science Applications, 11):1-88, 2019.
10

---

Under review as a conference paper at ICLR 2020
Ayan Sinha, Justin Lee, Shuai Li, and George Barbastathis, Lensless computational imaging through deep learning. Optica, 4(9):1117-1125, 2017.
Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. End-to-end optimization of opties and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG), 37(4): 14, 2018.
Shuran Song and Thomas Funkhouser. Neural illumination: Lighting prediction for indoor environ- ments. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2019.
Russell Stewart and Stefano Ermon. Label-free supervision of neural networks with physics and domain knowledge. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Yu Sun, Jiaming Liu, and Ulugbek S Kamilov. Block coordinate regularization by denoising. arXiv preprint 19X5v:1905.05// 2019.
Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. Recovering dine--dimnssioal shape around corner using ultrafast time-of- flight imaging. Nature communications, 3:745, 2012.
Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bound- ing box estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 244 253, 2018.
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. arXiv preprint arXiv: /903:/::900.113333 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.0157,, 2016.
A GENERALIZABILITY FROM PHYSICAL PRIOR
We present a VISUAL PHYSICS task to further illustrate why physical prior is significant to data- driven approaches in qualitative manner. The network tries estimate the future trajectory of paper ball being tossed from the initial three frames. We deploy similar physical prior shown Equation 3 This simple physical prior fails to generate reliable estimation due to the physical disparity from real world, such as the existence of air resistance and the deformation of the paper during traveling. From pure data-driven approach, we train a ResNet18 (He et al., 2016) with 32 tossing samples. This network still could not generate reliable estimation due to the limited number of training samples. Most of the predicted trajectories are not physically plausible, since the ball should not change its position suddenly in Frame 4. For the PBL method, design physics- informed R with physical estimation as an additional input (Physical Fusion) and a skip connection between the physical estimation and the output (Residual Physics). The PBL network is also trained with 32 samples. With the assistance of physical prior, the trajectory could be estimated accurately. Some typical predictions the above three methods are illustrated in Figure
B COMPLETE LIST OF SEARCHED ARCHITECTURES
Searched architectures of PhysicsNAS in Table I and Table displayed in Figure Figure 10
C EFFECTIVENESS OF EDGE WEIGHTS
We evaluate the effectiveness of edge weights, and exhibit the advantage PhysicsNAS as com- pared to the original NAS framework. We conduct comparison between two different NAS poli- cies: 1) the NAS framework with physical inputs and operations, yet without edge weights; 2) the proposed PhysicsNAS with both edge weights and physical modules. The initialized architectures
11

---

Under review as a conference paper at ICLR 2020
Pryses Predition
Plyska based loaniing prodiciin
Graund
Ground trufh
Grundl
Figure 8: Even from limited training samples, PBL has strong generalizability as compared with conventional data-driven approaches. As exhibited in the left, estimation from physics can not match the ground-truth trajectory due to the physical disparity. As a data hungry method, deep neural networks also fail to generate physically plausible estimation with limited training samples (the middle). As depicted in the right of the figure, PBL method solves the dilemma by utilizing knowledge from both data and physical prior. The predicted trajectory (marked as green dots) from PBL highly overlaps with the ground truth in the testing stage.
skip connect
Fc out
skip connect
rolu
skip connect
1.2tg
rolu
to roiu
skip connect
play
relu
T: 7P
k.7.
(a) Low mismatch, 32 training samples
(h) Low mismatch, 128 training samples
dup
Ship connect
Ik y_plyy
To mlu
Sllp connect
fo.rola
Ie rala
f: out
y_Pby
(c) High mismatch, 32 training samples
(d) High mismatch, 128 training samples
Figure 9: Searched architectures of tossing task under diversified physical conditions.
before neural architecture searching are illustrated in Figure IT and the corresponding searched architectures are illustrate in Figure 12 We can observe that if the architectures shallow in the initialization stage, there is a high probability that they remain shallow after searching. This phenomenon may come from the nature of differentiable NAS training process: once an operation is selected in a training update, it obtains privilege over other operations due to the backpropaga- tion process, which in turn increases its weight and probability of being selected in the following training updates. Consequently, if the edges are selected merely based on the operation weights, NAS frameworks without edge weights will lose their potential to generate deep architectures due to the preference caused by difference in numbers of operations inside edges. PhysicsNAS addresses this issue by introducing additional edge weights, and the initial preference caused by unbalanced amount of operations within edges is thus alleviated. Comparison in Figure IT and Figure [2 is con- ducted on the tossing task as detailed in Section 3.1 The range of random accelerations is [-1 1mmm/s. 1m/s²] ] and the damping factor is set to be 0.2. There are 128 training samples used for architecture searching and retraining.
D EFFECTIVENESS OF TASK-SPPECIIR/PPHYIICC-- BASED ADAPTATIONS
We conduct an ablation study to demonstrate that the task-specific adaptations (i.e., adding physical inputs and operations into search space) in PhysicsNAS improve the predicted result. To this end, we conduct a comparison between: 1) the NAS framework with edge weights only: 2) the proposed PhysicsNAS with both edge weights and physics-based adaptations. The comparison is conducted
12

---

Under review as a conference paper at ICLR 2020
X dup
relu
ee_rllu
y phy
fc relu
fc relu
relu
Ic out
dup
Fo rotooo
retu
relu
out
y hat
skip connect
pky
To out
ship connect
rerrluu
forward
hat
y phy]
pay
ralu
relu
(a) Low mismatch, 32 training samples
(b) Low mismatch, 128 training samples
[x. (x.__hyy]
fc relu
y_pby
fc relu
fc relu
relu
x dup
fc relu
Tc relu
fc. skip connect
fc relu
phy forward
y ply
relu
rolo
Bc out
hai relu
relu
[x V. y_ply)
dup
out
colu
(c) High mismatch, 32 training samples
(d) High mismatch, 128 training samples
Figure 10: Searched architectures collision task under diversified physical conditions.
fc out
relu
phy forward y hat
relu
[x. y_pbyy
fc relu
fc relu
phy
ou:
bat
relu
To relu
rela
fc rolu
y_phy
tc relu
[x. pbyy
fo: relu
T6 retu
fc relu
relu
Ic relu
dup
tc relu
dup
relu
(a) Without edge weights
(b) With edge weights
Figure 11: Initialized architectures different NAS policies. The initialized architecture of NAS without edge weights is generally shallow, since the number of the candidate operations between the input nodes and the hidden nodes is less than the number of candidate operations between hidden nodes themselves.
using the tossing task. As can be seen in Fig. 13 the introduction of physics-bar based adaptations ables PhysicsNAS to achieve significantly lower prediction error as compared to the NAS framework without task-specific adaptations.
13

---

Under review as a conference paper at ICLR 2020
skip connect
y_phy
y_hat
Ic relu
relu
relu
dup
relu
Tc relu
relu
x. phyl relu
(a) Without edge weights (Error: 0.205)
(b) With edge weights (Error: 0.097)
Figure 12: Searched architectures using edge weights alleviates model collapse. The searched ar- chitecture without edge weights remains shallow due to the initial preference caused by unbalanced amount of operations, and its corresponding testing error is inferior to PhysicsNAS. The proposed PhysicsNAS has the potential to generate both deep and shallow architectures, while NAS with- out edge weights would generally converge to shallow architectures. The searched architecture of PhysicsNAS thus superior to the one without edge weights in terms of the testing error.
0.6
0.45
wia physics-hasod idtepaations
wit physics-haae adaptatiens physicelused adoptation
0.5
physic-based adsptiations
0.40
Testing Mexa Etror
Tallog Mean Error
0.35
0.30
0.25
0.18
0.1
k:06
r:A3 1:155
r:A3 lcll.
Phystcal Mismatch Levels
Training Sample Amount
Figure 13: PhysicsNAS has lower errors compared with NAS framework without physics-
based adaptations over range of quality conditions in physics and data.
E DATA EFFICIENCY OF PHYSICSNAS
We show that the introduction of physical prior knowledge reduces the data amount for learn- ing/approximating an exact physical model by comparing PhysicsNAS with naive MLPs along the data dimension. This comparison is conducted using the aforementioned tossing task in the high mismatch level scenario. As shown in Figure 14 PhysicsNAS can achieve significant, high pre- diction precision with limited data. As for this model mismatch type and level, PhysicsNAS can decrease the demand for training data by approximately 75%. This demonstrates the capability of PhysicsNAS on fewer-shot learning, where training data are burdensome to acquire due to extreme environments.
Performance gain decreases as data amount increases. As also shown in Figure [4 the per- formance gap between PhysicsNAS and the naive MLP continuously decreases the amount of training samples increases. When the size of the training data is increased to 1024 the performence of the two models is nearly identical, with the MLP method slightly outperforming PhysicsNAS. This indicates that performance gain for applying PhysicsNAS is more evident when the amount of high-fidelity training data is limited.
14

---

Under review as a conference paper at ICLR 2020
0.7
MLP PhysiesNAS
0.6
Testing Mean Error
0.5
0.4
0.2
0.1
32
128
256
512
1024
Training Sample Amount
Figure 14: Data Efficiency of PhysicsNAS. With rough physical prior, the demand of high-fidelity training samples can be greatly reduced. PhysicsNAS only requires less than 64 training samples to reach the testing performance of Naive MLP with 256 training samples. It is reasonable deploy PhysicsNAS to alleviate the burden of data acquisition when there are limited training samples.
F ARCHITECTURE CHANGES AT DIFFERENT EPOCHS
We show the usage of physical operations along with the number of training epochs in this section. In Figure [5] the experiment is conducted in the tossing task with low mismatch level and 128 training samples, while Figure [6] shows the architecture changes in the collision task with high mismatch level and 128 training samples.
1.2Pg
meh
FLida
e ferwer
TE. PPU
(a) Epoch:
(b) Epoch:
Figure 15: Changes of searched topology in tossing task. Inaccurate physical operation is used at early epochs, however, is discarded as the training proceeds.
x_ x_dup
fc_out
X y_phhy]
fc relu
phy forward y_hat
fe_relu
relu
connect
ft.out
relre
reku
fc. relu
slip. connect
y_hak
y_phy
fc relu
fe_out.
skip_ sdip_connect
X_dup
y_phy
rdu
felu relu
skip,con conject
fc_ relu
(x. phy]
(a) Epoch: 39
(b) Epoch: 355
Figure 16: Changes of searched topology in collision task. The physical operation is adopted initially, yet the searched topology eventually adopts the physical operation.
15

---

Under review as a conference paper at ICLR 2020
G UNCERTAINTY OF DIFFERENTIABLE ARCHITECTURE SEARCH
DARTS (Liu et al. 2018) framework tries to find an optimal architecture by updating the architecture parameters and network weights simultaneously. Therefore, the inconsistency between the architec- ture optimizer and the network weight optimizer may impede this joint optimization process. The performance of searched results relies on both the hyperparameters and the initialization, and im- proper training process may lead to model collapse as addressed in (Liang et al. 22019). PhysicsNAS built based on the DARTS framework, and the model collapse issue also exists. Please refer to Figure 17 for some collapsed models of PhysicsNAS in the two demonstrative tasks. In our paper, we avoid the model collapse by tuning the learning rates of architecture and network optimizers, so that architecture parameters and network parameters can converge synchronously.
Ic rolu
fc relu
Fc out
fc: relu
y haa
fc relu
dup
Ro.out
[x. y_phyy
relu
fc relu
ply
Rc. relu
fc out
fc relu
nelu
x dup
rolu
fcc
y.payl
Se zclu
Seep connect
retu
(a) Model collapse in the tossing task.
(b) Model collapse in the collision task.
Figure 17: Model collapse of PhysicsNAS due to the uncertainty of training. The performance of PhysicsNAS depends the differentiable searching process. Model collapse may appear if hyperparameters are not selected carefully.
16


Reviews:
[{"review_id": "rklrUUyI5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors apply neural architecture search techniques to the problem of physics based learning. It is interesting because it cleverly tackles the challenge of manually designing priors and network architectures. The results are also impressive as the proposed method surpasses all the considered baselines. Despite of the above upsides, I have the following questions/concerns.\n1. There is limited technical novelty as the entire method is mainly based on previous work on neural architecture search. Nevertheless, it might be helpful to have some ablation study to show the improvement of the task-specific adaptations presented in the paper, with which I believe this could be a good paper on the application side.\n2. I'm curious about the performance of the baseline methods given the same amount of computation. For example, is it possible to perform intensive hyperparameter tuning for the baselines to also obtain improvement. It seems that the authors did not discuss the computational costs and whether different methods are compared given the same cost.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxVmBAy9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes PhysicsNAS, which proposes a method to automatically design architectures that\nincorporate domain expertise from phyiscs-based models while also accounting for potential\nmismatch between the model and real world due to unaccountable factors. While existing work seem\nto incorporate such information via one of 4 standard ways (given on page 2), the proposed work\nattempts to meld them so as to find the optimal combination for the problem at hand and the data\navailable.\n\nWhile I don't see anything fundamentally wrong with the paper, I do not feel that the technical\ncontributions are substantial enough to warrant acceptance at ICLR.\nMore specifically, the methodological novelty is limited and the experimental evaluation only\nevaluates the method on two fairly simple problems. \n\nOn a positive note, the authors have done a good job of illustrating the idea and have compared it\nto most natural baselines. I also thought that the illustrations of the architectures found\nfor different sample sizes (in the Appendix) quite insightful.\n\nI encourage the authors to pursue this line of work, but test this on more complex prediction tasks\nwhere entirely model-based approaches are unreliable, and entirely black-box estimators are sample\ninefficient. It also seems that the approach need not be confined to physics per se - in many\nproblems in chemistry, materials science etc. scientists are looking for ways to incorporate domain\nexpertise while accounting for model-mismatch. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lojMTjKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Overall comments ===\nThis paper proposes to generalize approaches to physics-based learning (PBL) by performing network architecture search (NAS) over elements from PBL models found in the literature. This entails including physical inputs to the network and the incorporation of new operations to the NAS. I think the idea has merit and rather like it. However, there are several aspects of the work that could be improved. The technical novelty is small, as the extension of the  existing NAS models to handle physical inputs and a few new operators is relatively straightforward. The experiments, while well designed, only explore uninteresting toy problems. While I appreciate the necessity to explore the methods performance in a more controlled setting, a more impactful testbed would be more convincing. Another drawback of the evaluation is the lack of a proper statistical analysis of the results, given the small data and model sizes.\n\n\n=== Relevance & Prior Work ===\n+ The related work gives a good summary and categorization of prior work in physics-based learning\n+ The problem (physics-based learning) is interesting and relevant to the community\n\n\n=== Novelty & Approach===\n+ application of NAS to physics based learning\n+ incorporation of physics solutions as inputs into differentiable NAS\n+ creation of physics-informed operation sets to merge physical models into network\n- technical steps to merge NAS and PBL are relatively straightforward\n\n\n=== Evaluation ===\nTwo representative physical simulations were chosen for evaluation, where elements of the physics model are intentionally omitted,  1) estimating trajectory of a ball in presence of wind and air resistance, and 2) a collision speed simulation where two objects collide, where sliding friction is not accounted for in the physics model.\n\nThe baselines consist of: a 3-layer MLP (data-driven), a 3-layer MLP with Physical Regularization, a 3-layer MLP with residual connection to the physics prediction, an MLP with two input branches, on for the data and one for the physics predictions (Physical Fusion), and the Embedded Physics model which estimates parameters for the physics modelu using a 3-layer MLP.\n\nPhysicsNAS can combine elements of the baseline models, but the total number of nodes is limited to 5.\n\n+ Experiments testing the dependence of the model on the numbers of samples and the strength of the physical inconsistencies were conducted. In both cases, PhysicsNAS outperformed the best specialized physics models.\n\n- The chosen testbed tasks are toy problems. While these types of experiments are necessary to understand the performance of the model, it would have been interesting to see PhysicsNAS applied to a more impactful task\n\n- Given the size of the networks and the training data, there is no reason why a more sophisticated statistical analysis of the results wasn’t performed (confidence intervals, t-test, p-value). Similarly, a more complete set of experiments with more sample amounts could be provided with little effort.\n\n\n=== Clarity & Other Comments ===\n- “precious nodes” -> previous nodes\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper constitutes interesting progress on an important topic; the reviewers identify certain improvements and directions for future work, and I urge the authors to continue to develop refinements and extensions.
Decision: Reject