Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors apply neural architecture search techniques to the problem of physics based learning. It is interesting because it cleverly tackles the challenge of manually designing priors and network architectures. The results are also impressive as the proposed method surpasses all the considered baselines. Despite of the above upsides, I have the following questions/concerns.
1. There is limited technical novelty as the entire method is mainly based on previous work on neural architecture search. Nevertheless, it might be helpful to have some ablation study to show the improvement of the task-specific adaptations presented in the paper, with which I believe this could be a good paper on the application side.
2. I'm curious about the performance of the baseline methods given the same amount of computation. For example, is it possible to perform intensive hyperparameter tuning for the baselines to also obtain improvement. It seems that the authors did not discuss the computational costs and whether different methods are compared given the same cost.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes PhysicsNAS, which proposes a method to automatically design architectures that
incorporate domain expertise from phyiscs-based models while also accounting for potential
mismatch between the model and real world due to unaccountable factors. While existing work seem
to incorporate such information via one of 4 standard ways (given on page 2), the proposed work
attempts to meld them so as to find the optimal combination for the problem at hand and the data
available.

While I don't see anything fundamentally wrong with the paper, I do not feel that the technical
contributions are substantial enough to warrant acceptance at ICLR.
More specifically, the methodological novelty is limited and the experimental evaluation only
evaluates the method on two fairly simple problems. 

On a positive note, the authors have done a good job of illustrating the idea and have compared it
to most natural baselines. I also thought that the illustrations of the architectures found
for different sample sizes (in the Appendix) quite insightful.

I encourage the authors to pursue this line of work, but test this on more complex prediction tasks
where entirely model-based approaches are unreliable, and entirely black-box estimators are sample
inefficient. It also seems that the approach need not be confined to physics per se - in many
problems in chemistry, materials science etc. scientists are looking for ways to incorporate domain
expertise while accounting for model-mismatch. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: === Overall comments ===
This paper proposes to generalize approaches to physics-based learning (PBL) by performing network architecture search (NAS) over elements from PBL models found in the literature. This entails including physical inputs to the network and the incorporation of new operations to the NAS. I think the idea has merit and rather like it. However, there are several aspects of the work that could be improved. The technical novelty is small, as the extension of the  existing NAS models to handle physical inputs and a few new operators is relatively straightforward. The experiments, while well designed, only explore uninteresting toy problems. While I appreciate the necessity to explore the methods performance in a more controlled setting, a more impactful testbed would be more convincing. Another drawback of the evaluation is the lack of a proper statistical analysis of the results, given the small data and model sizes.


=== Relevance & Prior Work ===
+ The related work gives a good summary and categorization of prior work in physics-based learning
+ The problem (physics-based learning) is interesting and relevant to the community


=== Novelty & Approach===
+ application of NAS to physics based learning
+ incorporation of physics solutions as inputs into differentiable NAS
+ creation of physics-informed operation sets to merge physical models into network
- technical steps to merge NAS and PBL are relatively straightforward


=== Evaluation ===
Two representative physical simulations were chosen for evaluation, where elements of the physics model are intentionally omitted,  1) estimating trajectory of a ball in presence of wind and air resistance, and 2) a collision speed simulation where two objects collide, where sliding friction is not accounted for in the physics model.

The baselines consist of: a 3-layer MLP (data-driven), a 3-layer MLP with Physical Regularization, a 3-layer MLP with residual connection to the physics prediction, an MLP with two input branches, on for the data and one for the physics predictions (Physical Fusion), and the Embedded Physics model which estimates parameters for the physics modelu using a 3-layer MLP.

PhysicsNAS can combine elements of the baseline models, but the total number of nodes is limited to 5.

+ Experiments testing the dependence of the model on the numbers of samples and the strength of the physical inconsistencies were conducted. In both cases, PhysicsNAS outperformed the best specialized physics models.

- The chosen testbed tasks are toy problems. While these types of experiments are necessary to understand the performance of the model, it would have been interesting to see PhysicsNAS applied to a more impactful task

- Given the size of the networks and the training data, there is no reason why a more sophisticated statistical analysis of the results wasn’t performed (confidence intervals, t-test, p-value). Similarly, a more complete set of experiments with more sample amounts could be provided with little effort.


=== Clarity & Other Comments ===
- “precious nodes” -> previous nodes

review_title: Official Blind Review #3

Meta-review: This paper constitutes interesting progress on an important topic; the reviewers identify certain improvements and directions for future work, and I urge the authors to continue to develop refinements and extensions.
Decision: Reject