Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The core concern of this paper is the synthesis problem of choosing design parameters in a complex engineering design process mapping a high level goal to a high dim description of how the process achieves it. This is studied using the examples of 3D printing path planning and trajectory generation for a soft robotic arm. 

The core contribution is to set up this problem in terms of two stages - mapping goals to a space of designs, and then an evaluation of designs in terms of a cost function. This breaks down one of the difficulties identified - the ill-posedness of the problem due to multi-valued maps. So, the authors set up an autoencoder architecture with an encoder that conjectures designs based on goals, and a surrogate model to evaluate the designs. 
main_review: This is a clearly written paper on a timely topic.

The core contribution is the formulation factoring the evaluation of the design from the mapping of goals to designs, which breaks down the problem of many-valued mapping. Both parts are defined in terms of a differentiable module, so that the training can be achieved jointly. The evaluation methodology addresses what I would have expected to see - setting up baselines to show the extent to which each piece is relevant. So, one baseline asks how much is achieved by just mapping from goals to designs from historical data, while another asks how much is achieved by optimizing designs given the trained surrogate or decoder. 

I have only a few points of critique that I would have liked to have seen expanded in this paper. This is based on the acknowledgement that the technically arguments seem essentially sound to me, so my comments are mainly on formulation.

(1) Building on the notion of multiple ways for a design to be good, many engineering designs have path-dependent constraints. For instance, while moving fast there may be a different cost to lateral deviations than when moving slowly. This is a combination of dynamics (something not considered explicitly in the examples) and other forms of heterscedasticity or path dependence. To the extent that the surrogate and the decoder are fairly simple MLPs, it is unclear if such phenomena can be handled. I think this is an important consideration, especially if the use cases include robotics and cyber-physical domains involving control.

(2) While the introduction talks about considerations beyond the basic path, e.g., "ease of manufacturing" mentioned in sec 3, they do not feature later on. So, for instance, the soft body reaching problem could have been solved to some extent by multi-rigid body assumptions without seriously needing to acknowledge the infinite-dimensional space. On the other hand, there are design problems involving fundamentally different designs with discrete and continuous choices, e.g., multiple body morphologies involving soft components each (just for argument's sake). It is not clear if the proposed method would apply as described in these settings - calling for a clarification of generality and scope. 

(3) In the same vein, the design objectives here do admit a clear quantitative description always, e.g., distance from a desired path and reach error. There are many design problems where the objective is qualitative (e.g., any design from a topologically defined class will suffice). I expect that this might be easier to encode in this architecture but an example of this kind would have been helpful.

In summary, this is a clearly written paper whose methodology is sound. It would be stronger if the formulation's assumptions and limits are more clearly delineated.


limitations_and_societal_impact: This paper is a methods contribution which is likely to be used in systems such as 3D printers and robots. I believe the authors have taken due care in setting up and disclosing their software use.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper addresses the challenge of synthesis - where the goal is to find a configuration of a physical system that satisfies a set of constraints and optimizes one or more objective functions representing the desiderata. The synthesis challenge has three components - slow and costly realization process for evaluation of each design configuration, the physical realization or its software simulation being usually non-differentiable, and many-to-one mapping from the space of design configurations to the specification (constraints + objective function). Differentiable surrogates are used to address the first two challenges (to the extent possible for a realization process). The third challenge makes supervised learning difficult because the many-to-one mapping from configuration to realization to goal can penalize learned for good configurations/designs just because these were different from the training set. Adding more alternatives in the training set would also not help because the supervised learner would learn to predict average over the configuration space, which would semantically be not meaningful. 

The solution proposed in this paper is similar to some other recent work in this space which use autoencoders - a decoder captures many-to-one mapping from design configuration to design goals, and another network that maps goal to a design.  As described in the limitations section, the paper's claim to novelty is not fully justified by their choice of benchmarks and baselines, and limited discussion of related work. Further, the reviewer is not convinced (for reasons described in the limitation section) that the presented approach would be practical even for physical design problems (ignoring the larger synthesis literature). 
main_review: + The paper addresses a very important challenge of automatically finding configurations that meet design constraints and are optimum with respect to a given set of objectives. 

+ The approach is demonstrated on a path planning (for a class of 3D printers) problem and a snake-like soft robotics inverse kinematics problem. 
limitations_and_societal_impact: - The baselines (direct-optimization, direct-learning) selected for comparison are rather crude and simplistic, and do not reflect the state-of-the-art in this area. The approach presented in this paper is described more generically as a general synthesis solution (and reviewer believes this to be the case too) and similar generic synthesis approaches have been developed with focus on slightly different application (proteins). For example, see Polykovskiy, Daniil, et al. "Entangled conditional adversarial autoencoder for de novo drug discovery." Molecular pharmaceutics 15.10 (2018): 4398-4405, Hawkins-Hooker, Alex, et al. "Generating functional protein variants with variational autoencoders." PLoS computational biology 17.2 (2021): e1008736. The reviewer is of the opinion that either the paper can be rewritten as one focussed on planning/robotics (with an explanation about why this domain is different from other synthesis problems) and the presented solution can be described as uniquely better suited to this domain, or synthesis techniques from chemistry/biochemistry need to be considered as baselines and the presented approach compared on some case-studies on that literature. These examples will also be more practically relevant challenges than the simple examples considered in the paper. In its present form, the paper's claim to novelty is not matched with its empirical analysis.

- More realistic benchmark problems in the robotics domain for evaluating the design approach would be to pick examples from state-of-the-art robotics papers on design: Campos, Thais, Samhita Marri, and Hadas Kress-Gazit. "Automated Synthesis of Modular Manipulators’ Structure and Control for Continuous Tasks around Obstacles." Robotics Science and Systems. 2020, Campos, Thais, et al. "Task-based design of ad-hoc modular manipulators." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

-  The paper is also lacking discussion of related work to identify its incremental novelty over approaches developed there. Some relevant papers from ML venues include: Brookes et. al. "Conditioning by adaptive sampling for robust design." International conference on machine learning. PMLR, 2019. Fannjiang et. al. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 33.

- Leaving aside the experimental evaluation and the core idea of using an autoencoder to go from goals to design space and back, the method proposed in the paper is rather straightforward and lacks any theoretical analysis of complexity or guarantees on the produced design. While a convincing experimental analysis to demonstrate practical utility would have been itself sufficient, the paper doesn't appear to have either major theoretical result or practical insight that could improve the state-of-the-art in synthesis. 

- In real world, if we randomly sample designs, a vast majority would be actually invalid. Imagine putting together battery, speed control, propeller and motors to create a drone. In many cases, these combinations would produce current enough to burn the motor or torque high enough to tear the propeller. So, how do authors suggest building a dataset described in line 128-135. One reason why synthesis papers focus on biochemistry is this challenge of creating a dataset with physical designs. Path planning and kinematics are niche problems where each configuration (and configuration space is defined in a way to exclude impossible positions) leads to some measurable performance. These examples are not reflective of typical physical design problems.

- The paper adopts a rather simplistic approach to combined multiple objectives, which are common in design problems. The path planning example has "reach" and "avoid" as objectives, which are being weighed using hyperparameters. Typical physical design problems have several (10-100s) of optimization criteria and constraints, for e.g. a vehicle design might need one to look at track accuracy, thermal characteristics of motor, speed, etc. Combining them with hyperparameters would not be practically feasible (discovery of good values of these hyperparameters would be itself a hard problem).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers two concrete problems in fabrication and design: path planning in 3D printing and soft robotic control. In both cases, there is trouble resulting from solutions that are not unique, in the sense that the inverse image from the codomain (solution space) is not injective. The paper proposes using a neural network architecture for feature space learning, which helps to collapse dimensionality in a way that allows the recovery of more robust (i.e. not the mean) solutions.  
main_review: This paper is well written and addresses an interesting problem. The authors showed two concrete design and fabrication problems where the use of surrogate objectives is useful for overcoming lack of uniqueness in synthesis problems. The results presented in table 3 seemed to convincingly show that the author's method outperforms naive baselines on a soft robotics problem. 

The soft robotics problem, in particular, was a good setting. And I quiet enjoyed reading about the difficulties that arise in PDEs with non-unique solutions at the boundary. This is a very real world problem with an interesting machine learning solution, and I think it is exactly the type of work we should encourage at this venue. 

Overall, the encoder-decoder architecture, such as it is, does not introduce anything new or interesting to the field. Indeed, the authors never make such claims. There are many alternative ways that the authors could go about this process, including adapting GAN-based and VAE-based approaches. Did the authors try any other approaches for their encoder-decoder architecture? Can you include a table with the dataset size and network parameters (optimizer, number of layers, etc)? I am a little confused about why the network took so long to train. 

The data generation process also leaves something to be desired. Generating synthetic test data seems highly problem dependent. And the experiments in soft robotics in particular left me quite worried that the authors are just showing that a neural net can fit to a piecewise linear model. Or a simple piecewise convex model. With synthetic data, you always have to be quite careful that you're not simply investigating the ability of neural networks to approximate linear functions, etc. Although if Figure 4 is to be believed, it does seem like this works quite well in the real world. 


The author's worked on interesting problems and developed a machine learning solution that significantly improves the final outcome. Surely, that's worth acknowledging. 
limitations_and_societal_impact: The authors discuss the societal impact of automation of labor. As they point out, this type of algorithm is more of an IA proposition. There is no serious concern. 

I would've liked to see further comparisons to different types of network architectures, even if only in an appendix. I'm guessing the network architecture is not very optimized. Since there was no benchmark presented where the author's method fails or even does somewhat poorly, it's hard to know where failure would occur. I always like to see more stress testing. Are neural networks even needed? What if we replace the encoder with a linear encoder? How well would that do? I'd just like to see the author's dig deeper into the model so that I can better understand why it works and where I can expect it to fail. 

Finally, I should point out that fabrication is not my speciality. And I may have missed something.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a novel method for addressing synthesis problems, where they define a synthesis as an object or configuration that meets a set of constraints while optimizing one or more objectives. The authors motivate the problem with manufacturing settings, such as an additive printing path and extruding sequence, as well as soft robotic motion problem that can be described by a partial differential equation (PDE). The authors aims to address a couple of challenges:
* Provide a differentiable surrogate for non-differentiable simulations in synthesis problem. The claim and show that this can help with learning a differentiable model of the process that can help learn how to generate viable design via a synthesis process.
* Propose and develop a method that can solve many-to-one synthesis problems using a two-stage where one neural network based model learns how to generate designs and another model learns how to evaluate designs. Since both models are differentiable, the authors can train their framework end-to-end, which they claim provides accuracy improvements compared with directly learning a design to evaluation map and compute improvement compared to direct optimization of design to evaluation using method like BFGS.
* Provide a series of objective functions that can evaluate various synthesis design for a diverse set of use cases, which the authors then use to inform their two-stage approach.

The authors outline the objective function for an extruder path planning case and a constrained soft robotic motion task. In both cases they optimize the objective and perform the synthesis task using their approach as well as the direct learning (design -> evaluation) and direct optimization (BFGS) baseline. The authors claim, and the experimental supports, that the proposed two-stage method has a performance advantage of direct learning and a compute advantage, in inference, over direct optimization. The authors also discuss how even the training cost for their method is higher compared to direct optimization, the training cost gets amortized in inference as the two-stage does not require re-training/re-running like direct optimization.
main_review: Nits:
* Checklist should start on a separate page after the references

Overall, I think the paper outlines a novel approach to a significant problem, provides thorough details on the method and relevant experiments, and has strong clarity in the writing and presentation. Please see the subsequent sections on suggestions on how to further improve the paper. 

**Originality:** The paper tackles a new and relevant problem with a novel deep learning based method that involves end-to-end training of a differential surrogate model and a design generator. I think the approach is novel approach to synthesis problems the authors describe, which are a new set of challenges for deep learning methods.

**Quality:** The authors provide mostly thorough details on their method, formulation and experimental test cases. In order to improve the score, I would to get more clarity on the following questions:
* How does the "sliding window" work to generate an extruder path? Since the is sequential and time dependent, does the "encoder" produce the full directly or does it produce it sequentially? If is sequentially generated, how does the input change over time?
* What are the size of the design spaces for both cases? This would help get a sense of the scale of problems the paper tackled. It would also be useful to have a discussion on how the current approach could scale to larger and more complex challenges. 
* Did you do an ablation of different loss functions? You discuss the importance of the Chamfer distance for the extrusion case, which is helpful, and I would be interested to learn more about how potentially naive loss function affect performance. I assume they would make things worse, but it would be helpful to get more detail on that. 
* A discussion on future work/directions of the work would be nice.

**Clarity:** Overall, the paper is well written and presents relevant equations, formulations and diagrams to understand most of the relevant elements of the study. As mentioned above, to further improve on the clarify it would be helpful to get more detail on the input & output setups and relevant dimensions of the design cases, especially the outputs. There are more details on the inputs in the appendix, but the outputs are still somewhat unclear.

**Significance:** The paper addresses the application of a novel deep learning method to a non-traditional challenges that is significant. 

limitations_and_societal_impact: The authors briefly address limitations in the last section of the paper. As I mentioned in the previous section, it would be good to get a sense of scale of problems that authors think their method would be able to tackle. While the presented problems provide a good basis for proof-of-concept, what challenges do you anticipate in moving to larger and more challenging synthesis problems?
For the societal impact perspective, it would be good to include a discussion on the safety and reliability of methods like the ones the authors proposed would affect the over quality and process of synthesis processes, particularly in the manufacturing challenges discussed.
review_title: null

Meta-review: Thank you for your submission to NeurIPS.  Even after discussion, there is some substantial disagreement on this paper, so I will need to express an opinion that is not a unanimous one.  But with this caveat, I will state that I ultimately come down very much on the "positive" side of the disagreement, and I am recommending the paper be accepted to NeurIPS as a spotlight.

The rationale for this is rather simple: although currently confined to rather simple domains, the paper makes a compelling case that introducing a differentiable surrogate into synthesis problems, in a disciplined and generic fashion, can provide substantial gains over direct learning on the system of interest.  The authors do an extremely good job presenting detailed (and real, even if they are simple) evaluations of this strategy.  Thus, in total the authors present a compelling (_in_ its simplicity, rather than this being a flaw) approach to constrained synthesis, and show that across multiple realistic domains it outperforms the most common baseline.  Those are all points of a strong paper.

Now let me address the negative reviewer's concerns, which in my mind boils down to the concern that 1) the domains evaluated here are not particularly challenging and 2) don't reflect the vast amount of work that is already done in incorporating neural networks into synthesis problems.   I think it would of course be good to mention some of these connections (in the discussion, not in direct comparison) in the final draft of the paper.  But it is also the case that the landscape of integrating neural networks into specific physical synthesis domains is _vast_, and attempting to consider the scope in which all these approaches have been applied previously wouldn't be feasible.  Yes, for any given domain there are likely multiple NN-based solutions to synthesis problems, but attempting to consider all of these, in my opinion, would make anything but extremely domain-specific papers impossible.  The current paper makes no promises of extending the state of the art in a given practical domain (say the chemistry domains mentioned by the reviewer), but does offer evidence of a generic and compelling approach, and I believe these should be welcomed even if they are not (yet) as in-depth within a given domain as would be ideal in the longer term.  I think this is likely to be a fundamental disagreement as to the "right" focuses of papers like these, so while I don't expect to resolve this dilemma (and I certainly understand the reviewer's opinion), I still lean strongly towards accepting the paper.
Decision: Accept (Spotlight)