Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate
Authors: Xingyuan Sun
Tianju Xue
Szymon Rusinkiewicz
Ryan P Adams
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Amortization
Inverse design
Differentiable surrogate
3D printing
Soft robot

Abstract: In design, fabrication, and control problems, we are often faced with the task of synthesis, in which we must generate an object or configuration that satisfies a set of constraints while maximizing one or more objective functions. The synthesis problem is typically characterized by a physical process in which many different realizations may achieve the goal. This many-to-one map presents challenges to the supervised learning of feed-forward synthesis, as the set of viable designs may have a complex structure. In addition, the non-differentiable nature of many physical simulations prevents efficient direct optimization. We address both of these problems with a two-stage neural network architecture that we may consider to be an autoencoder. We first learn the decoder: a differentiable surrogate that approximates the many-to-one physical realization process. We then learn the encoder, which maps from goal to design, while using the fixed decoder to evaluate the quality of the realization. We evaluate the approach on two case studies: extruder path planning in additive manufacturing and constrained soft robot inverse kinematics. We compare our approach to direct optimization of the design using the learned surrogate, and to supervised learning of the synthesis problem. We find that our approach produces higher quality solutions than supervised learning, while being competitive in quality with direct optimization, at a greatly reduced computational cost.

Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate
Xingyuan Sun', Tianju Xue², Szymon Rusinkiewicz', Ryan P. Adams
Department of Computer Science *Department of Civil and Environmental Engineering
Princeton University
(xs5, txue, smr, rpay@princeton.eer
Abstract
In design, fabrication, and control problems, we are often faced with the task of synthesis, which we must generate an object or configuration that satisfies a set of constraints while maximizing one or more objective functions. The synthesis problem is typically characterized by a physical process in which many different realizations may achieve the goal. This many-to-one map presents challenges to the supervised learning of feed-forward synthesis, as the set of viable designs may have a complex structure. In addition, the non-differentiable nature of many physical simulations prevents efficient direct optimization. We address both these problems with two-stage neural network architecture that we may consider to be an autoencoder. We first learn the decoder: differentiable surrogate that approximates the many-to-on physical realization process. We then learn the encoder, which maps from goal to design, while using the fixed decoder to evaluate the quality of the realization. We evaluate the approach two case studies: extruder path planning in additive manufacturing and constrained soft robot inverse kinematics. We compare our approach direct optimization of the design using the learned surrogate, and to supervised learning of the synthesis problem. We find that our approach produces higher quality solutions than supervised learning, while being competitive in quality with direct optimization, at a greatly reduced computational cost.
Introduction
One of the ambitions of artificial intelligence is automate problems in design, fabrication, and control that demand efficient and accurate interfaces between machine learning algorithms and physical systems. Whether it is optimizing the topology of mechanical structure or identifying the feasible paths for a manufacturing robot, we can often view these problems through the lens of synthesis. synthesis task, we seek configurations
physical system that achieve certain desiderata while satisfying given constraints; i.e., we must optimize a physically-eealizabl design.
In this work, design refers to the parametric space over which we have control and in which, e,g., we optimize. A reulization is the object that arises when the design is instantiated, while goal refers to its desired properties. For example, in fabrication, the design might be a set of assembly steps, the realization would be the resulting object, while the goal could be match target dimensions while maximizing strength. Synthesis, then, refers to finding a design whose realization achieves the goal.
Synthesis problems are challenging for several reasons. The physical realization process may be costly and tone-consuming, making evaluation of many designs difficult. Moreover, the realization process even simulation of it-is generally not differentiable, rendering efficient gradient-basse methods inapplicable. Finally, there may be a many-to-one map from the parametric space feasible and equally-desirable designs realizations; i.e., there may be multiple ways achieve the goal.
Surrogate modeling is widely used to address the first two challenges, though it can still be expensive because of the need for optimization, sampling, search algorithms find a feasible design. More seriously, the third challenge-lack of uniqueness-create difficulties for naïve supervised learning
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Target
Fixed sturting speed
Design
(b) carben fiber reinforced "dog bone"
Pavus
Without planning
Design 1
Obstacle
@: angle (want to salve)
larget distance (imput)
Design 2
Design 2
Extruder path
Fiber path
(d) case stady 2: constrained soft robot inverse kinematics
(a) toy example
(c) case study 1: extruder path planning
Figure I: (a) For fixed and large-enough starting speed, there exist exactly two angles such that the ball will hit the target, where the mean these two angles is TT/4 4. (b) Some 3D printers utilize fibers reinforce the thermoplastic print. (c) For such printers, fiber is laid out along an extruder path but deforms into smoothed version due to the fiber's high stiffness and low stretch. Our goal is to generate extruder paths that compensate for the smoothing. but multiple extruder paths can result in the same target shape, such square. (d) In soft robot inverse kinematics, Our goal is to reach the target while avoiding an obstacle but, as is illustrated, the solution is not unique two different designs are shown.
control the stretch ratios of both the left- and right-hand sides of snake-like rohot.
approaches to synthesis. Specifically, consider generating many design/realization pairs, evaluating the constraints and objectives on the realizations, and attempting to learn a supervised map from goal back to design. When multiple designs lead to the same realization, multiple realizations achieve the same goal, the supervised learner is penalized for producing designs that are valid but happen to not be the ones used to generate the data. Moreover, this approach may learn to produce an "average" design that is actually incorrect. Figure shows a cartoon example: if the goal to throw a ball to reach some target distance, there are two possible launch angles (designs) resulting in landing points (realizations) at the correct spot. Performing least-squares regression from distance to angle on the full set of distance/angle pairs, however, learns an average angle that does not satisfy the goal.
To address these challenges, we propose to use a two-stage neural network architecture that resembles an autoencoder. One stage (the decoder) acts as differentiable surrogate capturing the many-to-on physical realization process. The other stage (the encoder) maps from a goal back to a design but, eritically, it is trained end-to-end with loss in the space of realizations that flows back through the decoder. Thus the encoder- our central object of interest for synthesis is not constrained to match specific design in training dataset, but instead is tasked with finding any design that meets the desiderata of the realized output. The result a neural network that performs amortized synthesis: it is trained once, and at time produces design that is approximately optimal, using only a feed-forward architecture. Note that our method not an autoencoder, as the design is not a lower-dimensional representation of the goal, and the encoder and the decoder are trained in separate stages.
Our method places number of requirements on the synthesis problem. First, train the surrogate, we need data pairs of designs and realizations. Commonly, this would require us to generate designs, in which a substantial amount of designs are viable, and simulate them on simulator, Second, given our current setting, the physical realization process needs to be deterministic. Third, to train the encoder, the synthesis problem should have a clear objective function, or at least we can quantify the objective. Finally, given our current feed-forwar setting, we consider synthesis problems that need only one valid design, although we may further extend our method by using generative encoder.
In this work, we demonstrate this two-stage approach on a pair of specific design tasks. The first case study is extruder path planning for a class of 3D printers (the Markforged Mark Two) that can reinforce polymer layers with discrete fibers (Figure [b). Since the fibers are stiff, their shape is deformed after extrusion (Figure Dc. top row), and our task to find an extruder path that results in given fiber shape. As shown Figure [P. this problem has the many-to-om nature described above: for a small error tolerance on fiber path, there exist infinitely many extruder paths, which may even look very different. The second case study is constrained soft robot inverse kinematics. In this work, we use a simulation a snake-like soft robot as in Xue et al. [97], in which we can control the stretch ratios of each individual segment on both sides of the robot. The robot has reach
2

---

a target while avoiding an obstacle, and the locations of both are input goals. As before. there may be multiple solutions for given goal (i.e., locations of target and obstacle), as shown in Fig neural
For both case studies, we compare to two baseline algorithms. In direct-learninn network for the synthesis problem (i.e., from goal to design) is trained in a supervised manner on a set of designs. Since this effectively averages designs in the training dataset, as argued above, our method outperforms it significantly. The second baseline is direct-optimizz matiiiiiiooin which uses a gradient-basse based method (BFGS) to optimize for each new design separately, given access to the trained differentiable surrogate for the realization process (decoder). Our method is competitive with this rough "performance upper bound" while using dramatically lower computational resources.
2 Related work
Machine learning applications in synthesis problems. In synthesis tasks, the aim is find
design solution such that its realization achieves one or more given goals. Usually, the solution is non- unique, and only one is needed. In molecule discovery, one would like to find molecule which has
some desired properties (e.g., minimal side effects, efficacy, metabolic stability, growth inhibition) [20]
46
83]. See Vamathevan et al. [90], Chen et al. [66] for surveys on machine/deep learning in drug discovery. Challenges of molecule discovery include discrete design space [22 76 and limited data [47] due to difficulty in simulation. In materials synthesis, researchers seek materials with specific properties. See Bhuvaneswari et al. 7 for review. Similarly, to obtain enough training data, researchers have put efforts into parsing and learning from scientific literature in natural language [52
In 3D shape generation, one wants to find a 3D shape that has some desired properties: properties like 2.5l) sketches (95] that can be easily calculated and properties like functionality (34] that need to be numan-labele In topology optimization, researchers aim to maximize the system's 's performance by optimizing the material layout given boundary conditions, constraints, external loads, etc [80] 5 133. Machine learning has been used to infer properties (75), find representations of designs [48 99], and directly generate designs [55]. In program synthesis, the goal is to find programs that realize given intentions (e.g., generating 3D shapes, answering visual questions) [35, 88 98), where machine learning has been used to generate programs and execute programs. In this work, we propose approach to learn feed-forward neural network that can directly and efficiently produce a feasible solution to a synthesis problem that satisfies the requirements mentioned above.
45)
Surrogate/oracle-bast synthesis. Surrogate/oracle-bass synthesis uses an auxiliary model- surrogate (or sometimes called tracle)-that can evaluate qualities of a design without time- consuming laboratory experiments, while still being reasonably accurate (58) 9. Surrogate models can be physics-based or approximation-basee ing techniques for approximation- based surrogates, including polynomial regression, radial basis functions, Gaussian processes, and neural networks [58 39 Bhosekar and Ierapetritou O pro- vide a review of surrogate- baarog methods, and see Koziel and Leifsson [56) for general review of surrogate models in engineering. There are two major approaches: optimization and sampling. The most common approach is surrogate- surrogate-based optimization; see Forrester and Keane [29] for survey. Some application examples include: optimizing the parameters of a CPU simulator [72]; solving partial differential equations in service of PDE-constrained optimization 197: : and optimizing stochastic non-differentiable simulators (79. Researchers have also developed methods to deal with the challenge that inputs can be out-of-distribution for the oracle [27] 30 89). On the other hand, sampling- based methods have several advantages: the design space can be discrete and can generate multiple designs [9] 110 24). Researchers have successfully applied sampling methods to problems in chemistry [32) and biology [16 73.. In this work, we use surrogate- based optimization as one of our baseline algorithms, and our proposed method uses surrogate during training to optimize for network for the design problem.
[57 (i.e., empirical), and there are different model-
Differentiable surrogate of losses in machine learning. Since some loss functions in machine learning are not differentiable (e.g., IoU for rotated bounding boxes, 0-1 loss in classification), researchers have proposed to learn surrogates for them. Grabocka et al. (33) provided a formulation of surrogate loss learning and compared several learning mechanisms on some commonly used non-differentiable loss functions. Liu al. [62] proposed general pipeline to learn surrogate losses. Bao et al. (2) Hanneke et al. [40] provided theoretical analyses of surrogates for 0-1 loss in classification. Patel et al. 770,, Nagendar et al. (64), Yuan et al. [100] explored the use of surrogate losses in various real- world tasks, including medical image classification, semantic segmentation,
3

---

and text detection and recognition. In this work, due to the non non-uniqueness of design solutions, we further extend this idea from loss functions to more complex, physical realization processes.
Robot motion planning. Robot motion planning [59] can also be viewed as form of synthesis, and there are different types of approaches to One popular strategy is optimization. For exam- ple, researchers have used evolutionary algorithms (EA) [60,
12 42, including variants of genetic 53 28 and covariance matrix adaptation evolution strategy 50
algorithms (GA)
18
17 87
(CMA-ES) 337. Another example is constrained optimization [94 85 21 23]. which includes exten- sions like sequential quadratic programming (SQP) [15] 37 and the DIRECT algorithm [9]]. Other examples of optimization applied to robot motion planning include reinforcement learning [8] 25
and teaching-leanne learggllaannn-bbaae -banng optimization 82] Another popular class of methods is sampling, in-
cluding simulated annealing (SA) 4 69 101,, probabilistic roadmaps (PRM) 44 , rapidly exploring
random trees (RRT) [4] 4 [3), etc. Finally, search methods, e.g., A search [8 have been used to solve motion planning problems. These works usually model the robot's 's physies directly without using& a surrogate model and solve the design using methods including optimization, sampling, and search, which can be time-consuming. Our method amortizes the cost of inference, and can be used to solve planning problems with more complex physics that cannot he directly modeled. In our first case study, we plan for trajectory of the extruder, and our design space is not the speeds of the motors but rather the coordinates of points along the trajectory. In the second case study, rather than solving dynamic planning problem, we solve static planning task on soft robot, with the stretch ratios of all the controllable segments of the robot as the design space.
Path planning in 3D printing. Path planning is one of the most important problems in 3D printing. and people plan for different objectives: minimizing printing time, avoiding collision, faster planning, etc. Shembekar al. 778 proposed a planning algorithm to build complex shapes with multiple curvatures and can avoid collisions. Ganganath et al. [31] minimized the printing time by modeling
the task as a traveling salesman problem. Xiao et al. [96] speeded up path planning algorithms by
introducing efficient topology reconstruction algorithms. Stragiotti (84) provided an optimization- based algorithm that minimizes compliance of a printed part. Asif (I introduced a planning al algorithm for continuous fiber and can generate continuous deposition path. See Huang al. [3 for review of existing work in 3D printing path design. In this work, instead of the aforementioned objectives, we plan extruder path to compensate for the deformation caused by the printing process. We demonstrate increased run- run-time efficiency by amortizing the cost of physical simulation of the printing process, learning feed-for ferdffr network output the extruder path.
3 Method
We formalize synthesis as a constrained optimization problem, denoting the set of allowed designs as é and the set of possible realizations as U. There is physical process that maps from design to realization that we denote as U θ - U. Our goal may be a function of both the realization and the design, as designs may differ in, e.g., ease of manufacturing. Moreover, it may be appropriate to specify parametric family of goals to accommodate related tasks, e.g., different target locations in inverse kinematics. The user expresses (g-indexed) family of design goals via a cost function denoted Lg x U - R. The problem of interest is to optimize the cost function with respect to the design:
min Cg u) s.t. U()) = u. GE0
We can view this problem as a generalization of PDE-constrained optimization problems, where we have allowed for broader types of realizations than PDE solutions. Revisiting the challenges of synthesis problems articulated earlier, U may be expensive, non-differentiable, and non-injective, and Ly may not have unique minimum.
Recalling the toy problem of Figure D 0 is the angle at which the ball is thrown, U(A) = u is where it lands, the goal g is a desired distance, i.e., a target realization, and (design-independent cost function might be the squared difference between the desired and actual realizations: L, (0, u) = Ilg u u1.
Since the realization u U0) is unique for specific design 0, we propose using a two-stage approach, which can be viewed as an autoencoder. We first learn a differentiable surrogate (de- coder) (-) for the physical realization process from 0 to and then learn an encoder d(.) from goal g to design 0, evaluating the design quality with the trained decoder. To build a dataset, we have to randomly sample designs and calculate realizations and goals from them, since the physical realization process (-) is known, and the reverse direction (i.e., goal to design) is difficult as dis- cussed before and is what we are trying to learn. Thus we first sample D designs .,,... Op and

---

where u, == (6;) and 9: is the goal calculated from the realization u, The calculation of the goal gi summarizes properties that we care about in ua; and this process depends on the synthesis problem itself and how the cost function (:;-) is designed. We split D into Drrain: Dval, and Deest- We then train surrogate Ú.) such that (')+) arg min )~ (0) =ul2),
build our dataset as D = (01, (1.g1). (....)).....0))) (0p.
uD.gD));
U()
so that (.) serves differentiable surrogate of the physical realization function U(.). We then use the trained decoder (-) train an encoder ((-):
('() = arg min (..
(6(g), ((())))))()))))
(3)
Note that although our method resembles autoencoder, it is not autoencoder, since we learn the decoder first and then the encoder rather than jointly. Besides, the design space is usually not lower-dimensional representation of the goal space, and the cost function is usually not simply reconstruction loss.
4 Empirical evaluation approach
In this work, we use our method to solve two real problems as case studies: a task of optimizing the extruder path in additive manufacturing, and a task of actuating a soft robot in an inverse kinematics setting. While the details of the two differ, we evaluate them in the same way. In each case, we compare our algorithm with two baselines- poss-diree baselines-drrect werrt-leerninn and direct-optimizatinn evaluate our method in terms of relative quality and run time.
and
direct- lrarnnnle One natural baseline directly learn model from the goal g to the design model @ (-):
Thus we introduce the
(.) = arg min IE ()) [|0 (())1 Rai (o(g))
(4)
+(-)
Note that, since there is no surrogate, we do not have access to the realization u. We thus have to slightly adjust the cost function into squared Euclidean distance part and a regularizer part Ra (-) (if there is one), the latter which comes from the original cost function L. (...)). We expect our method to perform much better than
Since our surrogate is a neural network and therefore differentiable, an- other natural baseline is to directly optimize the design f with respect to the goal g by using gradient-based optimizer (e.g., BFGS), which provides us a rough performance "upper bound" on method:
sirect-optimization
Φso (g) = arg min Lg (0, (0)).
(5)
0
Note that AmorFEA 97 is state-of-the-art method for PDE-constrained optimization, which uses the baseline. We expect our method to have performance while running orders of magnitude faster.
same approach the dereettptimizzee
close to serrectoptimizateee
To evaluate our method and the baselines, iterate over all g from Diest, and evaluate the quality of each tuple (K(g), U (((g)) 9). where we use the physical realization function U(-) rather than
the learned surrogate Ü' (-). We run every experiment times with random initialization of neural networks. More details are provided in each case study and in the appendix.
5 Case study: extruder path planning
3D printing (a.k.a. additive manufacturing) is the process of creating a 3D object from a 3D model by successively adding layers of material. It has a wide variety of applications in areas including aerospace, automotive, healthcare, and architecture (77]. Popular 3D printers use thermoplastic polymers (PLA, ABS, nylon, etc.) as the printing material, but these have limited strength. To address this issue, some recent printers support using strong fibers to reinforce the composite. In this work, we explore a 3D printer (the Markforged Mark Two) capable of extruding discrete fibers (fiberglass, kevlar, or carbon fiber) along controllable path. However, since the fibers are stiff' and non-stretchable, the printed fiber path will be "smoothed" compared to the extruder path. As shown in Figure D. without path planning, the fiber path will be severely deformed. We seek to find a general method that, given any desired fiber path, plans an extruder path to compensate for the deformation caused by the printing process. To the best of our knowledge, there is no existing automated method for this task, so is worthwhile to tackle it using machine learning.
5

---

A1 1-- (1-)))))
Extraaer gatti
Flber path
Random generator
Train
Sample curves
Synthetic
Decoder (Surroaate) (*())
Data generation
training
Loss
Exrmder path
Extraler pathi
Fila path
path
Exumaeer puth
Fiber gutts
A31 71%
Fiber
Fiber
BPCS optimizer
Encoder ("()
Trained decoder si
Synthetic
Direct learning dant!
Trained decoderr
Direct kumming
Direct optimization
Figure 2: The pipeline for method and two baselines for extruder path planning. We first generate data by building simulator and calibrating using real printer, and train the decoder and direct-learne using the synthetic dataset. then train our method and build direct-optimizan using trained decoder.
5.1 Cost function
Following the notation in Section we denote the target fiber path as g =R= path is represented as a series of n points, and n varies from path to path (at the scale of hundreds in our experiments). We denote the extruder path (the design) 0 gnx2 and its realization fiber path as u _ Our cost function C. -.,..) is defined as:
(0 u) = وgg - u112 + R(O),,
(6)
where À is hyper-parameter and R(-) is smoothing regularizer that calculates the of squared empirical second-order derivatives of the extruder path:
R(0)= Σ
0:+1 0
0. 0:-1
(110
0:112
+ +10 0, 01-11l2 1112 ))
(7)
10
0.112
110. - 0 0.-1
2
where 0 E R is the i-th row of O.
5.2 Evaluation metric
The most intuitive way to evaluate the quality of extruder path 0 is to measure the distance between g. the desired fiber path, and the fiber path get by printing O. Note that it is likely the model under-estimates or over-estimate the deformation of fiber introduced by printing, so both cases can out of fiber before finish 0, there is still some
happen when we print with path θ: fiber remaining in the nozzle after we finish 0 (the total length of fiber is fixed given a desired fiber path g). In other words, the g:'s and u, 's might not be synchronized. Thus, directly measuring the distance between g: and u. does not necessarily reflect the difference between desired fiber path and the fiber path we get. Therefore, better measure the distance between g and during testing, we use Chamfer distance (26), which was first proposed by Barrow et al. [3] as an image matching technique, later developed as a commonly used (semi) measure the difference between two sets, and has been shown to have higher correlation with human judgment compared to intersection over union and earth mover's 's distance [86]: min lgg: u, 2112 + n (E(1.n)
deo(g, u)
Σ min |lg, uzll2)
2 n
JE(l,n]
5.3 Implementation
The pipeline is shown in Figure To generate the dataset for calibrating the decoder, we first use elliptical slice sampling (63) (New BSD License) to sample random extruder paths from a Gaussian process. We then use physical simulator built using Bullet (22) (zlib License), calibrated a real printer, to predict the realization (fiber path) for each extruder path. We generate 10,000 paths, split into 90% training, 5% validation, and 5% testing. For decoder, encoder, and direct-learning we use an MLP with 5 hidden layers and ReLU as the activation function. The MLP takes 61 points as input and produces point as output, and is applied
"sliding window" fashion over the entire path (details in appendix). We train every model with a learning rate of 1 x 10-3 for 10 epochs using PyTorch 68 and Adam optimizer (54). For direct-optimin mationn, we use the BFGS implementation in SciPy [92] More implementation details are included in the appendix.

---

Table 1: Path-planning evaluation of the average Chamfer distance on the set evaluated in simulation
Regularizer weight
0.1
0.3
0.6
1.0
1.5
direct-learning Ours
0.314+0.0008 0.0319=0.0000 0.0502==0.007 0. 0007 0.1457=0.0216 0.0180=000004 0.0157=0.0003 0.0164=0.0004 0.0158=0.0002
0.0156=0.0002
direct-optimization
0.0155:10.0002
Target fiber path
Extruder path
Simulated fiber path u
Dirrct learning
Ours
Direct leaminng
Ours
Distance from simulated puth desirod path
Distance from desined paih tu simmilated puth
on
Sample 1
Sample 2
Figure 3: Path-planning evaluation of sirect-learning vs. ours on the test set evaluated simulation. We also visualize the Chamfer distance for each point on both the simulated fiber path and the desired fiber path.
5.4 Experiments
Fiber path quality evaluated in simulation. To quantitatively evaluate the effectiveness of our ap- proach, the most straightforward way is to run its prediction the simulator and see how close the simulated fiber path is to the input fiber path. We compare the performance of approach (encoder), direct-learnimg and spreectopting imi canion on the test set of 500 paths, for different values of the reg- ularization parameter X. We report the average Cham-
Table 2: Path- suth-planning evaluation of the average running time on the first 50 samples in the test set
Avg. time (s)
direct-learning 7.96x 10
Ours 7.96x 10
direct-optimization
1.17x10
fer distance ($ 5.22 with the standard error among 3 runs in Table Note that directtpptimizat runs very slowly. so we instead tune its regularizer weight on the first 40 test samples, select the best regularizer weight, and report its performance on the whole test set using the selected regular- izer weight (more details in the appendix). The results demonstrate that our method significantly outperforms direct- lrrect-learnnng, and the performance is comparable to direct-optimgg cetti-pppiiiiiieee inntimiiitionn
As qualitative evaluation, Figure shows the predictions of both direct-learning and our method on samples from the test set. We select the run with minimum average Chamfer distance over the test set for both direct- better than direct-learnie distance from each individual point on one path to the other, and observe that the distances for our method are generally lower than for direct-learning
and our method, respectively. The results indicate that our method is on handling details in the fiber path. We also visualize the Chamfer
Running time comparison. To train needed neural networks, direct sotectttppiiizaiiio takes roughly 10 minutes, direct-learning takes roughly hour, and our method takes roughly days. Note that training costs are amortized, since we only need to train once. We then evaluate the inference time of the three algorithms on server with two Intel(R) Xeon(R) ES-2699 v3 CPUs running 2. Since small neural networks generally run faster the CPU, run all of the tests solely on CPU. We run every algorithm on the first 50 paths in the test set and report the average inference time in T As can see, both and di directleeanni below millisecond, while cerrecttottmzzzzee
achieve a running time runs orders of magnitude slower.
Fiber path quality evaluated on real printer. Lastly, real Markforged Mark Two 3D printer. We set the desired fiber path to a star, and print the star itself
our extruder path solutions on
7

---

Without planning
Direct leuming
Qurs
Run
Figure 4: Path-planning evaluation of "without planning" vs. direct-learning ours Markforged Mark Two, with star as the desired fiber. For "without planning", we have the same extruder path for the 2 runs; for direct-learning and ours,
are predictions from neural networks trained with different initializations.
(without planning) as well as solutions from both direct-learning and our method. We select regularizer weights based on [ ie.,, 0.1 for direct-learning 1.5 for our method, and we show two of the trained models. The results are visualized in Figure confirming that our method successfully improves the quality of the printed fiber.
6 Case study: constrained soft robot inverse kinematics
Partial differential equations (PDEs) powerful tool for describing complex relationships between variables and have been used widely in areas including physics, engineering, and finance. PDE- constrained optimization [8] the goal is optimize a cost function such that the constraints can be written as PDEs, i.e., the solutions are consistent with the relationships specified by the PDE. In the context of synthesis problems, we can consider the boundary conditions of the PDE as the design, and the solution of the PDE as the realization. Similar to other synthesis problems, different boundary conditions can result in the same PDE solution, and the cost function may not have unique minimum. In the above situations, we propose to apply our method. We test our method on specific PDE-constrained optimization problem- constrained soft robot inverse kinematics, which serves as representative use case of our method in this large category of problems.
Soft robots made of clastic materials, have received significant recent attention because of their reduced potential harm when working with humans 74). Researchers have explored a variety of applications, including surgical assistance [99. bio-mimicry [6]], and human-robot interaction [67] In this case study, in Xue et al. (97), we use snake-like soft robot with fixed bottom, in which we can control the stretch ratios on both sides of the robot. As shown
Tigure[i. the objective is for the midpoint at the top of the robot reach target, while making sure that the robot does not collide with fixed-size circular obstacle. The relationship between the robot's 's shape and the stretch ratios can be written PDE, and shown in Figure 11, there are different solutions to achieve the goal.
6.1 Cost Function
We adopt the soft robot from Xue et al. 997, which has an original height of 10 and an original width of 0.5, with its bottom is fixed. The goal vector g in 2 where gi E IR2 indicates the target location and 92 E I22 indicates the obstacle location. We denote the radius of the obstacle as which we set to 0.9. The design (control) vector is RD with n 40, with 0, E R indicating the stretch ratio of the i-th segment (e.g., 0 = 0.95 indicates the i-th segment is contracted by 5%). in gmxx with m 103, where u, E 1 indicates the location
The physical realization vector of the i-th vertex the soft robot's 's mesh. The location of the top midpoint is denoted as U R². Implicitly, the relationship between and u obeys a PDE that governs the deformation of the robot, as detailed in Xue et al. [97].
The cost function L. (:.))) defined as
(9) The first term ||g| utnll is the squared Euclidean distance between the top midpoint of the robot and the target. The second term, weighted by a hyper-parameter Àι (which we fix 0.5), enforces
Cg (0, u) = 2
1 |
u
Y1 Blu, + d2 R(0).

---

Table 3: Soft-robot evaluation of the number of successful cases (over 1,000) on test
Regularizer weight
0.03
0.05
0.07
0.09
direct-learning
907.743.1
918.3=3.4
910.7=3.4
912.3+3.8
Ours
9..
975.0=3.9
981.7=5.0
984.7-4.55
direct direct-optiiizzttio
997.0
998.0=0.0 998.3=0.7 097.7
Table Soft-robot evaluation of the average distance to the target on successful cases on test set
Regularizer weight
0.03
0.05
0.07
0.09
direct-learning
0.2171=:000016 0.2200=t0.0028 0.0657=000093
0.2236++0.0006 0.2179=:0.0336
Ours
0.0464+0.0018 0.059 0.0691=0.0224
direct-optimlzation
0.0233-0.0003
0.0240#0.0004
0.0241#0.0004 0.0242=000003
constraint via a barrier function 665 66
the obstacle Biu. g2):
Biu, 92) =
Σ
max(r + Ar l|: 92l12, 0))²
(10)
where Ar hyper-parameter (which we fix at 0.1). positive Ar provides penalty as well as nonzero gradients when the robot gets close to the obstacle. The last term contains another hyper-parameter X2, varied in our experiments, weighting a smooth regularization term R(0), with
R(O)=
Σ
0 - 0
0
01-1
(11)
2
2
ffn/2+1
where 0, for 1,2, ,n/2 corresponds to stretch ratios the left-hand side of the robot, and 0, for i = n/2 + 1,.... ,n corresponds to stretch ratios on the right-hand side of the robot. This regularizer prevents unphysical deformations with strong discontinuities.
6.2 Evaluation metric
Since there are two "rreathves=""racch"" and "avoid" this task, we have two evaluation metrics. The first metric the number of cases that successfully avoid the obstacle (i.e., have all vertex positions outside the obstacle circle). The second metric is the average Euclidean distance of the robot's top midpoint to the target for successful cases.
6.3 Implementation
Using the finite element method 44 and the code from Xue et al. 197 (MIT license). randomly generate 40,000 data samples, and split them into 90% training. 7.3% validation, 2.5% % testing. For encoder, decoder, and di direet-eean reect-lerrning, We train every model for 200 epochs with a learning rate of 1 x 10-3 using PyTorch (68) and Adam optimizer (54). For direct-opt cent-ooptmzzzeee miizin.,, we use the BFGS implementation in SciPy 92 More implementation details are included in the appendix.
MLP with hidden layers and ReLU activation.
6.4 Experiments
Design quality evaluation. We experiment with different regularizer weights X2 for our method and the two baselines. During training, we randomly sample the location of the obstacle, and we ensure the robot never collides with the obstacle for direct-learning, since does not have access to the realization vector and thus its loss function cannot contain the barrier function term for the obstacle (more details about direct-learning learningg are in the appendix). For a fair comparison, during testing, set the random seed to 0 such that for the same test sample. the obstacle will appear at the same location for all algorithms. The number of cases in which the robot successfully avoids the obstacle, with standard error for runs, is shown in Table The average Euclidean distance to the target for successful cases, with its standard error, is shown Table ( As the numbers show, our method is competitive Samples from the test set are shown Figure [3) (for both algorithms, we select the best run with regularizer weight of 0.5). We target more accurately than direct-leantnn
cerrectoppimizatee
and performs much better than direct-learning,
that method collides less frequently while reaching the

---

Tirget
Obsnele
Dircct lcoming
Ours
Direct optimization
Sample
Sample 2
Figure 5: Soft-robot evaluation direct- wreettlearning vs. ours vs. oppecttoptimzaaateee on examples from the test set. The direct learning baseline both violates the constraints (Sample 1) and fails to reach the target (Samples and 2), while the run time of direct-optimization is 4000 times that of our method.
Running time. To train the neural networks, direct-optimization takes roughly hours, direct-learning takes roughly 4 hours, and method takes roughly 4.5 hours. Note that train- ing costs are amortized, since only need train once. We then test all algorithms on a server with two Intel(R) Xeon(R) E5-2699 v3 CPUs running 2.30GHz. Everything runs solely on the CPU, max- imizing efficiency for the small neural networks we use. The results are shown in Table 5 showing that we successfully reduce the running time from over 1 second less than 1 millisecond. Note that the soft robot is relatively small (40 control variables), and the time complexity of BFGS grows quadratically w.r.t. the number of parameters. Therefore, for more complex soft robots or PDE-constrained optimization problems with a larger number of variables, the running time advantage of our method can be of even greater importance.
Table 5: Soft-robot evaluation of the average run- ning time on the test set
Avg. time (s)
direct-learning
3.12x10 10-
Ours
3.34x 10
direec-optimization
1.33
7 Discussion
In this work, provided amortized approach to synthesis problems in machine learning. To tackle the non-differentiabiiiy of physical system realizations, the huge computational cost realization processes, and the con-uniquesess of the design solution, we designed two-stage neural network architecture, where we first learn the decoder, a surrogate that approximates the realization processes, and then learn the encoder, which proposes design for input goal. We tested approach two case studies on fiber extruder path planning and constrained soft robot inverse kinematics, where we demonstrated that our method provides designs with much higher quality than supervised learning of the design problem, while being competitive in quality to and orders magnitude faster than direct optimization of the design solution.
Although the experiments in both case studies show the effectiveness of our approach, we would like to mention some limitations of our method. First, to effectively learn a differentiable surrogate for the realization process, we need to be able to generate a substantial number of viable designs. We also need a simulator calculate physical realizations of them, and the realization process has be deterministic, although extensions might consider probability distributions over realizations. Also, to train the encoder, we need the objective ("goal") to be quantifiable. Our method provides the greatest gains if the realization is computationally expensive and/or non-differentiable, or if our encoder can exploit the non-uniqueness of designs to choose one good option where supervised learning would have learned a poor "average" solution. Additionally, due to the cost of neural network training, amortization is only a good idea when we need to solve one design problem many times with different goals, or need fast inference. From societal point of view, the primary negative consequence is the potential for replacing human labor in design. We view the present approach, however, as part of larger human-int in-the-loop design processes in line with other software tools for modeling and fabrication.
10

---

Acknowledgements
We would like to thank Geoffrey Roeder for helping set up the 3D printer, Amit Bermano, Jimmy Wu, and members of the Princeton Laboratory for Intelligent Probabilistic Systems for valuable discussions and feedback, as well as Markforged. This work is partially supported by the Princeton School of Engineering and Applied Science, as well as the U.S. National Science Foundation under grants #IIS-1815000 and HIIS-2007278.
References
[1] Suleman Asif. Modelling and path planning for additive manufacturing of continuous fiber composites. PhD thesis, 2018.
[2] Han Bao, Clay Scott, and Masashi Sugiyama. Calibrated surrogate losses for adversarially robust classification. In Conference on Learning Theory, pages 408-451 PMLR, 2020.
[3] Harry G Barrow, Jay M Tenenbaum, Robert C Bolles, and Helen Wolf. Parametric cor- respondence and chamfer matching: Two new techniques for image matching. In IJCAI, 1977.
[4] Cenk Baykal and Ron Alterovitz. Asymptotically optimal design of piecewise cylindrical robots using motion planning. In Robotics: Science and Systems, volume 2017, 2017.
[5] Martin Philip Bendsoe and Ole Sigmund. Topology optimization: theory, methods, and applications. Springer Science & Business Media, 2013.
[6] Atharv Bhosekar and Marianthi Ierapetritou. Advances in surrogate based modeling, feasibility analysis, and optimization: A review. Computers & Chemical Engineering, 108:250-267, 2018.
[7] V Bhuvaneswari, M Priyadharshini, C Deepa, D Balaji, L. Rajeshkumar, and M Ramesh. Deep learning for material synthesis and manufacturing systems: a review. Materials Today: Proceedings, 2021.
[8] Lorenz T Biegler, Omar Ghattas, Matthias Heinkenschloss, and Bart van Bloemen Waanders. Large- scale aol-constrai optimization: an introduction. In Large-Scale PDE-Constraiie Optimization, pages 3-13. Springer, 2003.
[9] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pages 773-782. PMLR, 2019.
[10] David H Brookes and Jennifer Listgarten. Design by adaptive sampling. arXiv preprint arXiv:1810.03714 2018.
[11] JA Cabrera, A Simon, and M Prado. Optimal synthesis of mechanisms with genetic algorithms. Mechanism and machine theory, 37110):1165-1177 2002.
[12] JA Cabrera, A Ortiz, F Nadal, and JJ Castillo. An evolutionary algorithm for path synthesis of mechanisms. Mechanism and Machine Theory, 1672):127-141, 2011.
[13] Thais Campos and Hadas Kress-Gazit. Synthesizing modular manipulators for tasks with time, obstacle, and torque constraints. arXiv preprint 11Xi::21060094887 2021.
[14] Thais Campos, Jeevana Priya Inala, Armando Solar-Le7ama, and Hadas Kress-Gazit, Task- based design of ad-hoc modular manipulators. In 2019 International Conference on Roboties and Automation (ICRA), pages 6058-6064. IEEE, 2019.
[15] Thais Campos de Almeida, Samhita Marri, and Hadas Cress-Gazit Automated synthesis of modular manipulators' structure and control for continuous tasks around obstacles. Robotics: Science and Systems 2020, 2020.
[16] Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug discovery today, 23(6):1241-1500 2018.
11

---

[17] I-Ming Chen and Joel W Burdick. Determining task optimal modular robot assembly configu- rations. In proceedings of 1995 IEEE International Conference on Robotics and Automation, volume 1, pages 132-137. IEEE, 1995.
[18] Wan Kyun Chung, Jeongheon Han, Youngil Youm, and SH Kim. Task based design of modular robot manipulator using efficient genetie algorithm. In Proceedings of International Conference on Robotics and Automation, volume 1, pages 507-512, IEFE, 1997.
[19] Matteo Cianchetti, Tommaso Ranzani, Giada Gerboni, Thrishantha Nanayakkara, Kaspar Althoefer, Prokar Dasgupta, and Arianna Menciassi. Soft robotics technologies to address shortcomings today's minimally invasive surgery: the stiff-flop approach. Soft robotics, (2): 12):122-131, 2014.
[20] Connor W Coley, William H Green, and Klavs F Jensen. Machine learning in computer-aided synthesis planning. Accounts of chemical research, 51(5): 2018.
[21] Stelian Coros, Bernhard Thomaszewski, Gioacchino Noris, Shinjiro Sueda, Moira Forberg, Robert W Sumner, Wojciech Matusik, and Bernd Bickel. Computational design of mechanical characters. ACM Transactions Graphics (TOG), 32(4):1-12, 2013.
[22] Erwin Coumans. Bullet physics engine. Open Source Software: http://bulletphysics. org, 2010.
[23] Anubhav Dogra, Srikant Sekhar Padhee, and Ekta Singla. An optimal architectural design for unconventional modular reconfigurable manipulation system. Journal of Mechanical Design, 143(6):063333, 2021.
[24] Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning generate conditionally from unconditional generative models. In International Conference on Learning Representations, 2018.
[25] Michael Everett, Yu Fan Chen, and Jonathan P How. Motion planning among dynamic, decision-making agents with deep reinforcement learning. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3052-3059. IEEE, 2018.
[26] Haoqiang Fan, Hao Su, and Leonidas Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017.
[27] Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. Ad- vances in Neural Information Processing Systems, 33, 2020.
[28) Shane Farritor, Steven Dubowsky, Nathan Rutman, and Jeffrey Cole. A systems-level modular design approach to field robotics. In Proceedings of IEEE International Conference on Robotics and Automation, volume 4, pages 2890-2895. IEEE, 1996.
[29] Alexander IJ Forrester and Andy J Keane. Recent advances in surrogate- based optimization. Progress aerospace sciences, 45(1-3):50-79, 2009.
[30] Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. In International Conference on Learning Representations, 2020.
[31] Nuwan Ganganath, Chi-Tsun Cheng, Kai-Yin Fok, and K Tse Chi. Trajectory planning for 3d printing: A revisit to traveling salesman problem. In 2016 2nd International Conference Control, Automation and Robotics (ICCAR), pages 287-290. IEEE, 2016.
[32] Rafael Gómez-Bombareee Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobate Benjamin Sánchez-Lengeling Dennis Sheberla, Jorge aguilera-Iparraguire Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-276, 2018.
[33] Josif Grabocka, Randolf Scholz, and Lars Schmidt-Thieme, Learning surrogate losses. arXiv preprint arXiv:1905.10100 2019.
12

---

[34]
Yanran Guan, Han Liu, Kun Liu, Kangxue Yin, Ruizhen Hu, Oliver van Kaick, Yan Zhang, Ersin Yumer, Nathan Carr, Radomir Mech, and Hao Zhang, FAME: 3d shape generation via functionality-aware model evolution. IEEE Transactions on Visualization and Computer Graphics, 2020.
[35] Sumit Gulwani. Program synthesis. Software Systems Safery, pages 43-75, 2014.
[36] Anvita Gupta and James Zou. Feedback gan for dna optimizes protein functions. Nature Machine Intelligence, 1()):05-111, 2019.
[37] Sehoon Ha, Stelian Coros, Alexander Alspach, Joohyung Kim, and Katsu Yamane. Task-based limb optimization for legged robots. In 2016 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS) pages 2062-2068. IEEE, 2016.
[38] Sehoon Ha, Stelian Coros, Alexander Alspach, James M Bern, Joohyung Kim, and Katsu Yamane. Computational design of robotic devices from high-level motion specifications. IEEE Transactions on Robotics, 34(5):1 1200-121, 2018.
[39] Zhong-Hur Han, Ke-Shi Zhang, et al. Surrogate-base optimization. Real-world applications of genetic algorithmss, 343, 2012.
[40] Steve Hanneke, Liu Yang, et al. Surrogate losses in passive and active learning. Electronic Journal of Statistics, 13()):464644708 2019.
[41] Alex Hawkins-Hookee Chen, and David Bikard. Generating functional protein variants with variational autoencoders. PLoS computational biology, 17(2)=e0088736, 2021.
Florence Depardieu, Sebastien Baur, Guillaume Couairon, Arthur
[42] Gregory S Hornby, Hod Lipson, and Jordan B Pollack. Evolution of generative design systems for modular physical robots. In Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), volume 4, pages 4 4146-4151, IEEE, 2001.
[43] Jiaqi Huang, Qian Chen, Hao Jiang, Bin Zou, Lei Li, Jikai Liu, and Huangchao Yu. A survey of design methods for material extrusion polymer 3d printing. Virtual and Physical Prototyping. 15(2):148-16) 2020.
[44] Thomas JR Hughes.
The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
[45] Haoyan Huo, Ziqin Rong, Olga Kononova, Wenhao Sun, Tiago Botari, Tanjin He, Vahe Tshitoyan, and Gerbrand Ceder. Semi-supervised machine-learning classification of materials synthesis procedures. npj Computational Materials, 5(1):1-7, 2019.
[46] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, pages 4849-4859. PMLR, 2020.
[47] Wengong Jin, Jonathan M Stokes, Richard T Eastman, Zina Itkin, Alexey V 7akharov, James Collins, Tommi S Jaakkola, and Regina Barzilay. Deep learning identifies synergistic drug combinations for treating covid-19. Proceedings of the National Academy of Sciences, 118 (39), 2021.
[48] Nikos Ath Kallioras, Georgios Kazakis, and Nikos D Lagaros. Accelerated topology opti- mization by means of deep learning. Structural and Multidisciplinary Optimization, 62(3):
1 1185-1212, 2020.
[49] Sertac Karaman and Emilio Frazzoli. Sampling- based algorithms for optimal motion planning. The international journal of robotics research, 30(7):846-88)) 894, 2011.
[50] M Khorshidi, M Soheilypour, M Peyro, A Atai, and M Shariat Panahi. Optimal design of four-bar mechanisms using hybrid multi-objective ga with adaptive local search. Mechanism and Machine Theory, 46( (0(10))144 1453-1455,, 2011.
[51] Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan Frey. Generating and designing dna with deep generative models. arXiv preprint
2017.
13

---

[52] Edward Kim, Kevin Huang, Adam Saunders, Andrew McCallum, Gerbrand Ceder, and Elsa Olivetti. Materials synthesis insights from scientific literature via extraction and machine learning. Chemistry of Materials, 29(21):94369444 2017.
[53] J-O Kim and Pradeep K Khosla. formulation for task based design of robot manipulators. In Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (33,, volume 3. pages 2310-2311 IEEE, 1993.
[54] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[55] Hunter Kollmann, Diab W Abueidda, Seid Koric, Erman Guleryuz, and Nahil Sobh. Deep learning for topology optimization of 2d metamaterials. Materials & Design, 196: 2020.
[56] Slawomir Koziel and Leifur Leifsson. Surrogate-based modeling and optimization. Springer, 2013.
[57] Slawomir Koziel and Stanislav Ogurtsov. Surrogate saarog optimization. In Antenna Design by Iomulattio-Driveen Optimization, pages 13-24. Springer, 2014.
[58] Slawomir Koziel, David Echeverría Ciaurri, and Leifur Leifsson. Surrogate-based methods. In Computational optimization, methods and algorithms, pages 33-59. Springer, 2011.
[59] CannCCladde Latombe. Robot motion planning, volume 124. Springer Science Business Media, 2012.
[60] Chris Leger and John Bares. Automated task-based synthesis and optimization of field robots. 1999,
[61] Guorui Li, Xiangping Chen, Fanghao Zhou, Yiming Liang, Youhua Xiao, Xunuo Cao, Zhen Zhang, Mingqi Zhang, Baosheng Wu, Shunyu Yin, et al. Self-powered soft robot in the mariana trench. Nature, 6767784)) 2021.
[62] Lanlan Liu, Mingzhe Wang, and Jia Deng. A unified framework of surrogate loss by refactoring and interpolation. In ECCV, 2020.
[63] lain Murray, Ryan Adams, and David MacKay. Elliptical slice sampling. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 541-548. JMLR Workshop and Conference Proceedings, 2010.
[64] Gattigorla Nagendar, Digvijay Singh, Vineeth N Balasubramanian, and CV Jawahar. Neuro- iou: Learning a surrogate loss for semantic segmentation. In BMVC, page 278, 2018.
[65] Yurii Nesterov et al. Lectures convex optimization, volume 137. Springer, 2018.
[66] Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science Business Media, 2006.
[67] Gaoyang Pang, Geng Yang, Wenzheng Heng, Zhiqiu Ye, Xiaoyan Huang, Hua- Hua-Yong Yang, and Zhibo Pang. Coboskin: Soft robot skin with variable stiffness for safer human-robe collaboration. IEEE Transactions on Industrial Electronics, 68(4)3330333344 2020.
[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-perffrmance deep learning library. In Advances in neural information processing systems, pages 8026-8037, 2019.
[69] Sarosh Patel and Tarck Sobh. Task based synthesis of serial manipulators. Journal of advanced research, 6(3):479 2015.
[70] Yash Patel, TomáX Hodañ, and Jif Matas. Learning surrogates via deep embedding. In European Conference Computer Vision, pages 205-221. Springer, 2020.
14

---

[71] Daniil Polykovskiy. Alexander Zhebrak, Dmitry Vetrov, Yan Ivanenkov, Vladimir Aladinskiy, Polina Mamoshina, Marine Bozdaganyan, Alexander Aliper, Alex Zhavoronkov, and Artur Kadurin. Entangled conditional adversarial autoencoder for de novo drug discovery. Molecular pharmaceutics, 15(10):4398-4400 2018.
[72] Alex Renda, Yishen Chen, Charith Mendis, and Michael Carbin. Difftune: Optimizing cpu simulator parameters with learned differentiable surrogates. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 442-455. IEEE, 2020.
[73] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
[74] Daniela Rus and Michael T Tolley. Design, fabrication and control of soft robots. Nature, 521 (7553):467-475, 2015.
[75] Hidenori Sasaki and Hajime Igarashi. Topology optimization accelerated by deep learning. IEEE Trunsactions Magnetics, 55(6):1-5, 2019.
[76] Ari Seff, Wenda Zhou, Farhan Damani, Abigail Doyle, and Ryan P Adams. Discrete ob- ject generation with reversible inductive construction. In Advances in Neural Information Processing Systems, volume 32, pages 10353-10363, 2019.
[77] Nurhalida Shahrubudin, Te Chuan Lee, and Rhaizan Ramlan. An overview on 3d printing technology: technological, materials, and applications. Procedia Manufacturing, 35: 32:1286- 1296, 2019.
[78] Aniruddha V Shembekar, Yeo Jung Yoon, Alec Kanyuck, and Satyandra K Gupta, Trajectory planning for conformal 3d printing using non-planar layers. In International Design Engi- neering Technical Conferences and Computers and Information in Engineering Conference, volume 51722, page ATO2A020 American Society of Mechanical Engineers, 2018
[79] Sergey Shirobokov, Vladislav Belavin, Michael Kagan, Andrei Ustyuzhanin, and Atilim Gunes Baydin. Black- optimization with local generative surrogates. In Workshop Real World Experiment Design and Active Learning at International Conference on Machine Learning. 2020.
[80] Ole Sigmund and Kurt Maute. Topology optimization approaches. Structural and Multidisci- plinary Optimization, (0(6))0031-000 2013.
[81] Satinder P Singh, Andrew G Barto, Roderic Grupen, Christopher Connolly, et al. Robust reinforcement learning in motion planning. Advances in neural information processing systems, pages 655-655, 1994.
[82] Suwin Sleesongsom and Sujin Bureerat. Four-bar linkage path generation through self- adaptive population size maachnngleearing based optimization. Knowledge-Baud Based Systems, 135: 180- 191, 2017.
[83] Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell, 180(4):688-700 2020.
[84] Enrico Stragiotti. Continuous Fiber Path Planning Algorithm for 31) Printed Optimal Mechan- ical Properties. PhD thesis, Politecnico di Torino, 2020.
[85] Devika Subramanian et al. Kinematic synthesis with configuration spaces. Research in Engineering Design, 7(3):193-213, 1995.
[86] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2018.
15

---

[87] Saleh Tabandeh, William Melek, Mohammad Biglarbegian, Seong-hoon Peter Won, and Chris Clark. A memetic algorithm approach for solving the task-based configuration optimization problem in serial modular and reconfigurable robots. Robotica, 1449))1179-20000 2016.
[88] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenenbaum, and Jijjun Wu. Learning to infer and execute 3d shape programs. In International Conference on Learning Representations, 2018.
[89] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pages 10358-10368. PMLR, 2021.
[90] Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee. Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learning in drug discovery and development. Nature Reviews Drug Discovery, 18(6): 463-477, 2019.
[91] EJ Van Henten, DA Van't Slot, CWJ Hol, and LG Van Willigenburg. Optimal manipulator design for a cucumber harvesting robot. Computers and electronics in agriculture, 65(2): 247-257,
[92] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-211 2020. doi: 10.
[93] Michael Yu Wang, Xiaoming Wang, and Dongming Guo. A level set method for structural topology optimization. Computer methods applied mechanics and engineering, 192(1-2): 227-246, 2003.
[94] Julian Whitman and Howie Choset. Task-specific manipulator design and trajectory synthesis. IEEE Robotics and Automation Letters, 4(2):301-300 -0000,,11,,--00 2018.
[95] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenenbaum. MarNNet: 3D Shape Reconstruction via 2.5D Sketches. In NIPS, 2017.
[96] Hong Xiao, Wei Han, Wenbin Tang, and Yugang Duan. An efficient and adaptable path planning algorithm for automated fiber placement based on meshing and multi guidelines. Materials, 13(8))2209, 2020.
[97] Tianju Xue, Alex Beatson, Sigrid Adriaenssens, and Ryan Adams. Amortized finite element analysis for fast pde-constrained optimization. In International Conference on Machine Learning, pages 10638-10647. PMIR, 2020.
[98] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenen- baum. Neural-symbolt vqa: disentangling reasoning from vision and language understanding. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 1039-10500 2018
[99] Yonggyun Yu, Taeil Hur, and Jaeho Jung. Deep learning for topology optimization design. arXiv preprint 1801 05463, 2018.
[100] Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Robust deep auc maximization:
A new surrogate loss and empirical studies on medical image classification. arXiv preprint arXiv:2012.031733 2020.
[101] Lifeng Zhu, Weiwei Xu, John Snyder, Yang Liu, Guoping Wang. and Baining Guo. Motion- guided mechanical toy modeling. ACM Transactions on Graphics (TOG), 31(6):1-10, 2012.
16


Reviews:
[{"review_id": "tPEXfaKVsVG", "reviewer": "Reviewer_P3xV", "summary": "The paper addresses the challenge of synthesis - where the goal is to find a configuration of a physical system that satisfies a set of constraints and optimizes one or more objective functions representing the desiderata. The synthesis challenge has three components - slow and costly realization process for evaluation of each design configuration, the physical realization or its software simulation being usually non-differentiable, and many-to-one mapping from the space of design configurations to the specification (constraints + objective function). Differentiable surrogates are used to address the first two challenges (to the extent possible for a realization process). The third challenge makes supervised learning difficult because the many-to-one mapping from configuration to realization to goal can penalize learned for good configurations/designs just because these were different from the training set. Adding more alternatives in the training set would also not help because the supervised learner would learn to predict average over the configuration space, which would semantically be not meaningful. \n\nThe solution proposed in this paper is similar to some other recent work in this space which use autoencoders - a decoder captures many-to-one mapping from design configuration to design goals, and another network that maps goal to a design.  As described in the limitations section, the paper's claim to novelty is not fully justified by their choice of benchmarks and baselines, and limited discussion of related work. Further, the reviewer is not convinced (for reasons described in the limitation section) that the presented approach would be practical even for physical design problems (ignoring the larger synthesis literature). ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper addresses a very important challenge of automatically finding configurations that meet design constraints and are optimum with respect to a given set of objectives. \n\n+ The approach is demonstrated on a path planning (for a class of 3D printers) problem and a snake-like soft robotics inverse kinematics problem. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The baselines (direct-optimization, direct-learning) selected for comparison are rather crude and simplistic, and do not reflect the state-of-the-art in this area. The approach presented in this paper is described more generically as a general synthesis solution (and reviewer believes this to be the case too) and similar generic synthesis approaches have been developed with focus on slightly different application (proteins). For example, see Polykovskiy, Daniil, et al. \"Entangled conditional adversarial autoencoder for de novo drug discovery.\" Molecular pharmaceutics 15.10 (2018): 4398-4405, Hawkins-Hooker, Alex, et al. \"Generating functional protein variants with variational autoencoders.\" PLoS computational biology 17.2 (2021): e1008736. The reviewer is of the opinion that either the paper can be rewritten as one focussed on planning/robotics (with an explanation about why this domain is different from other synthesis problems) and the presented solution can be described as uniquely better suited to this domain, or synthesis techniques from chemistry/biochemistry need to be considered as baselines and the presented approach compared on some case-studies on that literature. These examples will also be more practically relevant challenges than the simple examples considered in the paper. In its present form, the paper's claim to novelty is not matched with its empirical analysis.\n\n- More realistic benchmark problems in the robotics domain for evaluating the design approach would be to pick examples from state-of-the-art robotics papers on design: Campos, Thais, Samhita Marri, and Hadas Kress-Gazit. \"Automated Synthesis of Modular Manipulators’ Structure and Control for Continuous Tasks around Obstacles.\" Robotics Science and Systems. 2020, Campos, Thais, et al. \"Task-based design of ad-hoc modular manipulators.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n\n-  The paper is also lacking discussion of related work to identify its incremental novelty over approaches developed there. Some relevant papers from ML venues include: Brookes et. al. \"Conditioning by adaptive sampling for robust design.\" International conference on machine learning. PMLR, 2019. Fannjiang et. al. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 33.\n\n- Leaving aside the experimental evaluation and the core idea of using an autoencoder to go from goals to design space and back, the method proposed in the paper is rather straightforward and lacks any theoretical analysis of complexity or guarantees on the produced design. While a convincing experimental analysis to demonstrate practical utility would have been itself sufficient, the paper doesn't appear to have either major theoretical result or practical insight that could improve the state-of-the-art in synthesis. \n\n- In real world, if we randomly sample designs, a vast majority would be actually invalid. Imagine putting together battery, speed control, propeller and motors to create a drone. In many cases, these combinations would produce current enough to burn the motor or torque high enough to tear the propeller. So, how do authors suggest building a dataset described in line 128-135. One reason why synthesis papers focus on biochemistry is this challenge of creating a dataset with physical designs. Path planning and kinematics are niche problems where each configuration (and configuration space is defined in a way to exclude impossible positions) leads to some measurable performance. These examples are not reflective of typical physical design problems.\n\n- The paper adopts a rather simplistic approach to combined multiple objectives, which are common in design problems. The path planning example has \"reach\" and \"avoid\" as objectives, which are being weighed using hyperparameters. Typical physical design problems have several (10-100s) of optimization criteria and constraints, for e.g. a vehicle design might need one to look at track accuracy, thermal characteristics of motor, speed, etc. Combining them with hyperparameters would not be practically feasible (discovery of good values of these hyperparameters would be itself a hard problem). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qLCcIKIiquj", "reviewer": "Reviewer_PZD7", "summary": "The core concern of this paper is the synthesis problem of choosing design parameters in a complex engineering design process mapping a high level goal to a high dim description of how the process achieves it. This is studied using the examples of 3D printing path planning and trajectory generation for a soft robotic arm. \n\nThe core contribution is to set up this problem in terms of two stages - mapping goals to a space of designs, and then an evaluation of designs in terms of a cost function. This breaks down one of the difficulties identified - the ill-posedness of the problem due to multi-valued maps. So, the authors set up an autoencoder architecture with an encoder that conjectures designs based on goals, and a surrogate model to evaluate the designs. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a clearly written paper on a timely topic.\n\nThe core contribution is the formulation factoring the evaluation of the design from the mapping of goals to designs, which breaks down the problem of many-valued mapping. Both parts are defined in terms of a differentiable module, so that the training can be achieved jointly. The evaluation methodology addresses what I would have expected to see - setting up baselines to show the extent to which each piece is relevant. So, one baseline asks how much is achieved by just mapping from goals to designs from historical data, while another asks how much is achieved by optimizing designs given the trained surrogate or decoder. \n\nI have only a few points of critique that I would have liked to have seen expanded in this paper. This is based on the acknowledgement that the technically arguments seem essentially sound to me, so my comments are mainly on formulation.\n\n(1) Building on the notion of multiple ways for a design to be good, many engineering designs have path-dependent constraints. For instance, while moving fast there may be a different cost to lateral deviations than when moving slowly. This is a combination of dynamics (something not considered explicitly in the examples) and other forms of heterscedasticity or path dependence. To the extent that the surrogate and the decoder are fairly simple MLPs, it is unclear if such phenomena can be handled. I think this is an important consideration, especially if the use cases include robotics and cyber-physical domains involving control.\n\n(2) While the introduction talks about considerations beyond the basic path, e.g., \"ease of manufacturing\" mentioned in sec 3, they do not feature later on. So, for instance, the soft body reaching problem could have been solved to some extent by multi-rigid body assumptions without seriously needing to acknowledge the infinite-dimensional space. On the other hand, there are design problems involving fundamentally different designs with discrete and continuous choices, e.g., multiple body morphologies involving soft components each (just for argument's sake). It is not clear if the proposed method would apply as described in these settings - calling for a clarification of generality and scope. \n\n(3) In the same vein, the design objectives here do admit a clear quantitative description always, e.g., distance from a desired path and reach error. There are many design problems where the objective is qualitative (e.g., any design from a topologically defined class will suffice). I expect that this might be easier to encode in this architecture but an example of this kind would have been helpful.\n\nIn summary, this is a clearly written paper whose methodology is sound. It would be stronger if the formulation's assumptions and limits are more clearly delineated.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper is a methods contribution which is likely to be used in systems such as 3D printers and robots. I believe the authors have taken due care in setting up and disclosing their software use. ", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "LWmztfgGSsf", "reviewer": "Reviewer_GztB", "summary": "This paper considers two concrete problems in fabrication and design: path planning in 3D printing and soft robotic control. In both cases, there is trouble resulting from solutions that are not unique, in the sense that the inverse image from the codomain (solution space) is not injective. The paper proposes using a neural network architecture for feature space learning, which helps to collapse dimensionality in a way that allows the recovery of more robust (i.e. not the mean) solutions.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and addresses an interesting problem. The authors showed two concrete design and fabrication problems where the use of surrogate objectives is useful for overcoming lack of uniqueness in synthesis problems. The results presented in table 3 seemed to convincingly show that the author's method outperforms naive baselines on a soft robotics problem. \n\nThe soft robotics problem, in particular, was a good setting. And I quiet enjoyed reading about the difficulties that arise in PDEs with non-unique solutions at the boundary. This is a very real world problem with an interesting machine learning solution, and I think it is exactly the type of work we should encourage at this venue. \n\nOverall, the encoder-decoder architecture, such as it is, does not introduce anything new or interesting to the field. Indeed, the authors never make such claims. There are many alternative ways that the authors could go about this process, including adapting GAN-based and VAE-based approaches. Did the authors try any other approaches for their encoder-decoder architecture? Can you include a table with the dataset size and network parameters (optimizer, number of layers, etc)? I am a little confused about why the network took so long to train. \n\nThe data generation process also leaves something to be desired. Generating synthetic test data seems highly problem dependent. And the experiments in soft robotics in particular left me quite worried that the authors are just showing that a neural net can fit to a piecewise linear model. Or a simple piecewise convex model. With synthetic data, you always have to be quite careful that you're not simply investigating the ability of neural networks to approximate linear functions, etc. Although if Figure 4 is to be believed, it does seem like this works quite well in the real world. \n\n\nThe author's worked on interesting problems and developed a machine learning solution that significantly improves the final outcome. Surely, that's worth acknowledging. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the societal impact of automation of labor. As they point out, this type of algorithm is more of an IA proposition. There is no serious concern. \n\nI would've liked to see further comparisons to different types of network architectures, even if only in an appendix. I'm guessing the network architecture is not very optimized. Since there was no benchmark presented where the author's method fails or even does somewhat poorly, it's hard to know where failure would occur. I always like to see more stress testing. Are neural networks even needed? What if we replace the encoder with a linear encoder? How well would that do? I'd just like to see the author's dig deeper into the model so that I can better understand why it works and where I can expect it to fail. \n\nFinally, I should point out that fabrication is not my speciality. And I may have missed something. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "8zQ669zPfG", "reviewer": "Reviewer_tJBF", "summary": "The paper proposes a novel method for addressing synthesis problems, where they define a synthesis as an object or configuration that meets a set of constraints while optimizing one or more objectives. The authors motivate the problem with manufacturing settings, such as an additive printing path and extruding sequence, as well as soft robotic motion problem that can be described by a partial differential equation (PDE). The authors aims to address a couple of challenges:\n* Provide a differentiable surrogate for non-differentiable simulations in synthesis problem. The claim and show that this can help with learning a differentiable model of the process that can help learn how to generate viable design via a synthesis process.\n* Propose and develop a method that can solve many-to-one synthesis problems using a two-stage where one neural network based model learns how to generate designs and another model learns how to evaluate designs. Since both models are differentiable, the authors can train their framework end-to-end, which they claim provides accuracy improvements compared with directly learning a design to evaluation map and compute improvement compared to direct optimization of design to evaluation using method like BFGS.\n* Provide a series of objective functions that can evaluate various synthesis design for a diverse set of use cases, which the authors then use to inform their two-stage approach.\n\nThe authors outline the objective function for an extruder path planning case and a constrained soft robotic motion task. In both cases they optimize the objective and perform the synthesis task using their approach as well as the direct learning (design -> evaluation) and direct optimization (BFGS) baseline. The authors claim, and the experimental supports, that the proposed two-stage method has a performance advantage of direct learning and a compute advantage, in inference, over direct optimization. The authors also discuss how even the training cost for their method is higher compared to direct optimization, the training cost gets amortized in inference as the two-stage does not require re-training/re-running like direct optimization.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Nits:\n* Checklist should start on a separate page after the references\n\nOverall, I think the paper outlines a novel approach to a significant problem, provides thorough details on the method and relevant experiments, and has strong clarity in the writing and presentation. Please see the subsequent sections on suggestions on how to further improve the paper. \n\n**Originality:** The paper tackles a new and relevant problem with a novel deep learning based method that involves end-to-end training of a differential surrogate model and a design generator. I think the approach is novel approach to synthesis problems the authors describe, which are a new set of challenges for deep learning methods.\n\n**Quality:** The authors provide mostly thorough details on their method, formulation and experimental test cases. In order to improve the score, I would to get more clarity on the following questions:\n* How does the \"sliding window\" work to generate an extruder path? Since the is sequential and time dependent, does the \"encoder\" produce the full directly or does it produce it sequentially? If is sequentially generated, how does the input change over time?\n* What are the size of the design spaces for both cases? This would help get a sense of the scale of problems the paper tackled. It would also be useful to have a discussion on how the current approach could scale to larger and more complex challenges. \n* Did you do an ablation of different loss functions? You discuss the importance of the Chamfer distance for the extrusion case, which is helpful, and I would be interested to learn more about how potentially naive loss function affect performance. I assume they would make things worse, but it would be helpful to get more detail on that. \n* A discussion on future work/directions of the work would be nice.\n\n**Clarity:** Overall, the paper is well written and presents relevant equations, formulations and diagrams to understand most of the relevant elements of the study. As mentioned above, to further improve on the clarify it would be helpful to get more detail on the input & output setups and relevant dimensions of the design cases, especially the outputs. There are more details on the inputs in the appendix, but the outputs are still somewhat unclear.\n\n**Significance:** The paper addresses the application of a novel deep learning method to a non-traditional challenges that is significant. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly address limitations in the last section of the paper. As I mentioned in the previous section, it would be good to get a sense of scale of problems that authors think their method would be able to tackle. While the presented problems provide a good basis for proof-of-concept, what challenges do you anticipate in moving to larger and more challenging synthesis problems?\nFor the societal impact perspective, it would be good to include a discussion on the safety and reliability of methods like the ones the authors proposed would affect the over quality and process of synthesis processes, particularly in the manufacturing challenges discussed. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to NeurIPS.  Even after discussion, there is some substantial disagreement on this paper, so I will need to express an opinion that is not a unanimous one.  But with this caveat, I will state that I ultimately come down very much on the "positive" side of the disagreement, and I am recommending the paper be accepted to NeurIPS as a spotlight.

The rationale for this is rather simple: although currently confined to rather simple domains, the paper makes a compelling case that introducing a differentiable surrogate into synthesis problems, in a disciplined and generic fashion, can provide substantial gains over direct learning on the system of interest.  The authors do an extremely good job presenting detailed (and real, even if they are simple) evaluations of this strategy.  Thus, in total the authors present a compelling (_in_ its simplicity, rather than this being a flaw) approach to constrained synthesis, and show that across multiple realistic domains it outperforms the most common baseline.  Those are all points of a strong paper.

Now let me address the negative reviewer's concerns, which in my mind boils down to the concern that 1) the domains evaluated here are not particularly challenging and 2) don't reflect the vast amount of work that is already done in incorporating neural networks into synthesis problems.   I think it would of course be good to mention some of these connections (in the discussion, not in direct comparison) in the final draft of the paper.  But it is also the case that the landscape of integrating neural networks into specific physical synthesis domains is _vast_, and attempting to consider the scope in which all these approaches have been applied previously wouldn't be feasible.  Yes, for any given domain there are likely multiple NN-based solutions to synthesis problems, but attempting to consider all of these, in my opinion, would make anything but extremely domain-specific papers impossible.  The current paper makes no promises of extending the state of the art in a given practical domain (say the chemistry domains mentioned by the reviewer), but does offer evidence of a generic and compelling approach, and I believe these should be welcomed even if they are not (yet) as in-depth within a given domain as would be ideal in the longer term.  I think this is likely to be a fundamental disagreement as to the "right" focuses of papers like these, so while I don't expect to resolve this dilemma (and I certainly understand the reviewer's opinion), I still lean strongly towards accepting the paper.
Decision: Accept (Spotlight)