Closed-Loop Control of Additive Manufacturing via Reinforcement Learning
Authors: Michal Piovarci
Michael Foshey
Timothy Erps
Jie Xu
Vahid Babaei
Piotr Didyk
Wojciech Matusik
Szymon Rusinkiewicz
Bernd Bickel
Venue: ICLR 2022 Submitted (2022)
Keywords: additive manufacturing
closed-loop
reinforcement learning
in-process

Abstract: Additive manufacturing suffers from imperfections in hardware control and material consistency. As a result, the deposition of a large range of materials requires on-the-fly adjustment of process parameters. Unfortunately, learning the in-process control is challenging. The deposition parameters are complex and highly coupled, artifacts occur after long time horizons, available simulators lack predictive power, and learning on hardware is intractable. In this work, we demonstrate the feasibility of learning a closed-loop control policy for additive manufacturing. To achieve this goal, we assume that the perception of a deposition device is limited and can capture the process only qualitatively. We leverage this assumption to formulate an efficient numerical model that explicitly includes printing imperfections. We further show that in combination with reinforcement learning, our model can be used to discover control policies that outperform state-of-the-art controllers. Furthermore, the recovered policies have a minimal sim-to-real gap. We showcase this by implementing a first-of-its-kind self-correcting printer.

Under review as a conference paper at ICLR 2022
CLOSED-LOOP CONTROL OF ADDITIVE MANUFAC- TURING VIA REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Additive manufacturing suffers from imperfections in hardware control and ma- terial consistency. As a result, the deposition of wide range of materials re- quires on-the-fly adjustment of process parameters. Unfortunately, learning the in- process control challenging. The deposition parameters are complex and highly coupled, artifacts occur after long time horizons, available simulators lack predic- tive power, and learning on hardware is intractable. In this work, we demonstrate the feasibility of learning closed-loop control policy for additive manufacturing. To achieve this goal, we assume that the perception of deposition device lim- ited and can capture the process only qualitatively. We leverage this assumption to formulate an efficient numerical model that explicitly includes printing imper- fections. We further show that in combination with reinforcement learning, our model can be used to discover control policies that outperform state-of-the-an controllers. Furthermore, the recovered policies have a minimal sim-to-real gap. We showcase this by implementing a single-layer self-correcting printer.
INTRODUCTION
A critical component of manufacturing is identifying process parameters that consistently produce, high-quality structures. In commercial devices, this typically achieved by expensive trial-and-error experimentation (Gao et al., (015). To make such an optimization feasible, a critical assumption is made: there exists set of parameters for which the relationship between process parameters and process outcome is predictable. However, such assumption does not hold in practice because all manufacturing processes are stochastic nature. Specifically in additive manufacturing, variability in both materials and intrinsic process parameters can cause geometric errors leading to imprecision that can compromise the functional properties of the final prints. Therefore, transition to closed-loop control is indispensable for industrial adoption of additive manufacturing (Wang et al. 2020).
Recently, we have seen promising progress in learning policies for interaction with amorphous ma- terials (Li al. 2019b] Zhang et al. 2020). Unfortunately, in the context of additive manufacturing, discovering effective control strategies is significantly more challenging. The deposition parameters have a non-linear coupling to the dynamic material properties. To assess the severity of deposition errors, we need to observe the material over long time horizons. Available simulators either lack pre- dictive power (Mozaffar et al., (2018) are too complex for learning (Tang et al. 2018 Yan al.) 2018). Moreover, learning on hardware is intractable as we require tens of thousands of printed samples. These challenges are further exaggerated by the limited perception of printing hardware, where typically, only a small in-situ view is available to assess the deposition quality.
In this work, we propose the first closed-loop controller for additive manufacturing based on rein- forcement learning deployed on real hardware. To achieve this we formulate a custom numerical model of the deposition process. Motivated by the limited hardware perception we make a key assumption: to learn closed-loop control it is sufficient to model the deposition only qualitatively. This allows us to replace physically accurate but prohibitively slow simulations with efficient ap- proximations. To ameliorate the sim-to-real gap, we enhance the simulation with a data-driven noise distribution on the spread of the deposited material. We further show that careful selection of input and action space is necessary for hardware transfer. Lastly, we leverage the privileged information about the deposition process to formulate a reward function that encourages policies that account for material changes over long horizons. Thanks to the above advancements, our control policy can

---

Under review as a conference paper at ICLR 2022
be trained exclusively in simulation with minimal sim-to-real gap. We demonstrate that our pol- icy outperforms baseline deposition methods in simulation and physical hardware with low or high viscosity materials. Furthermore, our numerical model can serve as essential building block for future research in optimal material deposition, and we plan to make the source code available.
2 RELATED WORK
To identify process parameters for additive manufacturing, it is important to understand the complex interaction between a material and deposition process. This is typically done through trial-and- error experimentation (Kappes al. 2018 Wang al. 2018 Baturynska al. 2018 Recently, optimal experiment design and, more specifically, Gaussian processes have become a tool for effi- cient use of the samples to understand the deposition problem (Erps et al., 202I). However, even though Gaussian Processes model the deposition variance, they do not offer tools to adjust the de- position on-the-fly. Another approach to improve the printing process is design closed-loop con- trollers. One of the first designs was proposed by (mithi--morn et al. (2015) that monitors each layer deposited by a printing process to compute an adjustment layer. Liu et al. (2017) built upon the idea and trained discriminator that can identify the type and magnitude of observed defects. A similar approach was proposed by Yao et al. (0188) that uses handerafted features to identify when a print significantly drops in quality. The main disadvantage of these methods is that they rely on col- lecting the in-situ observations to propose one corrective step by adjusting the process parameters. However, this means that the prints continue with sub-optimal parameters, and it can take several layers to adjust the deposition. In contrast, our system runs in in-process and reacts to the in-situ views immediately. This ensures high-quality deposition and adaptability material changes.
Recently machine learning techniques sparked a new interest in the design of adaptive control poli- cies MMiih al. 2013). particularly successful approach for high-quality in- in-process control is to adopt the Model Predictive Control paradigm (MPC) (Gu et al. 2016 Silver et al., 2017 Oh et al.) 2017 Srinivas et al., 2018 Nagabandi al. 2018). The control scheme of MPC relies on an obser- vation of the current state and a short-horizon prediction of the future states. By manipulating the process parameters, we observe the changes in future predictions and can pick a future with desirable characteristics. Particularly useful is to utilize deep models to generate differentiable predictors that provide derivatives with respect to control changes (de Avila Belbute- et al.) 2018 Schenek & Fox 2018 Toussaint al. 2018 Li et al. 2019a). However, addressing the uncertainties of the de- position process with MPC is challenging. In a noisy environment, we can rely only on the expected prediction of the deposition. This leads to conservative control policy that effectively executes the mean action. Moreover, reacting to material changes over time requires optimizing actions for long time horizons which is known weakness of the MPC paradigm (Garcia et al. 1989. As a result, MPC is not suitable for in in-process control in noisy environments.
Another option to derive control policies is to leverage deep reinforced learning (Raeeswaran al.
2017 Liu & Hodgins 2018 Peng et al., 2018 Yu et al. 2019 Lee et al, 2019 Akkaya et al. 2019.. L The key challenge in the design of such controllers is formulating an efficient numerical model that captures the governing physical phenomena. As a consequence, it is most commonly applied to rigid body dynamics and rigid robots where such models are readily available (Todorov) et al. 2012. Bender et al. 2014| Coumans & Bai 2016: Lee et al. 20.8). In contrast, learning with non-rigid objects is significantly more challenging as the computation time for deformable materials is higher and relies some prior knowledge on the task (Clegg et al. 2018, Elliott & Cakmak 2018 Ma et al.] 2018: Wu et al. 2019). Recently Zhang et al. (2020) proposed a numerical model for training control policies where rigid object interacts with amorphous materials. Similarly, in our work a rigid printing nozzle interacts with the fluid-like printing material. However, our model specialized for the printing hardware and models not only the deposition but also its variance. We demonstrate that this is an important component in minimizing the sim-to-real gap and design control policies that are readily applicable to the physical hardware.
HARDWARE PRELIMINARIES
The choice of additive manufacturing technology constraints the subsequent numerical modeling.
To keep the applicability of our developed system as wide as possible, we opted for a direct write
2

---

Under review as a conference paper at ICLR 2022
needle deposition system mounted a 3-axis Cartesian robot (inset). The robot allows us to freely control the acceleration and position of the dispenser. The dispenser can process wide range of viscous materials, and the deposition is very similar to fused deposition modeling. We further enhance the apparatus with two camera modules. The cameras lie on the opposite sides of the nozzle allow our apparatus perceive the location around the deposition. It is this locality of the in-situ view that we will leverage to formulate our numerical model.
Material
Nozzle
Camera
3.1 BASELINE CONTROLLER
To control the printing apparatus, employ baseline slicer. The input to the slicer is three-dimensional object. The output is se- ries of locations the printing head visits reproduce the model closely as possible. To generate single slice of the object, we start by intersecting the 3D model with a Z-axis aligned plane (please note that this does not affect the generalizability since the input can be arbitrarily rotated). The slice represented by polygon that marks the outline of the printout (Figure gray). To generate the printing path, we assume a constant width of deposition (Figure [ red) that acts
convolution on the printing path. The printing path (Figure [ blue) is created by offsetting the print boundary by half the width of the material using the Clipper algorithm (John- son 2015). The infill pattern is generated by tracing a zig-zag line through the area of the print (Figure green).
Material Width Outline Path Infill Path Target
Figure 1: Baseline slicer.
4 REINFORCEMENT LEARNING FOR ADDITIVE MANUFACTURING
The baseline control strictly relies on a constant width of the material. To discover policies that can adapt to the in-situ observations, we formulate the search in a reinforcement learning frame- work. The control problem is described by a Markov decision process (S, A, P, R), where the observation space, A is a continuous action space, P P(s'|s,a) is the transition function, and (s. + the reward function.
To learn a control policy we take model free approach by learning directly from printing. Un- fortunately, learning on a physical device challenging. The interaction between various process parameters can lead to deposition errors that require manual attention. As such discovering control policies directly on the hardware has too steep sample complexity to be practical. A potential so- lution is to learn the control behavior in simulation and transfer to the physical device. However, transfer from simulation to real world is a notoriously hard problem that hinges on applicability of the learned knowledge. In this work, additive manufacturing that facilitate the sim-to-real transfer. Our model has three key components that facilitate the generalization of the learned control policies.
propose a framework for preparing numerical models for
The first component is the design of the observation space. To facilitate the transfer of learning between simulation and physical device, we rely on an abstraction of the observation space (Kauf- mann et al. 2020p. Rather than using the direct appearance feed from our camera module we process the signal into heightmap. heightmap is a 2D image where each pixel stores the height of the deposited material. For each height map location, the height is measured as distance from the building plate to the deposited material. This allows our system to generalize to many different sensors such cameras, depth sensors, laser profilometers. However, unlike Kaufmann et al.) 22200. we do not extract the feature vectors manually. Instead, similarly to OpenAl et al. (2018), learn the features directly from the heightmap. In contrast to OpenAl al. (2018) do not randomize the observation domain. Additional randomization is not necessary in our case thanks to the controlled observation conditions of the physical apparatus.
A key insight of our approach is that the engineered observation space coupled with learned features can significantly help with policy learning. A careful design of the observation space can facilitate
3

---

Under review as a conference paper at ICLR 2022
the sim-to-real transfer, make the hardware design more flexible by enabling the of a range of sensors that compute similar observations, and remove the need to hand-craft the features. It is therefore worth wile to invest in the design of observation spaces.
The second component of our system is the design of the action space, Instead of directly controlling the motors of the printer we rely on high-level control scheme and tune coupled parameters such as velocity or offset from the printing path. This idea is similar in spirit to OpenAl et al. (2018).
OpenAl et al. (2018) suggest not using direct sensory inputs from the mechanical hand as observa- tions due to their noisiness and lack of generalization across environments. Instead, they use image data to track the robotic hand. Similarly, but instead in action space, we do not control the printer by directly inputting the typically noisy and hardware-specific voltages that actuate the motors of the apparatus. Instead, we control the printer by setting the desired velocity and offset and letting the apparatus match them to the best of its capabilities. This translation layer allows us to utilize the controller on broader range of devices without per-device training.
This idea could also be generalized to other robotic tasks, for example, by applying hierarchical divide and conquer approach to the action space. The control policies could output only high-level actions such as desired locations for robots actuators or deviations from baseline behavior. Low- level controllers could then exccute these higher-level actions. Such a control hierarchy can facilitate training by decoupling the higher-level goals from low-level inputs and transferring existing control policies to new devices through specialized low-level controllers.
The third and last component of system is approximative transition function. Rather than modelling the deposition process exactly we propose to approximate it qualitatively. A qualitative approximation allows us to design an efficient computational model. To facilitate the transfer of the simulated model to the physical device we reintroduce the device uncertainty in a data-driven fashion. This is similar to OpenAl al. (2018), but instead of covering a large array of options, we specialize the randomization. Inspired by Chebotar et al. (2019), we designed a data-driven LPC filter that matches the statistical distribution of variations observed during a typical printing process. This noise enables our control policies to adapt to changing environments and, to some extent, to changes in material properties such as viscosity.
Our approximative transition function shows that it is not necessary to reproduce the physical world in simulation perfectly. A qualitative approximation terns that translate to real-world experiences. This is an important observation for any task where we manipulate objects and clastic or frictional forces dominate the behavior. Relying on computation- ally more affordable simulations allows for applying existing learning algorithms to broader range of problems where precise numerical modeling has prohibitive computational complexity. More- over, by leveraging a numerical model it is possible to utilize privileged information that would be challenging if not impossible to collect in the real world. For full description of our methods please see Appendix A
sufficient as long as we learn behavior pat-
5 RESULTS
In this section, provide results obtained in both virtual and physical environments. We first show that an adaptive policy can outperform baseline approaches in environments with constant deposition. Next, we showcase the in-process monitoring and the ability of our policy adapt to dynamic environments. Finally, we demonstrate our learned controllers transferring to the physical world with a minimal sim-to-real gap.
5.1 COMPARISON WITH BASELINE CONTROLLER
We evaluate the optimized control scheme on a selection of freeform and CAD models sampled from Thingy 10k (Zhou & Jacobson 2016) and ABC (Koch et al. 2019) datasets (Appendix (A.6). In total, we have 113 unseen slices corresponding to 96 unseen geometries. We report our findings in Figure 2 For each input slice, report improvement on the printed boundary as the average offset. The average offset is defined as a sum of areas of under and over deposited material normalized by the outline length. More specifically, given image of the target slice T, printed canvas C, weight
4

---

Under review as a conference paper at ICLR 2022
mask W, and the length of the outline I, the average offset O is computed as:
C(--T) (1 C)TW + 1
The improvement is calculated as difference between the baseline and our policy, Therefore, value higher than zero indicates that our control policy outperformed the baseline. As we can see, policy achieved better performance in all considered models.
Average Offsel Improvement
25
20
(microns)
15
Validation Slices
Figure 2:
The relative improvement of our policy over baseline.
Next, we investigate the shapes where our control policy achieves the highest and the lowest gain, respectively (Figure B). Best performance achieved in smooth regions. The reason is that our policy is capable of adjusting the printing parameters based on the curvature while the baseline's constant speed more suitable for a limited range of curvatures. Conversely, our policy achieves the weakest performance material in sharp regions is too large for the desired feature scale, leading to over-deposition. If such thin features are desired to print regularly, a thinner material nozzle can alleviate this issue.
objects with sharp features. This is natural as the width of the deposited
Highest Gain
Lowest Gain
Figure 3: Representative deposited patterns from the evaluation dataset.
Finally, compare control policy with fine-tuned baseline. The baseline controller uses the same parameters for each slice. Different process parameters may be optimal for different slices. To this end, choose slices, freeform slice of a bird and CAD slice of a bolt and optimize their process parameters using Bayesian optimization, Figure 26 (for numerical details see Appendix [B. We can observe that the two control schemes require drastically different velocities (1.46 SU/s 0.89 SU/s) to maximize performance. Moreover, we can see that the control parameters are not interchangeable, Appendix B When switching the control parameters, we observe loss in performance, This loss is caused by each set of control parameters exploiting the local properties of the slice used for training. Lastly, we compare the individually optimized control parameters with our policy. Our policy improves upon both baseline solutions while maintaining generalizability. This is possible because our control policy relies on live feedback adjust the printing parameters on-the-fly.
5.1.1 ABLATION STUDY ON OBSERVATION SPACE
Our control policy relies on live view of the deposition system to select the control parameters. However, the in-situ view is technologically challenging addition to the printer hardware that requires carefully calibrated imagining setup. With this ablation study, we verify how important the individual observations are to the final print quality. We consider three cases: (I) no printing bed view, (2) no target view, and (3) no future path view. We analyzed the results from the pre-test
(full observation space = 9.7, σ = 4.9) and the post-tests (no canvas μ = 8.8, a = 5.7, no target 7.2, 5.5, path μ 8.4, = 4.8) printing task using paired t-tests with Holm- correction. The analysis indicates that the availability of all three inputs: the printing bed, the target,
Holm-Bonferront Bonferroni
and the path improved final printouts (P 0.01 for all three cases).

---

Under review as a conference paper at ICLR 2022
5.2 PERFORMANCE DYNAMIC ENVIRONMENTS
We use an identical random pressure variation profile to perform quantitative evaluation in envi- ronments with varying pressure. We use the same evaluation dataset as for constant-pressure policies and report the overall improvement over the baseline controller, (Figure 4. We can observe that in each the considered slices, our closed-loop controller outperformed the baseline.
Averagee Offset Improvement
35
30
(microns)
15
Figure The relative improvement of our policy over baseline.
We have also evaluated the infill policy in noisy environ- ment, (Figure 5). We can observe that the deposition noise leads to accumulation of material. The accumulation tually results in bulge of material in the center of the print, complicating the deposition of subsequent layers as the mate- rial would tend to slide off. In contrast, our policy dynamically adjusts the printing path to generate a print with significantly better height uniformity. As we can observe, the surface gen- erated by our policy almost flat and ready for deposition of potentially more layers.
Max
Heghh
Contians Frik
Contra Folky riki
Figure 5: Infill comparison.
5.3 ABLATION STUDY ON VISCOSITY
To verify that our policy can adapt to various materials, we trained three models of varying viscosity, (Figure (6. We observe that, without adaptive control scheme, the pressure changes sufficiently strong to cause local over- or under-deposition. Our trained policy dynamically adjusts the offset and velocity counterbalance the changes in the deposition. We can that policy particularly good at handling smooth width changes and quickly recovers from a spike in printing
width.
Baseline
Ours
Baseline
Ours
Baseline
Ours
Viscosity
Figure 6: Performance of our policy and baseline with varying viscosity.
5.3.1 ABLATION STUDY ON ACTION SPACE
To evaluate the need tweak both the print- ing velocity and the printing path, we trained two control policies with a limited action set to either alter the velocity or path offset. We analyzed the results from the pre-test (full ac- tion space f4 12.7, a 5.7) and the post- tests (velocity μ = 7.5, a = 2.5, displacement 5.6, = 8.3) printing task using paired t- tests with Holm- -onn-Boffrrroni correction. The analysis indicates that the availability of the full action space resulted in improvement in final printouts (P < 0.001 for both cases). The difference in performance depends on the inherent limitations of the individual actions. On the one hand, adjust- ing velocity is fast (under 6.6 milliseconds) but can cope only with moderate changes in material
Actiin Space
Velocity Only
Displacement Only
Figure 7: Action space ablation study.

---

Under review as a conference paper at ICLR 2022
width. This can be observed the larger bulges of over-deposited material in Figure middle. On the other hand, while offset can cope with larger material differences but it needs between 0.13 and 1.3 seconds to adjust. As result, offset adjustment cannot cope with sudden material changes, (Figure (Figure(righht) right). In contrast, by utilizing the full action space our policy can combine the advantages of the individual actions and minimize over-deposition, (eggtt).left)
5.3.2 ABLATION STUDY ON REWARD FUNCTION
Our reward function uses privileged informa- tion from the numerical simulation to evaluate how material settles over time. However, such information not readily available on physical hardware. One either evaluates the reward once at the end of each episode to include material flow at each timestep by disregarding long- term material motion. We evaluated how such changes to the reward function would affect our control policies. We analyzed the results from the 5.7) and the post-tests (delayed reward
Privieged
Delayed Reward
Immediate Reward
Figure 8: Reward function ablation study.
222.3, 8.6, immediate reward μ 9.2, a 8.0) printing task using paired t-tests with Holm- Bonferroni correction. The analysis indicates that the availability of the privileged information resulted in an improvement in final patterns (P 0.001 for both cases). The learning process for delayed reward is significantly slower, and is unclear if performance similar to our policy can be achieved, Appendix A.66 On the other hand, the immediate reward policy learns faster but cannot handle material changes over longer time horizons, (Figure
pre-test (privileged reward = 12.7,
5.4 PERFORMANCE ON PHYSICAL HARDWARE
Finally, we evaluate our control policies on physical hardware. The policies were trained entirely in simulation without any additional fine-tuning the printing device. To conduct the evaluation, we equipped our printer with a pressure controller. The pressure control was set to sinusoidal oscillatory signal provide controllable dynamic change in material properties. We used two materials, with high and low viscosity, and used two separate policies pretrained in simulation using those materials. We printed 22 slices, of which 11 corresponded
the simulation training set and 11 to the evaluation set. We monitor the printing process and use the captured images to run our evaluation function to capture quantitative results. We observe that our controllers improve the average offset over the baseline print in every scenario, (Figure 9.
Average Offset Improvement
100
Low Viscosity Material
High Viscosity Material
80
(microms)
60
40
20
Training Slices
Evaluation Slices
Training Slices
Evaluation Slices
Figure 9: The relative improvement of policy over baseline.
A sample of the fabricated slices can be seen in Figure The print target (white) is overlayed with a map of underdeposited (blue) and overdeposited (red) material. We further plot a histogram of under and over deposition.
We can see that our control policy transferred excellently to the physical hardware without any ad- ditional training. Our policy consistently achieves smaller over-deposition while not suffering from significant under-deposition. Moreover, in many cases our policy achieves histograms with smaller width suggesting achieved tighter control the material deposition than the baseline. This demonstrates that our numerical model enables learning control policies for additive manufacturing in simulation.

---

Under review as a conference paper at ICLR 2022
6 CONCLUSION
We present the first closed-loop control policy for additive manufacturing recovered via reinforce- ment learning. To learn an effective control policy, we propose custom numerical model of the deposition process. During the design of our model, we tackle several challenges. To obtain an printing
efficient approximation of the deposition process, we leverage the limited perception apparatus and model the deposition only qualitatively. To include non-linear coupling between pro- cess parameters and printed materials, we utilize data-driven predictive model for the deposition imperfections. Finally, to enable long horizon learning with viscous materials, we use the privileged information generated by our numerical model for reward computation. several ablation studies, show that these components instantaneous and long horizon material changes, handle materials with varying viscosity, and adapt the deposition parameters achieve printouts with minimal over-deposition and smooth top layers.
required to achieve high-quality printing, effectively react to
We demonstrate that our model can be used to train control policies that outperform baseline con- trollers, and transfer to physical apparatus with minimal sim-to-real gap. We showease this by applying control policies trained exclusively our policies fabricate several prototypes using low and high viscosity materials. The quantita- tive and qualitative analysis clearly shows the improvement of our controllers over baseline printing. This indicates that our numerical model can guide the future development of closed-loop policies for additive manufacturing. Thanks to its minimal sim-to-real gap, the model democratizes research on learning for additive manufacturing by limiting the need to invest in specialized hardware. Further- more, by expanding the simulator with other physical phenomena, e.g., abrasion, melting, or heat transfer, our numerical model can serve as a blueprint for learning closed-loop control policies of other manufacturing methods such as milling, direct energy deposition, or selective laser sintering.
simulation on a physical printing apparatus. We
8

---

Under review as a conference paper at ICLR 2022
Low Viscosity Material
High Viscosity Material
Baseline
Baseline
Ours
Baseline
Baseline
Ours
Ours
Ours
Underdeposition
Overdeposition
Underdepasition
Overdeposition
Baseline
Baseline
Ours
Baseline
Baseline
Ours
Ours
Ours
Underdeposition
Overdeposition
Underdeposition
Overdeposition
Baseline
Baseline
Ours
Baseline
Baseline
Ours
Ours
Ours
Underdepositton
Overdeposition
Underdeposition
Overdeposition
Baseline
Baseline
Ours
Baseline
Baseline
Ours
Ours
Ours
Underdeposition
Overdeposition
Underdeposition
Overdeposition
Figure 10: Deposition quality estimation of physical result manufactured with baseline and our learned policy.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.00 13, 2019.
Ivanna Baturynska, Oleksandr Semeniuta, and Kristian Martinsen. Optimization of process parame- ters for powder bed fusion additive manufacturing by combination of machine learning and finite element method: A conceptual framework. Procedia Cirp, 67:227-232, 2018.
Jan Bender, Matthias Müller, Miguel A Otaduy, Matthias Teschner, and Miles Macklin. A sur- vey position based simulation methods in computer graphics. In Computer graphics forum, volume 33, pp. 228 251. Wiley Online Library, 2014.
John Parker Burg. Maximum Entropy Spectral Analysis. Stanford Exploration Project. Stanford University, 1975.
Yevgen Chehotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973- 8979. IEEE, 2019.
Alexander Clegg, Wenhao Yu, Jie Tan, Karen Liu, and Greg Turk. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions on Graphics (TOG), 1-0 2018
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and Zico Kolter. End- to-end differentiable physics for learning and control. Advances neural information processing systems, 31:7178-7188 2018.
Sarah Elliott and Maya Cakmak. Robotic cleaning through dirt rearrangement planning with learned transition models. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1623-1630. IEEE, 2018
Timothy Erps, Michael Foshey, Mina Konaković Luković, Wan Shou, Hanns Hagen Goetzke, Herve Dietsch, Klaus Stoll, Bernhard von Vacano, and Wojeiech Matusik. Accelerated discovery of 3d printing materials using data-driven multi-objective optimization, 2021.
Wei Gao, Yunbo Zhang. Devarajan Ramanujan, Karthik Ramani, Yong Chen, Christopher B Williams, Charlie CL Wang, Yung C Shin, Song Zhang, and Pablo D) Zavattieri. The status, challenges, and future of additive manufacturing in engineering. Computer-Aided Design, 69: 65-89, 2015.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and prac- tice-a a survey, Automatica, 25(3):335-348, 1989.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pp. 2829-28334 PMLR, 2016.
Angus Johnson. Clipper an open source freeware library for clipping and offsetting lines and polygons. http:// http:///www... http//wwwaanggg www.w.nguj..... www/de//eeeeeeeo httilclipppeeeeeeeeeeeeep -hhp, 2015.
Branden Kappes, Senthamilaruvi Moorthy, Dana Drake, Henry Geerlings, and Aaron Stebner. Ma- chine learning to optimize additive manufacturing parameters for laser powder bed fusion of in- conel 718. In Proceedings of the 9th International Symposium on Superalloy 718 & Derivatives: Energy, Aerospace, and Industrial Applications, pp. 595-610. Springer, 2018.
Elia Kaufmmann, Antonio Loquercio, René Ranftl, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Deep drone acrobaties. arXiv preprint 2rXi::2006005766 2020.
10

---

Under review as a conference paper at ICLR 2022
Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Bur- naev, Mare Alexa, Denis Zorin, and Danicle Panozzo. Abc: A big cad model dataset for geometric deep learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
June 2019.
Jeongseok Lee, Michael X Grey, Schoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S Srinivasa, Mike Stilman, and C Karen Liu. Dart: Dynamic animation and robotics toolkit. Journal of Open Source Software, 3(22):500, 2018.
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation control. ACM Transactions On Graphics (TOG), 38(4):1-13, 2019.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019a.
Yunzhu Li, Jiajun Wu, Lun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pp. 1205-1211. IEEE, 2019b.
Chenang Liu, David Roberson, and Zhenyu Kong. Textural analysis- baase online closed-loop qual- ity control for additive manufacturing processes. IIE Annual Conference. Proceedings, pp. 1127-1133 1132. Institute of Industrial and Systems Engineers (IISE), 2017.
Libin Liu and Jessica Hodgins. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. Fluid directed rigid body control using deep reinforcement learning. ACM Transactions on Graphics (TOG), 37(4): 1-11, 2018
Miles Macklin and Matthias Müller. Position based fluids. ACM Transactions on Graphics (TOG), 32(4):1-12, 2013.
Larry Marple. A new autoregressive spectrum analysis algorithm. JEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4):441 454, 1980. doi: 10. 1991 1163429.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (187540):529-333 2015.
Mojtaba Mozaffar, Arindam Paul, Reda Al-Bahrani, Sarah Wolff, Alok Choudhary, Ankit Agrawal, Kornel Ehmann, and Jian Cao. Data-driven prediction of the high-dimensional thermal history in directed energy deposition processes via recurrent neural networks. Manufacturing letters, 18: 35-39, 2018.
Matthias Müller, David Charypar, and Markus Gross. Particle-based fluid simulation for interac- tive applications. In Symposium Computer animation, pp. 154-159, 2003.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-basee deep reinforeement learning with model-free fine-tuning, In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In NIPS, 2017.
M Andrychowicz OpenAL, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint 1108.01800.0011100 2(3)) 2018.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-basee character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
11

---

Under review as a conference paper at ICLR 2022
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:
2017.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamies for deep neural networks. In Conference on Robot Learning, pp. 317-335, PMLR, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06 2017.
David Silver, Hado Hasselt, Mattco Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Aroll,, David Reichert, Neil Rabinowitz, Andre Barreto, al. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, pp. 3191-3199. PMLR, 2017.
Pitchaya Sitthi-morn, Javier Ramos, Yuwang Wangy, Joyce Kwan, Justin Lan, Wenshou Wang, and Wojeiech Matusik. Multifab: machine vision assisted platform for multi-material 3d print- ing. Aem Transactions on Graphics (Tog), 34(4):1-11, 2015.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pp. 4732-4741. PMLR, 2018.
Chao Tang, Jie Lun Tan, and Chee How Wong. A numerical investigation on the physical mecha- nisms of single track defects in selective laser melting. International Journal of Heat and Mass Transfer, 126:957-968, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Mare A Toussaint, Kelsey Rebecca Allen, Kevin A Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for usel-use and manipulation planning. 2018.
Chengeheng Wang, Xipeng Tan, Erjia Liu, and Shu Beng Tor. Process parameter optimization and mechanical properties for additively manufactured stainless steel 3161 parts by selective electron beam melting. Materials & Design, 147:157-166, 2018.
Chengcheng Wang, XP Tan, SB Tor, and CS Lim. Machine learning in additive manufacturing: State-of-the-ard and perspectives. Additive Manufacturing, pp. 101538, 2020.
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv: /910. 13439, 2019.
Wentao Yan, Ya Qian, Wenjun Ge, Stephen Lin, Wing Kam Liu, Feng Lin, and Gregory Wagner. Meso-scale modeling of multiple-layer fabrication process in selective electron beam melting: inter-layer/track voids formation. Materials & Design, 141:210-219, 2018.
Bing Yao, Farhad Imani, and Hui Yang. Markov decision process for image-guided additive manu- facturing. IEEE Robotics and Automation Letters, 3(4):2792-27988, 2018.
Ri Yu, Hwangpil Park, and Jehee Lee. Figure skating simulation from video. In Computer graphics forum, volume 38, pp. 225 2344. Wiley Online Library, 2019.
Yunbo Zhang, Wenhao Yu, C Karen Liu, Charlie Kemp, and Greg Turk. Learning to manipulate amorphous materials. ACM Transactions on Graphics (TOG), 39(6):1-11, 2020.
Qingnan Zhou and Alec Jacobson. Thingi10k: dataset of 10,000 3d-printing models. arXiv preprint arXiv: 2016.
12

---

Under review as a conference paper at ICLR 2022
A METHODS
A.1 HARDWARE SETUP
In this work, we developed a direct write 3D printing platform with an optical feedback system that can measure the dispensed material real-time, in-situ. The 3D printer is comprised of a pressure driven syringe pump and pressure controller, a 3-axis Cartesian robot, optical imaging system, back- lit build platform, 3D- 3D-printer controller and CPU Figure I1 The 3-axis Cartesian robot is used to locate the build platform in x and y-direction and the print carriage in the z-direction. The pressure driven syringe pump and pressure controller used to dispense and optically opaque material onto the back-lit build platform. The back-lit platform is used to illuminate the dispensed material. The movement of the robot, actuation of the syringe pump and timing of the cameras are controlled via the controller. The CPU is used to process the images after they are acquired and compute updated commands send the controller.
Figure 11: The printing apparatus consisting of a 3-axis Cartesian robot, direct write printing head, and camera setup.
A.1.1 CALIBRATION
To enable realtime control of printing process we implemented an in-situ view of the material deposition. Ideally we would capture top-down view of the deposited material. Unfortunately, this not possible since the material obstructed by the dispensing nozzle. As a result the camera has to observe the printing bed from angle. Since the nozzle would obstruct the view of any single camera we opted to use two cameras. More specifically, we place two CMOS cameras (Basler AG, Ahrensburg, Germany) at 45 degrees on each side of the dispensing nozzle, Figure II We calibrate the camera by collecting a set of images and estimating its intrinsic parameters, Figure calibraaai. To obtain single top-down view we capture calibration target aligned with the image frames of both cameras, Figure 12 homography. By calculating the homography between the captured targets and an ideal top-down view we can stitch the images into single view from virtual over-the-top camera. Finally, we mask the location of each nozzle in the image (Figure T2 nozzle masks) and obtain the final in-situ view, Figure 12 stitched image.
The recovered in-situ view is scaled to attain the same universal scene unit size our control policies are trained in. Since we seek to model the deposition only qualitatively it is sufficient to rescale the in-situ view to match the scale of the virtual environments. We identify this scaling factor separately for each material. To calibrate a single material we start by depositing a straight line at maximum velocity. The scaling factor is then the ratio required to match the observed thickness of the line with simulation. To extract the thickness of the deposited material we rely on its translucency propertics. More precisely, correlate material thickness with optical intensity. We do this be depositing the
13

---

Under review as a conference paper at ICLR 2022
Left Right Combined
Right Left
Stitched
Homography
Nozzle Masks
Image
Stitched
Calibration
Image
Locations
Image
The calibration of the imaging setup. First intrinsic parameters are estimated from cal- ibration patterns. Next we compute the extrinsic calibration by calculating homographies between the cameras and overhead view. We extract the masks by thresholding photo of the nozzle. The final stitched image consists of 4 regions: (1) view only in left camera, (2) view only right camera, (3) view in both cameras, (4) view in camera. The final stitched image is show the right.
Figure 12:
material at various thickness and taking a picture with our camera setup. The optical intensity then decays exponentially with increased thickness which is captured by power law fit.
1mm
0, 0.1mm
Material Thickness [mm]
0.75
0.05mm
0.50
0.75mm
0.00
Intensity
Figure 13: Calibration images for correlating deposited material thickness with optical intensity and the corresponding fit.
The last assumption of control policy is that the deposition needle is centered with respect to the in-situ view. To ensure that this assumption holds with the physical hardware we calibrate the location of the dispensing needle within the field of view of each camera and with respect to the build platform. First, a dial indicator is used to measure the height of the nozzle in and the fine adjustment stage is adjusted until the nozzle is 254 microns above the print platform. Next, using calibration target located on the build platform and the fine adjustment stage, the nozzle is centered the field of view of each camera. This calibration procedure is done each time the nozzle replaced during the start of each printing session.
A.1.2 BASELINE CONTROLLER
To calibrate the baseline control we follow the same procedure simulation and physical hardware.
We start be depositing straight line at constant velocity, Figure [4] Next, measure the width of the deposited line at various locations to estimate the mean width. We use the width to generate the offset for outline printing and spacing of the infill pattern.
14

---

Under review as a conference paper at ICLR 2022
Print Boundary Nozzle Path Width w
Width w
Width Estimation
Baseline controller starts by estimating the width w of the deposited material. A control sequence for the nozzle is estimated by offsetting the desired shape by half the size of material width.
Figure 14:
A.2 CONTROL POLICY INPUT STATES
To define the input states, we closely follow the constraints of the physical hardware. We model our observation space as small in-situ view centered at the printing nozzle location. The view has size of 84 x 84 pixels which trans- late roughly 2.95 2.95 scene units (SU). The view contains either heightmap (for infill printing) or material segmentation (for outline printing). Since the location directly under the nozzle is obscured for the physical hardware, mask small central position in the view equivalent to 0.42 SU t of the in-situ view. To- gether with the local view, we also provide the printer with a local image of the desired printing target and the path the control policy will take in the environment. To further minimize data redun- dancy, we rotate the in-situ view such that the printer moves along the positive X-axis in the image. These three inputs stacked together into 3-channel image, (Figure
Govveed ty Nuzce
In-Situ Printing Bed
Desired Target
Nozzle Path
Figure 15: Control Policy Input.
A.3 ACTION SPACE
The selection of action space plays critical role in adapting a con- troller to the actual hardware. One possibility is to control and di- rectly modify the voltage input of individual motors. However, such an approach is not readily transferable between printing devices. The controls are tied too tightly to the hardware selection and would ex- aggerate the sim-to-real gap. Moreover, directly affecting the motor voltage would mean that the control policy must learn how to trace print inputs. Instead, propose strategy that leverages the body of work on designing baseline controllers. Similar to baseline, our control policy follows path generated by a slicer. However, enable dynamic modification the path. At each state, the printer can modify actions: (1) the velocity at which the printing head is moving and (2) displacement of the printing head in a direction perpendicular the motion, Figure 16 Such a formulation allows us to decouple the hardware parameters from the control scheme and apply same policy in both simulation and physical hardware by scaling the input units appropriately. In our simulation, we limit our velocity to the range of [0.2, 2] SU/s and the displacement to 0.2666 SU.
Displacement
Velocity
In-Situ Printing Bed
Figure 16: The action space.
A.4 TRANSITION FUNCTION
The transition function takes a state-action pair and outputs a new state of the environment. In our setting, this means need numerically model the fabrication process, which is notoriously
15

---

Under review as a conference paper at ICLR 2022
difficult problem. Here we leverage assumption that the observation space is localized that it can identify the deposited materials only qualitatively. Therefore, we can trade physical realism for visual fidelity and efficiency. This description fits the Barooon--aad--Dynne Müller 2013) framework, which a geometrical approximation to the equations of motion.
(PBD) (Macklin &
To model the interaction of the deposited material with the printing apparatus we rely Position- Based Dynamics (PBD). PBD approximates rigid, viscous, and fluid objects as collections of par- ticles. To represent the fluid we assume a of N partieles where each particle defined by its position p. velocity v, mass m, and a set of constraints C. In our setting we consider two con- straints: (1) collision with the nozzle and (2) incompressibillty of the fluid material. We model the collision with the nozzle as a hard inequality constraint:
C((p)) = (pi -n..
where q is the contact point of a particle with the nozzle geometry along the direction of particles motion v and ne is the normal at the contact location. To ensure that our fluids remain incompres- silbe we follow (Macklin & Müller 2013) and formulate a density constraint for each particle:
CP Pn)
Pi
1,
(3)
Po
P: = Σ
my m, h),,
where is the rest density and pj is given by Smoothed Particle Hydrodynamics estimator (Müller al. 2003) in which W is the smoothing kernel defined by the smoothing scale h.
We further tune the simulation parameters to achieve a wide range of viscosity properties. More specifically, we couple the effects of viscosity, adhesion, and energy dissipation into single set- ting. By coupling these parameters we obtain materials with optically different viscosity properties. Moreover, we noticed that the number of solving substeps has a significant effect on viscosity and surface tension of the simulated fluids. Therefore, we also tweak the number of substeps from 2 for liquid-like materials
for highly-viscous materials.
We replicate our printing apparatus in the simulation, inset. We model the nozzle as a collision object with a hard contact constraint on the fluid particles. Since modeling pressurized reservoir is computationally costly as it requires us to have many particles constant contact, chose to approximate the deposition process at the peak of the nozzle. More specifically, we model the depo- sition as particle emitter. To the volume and velocity of the particles, we use a flow setting. The higher the flow, the more par- ticles with higher initial velocities are generated. This qualitatively approximates the deposition process with a pressurized reservoir. The particle emitter is placed slightly inside the nozzle allow for realistic material buildup and delayed stop, similar to extrusion processes. Finally, we consider the printer to have only a finite acceleration per timestep. To accelerate to target velocity, we employ a linear acceleration scheme.
Nozzle
Material Emitter
Deposited Material
Printing Bed
Another important choice for the numerical model is the used dis- cretization. We have two options: (1) time-based and (2) distance- based. We originally experimented with time-based discretization. However, we found out that time discretization not suitable for printer modeling. As the velocity in simulation approaches zero, the difference in deposited material becomes progressively smaller until the gradient information completely vanishes, Figure [7 left. Moreover, a time-based discretization allows the policy to affect the number of evaluations of the environment directly. As result, it can avoid being punished for bad material deposition by quickly rushing the environment to finish. Considering these factors we opted for distance-based discretization, Figure 17 right. The policy specifies the desired velocity at each interaction point, and the en- vironment travels predefined distance (0.2666 SU) at the desired
Minimum Velocity Maximum Velocity
Time Based
Distance Based
New Material Between Timesteps
Figure 17: Discretization. speed. This helps to regularize the reward function and enable learning of varying control policies.
16

---

Under review as a conference paper at ICLR 2022
An interesting design element is the orientation of the control poly- gons created by the slicer. When the outline is defined as points given counter-clockwise, then due to the applied rotation, each view
Outline Hole
Clockwise Counterclockwweee
is split roughly into two half-spaces, (Figure 18). The bottom one corresponds to outside i.e., generally black, and the upper cor- responds to inside i.c., generally white. However, the situation changes when outlining hole. When printing hole the two half- spaces swap location. We can remove this disambiguity by chang- ing the orientation of the polylines defining holes in the model. By orienting them clockwise, we will effectively swap the two half-
spaces to the same orientation as when printing the outer part. As result, we achieve better usage of trajectories and more robust control scheme that does not need to be separately trained for each print's 's outer and inner parts.
Figure 18: Orientation.
To design a realistic virtual printing environment, the model needs to capture the deposition imper- fections. The source of these imperfections is the complex non-linear coupling between the dynamic material properties and the deposition parameters. Analytical modeling of this coupling challeng- ing as it requires deep understanding of these interactions. Instead, we adopted a data-driven model. We observe that the final effect of the deposition error is a varying width of the deposited material. To recover such model for our apparatus, we start by printing a reference slice over mul- tiple iterations, (Figure 19 left). At each iteration, we measure the width of the deposited material at specified cross-sections, (Figure 19 middle). This yields observations of how the material width evolves in time, (Figure 19 left). To formulate predictive generative model, we employ a tool from speech processing called Linear Predictive Coding (LPC) (Marple) 1980). The model assumes that signal is generated by buzz filtered by auto-correlation filter. We use this assumption to recover filter coefficients that transform white Gaussian noise into realistic pressure samples, (Figure 19 left).
To formulate a predictive generative model we employ a tool from speech processing called Linear Predictive Coding (LPC) (Marple. 1980). We can predict the next sample of signal as a weighted sum of M past output samples and a noise term:
M Σ
an-mTn-m Cn:
where x are the signal samples, e is the noise term, and aMm are the parameters of M-th order curo-correlltion filter. To find these coefficients Burg (1975) propose to minimize the following energies:
N-m
N-m
EM
lfM 12
Σ
bb
(6)
M
fM.
Σ
a OM
(7)
k=0
M
bM.
(8)
where * denotes the complex conjugate. After finding the filter coefficients with Equation we can synthesize new width variations with similar frequency composition to the physical hardware by filtering a buzz modeled intervals we further find a smooth interpolating curve that corresponds model the observed pressure variation. We use the proposed model to drive the flow setting of our simulator. This directly influences the width of the deposited material similarly to the imperfections in the deposition.
white Gaussian noise. Since we sampled the width variation discrete
A.5 REWARD FUNCTION
Viscous materials take significant time to settle after deposition. Therefore. to assess deposition emors, it is needed to observe the deposition over long horizons. However, the localized nature
17

---

Under review as a conference paper at ICLR 2022
Pressure Variation
Measurement LPC Model
V
End
Calibration Printouts
Sample Locations
Printing Episode
We performed nine printouts and measured the width variation at specified locations. We fit the measured data with an LPC model. Please note that since our model is generative, we do not exactly match the data or any observed resemblance a testament to the quality of our predictor.
Figure 19:
of the in-situ view makes such observations impossible on the physical hardware. As result, learning long-horizon planning has infeasible sample complexity. To tackle this issue, we leverage the fact that we utilize a numerical approximation of the deposition process with access to privileged information. At each simulation step, model the entire printing bed. This allows to formulate the reward function as a global print quality metric. More specifically, our metric is composed of two terms: (1) reward term for depositing material inside the desired slice and (2) a punishment term for depositing material outside of the slice. To keep the values consistent across slices of varying size, we normalize them by the length of the outline the infill area, respectively. We provide dense rewards as the difference between the metrics evaluated at two subsequent timesteps to accelerate the training further.
We consider two reward function in our setting one for outline printing and one for infill printing. Each reward function evaluates the print quality as a whole. To accelerate the learning we provide the algorithm with dense rewards as delta between the reward in-between steps R = R R".
To print the outline we want to follow the boundary as closely as possible without overfilling. To this end we compose our reward function of two terms. Given an image of the current printing bed C' and the desired target T we define the reward as Σ CT. While such formulation rewards the control policy for depositing material inside the printing volume it does not encourage tight outline fill. Indeed a potential strategy with such a reward would be to offset the printing nozzle as much inside possible and then move safely within the object bounds. To address this issue propose to include a weight map W that is computed as a thresholded distance transform of the target T. The final reward function is then: R R= - CTW. Using such formulation we put the highest weight on depositing directly on the outline boundary. The threshold cutoff then helps preventing a strategy of filling up the shape interior. To ensure that the printer deposits material inside the desired locations we include an additional punishment term P = Σ C(1 - T). Finally, both reward and punishment normalized by the length of the outline of our target.
For infill printing we compute the reward from the heightfield of the deposited material. We start by estimating how much of the slice was covered. To this end, we use a thresholded version of the canvas and compute the coverage as R = Σ CT. Similarly, we estimate the amount of over- deposited material as P C(1 - T). To keep these values consistent across different slices we normalize them by the total area of the print. Finally, to motivate deposition of flat surfaces suitable for 3D printing we add another penalty term as the standard deviation of the canvas heightfield.
A.6 TRAINING PROCEDURE
To train our control policy we start with g-code generated by a slicer. As inputs to the slicer we consider a set of 3D models collected from the Thingy 10k dataset. To train controller the input models need to be carefully selected. On the one hand, if we pick an object with too low frequency features with respect to the printing nozzle size then any printing errors due to control policy will have negligible influence on the final result. On the other hand, if we pick a model with too high frequency features with respect to the printing nozzle then the nozzle will be physically unable to reproduce these features. As result we opted for manual selection of 18 models that span wide variety of features, Figure 21 Each model is scaled to into a printing volume 18 x 18 SU and sliced at random locations.
18

---

Under review as a conference paper at ICLR 2022
Outline
Infill
Target Printout
Rewarded
Punished
Figure 20: The reward function.
Figure 21: Models in our curriculum. For full view of exemplar slices please see the supplementary material.
Our policy is represented
CNN modeled after Mnih al. (2015). The network imput is
84 x 84 x 3 image. The image is passed through three hidden layers. The convolution layers have the respective parameters: (32 filters, filter size 8, stride 4), (64 filters, filter size 4, stride 2), and (64 filters, filter size 3, stride 1). The final convolved image is linearized and passed through fully- connected layer with 512 neurons that connected the output action. Each hidden layer uses the nonlinear rectifier activation. We formulate our objective function as:
To, (atlsa) A (T,_,, (ar(st)
arg max EC
where is a timestep in the optimization, are the hyperparameters of neural network encoding our policy æ that generates an action a, based on a of observations st, A, is the estimator of the advan- tage function and the expectation EỘ is average of finite batch of samples generated by printing sliced models from our curriculum C. To maximize Equation ? we use PPO algorithm (Schulman al., (017). Each trajectory consists of a randomly selected mesh slice that fully printed out before proceeding to the next one. One epoch terminates when we collect 10000 observations. We run the algorithm for a total of 4 million observations but convergence was achieved well before that, Figure 22 For the training parameters we set the entropy coefficient to 0.01 and anneal it to- wards O. Similarly we anneal the learning rate from 3e-4 towards zero. Lastly, we picked a discount factor of 0.99 which corresponds to one action having a half time of 70 steps. This is equivalent to roughly 18.6 SU of distance traveled. our training set this corresponds 29-80 percent of the
total episode length.
Full Training
No Printing Bed
No Path
No Target
0.12
0.12
0.12
0.12
Reward
Reward
Reward
Reward
Iterations
4e
Iterations 4et
Iterations
4e
Iterations 4a*
Figure 22: Training curves for controllers with constant material flow.
19

---

Under review as a conference paper at ICLR 2022
We also experimented with training controllers for materials with varying viscosity, Figure 23 In general we have observed that the change in viscosity did not significantly affect the learning con- vergence. However, have observed drop in performance when training control policies for deposition of liquid materials. The liquid material requires longer time horizons to stabilize and has wider deposition making preeise tracing of fine features challenging.
0.12
0.12
0.12
Reward
Reward
Reward
Iterations
4e*
Iterations
Iterations
Viscosity
Figure 23: Training curves for controllers with increasing viscosity in an environment with noisy flow.
Lastly, we conducted ablation studies on action space and reward function in the environment with noisy deposition, Figure 24 We can see that employing the delayed reward had a negative effect on convergence and unclear a policy of sufficient quality would be achieved.
0.12 Velocity Only
0.12
Displacement Only 0.12 Delayed Reward 0.12 Immediate Reward
Reward
Reward
Reward
Reward
Iterations
2e
Iterations
2e
Iterations
2ea
0
Iterations
2
Figure 24: Training curves for controllers with variable material flow.
For evaluation we constructed separate dataset consisting of freeform and CAD geometries that not present in the training, Figure 25
Figure 25: Exemplar models from the evaluation dataset.
B BAYESIAN OPTIMIZATION FOR BASELINE CONTROL
While the baseline controller closely follows the printed boundaries is possible that there is more suitable policy to maximize our objective function. To verify this we use the environment described in Section search for velocity and offset that maximizes the reward function. More specifically we optimize a simplified objective of Equation limited to a single shape:
arg max E [TTa((a]s::).
(10)
v,d
where and d are the optimized velocity and displacement of the printing policy To.d, and E reduces to the expected cumulative reward of executing our proposed environment with a single slice. Maxi- mizing Equation [0 for single shape is challenging task due to the high cost associated with evaluating the objective function. Because of this we rely on Bayesian optimization to maximize the objective. We wtrm-start the optimization with 20 samples acquired through Latin sampling of
20

---

Under review as a conference paper at ICLR 2022
2-dimensional action space. We the optimization until convergence that define not improving upon the found maxima for over 300 iterations. We can see the optimized controllers for free-form bird model and a CAD model of bolt compared to optimized policy in Figure 26
Optimization
Optimization
Rewards: 10
5.3
8.5
8.9
17
10
Bayesian Optimization
Our Policy
Figure 26: Printouts realized using control policies recovered with Bayesian optimization (left and middle, blue square marks the optimized slice) compared to our trained policy (right).
C ADAPTATION TO VARYING VISCOSITY
We can observe that our policy learned on low-viscosity materials consistently under-deposits when used to print at higher viscosities. Conversely, our control policy learned on sigh-viscoiity material over-deposits when applied to materials with lower viscosities. From this observation we conclude that our policy learns the spread of the material post-deposition and uses this information to guide the deposition. Therefore, small viscosity variations are not likely to significat challenge for our learned poli- cies. However, if the learned material behavior is significantly violated the in-situ observation space limits the ability of our policy to adapt to before unseen material.
We evaluate how our learned controllers adapt to varying viscosity, (Figure 27
Viscosity
Baseline
Our Low Viscosity
Our Medium Viscosity
Our High Viszosily
Figure 27: We compare the baseline policy and our three learned policies on materials with varying viscosity.
21

---

Under review as a conference paper at ICLR 2022
D
DETAILED PHYSICAL RESULTS
Viecosity
Hight Viscosily
Buseline
Oura
Biseline
Ours
Basshine
Ours
Busalin
Oura
Figure 28: Policy evaluation on physical hardware.
22


Reviews:
[{"review_id": "vfJBvTMmjT", "reviewer": "Reviewer_wpL1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Listing pros and cons\n## Things I like about this paper:\n- **an interesting application**: I think this paper addresses a known problem in the unusual but yet interesting application of 3D printing. Cheaper 3D printers are often imprecise which limits their usability in practice. Finding ways to make them more robust and capable can unlock a lot of useful applications which is great. Solving this with sim-to-real is also an exciting approach.\n- **a thorough evaluation**: Studying the ability of the solution to work under different circumstances and providing a comprehensive ablation study is great. I commend the authors for the thorough evaluation of the proposed approach.\n- **exciting architectural choices**: Using camera feed for closing the loop seems like a great choice, defining both the state space and the reward for the RL agent using the camera feed makes it appealing and potentially applicable in practice too.\n\n## Things that can be improved:\n- **ambiguous prose**: I found the paper generally hard to read, each section heavily depends on the appendix and a lot of the gluing details were missing. Furthermore, there are a lot of ambiguities in the description too. More details can be found below. A direct way to fix this limitation is to provide additional clarifications.\n- **limiting assumptions**: The assumptions made in this work seem limiting at this point. The work evaluates on laying out only a single layer of a print assuming that there is a drastic difference in the pixel values between the printing bed and the depositing material but does not show the ability of the printer to complete an entire object's print. Currently, I doubt this is possible but yet, the paper claims to propose a fully functional self-correcting printer. If this is true, then perhaps toning down the claimed contribution and acknowledging the work's limitations can improve the clarity. Ideally, a working vision-based solution that is not limited to a single printed layer would be best. Perhaps this can be achieved by combining visual signal with the in-situ features mentioned in the paper and employed by prior work.\n- **questionable improvements on the physical task**: Although the paper reports on the average offset improvement of the proposed solution with respect to a baseline controller, the provided photos of the results from the physical experiment do not seem different to the baseline controller. The lack of adaptation of the solution and its direct applicability from simulation to the physical world also brings up questions regarding the usability on the full 3D printing task. More details and clearer illustration of the provided results can help alleviate this issue. \n\n# Claim and contribution\nThe overall paper claim seems to be a little too broad. The paper states that it introduces a first of its kind self-correcting printer but this is not true. Self-correcting printers is not that uncommon nowadays and works such as [1,2,3,4,5] seem to propose closed-loop control solutions too. In fact, [1] proposes a solution that actively controls the travel speed of printing by using an infra-red camera input, yet parallels were not drawn with any of these works. There also seem to be existing products that do similar things too - https://www.sciaky.com/additive-manufacturing/iriss-closed-loop-control. Perhaps, this can be further clarified\n\n# Methodology\nThis work proposes to use PPO trained in simulation as an adaptive policy to a baseline controller for 3D printing. There are however a number of ambiguous statements that I detail below. Providing more details and clarifying these will make the prose and contribution much clearer.\n## Observations\nmore details of what a heightmap is and how was the used segmentation obtained would be beneficial. Is the same RL policy used for both infilf and outline used. It seems like the images are assumed to be black and white and contain only 0s and 1s as values. Is this true, this seems limiting especially for applying multiple layers.\n## Transition function\n**using a model-free aproach**: Overall, I think it's great that there is so much information provided for the design and development of a reasonable transition function. However, given the amount of effort put to design a reasonable transition function, it is unclear why it is not being used in the learning process but instead the paper opts in for a model-free approach. Those approaches are generally useful in cases where the design and implementation of a reasonable transition function is impractical.\n\n**modelling the deposition imperfections**: This paragraph is not very clear. I could not understand what is the purpose of modelling these imperfections reported in Figure 6 if they are already coming from a simulated and therefore controlled environment. The use of an LPC filter seems unnecessary but yet confusing. In addition, it is unclear if this information is provided to the agent or it's used for the simulation only.\n## Reward function\nThe paper makes use of two types of rewarding- positive reward for depositing material inside the desired slice and a punishment term for depositing anywhere else. However, the work does not discuss how exactly are the rewarding and punishment terms combined together. The paper seems to make use of two different sets of rewards too: an infill and outline reward. However, it is currently unclear if the choice of using the infill or outline reward something that is manually assigned and whether this means that the same policy is trained using two different types of rewards? A discussion why this is ok/not ok would be beneficial. Furthermore, there is no additional information of what are the weights assigned on the outline reward. Calling the term W weights implies that those weights are also learnt, however there are no details associated with the specifics of how W is obtained or what values it takes all together. Finally, showing the resulted final reward, e.g. as a function of the rewarding and punishment terms can also be helpful to improve clarity.\n## The use of a curriculum\nAppendix F mentions the use of a curriculum of objects to train the RL agent. However, there were no details provided of the actual curriculum. Did the paper start training using simpler objects and gradually tried harder ones, or was there a different approach undertaken? More details can further clarify the training process undertaken.\n\n# Evaluation\n\nThe paper provides a very thorough evaluation which is great, however the details and presentation of the obtained results can be further improved. I also have additional questions related to the obtained results. To begin with, there is no formal definition of an average offset. The verbal description is ambiguous and given that this metric is not a conventional metric for evaluation in the learning community some additional details, including a formal definition of the metric will be useful. Right now it is not clear if this metric can detect negative performance, such as falsely obtained improvements - e.g. spilling deposited material towards the infill part of the object as opposed to outside the outlines. In other words, it is not clear how the authors measure negative performance.\n\nFigure 8 is the first time patterns are introduced in this work. However, it is not clear at all what the green lines show and why the left three plots are high gain and the right three plots are low gain. Details, including a more descriptive title will help understand this better. Broadly, the provided visual illustrations can be great to put things in perspective but are difficult source of measuring achieved performance when used on their own, especially for the provide ablations. Perhaps, reporting on the achieved average performance improvement in addition to the visual illustration can be useful. Right now, it is not very clear how much exactly 30microns of improvement is, for example, or how bad it is to get a little bit of spillage, as for example in Figure 12, last plot.\n\nThe description of the results obtained in Figure 12 are also ambiguous, what is it meant with 'On the other hand, modifying the printing path is slower and alone cannot handle sharp changes in material pressure'. How can this be seen from Figure 12, more details on this observation can be useful.\n\nSection 5.1 compares against a refined solution that uses Bayesian optimistion as a preprocessing step to the execution, but it is unclear why this is done, is this a standard procedure in the field, then additional referencing can be helpful. The paper states that 'We can observe that the two control schemes require drastically different velocities' but it is not clear where this can be seen and how exactly.\n\nPre-test and post-test were not introduced anywhere in this work, it is not clear what is meant by those terms. The means and standard deviations are also confusing, what do these represent? The paper uses a holm-bonferroni test but does not provide additional details to what the total number of pairs is and what were their p-values. This is another non-standard metric for evaluation, additional details as to why this decision was made and what exactly it is supposed to measure would be helpful. What was the reason not to opt for more conventional and simpler metrics, such as the total average offset improvement, for example?\n\nThe varying pressure experiment does not provide sufficient details of how this pressure was varied and why this is a useful experiment. Right now it is not clear if this experiment is representative of the physical-world variations that occur in 3D printing. Perhaps evaluating this on hardware can be more informative, is there a reason to not do this?\n\nFinally, the performance on the physical task shows some good results in terms of average offset improvement but those are not reflected in Figure 15. I find it hard to see why the proposed method is better than the baseline in this figure. However, it seems like this figure is the most important one from the whole paper as it showcases the applicability of the solution to physical tasks. Perhaps, highlighting and maybe further zooming in, as well as reporting the individual performance improvements as part of the figure can improve clarity.\n\n# Additional details on the limiting assumptions\nThis work evaluates the proposed approach on a single layer of a 3D printed model. However, it does not disclose what would happen if the whole object's printout was to be done. Even though, a baseline solution is not as accurate on a single layer of a print, it will keep doing the same thing and arguably do a good job at printing the entire object, e.g. when applying more layers but I am curious to learn what will happen with the rl agent as right now it is not clear if it will succeed.\n\n# Minor\n- The choice of using scene units instead of cm seems a bit unconventional. Instead, relying on universally used metrics will make the work more readable.  \n- Thingi10K is misspelled to Thingy10K. \n- The appendix should start on a new page and not directly after the references. \n- The paper states that the policy can adapt to different viscosities but it also seems to assume individual learning procedures for the different viscosities. Therefore, a single learnt model cannot be adapted to different viscosities, instead the solution can be trained for different viscosities seems like a more accurate description of what it does.\n- Fig. 4 description says left/right but should say first/second row\n- Not good practice to describe a result in the main body of the paper but to keep the figure illustrating it in appendix.\n\n# References\n[1] Farshidianfar, Mohammad H. et al.  \"Closed-loop control of microstructure and mechanical properties in additive manufacturing by directed energy deposition.\" _Materials Science and Engineering: A_ 803 (2021): 140483.\n\n[2] Cerón Viveros et al. \"Development of a closed-loop control system for the movements of the extruder and platform of a FDM 3D printing system.\" In _NIP & Digital Fabrication Conference_, vol. 2018, no. 1, pp. 176-181. Society for Imaging Science and Technology, 2018.\n\n[3] Chen et al. \"Data-Driven Adaptive Control for Laser-Based Additive Manufacturing with Automatic Controller Tuning.\" _Applied Sciences_ 10, no. 22 (2020): 7967.\n\n[4] Freeman et al. \"Beat the machine (learning): metal additive manufacturing and closed loop control.\" _Physics Education_ 55, no. 5 (2020): 055012.\n\n[5] Razaviarab et al. \"Smart additive manufacturing empowered by a closed-loop machine learning algorithm.\" In _Nano-, Bio-, Info-Tech Sensors and 3D Systems III_, vol. 10969, p. 109690H. International Society for Optics and Photonics, 2019.", "summary_of_the_paper": "This paper demonstrates the feasibility of obtaining a closed-loop control policy using reinforcement learning for additive manufacturing also known as 3D printing. The paper proposes a sim-to-real approach that relies on a novel simulated environment that allows for synthesising successful off-the-shelve RL policies capable of improving upon existing state-of-the-art print deposition controllers. An underlying assumption of this work is to rely on qualitative information only assuming that the difference in colour between the background and the applied deposition is sufficiently large. The work is evaluated in both simulation and sim-to-real and shows that the proposed method works well under different circumstances in simulation such as using static or dynamic depositing pressure variations and depositing material with different viscosity. In addition, the paper provides a comprehensive ablation study over the choice of observation and action spaces as well as the choice of reward. Finally, this work demonstrates the applicability of the approach on a sim-to-real task with no additional fine-tuning showcasing its ability to achieve a higher offset improvement than a baseline controller. I think that this work proposes a reasonable solution to an interesting problem that can potentially impact the 3D printing industry. Although there was not necessarily a novel contribution from a learning perspective, I think this work proposes a novel solution to a curious application and is therefore worthy of consideration. However, I have additional questions and concerns that prevent me from recommending this work for acceptance yet. I detail those below.", "summary_of_the_review": "In summary, I cannot recommend this paper for acceptance at its current state. Although the work introduces a potentially exciting solution to an interesting problem it has a number of ambiguous statements and lacks clarity on the otherwise extensive evaluation. In addition, some of the used assumptions in this work may prevent if from being actually applicable to full object 3D printing. Upon clearly and convincingly rebutting my review, I will be inclined to reconsider my recommendation.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Rx81IrZSt", "reviewer": "Reviewer_WUp8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a well-executed and evaluated system paper. I have learned a lot about the challenges of additive manufacturing from it. In addition, I found it very interesting that for such a complex system, end-to-end policies trained in simulation with off-the-shelf RL could work very well. \nAlong those lines, the main strengths of the paper are:\n1. Very clear description of the proposed system, both in terms of hardware and software. The paper is well written and clearly presents the challenges of additive manufacturing.\n2. An extensive evaluation of the sensorimotor controller both in simulation and in the real world. \n3. Numerous ablation studies to support the design decision.\n\nThat being said, I think that the paper's main weakness is its very fine-grained specialization to the task of additive manufacturing. What is now completely missing is a discussion on how the key technical components ( e.g., learning with RL from a simplified model, selecting the action space, etc.)  could benefit other tasks in the realm of robot manipulation. A (non-exhaustive) set of possible questions could be:\n1. Why is domain randomization not required to achieve sim-to-real transfer?\n2. What parts of the simulated model need to match the real system closely, and what can be inaccurate?\n3. What are the fundamental ingredients for transfer? \nIt would be extremely interesting to see these ideas generalized to other robots and tasks. In addition, some prior works observed that simple models are enough for training sensorimotor policies to control real robots (see for example [1] or [2]), so it would be nice to draw some connections with them as well. \n\nA few more detailed feedbacks:\n\n1. The related work section clearly shows that a lot of work has been done to improve additive manufacturing using traditional controller structures (like MPC) or end-to-end controllers. Why not include them in the experimental results? Having a quantitative comparison, at the very least in simulation, would strengthen the contribution.\n2. The numerous ablation studies never show a comparison with a baseline. I think including baselines would facilitate understanding the failure cases of the ablated version of the approach.\n3. The qualitative results are sometimes not very informative. For example, in Fig. 15, I have trouble understanding what works best. Maybe a better description focusing the attention of the reader in some specific parts could help.\n4. (Minor) It would be nice to clarify why the approaches have higher failure rates for low viscosity, particularly the baseline controller.\n\n\n\n[1] Deep Drone Acrobatics, Kaufmann, et al.\n[2] Learning Dexterous In-Hand Manipulation, Andrychowicz et al.\n\n\n\n \n\n\n", "summary_of_the_paper": "This paper proposes an approach to learn a closed-loop controller for additive manufacturing using reinforcement learning. The sensorimotor policy is trained exclusively in simulation and evaluated on a physical system in the real world without any fine-tuning with real-world data. To account for the difficulty to model the complex material deposition process, the approach builds a simplified simulator that can capture the process only qualitatively. Interestingly, even if only trained on the simple model, the approach generalizes to a real-world system.\nThe training methodology is described and evaluated in detail and compared against a baseline non-adaptive controller.", "summary_of_the_review": "Overall, I found this to be interesting in both its technical execution and presentation. However, at the moment, it reads more as a technical report than a scientific paper. Learning how to extend the proposed technical innovation to other systems or tasks could strengthen the contribution, in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6RumO89--pm", "reviewer": "Reviewer_3pgj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper was pretty convincing in showing that the proposed approach is effective. My main concerns are (1) the aspect of the paper most directly relevant to ICLR, policy learning, used an existing algorithm and (2) the explanation of the approach was not entirely easy for me to follow.\n\nI will readily admit that my lack of familiarity with additive manufacturing might be driving some of my confusion, but there were several parts in the modeling section of this paper that I found difficult to follow. In particular I found section 4.3 to be confusing and I think a few parts could be clarified there:\n- Several times in this paper it says that the deposition process only needs to be captured \"qualitatively\". What does this mean exactly? What is this in contrast to? I think this needs to be explained in more detail.\n- The discussion about time- vs. distance-based discretization was also confusing to me. I can understand some of the advantages of using distance-based discretization, but I find it hard to follow the part about gradient information vanishing and its relationship to Fig. 4. Can this be clarified?\n- Fig. 5 and the associated discussion were also not clear to me.", "summary_of_the_paper": "This paper presents a method for using RL for adjustment of process parameters in additive manufacturing. This allows for closed-loop control that outperforms the state-of-the-art in terms of printing quality. Multiple experiments showed this approach outperforming baselines, and it was also shown that the approach could be applied directly on physical hardware.", "summary_of_the_review": "Given that this paper is application-focused and light on algorithmic contributions, I think there is should be a higher burden for it to be clear and readable. At the moment, I think the discussion about modeling is not clear enough. For this reason, I will recommend rejection, but will remain open-minded to the author rebuttal and the opinion of other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the paper contains interesting experiments. However, as this paper is a systems paper without much algorithmic contributions, all reviewers felt that the paper felt short in terms of describing the results, has too many unsupported claims and it is unclear how the presented results transfer to slightly different domains. I therefore agree with the reviewers and recommend rejection of the paper.
Decision: Reject