Closed-Loop Control of Additive Manufacturing via Reinforcement Learning

1 INTRODUCTION . A critical component of manufacturing is identifying process parameters that consistently produce , high-quality structures . In commercial devices , this is typically achieved by expensive trial-and-error experimentation ( Gao et al. , 2015 ) . To make such an optimization feasible , a critical assumption is made : there exists a set of parameters for which the relationship between process parameters and process outcome is predictable . However , such an assumption does not hold in practice because all manufacturing processes are stochastic in nature . Specifically in additive manufacturing , variability in both materials and intrinsic process parameters can cause geometric errors leading to imprecision that can compromise the functional properties of the final prints . Therefore , transition to closed-loop control is indispensable for industrial adoption of additive manufacturing ( Wang et al. , 2020 ) . Recently , we have seen promising progress in learning policies for interaction with amorphous materials ( Li et al. , 2019b ; Zhang et al. , 2020 ) . Unfortunately , in the context of additive manufacturing , discovering effective control strategies is significantly more challenging . The deposition parameters have a non-linear coupling to the dynamic material properties . To assess the severity of deposition errors , we need to observe the material over long time horizons . Available simulators either lack predictive power ( Mozaffar et al. , 2018 ) or are too complex for learning ( Tang et al. , 2018 ; Yan et al. , 2018 ) . Moreover , learning on hardware is intractable as we require tens of thousands of printed samples . These challenges are further exaggerated by the limited perception of printing hardware , where typically , only a small in-situ view is available to assess the deposition quality . In this work , we propose the first closed-loop controller for additive manufacturing based on reinforcement learning deployed on real hardware . To achieve this we formulate a custom numerical model of the deposition process . Motivated by the limited hardware perception we make a key assumption : to learn closed-loop control it is sufficient to model the deposition only qualitatively . This allows us to replace physically accurate but prohibitively slow simulations with efficient approximations . To ameliorate the sim-to-real gap , we enhance the simulation with a data-driven noise distribution on the spread of the deposited material . We further show that careful selection of input and action space is necessary for hardware transfer . Lastly , we leverage the privileged information about the deposition process to formulate a reward function that encourages policies that account for material changes over long horizons . Thanks to the above advancements , our control policy can be trained exclusively in simulation with a minimal sim-to-real gap . We demonstrate that our policy outperforms baseline deposition methods in simulation and physical hardware with low or high viscosity materials . Furthermore , our numerical model can serve as an essential building block for future research in optimal material deposition , and we plan to make the source code available . 2 RELATED WORK . To identify process parameters for additive manufacturing , it is important to understand the complex interaction between a material and a deposition process . This is typically done through trial-anderror experimentation ( Kappes et al. , 2018 ; Wang et al. , 2018 ; Baturynska et al. , 2018 ) . Recently , optimal experiment design and , more specifically , Gaussian processes have become a tool for efficient use of the samples to understand the deposition problem ( Erps et al. , 2021 ) . However , even though Gaussian Processes model the deposition variance , they do not offer tools to adjust the deposition on-the-fly . Another approach to improve the printing process is to design closed-loop controllers . One of the first designs was proposed by Sitthi-Amorn et al . ( 2015 ) that monitors each layer deposited by a printing process to compute an adjustment layer . Liu et al . ( 2017 ) built upon the idea and trained a discriminator that can identify the type and magnitude of observed defects . A similar approach was proposed by Yao et al . ( 2018 ) that uses handcrafted features to identify when a print significantly drops in quality . The main disadvantage of these methods is that they rely on collecting the in-situ observations to propose one corrective step by adjusting the process parameters . However , this means that the prints continue with sub-optimal parameters , and it can take several layers to adjust the deposition . In contrast , our system runs in-process and reacts to the in-situ views immediately . This ensures high-quality deposition and adaptability to material changes . Recently machine learning techniques sparked a new interest in the design of adaptive control policies ( Mnih et al. , 2015 ) . A particularly successful approach for high-quality in-process control is to adopt the Model Predictive Control paradigm ( MPC ) ( Gu et al. , 2016 ; Silver et al. , 2017 ; Oh et al. , 2017 ; Srinivas et al. , 2018 ; Nagabandi et al. , 2018 ) . The control scheme of MPC relies on an observation of the current state and a short-horizon prediction of the future states . By manipulating the process parameters , we observe the changes in future predictions and can pick a future with desirable characteristics . Particularly useful is to utilize deep models to generate differentiable predictors that provide derivatives with respect to control changes ( de Avila Belbute-Peres et al. , 2018 ; Schenck & Fox , 2018 ; Toussaint et al. , 2018 ; Li et al. , 2019a ) . However , addressing the uncertainties of the deposition process with MPC is challenging . In a noisy environment , we can rely only on the expected prediction of the deposition . This leads to a conservative control policy that effectively executes the mean action . Moreover , reacting to material changes over time requires optimizing actions for long time horizons which is a known weakness of the MPC paradigm ( Garcia et al. , 1989 ) . As a result , MPC is not suitable for in-process control in noisy environments . Another option to derive control policies is to leverage deep reinforced learning ( Rajeswaran et al. , 2017 ; Liu & Hodgins , 2018 ; Peng et al. , 2018 ; Yu et al. , 2019 ; Lee et al. , 2019 ; Akkaya et al. , 2019 ) . The key challenge in the design of such controllers is formulating an efficient numerical model that captures the governing physical phenomena . As a consequence , it is most commonly applied to rigid body dynamics and rigid robots where such models are readily available ( Todorov et al. , 2012 ; Bender et al. , 2014 ; Coumans & Bai , 2016 ; Lee et al. , 2018 ) . In contrast , learning with non-rigid objects is significantly more challenging as the computation time for deformable materials is higher and relies on some prior knowledge on the task ( Clegg et al. , 2018 ; Elliott & Cakmak , 2018 ; Ma et al. , 2018 ; Wu et al. , 2019 ) . Recently Zhang et al . ( 2020 ) proposed a numerical model for training control policies where a rigid object interacts with amorphous materials . Similarly , in our work a rigid printing nozzle interacts with the fluid-like printing material . However , our model is specialized for the printing hardware and models not only the deposition but also its variance . We demonstrate that this is an important component in minimizing the sim-to-real gap and design control policies that are readily applicable to the physical hardware . 3 HARDWARE PRELIMINARIES . The choice of additive manufacturing technology constraints the subsequent numerical modeling . To keep the applicability of our developed system as wide as possible , we opted for a direct write Camera Material Nozzle needle deposition system mounted on a 3-axis Cartesian robot ( inset ) . The robot allows us to freely control the acceleration and position of the dispenser . The dispenser can process a wide range of viscous materials , and the deposition is very similar to fused deposition modeling . We further enhance the apparatus with two camera modules . The cameras lie on the opposite sides of the nozzle to allow our apparatus to perceive the location around the deposition . It is this locality of the in-situ view that we will leverage to formulate our numerical model . 3.1 BASELINE CONTROLLER Material Width Outline Path Target Infill Path Figure 1 : Baseline slicer . To control the printing apparatus , we employ a baseline slicer . The input to the slicer is a three-dimensional object . The output is a series of locations the printing head visits to reproduce the model as closely as possible . To generate a single slice of the object , we start by intersecting the 3D model with a Z-axis aligned plane ( please note that this does not affect the generalizability since the input can be arbitrarily rotated ) . The slice is represented by a polygon that marks the outline of the printout ( Figure 1 gray ) . To generate the printing path , we assume a constant width of deposition ( Figure 1 red ) that acts as a convolution on the printing path . The printing path ( Figure 1 blue ) is created by offsetting the print boundary by half the width of the material using the Clipper algorithm ( Johnson , 2015 ) . The infill pattern is generated by tracing a zig-zag line through the area of the print ( Figure 1 green ) . 4 REINFORCEMENT LEARNING FOR ADDITIVE MANUFACTURING . The baseline control strictly relies on a constant width of the material . To discover policies that can adapt to the in-situ observations , we formulate the search in a reinforcement learning framework . The control problem is described by a Markov decision process ( S , A , P , R ) , where S is the observation space , A is a continuous action space , P = P ( s′|s , a ) is the transition function , and R ( s , a ) → R is the reward function . To learn a control policy we take a model free approach by learning directly from printing . Unfortunately , learning on a physical device is challenging . The interaction between various process parameters can lead to deposition errors that require manual attention . As such discovering control policies directly on the hardware has too steep sample complexity to be practical . A potential solution is to learn the control behavior in simulation and transfer to the physical device . However , transfer from simulation to real world is a notoriously hard problem that hinges on applicability of the learned knowledge . In this work , we propose a framework for preparing numerical models for additive manufacturing that facilitate the sim-to-real transfer . Our model has three key components that facilitate the generalization of the learned control policies . The first component is the design of the observation space . To facilitate the transfer of learning between simulation and a physical device , we rely on an abstraction of the observation space ( Kaufmann et al. , 2020 ) . Rather than using the direct appearance feed from our camera module we process the signal into a heightmap . A heightmap is a 2D image where each pixel stores the height of the deposited material . For each height map location , the height is measured as a distance from the building plate to the deposited material . This allows our system to generalize to many different sensors such as cameras , depth sensors , or laser profilometers . However , unlike Kaufmann et al . ( 2020 ) , we do not extract the feature vectors manually . Instead , similarly to OpenAI et al . ( 2018 ) , we learn the features directly from the heightmap . In contrast to OpenAI et al . ( 2018 ) , we do not randomize the observation domain . Additional randomization is not necessary in our case thanks to the controlled observation conditions of the physical apparatus . A key insight of our approach is that the engineered observation space coupled with learned features can significantly help with policy learning . A careful design of the observation space can facilitate the sim-to-real transfer , make the hardware design more flexible by enabling the use of a range of sensors that compute similar observations , and remove the need to hand-craft the features . It is therefore worth wile to invest in the design of observation spaces . The second component of our system is the design of the action space . Instead of directly controlling the motors of the printer we rely on a high-level control scheme and tune coupled parameters such as velocity or offset from the printing path . This idea is similar in spirit to OpenAI et al . ( 2018 ) . OpenAI et al . ( 2018 ) suggest not using direct sensory inputs from the mechanical hand as observations due to their noisiness and lack of generalization across environments . Instead , they use image data to track the robotic hand . Similarly , but instead in action space , we do not control the printer by directly inputting the typically noisy and hardware-specific voltages that actuate the motors of the apparatus . Instead , we control the printer by setting the desired velocity and offset and letting the apparatus match them to the best of its capabilities . This translation layer allows us to utilize the controller on a broader range of devices without per-device training . This idea could also be generalized to other robotic tasks , for example , by applying a hierarchical divide and conquer approach to the action space . The control policies could output only high-level actions such as desired locations for robots actuators or deviations from a baseline behavior . Lowlevel controllers could then execute these higher-level actions . Such a control hierarchy can facilitate training by decoupling the higher-level goals from low-level inputs and transferring existing control policies to new devices through specialized low-level controllers . The third and last component of our system is an approximative transition function . Rather than modelling the deposition process exactly we propose to approximate it qualitatively . A qualitative approximation allows us to design an efficient computational model . To facilitate the transfer of the simulated model to the physical device we reintroduce the device uncertainty in a data-driven fashion . This is similar to OpenAI et al . ( 2018 ) , but instead of covering a large array of options , we specialize the randomization . Inspired by Chebotar et al . ( 2019 ) , we designed a data-driven LPC filter that matches the statistical distribution of variations observed during a typical printing process . This noise enables our control policies to adapt to changing environments and , to some extent , to changes in material properties such as viscosity . Our approximative transition function shows that it is not necessary to reproduce the physical world in simulation perfectly . A qualitative approximation is sufficient as long as we learn behavior patterns that translate to real-world experiences . This is an important observation for any task where we manipulate objects and elastic or frictional forces dominate the behavior . Relying on computationally more affordable simulations allows for applying existing learning algorithms to a broader range of problems where precise numerical modeling has prohibitive computational complexity . Moreover , by leveraging a numerical model it is possible to utilize privileged information that would be challenging if not impossible to collect in the real world . For full description of our methods please see Appendix A .

Summary: This paper demonstrates the feasibility of obtaining a closed-loop control policy using reinforcement learning for additive manufacturing also known as 3D printing. The paper proposes a sim-to-real approach that relies on a novel simulated environment that allows for synthesising successful off-the-shelve RL policies capable of improving upon existing state-of-the-art print deposition controllers. An underlying assumption of this work is to rely on qualitative information only assuming that the difference in colour between the background and the applied deposition is sufficiently large. The work is evaluated in both simulation and sim-to-real and shows that the proposed method works well under different circumstances in simulation such as using static or dynamic depositing pressure variations and depositing material with different viscosity. In addition, the paper provides a comprehensive ablation study over the choice of observation and action spaces as well as the choice of reward. Finally, this work demonstrates the applicability of the approach on a sim-to-real task with no additional fine-tuning showcasing its ability to achieve a higher offset improvement than a baseline controller. I think that this work proposes a reasonable solution to an interesting problem that can potentially impact the 3D printing industry. Although there was not necessarily a novel contribution from a learning perspective, I think this work proposes a novel solution to a curious application and is therefore worthy of consideration. However, I have additional questions and concerns that prevent me from recommending this work for acceptance yet. I detail those below.