Closed-Loop Control of Additive Manufacturing via Reinforcement Learning

1 INTRODUCTION . A critical component of manufacturing is identifying process parameters that consistently produce , high-quality structures . In commercial devices , this is typically achieved by expensive trial-and-error experimentation ( Gao et al. , 2015 ) . To make such an optimization feasible , a critical assumption is made : there exists a set of parameters for which the relationship between process parameters and process outcome is predictable . However , such an assumption does not hold in practice because all manufacturing processes are stochastic in nature . Specifically in additive manufacturing , variability in both materials and intrinsic process parameters can cause geometric errors leading to imprecision that can compromise the functional properties of the final prints . Therefore , transition to closed-loop control is indispensable for industrial adoption of additive manufacturing ( Wang et al. , 2020 ) . Recently , we have seen promising progress in learning policies for interaction with amorphous materials ( Li et al. , 2019b ; Zhang et al. , 2020 ) . Unfortunately , in the context of additive manufacturing , discovering effective control strategies is significantly more challenging . The deposition parameters have a non-linear coupling to the dynamic material properties . To assess the severity of deposition errors , we need to observe the material over long time horizons . Available simulators either lack predictive power ( Mozaffar et al. , 2018 ) or are too complex for learning ( Tang et al. , 2018 ; Yan et al. , 2018 ) . Moreover , learning on hardware is intractable as we require tens of thousands of printed samples . These challenges are further exaggerated by the limited perception of printing hardware , where typically , only a small in-situ view is available to assess the deposition quality . In this work , we propose the first closed-loop controller for additive manufacturing based on reinforcement learning deployed on real hardware . To achieve this we formulate a custom numerical model of the deposition process . Motivated by the limited hardware perception we make a key assumption : to learn closed-loop control it is sufficient to model the deposition only qualitatively . This allows us to replace physically accurate but prohibitively slow simulations with efficient approximations . To ameliorate the sim-to-real gap , we enhance the simulation with a data-driven noise distribution on the spread of the deposited material . We further show that careful selection of input and action space is necessary for hardware transfer . Lastly , we leverage the privileged information about the deposition process to formulate a reward function that encourages policies that account for material changes over long horizons . Thanks to the above advancements , our control policy can be trained exclusively in simulation with a minimal sim-to-real gap . We demonstrate that our policy outperforms baseline deposition methods in simulation and physical hardware with low or high viscosity materials . Furthermore , our numerical model can serve as an essential building block for future research in optimal material deposition , and we plan to make the source code available . 2 RELATED WORK . To identify process parameters for additive manufacturing , it is important to understand the complex interaction between a material and a deposition process . This is typically done through trial-anderror experimentation ( Kappes et al. , 2018 ; Wang et al. , 2018 ; Baturynska et al. , 2018 ) . Recently , optimal experiment design and , more specifically , Gaussian processes have become a tool for efficient use of the samples to understand the deposition problem ( Erps et al. , 2021 ) . However , even though Gaussian Processes model the deposition variance , they do not offer tools to adjust the deposition on-the-fly . Another approach to improve the printing process is to design closed-loop controllers . One of the first designs was proposed by Sitthi-Amorn et al . ( 2015 ) that monitors each layer deposited by a printing process to compute an adjustment layer . Liu et al . ( 2017 ) built upon the idea and trained a discriminator that can identify the type and magnitude of observed defects . A similar approach was proposed by Yao et al . ( 2018 ) that uses handcrafted features to identify when a print significantly drops in quality . The main disadvantage of these methods is that they rely on collecting the in-situ observations to propose one corrective step by adjusting the process parameters . However , this means that the prints continue with sub-optimal parameters , and it can take several layers to adjust the deposition . In contrast , our system runs in-process and reacts to the in-situ views immediately . This ensures high-quality deposition and adaptability to material changes . Recently machine learning techniques sparked a new interest in the design of adaptive control policies ( Mnih et al. , 2015 ) . A particularly successful approach for high-quality in-process control is to adopt the Model Predictive Control paradigm ( MPC ) ( Gu et al. , 2016 ; Silver et al. , 2017 ; Oh et al. , 2017 ; Srinivas et al. , 2018 ; Nagabandi et al. , 2018 ) . The control scheme of MPC relies on an observation of the current state and a short-horizon prediction of the future states . By manipulating the process parameters , we observe the changes in future predictions and can pick a future with desirable characteristics . Particularly useful is to utilize deep models to generate differentiable predictors that provide derivatives with respect to control changes ( de Avila Belbute-Peres et al. , 2018 ; Schenck & Fox , 2018 ; Toussaint et al. , 2018 ; Li et al. , 2019a ) . However , addressing the uncertainties of the deposition process with MPC is challenging . In a noisy environment , we can rely only on the expected prediction of the deposition . This leads to a conservative control policy that effectively executes the mean action . Moreover , reacting to material changes over time requires optimizing actions for long time horizons which is a known weakness of the MPC paradigm ( Garcia et al. , 1989 ) . As a result , MPC is not suitable for in-process control in noisy environments . Another option to derive control policies is to leverage deep reinforced learning ( Rajeswaran et al. , 2017 ; Liu & Hodgins , 2018 ; Peng et al. , 2018 ; Yu et al. , 2019 ; Lee et al. , 2019 ; Akkaya et al. , 2019 ) . The key challenge in the design of such controllers is formulating an efficient numerical model that captures the governing physical phenomena . As a consequence , it is most commonly applied to rigid body dynamics and rigid robots where such models are readily available ( Todorov et al. , 2012 ; Bender et al. , 2014 ; Coumans & Bai , 2016 ; Lee et al. , 2018 ) . In contrast , learning with non-rigid objects is significantly more challenging as the computation time for deformable materials is higher and relies on some prior knowledge on the task ( Clegg et al. , 2018 ; Elliott & Cakmak , 2018 ; Ma et al. , 2018 ; Wu et al. , 2019 ) . Recently Zhang et al . ( 2020 ) proposed a numerical model for training control policies where a rigid object interacts with amorphous materials . Similarly , in our work a rigid printing nozzle interacts with the fluid-like printing material . However , our model is specialized for the printing hardware and models not only the deposition but also its variance . We demonstrate that this is an important component in minimizing the sim-to-real gap and design control policies that are readily applicable to the physical hardware . 3 HARDWARE PRELIMINARIES . The choice of additive manufacturing technology constraints the subsequent numerical modeling . To keep the applicability of our developed system as wide as possible , we opted for a direct write Camera Material Nozzle needle deposition system mounted on a 3-axis Cartesian robot ( inset ) . The robot allows us to freely control the acceleration and position of the dispenser . The dispenser can process a wide range of viscous materials , and the deposition is very similar to fused deposition modeling . We further enhance the apparatus with two camera modules . The cameras lie on the opposite sides of the nozzle to allow our apparatus to perceive the location around the deposition . It is this locality of the in-situ view that we will leverage to formulate our numerical model . 3.1 BASELINE CONTROLLER Material Width Outline Path Target Infill Path Figure 1 : Baseline slicer . To control the printing apparatus , we employ a baseline slicer . The input to the slicer is a three-dimensional object . The output is a series of locations the printing head visits to reproduce the model as closely as possible . To generate a single slice of the object , we start by intersecting the 3D model with a Z-axis aligned plane ( please note that this does not affect the generalizability since the input can be arbitrarily rotated ) . The slice is represented by a polygon that marks the outline of the printout ( Figure 1 gray ) . To generate the printing path , we assume a constant width of deposition ( Figure 1 red ) that acts as a convolution on the printing path . The printing path ( Figure 1 blue ) is created by offsetting the print boundary by half the width of the material using the Clipper algorithm ( Johnson , 2015 ) . The infill pattern is generated by tracing a zig-zag line through the area of the print ( Figure 1 green ) . 4 REINFORCEMENT LEARNING FOR ADDITIVE MANUFACTURING . The baseline control strictly relies on a constant width of the material . To discover policies that can adapt to the in-situ observations , we formulate the search in a reinforcement learning framework . The control problem is described by a Markov decision process ( S , A , P , R ) , where S is the observation space , A is a continuous action space , P = P ( s′|s , a ) is the transition function , and R ( s , a ) → R is the reward function . To learn a control policy we take a model free approach by learning directly from printing . Unfortunately , learning on a physical device is challenging . The interaction between various process parameters can lead to deposition errors that require manual attention . As such discovering control policies directly on the hardware has too steep sample complexity to be practical . A potential solution is to learn the control behavior in simulation and transfer to the physical device . However , transfer from simulation to real world is a notoriously hard problem that hinges on applicability of the learned knowledge . In this work , we propose a framework for preparing numerical models for additive manufacturing that facilitate the sim-to-real transfer . Our model has three key components that facilitate the generalization of the learned control policies . The first component is the design of the observation space . To facilitate the transfer of learning between simulation and a physical device , we rely on an abstraction of the observation space ( Kaufmann et al. , 2020 ) . Rather than using the direct appearance feed from our camera module we process the signal into a heightmap . A heightmap is a 2D image where each pixel stores the height of the deposited material . For each height map location , the height is measured as a distance from the building plate to the deposited material . This allows our system to generalize to many different sensors such as cameras , depth sensors , or laser profilometers . However , unlike Kaufmann et al . ( 2020 ) , we do not extract the feature vectors manually . Instead , similarly to OpenAI et al . ( 2018 ) , we learn the features directly from the heightmap . In contrast to OpenAI et al . ( 2018 ) , we do not randomize the observation domain . Additional randomization is not necessary in our case thanks to the controlled observation conditions of the physical apparatus . A key insight of our approach is that the engineered observation space coupled with learned features can significantly help with policy learning . A careful design of the observation space can facilitate the sim-to-real transfer , make the hardware design more flexible by enabling the use of a range of sensors that compute similar observations , and remove the need to hand-craft the features . It is therefore worth wile to invest in the design of observation spaces . The second component of our system is the design of the action space . Instead of directly controlling the motors of the printer we rely on a high-level control scheme and tune coupled parameters such as velocity or offset from the printing path . This idea is similar in spirit to OpenAI et al . ( 2018 ) . OpenAI et al . ( 2018 ) suggest not using direct sensory inputs from the mechanical hand as observations due to their noisiness and lack of generalization across environments . Instead , they use image data to track the robotic hand . Similarly , but instead in action space , we do not control the printer by directly inputting the typically noisy and hardware-specific voltages that actuate the motors of the apparatus . Instead , we control the printer by setting the desired velocity and offset and letting the apparatus match them to the best of its capabilities . This translation layer allows us to utilize the controller on a broader range of devices without per-device training . This idea could also be generalized to other robotic tasks , for example , by applying a hierarchical divide and conquer approach to the action space . The control policies could output only high-level actions such as desired locations for robots actuators or deviations from a baseline behavior . Lowlevel controllers could then execute these higher-level actions . Such a control hierarchy can facilitate training by decoupling the higher-level goals from low-level inputs and transferring existing control policies to new devices through specialized low-level controllers . The third and last component of our system is an approximative transition function . Rather than modelling the deposition process exactly we propose to approximate it qualitatively . A qualitative approximation allows us to design an efficient computational model . To facilitate the transfer of the simulated model to the physical device we reintroduce the device uncertainty in a data-driven fashion . This is similar to OpenAI et al . ( 2018 ) , but instead of covering a large array of options , we specialize the randomization . Inspired by Chebotar et al . ( 2019 ) , we designed a data-driven LPC filter that matches the statistical distribution of variations observed during a typical printing process . This noise enables our control policies to adapt to changing environments and , to some extent , to changes in material properties such as viscosity . Our approximative transition function shows that it is not necessary to reproduce the physical world in simulation perfectly . A qualitative approximation is sufficient as long as we learn behavior patterns that translate to real-world experiences . This is an important observation for any task where we manipulate objects and elastic or frictional forces dominate the behavior . Relying on computationally more affordable simulations allows for applying existing learning algorithms to a broader range of problems where precise numerical modeling has prohibitive computational complexity . Moreover , by leveraging a numerical model it is possible to utilize privileged information that would be challenging if not impossible to collect in the real world . For full description of our methods please see Appendix A .

Summary: This paper proposes an approach to learn a closed-loop controller for additive manufacturing using reinforcement learning. The sensorimotor policy is trained exclusively in simulation and evaluated on a physical system in the real world without any fine-tuning with real-world data. To account for the difficulty to model the complex material deposition process, the approach builds a simplified simulator that can capture the process only qualitatively. Interestingly, even if only trained on the simple model, the approach generalizes to a real-world system. The training methodology is described and evaluated in detail and compared against a baseline non-adaptive controller.