Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper demonstrates the feasibility of obtaining a closed-loop control policy using reinforcement learning for additive manufacturing also known as 3D printing. The paper proposes a sim-to-real approach that relies on a novel simulated environment that allows for synthesising successful off-the-shelve RL policies capable of improving upon existing state-of-the-art print deposition controllers. An underlying assumption of this work is to rely on qualitative information only assuming that the difference in colour between the background and the applied deposition is sufficiently large. The work is evaluated in both simulation and sim-to-real and shows that the proposed method works well under different circumstances in simulation such as using static or dynamic depositing pressure variations and depositing material with different viscosity. In addition, the paper provides a comprehensive ablation study over the choice of observation and action spaces as well as the choice of reward. Finally, this work demonstrates the applicability of the approach on a sim-to-real task with no additional fine-tuning showcasing its ability to achieve a higher offset improvement than a baseline controller. I think that this work proposes a reasonable solution to an interesting problem that can potentially impact the 3D printing industry. Although there was not necessarily a novel contribution from a learning perspective, I think this work proposes a novel solution to a curious application and is therefore worthy of consideration. However, I have additional questions and concerns that prevent me from recommending this work for acceptance yet. I detail those below.
main_review: # Listing pros and cons
## Things I like about this paper:
- **an interesting application**: I think this paper addresses a known problem in the unusual but yet interesting application of 3D printing. Cheaper 3D printers are often imprecise which limits their usability in practice. Finding ways to make them more robust and capable can unlock a lot of useful applications which is great. Solving this with sim-to-real is also an exciting approach.
- **a thorough evaluation**: Studying the ability of the solution to work under different circumstances and providing a comprehensive ablation study is great. I commend the authors for the thorough evaluation of the proposed approach.
- **exciting architectural choices**: Using camera feed for closing the loop seems like a great choice, defining both the state space and the reward for the RL agent using the camera feed makes it appealing and potentially applicable in practice too.

## Things that can be improved:
- **ambiguous prose**: I found the paper generally hard to read, each section heavily depends on the appendix and a lot of the gluing details were missing. Furthermore, there are a lot of ambiguities in the description too. More details can be found below. A direct way to fix this limitation is to provide additional clarifications.
- **limiting assumptions**: The assumptions made in this work seem limiting at this point. The work evaluates on laying out only a single layer of a print assuming that there is a drastic difference in the pixel values between the printing bed and the depositing material but does not show the ability of the printer to complete an entire object's print. Currently, I doubt this is possible but yet, the paper claims to propose a fully functional self-correcting printer. If this is true, then perhaps toning down the claimed contribution and acknowledging the work's limitations can improve the clarity. Ideally, a working vision-based solution that is not limited to a single printed layer would be best. Perhaps this can be achieved by combining visual signal with the in-situ features mentioned in the paper and employed by prior work.
- **questionable improvements on the physical task**: Although the paper reports on the average offset improvement of the proposed solution with respect to a baseline controller, the provided photos of the results from the physical experiment do not seem different to the baseline controller. The lack of adaptation of the solution and its direct applicability from simulation to the physical world also brings up questions regarding the usability on the full 3D printing task. More details and clearer illustration of the provided results can help alleviate this issue. 

# Claim and contribution
The overall paper claim seems to be a little too broad. The paper states that it introduces a first of its kind self-correcting printer but this is not true. Self-correcting printers is not that uncommon nowadays and works such as [1,2,3,4,5] seem to propose closed-loop control solutions too. In fact, [1] proposes a solution that actively controls the travel speed of printing by using an infra-red camera input, yet parallels were not drawn with any of these works. There also seem to be existing products that do similar things too - https://www.sciaky.com/additive-manufacturing/iriss-closed-loop-control. Perhaps, this can be further clarified

# Methodology
This work proposes to use PPO trained in simulation as an adaptive policy to a baseline controller for 3D printing. There are however a number of ambiguous statements that I detail below. Providing more details and clarifying these will make the prose and contribution much clearer.
## Observations
more details of what a heightmap is and how was the used segmentation obtained would be beneficial. Is the same RL policy used for both infilf and outline used. It seems like the images are assumed to be black and white and contain only 0s and 1s as values. Is this true, this seems limiting especially for applying multiple layers.
## Transition function
**using a model-free aproach**: Overall, I think it's great that there is so much information provided for the design and development of a reasonable transition function. However, given the amount of effort put to design a reasonable transition function, it is unclear why it is not being used in the learning process but instead the paper opts in for a model-free approach. Those approaches are generally useful in cases where the design and implementation of a reasonable transition function is impractical.

**modelling the deposition imperfections**: This paragraph is not very clear. I could not understand what is the purpose of modelling these imperfections reported in Figure 6 if they are already coming from a simulated and therefore controlled environment. The use of an LPC filter seems unnecessary but yet confusing. In addition, it is unclear if this information is provided to the agent or it's used for the simulation only.
## Reward function
The paper makes use of two types of rewarding- positive reward for depositing material inside the desired slice and a punishment term for depositing anywhere else. However, the work does not discuss how exactly are the rewarding and punishment terms combined together. The paper seems to make use of two different sets of rewards too: an infill and outline reward. However, it is currently unclear if the choice of using the infill or outline reward something that is manually assigned and whether this means that the same policy is trained using two different types of rewards? A discussion why this is ok/not ok would be beneficial. Furthermore, there is no additional information of what are the weights assigned on the outline reward. Calling the term W weights implies that those weights are also learnt, however there are no details associated with the specifics of how W is obtained or what values it takes all together. Finally, showing the resulted final reward, e.g. as a function of the rewarding and punishment terms can also be helpful to improve clarity.
## The use of a curriculum
Appendix F mentions the use of a curriculum of objects to train the RL agent. However, there were no details provided of the actual curriculum. Did the paper start training using simpler objects and gradually tried harder ones, or was there a different approach undertaken? More details can further clarify the training process undertaken.

# Evaluation

The paper provides a very thorough evaluation which is great, however the details and presentation of the obtained results can be further improved. I also have additional questions related to the obtained results. To begin with, there is no formal definition of an average offset. The verbal description is ambiguous and given that this metric is not a conventional metric for evaluation in the learning community some additional details, including a formal definition of the metric will be useful. Right now it is not clear if this metric can detect negative performance, such as falsely obtained improvements - e.g. spilling deposited material towards the infill part of the object as opposed to outside the outlines. In other words, it is not clear how the authors measure negative performance.

Figure 8 is the first time patterns are introduced in this work. However, it is not clear at all what the green lines show and why the left three plots are high gain and the right three plots are low gain. Details, including a more descriptive title will help understand this better. Broadly, the provided visual illustrations can be great to put things in perspective but are difficult source of measuring achieved performance when used on their own, especially for the provide ablations. Perhaps, reporting on the achieved average performance improvement in addition to the visual illustration can be useful. Right now, it is not very clear how much exactly 30microns of improvement is, for example, or how bad it is to get a little bit of spillage, as for example in Figure 12, last plot.

The description of the results obtained in Figure 12 are also ambiguous, what is it meant with 'On the other hand, modifying the printing path is slower and alone cannot handle sharp changes in material pressure'. How can this be seen from Figure 12, more details on this observation can be useful.

Section 5.1 compares against a refined solution that uses Bayesian optimistion as a preprocessing step to the execution, but it is unclear why this is done, is this a standard procedure in the field, then additional referencing can be helpful. The paper states that 'We can observe that the two control schemes require drastically different velocities' but it is not clear where this can be seen and how exactly.

Pre-test and post-test were not introduced anywhere in this work, it is not clear what is meant by those terms. The means and standard deviations are also confusing, what do these represent? The paper uses a holm-bonferroni test but does not provide additional details to what the total number of pairs is and what were their p-values. This is another non-standard metric for evaluation, additional details as to why this decision was made and what exactly it is supposed to measure would be helpful. What was the reason not to opt for more conventional and simpler metrics, such as the total average offset improvement, for example?

The varying pressure experiment does not provide sufficient details of how this pressure was varied and why this is a useful experiment. Right now it is not clear if this experiment is representative of the physical-world variations that occur in 3D printing. Perhaps evaluating this on hardware can be more informative, is there a reason to not do this?

Finally, the performance on the physical task shows some good results in terms of average offset improvement but those are not reflected in Figure 15. I find it hard to see why the proposed method is better than the baseline in this figure. However, it seems like this figure is the most important one from the whole paper as it showcases the applicability of the solution to physical tasks. Perhaps, highlighting and maybe further zooming in, as well as reporting the individual performance improvements as part of the figure can improve clarity.

# Additional details on the limiting assumptions
This work evaluates the proposed approach on a single layer of a 3D printed model. However, it does not disclose what would happen if the whole object's printout was to be done. Even though, a baseline solution is not as accurate on a single layer of a print, it will keep doing the same thing and arguably do a good job at printing the entire object, e.g. when applying more layers but I am curious to learn what will happen with the rl agent as right now it is not clear if it will succeed.

# Minor
- The choice of using scene units instead of cm seems a bit unconventional. Instead, relying on universally used metrics will make the work more readable.  
- Thingi10K is misspelled to Thingy10K. 
- The appendix should start on a new page and not directly after the references. 
- The paper states that the policy can adapt to different viscosities but it also seems to assume individual learning procedures for the different viscosities. Therefore, a single learnt model cannot be adapted to different viscosities, instead the solution can be trained for different viscosities seems like a more accurate description of what it does.
- Fig. 4 description says left/right but should say first/second row
- Not good practice to describe a result in the main body of the paper but to keep the figure illustrating it in appendix.

# References
[1] Farshidianfar, Mohammad H. et al.  "Closed-loop control of microstructure and mechanical properties in additive manufacturing by directed energy deposition." _Materials Science and Engineering: A_ 803 (2021): 140483.

[2] Cerón Viveros et al. "Development of a closed-loop control system for the movements of the extruder and platform of a FDM 3D printing system." In _NIP & Digital Fabrication Conference_, vol. 2018, no. 1, pp. 176-181. Society for Imaging Science and Technology, 2018.

[3] Chen et al. "Data-Driven Adaptive Control for Laser-Based Additive Manufacturing with Automatic Controller Tuning." _Applied Sciences_ 10, no. 22 (2020): 7967.

[4] Freeman et al. "Beat the machine (learning): metal additive manufacturing and closed loop control." _Physics Education_ 55, no. 5 (2020): 055012.

[5] Razaviarab et al. "Smart additive manufacturing empowered by a closed-loop machine learning algorithm." In _Nano-, Bio-, Info-Tech Sensors and 3D Systems III_, vol. 10969, p. 109690H. International Society for Optics and Photonics, 2019.
summary_of_the_review: In summary, I cannot recommend this paper for acceptance at its current state. Although the work introduces a potentially exciting solution to an interesting problem it has a number of ambiguous statements and lacks clarity on the otherwise extensive evaluation. In addition, some of the used assumptions in this work may prevent if from being actually applicable to full object 3D printing. Upon clearly and convincingly rebutting my review, I will be inclined to reconsider my recommendation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for using RL for adjustment of process parameters in additive manufacturing. This allows for closed-loop control that outperforms the state-of-the-art in terms of printing quality. Multiple experiments showed this approach outperforming baselines, and it was also shown that the approach could be applied directly on physical hardware.
main_review: Overall, this paper was pretty convincing in showing that the proposed approach is effective. My main concerns are (1) the aspect of the paper most directly relevant to ICLR, policy learning, used an existing algorithm and (2) the explanation of the approach was not entirely easy for me to follow.

I will readily admit that my lack of familiarity with additive manufacturing might be driving some of my confusion, but there were several parts in the modeling section of this paper that I found difficult to follow. In particular I found section 4.3 to be confusing and I think a few parts could be clarified there:
- Several times in this paper it says that the deposition process only needs to be captured "qualitatively". What does this mean exactly? What is this in contrast to? I think this needs to be explained in more detail.
- The discussion about time- vs. distance-based discretization was also confusing to me. I can understand some of the advantages of using distance-based discretization, but I find it hard to follow the part about gradient information vanishing and its relationship to Fig. 4. Can this be clarified?
- Fig. 5 and the associated discussion were also not clear to me.
summary_of_the_review: Given that this paper is application-focused and light on algorithmic contributions, I think there is should be a higher burden for it to be clear and readable. At the moment, I think the discussion about modeling is not clear enough. For this reason, I will recommend rejection, but will remain open-minded to the author rebuttal and the opinion of other reviewers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach to learn a closed-loop controller for additive manufacturing using reinforcement learning. The sensorimotor policy is trained exclusively in simulation and evaluated on a physical system in the real world without any fine-tuning with real-world data. To account for the difficulty to model the complex material deposition process, the approach builds a simplified simulator that can capture the process only qualitatively. Interestingly, even if only trained on the simple model, the approach generalizes to a real-world system.
The training methodology is described and evaluated in detail and compared against a baseline non-adaptive controller.
main_review: Overall, this is a well-executed and evaluated system paper. I have learned a lot about the challenges of additive manufacturing from it. In addition, I found it very interesting that for such a complex system, end-to-end policies trained in simulation with off-the-shelf RL could work very well. 
Along those lines, the main strengths of the paper are:
1. Very clear description of the proposed system, both in terms of hardware and software. The paper is well written and clearly presents the challenges of additive manufacturing.
2. An extensive evaluation of the sensorimotor controller both in simulation and in the real world. 
3. Numerous ablation studies to support the design decision.

That being said, I think that the paper's main weakness is its very fine-grained specialization to the task of additive manufacturing. What is now completely missing is a discussion on how the key technical components ( e.g., learning with RL from a simplified model, selecting the action space, etc.)  could benefit other tasks in the realm of robot manipulation. A (non-exhaustive) set of possible questions could be:
1. Why is domain randomization not required to achieve sim-to-real transfer?
2. What parts of the simulated model need to match the real system closely, and what can be inaccurate?
3. What are the fundamental ingredients for transfer? 
It would be extremely interesting to see these ideas generalized to other robots and tasks. In addition, some prior works observed that simple models are enough for training sensorimotor policies to control real robots (see for example [1] or [2]), so it would be nice to draw some connections with them as well. 

A few more detailed feedbacks:

1. The related work section clearly shows that a lot of work has been done to improve additive manufacturing using traditional controller structures (like MPC) or end-to-end controllers. Why not include them in the experimental results? Having a quantitative comparison, at the very least in simulation, would strengthen the contribution.
2. The numerous ablation studies never show a comparison with a baseline. I think including baselines would facilitate understanding the failure cases of the ablated version of the approach.
3. The qualitative results are sometimes not very informative. For example, in Fig. 15, I have trouble understanding what works best. Maybe a better description focusing the attention of the reader in some specific parts could help.
4. (Minor) It would be nice to clarify why the approaches have higher failure rates for low viscosity, particularly the baseline controller.



[1] Deep Drone Acrobatics, Kaufmann, et al.
[2] Learning Dexterous In-Hand Manipulation, Andrychowicz et al.



 



summary_of_the_review: Overall, I found this to be interesting in both its technical execution and presentation. However, at the moment, it reads more as a technical report than a scientific paper. Learning how to extend the proposed technical innovation to other systems or tasks could strengthen the contribution, in my opinion.
review_title: null

Meta-review: All reviewers agreed that the paper contains interesting experiments. However, as this paper is a systems paper without much algorithmic contributions, all reviewers felt that the paper felt short in terms of describing the results, has too many unsupported claims and it is unclear how the presented results transfer to slightly different domains. I therefore agree with the reviewers and recommend rejection of the paper.
Decision: Reject