CL-DiffPhyCon: Closed-loop Diffusion Control of Complex Physical Systems

Authors: Long Wei, Haodong Feng, Yuchen Yang, Ruiqi Feng, Peiyan Hu, Xiang Zheng, Tao Zhang, Dixia Fan, Tailin Wu
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: physical systems control, closed-loop control, PDE, physical simulation, generative models

Abstract:
The control problems of complex physical systems have broad applications in science and engineering.  Previous studies have shown that generative control methods based on diffusion models offer significant advantages for solving these problems. However, existing generative control approaches face challenges in both performance and efficiency when extended to the closed-loop setting, which is essential for effective control. In this paper, we propose an efficient Closed-Loop Diffusion method for Physical systems Control (CL-DiffPhyCon). By employing an asynchronous denoising framework for different physical time steps, CL-DiffPhyCon generates control signals conditioned on real-time feedback from the system with significantly reduced computational cost during sampling. Additionally, the control process could be further accelerated by incorporating fast sampling techniques, such as DDIM. We evaluate CL-DiffPhyCon on two tasks: 1D Burgers' equation control and 2D incompressible fluid control. The results demonstrate that CL-DiffPhyCon achieves superior control performance with significant improvements in sampling efficiency. The code can be found at https://github.com/AI4Science-WestlakeU/CL_DiffPhyCon.

Decision: accept (poster)
Reviewer scores: 8, 10, 3

Reviews and discussion:
<thread>
creation date:
1730485436829

thread_type:
official_review

<thread_object>
creator: reviewer_pd6k

<content>
summary:
The paper studies the problem of designing feedback control algorithms using diffusion techniques. Differently from open-loop control strategies, which compute a pre-determined control sequence that is applied to a system over time, closed-loop methods use information about the state of the system to tune the control inputs to compensate for deviations of the system trajectories from the predicted one, due to unmodelled dynamics or noise. The paper focuses on the computational complexity of the methods, and it proposes a technique to render closed-loop diffusion-based control computationally efficient.

strengths:
The overall objective of the paper is interesting and timely: to design closed-loop diffusion-based control algorithms. The issue of computational complexity is important, since diffusion-based methods may work well but require extensive computations, making them difficult to use in feedback settings.

weaknesses:
The paper is poorly presented, to an extent that makes the contributions unclear and difficult to appreciate. This start with the definition of the problem (where what is known/unknown/available to the designer is unclear, where there is confusion between system and environment, where   the stated problem is inherently open-loop and the feedback structure is not evident), and continues into the description of the proposed method (the synchronous and asynchronous parts are vague and difficult to grasp). Additionally, and importantly, the paper doesn't seem to offer performance guarantees for the proposed method, which makes it difficult to advocate for the proposed method in any practical use.

questions:
A careful rewriting of the manuscript seems necessary to clarify the problem setting and contribution, and to appreciate the differences between the propose method and the existing ones. A characterization of the performance of the proposed method is necessary, otherwise the  method takes the form of a heuristic that may not be desirable in controls applications.

As a side note, the problem described in equation (1) is inherently an open loop problem, since no structure is imposed on the input w. In fact, the set of open-loop sequences include the set of closed-loop sequences, but the converse may not be true. Since the system dynamics are deterministic, it is also unclear why using a closed-loop strategy would be necessary. Finally, the used notation (w for control inputs and u for the system state) is very unusual and, to the best of my knowledge, never been used in any controls paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730487986159

thread_type:
official_review

<thread_object>
creator: reviewer_sltc

<content>
summary:
The paper proposes a novel method for using diffusion policies for closed loop control that speeds up the generation of a control sequence of a length up to a chosen horizon by avoiding full denoising of the latter steps in the sequence, as they will likely not be executed by a receding horizon controller. The proposed method demonstrates superior performance in terms of achieving control objectives, due to closed loop control using a diffusion policy, with computation time much shorter than other diffusion policy baseline controllers.

strengths:
The main strength of the proposed method is the idea to save computation by not fully denoising the latter steps of the control sequence, with the understanding that in receding horizon control, the controls for those latter stages will likely not be used, as new state observations will appear in the meantime and the control sequence will be replanned. This idea addresses one of the main shortcomings of controllers based on diffusion policies - the very long process of denoising during sampling, which is currently a major obstacle to the wider adoption of diffusion policies that have shown considerable advantages with respect to more traditional behavior cloning schemes.

weaknesses:
I believe the proposed method is excellent and clearly applicable to very difficult control problems involving PDEs. However, most control problems are not of this kind. Why not show the performance on a simpler control problem with a handful of variables, hopefully not requiring high-end GPUs? This could potentially increase the applicability and appeal of the method significantly.

Some minor typos:

P. 3, L. 112: "It" -> "They"
P. 3, L. 140: "has" -> "have"
P. 7, L. 376: "well well" -> "perform well"?
P. 10, L. 539: "done" -> "drone"?
P. 19, L. 1009: "128 grids" -> "128 cells"

questions:
I am not quite sure why the proposed method achieves a better control objective than DiffPhyCon-1 - isn't the latter replanning at each control step by means of full denoising, thus as closed-loop as the proposed method? Can the authors clarify this? 

The planning horizon for the two control tasks is fixed. Why exactly to these lengths? What happens if full denoising is done over shorter horizons, replanning at each time step, using DiffPhyCon-1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576719080

thread_type:
official_review

<thread_object>
creator: reviewer_gnga

<content>
summary:
This manuscript addresses a closed-loop diffusion control problem of complex physical systems. The proposed CL-DiffPhyCon (Wei et al., 2024) extends the work of DiffPhyCon (Wei et al., 2024).

strengths:
One advantage is to improve the computation efficiency, which is achieved by an asynchronous denoising framework. This aspect is validated using numerical simulations. Overall, this manuscript is well-written and organized.

weaknesses:
There are some technical limitations, including noise-free inputs, input constraints, and numerical validations by using full in-domain control and observation.

questions:
Detailed comments are summarized below:

1. How restrictive is the consideration of noise-free control? There are many noise issues arising from complex engineering systems and applications. Can the proposed method be extended to account for plant, control, and measurement noise? 

2. Another important aspect associated with complex and practical physical systems is physical constraints. For example, physical control actions are implemented via actuators which often have upper and lower limits. That is one major reason model predictive control is useful. Can the proposed method handle physical constraints associated with control actions? 

3.  From Figure 2 and the limitation mentioned in the appendix, the algorithm is trained offline. In that case, it is not clear how the control is designed and employed in a real environment, as controllers are typically designed and used in practice in an interactive manner. In Figure 2, the ending time instance is T, can it be changed to T+1, as a bigger loop, to make it useful in an online/interactive manner? 

4. In both examples, the controls are defined as a spatiotemporal function in the PDE model, which is too ideal to be implemented in a realistic system. For PDE system control, typically, the control is implemented as a temporal function. The corresponding actuator is located in the boundary or a small part of the entire spatial domain. It would be interesting to see the results in these more realistic settings by using the proposed method. 

5. In the Burgers’ equation example, there is no difference between the full observation and partial observation cases, because the latter case only observes and controls two parts of the entire domain. In the case considered in the manuscript, one can simply treat each observed sub-domain as a total entire domain and apply the algorithm on the PDE with new spatial coordinates/ranges/boundary conditions. Partial observation typically refers to the case using partial observation to control the entire domain. The control performance results of the unobserved state should be added to fully demonstrate the effectiveness of the proposed algorithm. Similarly. practical applications often have sensors installed in the boundary of PDE systems. It would be good to also add results by using boundary observation data. 

6. The limitations of the proposed method should be moved to the conclusion section for better readability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732348242484

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank reviewers for their dedicated time and constructive comments. We are pleased that the reviewers found our work to be **novel** (Reviewer sLTC), **timely** (Reviewer pd6k), **well-written** (Reviewers GNga and sLTC), and **applicable to very difficult control problems** (Reviewer sLTC). Reviewers GNga and sLTC particularly pointed out that our proposed asynchronous denoising method **effectively addresses the computational efficiency challenge** in diffusion-based controllers, which is currently a major obstacle to the wider adoption of these methods. 

Based on the reviewers' insightful comments, we have conducted new experiments and made several clarifications to address their concerns. In this revised manuscript, we have updated both the main text and appendices, with changes highlighted in blue. The major improvements and new experiments are summarized as follows:
- **Experiments on system, measurement, and control noise:** We add experiments to analyze the effect of system and measurement noise on the 1D task, and control noise on the 2D task, as suggested by Reviewer GNga. The results are added to Table 1 and Table 2 in our revised manuscript. The results further confirm the robustness and superiority of our method. For more details, see the responses to Reviewer GNga.
- **Experiments on physical constraints:** We add experiments on the effect of physical constraints on control actions, as suggested by Reviewer GNga. Our method still shows robust and steady performance even with limited control action ranges. The results are added to Table 1 in our revised manuscript. For more details, see the responses to Reviewer GNga.
- **Experiments on more realistic scenarios with partial/boundary control and new double pendulum task:** We add experiments of more realistic scenarios where control actions are restricted to boundaries or a small part of the domain, as concerned by Reviewer GNga. These additional scenarios include 1D and 2D tasks, as well as a new double pendulum task, which is classically challenging yet computationally efficient, in response to Reviewer sLTC. The effectiveness of our method is confirmed in these scenarios. The results are added to Table 1, Table 2, and Appendix F in our revised manuscript. For more details, see the responses to Reviewer GNga and sLTC.
- **Clarification on contributions and differences:** We further emphasized our contributions and clarified the differences from existing methods in detail, in response to Reviewer pd6k. We make modifications accordingly in the final paragraph of Section 1, and Section 2. For more details, see the responses to Reviewer pd6k.
- **Clarified problem setup:** In response to Reviewer pd6k, we refined our problem setup and replaced the term "environment" with "system dynamics" for better clarity. These improvements are reflected in Section 3.1. For more details, see the responses to Reviewer pd6k.
- **Description of the proposed method:** We present clarification of the description of the proposed method, as concerned by Reviewer pd6k. In particular, we justify that our method is clear to follow, given a solid understanding of the prerequisite materials on SDEs provided in Section 3.2 and Appendices A and B of the original submission, which are consistent with conventions in the diffusion model literature. Additionally, we emphasize that our method is not heuristic, based on our established theoretical analysis. For more details, see the responses to Reviewer pd6k.
- **Analysis of the model horizon $H$:** We add the analysis of the effect of the model horizon $H$, as raised by Reviewer sLTC. We perform additional evaluations for $H=6$ and $H=10$, and provided practical guidelines for selecting $H$. The results and analysis are shown in Appendix E.3. For more details, see the responses to Reviewer sLTC.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel method for closed loop control using a diffusion policy which preferentially denoises early steps of the control sequence over later ones. The proposed method demonstrates superior performance in terms of achieving control objectives, with computation time much shorter than other diffusion policy baseline controllers.

The insight of the proposed method is simple (avoid denoising later steps of a controller since they are unlikely to be executed in a closed-loop system), but insightful. The result is an approach which is likely to be widely applicable to many control problems, as the authors demonstrate in several low and high-dimensional control settings. The proposed method achieves better performance results than prior work, while also having faster computation times.

The biggest weakness of the paper is in the presentation. The reviewers were split on the quality of the presentation, but in my own reading, the method *is* hard to follow. One reviewer also declined to provide a review at all because they could not understand the paper's contents despite reading it several times. Several variables are not defined, and Figure 2 (demonstrating the method) is not sufficiently self-encapsulating as an explanation on its own. For those already very familiar with diffusion models (and specifically how they might be used in a control setting), the math might be easier to follow, but it seems unlikely that a reader less well-versed in diffusion modeling (or control) could figure out how the method works without spending a week reading the paper in depth. In either case, the method is likely not described clearly enough for someone to reimplement it.

However, given that there were very positive reviews on the novelty and success of the method, I lean towards acceptance (poster). The authors are encouraged to take the concerns about presentation quality seriously for the camera-ready version however.