DiffPhyCon: A Generative Approach to Control Complex Physical Systems
Authors: Long Wei
Peiyan Hu
Ruiqi Feng
Haodong Feng
Yixuan Du
Tao Zhang
Rui Wang
Yue Wang
Zhi-Ming Ma
Tailin Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Physical systems control
physical simulation
generative models
prior reweighting
Primary area: machine_learning_for_physical_sciences
TL;DR: We introduce a novel method for controlling complex physical systems using generative models, by minimizing the learned generative energy function and specified objective

Abstract: Controlling the evolution of complex physical systems is a fundamental task across science and engineering. 
Classical techniques suffer from limited applicability or huge computational costs. On the other hand, recent deep learning and reinforcement learning-based approaches often struggle to optimize long-term control sequences under the constraints of system dynamics. In this work, we introduce Diffusion Physical systems Control (DiffPhyCon), a new class of method to address the physical systems control problem. DiffPhyCon excels by simultaneously minimizing both the learned generative energy function and the predefined control objectives across the entire trajectory and control sequence. Thus, it can explore globally and plan near-optimal control sequences. Moreover, we enhance DiffPhyCon with prior reweighting, enabling the discovery of control sequences that significantly deviate from the training distribution. We test our method on three tasks: 1D Burgers' equation, 2D jellyfish movement control, and 2D high-dimensional smoke control, where our generated jellyfish dataset is released as a benchmark for complex physical system control research. Our method outperforms widely applied classical approaches and state-of-the-art deep learning and reinforcement learning methods. Notably, DiffPhyCon unveils an intriguing fast-close-slow-open pattern observed in the jellyfish, aligning with established findings in the field of fluid dynamics. The project website, jellyfish dataset, and code can be found at https://github.com/AI4Science-WestlakeU/diffphycon.

DiffPhyCon: A Generative Approach to Control Complex Physical Systems
Long Wei!*
Peiyan Hu
Haodong Feng' Rui Wang Yue Wang Zhi-Ming Ma
Ruiqi Feng
Yixuan Du Tailin Wulf
Tao Zhang'
School of Engineering, Westlake University. Academy of Mathematics and Systems Science, Chinese Academy of Sciences,
Jilin University, 4Fudan University, 'Microsoft A14Science
Iveilong, fengruiqi wutailin] Leliin@@@eeslake edu.
Abstract
Controlling the evolution of complex physical systems is a fundamental task across science and engineering. Classical techniques suffer from limited applicability or huge computational costs. On the other hand, recent deep learning and rein- forcement learning-basee approaches often struggle optimize long-term control sequences under the constraints of system dynamics. In this work, we introduce Diffusion Physical systems Control (DiffPhyCon), a new class of method to ad- dress the physical systems control problem. DiffPhyCon excels by simultaneously minimizing both the learned generative energy function and the predefined con- trol objectives across the entire trajectory and control sequence. Thus, it can explore globally and plan near-optimal control sequences. Moreover, we enhance DiffPhyCon with prior reweighting, enabling the discovery of control sequences that significantly deviate from the training distribution. We test our method on three tasks: ID Burgers' equation, 2D jellyfish movement control, and 2D high- dimensional smoke control, where our generated jellyfish dataset is released as benchmark for complex physical system control research. Our method outper- forms widely applied classical approaches and state-of-the-art deep learning and reinforcement learning methods. Notably, DiffPhyCon unveils intriguing fast- close-slow-open pattern observed in the jellyfish, aligning with established findings in the field of fluid dynamics. The project website, jellyfish dataset, and code be found at whttps://// ////ttttbbbb com/ Att./AAASSSieeeeee www.eeeeeeeeeeeeaalalaiaafaaapee
Introduction
Modeling the dynamics of complex physical systems is an important class of problems in science and engineering. Usually, we are not only interested in predicting a physical system's 's behavior but also injecting time- variant signals to steer evolution and optimize specific objectives. This gives rise to the complex physical control problem, fundamental task with wide applications, such controlled nuclear fusion [00. fluid control [22], underwater devices 77 and aviation [4), among others.
Despite its importance, controlling complex physical systems efficiently presents significant chal- lenges. It inherits the fundamental challenge of simulating complex physical systems, which are typically nigh-dimensional and highly nonlinear, as specified in Appendix A Furthermore, ob- served control signals and corresponding system trajectories for optimizing a control model are typically far from the optimal solutions of the specific control objective. This fact poses significant challenge of dilemma: How explore long-term control sequences beyond its training distribution
"Equal contribution. Work done an intern at Westlake University. 'Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to seek near-optimal solutions while making the resulting system faithful to the dynamics of the physical system?
To tackle physical systems control problems, various techniques have been proposed, yet they fall short of addressing the above challenges. Regarding traditional control methods, the Proportional- Integral-Derivatin (PID) control [33], is efficient but only suitable for a limited range of problems. Conversely, Model Predictive Control (MPC) [57]. despite wider range of applicability, suffers from high computational costs and challenges in global optimization. Recent advances in supervised
learning (SL) [22 24 and reinforcement learning (RL) [5 46 52 trained on system trajectories and control signals data, have demonstrated impressive performance in solving physical systems control problems. However, existing SL and model- based RL methods either fall into myopic failure modes (26 that fail to achieve long-term near near-optimal solutions, or produce ad adversarial state trajectories [72] that violate the physical system's 's dynamics. The main reason may be that they treat the continuous evolution of dynamics from an iterative view, both lacking long-term vision and struggling in global optimization. See Appendix A.2 for more related work on physical system control.
In this work, we introduce Diffusion Physical systems Control (DifiPhyCon), a new class of method to address the physical systems control problem. We take an energy optimization perspective over system trajectory and control sequences across the whole horizon to implicitly capture the constraints inherent in system dynamics. We accomplish this through diffusion models, which are trained using system trajectory data and control sequences. In the inference stage, DiffPhyCon integrates simulation and control optimization into a unified energy optimization process. This prevents the generated system dynamics from falling out of distribution, and offers an enhanced perspective over long-term dynamics, facilitating the discovery of control sequences that optimize the objectives.
An essential aspect of physical systems control lies in its capacity to generate near-optimal controls, even when they may deviate significantly from the training distribution. We address this challenge with the key insight that the learned generative energy landscape can be decomposed into two components: a prior distribution representing the control sequence and a conditional distribution characterizing the system trajectories given the control sequence. Based on this insight, we develop a prior reweighting technique to subtract the effect of the prior distribution of control sequences, with adjustable strength, from the overall joint generative energy landscape during inference.
In summary, contribute the following: (1) We develop DifiPhyCon, novel generative method to control complex physical systems. By optimizing trajectory and control sequences jointly in the entire horizon by diffusion models, Don facilitates global optimization of long-term dynamics and helps to reduce myopic failure modes. (2) We introduce the prior reweighting technique to plan control sequences that are superior to those in the training set. (3) We demonstrate the effectiveness of our method on ID Burgers' equation, 2D jellyfish movement control, and 2D high-dimensional smoke control tasks. On all three tasks, our method outperforms widely applied classical control methods and is also competitive with recent supervised learning and strong reinforcement learning baselines. particularly demonstrating advantages of DiftPhy in scenarios with partial observations and partial/indirect control. Notably, DiffPhyCon reveals the intriguing fast-close-slowopper pattern exhibited by the jellyfish, aligning with findings in the field of fluid dynamies (28] (4) We generate
dataset of 2D jellyfish based on a CFD (computational fluid dynamic) software to mimic the
movement of a jellyfish under control signals of its flapping behavior. To advance research in
controlling complex physical systems, we release our dataset as a benchmark.
2 Background
2.1 Problem Setup
We consider the following complex physical system control problem:
w" argmin J (u. w) s.t. Clu, w) = O.
Here u(t, x) : [0, T X Ω + Rdo is the system trajectory {u(t, ),, E [0. T1} defined on time range [0, 7] C R and spatial domain Q C RDD, and w(1,x) [0, T x x Ω + Rdw is the external control signal for the physical system with dimension dw- J(u, w) denotes the control objective. For example, I can be designed to measure the control performance towards a target state with cost constraints: J = ||u u" 2|| + |/||xxxxtt. Clu, w) = 0 denotes the physical
2

---

Training
Msp
Mi.0
trajectory Marr
diffuse
control seguence
denoise
W
Inference
Evaluation
(l.gPa(U.W) (reweighting) (y- 137 (w)
simalate
(guidance) A87(o.
Figure 1: Overview of DiftPhyCon The figure depicts the training (top), inference (bottom left), and evaluation (bottom right) of DiffPhyCon. Orange and blue colors respectively represent models learning the joint distribution pul w) and the prior distribution Po (ww).. Through prior reweighting and guidance, DiffPhyCon is capable of generating superior control sequences.
constraints. C(u, w) 0 can be specified either explicitly by PDE dynamic which describes how control signal w(1, x) drives the trajectory ul,, x) to evolve under boundary and initial conditions, or implicitly by control sequences and trajectory data collected from observation of the physical system. In the latter case, we may even have access to only partial trajectory data or engage in partial control. These situations collectively pose significant challenges to the physical system control task.
2.2 Preliminary: Diffusion Models
Diffusion models 19 are a class of generative models that learn data distribution from data. Diffusion
models consist of opposite processes: the forward process g(Xx+11 xx) - (xx+1; Vayxg, (1 aR) to corrupt clean data to Gaussian noise Xк ~ N (0, I), and the reverse parametrized A (x 1: ne (xx, k), rıII) denoise from standard Gaussian X5K N I). where {ax I km the variance schedule. To train diffusion models, [99] propose the DDPM method
process po (x--1] -
K
minimize the following training loss for the denoising network Cø, simplification the evidence
lower bound (ELBO) for the log- likeliooo of the data:
L
ee(va vI- āge, k)
where ag = I 0ig- Eo estimates the noise to removed recover data Xy. During inference, iterative application of from Gaussian noise could generate a new sample Xy that approximately follows the data distribution p(x). See Appendix A..3 for related work on diffusion models.
Notation. We use V(n.m [vn,".. Vm] to denote a sequence of variables. We use 2 to denote the hidden variable of 2 diffusion step k. For simplicity, abbreviate Wo,T--] UpiT] w, Concatenation of two variables is denoted via e.g., [u, w).
Method
introduce method including its training and inference. In Section 3.2 we further propose a prior reweighting technique to improve Don. The overview of DiftPhyCon is illustrated in Figure T
In this section, detail our method DiffPhyCon. In Section 3.1
3.1 Generative Control by Diffusion Models
DiftPhyCon takes an energy optimization perspective to solve the problem Eq. ( where PDE constraints can be modeled as a parameterized energy-based model (EBM) Eg (u, w, c) which characterizes the distribution p(u, w/c) of u and w conditioned on conditions e by the correspondence p(u, w/c) x exp(-Fx u, w,c)). Lower (u,u, w, w) or equivalently higher p(u, w/e), means better satisfaction of the PDE constraints. Then the problem Eq. ! can be converted to:

---

u w argmin [E, + [u, w)],
(3)
u,w
where is a hyperparameter. This formulation optimizes u and w of all physical time steps simultaneously. The first term encourages the generated and u to satisfy the PDE constraints C(u, w) = O. The second term guides optimization towards optimal objectives. The advantage of this optimization framework is that it tasks global optimization on and of all times steps, which may obtain better solutions that are faithful to the dynamics of the physical system. Details about the effect of the hyperparameter À are provided in Appendix [
Training. To train Eg, we exploit the diffusion model to estimate the gradient of Eg. For convenience, we introduce new variable z to represent the concatenation of u and as [u, w]. We use denoising network 6g [1)) which approximates E (z, c) 13 to learn the noise that should be denoised in each diffusion step l = 1,... K. The training loss of co is:
(4) where Eg is conditioned on c. Regarding training datasets, they could be generated by designing the conditions and control sequences followed by simulation when an explicit PDE form is available. Otherwise, they should be collected from observed pairs of control sequences and trajectories.
C = ER-
|||e ee + vI āgc, cxe,c,
Control optimization. After the denoising network is trained, the Eq. ( can be optimized by the Langevin sampling procedure as follows. We start from initial sample 2 NO, I), and iteratively run the following process
2 2 17 (Vz(Eo(Ze, + (T EE {~
(5)
where of and correspond to noise schedules and scaling factors used in the diffusion process, respectively. Here 2n is the approximate noise-free Zo estimated from Z₁ by:
2 (Z Âze((Ze c, k))//aa,
(6)
We calculate J in (5) based on 2n instead of directly using Zy because otherwise noise in zμ could bring errors J. Then V EEg can be replaced by our trained denoising network eq follows:
2k-1 2 7) (Eg(Zu. c, k) (VzJ(2z)) + E, & ~~J (O, I
Iteration of this denoising process for k = K,K - L,..., yields a final solution Zo = [M[1.7],0: W(o.T-1).0) for the optimization problem Eq. (3)
Guidance conditioning. In addition to the above introduced explicit guidance, conditioning is also widely used to guide sampling in diffusion models (0 (58) When the control objective can be naturally expressed in a conditioning form, e.g., the generated trajectory u is required to coincide with desired target u², can include u" condition in e such that the sampled trajectory u from diffusion models automatically satisfying u u²
Overall, in proposed DiffPhyCon framework, the control objective J can be optimized either using the explicit guidance V or guidance conditioning depending on the specific control objectives.
3.2 Prior Reweighting
Motivation. In physical systems control, a critical challenge lies in obtaining control sequences superior to those training datasets, which often deviate significantly from achieving the optimal control objective. Although guidance of the control objective is incorporated in our diffusion model, generated control sequences are still highly influenced by the prior distribution of control sequences in training datasets. This inspires to explore strategies to mitigate the effect of this prior, aiming to generate near-optimal control sequences.
We address this challenge with the key insight that the energy-based model E(u, w, c) can be decomposed into two components: one is (w, derived from the prior distribution p( ([w]c) of control sequences, and the other (t)((u, w, c) representing the conditional probability distribution p(u/w, c) of trajectories with respect given control sequences. This decomposition has the following form
(8) by the corresponding decomposition of distribution p(u, w/c) = (c/cc)/(u w, c). We propose prior reweighting technique, which introduces adjustable hyperparameter y > 0 as exponential
E(u, c, ((P) (u,)( + E(O) (u, w.

---

of p(w/c), allowing for the tuning of the influence of this prior distribution. Then we have reweighted version of p(u, w/c) as Py (u, w c) = p(w)c)"p(ulw, c)/Z, which is also a probability distribution and can be further transformed to
Py (u, w/c) = p(wlc)p p(u, w w/c//Z,
where Z is normalization constant. In particular, when 0 Y this approach is advantageous as it flattens the distribution p(u, w/c), thereby increasing the likelihood of sampling from low probability region of p(u, w/c), where the optimal solutions of the problem Eq. 3 probably lie.
Integrating Eq. 9 into Eq.
we have
g u, w. = - 1)E()) (w, c),c) (1-1)
+ Es(u, w. log
(10)
where EGO (u, w, c) = log (p, (u, w/e)) + const is the reweighted energy- based model associated
with (u, w, c) in Eq. can be transformed to
relying the hyperparameter yy Then the optimization problem Eq.
u w argmin u.w
E
E (uw.wor + ((
(11)
Optimization of this problem encourages sam- pling from the low likelihood region of p( (w/cc) while minimizing the control objective, which possesses the capability to generate control se- quences that are more likely to near-optimal than its degenerate version y 1 in the original optimization problem Eq. 3 The intuition of
of Parearay
prior reweighting is illustrated in Figure 2
Training. To learn the reweighted energy (u, w, c). we parameterize its gradient as a summation of two parts by taking the gradient of both sides of Eq. (IO):
Figure 2: Intuition of Prior Reweighting. The top surface illustrates the landscape of J(u, w). where the high-dimensional variables u and w are represented using one dimension. The middle and lower planes depict probability heatmaps for the reweighted distribution p (w )p(u/w)/2. Adjust- ing from 7 (middle plane) to < 7 (lower plane), better minimal of J (red dot in the lower plane) gains the chance to be sampled. This contrasts with the suboptimal red point in the middle plane highly influenced by the prior p(w).
V W. c) = (7 1)VE! (p)
(w.c) Eo w..,,
where Ede cw parameterizes the energy based model gl,((w,e corresponding to p(wcc). Note that Vogg Z vanishes here because is constant.
Notice that v Fa(u,, u, c) has already been trained by Eq. V [w, c) can be trained similarly by the following loss function
(1,K) (w). Eg(Va VI ane, k)ll)
(12)
where Eg is the conditional denoising network that approximates V VE (w,
Control optimization. With both eg and Ep trained, Eq. can be optimized by running:
- 24 77( 2g, c, k) + 2 Eτ, ~ (0,
(13)
Wk-1 Wk-1 n( 1) (ele(wees. (We, c, k) + & & ~((,00
(14)
iteratively, where 2 [ug, WA]. The difference between this iteration scheme and Eq. is that uses an additional step to update w based on the predicted noise of cg. This guides 2₁ [ug, w.] to move towards the reweighted distribution p- (u, w /) while aligning with the direction decrease the objective by its guidance in the iteration of z The complete algorithm is present Algorithm Detailed discussion and results about how
the hyperparameter
provided in Appendix
Theoretical Analysis. Consider pair (u, w) sampled using the prior reweighting technique with hyperparameter 7): [u, w] p.y (u, w) p(u(w)p' p" (w))/ Denote I as the global minimum
5

---

Algorithm Inference for DiffPhyCon
1: Require Diffusion models (Z,, c, k, and c (W. (W c, k). control objective I()), covariance matrix oiI, control conditions c, schedule ckg, hyperparameters X, 7, K
2: Initialize optimization variables Zk = [Kk wx] ~ (0,1
3: for ki K, do
4: àg (Zj; V1 (zk, c. k.)l/////
7x-1 7 -7(E0 (Z C, k) (Zx](z)) + f ~ (0, of)) transition to next diffusion step
6: WA-1 W - 17 () k) E2 (0, ofT) prior reweighting
end
8:
return u1,, w' = Zo
of the control objective J. Define ((s) to be the "c-optimal" solution set of (u, w) such that [u, w) J $ (c), whose complement set Q(a),, and denote ((((e) (u, w as its indicator function, i.c. Lo(e) (u, w) = if [u, w] E Q(s); otherwise O. Define Y be the random variable of "whether use J as guidance for sampling" namely,
(-((w,)) /7z
Y Y=1
pY
(15)
F(. /Z, Y 0.
Consider E(7) = () ( (u.w) (oote) (u, w) = 1], which indicates the expectation of getting an
e-optimal solution by using the prior reweighting technique with y under the guidance of J. Define
F(y) En.w To(e) w)p(Y- = 1/u, w) In(p(w))] w.. [T(ees (u, w)p(Y = 1/u, w) In(p(w))] Eu,w [l (u, w)p(Y 1|u, w)]
Wu,w (Q(e)) (u, w w)p((Y 1|u, w)]
then have the following theorem (please refer Appendix Theorem 3.1. Assume E(y) is a smooth function, then the following hold:
B for its proof):
F() O, there exists a 7- 1, s.t., E(7-) E(1):
F() O. there exists a 7+ s.t., E(++) > E()).
Remark: Here F(7) can be interpreted as some kind of difference between "entropies" in Q(s) and Q(e). When F(1) (1) < means that Q(e) has higher "entropies", implying that the training trajectories are far from optimal. As a result, we may need to flatten the distribution of training trajectories, which corresponds to using the prior reweighting technique with y < 1. Since this the most common case in real scenarios, we usually set 1.
The introduction of the prior reweighting technique DiffPhyCon involves training and evaluation of two models, thus bringing in additional computational cost. It is gratifying to note that we can balance the control performance and computational overhead of DiftPhyCon by adjusting the parameter ry. When y = 1, the model eg is not needed, and we denote this simplified version of DiffPhyCon as DiffPhyCon-lice
Experiments
In this section, we aim to answer the following questions: (1) Can DiffPhyCon present superiority over traditional, supervised learning. and reinforcement learning methods for physical systems control? (2) Does the proposed prior reweighting technique help achieve better control objectives? (3) Could answers to (I) and (2) be generalized to more challenging partial observation or partial control scenarios? To answer these questions, we conduct experiments three vital and challenging problems: ID Burgers' equation, 2D jellyfish movement control, and 2D smoke control problems.
The following state-of-the-arr control methods are selected as baselines. For the ID Burgers' equation, we use (1) the classical and widely used control algorithm
Cnnoortional-ntegga----eeee interacting with our trained surrogate model of the solver; (2) Supervised Learning method
(PID)
33
SSL. 24 RL methods including (3) Soft Actor-Critic (SAC) [16] with offline and surrogate-solver
versions; (4) Behaviour Cloning (BC) [4 and (5) Behavior Proximal Policy Optimization (BPPO)

---

Table Best Jactual achieved in 1D Burgers's equation control. Bold font denotes the best model, and underline denotes the second best model.
PO-FC
FO-PC
PO-PC
PID (surrogate-solver)
0.09115
0.09631
SL
0.09752
0.00078
0.02328
SAC (surrogate-solver)
0.01577
0.03426
0.02149
SAC (offline)
0.03201
0.04333
0.03328
BC
0.02836
0.00856
0,00952
ВPPO
0.02771
0.00852
0.00891
DiflPhyCon-lits (ours)
0.01139
0.00037
0.00494
DiflPhyCon (ours)
0.01103
0.00037
0.00494
10r
10
10"
(ourmgate- luffine)
101
10a
10a
107
DonPryConn lourst D Con rouns)
Without centrol
100
10.
10
10'1
10
10"1
10
10"
Jacta
Juzual
(40 Oosersatin,
Figure Pareto frontier of Jenergy vs. Jactual of different methods for 1D Burgers' equation.
73 Specifically, the surrogate-son solver version of SAC interacts with our trained surrogate model of the solver, while the offline version only uses given data. BC and BPPO are also in offline versions. For 2D jellyfish movement control, baselines include SI., SAC (offinee) SAC (surrogate-solver), BC, BPPO, and an additional classical multi-input multi-output algorithm Model Predictive Control (MPC) [57]. PID inapplicable to this data-driven task [2]. Detailed descriptions of baselines are provided in Appendix and Appendix !
4.1 1D Burgers' Equation Control
Experiment settings. The Burgers' equation is governing law occurring in various physical systems. We consider the 11) Burgers' equation with the Dirichlet boundary condition and external force w(t,,)), x). which is also studied in 24 42
+D 82 du uu Am O (t, ör
in [0, T
u(t, x) =0 u(0,zz) uo(x)
on 00, T| x an
(16)
in { = 0} x Ω.
Here the viscosity parameter, and (x(x) is the initial condition. Subject Eq. (16). given target state uz(x), the objective of control is to minimize the control error Juetual between and ta- while constraining the energy cost Jemergy of the control sequence w(t,, .)):
Jactual == J/1 |u(T,z) ?dx Jaworgy == - 10.T [t(t,
(17)
To make the evaluation challenging, we select three different experiment settings that correspond to different real-life scenarios: partial observation, full control (PO-FC), full observation, partial control (FO-PC), and partial observation, partial control (PO-PC), which are elaborated in Appendix D.2 and illustrated in Appendix C.T These settings are challenging for classical control methods such as PID since they require capturing the long-range dependencies in the system dynamics. Note that the reported metrics in different settings are not directly comparable. In this experiment, DiffPhyCon uses the guidance conditioning (Section 3.1) to optimize Jactual and explicit guidance optimize the energy cost, which elaborated in Appendix[D..
Results. In Table [ we report results of the control error Jnctual of different methods. It can be observed that Dif delivers the best results compared to all baselines. Specifically, Dif decreases Tneual of the best bascline by 30.1%, 52.6%, and 44.6% in the PO-FC, FO-PC, and PO-PC
7

---

Table 2: Jellyfish movement dataset outline.
training trajectories
test trajectories
resolution
#fluid features
trajectory length
30000
1000
128 x 128
3
40
DiffPhyCon and DiffPhyCon show little performance gap. This is because, the prior distribution of w that our DiffPhyCon-lie learned is conditioned both uo and uy, which fully determines the optimal w. Therefore, p(ul uw, ur) is intrinsically the optimal already deliver satisfactory performance.
settings respectively. From Table]] I
distribution and thus DiffPhyCon-lit
To compare the ability of different methods to optimize Jacuaa with constrained energy cost Tenergy- we compare the Pareto frontiers of different methods in F We vary the hyperparameter À to control the tradeoff between Jnctual and the energy cost since most baselines have this hyperparameter. As can be observed in Figure 3 the Pareto frontiers of DiftPhyCon are consistently among the best, achieving the lowest Tacuaa for most settings of the energy budget. Although SL performs well in full observation setting (b) where the system dynamics can be more easily predicted, it encounters difficulty in partial observation scenarios (a)().. The results demonstrate DiftPhyCon' 's ability to generate near-optimal control sequences compared to baselines. More visualization results provided in Appendix C.I More results of evaluation are presented in Appendix K.1 For efficiency evaluation of training and test phases, please refer to Table 23 in Appendix K.4
4.2 2D Jellyfish Movement Control
Experiment settings. This task is to control the movement of a flapping jellyfish with two wings in a 2D fluid field where fluid flows a constant speed. The jellyfish is propelled by the fluid when its wings flap. Its moving speed and efficiency are determined by the mode of flapping. This task is an important source of inspiration for the design of underwater and aerial devices, and its challenges come from complex vortice behavior and fluid-solid coupling dynamics [54] 28 For this task, fluid dynamics follows the 2D incompressible Navier-Stoker Equation:
dv
V. Vv V Vp=0
DF
Vv= (0,x) (x)
V
(18)
where v represents the 2D velocity of the fluid, and p represents the pressure, constituting the PDE state u = p). The initial velocity condition is (x(x) and the kinematic viscosity is v. We assume that each wing is rigid, so the jellyfish's boundary can be parameterized by the opening angle w, of wings. Long-term movement of jellyfish usually presents periodic flapping mode. Consequently, the control objective is maximize its average moving speed ū determined by the pressure of the fluid, under the energy cost constraint R(w) and the periodic constraint d(wr, Wo) of the movement:
I = - + < R(w) + d((r) Wo),
(19)
subject to Eq. (8) and the boundary condition that the velocity of fluid vanishes near the boundary. The hyperparameter { is set to be 1000. We evaluate in two settings: full observation, where the full state u p) is observed; and partial observation. where only pressure is observed. This task is very challenging due the complicated dynamics of fluid-solid interactions and vortices behaviour [54] 28) especially in the scenario of partial observation where the missing of v in Eq. (18) restricts the information available generate well-informed control signals. Details of the experiment provided in Appendix F
Open-source Dataset Description. We use the Lily-Pad simulator [66] to generate dataset describing the movement of jellyfish under the control of its flapping behavior. Statistics about the dataset are listed in Table The feature of this dataset is that it contains rich variation of vortices behavior and fluid-solid interaction dynamics, determined by violent changes in opening angles of wings and open-close phase ratio. It would serve as important benchmark for studying complex physical system control problems. Details about the dataset are provided in Appendix
Results. Evaluation results are presented in Table It can be seen that our method outperforms the baselines by large margin in optimizing the control objective J. cost of slightly increasing the control cost R( w). control sequences generated by method achieve much faster average speed
8

---

Table 3: 2D jellyfish movement control results. Bold font denotes the best model, and underline denotes the second best model.
Full observation
Partial observation
0
R(w) +
J 4
8
R(w)
J +
MPC SL
25.72
0.0112
109.17
-150.51
0.1791
329.59
-76.94
0.1286
205.57
-102.98
0.1188
221.79
SAC (surrogat--oolve))
166.96
0.0069
18.14
-13.009
0.0057
158.82
SAC (offline)
158.66
0,0069
165.58
-206.21
0.0058
211.96
BC
30.48
0.0629
32.44
20.08
0.0556
35.48
ВPPO
107.67
0.0867
-20.93
54.83
0.0518
-3.02
DiflPhyCon-lite (ours)
95.04
0.0746
-20.47
2.92
0.0779
74.97
DiffPhyCon (ours)
279.87
0.2058
-74.11
150.21
0.1269
-23.32
Can-PhyCnn-lite
DiflPhyCon
SAC (offline)
MPC
Figure 4: Comparison of generated control curves of three test jellyfish. The resulting control
objective for each curve is presented.
than baselines. In the full observation setting, the control objective achieved by DiffPhyCon -53.18 lower than the best baseline BPPO, while the average speed exhibits an increment of 172.2 over it. This demonstrates that diffusion models are effective for this challenging control task by performing global optimization of trajectory and control sequences in generative approach. Comparison between DiftPhyCon-lice and DiftPhyCon reveals that by flattening the prior distribution of control sequences, another significant improvement is further achieved. Even in the more challenging partial observation setting, DiffPhyCon still exhibits substantial advantages over existing methods. This reflects method has strong control capability under inadequate information. Configuration of the hyperparameter > in DiftPhyCon and performance with respect to varying is presented in Figure 17 in Appendix L.1 Details about the hyperparameter À can be found in Table 33 in Appendix L.2
Figure 4 visualizes generated opening angle curves of different methods on three test jellyfish. shape, which is proven to produce high speed in jellyfish movement [28]. The reason is that the fast closing of wings leads to thrust from fluid in the early state, resulting in a long-term high speed, and followed by a slow opening to reduce resistance. While this mode movement appears rarely in the training dataset, DiffPhyCon-Il
Opening angle curves of DiftPhyCon-ie show obvious cest-closeslow-opre
could generate such control sequences for most test samples. This reflects that diffusion models under guidance are effective in optimizing the control objective. Furthermore, DiffPhyCon makes this mode of movement more aggressive, with sharper change of the opening angle in the beginning and end stage within a period. This provides strong evidence that reweighting the prior distribution of control sequences allows for flexible sampling over flattened distribution, thus control sequences with low prior but good objectives are more likely to be sampled. The movement and the resulting fluid field of the jellyfish corresponding to the middle subtigure of Figure 4 controlled by DiftPhyCon is illustrated in Figure 55 and more examples are provided in Figure I0 in Appendix C.2 Conversely, opening angles obtained by basclines are inferior. The reason may be that they predict opening angles sequentially and hard to globally optimize the objective with three conflicting terms: average speed, R(w), and d( (Wr, wa). We further study such myopic failure mode of SAC in Appendix K.3] More comparisons about the variation of the weight are listed in Table 20 and Table 2 in Appendix K.2 For efficiency evaluation of training and test phases, please refer to Table 24 in Appendix K.4 We also extend this experiment to high-dimensional control signal setting, where the wings the jellyfish are assumed to soft. We find that our method is still competitive with basclines. Details about this evaluation are provided in Appendix M

---

20 t 10
15
Figure 5: Visualization of jellyfish movement and fluid field controlled by DiffPhy Con in the middle subfigure of Figure
t=0
t=6
t=12
[=18
L=
t=30
Figure 6: Visualization of smoke density and fluid field dynamics controlled by DiffPhyCon.
4.3 2D Smoke Indirect Control
Experiment settings. This task is to control smoke control in an incompressible fluid environment, following similar (but more challenging) setup of 22 Control forces were applied within 64 64 grid flow field, excluding a semi-enclosed region, to minimize the smoke failing to pass through the top middle exit (seven exits in total). For illustration of our settings, please refer Figure [4] in Appendix G This high-dimensional indirect control problem involves managing 2D forces at approximately 1,700 grid points every time step, resulting in about 100,000 control variables across 32 time steps, making it highly challenging.
Results. Evaluation results presented in Table Our method still has significant ad- vantages baselines in minimizing the denotes the second best model.
Table 4: 2D smoke movement control results. Bold font denotes the best model, and underline
trol objective. Furthermore, the prior reweight- ing technique achieves extra improvement over DiflPhyCon-lic. One test sample of smoke den- sity and fluid field dynamics is illustrated in Fig- ure d More visualization results of our method are presented in Figure Tiin Appendix G These results demonstrate that DiffPhy Con is capable of controlling high dimensional physical sys- tems even when control signals are indirectly applied to the system.
Method
BC
0.3085
BPPO
0.3066
SAC (surrogate-solver)
0.3212
SAC(ofllin)
0.6503
DifPPhCCn--liee (ours)
0.2324
DiffPhy (ours)
0.2254
Conclusion
In this work, we have introduced DiffPhyCon, novel methodology for controlling complex physical systems. It generates control sequences and state trajectories by jointly optimizing the generative energy and control objective. We further introduced prior reweighting to enable the discovery of control sequences that diverge significantly from training. Through comprehensive experiments. we demonstrated our method's superior performance compared to classical, deep learning, and reinforcement learning baselines in challenging physical systems control tasks. We discuss limitation and future work in Appendix N and state social impact in O
6 Acknowledgment
We thank Yuchen Yang for insightful discussions theoretical analysis. We thank the anonymous reviewers for providing valuable feedback on our manuscript. We also gratefully acknowledge the support of Westlake University Research Center for Industries of the Future and Westlake University Center for regorpaeffor Computing. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
10

---

References
[1] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
[2] Karl Johan Aström and Tore Hägglund. The future of pid control. In Digital Control Past, present, and future of PID Control, United States, 2000, Elsevier.
[3] Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations, 2023.
[4] Gerben Beintema, Alessandro Corbetta, Luca Biferale, and Federico Toschi. Controlling rayleigh-bénard convection via reinforcement learning. Journal of Turbulenee, 21(9-10):585 605, 2020.
[5] Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for PDE modeling. In The Eleventh International Conference Learning Representations, 2023.
[6] Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[7] Salva Rühling Cachay, Bo Zhao, Hailey James, and Rose Yu. Dyffussin: A dynamics-informed diffusion model for spatiotemporal forecasting, arXiv preprint arXiv:2306.0190 2023.
[8] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informet neural networks (pinns) for fluid mechanics: review. CoRR, abs/2105.09500 2021.
[9] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04133 2023.
[10] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):444- 419, 2022.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[12] Jie Ding, Min Wu, and Min Xiao. Nonlinear decoupling control with pi À d H neural network for mimo systems. IEEE Transactions Neural Networks and Learning Systems, pages 1-8, 2022.
[13] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jaseha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International Conference on Machine Learning, pages 1489-8510. PMLR, 2023.
[14] MA Elhawary. Deep reinforcement learning for active flow control around circular cylinder using unsteady-mode plasma actuators, arXiv preprint arXiv:2012.10165. 2020.
[15] masioud Farahmand, Salch Nabi, and Daniel N. Nikovski. Deep reinforcement learning for partial differential equation control. In 2017 American Control Conference (ACC), pages 31 3120-3127, 2017.
[16] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
11

---

[17] Elie Hachem, Hassan Ghraieb, Jonathan Viquerat, Aurélien Larcher, and P Meliga. Deep reinforcement learning for the control of conjugate heat transfer. Journal of Computational Physics, 436:1 10317, 2021.
[18] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, 36, 2024.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[20] Jonathan Ho and Tim Salimans. faasiffer-frre diffussion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[21] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. XXiv:22040034458 2022.
[22] Philipp Holl, Nils Thuerey, and Vladlen Koltun. Learning control pdes with differentiable physics. In International Conference on Learning Representations, 2020.
[23] Benjamin Holzschuh, Simona Vegetti, and Nils Thuerey. Solving inverse physics problems with score matching. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[24] Rakhoon Hwang, Jae Yong Lee, Jin Young Shin, and Hyung Ju Hwang. Solving pde-constr control problems using operator learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4504-4512, 2022.
[25] Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212. 1972, 2022.
[26] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162, pages 9902-9915, PMLR, 17-23 Jul 2022.
[27] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine Planning with diffusion for flexible behavior synthesis. Proceedings of Machine Learning Research, 162:9902-9915, 17-23 Jul 2022.
[28] Linlin Kang, An-Kang Gao, Fei Han, Weicheng Cui, and Xi-Yun Lu. Propulsive performance and vortex dynamics of jellyfish-like propulsion with burst-and-coas strategy. Physics of Fluids, 35(9), 2023.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690: 2014.
[30] Remi Lam, Alvaro Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 381(6677):1111-14211 2023.
Matthew Willson, Peter Wirnsberger, Meire Fortunato,
[31] Leon Lapidus and George F Pinder. Numerical solution of partial differential equations in science and engineering. John Wiley & Sons, 1999.
[32] A Larcher and E Hachem. A review on deep reinforcement learning for fluid mechanics: An update. Physics of Fluids, 34(11), 2022.
[33] Yun Li, Kiam Heong Ang, and G.C.Y. Y. Chong. Pid control system analysis and design. JEEE Control Systems Magazine, 26(1):32-44 2006.
[34] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadeneshelli Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
12

---

[35] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprini arXiv: (50.:./5999997 2015.
[36] Jacques Louis Lions. Optimal control of systems governed by partial differential equations, volume 170. Springer, 1971.
[37] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503-528, 1989.
[38] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439. Springer, 2022.
[39] Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. Fluid control using the adjoint method. ACM Transactions On Graphics (TOG), 23(3):449 456, 2004.
[40] Rajat Mittal and Gianluca Iaccarino. Immersed boundary methods. Annu. Rev. Fluid Mech., 37:239-261, 2005.
[41] Keith Morton and David Francis Mayers. Numerical solution of partial differential equations: an introduction. Cambridge university press, 2005.
[42] Saviz Mowlavi and Saleh Nabi. Optimal control of pdes using ihysics-informed neural networks Journal of Computational Physics, 473:111731, 2023.
[43] Alex Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. ArXiv, February 2021.
[44] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text text-guided diffusion models. Proceedings of Machine Learning Research, 162: 162:16784-16804, 17-23 Jul 2022.
[45] Guido Novati, Siddhartha Verma, Dmitry Alexeev, Diego Rossinelli, Wim M Van Rees, and Petros Koumoutsakos. Synchronisation through learning for two self ceff-propelled swimmers. Bioinspiration & biomimetics, 12(3):036001, 2017.
[46] Yangchen Pan, Amir-massoud Farahmand, Martha White, Salch Nabi, Piyush Grover, and Daniel Nikovski. Reinforcement learning with function-valued action spaces for partial differ- ential equation control. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3986-3995. PMLR, 10-15 Jul 2018
[47] Aditya A Paranjape, Jinyu Guan, Soon-Jo Chung, and Miroslav Krstic. Pde boundary control for flexible articulated wings on a robotic aircraft. IEEE Transactions on Robotics, 2953):62 2013.
[48] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonnzleez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.034099 2020.
[49] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
[50] Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Timo Ewalds, Andrew El-Kadi, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam, and Matthew Willson. Gencast: Diffusion- based ensemble forecasting for medium-rann raagee weather. arXiv preprint 27X75 2023.
[51] Bartosz Protas. Adjoint-based optimization of pde systems with alternative gradients. Journal of Computational Physics, 227(13):6490- 2008
[52] Jean Rabault, Miroslav Kuchta, Atle Jensen, Ulysse Réglade, and Nicolas Cerardi. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of fluid mechanics, 065:281-302. 2019.
13

---

[53] Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics-informet neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378:686-707, 2019.
[54] Leif Ristroph and Stephen Childress. Stable hovering of a jellyfish-like flying machine. Journal of The Royal Society Interface, 1 11 2014.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assise CittenttionMICCCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[56] Alvaro Sanzaez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks, In International conference on machine learning, pages 8459-8468. PMLR, 2020.
[57] Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology,
2021.
[58] Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for high- fidelity flow field reconstruction. Journal of Computational Physics,
2023.
[59] Sabrine Slama, Ayachi Errachdi, and Mohamed Benrejeb. Neural adaptive pid and neural indirect adaptive control switch controller for nonlinear mimo systems. Mathematical Problems in Engineering, 2019, 2019.
[60] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[61] Kiwon Um, Robert Brand, Yun Raymond Fei, Philipp Holl, and Nils Thuerey. Solver-in-the- loop: Learning from differentiable physics to interact with iterative pde-solvers. Advances in Neural Information Processing Systems, 33:6111-6122, 2020.
[62] Siddhartha Verma, Guido Novati, and Petros Koumoutsakos, Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proceedings of the National Academy of Sciences, 115(23):5849-5854, 2018.
[63] Pantelis R Vlachas, Georgios Arampatzis, Caroline Uhler, and Petros Koumoutsakos. Multi- scale simulations of complex systems by learning their effective dynamics. Nature Machine Intelligence, 4(4):359-336) 2022.
[64] Marin Vlastelica, Tatiana
Kelsey R Allen, Peter Battaglia, Arnaud Doucet, and Kim Stachenfeld. Diffusion generative inverse design. 2023.
[65] Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of para- metric partial differential equations with physics-informed deeponets. Science advances, 7(40))eabi8605, 2021.
[66] Gabriel D Weymouth, Lily pad: Towards real-time interactive computational fluid dynamics. arXiv preprint arXiv:1510006886 2015.
[67] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. In Sanmi Koyejo, S. Mohamed. A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, LA, USA, November 28 December 2022, 2022.
[68] Tailin Wu, Takashi Maruyama, Long Wei, Tao Zhang, Yilun Du, Gianluca laccarino, and Jure Leskovec. Compositional generative inverse design. arXiv preprint a 13X 2024.
[69] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. arXiv preprint 0rX1v:2403.03954, 2024.
14

---

[70] Shuang Zhang, Xinyu Qian, Zhijie Liu, Qing Li, and Guang Li. Pde modeling and tracking control for the flexible tail of autonomous robotic fish. IEEE Transactions on Systems, Man, and Cyberneties: Systems, 52(12):7618-7627 2022.
[71] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 01:3009-3623. 2022.
[72] Qingqing Zhao, David B Lindell, and Gordon Wetzstein. Learning solve PDE-constrained inverse problems with graph networks. Proceedings of Machine Learning Research, 162:26895- 26910, 17-23 Jul 2022.
[73] Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization, arXiv preprint arXiv:2332. //3/2 2023.
15

---

Appendix
Table of Contents
A Additional Related Work
17
A.1 Physical Systems Simulation
17
A.2 Physical Systems Control
17
A.3 Diffusion Models
18
B
Theoretical Analysis of Prior Reweighting
18
Visualization Results
C.I ID Burgers Equation Visualization
C.2 2D jellyfish control Visualization
................................
22
2D Smoke Control Visualization
22
D Additional Details for ID Burgers' Equation Control
25
D. Data Generation
25
D.2 Experimental Setting
25
D.3 Modell
26
D.4 Training and Evaluation
26
Jellyfish Movement Dataset
28
F
Additional Details Jellyfish Movement Control
29
F.1
Dataset Preparation
29
F2
Experimental Setting
................................
29
F3
Model]
30
F4
Training, Inference, and Evaluation
31
G
Additional Details for 2D Smoke Control
32
Dataset Preparation
32
Experimental Setting
34
Model
................................
34
Training, Inference, and Evaluation
34
ID Burgers' Equation Control Baselines
34
PID
................................
34
112
SAC
................................
35
H.3
Supervised Learning
................................
36
H.4
BC
37
11.5
BPPO
38
2D Jellyfish and Smoke Control Baselines
38
1.1
MPC and SL
38
1.2
SAC
................................
39
1.3
BC
40
BPPO
40
Surrogate Models
41
J.1
ID Surrogate Model
41
J.2
2D Jellyfish Force Models
41
J.3
2D Jellyfish Boundary Mask and Offsets Updater
42
16

---

J.4
2D Jellyfish Simulator
42
Additional Results of Experiments
43
Don ID Burgers' Equation
................................
43
K.2
2D Jellytish Movement
................................
43
K.3
Myopie failure modes SAC
44
K.
44
Efficiency Comparison
45
I.
Effect of Hyperparameters
45
L.1 Efffct of Y
45
Elfect
M Extensions to Fimer-grained Jellyfish Movement Control Task
48
M. Experimental Setting
48
M.2 Data Generation
48
M.3 Model
49
M.4 Training, Inference, and Evaluation
49
M.5 Results.
51
N Limitation and Future Work
51
0 Social Impact Statements
51
Additional Related Work
A.1 Physical Systems Simulation
Complex physical systems simulation forms the foundation of systems control. While classical numerical techniques for simulating physical systems renowned for their accuracy, they are often Recently, neural network-based solvers show significant advantage over classical solvers in accelerating simulations. They could be roughly divided into three primary classes: data-driven methods [34, 56, 4%
associated with significant computational expenses [4] 31
6 67
5
Physics-Informen 63 Most of them
30
Neural Networks (PINNs) [33 g 655, and solver-in-the- methods 61
iterative horizontal prediction framework. Instead, we treat the system trajectory as a whole variable and use diffusion models learn explicit simulator conditioned control sequences. A notable work is by (7]. which introduces diffusion models for temporal forecasting. While both our work and 7 employ diffusion models, we tackle different task of physical system control. Furthermore, we incorporate the control objective into the inference and introduce prior reweighting to tune the influence of the prior.
A.2 Physical Systems Control
For physical systems whose dynamics are described by PDEs, the adjoint methods [36] have been the most widely used approach for system control in the last decades. It is accurate but computationally expensive. Deep learning-based methods have emerged as powerful tool for modeling physical systems' dynamics. Supervised learning (SL. 22 24 trains parameterized models to directly optimize control using backpropagation through time over the entire trajectory. For example, [22] proposes hierarchical predietor-correcto scheme control complex nonlinear physical systems over long time frames. A more recent work proposed by (24) designs two stages which respectively learn the solution operator and search for optimal control. Different from these methods, we do not use the surrogate model, and learn both state trajectories and control sequences an integrated way. Reinforcement learning (RL) [15] 46 52)] treats control signals as actions and learns policies to make sequential decisions. Particularly in the field of fluid dynamics [32). reinforcement learning has been applied a multitude of specific problems including drag reduction [52) 14, conjugate heat transfer [4] [7 and swimming 45 62 But they implicitly consider physics information and sequentially make decisions. In contrast, we generalize the entire trajectories, which results in a global optimization with consideration of physical information learned by models.
17

---

Recently, PINNs are also incorporated in PDE control 422, but they require an explicit form of PDE dynamics, while our method data-driven and can deal with broader range of complex physical system control problems without explicit PDE dynamics.
A.3 Diffusion Models
Diffusion models [9 have significantly advanced in applications such as image and text generation
11 44]. inverse design 68 64]. inverse problem [23]. physical simulation 7 [50] and decision-
making 18) In particular, recent progress in robot control shows that diffusion models have
significant advantages over existing reinforcement learning methods for action planning 9
69
Generating diverse yet consistent samples poses challenge. For diversity, methods (38] 3
13
that integrate score estimates from various models have been effective. For consistency, guidance diffusion techniques [TT200) have been utilized generate condition-speet sppcciii samples. Our approach differs by flattening the joint distribution to achieve better control by slightly expanding beyond the prior distribution range.
B Theoretical Analysis of Prior Reweighting
Proof of Theorem 3.1
E(r) = / To(e) (u, w)p,(u, w/YY = 1)d(u, w)
/ Iqie)(u, w) p(Y 1/u, w)p, w) d(u, w)
p(Y =
(( Mlleelleeeeie (u, w)p(Y = 1/u, w)]
(u, (p(Y
uu, [.,
E (uw.:
Eu,w (L(c=) uu w)p(Y w)] Eu, ELu w)p(Y 1|u, w)]
1 2n.w PQwe w.]
E. ~ Lote 11, p w.
Define
Fu,w [T(ie (u, w)p(Y = 1/u, w)]
G(7)
Eu,w [owee (u, w)p(Y = 1/u, w)
Evp Em (To(e) (u, w)p(Y = w/ w] E En [ll(()) (u, w)p(Y 1|u, w]]
E. (u, w)p(Y = 1/u, w) (w)
J Eu Lole) (u, w)p(Y 1|u, (/l
Then
E(y) 1+ 1+000
Since G(y) > 0, E(y) and G(y) have the same monotonicity.
18

---

make the final state (7) f E. (o(e) (u, w)p(Y 1/u, w)/wlp"(ww In(p(w))d [T(e) (u, w)p(Y 1/u, w)] (Eu, [e(e) (u, w)p(Y 1/u, w)])
Slawel w)p(Y 1/u, w)] E. (Q(e) w)p(Y = 1 u, wwllpp((w)) (n)pp//(w)(((((((()))))) In(p(w))cc (E [ (u, w)p(Y = 1/u, w)])2
w (u, w)p(Y = i|u, w) In(p(w)) (T w)p(Y= = =1/,,w)
Bu. (u, w)p(Y 1/u, w)))?
Wu, (ut w)p(Y = 1|u, w) (ltee (u, w)p(Y = 1 w) In(((w))
(uu,([oeee (u, w w)p(Y = []),
By definition, F(7) is a positive multiple of (7)). which implics our conclusion:
If F1) < 0, then G'(1) < 0 and thus E'rr) decreases around 1. Hence, there exists 7- < 1,
s.t., B(T-) E(). thus holds;
otherwise, for similar reason, (ii) holds.
Remark: Here F(7) can be interpreted as some kind of difference between "entropies" in (()) and Q(=). When F(1 < means that Qe=) has higher "entropies", implying that the training trajectories are far from optimal. As result, may need to flatten the distribution of training trajectories, which corresponds to using the prior reweighting technique with < 1. Since this the most common case, usually set ry < 1.
C Visualization Results
C.1 ID Burgers' Equation Visualization
We present more visualization results of method and baselines under three settings: FOPC, POFC, and POPC in Figure Figure and Figure respectively. Under each setting, we present the results of five randomly selected samples from the dataset. The goal of control ur (7' = 10) close to the target state.
From these visualization results in Figure Figure g and Figure 9 it can be observed that under the control of our DiffPhyCon, the system state could converge smoothly to the target state given different initial states, and the final state uy always coincides with the target state. Furthermore. this observation is consistent under all three settings: FOPC, POFC, and POPC, implying that our DifiPhyCon is effective in addressing the partial observation and partial control challenges. In contrast, the baselines showed inferior results. Even the best baseline, BPPO, presents obvious mismatching with the target state
samples.
19

---

06
90
14
23 A
23
20
400
400
-12
23
-05
12
23 M
d6
34
15
23 12
34
400
+1.
12 C5
12
Figure 7: Visualizations results of ID Burgers' equation control under the FO-PC (full observa- tion, partial control) setting. The curve for the system state of each time step f = 0, ... 10 under control is plotted for our method DiifiPhConn) and baselines. The r-axis the spatial coordinate and the y-axis is the value of the system state.
20

---

X3 D4
83 24 23 08
2X
04 E4 03
02
suran 200rrinl
SMPONS, SIC
wrabbrghe 1.M.
Iedurge2
Fer Sempe 1.
2 BC
1rd sueple1
1.5
+1.0
Eedt Campr
Incearpie 2910
Lec creded 1940
Iod cample 490
122
for sende DiPlOr
sarplct Cef
Figure Visualizations results of 1D Burgers' equation control under the PO-FC (partial observation, full control) setting. The curve for the system state of each time step t = 0, ... 10 under control is plotted for our method (DifiPhyCon) and baselines. The r-axis is the spatial coordinate and the y-axis is the value of the system state.
21

---

Flburapeer
Riturmgen
22 10
22
22 34
E110
24
08 09
01 13
02 24
30
-10
-10
52
56
23
24
22
58
02 54
-10
52
on 50
12 21
01
21
12 24
64
02 08
02 24
-10
43
65
0
15
04
24 26 1.0
410
05
Li
Figure 9: Visualizations results of 1D Burgers' equation control under the PO-PC (partially observation, partially control) setting. The curve for the system state u, of each time step t= 0, 10 under control is plotted for our method (DifiPhyCon) and baselines. The r-axis the spatial coordinate and the y-axis is the value of the system state.
C.2 2D jellyfish control Visualization
We present more simulation results of our method in Figure 10 Each line represents an example from the test dataset. We plot five snapshots of boundary and fluid field for each example.
C.3 2D Smoke Control Visualization
We present fluid states and control signals generated by our method in Figure 1T Each line represents example from the test dataset. We plot six snapshots for each example.
22

---

t=
C
Figure 10: More examples of 2D jellyfish simulation controlled by our method.
23

---

Smoke denaty
Example J=0.0151
Control
Control
Smoke density
Control x
Example J=0,2746
Controly
Smoke density
Example J-0.3761
Control
Control
t=0
t=6
t=12
18
t=24
t=30
Figure 11: Examples of 2D smoke control results by method. We present three randomly selected test examples. For each example, we show the generated smoke density map and control force fields in horizontal and vertical directions. Each depicts six frames of movement. The smoke density in the first row corresponds to that in Figure 6
24

---

D Additional Details for 1D Burgers' Equation Control
D.I Data Generation
We use the finite difference method (called solver ground-truth solver in the following) to generate the training data for the ID Burgers' equation. Specifically, the initial value un(x) and the control sequence w(t, z) are both randomly generated, and then the states u(t, a) are numerically computed using the solver.
In the numerical simulation (using the ground-truth solver). domain of x E [0, 1]. E [0, is simulated. The space is discretized into 128 grids and time into 10000 steps. However, in the dataset, only 10 time stamps are stored. For the control sequence w, its refreshing rate is 0.1-1, i.e. x), & E [0. 0. + 1)], k E (O,.., 9} does not change with Therefore, the data size of each trajectory is (11, 128) for the state and [10, 1 128) for the control w.
In all settings, the initial value u(0, x) is a superposition of two Gaussian functions u(0,z) 42
E qee 20, 2),,2), 02 U(-2,0), 0), bj (0.2, 0.4), b2 (0.6, 0.8), 01 U00.05, 0.15), 02 (0.05, 0.15). Similarly, the control sequence w w(x, t) is also superposition of 8 Gaussian functions
where ai d;, oy are all randomly sampled from uniform distributions: 41 ~
w(b, Σ 0g6
2.
(20)
where each parameter is independently generated as follows: b1, ~ U(0, 1), b2 ~ U10, 1), Ti ~ U0.05, 0.2), σ2 (0.05, 0.2), while a U(-1.5, 1.5) and for i 2 2, U-11,, 1.5) or 0 with equal probabilities. xf,, (t + 0) is then numerically simulated (using the ground-trun -trtt solver) given u(0. and a(t, based on Eq. (16). The setting of the dataset generation is based on a previous work [24]. We generated 90000 trajectories for the training set and 50 for the testing set. Each trajectory takes up 32KB space and the size of the dataset sums up 2GB.
D.2 Experimental Setting
During inference, alongside the control sequence w(4, 2)), our diffusion model generates states xt,, and some models produce surrogate states μtt, x) when feeding the control w(t, x)) into the corresponding surrogate model. However, our reported evaluation metric Jacwal is always computed by feeding the control w(t, x) into the ground truth numerical solver to get Uga. (t, x) and computed following Eq. (7).. Followings are three different settings of our experiments.
D.2.1 Partial Observation, Full Control
In realistic scenarios, the system is often unable to be observed completely. Generally speaking, it is impractical to place sensors everywhere in system, so the ability of the model to learn from incomplete data is imperative. To evaluate this, we hide some parts of u in this setting and measure the Tnctual of model control.
Specifically, u(t, x), x E [ } is set to zero in the dataset during training and un (4), x E [1 is also set to zero during testing. In this partial observation setting ? = [1, 11 U 12 1]. Since no information in the central 1 space is ever known, the model does not know what will influence the control outcome of the unobserved states. Therefore, controlling the unobserved states is not reasonable task and they are excluded from the evaluation metric.
This setting is particularly challenging not only because of the uncertainty introduced by the unob- served states but also the generation of the control in the central locations that implicitly affect the controlled x E Ω.
D.2.2 Full Observation, Partial Control
This is another setting of practical relevance, where only fraction of the system can be controlled. The control sequence is enforced to be zero the central locations of x E []- 3) Ω still [0, 1). and J is evaluated on all of the observed states, though.
25

---

Some modifications to the dataset should be mentioned. The generation of the data involves first generating w before, followed by setting the central of w zero. To compensate for the decreased control intensity so that the magnitude of can be roughly comparable to the full control setting, we double the magnitude of w. During the evaluation, the output control sequence is also post-processed to be zero in x E (1
worth noting that in this setting, even when the control energy not limited all, is still challenging to find a perfect control since the model has to learn how to indirectly impose control on the central locations.
D.2.3 Partial Observation, Partial Control
The final setting is the combination of the previous two settings. Only Q [0, D 12 1] is observed, controlled and evaluated.
It is worth noting that some models require accessing the current state to produce output. If the model interacts with the ground truth solver instead of a surrogate model, then the result would be unfairly good since the information of the unobserved states is leaked through the interaction.
D.3
Model
Since the training of models Eg s V log p(w) and Eø s Vu,w lou w) are essentially the same and the latter model is exactly DiftPhyCon-lite, will introduce DifiPhyCon-lite first.
D.3.1 DifrPhyCon-lite
In general, DiftPhyCon follows the formulation of [19] which is also described in the main text. The data of u and is fed in images of size (Ne, N₂)) where N, is the number of time steps (11 and 10 respectively) and N₂ is the spatial grids (128). Since the two N&s for u and w are inconsistent, we zero-pad them into the size of 16. Then, and w are stacked as two channels and fed into the 2D DDPM model.
2D UNet Eg is used to learn predict It is structured into three main components: the downsampling encoder, the central module, and the upsampling decoder. The downsampling encoder is made up of four layers, each layer consisting of two ResNet blocks, one linear Attention block, and one downsampling convolution block. The central module also consists of two ResNet blocks and one linear Attention block. Each upsampling layer is the same as the downsampling layer except the downsampling block is replaced by the upsampling convolution block.
In experiments, we found that the control result is best when learning the conditional probability distribution of control w, step ug and uy input, and predicts the noise of u and w. Note that it is not trained to predict Ug and ur which are used as a condition, but there are still model outputs at the corresponding locations for the data shape consistency across different design choices of DiftPhyCon-lite. The hyperparameters in different settings are listed in Table
1], (1,T-1) un u.y. In summary, Eø takes the current trajectory u.
D.3.2 Don
In terms of implementation, DiffPhyCon is simply adding co (w) to ca(u, w) during inference as shown in Section 3.2 where is the output of the denoising network in DiftPhyCon-lit while es is a new denoising network that is trained to generate w following the dataset distribution. Therefore, we only describe the model of eg here.
Ey takes input of w, is as in the standard DDPM and ug. uy as guidance conditioning. The output of es (w) is of the same shape as w, so it can be treated as network learning to sample from p(w) = p(u, w w)du The output of eg (w) the locations of thus filled with zeros. The model hyperparameters are also listed in Table 5
D.4 Training and Evaluation
Training During training, the un and u, without noise are fed into the model and the model outputs at the corresponding locations are excluded from the loss. In the partial observation settings, the
26

---

Table Hyperparameters the UNet architecture and training for the results of ID Burgers'
equation in Table 1
Full observation Partial Control
Partial observation
Partial observation Partial Control
Hyperparameter name
Full Control
UNe (W]
Initial dimension
32
32
32
Downsampling/U/ee Aammpi/inn/nnnnngggggg...m layers
Convolation kernel size
Dimension multiplier
[1, 2, 4,
[1, 2, 4, 8]
[1, 2, 8]
Resnet block groups
Attention hidden dimension
32
32
32
Attention heads
UNe Es w)
Initial dimension
128
128
64
Downsampling/ Upsampliinnnnng layers
Convolution kernel size
3
Dimension multiplier
[,
[,,2,4,8]
[1, 2, 8]
Resnet block groups
8
Attention hidden dimension
32
32
32
Attention heads
4
4
Training
Training hatch size
16
16
16
Optimizer
Adam
Adam
Adam
Learning rate Training steps Learning rate scheduler
le-4
le-4
le-4
190000
170000
190000
cosine annealing
cosine annealing
cosine annealing
Inference
Sampling iterations
1000
1000
1000
Intensity of energy guidance J f //] daddt
Scheduler of energy guidance J = |[w] [
cosine
cosine
cosine
unobserved data is invisible to the model during both training and testing as introduced in Appendix
D.2 We simply pad zero in the corresponding locations of the model input and also exclude these locations in the training loss. Therefore, the model only learns the correlation between the observed states and control sequences. In the partial control setting, we train our DiffPhyCon on the dataset with control being zero in x E 1 7.1 In this way, the model naturally learns to output zero at the "non-controllablee" locations.
We use the MSE loss to train the denoising UNets and other training hyperparameters are listed in T
Inference During inference, ug and ur are set to the target ug and un so that the DDPM generates samples satisfying the physical constraint that is also conditioned on the target (u,) or the constraint (ug). In the partial observation setting, the ug and uy drawn from testing set are all filled zero at the unobserved locations E [ $1, which is the same the data used train the UNets. In the partial control scenarios,
During inference, we replace the denoising network's output (u, w) with (u, w) ( 1)e,(w) (w). is worth noting that co (u, w) denoises and w simultaneously while (w()) only denoises w. In our experiments, we found that adding a schedule to the output of the network is beneficial. The results in Table are generated with reverse Sigmoid schedule following
Eg (u We, k. ug. uz
+ (7
(Wk, k, up uy).
(21)
where 3 is defined as the noise schedule in 19). The inference DiffPhyCon-lite is simply setting y 1, which neglects the effect of the model (w).
When trying regulate the control energy, however,
not natural learn conditional
model. Therefore, we use the external guidance J = w(z, f)drdd to produce cost-limited control sequences that are shown in Figure 3 The gradient of the external guidance is computed and added to in Eq. (21). Note that we use the predicted clean sample We and új at the k-tth step to compute V since they suffer less from being noisy and leading to deviated guidance. ug and Wx
are computed following X x X4
eN where x represents u or w as in [9] In
VaL
V aa
27

---

our experiments, we use a cosine scheduling of the external guidance, and thus the final predicted noise would be E + Aag J where B3 the noise schedule in 19 with the cosine schedule.
Evaluation After generating the trajectory u and control sequence w, we feed the control sequence w into the ground-truth solver and simulate the final state u given the generated w and the initial condition un directly drawn from the testing dataset. The solver is the same as the one used in data generation in Appendix [.I] Finally, we compute Jactual following Eq. (17). In the partial observation setting, the MSE is computed only on the observed region, and in the control setting, the generated control will first be set zero in the uncontrolled region before being fed into the solver.
E Jellyfish Movement Dataset
We use the Lily-Pad simulator [66] to generate the Jellyfish Movement dataset, which serves as benchmark for physical system control research and also the dataset for our 2D evaluation task. Lily-Pad adopts the Immersed Boundary Method (IBM) [40] to simulate fluid-solid dynamics. The resolution of the 2D flow field is set be 128 x 128. The flow field assumed to be boundless in Lily-Pad. The head of the jellyfish is fixed at (25.6, 64). Its two wings are represented by two identical ellipses, where the ratio between the shorter axis and the longer axis 0.15, At each moment, the two wings are symmetric about the central horizontal line = 64. For each wing, we sample M- 20 points along the wing to represent the boundary of the wing. The opening angle of the wings is defined as the angle between the longer axis of the upper wing and the horizontal line. It acts as the control sequence w in 2D jellyfish control experiment.
Each trajectory starts from the largest opening angle and follows a cosine curve periodically with period T' = 200. Trajectories differ in initial angle, angle amplitude, and phase ratio (the ratio between the closing duration and whole pitching duration). For each trajectory, the initial angle Wo is generated as follows: first, sample random angle, called mean angle w(m)) E [20", ,40), then sample random angle amplitude w(*) E [10", min(w("), 60% w (m)) }} The initial Wo is set as Wo ((1))) + w(a), The phase ratio is randomly sampled from j0.2, 0.8]. The opening angle We of step decreases from w(m) w(4) w(m) w(() as L grows from 0 to rT'; then w, increases from w(mm) -w(a) to w(m)) w(a) as grows from -T' to T', Afterwards, w, varies periodically for T'. The range of w, is [w(m) w(a), w(m) + w(a]] C [10", 60%. For each trajectory, we simulate for 600 simulation steps, i.e., 3 periods. To save space, we only save the piece of trajectory from T' = 200 to 3T' = 600 steps with step size 10 because the simulation from = 0 to T' 200 is for initialization of the flow field. Then each trajectory is saved as T = (600- - 200) = 40 steps long sequence. An example of the simulated fluid field and the corresponding curve of opening
angles are shown in Figure 12
Besides the positions of the boundary points of wings and the opening angles w, also use another kind of image-like representation of the boundaries of wings as this representation contains spatial information that can be more effectively learned along with physical states (fluid field) by convolution neural networks. For each trajectory, this image-like boundary representation is compatible with physical states in shape. At each time step, boundaries of two wings are merged and then represented as tensor of shape [3, 64, 64], where it has three features for each grid cell: a binary mask indicating whether the cell is inside a boundary (denoted by 1) or in the fluid (denoted by 0), and a relative position (Ax, Ay) between the cell center to the closest point on the boundary. For each trajectory. we save system states, opening angles, boundary points, boundary masks and offsets, and force data. They are specified as:
system states U: shape [T,, 3, 64, 64). For each step. we save the states of the fluid field consisting of velocity in x and y directions and pressure. To save space, we downsample the resolution from 128 x 128 to 64 x 64.
- vvlooiity [T] 2, 64, 64].
pressure: [T, 1, 64, 64].
opening angles w: shape [ř]. For each step, we save the opening angle in radians.
boundary points: shape [7,2, 2). For each step, we save the boundary points on the upper and lower wings. Each wing consists of M 20 points and each point consists
28

---

30 20
Figure 12: Example of a flapping jellyfish in Lily-Pad simulator (a) and the corresponding curve of opening angle (b). The control signal is the opening angles w the wings of a jellyfish.
of coordinates. To make boundary points compatible with the downsampling of states, coordinates of x and y directions are shrunk to half (64/ 128) of the original values.
boundary mask and offsets b: [ 3, 64, 64). For each step, save the mask of merged wings with half coordinates of boundary points and offsets both x and y directions. The resolution is 64 x 64, compatible with that of the states.
-mask: [T. 1, 64, 64].
offsets: [7. 2, 64, 64].
force: shape [T: 2]. For each step, the simulator outputs the horizontal and vertical force from the fluid to the jellyfish. The horizontal force is regarded as a thrust to jellyfish if positive and a drag otherwise.
We generate 30, 000 training trajectories and 200 testing trajectories. Trajectories differ in the above specified parameters w()), (m) and т. Each trajectory occupies about 2MB of storage and the total dataset costs about 100GB.
Additional Details for 2D Jellyfish Movement Control
F1
Dataset Preparation
Based on our generated dataset in Appendix we prepare training samples for the 2D jellyfish movement control task as follows. We use sliding time windows that contain T = 20 successive sample, which corresponds to T 200 original simulation
time steps of states and boundaries steps and constitutes exactly a period wing movement. In this way, each trajectory can produce 20 samples. Therefore, we get 6 million training samples in total. In each training sample, the initial and the final time steps share the same opening angle due to periodicity, which serves as the conditions for control. For each test trajectory, select the opening angle of the jellyfish in the initial time and the initial states as the control condition for both the initial and final time and state initial condition.
F.2
Experimental Setting
F.2.1 Full Observation
In this setting, we assume all the states of the fluid field are observable. That is, both the velocity of and y directions and pressure are available in all the time steps of the training dataset and the initial time of the testing dataset.
F.2.2 Partial Observation
In this setting, we assume only partial states are observed. typical scenario in fluid simulation and control is that can only observe pressure data while the velocity data is not easy access. That is, only pressure is available in all the time steps
training samples and the initial time of the testing samples, hence the state tensor is of shape [T', 1, 64, 64). Notice that even only pressure is available, can still compute the force of fluid on the jellyfish and consequently the control objective because
29

---

force is fully determined by the shape of the jellyfish and pressure. The challenge of this partial observation setting that the velocity variable missing Eq. (T8). which makes the traditional numerical solver no longer applicable to solve this physical system control problem. However, this challenge could be well addressed by our method since could learn the relationship between control and pressure despite missing of the velocity data, and use the accessible control objective as guidance for flapping control.
F.3 Model
F.3.1
Architecture
We use a 3D U-Net as the backbone of our diffusion model, in both DiffPhyCon-lite and DiffPhyCon methods (detailed in the following subsection). In this paper, the architecture of the 3D U-Net we employed is inspired by (21) To better capture temporal conditional dependencies, we modify the previous space-only 3D convolution into space-time 3D convolut ion. Notably, we did not perform any scaling on the temporal dimension during downsampling or upsampling, Specifically, our U-Net consists of three main modules: the downsampling encoder, the middle module, and the upsampling decoder. The downsampling encoder is composed of three layers, each incorporating two residual modules, one spatial attention module, one temporal attention module, and one downsampling module. The middle module consists of two residual modules, one spatial attention module, and one temporal attention module. Mcanwhile, the upsampling decoder consists of four layers, each containing two residual modules, one spatial attention module, one temporal attention module, and one upsampling module. The input shape of U-Net is [batch size, frames, channels, height, width]. During convolution, the operation is performed on the [frames, height, width] dimensions. The output shape follows the same structure. Further details are provided in Table
Table 6: Hyperparameters of 3D-Unet architecture.
Hyperparameter name
Value
Kernel size of conv3d
(3, 3,3,3) 3)
Padding of conv3d
(1,1,1)
Stride of conv3d
(1,1,1)
Kernel size of downsampling
(1,4,4)
Padding of downsampling
(1,2,2)
Stride of downsampling
1,
Kernel size of upsampling
(1,4,4)
Padding of upsampling
(1,
Stride of upsampling
(0,
attention heads
4
F.3.2 DiniPhyCon-lite
The DiftPhyCon-lite method learns the denoising network of the joint distribution p(u, w/c) where u is physical states, w is the opening angle, and the conditions e consist of the initial angle wo. the initial state un and the final angle wr wn- We adopt the 3D U-Net the backbone. To make the opening angle (of shape [T]) align with physical states (of shape [T. 3, 64, 64) in full observation setting and [T,, 64, 64] in partial observation setting) in shape, we expand the opening angle to shape [T' 1, 64, 64] along spatial dimension by value copy. Besides, we also adopt the boundary mask and offsets representation, whose shape is [T", 3, 64, 64), determined by the opening angles as an auxiliary model input because they contain explicit spatial features, which makes model learning more effective. Then states, boundary mask and offsets, and expanded opening angle are stacked along the channel dimension and we get tensor of shape [T. 7, 64, 64] in full observation setting or [T, 5, 64, 64) in partial observation setting as the model input. The model output contains predicted noise of states and open angles. Thus its shape is [T, 4, 64, 64] in the full observation setting or
[T,2, 2, 64, 64) in the partial observation setting, where the last channel corresponds to the predicted
noise of opening angles and other channels correspond to predicted noise of states.
30

---

Marginal distribution (iw] Wa)
Ino 7
Prounge
Denoise (we)
Network &
Denoise under guidance
Joint distribution p(u,
Mporra Diarik (ouriAA
Denoise Network &g
Boundary updater
Wo. Do
Ppazjk-1
From inference step k to
Figure 13: Inference of our DiffPhyCon-lit and DiffPhyCon in the 2D experiment.
Table 7: Hyperparameters of network architecture and training for the 2D experiment.
Hyperparameter name
full observation
partial observation
Batch size
16
16
Optimizer
Adam
Adam
Initial learning rate
0.001
0.001
Loss function
MSE
MSE
F.3.3 DiffPhyCon
DiffPhyCon learns the denoising network of the joint distribution p(u, w/c) and the marginal distri- bution p(w/c). The denoising network of p(u, w/c) is exactly the same the introduced in the DifiPhCConllite method. The denoising network of p(w]c) also adopts the 3D U-Net architecture. Its input size is the same that of p(u, w w/c) in both full and partial observation settings. The difference is that the input states feature is replaced by the expansion of the initial state un along the time dimension by value copy. The output is the predicted noise of opening angles, whose shape is [T', 1, 64, 64], no matter the full observation setting or partial observation setting.
F,4 Training, Inference, and Evaluation
Training. We use the MSE (mean squared error) between model prediction and the Gaussian noise as the loss function Eq. 4 The batch size chosen
and the training involves 200,000 iterations. The learning rate starts from 10-3 and multiplies a factor of 0.1 at the 50000th and 155000th iterations. Training details are provided in Table The training is performed on two NVIDIA Tesla A100 GPUs with 80 GB memory for about days.
Both diffused variables Ufo.r] and W are initialized from Gaussian prior and gradually denoised from denoising step k = 1000 to & = 0 based on denoising networks and guidance. Because we introduce the boundary mask and offsets as auxiliary inputs, the model input and output are not consistent in shape. Thus we introduce surrogate model (shown as "Boundary updater" block in Figure 13) to update boundary mask and offsets b.o.T..u for each denoising k. Specifically, at each time step & [0. T), br,k is estimated by the initial boundary mask and offsets bp. and the difference of opening angle wt,k Wo from time step O to I, which is presented in the right part (after "Denoise under guidance") of Figure 13 Details about this surrogate model are presented J.3 Notice that although this surrogate model trained on noise-frr free data, we do not worry too much about its generalization to the noisy scalar w in
Inference. The pipeline of inference shown in Figure 13
31

---

inference because the estimated & does not deviate from the normalized range of noisy free w too much.
Our method introduces two kinds of inference: DiffPhyCon-lice and DiffPPhCCn. In Din-PhyCon-lite, only use the denoising network of the joint distribution p(u, w uu; Wa) for inference, while in DiftPhyCon, we use the additional denoising network of the marginal distribution p(w No. wo) together with that of the joint distribution for inference. These two branches are plotted in the left part of Figure 13 where the notation [uo] x T means expand initial state uo (of shape [3, 64, 64)) along time dimension by value copy to form a tensor of shape [T, 3, 64, 64).
As for guidance, we use a surrogate force model to approximate the force of fluid on jellyfish. This model is detailed in Subsection J.2] In denoising step k, its input consists of two parts: the first one is the noise-free state ([ooT] estimated from U[0.77, by Eq. (6;; the second one is the noise-free
boundary mask and offsets b10.21.* estimated from noise-free W1,.2 by the surrogate model to update boundaries, where W[0.7] is also estimated from Woo.T]. &. as in Eq. (6). The model output is force. Here we only use the horizontal force. Notice that the force could be computed via the surrogate force model no matter whether states are fully or partial observation in that force is irrelevant to the velocity of the fluid. The control objective J in Eq. (19) is computed as a summation of force and R( (o,rr) ). We fix Ç = 1000 as a default setting in Eq. (99) because this value can achieve a balance between scales of the average speed and the regularizer R(w). We also study the Pareto performance of varying ( in Table Then the gradients of the objective T in terms of újo.T and W[,,7 are computed and used in gurdance. For DiftPhyCon-lite these gradients are substracted from (.)..)) W(o.T].A] to generate [u [ W [o x-1] For DiffPhyCon, an additional term of noise (7 - 1)&s predicted from the denoising network of the marginal distribution p( ( Uoo Wo) should also be subtracted, shown in the upper left part of Figure 13 The effect of this term is controlled The inference is performed on one NVIDIA Tesla-V V100 GPU with 32 GB memory for about 3 hours for 50 testing samples.
by the scale of the hyperparameter y, as studied in Appendix L
Evaluation. The inference outputs opening angles Wio.T] of T 20 steps for 50 testing samples. In simulation, for each testing sample, the ground-truth first T = 20 steps of the opening angles (which corresponds to 200 simulation steps) are directly input to the Lily-Pad simulator for the reason of generating initial states of fluid, which is followed by the predicted control sequences of opening angles (interpolated to 200 steps of opening angles). The simulator outputs the horizontal force of fluid on the jellyfish for each simulation step. Finally, average speed U. energy cost R( R(w). and objective J are computed as metrics. The average speed = T fo r,dt as tia + 1 IT- F) 5, where vo is the initial speed and F; is the horizontal thrust from the fluid. The mass of the jellyfish is assumed
T
be 1. The energy cost term R(w) = 2
w.)? where the
control sequence w = (WI, Wr) represents the predicted opening angles. The periodic term d(wo, wr) max |T wol e, 0) is the constraint of periodic opening angles with small threshold e = 0.01.
G Additional Details for 2D Smoke Control
G.1 Dataset Preparation
We use the Phiflow solver [22] to generate the incompressible fluid with an indirect control dataset. The resolution of the 2D flow field is set to be 64 x 64. The flow field set to be boundless in Phiflow. We placed obstacles and absorbing areas in the middle of the fluid domain. In Figure we illustrate the locations of the exits, obstacles, and controllable areas. The specific positions of the obstacles and absorbing areas are detailed in Table 8 & Table
At the beginning of each trajectory, we set the horizontal velocity component v, of the entire flow field to zero and the vertical component 20 to an upward velocity of 1.0. Additionally, we randomly initialize a square smoke patch with dimensions of x 4 in the area below the horizontal obstacles, specifically within the coordinates (11:50, 11:14). In the trajectories we designed, the smoke is required to make four turns. We randomly determine the positions where the smoke will turn, and calculate the smoke's 's horizontal (v₂)) and vertical (vy) velocity components, assuming a constant magnitude of velocity. By adjusting the parameters, we observed that when the e, added at the turning points is sampled from a Gaussian distribution N (mu,,, m² u22 16) and vy is sampled from
32

---

64
Exit
Non-controllable Area
Target
(a) Locations of exits and obstacles
(b) Locations of controllable area
Figure 14: 2D smoke indirect control task. There are seven exits total and the top middle one the target exit (a). The control signals are only allowed to apply to peripheral regions (b). The control objective is to minimize the proportion of smoke failing to pass through the target exit.
Table Obstacle Positions
Table Absorb Area
Category
Position
Category
Absorb Area
Bottom
(8:56,8:9)
(0:8, 11:21) (0:8, 27:37)
Left
(8:9,8:12)
Left
(8:9,20:28)
(56:64, 11:21)
Right
(8:9,36:56)
(56:64, 27:37)
(56:57,8:12)
(11:21, 56:64)
Right
(56:57,20:28)
Up
(27:37, 56:64)
(56:57,36:56)
(43:53, 56:64)
(8:12,56:57)
(20:28,56:57)
Up
(36:44,56:57)
(52:56,56:57)
(24:25.32:40)
(24:25,48:56)
Inside Obstacles
(40:41.32:40)
(40:41,48:56)
(20:44, 20:21)
a Gaussian distribution (6 9o. /4), where is m sampled from a uniform distribution (2, 7), the success rate of the smoke navigating through is approximately 53.87% in the training set and 52.86% in the testing set, which meets our requirements. During the motion of the smoke, at turning points, the velocity
the central region the velocity in the surrounding regions is assigned by randomly sampling from the distributions (Vturrn 2 2/100). At non-turning points, velocity the central region remains the velocity from the previous moment, but the velocity in the surrounding areas is adjusted to the previous velocity plus noise sampled from (0, 0.01). This procedure ensures that each control parameter varies every moment and remains distinct from others at any given time. Such variability is beneficial for the subsequent training and generation of control parameters.
the velocity from the previous moment, while
We duplicated the density field into two versions: the original density field and the set-zero density field. The original density field used for model training and remains unaffected by the absorbing areas. In contrast, the set-zero density field is employed to calculate the amount of smoke passing through each bucket. When set-zero density present in the absorb areas, we sum up and record this density. Once the recording is complete, we reset the set-zero density in the absorb areas to zero to prevent the double-counting of emitted smoke. Ultimately, we document the quantity of smoke emitted from each bucket at every moment.
33

---

Our experimental data records the velocity and density of the entire flow field at each moment, as well as the velocity the peripheral flow field (control field) after noise addition. We generated 40,000 training trajectories and 500 test trajectories, with each trajectory approximately 10 MB in size. Given total of 40,500 trajectories, the entire dataset is estimated be around 400 GB in size.
G.2 Experimental Setting
In smoke control, we only consider the full observation and full control setting. Namely, all fluid features, including smoke density, horizontal and vertical velocities are observable; and green areas shown in Figure 14 are controllable.
G.3 Model
G.3.1 Architecture
Similar to 2D jellyfish control, we use 3D U-Net as the backbone of our diffusion model, both DiffPhyCon-lit lite and DiffPhyCon methods (detailed in the following subsections). Details of the architecture are same to Table
G.3.2 DinlPhCon-lite
Except for three fluid features, we use the accumulated proportion of smoke passing through the target exit as another feature. To make this feature (of shape [T), align with fluid features (of shape [T,3,64 64, 64)) in shape, we expand it to shape [T, 1, 64, 64) along spatial dimension by value copy. The horizontal and vertical control signals are represented by tensor of shape [T. 2, 64, 64), where uncontrollable positions are filled with zeros. Then these four state features and control signals are stacked along the channel dimension and we get tensor of shape [T', 6, 64, 64] the model input. The model output contains predicted noise of the state features and control signals. Thus its shape is also [T', 6, 64, 64].
G.3.3 DiffPhyCon
The denoising network of p(wlc) also adopts the 3D U-Net architecture. The input and output size is [T, 64).
G.4 Training, Inference, and Evaluation
Training. The batch size is chosen as 6. The number of iterations and learning rate schedule are same to 2D jellyfish control. The training performed on two NVIDIA Tesla A6000 GPUs with 48 GB memory for about 2 days.
Inference. The pipeline of inference is similar to 2D jellyfish control shown in Figure 13 The difference that we do not involve any surrogate model in this task. In each sampling step, we use the estimated proportion of smoke failing to exit through the target exit guidance. The inference is performed on one NVIDIA Tesla A6000 GPU with 48 GB memory for about hours for 50 testing samples.
Evaluation. The inference outputs the control force Wio,7 of T = 64 steps for 50 testing samples. In simulation, for each testing sample, the initial state and the generated control signals are input to the simulator. The simulator outputs the proportion of smoke failing to exit through the target exit for the final time step. Then the proportions are averaged over 50 test samples.
H 1D Burgers' Equation Control Baselines
H.1
PID
Proportional Integral Derivative (PID) control is versatile and effective control method widely used in various real-world control scenarios. It operates by utilizing the difference (error) between the desired target and the current state of system. PID control often considered the go-to option for many control problems due to its simplicity and usefulness. However, despite its popularity, PID
34

---

Up
Qa tanget state
Bri:
Kd.
ft
U4+1
Ugi state time
Ery timee
NN
[K Kig Kdg] PID parameters fo For at ciise
UL
Figure 15: The architecture of ANN-PID. To use MIMO PID controller to control U+ to Us. we train ceural-network--oed PID parameter planner output MIMO PID parameters based on Err.. then use the PID controller to output the control sequence fe.
control does encounter certain challenges, such as parameter adaptation and limitations when applied to Single Input Single Output (SISO) systems. In our specific context, the ID Burgers' Equation Control problem presents Multiple Input Multiple Output (MIMO) control scenario, which makes it infeasible to directly employ PID control to regulate the Burgers' equation. Inspired by the early works (59 [2] using neural network as a PID parameter adapter, we have integrated deep learning with PID control to tackle the MIMO control problem. As shown in Figure 15] ANN(artificial neural network) PID uses a neural network as a PID parameter adapter to output multiple sets of PID parameters and do multiple sets of SISO PID control.
The neural network to output PID parameters comprises two ID convolutional layers, fully con- nected layers, and 4 corresponding activation layers. We use the L1 loss of the current state and target state as training loss and the Adam optimizer (2 train the model. Detailed information can be found in Table 10
Table 10: Hyperparameters of network architecture and training for ANN-PID.
Hyperparameter name
Full observation
Partial observation
Kernel size of conv Id
3
3
Padding of cony ld
1
1
Stride conv ld
1
1
Activation function
Softsign
Softsign
Batch size
16
16
Optimizer
Adam
Adam
Learning rate
0.0001
0.0001
Loss function
MAE
MAE
As PID itself is SISO control method, ANN-PID uses a neural network to get multiple sets of PID parameters to do multiple SISO PID controls for MIMO control in the Burgers' equation. But here, ANN-PID requires the dimensions of inputs and outputs to be the same, it can only cope with full observation, full control control problems, and partial observation, partial control control problems. Besides, the ANN-PID controller has 2 training setups, including directly interacting with the solver and interacting with the 1D surrogate model in Appendix
H.2 SAC
The Soft Actor-Critic (SAC) algorithm [6 is cutting-edge reinforcement learning method. Con- ceptualized as an improvement over traditional Actor Critic methods. SAC distinguishes itself by introducing an entropy regularization term into the loss function, which encourages the policy to explore more efficiently by maximizing both the expected cumulative reward and the entropy of the policy itself.
Compared with Deep Deterministic Policy Gradient (DDPG) algorithm 35 46), SAC's entropy regularization encourages more effective exploration and prevents early convergence to suboptimal policies, a limitation often seen with DDPG's deterministic approach. Additionally, SAC's twin Q- networks mitigate the overestimation bias that can affect DDPG's value updates, leading to more stable learning. The automatic tuning of the temperature parameter in SAC further simplifies the delicate balance between exploration and exploitation, reducing the need for meticulous hyperparameter
35

---

Table 11: Hyperparameters ID SAC. The full observation partial control, partial observation full control, and partial observation partial control settings share the same hyperparameters.
Hyperparameter name
Value
Hyperparameters for ID Burgers' equation control:
Discount factor for reward Target smoothing coefficient Learning rate of critic loss Learning rate of entropy loss Learning rate of policy loss
0.5
0.05
0.0003
0.003
0.003
Training batch size Number of episodes
8192
1500
Number of model updates per simulator step
50
Value target updates per step
15
Size of replay buffer
1000000
Number of trajectories interacted with the environment per step
Number of layers of critic networks
3
Number of hidden dimensions of critic networks
4096
Number of layers of the policy network
5
Number of hidden dimensions of the policy network
4096
Activation function
ReLU
Clipping's range of policy network's standard deviation output
20 e2
adjustments. Consequently, these features render SAC generally more sample-efficient and robust, particularly in complex and continuous action spaces,
During training, experience for training is stored in replay buffer and sampled randomly to update the networks. All data in the training set are the replay buffer at the beginning. For offline SAC, the replay buffer is unchanged, while online SAC alternates between collecting experience by interacting with the environment and updating the networks with the replay buffer. And offline SAC only uses the surrogate model trained with the training instead of the real environment collect experience. The policy network is updated to maximize the expected return, considering both the Q-value and the entropy term. The critic networks predictions and the target Q-values. SAC also employs target critic network for the critic networks, which are slowly updated with the weights of the main critic network stabilize training, For the inference, SAC uses the policy network to determine the action by selecting the action with the highest probability.
updated minimize the distance between their Q-value
In practice, to help the system approximate the target state accurately and quickly, we need to include the distance between the states of every time step and the target state in the reward. So the reward function of time step i, state wt, target state ur and action w, here is defined as
rl. Uz UT, w.) - -/o In ua|2xx
S |wr|?
where the space domain and a is the weight of energy. We take the Adam optimizer 22 to train the networks and update the temperature parameter. The detailed values of hyperparameters are provided in Table
H.3 Supervised Learning
The paper 24
wewpiieedd-eanin--bbbe control algorithm that takes neural operator as a surrogate model to solve control problems. contains two stages. the first stage, we take a neural operator to learn the physical constraint as Appendix J The three CNNs respectively reconstruct u, reconstruct w and learn the transition from u, to Ut+ More details are Appendix [ In the second stage, these three neural networks are used as surrogate models to calculate the gradient of the objective function with respect to the control input. We consider the control w as learnable parameter and update it with the gradient.
proposes
36

---

To enhance the accuracy, we adopt the LBFGS optimizer [37], which is more accurate while slower than the Adam optimizer. We record the hyperparameters of the second stage in Table T2
Table 12: Hyperparameters of the second stage of the ID supervised learning method.
Hyperparameter name
Valuc
Hyperparameters for ID Burgers' equation control: (full observation partial control)
Learning rate of w updating
0.1
Number of epochs
300
Weight of objective function loss
500
Weight reconstruction loss
0.03
Termination tolerance on first order optimality of LBFGS optimizer
4 x 10-7
Termination tolerance on parameter changes LBFGS optimizer
x 10
Hyperparameter name
Value
Hyperparameters for ID Burgers' equation control: (partial observation partial/full control)
Learning rate of w updating
0.1
Number of epochs
300
Weight of objective function loss
50000
Weight of reconstruction loss
3 x 10-7
Termination tolerance on first order optimality of LBFGS optimizer
Termination tolerance on parameter changes LBFGS optimizer
x 10
H.4 BC
The Behavior Cloning (BC) algorithm 44]
fundamental imitation learning method. BC aims to learn policies directly from expert demonstrations, leveraging supervised learning to map states to actions. This approach bypasses the need for exploration typically required in reinforcement learning by directly mimicking the behavior observed in the provided demonstrations. BC does not require interaction with the environment during training, which simplifies the learning process and reduces the computational resources needed.
In Behavior Cloning, the policy network is trained using supervised learning techniques, where the objective is to minimize the difference between the predicted actions and the actions taken by the expert in the training data. The loss function is typically the mean squared error between the predicted and expert actions. The training data, consisting of state-action pairs, is collected from expert demonstrations and stored in dataset. In the inference, function as the SAC, which is the MSE between the final state after T 1 steps' control and the target. For the partially observed or partially controlled settings, we concatenate zeros in the masked dimensions to make the same dimension as the input of the policy network, but the zeros are not considered during the calculation of the metrics. The detailed values of hyperparameters are provided in Table [
evaluated the same objective
Table 13:
Hy dyyerparameters of 1D BC. The full observation partial control, partial observation full control, and partial observation partial control settings share the same hyperparameters.
Hyperparameter name
Value
Hyperparameters for ID Burgers' equation control:
Learning rate
x 10
Training batch size
512
Number of episodes
5 x 105
Size of replay buffer
2 x 106
Number of layers of policy networks
2
Number of hidden dimensions of policy networks
1024
Activation function
ReLUU
37

---

H.S BPPO
The Behavior Proximal Policy Optimization (BPPO) algorithm [73] is an advanced reinforcement learning method that builds upon the strengths of Proximal Policy Optimization (PPO) while incorpo- rating elements from behavior cloning. BPPO is an offlin algorithm that monotonically improves behavior policy in the manner of PPO. Owing to the inherent conservatism of PPO, BPPO restricts the ratio of learned policy and behavior policy within a certain range, similar to the offline RL methods which make the learned policy close to the behavior policy. By leveraging the inherent conservatism of online on-policy algorithms, BPPO addresses the overestimation issue commonly encountered in offline RL settings. The algorithm starts by estimating behavior policy using behavior cloning and then iteratively improves a target policy using the Proximal Policy Optimization (PPO) objective with behavior constraint. Through this process of policy improvement, advantage estimation, and policy update, BPPO aims to refine the target policy while ensuring it remains close to the behavior policy. By combining the strengths of online on-policy methods with tailored offiin RL techniques, BPPO demonstrates promising results on the D4RL. benchmark, outperforming state-of-the-al offline RI. algorithms.
During the training, BPPO first initializes the behavior policy TH and target policy T.. Then, it estimates the behavior policy Ts using behavior cloning to mimic the behavior demonstrated in the offline dataset. Next, the target policy To is optimized by the PPO objective with a behavior constraint, ensuring that the target policy remains close the behavior policy. After that, it estimates the advantage function A using the behavior policy T to evaluate the quality of actions taken by the target policy. Finally, updating the target policy by maximizing the PPO objective with that estimated advantage function, and adjusting the policy parameters to improve performance. In the implementation, state value network and Q value network pre-trained using the state, action, and reward from the offline dataset.
In practice, to help the system approximate the target state accurately and quickly, need to include the distance between the states of every time step and the target state in the reward. So the reward function of time step i, state We, target state ud and action w, here is defined as
(t. Ue Ud, w.) = / Iu: ual
/ |w, /wi/dxx,
where Ω is the space domain and a is the weight of energy. We take the Adam optimizer [29 to train the networks and update the temperature parameter. The detailed values of hyperparameters are provided in Table [4]
I
2D Jellyfish and Smoke Control Baselines
1.1 MPC and SL
control strategy that solves an optimization problem repeatedly to determine the optimal control inputs for a dynamic system. It operates over a finite prediction horizon, optimizing a cost function and applying only the first control action. In the 2D jellyfish movement control problem, MPC uses the control sequences w and the fluid states as internal state variables. Without the need train a control agent model, MPC relies on 2D surrogate models mentioned in Appendix to estimate future states based on the current state and control. We use backpropagation to compute the gradient, update the control action sequences, and optimize the control objective J in Eq. (19). For every time step, we get the optimized sequences from this time step forward in this way, and only the first control sequence of the optimized control sequences is applied. Compared with MPC, the Supervised learning (SL) method [24] only optimizes the entire control sequences and employs the entire sequences.
Model Predictive Control (MPC) [57
MPC is an optimization technique that aims to optimize the control of complex dynamic systems by considering future predictions. This approach can optimize performance measures over a future time horizon, handle systems with multiple variables and constraints, adapt to changes in the system behavior, and offer good performance even the presence of nonlinearity. Nevertheless, using MPC comes with high cost, both in terms of computational resources and time. Additionally, adapting the optimization hyperparameters for MPC can be a challenging task, In our experiment, both MPC and SL face difficulties when trying to generate smooth opening angle control curves,
38

---

Table 14: Hyperparameters 1D BPPO. The full observation partial control, partial observation
full control, and partial observation partial control settings share the same hyperparameters.
Hyperparameter name
Value
Hyperparameters for ID Burgers' equation control:
State value network:
Learning rate of value network
1 x 10
Steps of value network
2 x 106
Number of layers of value network
3
Batch size of value network
512
Number of hidden dimensions of value network
512
Q value network:
Learning rate of Q network
1 x 10-4
Steps of Q network
2 x 10
Number of layers of Q network
2
Batch size of Q network
512
Number of hidden dimensions of Q network
1024
Target Q network updates per step
2 0.005
Soft update factor
Discount factor for reward
0.99
Behavior cloning:
Learning rate of BC
1 x 10 512 5 x 105
Training batch size of BC
Number of episodes of BC
BPPO:
Number of episodes of BPPO
1 x 102 2 1024
Number of layers of policy networks
Number of hidden dimensions policy networks
Learning rate of BPPO
x 10
Training batch size of BPPO
512
Clip ratio of BPPO Weight decay factor Weight of advantage function
0.25
0.96
0.9
Size of replay buffer
2 x 106
Activation function
ReLU
when constraints R(w) are included the optimization objective J. In the case multi-objective optimization problems, it becomes even more challenging for them simultaneously achieve both the higher speed (bigger ū) and the control curves smoothness (smaller R(w)).
1.2
SAC
For the 2D case, the algorithm and basic architecture of SAC the same the ID case in Appendix When designing the reward function for the 2D jellyfish movement control, we find the periodic condition of the opening angle curves is hard to constrain. So to satisfy the periodic condition better, we include the distance between w, of every time step & and Wp- Also, we both consider the squared and absolute error of (w, wo) since they respectively constrain the periodic condition when (W,- wo) is large and small. As a result, the reward function of time step 4, force Fr, opening angle (w,-1, 1, w.) and condition angle wo is defined
H.2
r(t, W1-1, w. Wo) (T + X Aw: - - + |w: -wol).
where '₁ A2 are weights of different terms. As for the 2D incompressible fluid control, we set the reward function the percentage of smoke passing through the target bucket.
We take the Adam optimizer (29] update the weights of networks and the temperature parameter as in the ID experiment. The hyperparameters reported in Table [5 In particular, take the best
39

---

checkpoint to evaluate the final performance, thus the actual number of training episodes for each setting ranges from about 100 to about 300.
Table 15: Hyperparameters of 2D SAC. The full observation and partial observation settings share the same hyperparameters.
Hyperparameter name
Value
Hyperparameters for 2D Jellyfish movement control:
Weight of the constraint of periodic condition / Discount factor for reward Target smoothing coefficient Learning rate of critic loss Learning rate of entropy loss Learning rate of policy loss Training batch size Number of episodes
0.001
0.5
0.05
0.0003
0.0003
0.0003
2048
350
Number of model updates per simulator step
20
Value target updates per step Size of replay buffer
15
11400001
Number of trajectories interacted with the environment per step
5
Activation function Clipping's range of policy network's s standard deviation output
ELU
e- [e-5,6-
1.3 BC
For the 2D task, the algorithm and basic architecture of BC are the same as the ID case in Appendix H.4 In Behavior Cloning, the policy network is trained using supervised learning techniques, where the objective is to minimize the difference between the predicted actions and the actions taken by the non-expert in the training data. The loss function is typically the mean squared error between the predicted actions and actions in training data. The training data, consisting of state-action pairs, collected from non-expert random demonstrations and stored in dataset. In the inference, we evaluated the same objective function as the SAC. The detailed values of hyperparameters are provided in Table 16
Table 16: Hyperparameters of 2D BC. The full observation and partial observation settings share the same hyperparameters.
Hyperparameter name
Value
Hyperparameters for 2D control:
Learning rate Training batch size
x 10
512
Number of episodes
x 103
Size of replay buffer Activation function
2 x 106
ELU
Clipping's range of policy network's standard deviation output
5, -2
1.4 BPPO
For the 2D task, the algorithm and basic architecture of BPPO are the same as the ID case in Appendix H.5) In jellyfish movement control, when designing the reward function, we find the periodic condition of the opening angle curves is hard to constrain. So to satisfy the periodic condition better, we include the distance between
of every time step and wo- Also, we both consider the squared and absolute error of (w, Wo) since they respectively constrain the periodic condition when (w: wo) is large and small. As result, the reward function of time step t, force
40

---

Ft, opening angle (w,-1, wi) and condition angle Wo is defined
r(t, WI-1, Wa, wo) (T' - Pr Ai(w, w.-1)² >2((w, -o)2 + |w: wol),
where A1, A2 are weights of different terms. The hyperparameters reported in Table 17
Table 17: Hyperparameters of 2D BPPO. The full observation and partial observation settings share the same hyperparameters.
Hyperparameter name
Value
Hyperparameters for 2D control:
State value network:
Learning rate of value network
1 x 10
Steps of value network
2 103
Number of layers of value network
Batch size of value network
512
Q value network:
Learning rate of Q network
1 10
Steps of Q network
2 103
Number of layers of Q network
Batch size of Q network
512
Target Q network updates per step Soft update factor
2 0.005
Discount factor for reward
0.99
Behavior cloning:
Learning rate of BC Training batch size of BC
1 x 10-4
512
Number of episodes of BC
5 x 103
ВPPО:
Number of episodes of BPPO
1 x 102
Number of layers of policy networks
3
Learning rate of BPPO
1 x 10-5
Training batch size of BPPO
512
Clip ratio of BPPO
0.25
Weight decay factor Weight of advantage function
0.96
0.9
Size of replay buffer
2 10
Activation function
ELU
Surrogate Models
J.1
1D Surrogate Model
For the control problem of 1D Burgers' equation, our 1D surrogate model is based on the previous paper (24), which uses autoencoders to model dynamics the latent space. The neural simulator architecture and training details are shown in Table [8
J.2 2D Jellyfish Force Models
Dataset. In 2D jellyfish movement control experiments, we train force surrogate to approximate the computation of the average speed of the jellyfish for the guidance of inference, which is implemented by neural network and is thus differentiable. The training data consists of pressure, boundary, and force data in the training trajectories. Each training trajectory amounts to T 40 training samples. Therefore, have 1.2 million training samples and 4 thousand testing samples in total.
Model. The model's s input contains pressure, boundary mask, and offsets with shape x 64 at certain time step, and the output is the corresponding forces of and y directions. The model
41

---

Table 18: Hyperparameters of ID surrogate model.
Full observation,
Partial observation,
Partial observation,
Hyperparameter name
partial control
full control
partial control
Autoencoder of state
Convolution kernel size
5
5
5
Convolution padding
2
Activation function
ELU
ELU
ELU
Latent vector size
256
128
128
Autoencoder of force
Convolution kernel size
5
Convolution padding
2
Activation function
ELU
ELU
ELU
Latent vector size
256
256
256
Training
Training batch size
5100
5100
5100
Optimizer
Adam
Adam
Adam
Learning rate Training epochs
le-3
le-3
1e-3
500
500
500
Learning rate scheduler
cosine annealing
cosine annealing
cosine annealing
architecture is the down-sampling part of U-Net [55 that embeds the input features into a 512- dimensional hidden representation; then we use linear function with output dimension two to output forces.
Training. We MSE (mean squared error) loss between the ground truth and predicted forces to train the force surrogate model. The optimizer is Adam [29]. The batch size is 64. The model is trained for 10 epochs. The learning rate starts from 1 10-1 and multiplies factor of 0. every three epochs. After training, the relative la test error is 0.4%.
J.3
2D Jellyfish Boundary Mask and Offsets Updater
Dataset. In 2D jellyfish movement control experiments, train boundary mask and offsets updater surrogate to approximate the transition boundary mask and offsets from time step to Thus each training trajectory amounts to
39 training samples.
Model. The input is the boundary mask and offsets at time step 0 with shape 3 x 64 x 64, and the difference of the opening angle from to f. The output is the boundary mask and offsets time step f with shape 3 64 x 64. The model architecture is the U-Net [55] with additional scalar input, similar the denoising network in DDPM [19, where the input scalar diffusion step is replaced by the angle difference in our model.
Training. We use MSE (mean squared error) loss between the ground truth and predicted boundary mask and offsets to train this surrogate model. Hyperparameters of training are the same as those of the force surrogate model.
J.4 2D Jellyfish Simulator
Dataset. 2D jellyfish movement control experiments, we need to train a surrogate model as a solver of the physical dynamics for the baseline methods like SAC (online) and MPC, because of their iterative nature. Conversely, our diffusion method does not need this surrogate model. This model approximates the transition of states under the boundary condition from time step t to t 1. Thus each training trajectory amounts to T - 1 39 training samples. We train two versions of this model for full/partial observation settings.
Model. This surrogate model is also implemented by the U-Net [55] The model input is the states, boundary mask, and offsets at time with shape 6 x 64 64 for the full observation setting and 4 x 64 x 64 for the partial observation setting. The output is the predicted states at time step / 1, with shape 3 x x for the full observation setting and 64 64 for the partial observation setting.
42

---

Table 19: More results ID Burgers's equation control. Italic font denotes unfair results, bold font denotes the best model among fairly evaluated baselines, and underline denotes the second best model among them.
PO-FC J
FO-PC
PO-PC J 4
J
SAC (online)
0.01567
0.034092
0.02768
DifiPhyCon-lite
0.01139
0.00037 0.00037
0.00494
DiffPhyCon
0.01103
0.00494
10
10"1
10"
-0
4
10
10"1
Evelusted samples
10-t
Evaluated samples
Evaluated samples
10"4
Z J
10-1
10
10a
10"
(a) PO,
[B]
Figure 16: and state MSE & of different samples in different settings.
Training. We use MSE loss between the ground truth and predicted states train this surrogate model. Hyperparameters of training are the same as those of the force surrogate model.
K
Additional Results of Experiments
K.1
DiffPhyCon Con for 1D Burgers' Equation
Since online RL methods are effective in a wide domain of control tasks, we tested the performance of the classical RL method, SAC (online), shown in Table 19 The online SAC belongs a different setting than the offline training setting considered in our work. Even with access to more information from the environment, online SAC is less competitive our DiffPhyCon in Burgers' control task.
As DiffPhyCon can jointly generate the trajectory u and control sequence w, we demonstrate the MSE of the diffused trajectory from the trajectory computed by the ground-truth solver Uground tuth given the diffused control sequence w. This should be differentiated from J reported Table which is the MSE of the trajectory computed by the ground-truth solver given from the target trajectory u.. The former MSE is denoted as E == Iu Uggonnd truhh in Figure 16
It can be observed from Figure 16 that there is a clear correlation between J and E. This is because, the higher the prediction error is, the less accurate the solution to the inverse control problem would be. The correlation proves that our DiftPhyCon learns the dynamics of the system (i.e.. the dependency partial observation settings,
of on w), based on which the control sequence is generated. the randomness is larger than in the full observation setting, which demonstrates the challenge of controlling partially observed systems. The reference line denotes J E where the control objective completely depends the prediction accuracy. In Figure (b), samples cluster tightly around the line, which implies DiffPhyCon can find near-optimal control sequence based on the predicted trajectory in the corresponding level of prediction accuracy. This ability is remarkable in that DiffPhyCon does not rely on explicit gradient descent like those in the adjoint method and SI.
K.2 2D Jellyfish Movement
In this subsection, we record extensive results of 2D jellyfish movement control. further compare baselines with DiffPhyCon in more settings, we change Ç in J and results of Ç 500, 1000 and
43

---

Table 20: Full observation setting of 2D jellyfish movement control. Bold font denotes the best model. and underline denotes the second best model.
500
1000
( = 2000 R(w) 4
R(w)
R(w)
8
4
MPC
-38.21
0.1743
212.49
25.72
0.0112
109.17
44.24
0.0755
31.29
90.09
0.2570
166.89
-76.94
0.1286
205.57
-62.96
0.0648
127.8
SAC (pseudo-online)
133.13
0.0204
154.49
-166.96
0.0069
178.14
-309.18
0.0040
313.19
SAC (ofline)
159.83
0.0119
171.73
-188.66
0.0069
165.58
-278.52
0.0051
283.58
BC BPPO
15.61
0.0589
43.33
30.48
0.0629
32.44
33.5
0.0677
34.23
88.64
0.0776
-11.07
107.67
0.0867
-20.93
-20.12
0.0621
82.27
DifiPhyCon-lite
151.57
0.0939
-57.69
95.04
0.0746
-20.47
150,53
0.0923
-58.23
DiffPhyCon
310.02
0.2344
-75.64
279.87
0,2058
-74.11
270.56
0.1755
-95.08
Table 21: Partial observation setting of 2D jellyfish movement control. Bold font denotes the best model, and underline denotes the second best model.
500
1000
2000 R(w) 1
I((w)
8
R(w) 4
J
8
J 4
MPC
-179.85
0.2534
433.3
-150.51
0.1791
329.59
-116.75
0.1397
256.46
ST.
44.04
0.2895
245.43
-102.98
0.1188
221.79
-152.45
0.0661
218.53
SAC (pscudo-online)
21.87
0.0798
57.96
-153.09
0.0057
158.82
-206.43
0.0048
211.2
SAC (offline)
-149.32
0.0156
164.93
-206.21
0.0058
211.96
-258.96
0.0029
261.87
BC
35.49
0.057
21.48
20.08
0.0556
35.48
35.14
0.0594
24.27
BPPO
104,28
0.0746
.29677
54.83
0.0518
-3.02
16.97
0.0598
42.83
DiflPhyCon-lite
-0.35
0,0782
78.56
2.92
0,0779
74.97
-10.89
0.0760
86,93
DiffPhyCon
173.27
0.1636
-9.66
150.21
0.1269
-23.32
101.25
0.25
1,21
= 2000 are listed in Table 20 Table 21 The results show that DiffPhyCon outperforms other baselines in diverse settings with different objectives.
Table 22: Results of myopic failure modes of SAC on 2D jellyfish movement control.
Average speed (8)
R(w)
objective J
periodicity error
SAC (weight=100)
59.82
0.0263
-33.48
0.35187
SAC (weight=1000)
-166.96
0.0112
178. 14
0.02299
DiftPhyCon
279.87
0.2058
-74.11
0.01157
K.3 Myopic failure modes of SAC
To further confirm the advantage of diffusion models over the baseline SAC in alleviating the myopic failure modes, we perform additional 2D experiments. Specifically, when solving the jellyfish movement problem using SAC, we incorporated constraint achieve periodic motion of the jellyfish in the reward function, namely, we added hyperparameter as the weight of the term d(wr, wo) in Eq. 19.. The challenge caused by the periodicity condition is that the average speed and R(w) are calculated over the whole horizon while the periodicity condition is only evaluated the final time step of the horizon. Thus, when the control objective consists of all three terms, requires global optimization over the whole horizon to balance different terms. In Table 22 we present the original results (weight=1000, corresponding to Table 3 and the results after reducing the weighting of this term (weight=100). It can be observed that while the speed of the jellyfish increases (though still not reaching the results of DiffPhyCon) the R(w) and constraint term for periodicity sharply rise. From these results and the additional results in Table an and Table 21 where we test the performance of SAC under different weights Ç of R(w). we conclude that SAC struggles provide satisfactory policies that simultaneously satisfy multiple conflicting constraints with different time coverage, resulting in a myopic failure model. The reason may that difficult to estimate the effect of each term in the control objective in each time step in the iterative planning fashion []. which this issue could well addressed by our diffusion models with an inherent nature of global optimization.
44

---

Table 23: Efficiency comparison on 1D Burgers' equation. Inference time is tested on Tesla- V100 GPU with 8 CPUs.
Methods
Training time (hours)
Inference time (seconds)
DiftPhyCon-lice
1.7 A100-80G GPU, CPUs)
21.13
DiffPhyCon
4.4 (1 A100-80G GPU, CPUs)
58.97
DifiPhyCon-DDD DDIM (8 sampling steps)
1.7 (1 A100-80G 80G GPU, 8 CPUs)
0.53
PTD
0.1 (1 A100-80G GPU, 8 CPUs)
3.61
SI.
2.6 V100-32G GPU, 12 CPUs)
74.85
SAC
10.5 (1 A6000-48G GPU, CPUs)
0.11
BC
8.8 (1 V100-32G GPU, 12 CPUs)
1.22
BPPO
8.9 (1 V100-32G GPU, 12 CPUs)
0.82
Table 24: Efficiency comparison on 2D jellyfish movement. Inference time is tested on Tesla- V100 GPU with 8 CPUs.
Methods
Training time (hours)
Inference time (seconds)
DiffPhyCon
62 (2 A100-80G GPUs, CPUs)
252.2
DinPhyCon-DDIM (8 sampling steps)
62 (2 A A100-80G GPUs, 32 CPUs)
12.6
MPC
52.1 (1 A A100-80G GPU, 16 CPUs)
1401.7
SL
52.1 (1 A 1100-80G GPU, 16 CPUs)
133.5
SAC
9.5 (1 A100-80G, 16 CPUs)
0.2
BC
2.8 (1 A100-80G, 16 CPUs)
0.99
BPPO
3.0 (1 A100-80G GPU, 16 CPUs)
1.08
K.4
Efficiency Comparison
We test the training and inference (control) efficiency of DiffPhyCon and baselines on both ID Burgers' equation and 2D jellyfish movement tasks. On the ID Burgers' equation task, we test the efficiency of partial observation and full control (PO-FC); on the 2D jellyfish movement task, we test the efficiency of full observation (FO).
The results are presented in Table 23 and Table 24 For training efficiency, since models with different sizes trained on different machines, report the training time of models along with the machine information. The inference efficiency is tested on a single Tesla-V100 GPU with 8 CPUs and the average inference times over all test samples are reported. For training efficiency, we have the following observations. On ID Burgers' equation, our DiffPhyCon is comparable with SL and more efficient than reinforcement learning (RL) methods like SAC, BC, and BPPO. On 2D jellyfish movement, our DiftPhyCon costs similar training time compared with MPC and SAC, but more time than RL methods. For inference efficiency, these results reveal that although DiffPhyCon is not as efficient as RL methods, it has competitive efficiency compared to SL and MPC. In particular, by using the fast sampling method DDIM 60), DiffPhyCon could be accelerated significantly by reducing the number of sampling steps.
L Effect of Hyperparameters
L.1 Effect of
ID Burgers' Equation control. In the ID Burgers control task, the prior reweighting is intrinsically unnecessary. The evaluated control target follows the same distribution the training set, and the samples in the training set are defined to be the optimal control. Therefore, the generated control sequences following pl p(w[ug, ur) are already the (near) optimal distribution which alleviates the need for prior reweighting.
We tested different scheduling of the prior reweighting and found their performance similar. Finally, we use the same cosine schedule as that in the DDPM noise for clarity. Our experiment results also revealed that the performance of DiffPhyCon in 1D Burgers equation control is insensitive to the prior reweighting intensity 7 as we reported T Besides, we conducted a more comprehensive experiment the prior reweighting intensity y in all three settings (FO-PC; PO-FC; PO-PC) as
45

---

Table 25: Results of different y in Dif on FO-PC ID Burgers equation control task.
0.0
0.3
0.5
0.7
1.0
Jactual
0.00038
0.00037
0.00037
0.00037 0.00028
0.00037
State MSE
0.00063
0.00043
0.00034
0.00025
Table 26: Results of different y in DiffPhyCon on PO-FC ID Burgers equation control task.
0.0
0.3
0.5
0.7
1.0
Tnctunl State MSE
0.01159
0.01155
0.01152
0,01 0.02641
0.01139 0.02639
0.02643
0.02642
0.02642
in Table 25
26 and 27 The results show that prior reweighting does not significantly impact the performance of DiftPhyCon in ID Burgers control task, where State MSE denotes the deviation between generated u and the ground truth Ugt given by the numerical simulator based on generated f.
2D jellyfish movement control. Performance of DiftPhyCon is determined by the hyperparameter 7. Since diffusion models denoise gradually, we use varying sequence of 7 {7x ) to subtract prior in DifiPhyCon. Specifically, the schedule of y set as 7h = - f. 3 k = 1,..,, K, k= where & is a fixed coefficient to control the scale of and 8 { is the schedule of variances of noise in DDPM [9].. In our implementation, we use the sigmoid schedule of B [25). The total number of inference steps is K 1000. Thus only need to tune E to examine the effect of y. When & < 0. DiffPhyCon is prone to restrict w within its prior distribution training dataset in inference. When > 0, DiffPhyCon is more likely generate
kinds of beyond training ones. When ç = 0, DiffPhyCon degenerates to DiftPhyConlitt In 2D experiments, We set default & = 0.3 and the corresponding "/1 0.7 we empirically find this value performs well and steadily. We present the performance of DiftPhyCon on the 2D jellyfish movement control task under different y in Figure and Table 28 We observe that the performance increases along with decreasing of 71. When 71 < 0.6, invalid generated control sequences emerge because the prior is largely overlooked. Thus the valid interval for y of prior reweighting on this task [0.6, 1.0]. J. is interesting
to find that when > 1, the performance decreases. This may be caused by the strict constraint of
the prior distribution of p( o(W, c), which results in generating control sequences similar those from training datasets and thus not good.
L.2 Effect of X
ID Burgers' Equation control. In ID Burgers control, DiftPhyCon uses guidance conditioning since it empirically performs better than explicit guidance as shown in Table 29 Therefore, our model does not use À to generate the optimal control. However, when we consider the energy cost as in Figure 3 we use explicit guidance to control the magnitude of control signals as it more natural way to incorporate the objective >.Teneryy into the overall diffusion objective E: (u, w, c) + AJennrgy where the condition e ur Udd.
where À = Au A: = 1, K where 3g starts The guidance intensity uses a cosine schedule 43
from 81 0.001 to 8x 1. The schedule decreases the diffusion step decreases during inference, where the gradually decreasing guidance intensity provides strong guides in the initial stage while reducing the impact the near-clean samples.
Table 27: Results of different y in Din on PO-PC ID Burgers equation control task.
7
0.0
0.3
0.5
0.7
1.0
Jactual
0.00493
0.00493
0.00493
0.00493
0.00494
State MSE
0.00675
0.00667
0.00665
0.00664
0.00663
46

---

Table 28: Results of different y in Con on 2D jellyfish movement control task.
?1
Average speed (i)
R(w)
objective J
0.6
0.4
410.6
0.2581
-152.51
0.7
0.3
279.87
0.2058
-74.11
0.8
0.2
197.18
0.1312
-65.99
0.9
0.1
76.97
0.0741
-2.84
1.0
95.04
0.0746
-20.47
1.1
-0.1
81,41
0.0742
-7.21
1.2
-0.2
84.56
0.0736
-10.93
1.3
-0.3
65.12
0,0725
7.38
1.4
-0.4
65.02
0.0734
8.43
1.5
-0.5
64.07
0.0752
11.1
1030-R(A0
25
RR())
-50
-12
150
0.6
1.0
2.2
V/2
Figure 17: Results of different y in DiffPhy Con on 2D jellyfish movement control task.
In such a setting, varying À changes how the generated samples are shifted towards lower energy regions, but an excessively large À causes the sample to deviate from the physically feasible region. Results in Table 30 31 and 32 show that the behavior of DiffPhyCon under different À is expected. Note that "physically feasible" samples here are samples with lower Jactual-
Table 29: Results of two different guidance type of DiffPhyCon in 1D Burgers equation control task.
Guidance Type
FO-PC
PO-FC
PO-PC
Explicit Guidance
0.02789 0.00037
0.03257 0.01139
0.05584
Conditioning Guidance
0.00494
2D jellyfish movement control. our jellyfish movement control task, we use explicit guidance with À involved. Since diffusion models denoise gradually, similar the hyperparameter 7, we use varying sequence of X = > Ao x 3x. where 3₁ is the schedule of variances of noise in DDPM [2] increasing monotonically from B1 0.0003 to 3x 1. The motivation of this schedule is that the confidence of gradient estimation becomes stronger along with the denoising process from k = K to We can observe that as À (which determines the schedule) increases from 0 to 0.6, the performance of DiffPhyCon improves consistently. Compared to Table 3 DiffPhyCon always outperforms baselines for wide range of 0.1 S λo S 0.5. When An 2 0.6, the results are no longer valid since the generated control sequences may be unfeasible. This reflects that should not be overly large to make safe, which is consistent conclusion with previous analysis.
1. The results are shown in the Table 33
47

---

Table 30: Results of different for Tenergy DiffPhyCon in FO-PC ID Burgers equation control
task.
A
0
10
100
500
1000
10000
100000
1000000
Tanul
0.00037
0.000337
0.00037
0,00036
0,00037
0.00147
0.02202
0.06444
State MSE
0,00025
0.00025
0,00024
0,00024
0.00024
0.00083
0.00802
0,01838
Tenrgyy
1320.98206
1309.31104
1237.06958
1061.67766
946.07617
584.32098
225.50281
38.25005
Table 31: Results of different for Tenergy of CoffPhyCon in PO-FC ID Burgers equation control
task.
0
10
100
500
1000
10000
100000
1000000
Jacual 0.01139
0.01152
0.01252
0.01906
0,02100
0.04411
0,08472
0.11934
State MSE
0.02639
0.02641
0.02661
0.02919
0.02990
0.03857
0.05472
0.06471
Jewergy
862.98633
860.0516 164
839.11932
770.45819
704.68665
311.21915
69.87619
8.93373
M Extensions to Finer-grained Jellyfish Movement Control Task
To test the scalability of DiffConPDE in handling nigh-dimensional and complex dynamic systems, we extend the jellyfish movement control experiment from rigid boundaries of wings to soft ones.
M.1 Experimental Setting
Similar to previous setups in Appendix [F] this extension aims to control the movement of flapping jellyfish with two wings in a 2D fluid field where fluid flows with constant initial speed. However, in this task, we parametrize the jellyfish's boundary as the coordinate change (Aæ, Ay) for each cell within the boundary, which serves as the control sequence w. Undoubtedly, this a high-dimensional and complex control task. Firstly, the control sequence w is elevated to three dimensions. In addition ensuring optimization towards the control objective, maintaining consistency the movement of different cells within the jellyfish boundary is also crucial. Bearing this in mind, we have chosen to employ DiffConPDE-lite for this experimental setup to minimize potential disruptions to the jellyfish boundary. Regarding the PDE state, the experiment adopts the full observation setup. All other settings remain the same with Appendix
M.2 Data Generation
In this extension, we leverage data previously generated under the vanilla setting. Additionally, we utilize opening angles O, The coordinates of the rotation center h, and the boundary mask generate the required control sequence w. We represent w tensor of shape [T, 2, 64, 64). Each cell has two features representing the changes in the two coordinates. For each trajectory, w is generated as follows: Firstly, for the initial wo and final Wr. we set all elements in the tensor to 0. Then, for each W[] corresponding to the cells within the boundary, we first obtain the relative coordinates c= (z, y) of the cell with respect to the rotation center using the boundary mask and the coordinates of the rotation center h. Afterwards, we construct the rotation matrix Mc(O,) at different time steps using the opening angle θ p17-
cos cos(O,) sin sin(O,) sin O.))
].
Mc(O,)
cos (
The coordinate change w, can be obtained using the following formula:
Wl cM2-
Table 32: Results of different À for Jenergy of Con in PO-PC 1D Burgers equation control task.
0
10
100
500
1000
10000
100000
1000000
Jarua 0.00494
0.00500
0.00549
0.00794
0.01426
0.02411
0.03979
0.08153
State MSE
0.00663
0.00676
0.00768
0.01048
0.01525
0.02766
0.03635
0.05459
Jexeray
1314,90698
1297.32849
1192.89026
945.77179
793.28442
331.45758
108.39325
16.13734
48

---

Table 33: Results of different À on 2D jellyfish movement control.
Àu
Average speed (6)
R(w)
objective J
0.6
0.5
362.82
0.192
1170.79
0.4
322.04
0.182
-140.04
0.3
279.87
0.206
-74.11
0.2
192.77
0.114
-78.57
0.1
103.89
0.091
-12.32
-86.69
0.042
128.67
Additionally, for cells that are not within the boundary, the corresponding W[1,--1] is also directly set to (0, 0). Therefore, all the data involved in this experiment are as follows:
Coordinate change of cells w: shape 2, 64]. For each step, we save the coordinate change of each cell within the boundary.
PDE states w: shape [T' 3, 64, 64). For each step, we save the states of the fluid field consisting of velocity in æ and y directions and pressure. To save space, downsample the resolution from 128 x 128 to 64 x 64.
- -velciit: [T', 2, 64, 64].
pressure: [T', 1, 64, 64].
opening angels O: shape [T]. For each step, save the opening angle in radians.
boundary mask and offsets b: [T, 64, 64]. For each step, we save the mask of merged wings with half coordinates of boundary points and offsets in both and y directions. The resolution is 64 x 64, compatible with that of the states.
mask: [ 1, 64, 64].
offsets: [T' 2, 64, 64].
force: shape [ 2]. For each step, the simulator outputs the horizontal and vertical force from the fluid to the jellyfish. The horizontal force is regarded as a thrust to jellyfish if positive and a drag otherwise.
M.3
Model
Similar to the vanilla setting, this experiment also employs U-net as the backbone for the diffusion model. The U-net architecture remains consistent. The input the U-net includes PDE state u(TT,3, 3, 64, 64]), control sequence w(T, 64, 64]), initial boundary mask and offset ([1, 64, 64]). is worth noting that to align the initial mask and offset with others terms of shape, we expand them along the time dimension, resulting in the final shapes of [T. 3, 64, 64). Therefore, the shape of the input to the model is [T' 8, 64, 64], and the output the model includes both noise of the predicted state and the control sequence, with shape of [T, 64, 64).
M.4 Training, Inference, and Evaluation
Training. During the training phase, the diffusion-flow task adopts similar setup, utilizing the Mean Squared Error (MSE) between the model predictions and Gaussian noise the loss function (Fq[4). The batch size is chosen as 16, and the training involves 180,000 iterations. The learning rate starts 10- and increases by a factor of 0.1 at the 50,000th and 150,000th iterations. More training details are presented in Table 34
The pipeline of inference shown in Figure Both diffused variables u and W|o.T] are initialized from Gaussian prior and gradually denoised from denoising step k = 1000 to l: 0 based on denoising networks and guidance. Similar the vanilla setting, due to the introduction of the initial boundary mask and offset additional inputs, the number of channels in
Inference.
49

---

Table 34: Hyperparameters network architecture and training for the Finer-grained Jellyfish
Boundary Control Task.
Hyperparameter name
full observation
Batch size
16
Optimizer
Adam
Learning rate
0,001
Loss function
MSE
the model's input and output are inconsistent. For guidance, we X = 5 10-5 and we also utilized surrogate model to approximate the force of fluid on the jellyfish. In the k-th denoising step, the inputs of this model include the noise-free state [oo.T]..- W W[o.T].K estimated from (. by Eq. (6), initial boundary mask and offset. The model output is the force. The remaining aspects such as model architecture, training details, etc., remain the same. As for the regularization term R ((w). we approximate the opening angle of the jellyfish's wings at time t using the change of the coordinate (Ar,, Ayr) and the initial coordinates (æ, y) of each cell to achieve regularization. Firstly, we utilize the following formula to approximate the angle change of each cell:
ù. - - (Ar / y²)
Then, we calculate the average value the angles of different cells, which serves as an approximation of the angle between the two wings of the jellyfish at time t, denoted as θ,, It evident that optimizing this term intuitively minimizes the coordinate changes of each cell, aligning with the objective of regularization, The control objective J in Eq. (99) is computed as a summation of force and (( (o.r]) ). and we fix 1000. Then the gradients of the objective J in terms of u TJ and W[o,2] are computed and used in guidance. Because this experiment selects DiffConPDE-lite, all these gradients are directly subtracted from [u Woo.7J.l generate [M[o,T].--1] 1, W [oo.T].e-11
[la] x [ba] Stpaas Miart
Joint distributio WiltoWg..he
Denoise under guidance
L3-3
x
x
From inference step k to k
Figure 18: Inference our finer-grained jellyfish boundary control task.
Evaluation The inference outputs coordinate change for each cell of T' 20 steps for 50 testing samples. To align with the evaluation metrics of the vanilla setting, we first need to calculate the opening angle θ, using the coordinate change of the cells time t. Firstly, at time E [1, 1]. for each cell, we obtain the rotated coordinates (æ1, 3/) using the initial coordinates (x, y) and the coordinate change (Ar,, Ay:). Then, combined with the coordinates of the rotation center h, we can respectively obtain the radial distance of the cell relative to the rotation center before and after rotation, denoted as r and r'. Then, according to the law of cosines, we can obtain the rotation angle 50 of the cell at time compared to the initial time:
A 7/2 + (dx? + 5y.)
2r/r=
Afterward, we take the average of the angle changes of all cells within the boundary as the angle change of the jellyfish's two wings at time t, denoted as AQ,. Finally, based the average value of
50

---

the flow in the z-direction at time t, the opening closing status of the jellyfish relative to the initial moment is determined. If it is negative, indicates opening; otherwise, it suggests closing. This process allows us to obtain the specific angle change for each time step compared to the first moment:
c | Oo- |
A.,
AT,
O
AT, = 0.
After obtaining the theta sequence, we can follow the vanilla evaluation procedure.
M.5 Results
We present the results in Table 35 From the table, it can be observed that our method optimized the objective J to 84.31 for this task. Compared to the results based on the original setting in Section 4.2, our method is still competitive with baselines (shown in Table in this more challenging setting. An example of optimized soft boundary shapes is illustrated in Figure 19 These results demonstrate the good scalability of our method when facing higher-dimensione and complex control tasks.
Table 35: Performance of DiffPhyCon-Ien on the finer-grained jellyfish boundary control task.
Average speed (ö)
R(w) objective (T)
-33.91
0.0504
84.31
Jelyfish Boundaey
jelyfih
t-10
L-15
L-19
51-
50
27
20
22
Figure 19: Inference results of soft boundaries on the finer-grained jellyfish control task.
N Limitation and Future Work
There are still several limitations of DiffPhyCon that inspire future work. Firstly, our approach is data-driven, so it is not guaranteed to achieve optimal solutions and also lacks estimation of the error gap between the generated control sequence and the optimal ones. Secondly, the training of DiftPhyCon is currently conducted in an offline fashion, without interaction with a ground- -ruthh solver. Incorporating solvers into the training framework could facilitate real-time feedback. enabling the model to adapt dynamically to the environment and discover novel strategies and solutions. Furthermore, our proposed DiffPhyCon presently operates an open-loop manner, does not consider real-time feedback from solvers. Integrating such feedback would empower the algorithm to adjust its control decisions based on the evolving state of the environment.
Social Impact Statements
The method we propose offers means to actively interact with the physical world and achieve specific control objectives. This approach presents significant opportunities for various domains, including fluid control, robotic control, controllable nuclear fusion, and more. However, it is imperative to remain vigilant about potential negative consequences that may arise from this technology to prevent its application unethical or illegal control issues.
51

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: Our paper's contributions and scope are presented in the abstract and introduc- tion accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations our work are discussed in Section N
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
52

---

Justification:
The theoretical result is presented in Section 3.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details of experiments are presented Appendices D
Mnd These details ensure the reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
53

---

Answer: [Yes]
Justification: Our code available at htttts:://githiii com/ AIwScience----eeeeeeceeieee /
diffphycon which also includes the link our 2D jellyfish movement data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/guuu guwde//ddee/e/e Cowe/ooodeeeieseeeeeeee wwtprbmmsssiooooiiooiooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details of training and test details are presented in Appendices D
K and M These details ensure the reproducibility of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars because multiple rumning of our method and all the baselines on the three tasks under different settings are computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
54

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details about the compute resources are provided in Appendix K.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts of our research are discussed Appendix O
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
55

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our released data are generated by open-source simulator. Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our released data are generated by simulator, which is properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
56

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide details about the dataset in the Section of Experiment and Appen- dices F Gnd
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
57

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
58


Reviews:
[{"review_id": "naPFp01309", "reviewer": "Reviewer_8Er5", "summary": "The method learns the energy function, $\\epsilon_\\theta$, which is used in the energy optimization target to generate the control sequences and system trajectory. A denoising network is trained to approximate the gradient of $\\epsilon_\\theta$. The network and optimization framework takes a global state of u and w of all times steps. The training and inference follow the process of a diffusion model. \n\nThe paper also introduces prior reweighting to enable the discovery of control sequences that diverge significantly from training. Another network $\\epsilon_\\phi$ to learn the prior distribution of the control sequences is introduced.\n\nThe proposed method is tested by two systems (1D Burgers’ equation and 2D jellyfish movement control). The experiments compared different control methods.", "questions": "- Compared with other methods, does the proposed method use less, similar, or more time to train (for the learning-based methods) and inference?\n- Is it easy or hard to extend this method to 3D systems? In the 2D example, a 3D U-Net is used. Does this make it hard to apply this method to 3D systems as it requires a much larger 4D network?\n- In the prior reweighting method, are $\\epsilon_\\theta$ and $\\epsilon_\\phi$ trained jointly, or  is $\\epsilon_\\theta$ trained first and then $\\epsilon_\\phi$?", "limitations": "The authors discussed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper develops a generative method to control complex physical systems. The method optimizes system trajectory and control sequences jointly in the entire horizon by diffusion models. A prior reweighting technique is also proposed to improve the model's generalization ability.\n- The results of the jellyfish systems are strong. The proposed methods generates realistic control sequences and trajectory that align with the established findings in fluid dynamics. \n- The paper generated a dataset for the jellyfish system, which contributes to the benchmark of this area.", "weaknesses": "- The experiments are relatively limited - only include one 1D example and one 2D example.\n- I have minor reservations about the novelty of the paper since the primary method relies on the existing diffusion model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments. We are glad that the reviewer appreciates our jellyfish results and dataset contribution. Below, we address the reviewer’s questions one by one.\n>**Comment 1**: The experiments are relatively limited - only include one 1D example and one 2D example. \n\n**Answer**: Our two tasks align with existing data-driven physical control papers [1-3]. For further evaluation, we **conducted an additional 2D incompressible fluid control experiment** using a setup similar to [3]. Control forces were applied within a 64x64 grid flow field, excluding a semi-enclosed region, to minimize the smoke failing to exit through the top middle exit. This **high-dimensional indirect control problem** involves managing 2-dim forces at approximately 1,700 grid points per time step, resulting in about **100,000 control variables** in total across 32 time steps, making it highly challenging.\n\nWe generated 20,000 training trajectories, with average control objective 49.9%. We evaluated 50 test samples. The results (Table 1) show that **our method still has significant advantages**, especially in validating the role of prior reweighting. For **visualization** of the results, please refer to Figure 1 in the PDF file in **General Response**. These updates will be added to the final manuscript.\n\nTable 1: Performance comparison on the new 2D indirect fluid control task.\n| Method | Control Objective $\\downarrow$ |\n| - | - |\n| BC | 0.3085|\n| BPPO | 0.3066 |\n| SAC (surrogate-solver) | 0.3212 |\n| SAC(offline) | 0.6503 |\n| DiffPhyCon-lite | 0.2324 |\n| DiffPhyCon ($\\gamma$=0.96) | **0.2254**|\n\n>**Comment 2**: I have minor reservations about the novelty of the paper since the primary method relies on the existing diffusion model.\n\n**Answer**: \nOur method, while relying on diffusion models, introduces the following three key innovations:\n1. **New Application Area**: We leveraged diffusion models' advantages, such as ease of generalization and global optimization over time, for physical system control. This establishes a new application area for diffusion models.\n2. **Prior Reweighting Technique**: We addressed the challenge of generating control sequences that outperform those in the training set by introducing prior reweighting. This is a novel technique in the diffusion model literature, marking a significant technical contribution.\n3. **Theoretical Analysis**: During the rebuttal, we conducted additional theoretical analysis: for tasks where training set control sequences are far from optimal, using a 𝛾<1 in prior reweighting increases the probability of obtaining near-optimal control sequences compared to not using the technique (𝛾=1). This aligns with our intuitive explanation in the original manuscript and will be included in the final manuscript. Due to space limitation, for formal theoretical analysis, please refer to **Official Comment**.\n\n>**Comment 3**: Compared with other methods, does the proposed method use less, similar, or more time to train (for the learning-based methods) and inference? \n\n**Answer**: \nThe efficiency comparison for inference on two tasks was detailed in **Appendix I of the original submission**. Those resutls are combined with training time comparisons in the following Table 2 and Table 3. Here are the key points:\n- **Inference Time**: Our method is competitive among most methods, except SAC. By adopting the fast sampling method DDIM (DiffPhyCon-DDIM), our method's efficiency significantly improves, nearing the fastest models in the 1D task.\n- **Training Time**: The training time for our method is smaller (1D task) or comparable (2D task) to other learning-based methods.\n\nThese details will be included in the final manuscript.\n\nTable 2: Training and inference time comparison on 1D task.\n| Method | Training Time / hours | Inference Time / seconds |\n| - | - | - |\n| DiffPhyCon-lite | 1.7 (1 A100-80G GPU, 8 CPUs) | 21.13 |\n| DiffPhyCon | 4.4 (1 A100-80G GPU, 8 CPUs) | 58.97 |\n| DiffPhyCon-DDIM (8 sampling steps) | 1.7 (1 A100-80G GPU, 8 CPUs) | 0.53 |\n| BPPO | 8.9 (1 V100-32G GPU, 12 CPUs) | 0.82 |\n| BC | 8.8(1 V100-32G GPU, 12 CPUs) | 1.22 |\n| SAC | 10.5 (1 A6000-48G GPU, 16 CPUs) | 0.11 |\n| SL | 2.6 (1 V100-32G GPU, 12 CPUs) | 74.85 |\n\nTable 3: Training and inference time comparison on 2D task.\n| Method | Training Time / hours | Inference Time / seconds |\n| - | - | - |\n| DiffPhyCon | 62 (2 A100-80G GPUs, 32 CPUs) | 252.2 |\n| DiffPhyCon-DDIM (50 sampling steps) | 62 (2 A100-80G GPUs, 32 CPUs) | 12.6 |\n| BPPO | 3.0 (1 A100-80G GPU, 16 CPUs) | 1.1 |\n| BC | 2.8 (1 A100-80G, 16 CPUs) | 1.0 |\n| SL | 52.1 (1 A100-80G GPU, 16 CPUs) | 133.5 |\n| SAC | 9.5 (1 A100-80G, 16 CPUs) | 0.2 |\n| MPC | 52.1 (1 A100-80G GPU, 16 CPUs) | 1401.7 |\n\n\n>**Comment 4**. Is it easy or hard to extend this method to 3D systems? In the 2D example, a 3D U-Net is used. Does this make it hard to apply this method to 3D systems as it requires a much larger 4D network? \n\n**Answer**:\nOur method is agnostic to the neural network backbone of the diffusion model. For a 3D system, we can switch to an appropriate neural network like a 4D U-net.\n\nRegarding efficiency, controlling 3D complex physical systems is challenging for all current methods. Although autoregressive models require only a 3D neural network, they evaluate iteratively at each physical time step [1,4]. Our method improves temporal efficiency by generating the full trajectory simultaneously.\n\n>**Comment 5**. In the prior reweighting method, are 𝜖𝜃 and 𝜖𝜙 trained jointly, or is 𝜖𝜃 trained first and then 𝜖𝜙?\n\n**Answer**: These two models can be trained simultaneously as they are independent.\n\n\n\n[1]Solving pde-constrained control problems using operator learning, 2022.\n\n[2]Optimal control of PDEs using physics-informed neural networks (PINNs), 2021.\n\n[3]Learning to Control PDEs with Differentiable Physics, 2020.\n\n[4]Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZwU7j3JJh0", "reviewer": "Reviewer_DFgi", "summary": "The model proposed a variant of diffusion models to control complex dynamical systems. Its contributions are threefold. First, the proposed model could optimize the trajectory and control sequence simultaneously. Second, it proposed a reweighting prior technique to generate a superior control sequence to the training dataset. Third, it contributes a benchmark dataset, jellyfish movement control, for the complex dynamical control community.", "questions": "1. How is the offline dataset generated? Does it include the optimal control trajectory? What if the offline collected training control trajectory is too far from the optimal one? I am trying to understand if your model \"finds\" the existing trajectory in the dataset or if it \"stitches\" different pieces of trajoetiers and generates one that doesn't exist in the dataset.\n\n2. For the cases in this paper, using the online control algorithm is not too expensive. Online RL for these cases could also give global optimal control results. Could you compare the computational cost of your proposed method and an online RL method to demonstrate the benefits against the online RL algorithm?  \n\n3. For the diffusion-generated trajectory, how did you get Figure 4? Did you input the diffusion-generated trajectory to the numerical solver (Lilly-Pad) to get the physics simulation result? Or did you take the diffusion-generated flow field result as the \"physics trajectory\"? \n\n4. I didn't quite get the intuition behind Figure 2. Why does reducing $\\gamma$ to less than 1 shift the red point from margin to center?", "limitations": "The limitations are explicitly mentioned in the paper.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: The novelty lies in two parts. A variant of the diffusion model is proposed to minimize the energy function and control objectives simultaneously. Moreover, it proposes a prior reweighting algorithm to allow the sample to enhance the sampling of good but low-probability trajectories. \n\nClarity: It is clear in illustrating the methodologies and the experiments. \n\nQuality: It provides comprehensive details on the background of the model and dataset. The experiment results substantiate its claim. \n\nSignificance: The proposed method can potentially control complex systems with a generative model with reduced cost and good long-term control sequences.", "weaknesses": "Several questions need to be clarified; see details in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We are glad that the reviewer recognizes the novelty, clarity, results, and significance of our work. Below, we address the reviewer’s questions one by one.\n\n>**Comment 1**. How is the offline dataset generated? Does it include the optimal control trajectory? What if the offline collected training control trajectory is too far from the optimal one? I am trying to understand if your model \"finds\" the existing trajectory in the dataset or if it \"stitches\" different pieces of trajoetiers and generates one that doesn't exist in the dataset.\n\n**Anwser**: \nAs detailed in **Appendices C.1 and D of the original submission**, for our 1D and 2D data, we input initial conditions and control sequences with random variations into the solvers to generate trajectories. For instance, in the 2D case, control angles are cosine curves with varying amplitude, mean, and duty cycle. These random variations are necessary as expert control sequence data are hard to obtain for nonlinear and high-dimensional physical systems. Consequently, generated sequences are typically far from optimal.\n\nThis issue **motivates our prior reweighting technique**, which reduces the weight of prior control sequences to increase the probability of sampling near-optimal solutions. In this rebuttal process, our **additional theoretical analysis** shows that for tasks with suboptimal training set control sequences, using prior reweighting with 𝛾<1 increases the likelihood of obtaining near-optimal control sequences compared to not using it (𝛾=1). Due to space limitations, for details of the formal statement of the theorem and proof, please refer to **Official Comment** on top of this webpage.\n\nDespite the offline training control sequences not being optimal, **diffusion models can combine segments of these trajectories** through conditional generation or control target guidance, as illustrated in [1]. In our experiments, a close look at Figure 4 reveals that although the entire curve has not appeared in the training set, its segments resemble parts of a cosine control curve from the training set.\n\n>**Comment 2**: For the cases in this paper, using the online control algorithm is not too expensive. Online RL for these cases could also give global optimal control results. Could you compare the computational cost of your proposed method and an online RL method to demonstrate the benefits against the online RL algorithm?\n\n**Anwser**: \nFirst, online training of SAC on our 2D Jellyfish control case is so **computationally demanding** that it is hardly feasible. However, we conducted a new experiment using SAC on our 1D Burgers' control task. The results are shown below. Our method is about **twice as fast as SAC and performs better**.\n\n| Method | Training time / hour | $J_{actual},\\downarrow$ |\n| - | - | - |\n| DiffPhyCon (ours) | 4.4 | 0.01103|\n| DiffPhyCon-lite (ours) | 1.7 | 0.01139|\n| SAC-online | 10.5 | 0.01567 |\n| SAC-offline | 8.0 | 0.03201 |\n\nSecond, the cases in our paper are for evaluation purposes. Our method can be applied to more complex tasks, such as the **new high-dimensional indirect fluid control task** (see Figure 1 in the PDF of General Response), where online RL training is also infeasible here due to high costs. Moreover, online RL is impractical in many settings, like dangerous exploration (e.g., autonomous underwater vehicles) [2], whereas offline learning can avoid these issues. Thus, **our method has broader applications compared to online RL**.\n\n>**Comment 3**: For the diffusion-generated trajectory, how did you get Figure 4? Did you input the diffusion-generated trajectory to the numerical solver (Lilly-Pad) to get the physics simulation result? Or did you take the diffusion-generated flow field result as the \"physics trajectory\"? \n\n**Anwser**: \nFigure 4 shows the control results of three jellyfish, each with different initial conditions. Control sequences obtained from each method are input into Lilypad solver to simulate the flow field trajectories, based on which we calculated the average movement speed and control objective $J$.\n\nTherefore, **we did not use the flow field trajectories generated directly by the diffusion model as the \"physical trajectory\"**. The reason for this is that the evaluation of these control methods, both our method and the baselines, should be based on simulated trajectory of the same solver, to compare them in a fair footing. **Details are provided in Appendix E.4 (Line 853 to Line 864)  of our original submission.**\n\n>**Comment 4**: I didn't quite get the intuition behind Figure 2. Why does reducing 𝛾 to less than 1 shift the red point from margin to center? \n\n**Anwser**: We apologize for any confusion. In the figure, the red dot at the margin indicates a local minimum of $J$, while the red dot at the center represents the global minimum. In the joint distribution $p(u,w)=p(w)p(u|w)$, we assume the global minimum of $J$ has a lower probability than the local minima, as global optimal trajectories rarely appear in the training dataset. By setting 𝛾<1 and then normalizing by a constant $Z$, the distribution $p(w)^\\gamma p(u|w)/Z$ flattens $p(u,w)$, increasing the probability of sampling at the global minimum. Thus, the figure illustrates this ideal situation: with 𝛾=1, we sample a local minimum by using DiffPhyCon-lite; with 𝛾<1, we sample the global minimum. However, this ideal scenario is only for intuitive illustration. Rigorous theoretical analysis shows that with 𝛾<1, the probability of sampling near the global minimum increases compared to 𝛾=1. Due to space limitations, for details of the formal theoretical analysis, please refer to **Official Comment** on top of this webpage.\n\n[1] Janner, et al. Planning with diffusion for flexible behavior synthesis., ICML 2022.\n\n[2] Levine, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv:2005.01643, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGy9fBOavQ", "reviewer": "Reviewer_n6Yf", "summary": "This paper introduces Diffusion Physical Systems Control (DiffPhyCon), where diffusion models are used to generate a near-optimal controller for a system described by a partial differential equation (PDE). In this generative approach, a learned generative energy function and control objective is minimized. Additionally, a prior reweighting technique is developed to mitigate the effect of prior distribution of control sequences.", "questions": "Besides the big question of novelty in comparison to the \"Generative PDE Control\" paper from ICLR, I have the following comments for the authors:\n\n1. The authors acknowledge the limitation that DiffPhyCon presently operates in an open-loop manner. This limitation by itself is not a problematic, but the claims made in the introduction need revision in that case. Due to robustness concerns, a controller is seldom implemented open-loop even for simple systems, let alone complex physical systems. However, open-loop optimal control design can be applied to motion planning or trajectory generation objectives. In the context of this paper, the generative approach could be used to construct a nominal pre-planned open-loop control sequence term which can then be added to a feedback term (e.g. PID). Thus, I suggest to revise the introduction to claim planning as the objective. For more information about motion planning, the authors are referred to Chapter 12 (Motion Planning for PDEs) from the reference:\n\nKrstic, M. and Smyshlyaev, A., 2008. Boundary control of PDEs: A course on backstepping designs. Society for Industrial and Applied Mathematics.\n\n2. In line 72 of page 2, the authors claim \"...DiffPhyCon facilitates global optimization of long-term dynamics...\". It is not clear how the globality (or even non-triviality) of near-optimal control sequence is guaranteed to be achieved. I am not sure why Algorithm 1 would not yield local optima.", "limitations": "Yes, the authors have stated the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and the method is well-motivated. The performance of the developed generative approach is demonstrated using extensive simulation studies on 1D Burgers equation and 2D Jellyfish movement control. The efficiency of this method is compared with a decent number of baselines. Additional experiments are provided to study the effects of hyperparameters.", "weaknesses": "The contribution of the paper is not clear compared to the following recently published result in ICLR: \n\nWei, L., Hu, P., Feng, R., Du, Y., Zhang, T., Wang, R., Wang, Y., Ma, Z.M. and Wu, T., Generative PDE Control. In ICLR 2024 Workshop on AI4DifferentialEquations In Science.\n\nThe method looks identical to the generative PDE control result in the above reference which is published and peer-reviewed. The additional contribution, if any, is too minor for NeurIPS. For this reason, I cannot recommend the paper to be accepted for NeurIPS. However, I do think this paper would have been a contribution to NeurIPS otherwise.\n\n-----\nUPDATE: It appears the ICLR workshop is not considered an archival publication. With this consideration, I update my score to 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and helpful suggestions. We are pleased to hear that the reviewer finds our paper well-written, well-motivated, and extensive in its results. Below, we address the reviewer’s questions one by one.\n\n>**Comment 1**:  ... Thus, I suggest to revise the introduction to claim planning as the objective. For more information about motion planning, the authors are referred to Chapter 12 (Motion Planning for PDEs) from the reference:\nKrstic, M. and Smyshlyaev, A., 2008. Boundary control of PDEs: A course on backstepping designs. Society for Industrial and Applied Mathematics.\n\n**Answer**: Thanks for this constructive suggestion. Based on your advice and the discussions in the referenced literature, our problem setting indeed aligns more closely with a planning task, where future multi-step actions are generated at once. We will revise the introduction and other relevant sections of the paper to **change the keyword \"control\" to \"planning\"**.\n\n>**Comment 2**: In line 72 of page 2, the authors claim \"...DiffPhyCon facilitates global optimization of long-term dynamics...\". It is not clear how the globality (or even non-triviality) of near-optimal control sequence is guaranteed to be achieved.\n\n**Answer**: We apologize for not clearly explaining global optimization. By this, we mean treating the state trajectory and control sequences **over all physical time steps** as a single variable during diffusion/sampling. This approach is chosen because control objectives are typically defined over the entire time span. In particular, when the objective function includes several conflicting terms, our method can **relieve the myopic failure modes** [1] that may exist in reinforcement learning's iterative decision-making. For instance, in our 2D control task, maximizing the jellyfish's average speed requires sharp angle changes in the early stage, conflicting with minimizing the total energy cost $R(w)$. More details are in **Lines 302-307 and Appendix H.7** on myopic failure modes of SAC **in our original submission**.\n\n>**Comment 3**: I am not sure why Algorithm 1 would not yield local optima\n\n**Answer**: We do not guarantee that Algorithm 1 yields local optima. However, our **further theoretical study during rebuttal** shows that **prior reweighting** (Line 6 of Algorithm 1) **enhances the probability of obtaining global near-optimal solutions**. Informally speeking, for tasks with suboptimal training control sequences, there exists a hyperparameter $\\gamma < 1$ such that using prior reweighting with this $\\gamma$ increases the likelihood of obtaining near-optimal control sequences compared to not using this technique ($\\gamma = 1$). Due to space limitations, for details of the formal statement of the theorem and proof, please refer to **Official Comment** on top of this webpage.\n\n[1] Janner, et al. \"Planning with diffusion for flexible behavior synthesis.\", ICML 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f5i0m0Ca2d", "reviewer": "Reviewer_Mp9v", "summary": "This paper proposes an algorithm for controlling complex physical systems, particularly in long-term settings. The method is based on diffusion models and energy methods, utilizing data generated by traditional finite difference methods.", "questions": "* What is the energy term $E_\\theta(u, w, c)$?\n\n* In the 2D Jellyfish task, what is the value of the kinematic viscosity $\\nu$?\n\n* Is it possible to make the proposed algorithm independent of classical methods (e.g., the finite difference method used in this paper) for data generation?", "limitations": "This paper has addressed its limitations in the main text.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The problem studied in this paper is important and well-motivated.\n\n* The presentation is clear and easy to follow.\n\n* Detailed experimental results of the proposed algorithm are presented in both main text and appendices.", "weaknesses": "* The method proposed in the paper does not demonstrate its strength in challenging tasks. I agreed with the claim that the fundamental challenge of simulating complex physical systems, which are high-dimensional and highly nonlinear. However, the current paper does not address these problems convincingly for the following two reasons: \n\n* First, both numerical examples are low-dimensional so that the curse of dimensionality does not affect much. \n\n* Second, their dynamics are not very complex. At first glance, the 2D Jellyfish task appears complex, as it involves solving the 2D Navier-Stokes equation, which might yield turbulent solutions. However, I realized that the optimization focuses on the jellyfish itself. I was wondering if there is a chance to obtain a good solution even if the simulation of fluid dynamics is not accurate. If this is the case, the results of this example may not provide strong evidence that the algorithm can handle complex dynamics. Additionally, this paper does not present the error analysis for the fluid dynamics simulation, only stats of energy comparison are provided.\n\n* This algorithm depends on the data generated by finite difference methods, hence suffering from the curse of dimensionality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed comments. We are glad that the reviewer finds our work clear and well-motivated with detailed results. Below, we address the reviewer’s questions one by one.\n\n> **Comment 1**: ...The current paper does not address these problems convincingly for the following two reasons:\n>\n>-- First, both numerical examples are low-dimensional so that the curse of dimensionality does not affect much.\n>\n>-- Second, their dynamics are not very complex. ...I was wondering if there is a chance to obtain a good solution even if the simulation of fluid dynamics is not accurate. If this is the case, the results of this example may not provide strong evidence that the algorithm can handle complex dynamics. Additionally, this paper does not present the error analysis for the fluid dynamics simulation, only stats of energy comparison are provided.\n  \n**Answer**: \nOur tasks align with existing data-driven physical control papers [1-3]. Controlling Burgers' Equation is challenging due to shock wave tendencies, and our setup is more difficult with partial observation/control combinations. The 2D jellyfish control task includes challenges such as (1) asymmetric wake vortex formation from symmetric structures and motion modes [4]; (2) the nonlinear, complex interaction of vortices with the shell [5].\n\nStill, to make our experiments more convincing, we **conducted new 2D incompressible fluid control experiments**, following a similar (but more challenging) setup of [3]. Control forces were applied within a 64x64 grid flow field, excluding a semi-enclosed region, to minimize the smoke failing to exit through the top middle exit (seven exits in total). For illustration of settings, please refer to subfigures (a) and (b) of Figure 1 in the PDF file at the end of **General Response**. This **high-dimensional indirect control problem** involves managing 2-dim forces at approximately 1,700 grid points per time step, resulting in about **100,000 control variables in total** across 32 time steps, making it highly challenging. We generated 20,000 training trajectories (including features of the smoke density field, velocity fields, control force fields and smoke proportion field) after filtering low quality trajectories. The average control objective in the training set is 49.9%. We evaluated 50 test samples, and the results are shown in the following Table 1. It shows that **our method still has significant advantages**, especially in validating the role of prior reweighting. **The average relative L2 error of fluid dynamics simulation of our method is 19.2%**. In particular, **the curse of dimensionality is effectively addressed by our method**. For **visualization** of the generated control and density map of smoke, please refer to subfigure (c) of Figure 1 in the PDF file at the end of **General Response**. These updates will be added to the final version of our manuscript.\n\nTable 1: Performance comparison on the new 2D indirect fluid control task.\n| Method | Control Objective $\\downarrow$|\n| - | - |\n| BC | 0.3085|\n| BPPO | 0.3066|\n| SAC (pseudo-online) | 0.3212|\n| SAC(offline) | 0.6503|\n| DiffPhyCon-lite | 0.2324|\n| DiffPhyCon ($\\gamma$=0.96) | **0.2254**|\n\n>**Comment 2**: This algorithm depends on the data generated by finite difference methods, hence suffering from the curse of dimensionality; \n>\n>**Comment 3**: Is it possible to make the proposed algorithm independent of classical methods (e.g., the finite difference method used in this paper) for data generation?\n\n**Answer**: \nOur method does not depend on the data generated by finite difference methods. Our method belongs to the domain of deep learning-based surrogate modeling, and like any other methods in this domain, is **decoupled from the data-generating process**. The data can come from classical solvers (not limited to finite difference methods) or actual physical observations, e.g., 1D data from finite difference methods while 2D data from the finite volume method [6] in our paper.\n\nMoreover, like any other deep learning-based surrogate modeling method, the goal is exactly to deal with complex, high-dimensional processes. It is exactly our generative control method that can better learn the high-dimensional dependencies from the given data.\n\n>**Comment 4**:What is the energy term $E_\\theta (u,w,c)$?\n\n**Answer**:  It is the parameterized energy-based model of the joint distribution of $p(u, w, c)$, where $w$ is the control sequence, $u$ is the state trajectory, and $c$ is the set of conditions, like initial conditionals and boundary conditions. $E_\\theta(u,w,c)=-\\log(p(u,w,c))+\\text{const.}$ characterizes $p(u,w,c)$. Given $c$, we aim to generate $(u,w)$ that lies on the data manifold with high probability, with the optimal objective $J$. To achieve this goal, we train a diffusion model $\\epsilon_\\theta$ to approximate $\\nabla E_\\theta(u,w,c)$. During inference, the sampling starts from a Gaussion noise $(u_K, w_K)$ and travels along $\\nabla E_\\theta(u,w,c)$ by applying $\\epsilon_\\theta$ iteratively to achieve a final sample $(u_0,w_0)$ with near-minimal energy $E_\\theta(u_K, w_K, c)$, under the guidance of $J$.\n\n>**Comment 5**: In the 2D Jellyfish task, what is the value of the kinematic viscosity ν?\n\n**Answer**: In our paper, ν is set to 0, considering an inviscid fluid, similar to other flow field control papers [3]. Our method can also be applied to viscous fluids and does not restrict the magnitude of ν.\n\n[1]Solving pde-constrained control problems using operator learning,2022.\n\n[2]Optimal control of PDEs using physics-informed neural networks (PINNs), 2021.\n\n[3]Learning to Control PDEs with Differentiable Physics, 2020.\n\n[4]Stable hovering of a jellyfish-like flying machine, 2014.\n\n[5]Propulsive performance and vortex dynamics of jellyfish-like propulsion with burst-and-coast strategy, 2023.\n\n[6]Conservative Volume-of-Fluid method for free-surface simulations on Cartesian-grids, 2010.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We thank the reviewers for their thorough and constructive comments, as well as AC's organization in reviewing our paper. 
We are glad that the reviewers think that our paper is **well-written** (DFgi, n6Yf, Mp9v) and **well-motivated** (n6Yf, Mp9v). Reviewers 8Er5 and DFgi recognized the **novelty of our paper**: a generative method to control complex physical systems, and a prior reweighting technique (8Er5 and DFgi). Reviewers appreciate the **strong results of the jellyfish systems** (8Er5) and the **comprehensive details of experiments** (DFgi, n6Yf, Mp9v). Reviewer 8Er5 also recognizes the **importance of our contributed dataset** for the jellyfish system. 

Based on the reviewers’ valuable feedback, we have **conducted several additional experiments** and **made additional theoretical analysis**. Below, we address the issues pointed out by the reviewers and resolve possible misunderstandings: 
1. We **present an additional theoretical analysis of the prior reweighting technique** to illustrate its effectiveness. The theoretical conclusion is: for control problems where control sequences in the training set are far from optimal, there exists a $\gamma<1$ such that using the prior reweighting technique with this  $\gamma$ shows improvement compared to not using it. For **details of the formal statement of the theorem and proof**, please refer to **Official Comment**.
2. We **introduce a new 2D incompressible fluid control task** to further demonstrate the effectiveness of our method, in response to Reviewer 8Er5 and Mp9v. This is a **high-dimensional indirect** control task, thus very challenging. Our method still shows **significant improvement compared to those baselines**. For details, please refer to the responses to Reviewers 8Er5 and Mp9v. For **visualization** of the generated control fields and the generated smoke density field, please refer to **Figure 1 in the attached PDF file**.
3. We **further clarify our contributions**, in response to Reviewer 8Er5's minor reservations about the novelty of the paper. Our contributions on top of the diffusion models include **three aspects**: expanding the application of diffusion models, proposing a novel prior reweighting technique, and providing theoretical analysis of the technique. For details, please refer to the responses to Reviewer 8Er5.
4. We **add online RL results of the 1D control task**. The results show that our method has higher training efficiency and outperforms online SAC. For details, please refer to the responses to Reviewer DFgi.
5. We **add a comparison of the training efficiency** of DiffPhyCon and those learning-based baselines. For details, please refer to the responses to Reviewers 8Er5.
6. We **agree to revise the Introduction and others related sections**, as suggested by Reviewer n6Yf. We will change the keyword "control" to "planning".

**The above updates will be presented in our final manuscript**. We now individually address the concerns of reviewers. Please see the responses below each review.

Meta-review: The paper presents a new use of diffusion models for open loop control
(planning) in physical system models.  It observes one can adjust the
weighting in the energy function to de-emphasize the prior probability in
the data in favor of the control objective.  This allows the result to
outperform the demonstrations.  Some issues remain unaddressed such as the
case of a noisy system and/or an inaccurate dynamics model, which would
require actual closed loop control.  Nonetheless, the idea is interesting
and the empirical results show a nice improvement over other methods.
Decision: Accept (poster)