Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings
Authors: Hongpeng Cao
Yanbing Mao
Lui Sha
Marco Caccamo
Venue: ICLR 2024 spotlight (2024)
Keywords: Physics-informed deep reinforcement learning
Safety-critical autonomous systems
Primary area: reinforcement learning
TL;DR: Physics-regulated DRL

Abstract: This paper proposes the Phy-DRL: a physics-regulated deep reinforcement learning (DRL) framework for safety-critical autonomous systems. The Phy-DRL has three distinguished invariant-embedding designs: i) residual action policy (i.e., integrating data-driven-DRL action policy and physics-model-based action policy), ii) automatically constructed safety-embedded reward, and iii) physics-model-guided neural network (NN) editing, including link editing and activation editing. Theoretically, the Phy-DRL exhibits 1) a mathematically provable safety guarantee and 2) strict compliance of critic and actor networks with physics knowledge about the action-value function and action policy. Finally, we evaluate the Phy-DRL on a cart-pole system and a quadruped robot. The experiments validate our theoretical results and demonstrate that Phy-DRL features guaranteed safety compared to purely data-driven DRL and solely model-based design while offering remarkably fewer learning parameters and fast training towards safety guarantee.

Published as a conference paper at ICLR 2024
PHYSICS-REGULATER DEEP REINFORCEMENT LEARN- ING: INVARIANT EMBEDDINGS
Hongpeng Cao', Yanbing M Lui Sha & Marco Caccamo' TUU, Germany, PWSU, United States, $UIC, United States
ABSTRACT
This paper proposes the Phy-DRL: a physics deep reinforcement learning (DRL) framework for safety-critical autonomous systems. The Phy-DRL has three distinguished invariant-embedding designs: i) residual action policy (i.e., integrating data-driven- DDRvenn-RRLL- action policy and physics-modes based action policy), automatically constructed sabety-embedded reward, and iii) physics -moel- guided neural network (NN) editing, including link editing and activation editing. Theoretically, the Phy-DRL exhibits 1) mathematically provable safety guarantee and 2) strict compliance of critic and actor networks with physics knowledge about the action-value function and action policy. Finally, we evaluate the Phy-DRL on a cart-pole system and a quadruped robot. The experiments validate our theoretical results and demonstrate that Phy-DRL features guaranteed safety compared purely data-driven DRL and solely model-based design while offering remarkably fewer learning parameters and fast training towards safety guarantee.
INTRODUCTION
1.1 MOTIVATIONS
Machine learning (ML) technologies have been integrated into autonomous systems, defining learning- enabled autonomous systems. These have succeeded tremendously in many complex tasks with high-dimensional state and action spaces. However, the recent incidents due the deployment of ML models overshadow the revolutionizing potential of ML, especially
the safety-critical autonomous systems Zachary & Helen (202T). Developing safe ML is thus more vital today. In the ML community, deep reinforeement learning (DRL) has demonstrated breakthroughs in sequential decision-male making in broad arcas, ranging from autonomous driving Kendall et al. (2019) to games
Silver et al. (0188) This motivates us to develop DRL-b safe learning framework for achieving safe and complex tasks of safety-critich autonomous systems
Despite the tremendous suc- cess of DRL in many autonomous systems for complex decision-making applying DRL to safety- critical autonomous sys- tems remains a challenging problem. It has a deep root to the action policy of DRL being parameterized by deep neural networks (DNN), whose behaviors hard to predict [Huang
Phy Agent
Critic Networkk
Actor Networkk
policy 4k)
Mate a(L)
Embedding Physics- Koollc--nowweggeeeoo Enhanced DNN
Environment
Imcriant Embelding
action a(k)
Resigual Action Pollcy: a(k) - a. (k) ag
NN Input Augmentation
setion
limarizzd
apey
modil
Inveriant Embedding
Physics Model Knowledge:
Safety Embedded
et al. a and verify Katz
Phy-DRL Framework, applied to quadruped robot. safety concern. The second safety concern stems from the purely data-driven DNN that DRL adopts
al. 2017), raising the first
Figure 1:
"Equal contribution.
1 "Correspondence Yanbing Mao: macyanbl hg..eth@mma com.

---

Published as a conference paper at ICLR 2024
for powerful function approximation and representation learning of action-value function, action policy, and environment states Mnih et al. (2015); [iiver et al. (2016). Specifically, recent studies revealed that purely data-driven DNN applied to physical systems can infer relations violating physies laws, which sometimes leads to catastrophic consequences (e.g., data-driven blackout owning to violation of physical limits Zachary & Helen (202T).
1.2 CONTRIBUTIONS
To address the aforementioned safety concerns, we propose the Phy-DRL: physics-regulated deep reinforcement learning framework with enhanced safety assurance. Depicting in Figure I Phy-DRL. has three novel (invarian-embeedding)) architectural designs:
Residual Action Policy, which integrates data-driven- DRL action policy and physics-modee based action policy.
Safety-Embeddee Reward, in conjunction with the Residual Action Policy, empowers the Phy-DRL with mathematically provable safety guarantee and fast training.
Critic and Actor Networks, whose neural architectures have two key components: i) NN input augmentation for directly capturing hard-to-learn features, and ii) NN editing, including link editing and activation editing, for guaranteeing strict compliance with available knowledge about the action-value function and action policy.
1.3 RELATED WORK AND OPEN PROBLEMS
Residual Action Policy. The recent research on DRL for controlling autonomous systems has shifted to integrating data-driven DRL and model-based decision making, leading to residual action policy diagram. In this diagram, the prodel-based action policy can guide the exploration of DRL agents during training. Meanwhile, the DRL policy learns to effectively deal with uncertainties and compensate for the model mismatch of the model-based action policy. The aims of existing residual frameworks Rana et al.; Li et al. (a): Cheng et al. (2019b); Johannink et al. (2019) mainly focus stability guarantee, with the exception being Cheng et al. (2019a) Moreover, physics models are nonlinear, which poses difficulty in empowering analyzable and verifiable behavior. Furthermore, the model knowledge has not yet been explored to regulate the construction of DRL towards safety guarantee. In summary, the open problems in this domain are
safety guarantee.
Problem 1.1. How to design a residual action policy to be a best-trade-off between the model-based action policy and the Lata-driven-DRR action policy?
Problem 1.2. How can the knowledge of the physics model be used to construct a DRL's reward towards a safety guarantee?
Reward. A safety-embedded reward is crucial for DRL to search for policies that are safe. To achieve a safety guarantee, control Lyapunov function (CLF) is the potential
safety-embedded reward Perkins & Barto (2002); Berkenkamp et al. (2017); Chang & Gao (202 I); Zhhao et al.] (2033 2024). Meanwhile, the seminal work Westenbroek et al. (002) discovered that if DRL's reward CLF-like, the systems controlled by well-trained DRL policy can retain mathematically-prove
stability guarantee. Moving forward, the question of how to construct such a CLF-like reward for achieving safety guarantee remains open, i.c..
Problem 1.3. What is the systematic guidance for constructing the safety-embedded reward (e.g.. Cik-like reward) for DRL?
Bhowledge-Enhaneee Neural Networks. The critical flaw of purely data-driven DNN, i..., violation
of physics laws, motivates the emerging research on mayyscc--nhannee
DNN, Current frameworks
include onysics-informed NN Wang & Yul Willard et al. (2021) ): Jia et al. (021)) ): Lu et (202T):
Chen et al. (202T); ; Xu & Darve (2022); Karniadakis et al. (2021) >; Wang et al. (2020); Cranmer
ct al. (2020) and physics-guided NN architectures Muralidhar et al. ; Masci et al. (015)):
Monti et al. (2017): Horie et al.; Wang (2021); Li et al. (b). Both use compact partial differential equations (PDEs) for formulating training loss functions and/or architectural components. These frameworks improve consistency degree with prior physics knowledge but remain problematic in applying to DRL. For example, we define DRL's reward in advance. The critic network of DRL is to
learn or estimate the expected future reward, also known as the action-value function. Because the
2

---

Published as a conference paper at ICLR 2024
action-valus function involves unknown future rewards, its compact governing equation is unavailable for physics-innone informed networks and about the value function and action policy is available, which thus motivates the open problem: Problem 1.4. How do we develop end-to-ent critic and actor networks that strictly comply with partially available physics knowledge about the action-value function and action policy?
architectures. In summary, only partial knowledge
1.4 SUMMARY: ANSWERS TO PROBLEM 1.1 PROBLEM T.4
The proposed Phy-DRL answers Problem Problem simultancously. As shown in Figure T the residual diagram of Phy-DRL simplifies the model-based action policy to be an analyzable and verifiable linear one, while offering fast training towards safety guarantee. Meanwhile, the linear model knowledge (leveraged for computing model-based policy) works as model-baset guidance for constructing the safery-embedder reward for DRL towards mathematically provable safety guarantee. Lastly, the proposed NN editing guarantees the strict compliance of critic and actor networks with partially available physics knowledge about the aetion-value function and action policy.
PRELIMINARIES
Table in Appendix A summarizes notations that are used throughout the paper.
2.1 DYNAMICS MODEL OF REAL PLANT
The generic dynamics model of a real plant can be described by
s(k + 1) = A s(k) + B a(k) + ((k))) ke
(I)
where ((s(k), ak)) E R the unknown model mismatch, Raxa and E Raxm denote known system matrix and control structure matrix, respectively, s(k) R" is the system state, a(k) 1T is the applied control action. The available model knowledge pertaining to real plant (I) represented by (A, B).
2.2 SAFETY DEFINITION
The considered safety problem stems from safety regulations or constraints on system states, which motives the following definition of safety set X.
Safety Set: X A {
D DER V.V.VER))
(2)
where D, and v given in advance for formulating h safety conditions. Considering the safety set, we present the definition of a safety guarantee.
Definition 2.1. Consider the safety set X in Equation 2 and its subset Ω. The real plant T is said
to be safety guaranteed, if for any s(1) E Ω2 C X, then s(k) E C X, Ak > N.
Remark 2.2 (Role of (2). The subset Ω called the safety envelope, whose details will be explained in Section The Ω will bridge many (i.e., high-dimensional) safety conditions in safety Equation ( and oinedimnnsi guarantee means the Phy-DRL successfully searches for policy that renders Ω invariant (i.e..
embedded reward. Mcanwhile, Definition 2.1 indicates that safety
operating from any initial sample inside Ω2 system state never leaves Ω at any time).
DESIGN OVERVIEW: INVARIANT EMBEDDINGS
In this paper, an 'invariant' refers to a prior policy, prior knowledge, or designed property indepen- dent of DRL agent training. As shown in Figure the proposed Phy-DRL can address Problem [.I-- Problem because of three -embedding designs. Specifically. 1) residual action policy, which integrates data-driven action policy with an invariant model-based action policy that completely depends on prior model knowledge (A, B). ii) Safety-embedded reward, whose off-line-designed inequality (shown in Equation (8) for assistance in delivering the mathematically provable safety guarantee is also completely independent of agent training. iii) Physics- www.lldge-enanncee. DNN, whose NN editing embeds the prior invariant knowledge about the action-vaile function and action policy to the critic and actor networks, respectively. The following Section 4 Section (5 and Section detail the three designs, respectively.
3

---

Published as a conference paper at ICLR 2024
4 INVARIANT EMBEDDING 1: RESIDUAL ACTION POLICY
Showing in Figure ! the applied control action akk) from Phy-DRI is given in the residual form:
k)
adnn +
Aphy(k)
(3)
data-driven
invariant: model based
where the (k) denotes date-driven action from DRL, while the aphy (k) is model-based action, computed according to the invariant policy:
Bphy (k) F s(k),
where the computation of F is based on the model knowledge (A, B), carried out in Section 5
The developed Phy-DRL is based on actor-critic architecture in DRL et al. (2016); Schulman et al.; HHarnoja et al. (018) for searching an action policy (kr π (s(k)) that maximizes the expected return from the initial state skk):
(*((s)), add (k)) = Es(k)>p, (k)
Σ r- (s(1), aan) Link
(5)
where represents the initial state distribution, R(-)) maps state-action to a real-value reward, and action policy (-.) are parameterized by
y E [0, 1] the discount factor. The expected return (5)
the critic and actor networks, respectively.
5 INVARIANT EMBEDDING 2: SAFETY-EMBEDDEE REWARD
The current safety formula (2) is not ready for constructing the safety-embedded reward yet, since has multiple safety conditions while the reward R (-) in Equation (5) is one-dimensional. To bridge the gap, we introduce the following safety envelope, which converts multi-dimensional safety conditions into scalar value.
Safety Envelope: Q é { R2 P 0}
(6)
The following lemma builds a connection: safety envelope is subset of the safety set (also required in Definition 2. Its formal proof appears in Appendix C
Lemma 5.1. Consider the sets defined Equation
(2) and Equation (6)
We have 12 X. if
D
P
D
< and [D,, PP- -P-1.
[M._DT] [ [D]..-
- 221, [d] =1
11, [d]. --1 1E,1, h} (7)
where D
D D
and d, A and A are defined below for i,j E {1,2...., h}:
N'
A
0,
iØj [v+v] >0
0,
iØj
y+v| >0
[+
v++
[v+v],> >0
[d]: A
1, v+v], <0, A
[A].
+ 5. [v+v], <
v
[v+v], <0
1,, otherwise
otherwise
otherwise
Referring to model knowledge (A, B), Equation 6 and Equation (4. the proposed reward is
R(s(k), and (k:) = s (k) A P A s(k) - s (k + 1) P s(A 1) + w(s(k), am (k)), (8)
r(((k), u(k++))): invariant pruperty P - X PPA 0
where the sub-reward r(s(k), + 1)) is safety-embedded, and we define:
A A
(9)
(8) the safety-embedded sub-reward r(s(k), s(A + 1)) is critical for keeping a system safe, such as avoiding car crashes, and preventing car sliding and slipping in icy road. The sub-reward w(s(k), (k)) aims at nigh-performance operations, such as minimizing energy consumption of resource-limine limited robots Yang et al. (2022), which can be optional in some time- and safety-critical environments.
Remark 5.2 (Sub-rewards). In Equation
4

---

Published as a conference paper at ICLR 2024
Moving forward, present the following theorem, which states the conditions matrices F, P and reward for safety guarantee, whose proof is given in Appendix [.I]
Theorem 5.3 (Mathematically Provable Safety Guarantee). Consider the safety set X (2), the safety envelope Q2 10,, and the system
7 under control of Phy-DRI. The matrices F and P involved in the model-basee action policy ( and the safety-embeddee reward (8) are computed according to
F F-R-Q- P P=Q
(10)
where R. and Q-1 satisfy the inequalities 7
and
a a.Q Q A A.QBB.R
Q.A .
00, with a given a (0, 1).
(11)
Q
Given any s(1) E St the system state s(k) E S S X holds vk N (i.e., the safety of system (1) is guaranteed), if the sub-reward r(s(k), s(k - 1)) in (8) satisfies satisfies r(s(k), s(k 1)) 2 a a- Vk E N.
Remark 5.4 (Solving Optimal R. and Q). Equation (4), Equation (8) and Equation ( imply that the designs of model-based action policy and safety-embedded reward equate the computations of F and P. While Equation (T0) shows the computations depend on R and Q only. So, the remaining work is obtaining R and Q There inequalities (L.MIs) 7 and (TT), such as MATLAB's s LMI Solver [oyd et al.] (1994). What we are more interested in is finding optimal R and Q that can maximize the safety envelope. We note the volume of safety envelope (6) is proportional
multiple toolboxes for solving R and Q from linear matrix
det (P-1), the interested optimal problem is thus typical analytic centering problem, formulated as given a E (0,1), 1),
arg min {log det (Q-1)- arg max {log det (P-1)) subject to LMIs 7 and (11 Q, R
(12)
Q. R
from which, optimal R and Q can be solved via CVX toolbox Grant et al. 2009
Remark 5.5 (F is given). Equation (12) also works in the scenario of a given model- action policy (i.e., F), which is carried out in Section 7.2 as an example.
Remark 5.6 (Provable Stability Guarantee). Following the same proof path of Theorem 5.3 Phy-
DRL also exhibits the mathematically provable stability guarantee, which is presented in Appendix E
Remark 5.7 (Fast Training). The proof path of Theorem 5.3 is leveraged to reveal the driving factor of DRL's fast training towards safety guarantee, which presented in Appendix [D.2]
Remark 5.8 (Obtaining (A, B)). For a system with an available nonlinear dynamics model, the model knowledge (A, B) can be directly obtained by simplifying the nonlinear model to linear one. While for a system whose dynamics model is not available, (A, B) can obtained via system
identification Oymak & Ozay 2019), as used in social systems Mao et al. (0222
6 INVARIANT EMBEDDING 3: PHYSICS- WOWW
DNN
The Phy-DRL is built on the actor- critic architecture, where a critic and actor network are used to approximate the action-value function (i.e., Q (s(k), aun (k)) in Equation (5)) and learn an action policy (i.e., (ka(k)
[a) -nosles-Knowwedge Enhanced DNN
Physice-Model-Guier NN Editing
x
PhyN
Yis
PhyN
Yg-13
PhyN
Fin
(s(k))),
(b) PhyN Architecture
respectively. We note from Equa- tion (5) that the action-value func- tion is a direct function of our defined reward but involves un- known future rewards. So, some invariant knowledge exists that the critic and/or actor networks shall strictly comply with, which moti- vates Problem 1.4] To address this problem, shown in Figure we develop physics-enhama critic and actor networks for Phy-DRL. The proposed networks are built on physics-know knwwledge knowledge-enhanned enhanced DNN, whose architecture is depicted in Figure The DNN has two innovations in neural architecture:
Yig
MN Input
Augmentation
Link Editing
Activation Editing
Augmentation Order
Editing
Figure 2: Koossic-Knowllede-FEnenea
DNN architecture.
5

---

Published as a conference paper at ICLR 2024
i) Neural Network (NN) Input Augmentation described by Algorithm and ii) Physics-Model- Guided NN Editing described by Algorithm 2
To understand how Algorithm I and Algorithm 2 address
Problem 1.4, we describe the ground-truth models of action-value function and action policy
(s, aar))
Ag
m(s, adri ro)
+ p(s, adr)
E R
(13)
weight matrix
mode represeniation vecior
unknown model mismattch
I (S)
m(s,
p(s)
E (elen(((s)),
(14)
weight matrix
mode-ceresenttaiion vector
unknown model mismatch
where the vectors m(s, adri Tq) and m(s, r>) are, respectively, augmentations of input vectors [s; and and s, which embraces all the non-missing and non-redundant monomials of Taylor series. One motivation behind the augmentations is that according to Taylor's Theorem in Appendix G the Taylor series can approximate arbitrary nonlinear functions with controllable accuracy via controlling series orders: rg and Fm- The second motivation our proposed -mbeeded reward r(s(k), s(k: + 1)) in Equation (8) is typical Taylor series and is pre-defined. If using the Taylor series to approximate the oction-valut function and an action policy, we can also discover hidden invariant knowledge.
Algorithm needs inputs of knowledge sets Ko and K., which include available knowledge about governing Equation (13) and Equation (14), respectively. The two knowledge sets are defined below.
Ke è {[Ae], no [m(s, aun, re)l in p(s, ami), i E (Inmm,,,een(m(,, aus, rel)? }, (15) K, - (1([AT))) no [m(s.)=)), in [p(s)],, iE (en(()))en((s))))
len(mss, r=))]]- (16)
Remark 6.1 (Toy Examples: Obtaining Knowledge Sets). Due to the page limit, an example for obtaining Kg via Taylor's theorem is presented in Appendix H. This example is about obtaining K3 According to the dynamics and control of vehicles, the throttle command of traction control for preventing sliding and slipping depends on longitudinal velocity (denoted by v) and angular velocity
(denoted by w) only Rajamani 2011 : Mao et al. 2023). For simplification, we let = [U, w, <l where ( denotes yaw. While action policy (s) E F. with [ (s)]1 and [it (s)]2 denote the throttle command and steering command, respectively. By Algorithm with r(e) = -r>
and y(o) = s,
we have m(s rx) [1, U w. 2², uw, Uc, w² "C, S Considering Equation (1).. the m(s, rx). in conjunction with the knowledge "[π (s)]ı depends on w and v only", leads to the information: 1) [p())] in Equation (14) in this example does not have monomials: 1, mC. wC, and ç², and 2)
Aa
0
W1
W/2
0
W3
W4
0
W5
0
0
We
W7
WFo0
W11
WF1
W13
W14
W11
where W1, Wi5 are learning weights. Referring to Equation (6)), we then have K, {Ax]], = 0, [A,] wi...., [A
0}.
With knowledge sets at hand, we can introduce two design aims for addressing Problem 1.4
DNN in Figure 2 where x = aan) and y Q (s, agd) (i.e., y approximates Q (s, аал)). The end-to-end input/outpul of the critic network strictly complies with available knowledge about the governing Equation 13), i.e., if [Q) E K. then y does not have monomials [m(s, adn; r2)];.
Aim 6.2. Given Ke (5). consider the critic network built on dhysii--nnoledde-ennaaam
Aim 6.3. Given K, (T6), consider the actor network built on
enhanned DNN in Figure 2 where x = S and y = स (s) (i.e., y approximates π (s)). The end-to-enu input/output of the actor network strictly complies with available knowledge about the governing Equation (14) ie., if
[An]is E IK, .ttthen| [y]; does not have monomials [m(s, r=)];-
Algorithm 2 in conjunction with Algorithm [ is able to deliver Aim 6.2 and Aim 6.3 stated in Theorem whose proof appears in Appendix[]
It is formally
Theorem 6.4. If the critic and actor networks are built on
DNN
(described in Figure [ whose NN input augmentation and NN editing are described by Algorithm 7
and Algorithm [ respectively, then Aim 6.2 and Aim 6.3 are achieved.
Finally, we refer to the toy example in Remark 6. for an overview of NN editing. For the end-to-end mapping [y]= = [ (s)]1, given K,, and output of Algorithm [ the link editing of Algorithm 2 removes all connections with node representations 1, Ç, Vć, wc, 2 and maintain link connections
with v, w,v?, aw and w Meanwhile, the action editing of Algorithm 22 guaranttess the usages of
action functions in all PhyN layers do not introduce monomials of ( to the mapping [y): =[ (s)]

---

Published as a conference paper at ICLR 2024
Algorithm 1 NN Input Augmentation
D Aim representation embrace all the missing and redincdant
monomials of the Taylor scries.
1: Input: sugmentation order rea-- y
indicates layer number, e:g. denotes the second NN layer.
2: Generate index vector input: 4- [I: ... (o
3: Initialize augmentation vector: ((e))))))))))) + Mief
4:
(ee)) 5: for to len (e5)
6:
Compute temp: ty + co)l xee [1) tes
> Capture hard to learn nonlinear representations, in the
Form of Equation
monomials of the Taylor series, such the 2nd-order monomials of sub-reward ((s((()))))) (A), s(k + 13)
7:
if
then Generate terng:: Eo +- Eai
8:
9:
else
10:
Generate temp: in [as Ea]:
Avaid missing redundant monomials.
11:
end if
12:
Update index entry: (1), (yn()))))
13:
Augment: (() ((()))())))))) (m Fo
14:
end for
15:
end for
Line Line 15 generate node-representation vector that embraces all the non-missing and non-redundant monomials of the
Taylor
One illustration example Figure Appeadix
16: Output: m (2)> *(o) )
m(y (13. Ftc
Controllable Model Accuracy: algorithu provides one option
approximating the ground truth models 11
and [1 via Taylor series. According to Taylor's Theorem (Chupler 2.4 244Kimig Kiiigsberggrr 20133)
controllable accuracy by controlling augmentation orders r(t) (see Appendis G for expl explanations).
Algorithm 2 Crosiiss-odeel-Guiie Neural Network Editing
D Perform deep PhyN, and each needs
Algorithan D for generating nederrepeescnationn vectors. Detailed explanations of Algorithm 2 appear in Appendix
1: Imput: Network type T {' knowledge &o 15 and Xa number of PhyNs po and Pry. origin inpul
augmentation orders re and r., model matrices Ag and A,, terminal ouiput dimension len(y).
2: Choose network type
3: Specify augmentation order of the first PhyN: reas Pmi
4: for to do
5: ==
6:
Generate made reprrssepeennninn vector mi m(1)) Algorithm
Comreponding m(s, ir)) and m(s, in the
ground-truth models
13 and
because T(1) 4-
7:
Generate weight matrix gradient descent algorithm: W (ay Raw weight matrix usually responds fully-comnected NN
layer, which violate physics knowledge.
[A [Amliy E Km,
8:
Generate knowledge matris K Kaa); Kayle
b Include all elements in knowledge sel.
0.
otherwise
0.
l E K
9:
Generate weight-masking matrix Mas
[Mesy]au
2,
otherwise
0,
[Malle = 0.VJ E (1...., (axsm(x
10:
Generate activatioo-mskknng vector a
[am1)]e
otherwise
11:
12:
Generate raw weight matrix via gradient descent algorithm: W (ays Raw weight matrix usually responds to fully-conneeted layor, which can violate physics knowledge.
NN
13:
Generate node-repeosentation voctor m (e-1) ((e)) via Algorithm
Ogn(ar)
Tun(y))
O Ouely)) (ex))) (enf)))
14:
Generate knowledge matrix: K(
O (ein))))))))))))))xx///x//---
15:
Generate weight-musking matrix (x):
0.
0 (
of and [Mas)a,, v (enmmmxx))) (amta.r
MMeele.
1, otherwise
16:
Generate activation-maskia masking vector nis))
17:
end
18:
Generate uncertainty matrix Usy +- May O W ces:
19:
Compute output: y(c) - K m(y (2 ace @ not (U:o m (vr-))
20:
end for
21:
Output: & +- Y(s)-

---

Published as a conference paper at ICLR 2024
1.00
1.00
1.09
0.75
0.75
535
0.03
-0.2
0.25
-325
-0.50
0050
-55
0.75
-035
2.0
(a) Pby-DRL Policy
Model-based Policy
(c) DRI. Policy
Figure 3: Blue: area of IE samples defined in Equation (18). Green: area of EE samples defined in
Equation I9 Rectangular area: safety set. Ellipse area: safety envelope.
EXPERIMENTS
7.1 CART-POLE SYSTEM
We take the cart-pole simulator provided in Open-Al Gym Brockman et al. (2016). Its mechanical analog is shown in Figure II in Appendix K characterized by pendulum' angle 0 and angular
velocity _ 0, and cart's position and velocity I. Phy-DRL's action policy is to stabilize the
pendulum at equilibrium s* =Ixt,, v² w*] = [0, 0, 0, while constraining system state to
Safety Set: (s [ 0.9 S 0.9, 0.8
0.8).
(17)
To demonstrate the robustness of Phy-DRL, we intentionally create a large model mismatch for obtaining a model-based action policy of Phy-DRL. Specifically, as explained Appendix K.] the physics-model knowledge represented by (A, B) is obtained through ignoring friction force, and letting f sin show that the sole model-basee action policy does not guarantee safety.
and w² sin 9 = The system trajectories Figure in Appendix K.4
The computations of F, P and A are presented in Appendix K. Appendix K.3
Meanwhile, for an Finally, to
the high perfforrreee sub-reward in Equation (8). we let (s((s),, aai (A:) validate our theoretical results and present experimental comparisons, we define two safe samples:
Safe Internal- Envelloe (IE) Sample 5: if s(1 = 3 E Q then s(k)s())))))) Q2 V.
(18)
Safe Exxernal--Envelee (EE) Sample 5: if (11 3 E X, then s(k) E X Ω2 36 N.
(19)
We consider a CLF (control Lyapunov function) reward, proposed Westenbroek al. (0022). as R(-) = s (k) P ssk) s (k 1) P s(k + 1) + wsskk), a(k)). where the P is the same as eebeededd reward. We mainly compare our Phy-DRL with purely
the one in the Phy-DRL's data-driven DRL having the CLF reward for testing. Both models are trained for 75000 steps and have same configurations
critie and actor networks, presented in Appendix K.5 The performance metrics are the areas of TE and EE samples. Figure shows that i) Phy-DRL successfully renders the safcty envelope invariant, demonstrating Theorem 5.3 (r(s(k), ((ss(())s(() s( + 1)) 2 a holds final training episode), and ii) the safety areas of the sole model-based policy and the purely data-driven DRL are much smaller. Additional comparisons with incorporating model for model-based DRL (with the proposed CLF reward) and our Phy-DRL for state prediction presented in Appendix K.7
7.2 QUADRUPED ROBOT
In this experiment, action policies' missions are concurrent safe center- gravvty management, safe lane tracking alone x-axis, and safe velocity regulation. We define safety constraints as:
X = { CoM z-height-0.244 m.24m| < 0.13m, lyaw] < 0.17 rad, [CoM x-velocity -rxl < |nx] (20) Targeted Equilibrium: S* [0; 0; 0.24m; 0; O; rx: O; O; 0: 0; 0),
(21)
where the denotes the robot's ''s state vector (given in Equation (74h in Appendix L.2 the rs denotes the desired CoM x-velocity. The system state of the model
expressed as
S*
The designs of model-based policy and reward appear in Appendix L.3 and the training details are presented in Appendix [.66 and Appendix [..7 To demonstrate the performance of trained Phy-DRI.
8

---

Published as a conference paper at ICLR 2024
(a) Velocity más, Snow Road
(b) Velocity 0.5 mis. Snow Road
0.2
Safety Set
Linear
DAL
PD
Phy-ORL
Safely Envelope
0.4
0.2
-0.2
Yow
Safety Set
0.4
-0.2
-0.6
Linear
DRL
-0.4
Safety Envelope
Phy-ORL
-0.4
-0.3
0.0
0.2
0.4
Height
Height
(c) Velocity -1.4 mis, Wet Road
(d) Velocity -0.4 m/s, Wet Road
1.0
Linear
DRL
0.6
Linear
DRL
DRy
PD
Phy-DRL
0.4
Safety Envelope
0.2
Safety Envelope
Safety
0.4
0.2
-0.2
Safety
0.0
-0.6
0.2
0.5
Hoight
Height
Figure 4: Phase plots of models running different environments, given different velocity commands.
we consider the comparisons of four policies:
Phh--DRL policy, whose network configurations summarized in the model PKN-15 in Table in
Appendix L.6 Its total number of training steps is only 10
DRL policy, denoting a purely data-driven action policy trained standard DRL. Its network configurations summarized in the model FC MLP in Table 2 Its reward for training is the CLF
proposed in Westenbroek et al. (2022) and given Equation 84 where the P is the same as the one in the Phy-DRL's safety-embedded reward. Its number of training steps is large 10?.
-PD policy, denoting default proportionaldderiatte controller developed in Da al. (2221)
Linear policy, which is the sole model-base based action policy equation used in Phy-DRL.
We compare the four policies in four testing environments: a) r, 1 m/s and snow road, b) r. 0.5 m/s and snow road, c) rx = -1.4 m/s and wet road, and d) rx = -0.4 m/s and wet road. The links to demonstration videos are available Appendix L.9 with Figure showing that Phy-DRL successfully constraints the robot's states to safety set. Given more reasonable velocity commands in environments b) and d), Phy-DRL can also successfully constrain system states to the safety envelope. The Linear and PD policies can only constrain system states to safety envelope in environment d). The DRL policy violates the safety requirements in all environments, which implies that purely data-driven DRL needs more training steps to search for safe and robust policy. Meanwhile, Appendix [L.4, Appendix [L.6] and Appendix [L.7] show that Phy-DRL features remarkably better velocity-regulation performance, fewer learning parameters, and fast and stable training.
CONCLUSION AND DISCUSSION
This paper proposes Phy-DRL: a physics-regulated deep reinforcement learning framework for safety-critical autonomous systems. Phy-DRL exhibits a mathematically provable safety guarantee. Compared with purely data-driven DRL. and solely model-based design, Phy-DRL. features fewer learning parameters and fast and stable training while offering enhanced safety assurance.
We recall the computation of matrix P (used in defining reward and safety envelope) depends linear model. However, if the linear model's mismatch is large, no safe policies may exist to render the safety envelope defined by P invariant. How to address safety concerns induced by faulty P constitutes our future research. We also note the derived condition of safety guarantee in Theorem 5.3 is not yet ready for practical testing procedure due to the necessary full coverage testing within the domain Q. Transforming the theoretical safety conditions into practical and efficient ones will be another future research.

---

Published as a conference paper at ICLR 2024
9 REPRODUCIBILITY STATEMENT
The code to reproduce our experimental results and supplementary materials are available at https 7/ thub. com/ HP Ch00phy _r
The experimental settings are described in Appendix K.5
Appendix
K.6
and Appendix L.8
10 ACKNOWLEDGEMENT
We would like to first thank the anonymous reviewers for their helpful feedback, thoughtful reviews, and insightful comments. We appreciate Mirco Theile's helpful suggestions regarding the technical details, which inspire our future research directions. We also thank Yihao Cai for his help in deploying Phy-DRL on physical quadruped robot.
This work was partly supported by the National Science Foundation under Grant CPS-2311084 and Grant CPS-C31 2085 and the Alexander von Humboldt Professorship Endowed by the German Federal Ministry of Education and Research.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, and Matthieu Devin. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint https::tp//oc...t.. https////wwwwww///aeaee 1603. 04467]
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in Neural Information Processing Systems, 30, 2017.
Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear matrix inequalities in system and control theory. SIAM, 1994.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. 2016.
Ya-Chien Chang and Sicun Gao. Stabilizing neural control using self-learned almost Iyapunov critics. In 2021 IEEE International Conference on Robotics and Automation, pp. 1803-1809, IEEE, 2021.
Yuntian Chen, Dou Huang, Dongxiao Zhang, Junsheng Zeng. Nanzhe Wang, Haoran Zhang, and Jinyue Yan. Theory-guided hard constraint projection (HCP): A knowledge-bel -based data-driven scientific machine learning method. Journal of Computational Physics, 445: 10624, 2021.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3387-3395, 2019a.
Richard Cheng. Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. International Conference on Machine Learning, pp. 1141-1150, 2019b.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Xingye Da, Zhaoming Xie, David Hoeller, Byron Boots, Anima Anandkumar, Yuke Zhu, Buck Babich, and Animesh Garg. Learning contact-adaptive controller for robust, efficient legged locomotion. In Conference on Robot Learning, pp. 894. PMLR, 2021.
Jared Di Carlo, Patrick M Wensing, Benjamin Katz, Gerardo Bledt, and Sangbae Kim. Dynamic locomotion in the mit cheetah through convex model-predictive control. In 2018 IEEE/RSJ international conference intelligent robots and systems (IROS), pp. 1-9. IEFE, 2018.
10

---

Published as a conference paper at ICLR 2024
Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83)11-5, 2016.
Razvan V Florian. Correct equations for the dynamics of the cart-pole system. Center for Cognitive and Neural Studies (Coneural), Romania, 2007. doi: http:///ooeurrllooofforin/pppppp _ pole.pdf.
Michael Grant, Stephen Boyd, and Yinyu Ye. Cvx users' guide. edu/boyd/software. himl, 2009.
online: http://ww:. stanford.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-18700 PMLR, 10-15 Jul 2018. URL htttps://proceedingg mlr www.:sa............mmm.m www.pperesssseeeeeeeeeeee
html
Masanobu Horie, Naoki Morita, Toshiaki Hishinuma, Yu Ihara, and Naoto Mitsume. Isometric transformation invariant and equivariant graph convolutional networks. arXiv:2005.06316. URL
htttps://arxv..o wttpeiii.rrg/ab//////// /
Sandy H. Huang. Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Picter Abbeel. Adversarial
attacks on neural network policies. In 5th International Conference Learning Representations, ICLR 2017, Workshop Track Proceedings, 2017. URL /o..nttss///oeeereeee.....t form? [d=ryvIRyBK]
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan S Read, Jacob A Zwart, Michael Steinbach, and Vipin Kumar. poysics-guiden machine learning for scientific discovery: An application in simulating lake temperature profiles. ACM/MMS Transactions on Data Science, 2(3): 2021.
Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), pp. 6023-6029. IEEE, 2019.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, pp. 97-117. Springer, 2017.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 International Conference on Robotics and Automation, pp. 8248-8254. IEEE, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
https:://axxi. o abs/ 1412. 6980
Konrad Königsberger. Analysis 2. Springer- Verlag, 2013.
Tongxin Li, Ruixiao Yang, Guannan Qu, Yiheng Lin, Steven Low, and Adam Wierman. Equipping blxckboox policies with model-based advice for stable nonlinear control. arXiv preprint https
01341 pdf a.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compoostion Koopman operators for model-based control. arXiv:1910008264, URL httpts://aaxiv..ottt 1910. 08264
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference Learning Representations, ICLR, 2016.
11

---

Published as a conference paper at ICLR 2024
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G Johnson. Physics-informent neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing. (3(6):B1105-B111 2021.
Yanbing Mao, Naira Hovakimyan. Tarek Abdelzaher, and Evangelos Theodorou. Social system inference from noisy observations. IEEE Transactions Computationnl Social Systems, pp. 1-13, 2022. doi: 10. 1119/TCSS..022222225555
Yanbing Mao, Yuliang Gu, Naira Hovakimyan, Lui Sha, and Petros Voulgaris. -im-sipplex: Safe velocity regulation of self-driving vehicles dynamic and unforeseen environments. ACM Traasaations Cyber-Physicad Systems, 7(1):1-24, 2023.
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks on riemannian manifolds. In Proceedings the IEEE international conference on computer vision workshops, pp- 37-45, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):52-533,, 2015.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5p.5115 5115-5124, 2017.
Nikhil Muralidhar, Jie Bu, Ze Cao, Long He, Naren Ramakrishnan, Danesh Tafti, and Anuj Karpatne. PhyNet: Physies guided neural networks for particle drag force prediction in assembly. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp. 559-567, 2020.
Samet Oymak and Necmiye Ozay. Non-asymptotic identification of Iti systems from single trajectory. In 2019 American control conference, pp. 5655-5661. IEEE, 2019.
Theodore J Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research, 30Dec))803-83,, 2002.
Rajesh Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
Krishan Rana, Vibhavari Dasagi, Jesse Haviland, Ben Talbot, Michael Milford, and Niko Sünderhauf, Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics. arXiv preprint /att:s://arxtvv.ooo////////
09822. pdf
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifrre, George Van Den Driessche, Julian Schrittwesser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with 2016.
deep neural networks and tree search. Nature, 529(7587):484
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140-1144, 2018.
Wang. Incorporating symmetry into deep dynamics models for improved generalization. In International Conference Learning Representations, 2021.
Rui Wang and Rose Yu. Physics-guided deep learning for dynamical systems: A survey. 10X1v:2107.01222. /
01272.pd2
12

---

Published as a conference paper at ICLR 2024
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1457-1466, 2020.
Tyler Westenbroek, Fernando Castaneda, Ayush Agrawal, Shankar Sastry, and Koushil Sreenath. Lyapunov design for robust and efficient robotic reinforcement learning. arXiv:2208. 00222) 2022. URL
06721. pdf
Jared Willaad,, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Computing Surveys, 2021.
Kailai Xu and Eric Darve. Physics constrained learning for data-driven inverse modeling from sparse observations. Journal of Computational Physics, pp. 110938, 2022.
Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, and Xiaolong Wang. Learning vision- guided quadrupedal locomotion end-to-end with cross-modal transformers. 2022 International Conference on Learning Representations, 2022.
Arnold Zachary and Toner Helen. Accidents: An emerging threat. Center for Security and Emerging Technology, 2021. URL BRRLhttps: http://......111111111.
Fuzhen Zhang. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006.
Liqun Zhao, Konstantinos Gatsis, and Antonis Papachristodoullu. Stable and safe reinforcement learning via barrier-lyapuse actor-critic approach. 2023 62nd IEEE Conference on Decision and Control (CDC). pp. 1320-13255 IEEE, 2023.
Liqunn Zhao, Keyan Miao, Konstantinos Gatsis, and Antonis Papachristodoulou. Nlbac: A neural ordinary differential equations-base based framework for stable and safe reinforcement learning. arXiv preprint 13114824 2024.
13

---

Published as a conference paper at ICLR 2024
Appendices
Notations throughout Paper
16
Auxiliary Lemmas
17
Proof of Lemma 5.1
18
Theorem 5.3
D.I Proof of Theorem 5.3
21
D.2
Extension: Explanation of Fast Training Toward Safety Guarantee
22
Extension: Mathematically-Provabbb Safety and Stability Guarantees
24
E. Safety versus Stability
24
E.2 Provable Safety and Stability Guarantees
25
NN Input Augmentation: Explanations and Example
26
Controllable Model Accuracy
27
H Example: Obtaining Knowledge Set 1o
28
Plysies-Model-Guided Neural Network Editing: Explanations
29
Activation Editing
29
1.2
Knowledge Preserving and Passing
29
Proof of Theeore 6.4
31
Experiment: Cart-Pole System
32
K.I
Physics-Model Knowledge
32
K.2
Safety Knowledge
32
K.3
Model-Based Action Policy and DRL Reward
33
K.4
Sole Model-Based Action Policy: Failure Due to Large Model Mismatch
33
K.S
Configurations: Networks and Training Conditions
34
K.6
Training
34
K.7 7
Testing Comparisons: Model for State Prediction?
34
L
Experiment: Quadruped Robot
37
Overview: Best- Trade-Ort Between Model-Based Design and Data-Driven Design
37
Real System Dynamics: Highly Nonlinear!
37
L.3
Simplifying Model-Based Designs
38
Testing Experiment: Velocity Tracking Performance
39
14

---

Published as a conference paper at ICLR 2024
L.5
Safety-Emeedded Rewards
40
L.6
Boatees-nolleee--nnnen Critic Network
40
L.7
Reward Comparisons
41
Training
45
Links: Demonstration Videos
45
15

---

Published as a conference paper at ICLR 2024
A
NOTATIONS THROUGHOUT PAPER
Table 1: Notation
set of n-dimensional real vectors
N
set of natural numbers
len(s)
length of vector s
X
i-th entry of vector x
sub-vector formed by the i-th to j-th entries of vector
W
i-th row of matrix W
element at row and column j of matrix W
P
matrix P is positive definite
P R
0
matrix P is negative definite
matrix or vector transposition
I,
n--iienniional identity matrix
1
n-dimensional vector of all ones
On
n-dimenniioall vector of all zeros
Hadamard product
X y
stacked (tall column) vector of vectors x and y
act
activation function
ru-dimensional matrix
mx
an element in knowledge set
a learning element
X $2
complement set of 52 with respect to X
16

---

Published as a conference paper at ICLR 2024
B AUXILIARY LEMMAS
A
B
Lemma B.1 (Schur Complement Zhang 20061). For any symmetric matrix M
then
B
C
MX > 0 if and only C and A BC B
Lemma B.2. Corresponding the set X defined in Equation
define:
2 -
R" 1B S <
D.s, and D.S 1m
(22)
The sets X x if and only if D and D A' where d, A and are defined in Lemma 5.1
Proof. The condition of set defined in Equation 2 is equivalent to
v + vl, - Dia, + e (1,2.....hh
(23)
based on which, we consider three cases.
Case One: If V v], 0, obtain from Equation 23 that ( v],
0 as well, such that the
Equation 23 can be rewritten equivalently
D
[D,,
[ D [A]
1, and
2 21 [d];, E (1,2....,h), (24)
V vl,
A
vv + vl,
which obtained via considering the second items of [A] and [ and the first item of [d]; presented in Lemma 5.1
Case Two: If V + v), < O, we obtain from Equation (33 that V v), < as well, such that the
Equation can be rewritten equivalently as
D
D 11,
D V v].
D 21 = [d];, E (1,2...., ...
and
(25)
AA
Ali,
which is obtained via considering the third items of [A]k, and [A]k, and the second item of (d]. presented in Lemma
Case Three: If [v vl, > O and [v + v]; < 0, the Equation
23 can be rewritten equivalently as
D c. v),
D S 1, and
[ f.: v).
[D] [
-1 [d (1,2,..., k,. (26)
A]
A i
which is obtained via considering the fourth items of [A]k and [Ak, and the third item of [d]., presented in Lemma 5.1
We note from the first items of A and Ali in Lemma 5.1 that the defined A and A are diagonal matrices. The conjunctive results (23) (26) can thus be equivalent described by
D π
1, and D.
d 2 1
substituting D and D D into which, we obtain 1, d D s, and D 1a, which is the condition for defining the set X Equation (22). We thus conclude the statement.
L
17

---

Published as a conference paper at ICLR 2024
C PROOF OF LEMMA 5.1
In light of Lemma B.2 in Appendix we have = Therefore, to prove f C X, we consider the
proof of f C %. which is carried out below. The defined in Equation (22) relies on two conjunctive conditions: - 11, v d < D 8 and D v 1, based on which the proof is separated into two cases.
Case One: D 1 which can be rewritten as [D S} S 1,
hj.
We next prove that max SEi)
[ s] - D. D.P-1.
ie 2 ........... hJ. To achieve this, let
1,
consider the constrained optimization problem:
max
D s],}, subject to PPS51, iE (1,2.... h}.
Let s" be the optimal solution. Then, according the Kuhn-Tucker conditions, have
D
2A P =,=0,
X- 1111-55)) P P-)") =0
which, in conjunction with
0, lead to
2A P s" =D],
(27)
(s") T P p's'=1 =1
(28)
Multiplying both left-hand sides of Equation (7 by (S*)T yields 2A (8*) .P.s" = T
[
which, in conjunction with Equation (28) results in
2A (s*)T D k, 0.
(29)
Multiplying both left-hand sides of Equation both left-hand sides of which by D
27 by P-1 leads to 2A. =P-1. [ multiplying
we arrive
2A
D s.:
S
D i, P- D i.:"
(30)
Substituting Equation (2) into Equation 30p, we obtain (s*) [D]
[D] k, s* = [D
P-1. from which we can have (s*)
D
P-1. D > 0, which with
=
Equation 29 indicate:
2A
D bi
P-1,
[D]
(31)
We note that the Equation (27) is equivalent s* P-1 D f.
substituting Equation 311 into
which results in s*
D
multiplying both sides of which by
D
p
D
means
D
P--1
D
max { [[, sEll
max sEil
[D], .s) -D S
D i:: .P-1,
D
i,:
D i,:
P-1
D
i (1,2,.....
which means
max D $) < if and only if D sEit
D)a."
-P-1.
D
1
iE (1. h},
(32)
:
which further implies that
D 1, if [D,,
p-1
D
11, ie (1.2....,A
(33)
18

---

Published as a conference paper at ICLR 2024
Case Two: - 1 2 d < D which includes two scenarios: [d]; and [d]; - referring d
in Lemma 5.1 We first consider [D s|: 2 -1, which can be rewritten as [D S -[d]; = 1,
21, ................................ h), with D D. Following the same steps to derive Equation (32). obtain
max BEf
D
< 1 if and only
P
D S
<1, i (1)......
which with D indicate that
min
D $| 2 -1 iff Dh::" P-1. <1, [d] --1, i E (1,2,....,1)
(34)
SEi?
We next consider [d] = 1. We prove in this scenario, min
D s|,} =
D P-1 D T
To
sE
achieve this, let us consider the constrained optimization problem:
min [D s} subject to T 1
Let S* be the optimal solution. Then, according to the Kuhn-Tucker Tucker conditions, have
D + 2 P 8* = =0, Å. (1 -1-1-(1 - (5*) P P.S 0,
which, in conjunction with
0, leads
2 P S* = D
(35)
(s*) T P =
(36)
Multiplying both left-hand sides of Equation (35 by (S*) yields 2Å (S*) .P."" - - which, in conjunction with Equation (36). results
D
1.
21 -
O.
(37)
Multiplying both left-hand sides of Equation 35 by P leads to 2 = - D multiplying
both left-hand sides of which by DD we arrive in
2 D b,:
D PP-1 D 6,:
(38)
Substituting Equation (37) into Equation 38 we obtain T
D
8*
D
P
[ T which together with Equation
37 indicate the solution:
21 -S*)
D
[D]
P-1.
D
(39)
We note that the Equation
35 is equivalent to 8*
.P-1
D j.r
substituting Equation
39
2X
into which results in S*
P
D
multiplying both sides of which by D
means
PP11
D 1.
min xEi2
[ s|,} min
[
[k,, 8
i.:
sen
D)., P-1. -..
Dk. P-1 D !,,
i (1,........
which means
min sei2
s. 2
D
p-1.
D
1, [d]. =, i (1,2,....,
(40)
Summarizing Equation we conclude that
34b and Equation
440
with the consideration of d presented in Lemma 5.1
2 1, if [d] =1
min sEl2
D
+)
>
[d]
iff
D
P
D
< 1, if [d] - iE 21,,,......
19

---

Published as a conference paper at ICLR 2024
which, in conjumction with the fact that D and d are vectors: implics that
set....... .4. (41)
1, if ld] - 1
D D.s Is: if pl.
pp-.
D
if [d]] --1
:.3
1,
.rom......
We now conclude from Equation for any i E (1,2,... (1,2......A h).
33 and Equation
41 that D
1 and D s d -1, hold,
1 if = < if [d]: =
D in
P
D
I and D P
D
:.a
:;
Meanwhile, noticing the D s I 1, and D
2 11, is the condition of forming the set X in
Equation (22). we can finally obtain Equation 7
20

---

Published as a conference paper at ICLR 2024
D THEOREM 5.3
D.1 PROOF OF THEOREM
We note that = Q" Equation yields
T and the F R Q is equivalent the Q R, substituting which into
a Q + B F)
Q (A+
0.
(42)
(A
Q
We note Equation (42) implies and Q 0. Then, according to the auxiliary Lemma B.1 Appendix we have
a Q Q (A + B F) Q (A B- Q 0. 0.
(43)
Since P = Q multiplying both left-hand and right-hand sides of Equation 43 by P we obtain
P (A + B F)" P. ( > 0,
which, in conjunction with A defined in Equation 9 leads
oP P P.A 0.
(44)
We now define a function:
V (s(k)) = T(((k) (k)-P s(k)..
(45)
With the consideration of function in Equation (45) along the real plant ( with Equation Equation 4 we have
and
V Vss(k + 1))
(B and (k) + a(s))) -P- (B am) + ((s()
+ 2s (k) A P- (B (k) + f(s(((()))))))))))))())) a(k)) s (k: (A -P-A) s(k)
(46)
< (B and (k) + f(s(k), a (k)))) P- and (k) + f((kk) a (kk)))
+ 2s T (k) A P- and (k) + f((()) a()))) a V(s(k)),
(47)
where the Equation is obtained from its previous step via considering Equation Equation 44 Observing Equation (46). we obtain
455 and
-ss(k)ss(k + 1)) = V(s(k: l + 1)) s (k) A P. s(k)
(B aad (k) + ((s(k),a(k)) P (B Ban (k) f(s()) a(k))
2s (k) A P. (B aad aan(k) (k) fs(()( akk))
(48)
where r(s(k),s(k: + 1)) is defined in Equation S yields
Substituting Equation 48 into Equation
47
V(s(k: + 1)) a V(s(k)) r(s(s))sss))) 8( 1)),
which further implies that
V(s(k 1)) - V(s(k)) (a- - 1). V(s(k)) ((k 1)).
(49)
Since < a < 1, we have a - 1 < 0. So, the (a - 1) V (s(k)) - r(s(A). s(k + 1)) 2 0 means
(s(k)) < r(k(k) Therefore, if r(+(k)).((++) 1, we have a-1
a-1 V (s(+)) 1, and (a- V (s(k) r(s(k((s))))))))) s(k: + 1)) 2 0.
(50)
where the second inequality, in conjunction with Equation (49 implies that there exists scalar 0 such that
Vis(k: 1)) V(s(k) 0, with 0 > 0.
(51)
21

---

Published as a conference paper at ICLR 2024
The result in Equation 511 can guarantee the safety of real plants. To prove this, let's consider the worst-case scenario that the (ss(k) is strictly increasing with respect to time E N. So, starting from r(s(k),((<+1) a-1 S 1, V (s())) will increase to V (s(g)) r((()))((++) 1, where q > k E N. Meanwhile, we note that the (a-1) 1) VV (s(k:) - r(((k)), s(k a-1
system state s(k) satisfying V (s(k)) <
1)) & 0 equivalent to V (s(k)) 2 r(s(k),s(b++)) and also in conjunction with Equation (49) implies that (ss(A: + 1)) (s)) < 0. These mean that if V (s(g)) r(e(g).s(a+1))
a-1
a-1 < 1 is achieved, the V (s(k)) will start decreasing | We thus conclude here that in the worst-case scenario, if starting from a point not larger than r(8(R) (((())a(k-1)) i..., V (s(+)) < ((s(k),a(k++)) I 1, we have
(s())) 1 VKEN.
(52)
We now consider the other case, i.e., 2 V (s(k:) > r(s(k),a(k+1)) Recalling that in this case V(s(k - 1)) (s(k)) < O which means V (s(A)) is strictly decreasing with respect to time k. until (s())) < r(s(g).s(g+ii) <1,9 > E N. Then, following the same analysis path of the worst a-
case, we can conclude Equation (52) consequently, In other words,
V (s(+)) S Wk E N, if (s(1)) S 1,
which, in conjunction with Equation (6) and Equation 45 lead to
s skk) E Ω vk E, if s(1) E &Q
(53)
This proof path is illustrated in Figure 5
Finally, note from Lemma 5.1 that the condition in Equation is to guarantee that Q C X, which with Equation (53) result in s(k) Q C X, Vk E N. if E 2, which completes the proof.
r (s(k), s(k + 1))
a a
Safety Envelope
Figure 5: Illustration of the proof path of Theorem 5.3
D.2 EXTENSION: EXPLANATION OF FAST TRAINING TOWARD SAFETY GUARANTEE
The experimental results demonstrate that compared with purely data-driven DRL, our proposed Phy-DRL has much faster training toward safety guarantee, which can be explained by Equation 44
In practice the control command at one control period should not drive system to escape the safety envelop Q from V (s(g)) r(s(g),s(g+1))
22

---

Published as a conference paper at ICLR 2024
Equation 45 and Equation (46). Specifically, because of them and < 1, we have
((s(k + 1)) ((s(k))
S Vs((k 1)) a ((s(k))
(B a.nn (k) f(s(k), ak))) P.
(B and (k) + ((s(()))(()))
+ 2s (k) A P (B ( (k) + f(s(((()))))))))))))))) a (k))) + (k) (A' PPA- () -(k).. (54)
The (off-line designed) model-based property Equation implies that Equation 54) has
constant negative term, i.c.. S (k) (A P P) s(k) As depicted by Figure the (k) TT(k) A P-A .. s(K) 0 can be understood that it puts global attractor insides safety envelope, which generate attracting force toward safety envelope. Obviously, because of the always-existing attracting force, system states under the control of Phy-DRL are more likely and quickly to stay inside the safety envelope, compared with purely data-driven DRL frameworks.
In summary, the driving factor of Phy-DRL's fast training toward safety guarantee is the concurrent safety-embedded reward and residual action policy.
Training
Goal: Safety Guarantee
System states do not leave safety envelopel
Phy-DRL: Training
Safety Envelope
Global Attractor, due to (Ā P P) s=0 Generating attracting force toward safety envelope!
Figure 6: Explanation Phy-DRL's fast training: the root reason the (off-line designed) model- based property in Equation (4). which generates always-existing attracting force toward the safety envelope.
23

---

Published as a conference paper at ICLR 2024
E EXTENSION: MATHEMATICALLYYPROVAARR SAFETY AND STABILITY GUARANTEES
We first present the definition of stability guarantee.
Definition E.I. The real plant (1) said to stability guaranteed. given any s(1) E R", then lim skk) O,,.
The relation between safety guarantee and stability guarantee is presented in Appendix E.1
E.I SAFETY VERSUS STABILITY
(a)
Safety Set
Equilibrium Point
(b)
Safety Set (c)
Equilibrium Point
Safety Set
Equilibrium Point
Figure 7: Explanations safety and stability via phase plots: (a) only stability is guarantced, (b) only safety is guaranteed, (c) both safety and stability guaranteed.
24

---

Published as a conference paper at ICLR 2024
According to Definition 2.1 and Definition E.1 the phase plots in Figure well depict the relation between safety and stability:
Figure 7 (a): Only Stability guaranteed. Operating from any initial condition (inside or outside the safety set), the system state will converge to the equilibrium (zero). But safety is not guaranteed, since operating from an initial condition inside the safety set, the system will leave the safety set later.
Figure 7 (b): Only Safety is guaranteed. Operating from any initial condition inside the safety set, the system state never leaves the safety set but does converge the equilibrium point.
Figure (c): Both Safety and Stability are guaranteed. Operating from any initial condition inside the safety set, the system state never leaves the safety and converges to the equilibrium.
E.2 PROVABLE SAFETY AND STABILITY GUARANTEES
Thanks to the residual action policy and safety-embedded reward, the proposed Phy-DRL exhibits mathematically--o proovvbbl safety and stability guarantees, which is formally stated in the following theorem.
Theorem E.2 (Mathemattcally-Provabb Safety and Stability Guarantees). Consider the safety set X 2), the safety envelope Ω (6) and the system ① under control of Phy-DRL. The matrices F and P involved in the model-based action policy (4) and the safety- dnfee--embedded reward (8) are computed
according to Equation (11, where R and Q satisfies the inequalities 77 and 17). (7). Both the safety
and stability of the system (7) are guaranteed, if the sub-reward r(s(k)) (.kk),s( s(k 1]) in Equation
satisfies rsk(k), s(k + 1)) aa 1) (k) s(k).
Proof. This proof is straightforward and is based on the Proof of Theorem 5.3 in Appendix D.1 If (a () V (s(k)) r(s(k), s(k 1)) O, we obtain from Equation 49 that
V (s( 1)) V (s(A)),
4k E,
which implies that V (s(k) is strictly decreasing with respect to time k E N. The Phy-DRL in this condition thus stabilizes the real plant { Additionally, because V (s(k)))s strict decreasing, obtain Equation (53) via considering the Equation (6). In light of Lemma 5.1 the condition Equation 7 is to guarantee that C X, which with Equation (53) result in s(k) S2 S X, Vk e N, if s(1) sT. We thus conclude that in this condition, both safety and stability are guaranteed, which completes the proof.
25

---

Published as a conference paper at ICLR 2024
F NN INPUT AUGMENTATION: EXPLANATIONS AND EXAMPLE
finalll stacks vector with one. This operation means a PhyN node will be assigned to be one, and the bias will be thus treated link weights associated with the nodes of ones.
Line shows that Algorithm| I
As an example shown in Figure the NN input augmentation empowers PhyN capture well nonlinearities of physical quantities such as kinetic energy ( m and aerodynamic drag force @ 1 p that drive the state dynamics of physical systems and then represent or approximate physical knowledge in form of the polynomial function.
Line 6 Line 13 of Algorithm| T guarantce that the generated node-represennation vectors embrace all the non-missing and non-redundant monomials of polynomial function. One such example is shown in Figure 8 this example, the [x]? [x]3 is generated only by [x] ([x) [x|3), not including
others (see e.g.. [x|2 12 ([x]2 [x]))). Meanwhile, can be straightforward verify from the compact
example in Figure that all the generated monomials are non-missing and non-redundant.
2/03
21
21
X
WIMS
x
23
x
212
X
2/23
X
222
x
zero order
1st Order
2nd Order
3rd Order
m (x,
S)
15 (x]as [x]a >
(xt) [x (x) [xlas [x)]] [x] axxlas [x]]]
soco ordee
xX] [x]]
[x]]] [x]ss [x]] [x]8:
[xc) [x] [x]=[x]ss [x0]] [x]3 [x]] Mist
[x [x]]
Figure 8: An example of Algorithm in TensorFlow framework, where the input x E R and the augmentation order r 3.
26

---

Published as a conference paper at ICLR 2024
G CONTROLLABLE MODEL ACCURACY
The Taylor's ' Theorem offers a series expansion of arbitrary nonlinear functions, as shown below.
Taylor's Theorem (Chapter 2.4 Königsherger (2013) ): Let g IR + R be a r-times continuously differentiable function the point R. Then there exists ha R" R, where |o|| = r, such that
8 g()
g(x) = Σ
xx - -)" + Σ (x) - - and lim ha (x) = 0, x-00
(55)
ad
Sr
|a|=r
n Σ
where a Jan; 02: ... anl. |o||
ai
11 ass x
II x and 0e
1=1
0x -
Given ha (x) is finite and ||x o|| < 1, the error (x) o) for approximating the ground
truth g(x) will drop significantly as the order r = |a| increases and lim (x) - o) = 0.
-a/=r>o
This allows for controllable model accuracy via controlling the order
27

---

Published as a conference paper at ICLR 2024
H EXAMPLE: OBTAINING KNOWLEDGE SET Kg
We use simple example to explain how K is derived from Equation (5) according to Taylor's
theorem Königsberger (2013). For simplification, we let s(k) E R. and IR and re = 2. By
Algorithm with rie ro and (1) = [s(k), s(k + 1), (dn(k)] T, we have
m(s(k), (a)(k), [kk) s(k+1), (k), (k),, s(k) s(k+ s(k) (kn sk s(k+1) aad (k), a (+:] (56)
Observing Equation Q" ("((s(),, (k)).. For our reward, we let
also denote the action-value function as Q" (R (s(B). aud (k))) -
R(s(k), (k)) = s (R) A P A - s s +1 P s +
(57)
Right now, according to Taylor's theorem in Appendix G expanding the action-value function @ (R (s(k), (a)())) around the R (s(k), aan (k)). we have
2" (R (s(k), aan wn R (s(k), and (k)) + u2 R2 (s(k); (k)) +
a(((())
Ag m(s(k), aad (b)), re) p(s(k), adn (k)).
(58)
Recalling Equation (56). Equation 577. and Taylor's theorem in Appendix G Equation
we then conclude from
(58) that
Weight matrix A
2wi [Ph,2 wi [Ph,,
Wi 1P]2,22,
where b and Wi are learning parameters.
The unknown ((s(k), aun(k)) does not include any monomial in Equation (56). So, the elements of K this example are all entries of Ag, i.e. Kg ([A A ehoo)
28

---

Published as a conference paper at ICLR 2024
1 PHYSICS-MODEL-GUIIE NEURAL NETWORK EDITING: EXPLANATIONS
I.1 ACTIVATION EDITING
For the edited weight matrix W (t), if its entries in the same row are all in the knowledge set, the associated activation should be inactivated. Otherwise, the end-to-end input/output of DNN may not strictly preserve the available physies knowledge due the additional nonlinear mappings induced by the activation functions. This thus motivates the weyssics-kowwledee. preserving computing, i.e., Line 19 of Algorithm Figure summarizes the flowchart of NN editing single PhyN layer:
Given the node-ren reniee--ppesnntioonn vector from Algorithm | the original (fully- colly--ooneeted) weight matrix is edited via link editing to embed assigned physics knowledge, resulting in
W (t).
The edited weight matrix W Wes is separated into knowledge matrix K(r) and uncertainty matrix U (t), such that Wa (x) {e) + (e)- Specifically, the ( generated in Line 8 and
Line 14 of Algorithm 2 includes all the parameters in the knowledge set. While the M generated in Line 9 and Line 15, is used to generate uncertainty matrix U (c) (see Line T8)
to include all the parameters excluded from knowledge set. This is achieved by freezing the parameters of W (t) that are included in the knowledge to zeros.
The K (t). M and activation-masking vector a (generated in Line 10 and Line 16) are used by activation editing for the peysscal-knowledge knowledge--reeervving preserving computing of output in each PhyN layer. The function of a()) is avoid the extra mapping (induced by activation)
that prior physics knowledge does not include.
Original
Rode-Repressenatiio Algorithm
Maisation-Masknng Vector
Weight Matrix
m DY
a(o
Link Editing
Physics-guided NN Editing: Algorithm 2
Activation Editing
e.g.
Pryylcchikkooolllde-PPPeeinnnn Computing
Vee Kuy m(Vo-sy ag
O act (Unz m mYiic-syy (())
W/o
Knowledge Matrix KJp
Uncertainty Matrix Ug:
Figure 9: Flowchart of NN editing in single PhyN layer, where E and denote parameter included in and excluded from knowledge set, respectively.
I.2 KNOWLEDGE PRESERVING AND PASSING
The flowchart of NN editing operating in cascade PhyNs is depicted in Figure TO Lines 5-9 of Algorithm 2 means that Ag K + M(1) @ Aw, leveraging which and the setting T(1) = Two the
ground-truith model (33 and (44 can be rewritten as
y (K (1) May O A=) m(x,r) + p(x)
K()) m(x, r()) + (Ma) @ Aw) m(x,r(1)) + p(x).
(59)
where we define:
as,
Q'
[s: aai]
: ' w 'm'
re. == '
(s).
3 'x'
s,
Tr: w = 'T'
We obtain from Line 19 of Algorithm that the output of the first PhyN layer is
Y() K()) m(x, r()) + a @ act (U m (x, (x)))
(60)
Recalling that K includes all the entries of A while the U() includes remainders, conclude from Equation (59) and Equation (60) that the available physics knowledge pertaining to the ground- truth model has been embedded the first PhyN layer. As Figure 10 shows the knowledge embedded
29

---

Published as a conference paper at ICLR 2024
in the first layer shall be passed down the remaining cascade PhyNs and preserved therein, such that the end-to-end critic and actor network can strictly comply with the physics knowledge. This knowledge passing is achieved by the block matrix K (p) generated in Line 14, thanks to which, the output of 1-th PhyN layer satisfies
[y (1)l
(1) m(x, r(a)
(a)) O (C)(U m( T [1)) VIE .2.....pp) (61)
knowledge passing
knowledge preserving
Meanwhile, the (e) M
()) means the masking matrix M generated in Line 15 is to
remove the spurious correlations in the cascade PhyNs, which is depicted by the cutting link operation Figure T0
and Preserve Knowledge
Pass and Preserve Knowledge (Kig Myy) Way " Kgy Miy, @ Waz
Embedd Knowledge (Kay May): Wa - Kar Mjy Wan
Augmentation
p
Augmentation
Augmentation
x(k)
98
Wa
Wa
Wa
am
8
OF
m
You
Yoo
Vm
SP
Via)
m
F2
PhyN Layer 1: Faz 2
PhyN Layer 2:
PhyN B:rpy
Figure Example of NN editing, i.e., Algorithm (i) Parameters excluded from the knowledge set are formed by the grey links, while the parameters included in the knowledge set are formed by the and blue links. (ii) Cutting black links is to avoid spurious correlations. Otherwise, the links can lead to violation of physics knowledge about the governing Equation (13) and Equation (14).
30

---

Published as a conference paper at ICLR 2024
J PROOF OF THEOREM 6.4
Let us first consider the first PhyN layer, i.e., the case t = 1. Line 8 of Algorithm 2 means that the knowledge matrix ( includes parameters included in knowledge sets, whose corresponding entries in the masking matrix May (generated in Line 9 of Algorithm 2 are frozen to be zeros.
Consequently, both M(y o A and U11 M @ ((1) excludes all the parameters of knowledge
matrix K (1)- We thus conclude that M A= T(1))(1)) p(x) in the ground-truth model (59)
and (11) @ act (U))) m (x, r(1))) in the output computation in Line 21 are independent of the term (1)) m(x, T(1))-
Moreover, the activation mawaatiiin-mmiinggggg.. vector (generated in Line 10 of Algorithm 2 indicates that if all the entries in the i-th row of masking matrix are zeros (implying all the entries in the i-th row of weight matrix are included in the knowledge set K,), the activation function corresponding to the output's i-th entry is inactive. Finally, we arrive in the conclusion that the input/output of the first PhyN layer strictly complies with the available physics knowledge pertaining to the ground truth (59) ie., if the [A=]a K.. the (a does not have monomials [m (x, Tw)li-
We next consider the remaining PhyN layers. Considering Line 19 of Algorithm
we have
y (p)ltten(y)
(m) m(y 1) (w)) Meseery + (m))) O act(U (x) m(y (p-1) 1), (ip) (itenyy))
Ieenly) (m()(p-1), (pp) (2:()))())) + [a(p) act(U (p) m(y (p-)) T(p) ((p)))
(62)
[en(ty) (V(e-1))asenty) (p)) o act(U (p) 1)1)) T(p) (2sea(y)
(63)
[y(p-1)] 17]1stemy)) + am)) act ac (Uip) (n) m(y 1), (p)) (dmn(y))
= (x(--) m( 2) T(p T(p- | + (a @ act U (p) m(y r(n) latett))
= [en(y)) (m(((=22) (p-2)) (pp (2 + Apy @ act U (p) (p-)) (
[on(y)) 2) App @ act U (p) m(y (p-11 1), (p))
(((e=2)laaem()) + (p) act U (p) m( 1) ((p) [
(ycy)lattn(y) + (a(p) C act (U)(
(r) m (p-)) T (p) (n) Mate(yy)
(K()) m(x (()) ) lon(() (a @ act(U (p) m(y (p-1)>
K()) m(x, r(1))) + (a))) C (U (p) m(y (y-)) 1); (w)) Lste(y))
(64)
where Equation (62) and Equation (63 are obtained from their previous steps via considering the structure of block matrix K (generated in Line 14 of Algorithm 2 and the formula of augmented
monomials: m)
[1; y; [m(y,r) ())ny)))(((())22)))e))) (()n(x)))+)))e((mxxm))) (generated via Algorithm 2b. The
remaining iterative steps follow the same path.
The training loss function is to push the terminal output of Algorithm 2 to approximate the real output y, which in light of Equation (64) yields
(1) m(x, ray) + (a(p) O act((U (g) (U()) m(y (o-)))-1), T(p) 11)sem(y)))
= (i) m(x, + (pp) C aCt(UUn) (p) m(y (p1))))))
(65)
where Equation (65) from its previous step is obtained via considering the fact Icn(ý) = Ien(y)
len(y (,)) Meanwhile, the condition of generating a weight-masking matrix in Line 15 of Algorithm removes all the node-represennaaions connections with the parameters of knowledge set included conclude that in the terminal output computation in Equation (65), the
in K (1) Therefore,
term app) @ act U (p) m(y (p-1), P(pp)) )) does not have influence on the computing of knowledge term K11 m(x, r(1)). Thus, the Algorithm ( strictly embeds and preserves the available knowledge
pertaining to the physics model of ground truth in Equation (59)
31

---

Published as a conference paper at ICLR 2024
K
EXPERIMENT: CART-POLE SYSTEM
mp
21
F
friction force
M c
Figure 11: Mechanical analog of inverted pendulums.
K.1 PHYSICS-MODEL KNOWLEDGE
To have the model-based action policy, the first step is to obtain system matrix A and control structure matrix B in real plant (T). In other words, the available model knowledge about the dynamics of cart-pole system linear model:
s(k + 1) A 5(k) a(k),
KeN.
(66)
We refer to the dynamics model of cart-pole system described Florian (2077) and consider the approximations cos 0 as 1, sin x= and w sin / x= 0 for obtaining (A, B as
0.0333
O
0
1
0.0565
A
B 10 0.0334 0.0783)
(67)
0
0
0.0333
0.8980
K.2
SAFETY KNOWLEDGE
Considering the safety conditions Equation
17 and the formula of safety set in Equation
we
have
10001 0 0010
0 v=
0.9
0.9
v=
0.8
v
0.8
(68)
based on which, then according to the A and defined Lemma
5.1
we have
=
0.9
d
Σ A
(69)
0.8
from which and D given in Equation
(68
we then have
D
10
D
D
(70)
A
0
32

---

Published as a conference paper at ICLR 2024
K.3 BODEL-BASSE ACTION POLICY AND DRL REWARD
With the knowledge given in Equation (69) and Equation (70), the matrices F and P are ready to be obtained through solving the centering problem in Equation 12). We let 0.98. By the CVXPY toolbox Diamond & Boyd (2016) Python, we obtain
0.66951860
-0.69181711
-0.27609583
0.55776279
0.69181711
9.86247186
0.1240829
-12.4011146
Q
0.27609588
0.1240829
0.66034399
-2.76789607
0.55776279
-12.4011146
-2.76789607
32.32280039
R 6.40770185
-18.97723676
6.10235911
31.03838284
].
based on this, we then have
4.6074554
1.49740096
5.80266046
0.99189224
1.49740096
0.81703147
2.61779592
0.511796422
P --
(71)
5.80266046
2.61779592
11.29182733
1.87117709
0.99189224
0.51179642
1.87117709
0.37041435
F R
8.25591599
6.76016534
40.12484514
6.84742553
(72)
0.03333333
0.2 0.27592037
1.22590363
1.2843559
0. 0.2288196 196
A
(73)
0.03333333
-0.64668827
-0.52946156
2.24458365
0.46370415
With these solutions and letting w(s(k)), a(k)))))(())) = - (k), the model- baseld action policy (4) and the sabety-embedded reward (8) are then ready for the Phy-DRL.
K.4 SOLE MODEL-BASER ACTION POLICY: FAILURE DUE TO LARGE MODEL MISMATCH
The trajectories the cart-pole system under the control of sole model-based action policy, i.e., a(k) Aphy (k:) F s(k), shown in Figure 12 The system's initial condition lies in the safety envelope, i.e., s(1) E Ω. Figure 12 shows the sole model-based action policy cannot stabilize the system and cannot guarantee its safety. This failure due large model mismatch between the simplified linear model (66) and the real system having nonlinear dynamics. While Figure shows the Phy-DRL successfully overcomes the large model mismatch and renders the system safe and stable, tested with many initial conditions s(1) E X.
1.0
x 0.5
0.5
-0.5
2.5
0.0
10
-10
200
300
400
500
600
Time steps (k)
Figure 12: System trajectories of the cart-pole system: unstable and unsafe.
33

---

Published as a conference paper at ICLR 2024
K.5 CONFIGURATIONS NETWORKS AND TRAINING CONDITIONS
In this case study, the goal of action policy is to stabilize the pendulum at the equilibrium S [a",, o", 000 _h| [0, 0, 0J T while constraining system state the safery set in Equation 17 We convert the measured angle 0 into sin(0) and cos(O) to simplify the learning process. Therefore, the observation space can be expressed as
s = [ v, (0) (i), w]
We also added terminal condition the training episode that stops the running of the cart-pole system when a violation of safety occurs for both DRL and Phy-DRL in training, depicted as follows:
1, if [x< 0.9 [0(k)] 2 0.8
B(s(k))
0, otherwise.
Specifically, the running of the cart-pole system (starting from an initial condition) during training terminated if either its cart position pendulum angle exceeds the safety bounds or the pendulum falls. During training, we reset episodes the system running from random initial conditions inside the safety set if the maximum step of system running is reached, ((s(A)) =
The development Phy-DRL based on the DDPG algorithm. The actor and critic networks in the DDPG algorithm implemented a Multi-Layer Layer Perceptron (MLP) with four fully connected layers. The output dimensions of critic and actor networks are 256, 128, 64, and 1, respectively. The activation functions of the first three neural layers are ReLUU, while the output of the last layer is the Tanh function for the actor-network and Linear for the critic network. The input of the critic network [s; a). while the input of the actor-network
K.6 TRAINING
For the code, we use the Python API for the TensorFlow framework Kingma & Ba and the Adam optimizer Abadi et al. for training. This project using the settings: 1) Ubuntu 20.04, 2) Python 3.7, 3) TensorFlow 2.5.0, 4) Numpy 1.19.5, and 5) Gym 0.20.
For Phy-DRL, we let discount factor y 0.4, and the learning rates of critic and actor networks are the same as 0.0003. We set the batch size to 200. The total training steps are 10 and the maximum step number of one episode is 1000. Each weight matrix is initialized randomly from a (truncated) normal distribution with zero mean and standard deviation, discarding and re-drawing any samples more than two standard deviations from the mean. We initialize each bias according to the normal distribution with zero mean and standard deviation.
K.7 TESTING COMPARISONS: MODEL FOR STATE PREDICTION?
We perform testing comparisons of two Phy-DRLs (with and without a model inside for state prediction) and two DRLs (with and without model inside for state prediction). The two DRLs use the same CLF reward as R(-) = s (k:) (a).PP.so Pskkk) - (A: + 1) I).P. p-s(k+1))
(s(ks)), a (k))) (proposed in Westenbroek et al. (022)), where the P is the same the one in the Phy-DRL's safety-embedded reward. The model used for state prediction is the one in Equation (66). The testing results are
presented in Figure 13 We note in Figure that
mf-Phy-DRL, denotes policy trained via our Phy-DRL. that does not adopt the model in Equation (66) for state prediction.
mb-Phy-DRL, denotes policy trained via our Phy-DRL that adopts the model in Equa-
tion (66) for state prediction.
mf-DRL, denotes policy trained via DRI. that does not adopt the model in Equation 66 for state prediction.
mb-DRL, denotes policy trained via DRL that adopts the model in Equation (66 for state prediction.
Besides, all the training models of mf- inf-Phy-DRL, mb- Hob--hhy-RLL, mf-DRL and mb-DRL have the same configurations of critic and actor networks, presented in Appendix The performance metrics are the areas of IE and EE samples, defined in Equation (18) and Equation (19). respectively.
34

---

Published as a conference paper at ICLR 2024
639
6.9
6.9
6.99
63
6.7
6.7
6.71
6.79
-00
110
403
02
16
-385
30
485
1.
-3.0
100
-85
10
-3.00
-10
-85
2-
(a) mffPhhy-RRL 110) (b) mb-Phy Huh-Phy-DRI. (5 10
(c) mf)-DRL (5 100))
mb-DRI. 5--10))
632
63
6.9
6.91
-035
0.25
-022
4.25
4.90
400
-10
-05
06
10
4.
-30
-05
0.8
35
10
-3.00
-30
-65
0.8
10
-3.00
-30
-65
63
10
(e) mff-hh--RRL (7.5 10 mb--Phy-DRL. (7.5 10*)
mf-DRL (7.5 100)
(h) mb-DRL (7.5 -7.5-10")
1.50
1.50
1.50
3.00
629
0.2
8.2%
-032
93
-38
-00
-10
-65
10
-10
1.0
(i) mmf-Phy-DRR (10*)
mm--hhy-DRL HRL (10")
(k) mf-DRL (10")
()) mb-DRL (10")
1.80
1.81
3.81
3.50
0.5
0.75
025
0.5
0.25
0.25
455
4550
66.0
(m) mf-Phy-DeRR DRL (2 100) (n) mb- Hn--hhyDRLL (2 10")
(0) mf-DRL (2 10%)
mb-DRL (2 10")
Figure 13: Blue: area of IE samples defined in Equation 18
Green: area EE samples defined in Equation 19 Rectangular area: safety set. Ellipse area: safety envelope. (a)-dd): all the policies are
obtained via training for 10 steps. (e)-(h): all the policies are obtained via training for 7.5 10* steps. (i)-(l: all the policies obtained via training for 10y steps. (m)-(p): all the policies obtained via training for 2 10 steps.
35

---

Published as a conference paper at ICLR 2024
Observing Figure
13
conclude that
Our Phy-DRL that does not adopt model for state prediction can quickly complete training (only 5 10* steps) for rendering safety envelope invariant. While our Phy-DRL that adopts model for state prediction is slightly slow, 7.5 10st training steps can complete the safety task.
Safety areas of DRL policies are much smaller than our Phy-DRL policies. Even increasing training to 2 10 steps, mf-DRL and mb-DRL policies cannot render the safety envelope invariant, i.e. provide safety guarantee.
Incorporating linear model into DRL for state prediction does not improve system performance, a root reason was revealed in Janner et al. (2019) that the performance of model-based RL is constrained by modeling errors or model mismatch. Specifically, if model has a large model mismatch with the nonlinear model of real system dynamics, relying on the model for state prediction (in model-based RL) may lead to potential sub-optimal performance and safety violations.
36

---

Published as a conference paper at ICLR 2024
L EXPERIMENT: QUADRUPED ROBOT
CoM
Figure 14: Quadruped robot: 3D single iggi-booy model.
The developed package for training the robot via DRL is built Python-based framework for the Al robot from Unitree, released GitHub) The original framework includes a simulation based on Pybullet, interface for direct sim-to-real transfer, and an implementation of the Convex MPC Controller for basic motion control. For the quadruped robot, the outputs of the designed action policies the desired positional acceleration and rotational accelerations. The computed accelerations are then converted to the low-level motors' torque control.
L.1 OVERVIEW: BEST-TRADE-OFF BETWERN MODEL-BASED DESIGN AND DATA-DRIVEN DESIGN
In the following sections, we demonstrate that the residual action policy of Phy-DRL is a best- trade-off between the model-based policy and the data-driven DRL policy for safety-crities control. Specifically, Appendix [.. expliciity shows the dynamics quadruped robot highly nonlinear, directly leveraging which for designing analyzable and verifiable model-based action policy extremely hard. While Appendix L.3 shows the residual action policy of Phy-DRL allows the model- based design to be simplified be analyzable and verifiable linear one, while offering fast and stable training (see Appendix L.6 and Appendix
L.2 REAL SYSTEM DYNAMICS: HIGHLY NONLINEAR!
The dynamics model of the robot is based on single rigid body subject to forces at the contact patches Di Carlo al. (2018). Referring to Figure [4] the considered robot dynamies is characterized by the position of the body's center mass (CoM) p [piiPyi P=] E R3 CoM velocity v è P E R3, the Euler angles [ø; 0: v e F with ø, and & being roll, pitch and yaw angles, respectively, and the angular velocity in world coordinates w e R The robot's 's state vector is
Ø = [CoM x-position; CoM y-position; CoM z-heigh; roll; pitch; yaw; CoM x-velocity;
CoM y-velocity; CoM z-velocity; angular velocity E R$). (74)
Before presenting the body dynamics model, introduce the set of foot states [Left Front (LF), Right Front (RF), Left Rear (LR), Right Rear (RR)), based on which we define the following two footsteps for describing the trotting behavior of the quadruped robot:
Step, 1,F==1RFFO00 LR = 0, =R=1), Step2
RF= 1,
(75)
where '0' indicates that the corresponding foot is stance, and '1' denotes otherwise. The considered walking controller lifts two feet at a time by switching between the two stepping primitives in the following order:
Step, Step, Step, -> Step, - Step, Step2 Step,
Step, + repeating ....
37

---

Published as a conference paper at ICLR 2024
According the literature Di Carlo et al. (2018), the body dynamics of quadruped robots can be described by
P
O3
03
Ig
Og
03
d
O3
03
03
R(6,0,
((6,0,0) v)
03
+ Br(()) (x(())
+f($), (76)
dt
V
Og
0g
03
33
V
0g
O3
03
03
Og
g
2 3
where g = 00:0; 0: g] E 2 with @ being the gravitational acceleration, f(S) denotes model mismatch,
the R. 0, v) = (v) R Rx( E R3x3 with R,(a) R being the rotation of angle
about axis i. The az(1) R (t) S ".tep,.Sepp.) in the dynamics Equation (76) are the switching action commands, i.e.,
asup,
[RE:
asepp = [LL: fRR| E R.
(77)
where the flr. fRE, fRR, fLR E & are the ground reaction forces. While the Bo(t) E R R2x6 denote the corresponding switching control structure matrices:
3
O
Og
O3
Bsuep
1.
-13
(78a)
I
(Ø, 0,
[FRF]
I-1 (db 0, v)
[FER]x
O3
O3
O3
O3
Bsupp
(78b)
1a
La
(Ø) 0, b) [r
I-1 (#, v)) [RR]]
where I (Ø) rb)
R& the robot's inertia tensor, FLF, PRF, FIR, FRR R denote the four foots' positions relative to CoM position, and [rolx is defined as the skew-symmetric matrix:
0
[rols
Toly rrlz
[rolx =
[rol rroly
0
OE [F,,
[ro]
0
L.3 SIMPLIFYING MODEL-BASED BASED DESIGNS
To have the model knowledge represented by (A, B) pertaining to robot dynamics 76), we make the following simplifications.
O3
0
O
Og
R( ( 0.4) Ig.
B A
(79)
I
O3
Og
Ig
where the ()) Ig is obtained through setting the zero angels of roll, pitch and yaw, i.e.. 0.
④ = 0
Referring to the matrices Equation (7)). with the simplifications in Equation 79 at hand and the ignoring of unknown model mismatch of the ground -ruth model, we can obtain simplified linear model pertaining robot dynamics (76):
P
O
0
I3
O
d
O3
03
03
I
B ūb(t)
(80)
O3
0
0
Os
Og 03
03
Oa
w
A A
where ussp
LRFi fiR]
R and Usue
[LFi fRR] E R6
38

---

Published as a conference paper at ICLR 2024
In light of the equilibrium point s" in Equation (1 and 5 given in Equation
80 define s
It is then straightforward to obtain dynamies from Equation (80)
A B (t)))= which
transforms to a discrete-time model via sampling technique:
s(k + 1) = A k(k) + B (o) with A - I12 +T and B = TT B,
(81)
(A))
where T 0.001 sec is the sampling period.
Considering the safety conditions Equation 20 where
we obtain the safety set defined in Equation
0
0
1
0
0.17
00.17 17
D
0
0
0
0
0
V=
0.24
v=
0.13
v=
0.13 [xx
(82)
0
0
1x
[xx]
We obtain the following model- solutions, which satisfy the LMIs in Equation Equation
and
0.016
0.023
0
0
0.102
0.003
0
0.015
0.001
0
0
0.002
0
0.001
0.023
0.001
226.355
0.078
0.082
55.206
0.017
0.003
0.078
0.641
0.001
0.118
0
0
0.082
-0.001
0.638
0.125
0.102
0.002
55.206
0.118
0.125
247.803
0.045
0.003
0.003
0.017
0
0.045
1.065
0.001
0.003
0
0
0.003
0
0.03
0.211
-0.008
31141163
-0.73
0.789
483.6 $83.681
0.168
-0.021
0.006
0.042
0
0.004
0
0
0.003
0
0.042
0.003
0.002
1.249
0.003
0.003
5.547
0
0
-0.211
0
0
0.002
-0.008
314.163
0.006
0.003
1.249
-0.73
0.042
0.003
-0.789
0
0.042
0.003
483.681
0.004
0.003
5.547
0
0
-0168
0
0
0
-0.021
0
3229.212
-0.049
0.018
10.901
-0.049
0.017
00.088
0
0.017
0 0.169
100901
0.1
0
0
0
0
0
40
0
0.1
0
0
0
0
30
0
0
0
100
0
10
F
0
0
0
100
0
0
0
-
10
0
0
100
0
0
0
0
100
0
0
-30
with which and matrices and Equation and safety-embedded reward (8)
(81)
we are able deliver the model-based policy
L.4 TESTING EXPERIMENT: VELOCITY TRACKING PERFORMANCE
The velocity trajectories of the four models (defined in Section 7.2 running in Environments 1-4 shown in Figure 15 Observing the Figure 15 straightforwardly discover that the trained Phy-DRL can lead to much better performance of velocity regulation or tracking, compared with solely model-based action policies and purely data-driven DRL's action policy.
39

---

Published as a conference paper at ICLR 2024
(a) Velocity Reference 1 mis, Friction Confficient 0.44
(b) Velocity Reference 0.5 m's, Friction Coefficient 0.44
1.25
0.8
Reference
1.00
Linear
Phy-DAL
0.75
0.50
Velocity
0.25
Velocity
0.4
0.00
0.2
-0.25
Reference
DRL Phy-DRL
-0.75
PD
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Time Steps
Time Steps
(<) Velocity Reference -1.4 mvs. Friction Coefficient 0.6
(d) Velocity Reference -0. 4 mis, Friction Coefficient 0.6
Reference Linear
Reference
Phy-DRL
0.6
Linear PD
Phy-DRL
0.25
0.00
0.4
Velocity
-0.25
Velccity
0.2
0.50
-1.00
-1.25
-0.4
1000
2000
3000
4000 5000
6000
7000
8000
1000 2000
3000
4000
5000
6000
7000
8000
Time Stops
Time Steps
Figure 15: Velocity trajectories of quadruped robot in four different environments defined in Sec- tion
L.5 SAPETY-EMBEDDED REWARDS
For the aim of mathematically provablee safety guarantee, the reward that is most similar the CLF reward proposed in Westenbroek et al.] (2022). We also perform the comparisons. To simplify the comparisons, we do consider the high-performance sub-reward, which means both rewards degrade to
Ours (ss(k)), (k)) = s (k) T(k) A P. s(k) - (1) P P-s(k + +1,, T
(83)
CLF Reward R(s(k), ann (k)) S (k) P. s(k) -s (k+ +1++1 1) P s(k + +1).
(84)
L.6 PHYSICS-KNOWLEIN SANOWLEDGE-ENHANEE) CRITIC NETWORK
To apply the NN editing, we first obtain the available knowledge about the valur function and action policy. Referring to Equation (5), the action-value function can be re-denoted by
(*(((k), aad (k:) = Q" (R (s(k), ak)())))
(85)
According to Taylor's theorem in Appendix G expanding the action-valus function in Equation 85
around the (one-dimensional real value) R k)) (k)())), and recalling Equation (83). we conclude Include the action-value function does not include any odd-order monomials of [s(k)],, II... 12, and independent of aan (k). The critic network shall strictly comply with the knowledge. This knowledge compliance can be achieved via our proposed NN editing. We do not obtain the invariant knowledge about action policy in this example. In other words, according to analysis. the action policy depends on all the elements the system state s(k). So, in this example, we only need to design a physics-knowleee knowledge-enhanced critic network.
40

---

critic network with Published as a conference paper at ICLR 2024
The architecture of the considered physics-know wanoweede--nhhnneee critic network in this example is shown in Figure [6 where the PhyN architecture is given in (igure We compare the performance of feriissknowweddeee knwwweeee-enhhnneeeee
critic networks with fully-connected multi-layer perceptron (FC MI.P). Figure 16 shows that different critic networks can be obtained by only changing the output dimensions We here consider three models:
10 (PKN-10), physics-knowd wersss-ooooodeeeennneeeeee knowledge-enhaneed critic network with 20 (PKN-20). The parameter numbers of all network models are summarized in Table
critic network with n 15 (PKN-15), and physics-
Physics-Knowleegg--Enaaaeee Critic Network
Editing
Input
PhyN T(1)=2 2
dimention n
PhyN ((z)
dimention n
PhyN T
Output
Figure 16: Considered wewpiiis-knollldeeeeeooo critic network.
The trajectories of episode reward of the four models in Table 2 are shown in Figure 17 which, in conjunction with Table show that except for the smallest knowledge-enhanced critic network (i.e., model PKN-10), other networks (i.e., models PKN-15 and PKN-20) outperform the very large FC MLP model, viewed from perspectives of parameter numbers, episode reward and stability of training. This, the other hand, implies that the avoid significant spurious correlations via NN editing.
critic network can
Table 2: Model Parameters
Layer
Layer
Layer
Layer
Model ID
#weights
#bias
#Weights
#bias
#weights
#bias
#weights
#bias
#sum
PKN-10
1710
10
650
10
10
2391
PKN-15
2565
15
2025
15
15
1
4636
PKN-20
3420
20
4600
20
20
1
8081
FC MLP
2304
128
16384
128
16384
128
128
1
35585
L.7 REWARD COMPARISONS
We next compare the two rewards in Equation 83 and Equation (84) from the perspectives of design differences and experiments. To have a fair experimental comparison, we compare the two rewards in the same Phy-DRL package. In other words, we use two Phy-DRL models to train the robot; the only difference is their reward: one is our proposed reward in Equation (83) while the other one is the CLF reward in Equation (84).
L.7.1 COMPARISON: DESIGN DIFFERENCES
Along the ground-truth model of real plant T with the consideration of Equation and Equation g have
Equation
s' (l: +11 s(k +1
(B aai (k) +f f(s((k), a(k)) P- (B (k) + f(s(k), a(k))
2 T (k) A P (B am (k) ((s(() + T (A .P.A) s(k).
(86)
41

---

Published as a conference paper at ICLR 2024
(a)
-0.010
mean
PKN-10: CI
-0.015
FC MLP: mean
FC MLP: 95% CI
-0.020
Episode Reward
-0.025
-0.030
-0.035
-0.040
-0.045
-0.050
200000
400000
600000
800000
1000000
Training Steps
PKN-15: mean FC MLP: mean
PKN-15: 95%
-0.01
FC MLP: 95%
-0.02
Episode Reward
-0.03
-0.04
600000
Training Steps
(c)
PKN-20: mean
PKN-20: 95% CI
-0.01
FC MLP:
FC MLP: 95%
Episode Reward
-0.02
-0.03
-0.04
200000
400080
600000
800000
1000000
Training Steps
Figure 17: Trajectories of episode reward in training: smoothing rate 0.15 and random seeds.
42

---

Published as a conference paper at ICLR 2024
Table Episode Numbers
Ours
CLF Reward
Model ID
PKN-15
PKN-20
PKN-15
PKN-20
Episode Number: average
351
348
376
409
We next define two invariants and one unknown:
invariant-1 _ (k) A P A s(k),
(87)
invariant-2 s P A P P s(k).
(88)
unknown - (B aan + f(s(k). a a(k)) T P (B (k) ) f(((()), a(k)))
+ 2s (k) A P (B an (k) a(s)))
(89)
We note the formulas in Equation 87 and Equation 88 are named 'invariants' because all the terms in their right-hand sides (i.e., designed matrices A and P) are known to and their properties are not influenced by training. While the formula Equation (89 is defined 'unknown' since the terms in its right-hand side are unknown to us due the unknown model mismatch (s(k),, a(k)
and unknown data-driven action policy (Aa()) during training.
Using the definitions in Equation
(87 E-quation
89 the formula in Equation 86 is rewritten as
s (k + 1) P s(k 1) unknown + invariant-1,
by which and recalling Equation Equation the two rewards in Equation and Equa- tion (84) are equivalently rewritten
Ours: R(s(k),aall) invariant-1 - (I: + P s(k + 1) unknown,
(90)
CLF Reward: R(s(k), aan(k) (k) P s() - (I: 1) . p-s(k +
invariant-2 unknown.
(91)
discover critical difference between our proposed reward in Equation 83 and the CLF reward in Equation (84): our reward decouples invariant and unknown for learning (i.c., data-driven DRL only learn the unknown), while the CLF reward mixes invariant and unknown (i.e., data-driven DRL learn both unknown and invariant).
Observing the formulas in Equation 90 and Equation 91
L.7.2 COMPARISON: TRAINING
We next present the training behavior. We note our reward in Equation (90) and the CLF reward in Equation (91) have different scales. To present fair comparisons, we process the raw episode reward via unionization. Specifically, the proceeded episode reward called the united episode reward, defined as
R(m)
R(m) min 1,2
(92)
where R(m) denotes the raw episode reward at the episode index m.
We consider two models. PKN-15 and PKN-20, whose network configurations are summarized in Table Each model has three seeds. For all the training of DRL and Phy-DRL, we set the maximum step number of an episode as 10200, while an episode will terminate if the [CoM height S 0.12 m (robot falls). The two models' averages of episode number over the 10 training steps and the three random seeds are presented Table The smaller average value therein means the more successful running time of the robot or the fewer times the robot falls. Meanwhile, the trajectories of the processed episode rewards are shown in Figure [8] observing which and Table we can discover that our proposed reward leads to much more stable and safe training. The root reason can be that reward decouples invariant and unknown and only lets data-driven DRL learn the unknown defined in Equation (89).
43

---

Published as a conference paper at ICLR 2024
(a) PKN-15
CLF reward: mean
CLF reward: 95% Ci
Ours: mean
Ours: 95%
2.5
0.0
Episode Reward
-2.5
-5.0
-7.5
-10.0
-12.5
200000
400000
600000
800000
1000000
Training Steps
(b) PKN-20
CLF reward: mean
CLF reward: 95% Cl
Ours: mean
Ours: 95% CI
Episode Reward
-2
-4
-6
-10
200000
400000
600000
800000
1000000
Training Steps
Figure 18: Trajectories of united episode reward: smoothing rate 0. and random seeds.
44

---

Published as a conference paper at ICLR 2024
L.8 TRAINING
For the code, we use the Python API for the TensorFlow framework Kingma & Ba and the Adam optimizer Abadi et al. for training. This project using the settings: Ubuntu 22.04, 2) Python 3.7, 3) TensorFlow 2.5.0, 4) Numpy 1.19.5, and 5) Pybullet.
For Phy-DRL, the observation the policy a 12-dimensional tracking error vector between the robot's state vector and the mission vector. The agent's actions offset the desired positional and lateral accelerations generated from the model-based policy. The computed accelerations are then converted to the low-level motors' torque control.
The policy is trained using DDPG algorithm Lillicrap et (2016). The actor and critic networks are implemented as Multi-Layer Perceptron (MLP) with four fully connected layers. The output dimensions of the critic network are 256, 128, 64, and The output dimensions of actor networks are 256, 128, 64, and 6. The input of the critic network is the tracking error vector and the action vector. The imput of the actor network the tracking error vector. The activation functions of the first three neural layers ReLU, while the output of the last layer is the Tanh function for the actor network and Linear for the critic network.
We let discount factor y 0.2, and the learning rates of critic and actor networks the 0.0003. We set the batch size to 300. The maximum step number one episode 10200. Each weight matrix is initialized randomly from (truncated) normal distribution with zero mean and standard deviation, discarding and re-drawing any samples more than two standard deviations from the mean. We initialize each bias according the normal distribution with mean and standard deviation.
L.9 LINKS: DEMONSTRATION VIDEOS
Environment 1): velocity command: v, = m/s and snow road. demonstration video is available
at dt/ttps::/wwwz wwwp://wwwww//mm/////////////w www.t:/..ww.bbbee.m....o.mm wttp:/ecccwcaaaaaaeeaeeeeeeee htt.://tw....ccce.caaaaaaaatttta
Environment ii): velocity command: rx = 0.5 m/s and snow road. demonstration video
available at https::////// h/wp::://wwwww youtube. com
Iat=
I&t=21s
Environment iii): velocity command: т, 1.4 m/s and wet road. demonstration video
available at htttp:: http::::://www...ttuu
com. wwtoom/shrts/..oo http://sotttptcccceee1//
Environment iv):velocity command: Tx -0.4 m/s and wet road.
demonstration video is
available at https::/ http::::///www./// www. http:///wwwwyu.ubet.mmmommmm. youtube. com http:/shottss/see Shhoortss/uQQYve..JU..
45


Reviews:
[{"review_id": "qkgsxWgQI8", "reviewer": "Reviewer_GsDG", "summary": "Purely data driven learning has a pitfall of tending to violate known conditions of the environment. To address this, prior work has looked at modeling the known transition dynamics of the problem setting and using this to regularize the behavior of the data-driven model. This paper introduces a new technique (Phy-DRL) to improve safety assurance in Reinforcement Learning (RL) systems by means of invariant embeddings tackling three separate issues : using physics knowledge in learning, using safety violation information, and action policies violating known physics laws. Experimental results show the effectiveness of the approach in utilizing existing knowledge in both Cart-Pole and a complex quadrupedal robot.", "questions": "1. Fig. 4 does not have comparisons with any modern methods for MBRL or physics based RL. Is there a reason why this was omitted?\n2. Is ODE-based regularization [3] significantly more restrictive than the proposed approach? Why is a direct comparison not considered?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Considers safety information and known system dynamics in the final policy.\n- Introduces novel scheme for neural network policy and action-value function editing using the known dynamics of the environment.", "weaknesses": "- The paper addresses the case of RL with partially known environment dynamics but does not adequately consider comparative approaches such as  model-based RL  [1, 2]  or ODE-regularized RL [3]  in all the experiments.\n- The effect of each of the invariant embeddings is not shown empirically. An included ablation analysis would be useful to determine this.\n- Presentation could be improved slightly, perhaps by including some pseudo code in the Appendix.\n\n[1]  End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks, Cheng et al., AAAI 2019\n\n[2] Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning, Westenbroek et al., CoRL 2022\n\n[3] Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs, Du et al., NeuRIPS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Jlx36zaAw", "reviewer": "Reviewer_4Lie", "summary": "In this paper, the authors introduce the \"Phy-DRL\" framework which integrates physics knowledge into deep reinforcement learning (DRL) for safety-critical autonomous systems. Key features of Phy-DRL include:\n\nAn action policy that combines both data-driven learning and physics-based modeling.\nA reward system embedded with safety considerations.\nA neural network (NN) which is guided by physics modeling.\nTheoretical benefits of Phy-DRL are that it offers mathematically provable safety guarantees and aligns well with physics knowledge for computing action values. The authors tested their framework on a cart-pole system and a quadruped robot, finding that Phy-DRL offers safety benefits over purely data-driven DRL while requiring fewer learning parameters and enabling stable training.", "questions": "Please clarify what is meant by invariant embeddings here.\n\nHow does Phy-DRL differ from existing DRL frameworks in terms of its integration with physics knowledge? Is this a completely new framework, or an extension of existing ones?\n\nCould you provide more insight into the \"residual action policy\"? How does it ensure invariance with respect to training?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Addressing Safety in DRL: The paper introduces a DRL framework specifically designed for safety-critical autonomous systems, highlighting the significance of adhering to physical laws in AI applications.\n\nMathematically-Provable Safety Guarantee: One of the notable features of the Phy-DRL is its mathematically provable safety guarantee, which is crucial for real-world applications where safety is paramount. (although I am taking the claim of authors at face value and couldn't do a thorough analysis of the mathematical proofs and theorems given in the paper)", "weaknesses": "The thing I feel which is missing is where is the definition of invariant embeddings. \nWhat are they invariant to? How is adding a residual to a model-based making it invariant (and again invariant to what? state space?) Where is the invariant embedding principles coming from? (please cite any papers)\n\nThe lack of real-world results is also concerning, given the efforts by the community to test robots in the real world. \n\nThe presentation also needs to be improved. Specifically, Important terms and at least some experiment related information should be added in introduction to make the paper and its application easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWTmNBv4ZP", "reviewer": "Reviewer_BXs3", "summary": "This paper proposes a physics-regulated DRL framework for safety-critical tasks. The framework contains 1) residual action policy fusing model-based and model-free policies 2) safety-embedded reward by incorporating the knowledge from the approximated linear model dynamics 3) physics-enforced actor-critic algorithm including editing and activation editing, and achieves mathematically-provable safety guarantee.", "questions": "The reviewer is a bit confused about the Section 6, \nQuestion 1: How does NN input augmentation help embed what physics knowledge into AC?\n2: how does NN editing help embed what knowledge into AC? \n3. How do algorithms (1)(2) achieve NN input aug and NN editing?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated and well-organized. It is in general easy to follow. \n2. The approach, as the reviewer can tell, is sound and solid with remarkable provable safety guarantees. \n3. This framework is novel, specifically for the part of RL plus linearised model with a systematical reward construction approach to achieve provable safety guarantees. \n4. The experiments are significant compared to pure DRL and pure model-based approach.", "weaknesses": "1. In general, the paper is easy to follow. However,  Section 6 can be a bit confusing to the reviewer and needs back-and-forth checking while reading. This section can be improved by adding more overview, intuition, and connecting intro between and among subsections. It is somewhat difficult to accept a bunch of symbols such as in Equation (13), (14), (15), (16). \n2. It is somewhat hard to understand Algorithms (1) and (2). There is a lot of white space in Algorithm (1) and (2), why don't use it to add an overview, explanations, and comments for better understanding? \n3. In the experiments, the authors compared their approach to model-free RL, model-based control. For the reviewer, It is a bit unfair as the authors did not compare their approach to model-based RL such as RL + linearised model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agx2s8oNq9", "reviewer": "Reviewer_bsXP", "summary": "This work proposes three invariant-embedding principles to achieve a physics-regulated deep RL framework for safety-critical systems. The three principles include residual action policy, safety-embedded reward, and physics-model-guided neural network editing & augmentation. The main claim on safety-embedded reward and the editing are supported by theoratical proofs. Experimental results show that the proposed Phy-DRL achieve better performance in simulated cartpole and quadruped robot environments", "questions": "Please see weaknesses part.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work combines the safety requirement with neural network editing, which is novel to my understanding\n\n- This work provides proofs to support the designed safety-embedded reward.\n\n- The experimental results show that the proposed method achieves some good policies that follow safety requirements.", "weaknesses": "1. It's unclear how critical the residual physics policy is needed.\n\n2. The proposed method is only compared to methods without consideration on safety requirements.\n\n3. The method relies on known state definitions. If the state can not be obtained from the environment or definition, how this framework will be affected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)