Title: Boosting Verification of Deep Reinforcement Learning via Piece-Wise Linear Decision Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep reinforcement learning

Reachability analysis

Hybrid system

State abstraction
Abstract: Formally verifying deep reinforcement learning (DRL) systems suffers from both inaccurate verification results and limited scalability. The major obstacle lies in the large overestimation introduced inherently during training and then transforming the inexplicable decision-making models, i.e., deep neural networks (DNNs), into easy-to-verify models. In this paper, we propose an inverse transform-then-train approach, which first encodes a DNN into an equivalent set of efficiently and tightly verifiable linear control policies and then optimizes them via reinforcement learning.  We accompany our inverse approach with a novel neural network model called piece-wise linear decision neural networks (PLDNNs), which are compatible with most existing DRL training algorithms with comparable performance against conventional DNNs. Our extensive experiments show that, compared to DNN-based  DRL systems, PLDNN-based systems can be more efficiently and tightly verified with up to $438$ times speedup and a significant reduction in overestimation.  In particular, even a complex $12$-dimensional DRL system is efficiently verified with up to 7 times deeper computation steps.

Boosting Verification of Deep Reinforcement Learning via Piece-wise Linear Decision Neural Networks

Abstract

Formally verifying deep reinforcement learning (DRL) systems su ff ers from both inaccurate verification results and limited scalability. The major obstacle lies in the large overestimation introduced inherently during training and then transforming the inexplicable decision-making models i.e., deep neural networks (DNNs), into easy-to-verify models. In this paper, we propose an inverse transform-then-train approach, which first encodes a DNN into an equivalent set of e ffi ciently and tightly verifiable linear control policies and then optimizes them via reinforcement learning. We accompany our inverse approach with a novel neural network model called piece-wise linear decision neural networks (PLDNNs), which are compatible with most existing DRL training algorithms with comparable performance against conventional DNNs. Our extensive experiments show that, compared to DNNbased DRL systems, PLDNN-based systems can be more e ffi ciently and tightly verified with up to 438 times speedup and a significant reduction in overestimation. In particular, even a complex 12-dimensional DRL system is e ffi ciently verified with up to 7 times deeper computation steps.

1 Introduction

Deep neural networks (DNNs) have been exhibiting appealing advantages in decision-making and control for deep reinforcement learning (DRL) systems [1-4]. Nonetheless, the complexity and inexplicability [5, 6] of DNNs render the formal verification of their hosting systems, quite often even themselves, inaccurate and unscalable. Most existing approaches [7-10] over-approximate both embedded DNNs and non-linear environment dynamics to build verifiable models, which inevitably introduces dual overestimation. In particular, DNN-specific overestimation is extremely unpredictable due to many factors such as the dimension of system states, the complexity of environment dynamics, and the size, weight, and activation function of a neural network. For example, the verification results may deviate significantly even if the DNNs of the same DRL system di ff er only in their weights (as we also observed; see Appendix A.4 ). Unsurprisingly, verifying high-dimensional DRL systems would only exacerbate the problems of large overestimation and limited scalability.

Common practice for formally verifying DRL systems is to train and then transform the embedded DNNs into easy-to-verify models where, for any input set, output ranges can enclose the outputs of the over-approximated DNNs [7-10]. Taylor models [11] are widely adopted due to their preservation of input-output dependencies and less overestimation (accumulated in multiple steps) than the range analysis approaches such as interval over-approximation [12, 13]. However, they are still prone to intractable overestimation as the accuracy of verification depends heavily on the weights of DNNs

whose e ff ects are di ffi cult to quantify. Several other approaches attempt to extract approximated state-action policies, e.g., decision trees [14, 15], from DNNs via model compression [16] and distillation [17] techniques. However, no equivalence guarantee is established between DNNs and the extracted policies. Consequently, verification results are just probably approximately correct [18].

Inspired by recent advances [19-22] in training near-optimal policies even with reduced training state space imposed by aggregated adjacent states, we propose a novel, inverse transform-then-train approach: encoding a DNN into an equivalent set of easy-to-verify linear control policies and then optimizing them by training the DNN using reinforcement learning. We accompany our inverse approach by devising a novel neural network model called piece-wise linear decision neural networks (PLDNNs), which make linear decisions on each abstract state. Unlike conventional DNNs which build a state-action relation for each actual state, a PLDNN defines a linear relationship, called Linear Control Unit (LCU), between actions and actual states associated with the same abstract state. To this end, a PLDNN is essentially a set of LCUs for all abstract states. In contrast to DNNs, LCUs are more explainable and verifiable without any over-approximation. Moreover, PLDNNs are compatible with most existing DRL training algorithms as both share the same input and output layers for the same control task.

We extensively assess PLDNN, along with the state-of-the-art tools, with respect to both performance (in terms of cumulative rewards and system robustness) and verifiability (in terms of overestimation and time cost for the reachability analysis of trained systems) on a collection of benchmarks, including a 12-dimensional control task. Our experimental results show that, compared to the DNN-based systems, the PLDNN-based systems can be verified more precisely, with significantly less overestimation, and more e ffi ciently, with up to 438 times speedup, while achieving comparable performance. Moreover, compared to the state-of-the-art tools, the complex 12-dimensional control task can be trained and verified with up to 7 times deeper computation steps, along with notable tightness improvement.

Main Contributions. Overall, we provide: (i) a novel inverse approach for boosting the formal verification of DRL systems by learning e ffi ciently and directly (without over-approximation) verifiable piece-wise linear policies with comparable performance; (ii) a novel neural network model to realize the learned piece-wise linear policies, which is compatible with most existing DRL algorithms; and (iii) a prototype called L in C on , along with an extensive assessment which demonstrates its tightness in verification results, outperformance over the state-of-the-art tools (up to 438 times speedup), and scalability (up to a 12-dimensional control task).

2 Problem Formulation and Motivation

A DRL system is driven by a DNN-implemented controller π , which is trained for decision-making, and a physical model defined by the ordinary di ff erential equations (ODEs) ˙ s ( t ) = f ( s ( t ) , a ( t )), with s the state variables and a a control action. In what follows, we omit the time variable t . Typically, for DRL systems, continuous time is discretized, and we have a = π ( s ) at state s and assume ˙ a = 0 during a small time step e.g., δ . At the time point k δ, k ∈ N , the decision network receives the current state sk and outputs an action ak = π ( sk ). The state variables then evolve according to the physical model during the time interval [0 , δ ]. The reachable state sk + 1 at δ from sk is sk + 1 = sk + R δ 0 f ( s , ak ) dt which is called the successor state of sk . Note that the system evolves continuously from sk to sk + 1 . The intermediate states can be computed by substituting the time elapses for δ in the above formula.

Definition 1 (Path of DRL systems) . Given a DRL system D with an environment dynamics f , decision network π , time step size δ and initial state set S 0 , a path of D is a finite or infinite state

A DRL system is essentially a DNN-controlled hybrid system. Definition 2 gives a formal definition of regular hybrid systems. The state space of a hybrid system is the Cartesian product of a set L of discrete locations and state space of n real-valued variables Vc . At each location l ∈ L , the n continuous variables evolve continuously according to a dynamical law ˙ v = f l ( v ). When the guard condition on the transition between locations ( l 1 , l 2 ) ∈ T is triggered, the system moves to l 2, and the continuous variables are reset by R . There are two steps involved in the state transition from ( l i , vi ) to its successor state ( l i + 1 , vi + 1 ): first, from ( l i , vi ) to its time successor ( l i , φ f l i ( vi , t i )), and then, to

( l i + 1 , vi + 1 ) that is the transition successor of ( l i , φ f l i ( vi , t i )), where φ f l i is the solution of f l i with initial condition v (0) = vi , mapping the initial state vi to the state φ f l i ( vi , t ) (i.e., the reachable state at time t from vi ). Accordingly, the state of a hybrid system can be changed in two ways [23]: (i) by a time delay that changes only the value of continuous variables according to the dynamics of the current location defined in F ; and (ii) by a discrete and instantaneous transition that changes both location and continuous variables according to the rules in T .

Definition 2 (Hybrid Automata [9]) . A hybrid automaton is an 8-tuple H = ⟨ L , Var , Inv , F , T , G , R , I 0 ⟩ , where:

L is a finite set of discrete locations;

Inv : L → 2 Vc is a function assigning to each location an invariant condition;

Var is a finite set of n real-valued variables with state space Vc ⊆ R n ;

F : L → ( Vc → R n ) is a function associating each location l to a continuous dynamics ˙ v = f l ( v ) ;

G : T → 2 Vc is a function assigning each transition ( l 1 , l 2 ) ∈ T a guard condition G ( l 1 , l 2 ) ⊆ Inv ( l 1 ) ;

T ⊆ L × L is a set of transitions between locations;

R : T → 2 Vc is a function assigning each transition ( l 1 , l 2 ) ∈ T a reset R ( l 1 , l 2 ) ⊆ Inv ( l 2 ) ;

I 0 ⊆ L × Vc is an initial state set.

Definition 3 (Path of Hybrid Automaton [23]) . Let H = ⟨ L , Var , Inv , F , T , G , R , I 0 ⟩ be a hybrid automata. A path of H is a finite or infinite sequence of location and state value pairs, starting from an initial pair ( l 0 , v 0 ) ∈ I 0 , i.e., ( l 0 , v 0 ) t 0 - - → f l 0 ( l 1 , v 1 ) t 1 - - → f l 1 ( l 2 , v 2 ) t 2 - - → f l 2 ( l 3 , v 3 ) t 3 - - → f l 3 · · · such that:

∀ 0 ≤ t ≤ t i , φ f l ( vi , t ) ∈ Inv ( l i )

i 2. ( l i , l i + 1 ) ∈ T ∧ φ f l i ( vi , t i ) ∈ G ( l i , l i + 1 ) ∧ vi + 1 ∈ R ( l i , l i + 1 )

Theorem 1 (Modeling DRL Systems as Hybrid Automata) . A DRL system with an environment dynamics f, decision network π , time step size δ , and initial state set S 0 , state space S , can be equivalently modeled as the following hybrid automaton:

Var: state variable s, action a, and clock variable tc · I 0 : { ( l 0 , ( s ∈ S 0 , a = 0 , t c = δ )) }

T : { ( l 0 , l 0 ) } · G : G ( l 0 , l 0 ) = { t c = δ }

L : { l 0 } · Inv : Inv ( l 0 ) = { s ∈ S , t c ≤ δ }

F : F ( l 0 ) = { ˙ s = f ( s , a ) , ˙ a = 0 , ˙ t c = 1 }

R : R ( l 0 , l 0 ) = { t c = 0 , a = π ( s ) }

Proof. We first analyze the path of the modeled hybrid automaton H . For an arbitrary initial state ( l 0 , [ s 0 , 0 , δ ]) , s 0 ∈ S 0, since t c = δ , the only transition with guard condition { t c = δ } will be triggered and transition to ( l 0 , [ s 0 , a 0 , 0]) where a 0 = π ( s 0). Then ( l 0 , [ s 0 , a 0 , 0]) will move to its time successor ( l 0 , [ s 1 , a 0 , δ ]) where s 1 = s 0 + R δ 0 f ( s , a 0 ) dt . Next, the transition with guard condition { t c = δ } will be made and conduct the reset operation t c = 0 , a = π ( s 1) in R to obtain the transition successor ( l 0 , [ s 1 , a 1 , 0]) in which a 1 = π ( s 1). Repeating the evolution to the time successor and the transition successor yields the following sequence:

According to Definition 1, there exists the same transition relation si + 1 = si + R δ 0 f ( s , ai ) dt with ai = π ( si ) in D and the modeled hybrid automaton H at each time point i δ . Thus, given an arbitrary initial state s 0, the value of state variable and action of D and H remain the same at each δ . Then with the same state-action pair and dynamics f at each δ , D and H also have the same state-action value during each time interval [ i δ, ( i + 1) δ ).

Figure 1 depicts the hybrid automaton defined in Theorem 1. There exists only one location l 0 . The invariants in Inv claim that any state belongs to the state space S and the clock variable t c is less than or equal to the time step size δ . The flow in F defines the dynamics f of the system. The only transition is triggered when t c = δ , updating the action a and resetting t c as defined in R . The continuous change happens during the time interval [ i δ, ( i + 1) δ ], with i ∈ N , and the discrete change of actions occurs at each δ .

Unfortunately, the hybrid automaton of a DRL system cannot be verified by using existing hybrid automata model checkers such as Flow* [24], Ariadne [25], and CORA [26]. The reason is that

the action a in R depends on the uninterpretable DNN π by a : = π ( s ), and ˙ s = f ( s , a ) can not be expressed in a known closed-form, which however is required by regular hybrid automata supported by these tools [27]. Hence, almost all reachability-based verification methods for DRL systems such as Polar [10], Sherlock [7], and ReachNN [28] inevitably over-approximate π using a Taylor model, at the cost of large overestimation and time overhead.

3 Piece-Wise Linear Control Policies

To bypass the crux of over-approximating DNNs, we devise a novel, alternative neural network model which essentially realizes a set of linear control policies. Our approach bases on the common assumption that there exists a near-optimal linear control policy for every small region of the entire state space [29, 30, 21, 31]. Our objective is then to discretize the state space S of a DRL system and to train a linear control policy for each discretized region. Specifically, given an n -dimensional DRL system with m -dimensional control input, we train a DNN which implements a linear control function aj = b j + c j 1 x 1 + c j 2 x 2 + · · · + c j n xn for each control dimension 1 ≤ j ≤ m and each discretized region.

3.1 Abstracting State Space via Abstract Interpretation

Abstract Interpretation [32] is an e ff ective technique for scaling up formal verification of complex systems or programs by reducing the system space while preserving the soundness of verification results. For instance, an infinite state space [ -2 , 0] × [0 , 2] can be abstracted to be an abstract state represented as ( -2 , 0 , 0 , 2), when all the states in [ -2 , 0] × [0 , 2] share a same property. In general, given a system state space S , we denote S ϕ as a finite set of abstract states (each abstract state represents a possibly infinite set of actual system states in S ). Let ϕ : S → S ϕ be an abstraction function that maps each actual state s in S to the corresponding abstract state in S ϕ , and ϕ -1 : S ϕ → 2 S be the inverse concretization function such that ϕ -1 ( s ϕ ) = { s | s ∈ S , ϕ ( s ) = s ϕ } .

For state space abstraction, we choose the very primitive but e ff ective abstraction approach which abstracts actual system states as intervals. It is known as interval abstract domain and has been well studied for system [33] and program verification [34] and even the approximation of neural networks [35]. Specifically, let Li and Ui be the lower and upper bounds for the i -th dimension value of S . We first define the abstraction granularity as an n -dimensional vector γ = ( d 1 , d 2 , . . . , dn ). Then the i -th dimension will be divided evenly into ( Ui -Li ) / di intervals which means each abstract state can be represented as a 2 n -dimensional vector ( l 1 , u 1 , . . . , l n , un ).

Definition 4 (Interval-Based Abstraction Function) . Given an n-dimensional continuous state space S and an abstract state space S ϕ which discretizes S based on abstraction granularity γ , ϕ : S → S ϕ is called an interval-based abstraction function such that, for every actual state s = ( x 1 , . . . , xn ) ∈ S and abstract state s ϕ = ( l 1 , u 1 , . . . , l n , un ) ∈ S ϕ , we have ϕ ( s ) = s ϕ if and only if l i ≤ xi < ui holds for each dimension 1 ≤ i ≤ n.

Example 1 (Running Example) . Consider a 2-dimensional system in [36] with state space [ -2 , 2) × [ -2 , 2) . The dynamics f is defined by following ordinary di ff erential equations (ODE) i.e., ˙ x 1 = x 2 -x 3 1 and ˙ x 2 = a. The sign a means the control action. The objective is to train a DNN for determining action a based on ( x 1 , x 2 ) so that the agent can move from the initial region x 1 ∈ [0 . 7 , 0 . 9] and x 2 ∈ [0 . 7 , 0 . 9] to the goal region x 1 ∈ [ -0 . 3 , 0 . 1] and x 2 ∈ [ -0 . 35 , 0 . 05] as soon as possible.

Suppose that the abstraction granularity is γ = (2 , 2). The continuous state space [ -2 , 2) × [ -2 , 2) is then partitioned into four regions, corresponding to four abstract states represented by S ϕ = { ( -2 , 0 , -2 , 0) , ( -2 , 0 , 0 , 2) , (0 , 2 , -2 , 0) , (0 , 2 , 0 , 2) } , respectively.

3.2 Piece-Wise Linear Decision Neural Networks

We devise an alternative DNN model called piece-wise linear decision neural networks (PLDNNs). Unlike conventional DNNs, a PLDNN contains an abstraction layer between the input layer and the first hidden layer. The abstraction layer is used to convert an actual system state into its corresponding abstract state. Then the output of a PLDNN is the control action that is the dot product result of state variables of the actual state and the linear coe ffi cients determined by the corresponding abstract state.

l

0

;

s

˙

s

=

f

(

s

;

a

)

˙

a

=

0

˙

t

c

=

1

G

Inv

F

R

s

2

S

t

c

GLYPH<20>

GLYPH<14>

a

:

=

GLYPH<25>

(

s

)

t

c

:

=

0

t

c

=

GLYPH<14>

Algorithm 1: The Training Procedure based on the DDPG algorithm

6 return π

Figure 2 exemplifies the architecture of the PLDNN π for a two-dimensional DRL system. The decision-making of π is based on a coe ffi cient network π c that outputs the linear coe ffi cients. The second layer of π c is the inserted abstraction layer which consists of the blue neurons and the red neurons.

The output layer of π c contains n + 1 neurons that output the n + 1 linear coe ffi cients depicted as purple neurons. As for the weights setting between the input layer and the abstraction layer, the weights of the connections between the i -th neuron in the input layer and the (2 i -1)-th and 2 i -th neurons in the abstraction layer are set to 1 which are represented by blue lines and red lines, respectively. While the weights of other connections denoted by the black dashed lines are set to 0. Under this setting of weights, the inputs to both (2 i -1)-th and 2 i -th neurons in the abstraction layer are xi . Moreover, the activation function of the (2 i -1)-th neuron in

1

x

1

x

2

GLYPH<15>

a

b

c

1

c

2

GLYPH<25>

c

(

s

)

x

1

x

2

w

=

1

w

=

1

w

=

0

Abstraction Layer

a = GLYPH<25> ( x 1 ; x 2 ) = GLYPH<25> c ( x 1 ; x 2 ) GLYPH<1> (1 ; x 1 ; x 2 )

the abstraction layer is set to ϕ u with the responsibility of computing the upper bound ui , and that of the 2 i -th neuron is set to ϕ l for calculating the lower bound l i . Specifically, for a continuous state space partitioned by the abstraction granularity γ = ( d 1 , . . . , dn ), the activation functions for the (2 i -1)-th and 2 i -th neurons in the abstraction layer can be formulated as follows:

With the above activation functions, the abstraction layer can output the same abstract state s ϕ = ( l 1 , u 1 , . . . , l n , un ) for ∀ s ∈ ϕ -1 ( s ϕ ). The abstract state s ϕ is then propagated to the fully connected layers of π c to generate the linear control coe ffi cients ( b , c 1 , c 2) denoted by π c ( s ).

To obtain the final output of action a , an additional dot product operation between π c ( s ) and [1 , s ] is performed with the result of the operation as the control action a where [ · , · ] is the concatenation operation. For multiple dimensional control action a = ( a 1 , · · · am ), we only need to modify the output dimension of π c to m ( n + 1), such that each n + 1 neurons output the linear coe ffi cients of one dimension of a . More specifically, we can obtain aj as follows:

where vector [ start : end ] denotes the slicing operation that extracts the elements of vertor from index start up to but not including index end .

With the additional abstraction layer that can output an identical vector into the fully connected layers of π c for ∀ s ∈ ϕ -1 ( s ϕ ), we can ensure that π c always produces the same coe ffi cients for all actual states located in the same abstract state. Consequently, we can extract a piecewise linear decision function with this structure of π on each abstract state.

3.3 The Training Procedure

Training a PLDNN can be achieved by extending existing deterministic policy gradient algorithms such as Deep Deterministic Policy Gradient (DDPG) [37] and Twin Delayed Deep Deterministic Policy Gradient [38] since the modifications made stay inside neural networks and are invisible to the DRL algorithms. The pseudo code of the training procedure is given in Algorithm 1, where we take the DDPG algorithm as an example. The training procedure starts with defining the abstraction

function ϕ according to γ (Line 2), initializing PLDNN with an abstraction layer based on ϕ (Line 3), and, following [37], initializing the critic network and the two target networks (Line 4). The procedure then invokes the DDPG algorithm with the networks as arguments (Line 5) since the PLDNN has the same input and output as the actor network implemented by DNN. During this procedure, we freeze the parameters between the input and the abstraction layers of π . The parameters in the fully connected layers are trained based on backpropagation [39] and gradient descent optimization [40].

4 Equivalent Policy Extraction and Verification

After training, we can extract | S ϕ | LCUs based on the learned coe ffi cients of linear control policies for the abstract states in S ϕ . Specifically, we choose an arbitrary actual state s ∈ ϕ -1 ( s ϕ ) for each abstract state s ϕ and feed it to a PLDNN to obtain the coe ffi cients defined on the abstract state. For instance, we can feed ( -1 , -1) to the PLDNN in Example 1 and obtain the coe ffi cients ( -0 . 16610657 , -1 . 7437580 , -1 . 8227874) of the linear control policy for the region [ -2 , 0) × [ -2 , 0). Figure 3 shows the LCUs extracted from a trained PLDNN in Example 1. They are depicted by planes with di ff erent colors in Figure 3. These four planes denote the following linear control functions:

The underlying x 1 × x 2 plane in Figure 3 is the projection of the four LCUs. Under the control of these four LCUs, the agent can reach the goal region (orange box). The two sequences of purple boxes represent the range of reachable states from corresponding initial regions to the goal region.

With exacted LCUs from a PLDNN, we can build a verifiable hybrid automaton for the system by substituting equivalently the neural networks using corresponding LCUs. Theorem 2 formulates the hybrid automaton after a decision network π is substituted by LCUs. The di ff erences from Theorem 1 include the definitions of transitions T , guard condition G and reset formula R . For the PLDNN controlled systems, We use | S ϕ | transitions each of which contains a guard condition and a reset formula to update the action a to π c ( s ) · [1 , s ] at each δ . Notice that, π c ( s ) is a determined vector since ∀ s ∈ ϕ -1 ( s i ϕ ), the outputs of π c are the same according to the dedicated structure of π c . Thus, the reset formula for a is simplified to an a ffi ne mapping.

π(C1,C2)

5

0

2

-5

1

0

C2

-2

-1

-1

0

C1

1

2

-2

Theorem 2. Given a DRL system with environment dynamics f, PLDNN π , time step size δ and initial state set S 0 , it can be equivalently modeled as a hybrid automaton defined as follows:

Var: state variable s, action a, clock variable tc · I 0 : { ( l 0 , ( s ∈ S 0 , a = 0 , t c = δ )) }

T : { ( l 0 , l 0 ) , · · · , ( l 0 , l 0 ) } where | T | = | S ϕ |

L : { l 0 } · Inv : Inv ( l 0 ) = { s ∈ S , t c ≤ δ } · F : F ( l 0 ) = { ˙ s = f ( s , a ) , ˙ a = 0 , ˙ t c = 1 }

G : G ( T [ i ]) = { t c = δ, s ∈ ϕ -1 ( s i ) } where 0 ≤ i < | T | ∧ s i ∈ S ϕ

ϕ

ϕ

R : R ( T [ i ]) = { t c = 0 , a = π c ( s ) · [1 , s ] } where 0 ≤ i < | T | ∧ s ∈ ϕ -1 ( s i )

ϕ

Proof. We prove that the path of hybrid automaton H 1 in Theorem 1 is the same as the corresponding hybrid automaton H 2 in Theorem 2 for an arbitrary initial state ( l 0 , [ s 0 , 0 , δ ]). The path of H 2 can be described by the following sequence:

where si + 1 = si + R δ 0 f ( s , ai ) dt and ai = π c ( si ) · [1 , si ]. With the dedicated structure of PLDNN π , we have π ( s ) = π c ( s ) · [1 , s ]. Thus, at each δ , the discrete transitions of H 1 and H 2 change both location

and continuous variables in the same way. We can conclude that the paths of H 1 and H 2 are exactly the same. By Theorem 1, it is obvious that D , H 1 and H 2 produce the same state-action value for the same initial state.

According to Theorem 2, we can build a hybrid automaton that is equivalent to the DRL system in Example 1. Assuming the trained system has four linear control units as shown in Formulas (LCU1LCU4) and δ = 0 . 2, we construct the corresponding hybrid automaton as depicted in Figure 4. The four transitions in the automaton correspond to the four LCUs, respectively. The guard of each transition represents the condition of triggering the corresponding policy.

Thanks to the linearity of control policies in R , a hybrid automaton built for a PLDNNcontrolled system can be e ffi ciently verified by state-of-the-art tools. For instance, Flow* [24] is a representative tool for the reachability analysis of hybrid systems. In this paper we are focused on the verification of both goal-reach and reach-avoid properties. The former means that given a set of initial states, a system must eventually reach the goal region from any initial state. The latter means that the system never enters unsafe regions within a specific time horizon. Both properties can be verified via reachability analysis.

5 Experimental Evaluations

We prototype our approach into a tool called L in C on , with DDPG as the backend DRL algorithm and Flow* as the verification engine. We extensively assess it, along with the state-of-the-art tools. Our goal is to show, for the PLDNN-based training, (i) the reduction in the number of partitions with comparable cumulative rewards, robustness, and time overhead with respect to conventional DNNbased training; (ii) its high verification performance including the tightness of over-approximation sets and the e ffi ciency of verification; and (iii) its scalability for large-sized neural networks and systems with complex dynamics and high-dimensional state space.

Experimental Setup. All experiments were conducted on a workstation equipped with a 32-core AMDRyzen Threadripper CPU @ 3.6GHz and 256GB RAM, running Ubuntu 22.04.

Benchmarks. We choose eight benchmarks, including six regular benchmarks from Verisig 2.0 [9] (B1-B5 and Tora) and two complex benchmarks (CartPole with extreme complex dynamics from OpenAI Gym [41] and quadrotor (QUAD) with 12-dimensional state space and 3-dimensional action space from [10]). For fair comparisons, we use the same training configuration and guarantee that all trained systems reach the specified reward threshold. See Appendix A.1 for the detailed setting. For B1-B5 and Tora the environment dynamics are the same with [9]. The dynamics of QUAD are the same with [10]. We omit their detailed definition here. However, the dynamics of CartPole are originally represented as a discrete-time model in [41] by di ff erence equations. In our experiment, we consider its dynamics as a continuous-time model and formalize it with the following ODEs:

5.1 Performance Evaluation

We assess the performance of PLDNN, together with the conventional DNNs, in terms of cumulative reward, robustness, and training time under the same training configuration. We also measure the number of abstracted states required for training linear control policies and constant policies [21]. Due to space limitations, we present the experimental results only for B1, B2, and two complex cases

l

0

;

x

1

;

x

2

a

:

=

t

c

:

=

0

G

2

I

F

R

2

a

:

=

LCU3

t

c

:

=

0

a

:

=

LCU4

t

c

:

=

0

a

:

=

t

c

:

=

0

G

4

G

3

G

1

R

1

R

3

R

4

t

c

=

0

:

2

x

1

2

[

GLYPH<0>

2

;

0]

x

2

2

[

GLYPH<0>

2

;

0]

t

c

=

0

:

2

;

x

1

2

[

GLYPH<0>

2

;

0]

;

x

2

2

[0

;

2]

t

c

=

0

:

2

;

x

1

2

[0

;

2]

;

x

2

2

[0

;

2]

˙

x

1

=

x

2

GLYPH<0>

x

3

1

˙

a

=

0

˙

t

c

=

1

˙

x

2

=

a

x

1

2

[

GLYPH<0>

2

;

2]

t

c

GLYPH<20>

0

:

2

x

2

2

[

GLYPH<0>

2

;

2]

t

c

=

0

:

2

x

1

2

[0

;

2]

x

2

2

[

GLYPH<0>

2

;

0]

B1

B2

CartPole

QUAD

Performance

Robustness

100

1000

Reward

0

50

500

500

-100

0

0

DNN

DNN

0

DNN

DNN

-200

PLDNN

-50

PLDNN

PLDNN

-500

PLDNN

0

100

200

300

0

100

200

300

0

1000

2000

3000

0

1000

2000

Episode

Episode

Episode

Episode

60

90

900

1000

Reward

58

80

800

800

DNN

70

DNN

DNN

DNN

56

PLDNN

60

PLDNN

700

PLDNN

600

PLDNN

0

50

100

0

25

50

75

100

0

25

50

75

100

0

50

100

Noise (0.0004)

Noise (0.0002)

Noise (0.0001)

Noise (0.0005)

(i.e., CartPole and QUAD). The associated conclusions (from Table 1 and Figure 5) also apply to the other four cases; see Appendix A.2 for the detailed experimental results.

Cumulative Reward. Figure 5 plots the system cumulative reward (the average of five trials) during the training process. The solid lines and shadows refer to the average reward and 95% confidence interval, respectively. Apparently, the trends of accumulative rewards by PLDNNs and DNNs

are comparable. Despite slightly more time required for training with PLDNN due to its additional conversion from actual state to abstract state (see Table 1, 'Training Time'), we can, however, obtain a significant advance in the e ffi ciency and tightness of verification (see Section 5.2).

Robustness. We also evaluate the robustness of PLDNNs as training the linear controller on each partition may lead to discontinuity of control decisions in the boundaries of partitions. For a current state s = ( x 1 , . . . , xn ), we add a Gaussian noise X 1 , ... Xn to s and obtain a perturbed state s ′ = ( x 1 + X 1 , . . . , xn + Xn ) for calculating the control action, where Xi ∼ N ( µ i , σ 2 i ) with 1 ≤ i ≤ n and µ i = 0. For each benchmark, we train 10 di ff erent policies and evaluate their robustness under 100 di ff erent perturbation levels to obtain the average and 95% confidence interval of the cumulative reward. Figure 5 depicts the reward trend with the increasing perturbation level. As σ increases, the decline ratio of the system with PLDNNs is comparable to that with DNNs, which implies that both achieve similar robustness.

Reduction in the Number of Partitions. We measure the e ff ect of reducing the number of partitions by utilizing a linear policy, instead of the constant action on each partition, which is adopted by the work [21]. In both cases, we start training from a coarse-grained abstraction granularity (with only one partition) and gradually increase the number of partitions until the preset reward threshold can be reached by the trained PLDNN. As shown in Table 1, linear policies significantly reduce the number of partitions required for reaching the reward threshold, which benefits both the verification e ffi ciency and accuracy as we will see in the next section.

5.2 Verification E ffi ciency and Tightness

We evaluate the verification e ffi ciency and tightness for the PLDNN-based and DNN-based DRL systems, respectively. Regarding tightness, we choose two state-of-the-art tools, i.e., Polar [10] and Verisig 2.0 [9], for the reachability analysis of DNN-controlled systems. We do not consider ReachNN* which both Polar and Verisig 2.0 have been demonstrated to outperform [9, 10]. For e ffi ciency, we employ Flow* to perform reachability analysis, which is used by both Polar and Verisig 2.0 as the backend reachability analysis tool.

E ffi ciency. Table 2 presents the comparison results for the verification e ffi ciency. For each regular case, we choose four di ff erent network configurations: two smaller networks (e.g., Tanh2 × 20) from [10] and two larger networks (e.g., Tanh3 × 100). Our approach L in C on can handle all 26 instances including the two complex instances, while Polar succeeds only in 20 cases. Verisig 2.0 is not applicable to

Remarks. Impr. : time speedup of L in C on compared to Verisig or Polar (Verisig or Polar / L in C on ). Tanh / ReLU n × k : a DNN with the activation function Tanh / ReLU, n hidden layers, and k neurons per hidden layer. VR : verification result. ✓ : the reachability problem is successfully verified. ✗ type : the reachability problem cannot be verified due to type : ( a ) large over-approximation error, ( b ) the calculation did not finish, ( c ) not applicable. -: no data available due to ✗ b or ✗ c .

ReLU networks (marked by ✗ c ) and succeeds only in 9 instances. Overall, L in C on outperforms both Polar (up to 47 . 7 × speedup) and Verisig 2.0 (up to 438 . 6 × speedup). In particular, L in C on achieves even up to 58 . 1 × speedup compared to Verisig 2.0 accelerated by 20-core parallelization. For our approach L in C on , the only time overhead for encoding networks stems from extracting LCUs from PLDNNs, which is polynomial and negligible (less than 0.05s). Consequently, L in C on can scale up to large-sized neural networks.

Tightness. We compare the tightness of the over-approximation sets computed by di ff erent approaches. Figure 6 plots the experimental results, along with the corresponding simulation trajectories. For B2 and Tora, L in C on has a significant tightness improvement over Polar: the range of the over-approximation sets (red boxes) calculated by Polar far outreaches the range of the reachable states obtained from the simulation. Verisig 2.0 terminates prematurely in Tora since the range of the action reaches 10 7 during the calculation which is too large to proceed. We defer to Appendix A.3 the experimental results for the remaining four cases where all three tools obtain similar tightness results.

Discussion on CartPole and QUAD. We discuss the verification results of the two complex cases, namely CartPole and QUAD. Both Polar and Verisig 2.0 fail to verify them. As shown in Figure 6, Polar and Verisig 2.0 abort after 20 steps due to the huge over-approximation error in CartPole. In contrast, such a complex policy trained with our approach can be e ffi ciently and tightly verified. In particular, the trajectories diverge first and finally merge. The computed reachable states tightly over-approximate these trajectories, and the computation takes only 151 seconds. Regarding the 12-dimensional QUAD case, Polar and Verisig 2.0 time out (two hours) after only two steps, while L in C on produces a very tight set of reachable states in 1054 seconds even after 15 steps, which is 7 times deeper than Polar. To the best of our knowledge, this is the first time that QUAD can be formally verified more than ten steps under various decision networks. Note that the trained policies used in the comparison di ff er due to di ff erent decision networks, and thus agents may follow di ff erent paths to the goal region (see the simulated trajectories in Figure 6). Hence, for a fair comparison, we conduct more evaluations, from which we can draw the same conclusion as from Figure 6. The results are given in Appendix A.3.

B2

Tora

CartPole

QUAD

L

in

C

on

Polar

Verisig 2.0

1.0

1.0

1.0

0.5

0.5

0.02

0.8

X

X

0.0

0.00

0.6

0.0

-0.5

0.4

-0.5

-0.02

-1.0

0.2

0.5

0.0

0.5

1.0

-1.0-0.5

0.0

0.5

1.0

0

1020304050

0

5

10

15

x1

x1

Time step

Time step

1.0

1.0

1

0.02

0.5

0.00

0.8

0_0.02

0.0

20.6

-0.04

-0.5

0.4

-0.06

-1.0

-1

0.2

-0.5

0.0

0.5

1.0

-1

0

1

0

1020304050

0

5

10

15

x1

x1

Time step

Time step

1.0

0.02

1.0

0.5

0.00

0.8

0.0

0

0_0.02

2 0.6

-0.04

0.4

-0.5

-0.06

1

0.2

-0.5

0.0

0.5

1.0

-1

0

0

1020304050

0

5

10

15

x1

x1

Time step

Time step

6 Related Work

Policy Synthesis. Several works adopt programmatic policies (e.g., decision trees and program controllers) which are more interpretable and amenable to formal verification than neural policies. Bastani et al. [15] construct a decision tree to represent a DNN policy based on imitation learning [42]. Verma et al. [43, 44] follow a similar routine, distilling neural network policies into predefined program templates. Trivedi et al. [45] use a two-stage learning scheme to synthesize programmatic policies. Some e ff orts are dedicated to exploring the combination of training and verification. Zhu et al. [30] propose an inductive framework for synthesizing a deterministic policy program from neural policies. Wang et al. [31] learn programmatic controllers based on verification feedback to avoid safety violations. Our proposed PLDNN is essentially a DNN-based implementation of programmatic controllers, which could be integrated with these verification-guided synthesis approaches.

Reachability Analysis. Our work is also built atop the approaches for reachability analysis of neuralnetwork-controlled systems. NNV [46] utilizes star set [47] to perform range analysis of decision networks. JuliaReach [48] uses zonotope propagation to cover the output of a decision network. Verisig [49, 9] models a decision network with di ff erentiable activation functions (e.g., Tanh) as a hybrid system and analyzes its reachability for over-approximating the network. ReachNN* [28, 8] abstracts the input-output mapping of a decision network with a Bernstein polynomial, together with an error bound on the approximation. Sherlock [7] focuses on ReLU-based networks and computes tight Taylor models via rule generation. Polar [10] integrates the Taylor and Bernstein approximation techniques for building a Taylor model which over-approximates decision networks. All these e ff orts over-approximate the embedded DNNs, which limit their scalability and verification accuracy.

7 Conclusion and Future Work

We have presented PLDNN that seamlessly integrates DNN and programmatic controls via state abstraction for boosting the reachability analysis of DRL systems. Unlike traditional train-thentransform approaches, PLDNN accompanies a novel inverse training and verification method, in which a DNN is first transformed into an equivalent set of linear control policies and then trained to optimize them. Experimental results have shown that PLDNN-controlled systems can be more e ffi ciently and tightly verified than DNN-based systems, with up to 438 times speedup and 7 times deeper computation steps for a 12-dimensional control task.

Our work sheds light on a promising direction towards developing dependable DRL systems: learning easy-to-verify and high-performance control policies via DRL and abstraction techniques. Given the encouraging results of linear control policies, our work would also stimulate a passion for substituting them with, e.g., polynomial control policies, for training and verifying more complex DRL systems.

Acknowledgment

We thank all the anonymous reviewers for their valuable feedback on this work. The work has been supported by the National Key Research Program (2020AAA0107800), NSFC-ISF Joint Program (62161146001,3420 / 21), Huawei Technologies Co., Ltd., the Shanghai International Joint Lab of Trustworthy Intelligent Software (Grant No. 22510750100), the Shanghai Trusted Industry Internet Software Collaborative Innovation Center, the Engineering and Physical Sciences Research Council (EP / T006579 / 1), the National Research Foundation (NRF-RSS2022-009), Singapore, and the Shanghai Jiao Tong University Postdoc Scholarship.

References

Z. Zhang and K. P. Lam, 'Practical implementation and evaluation of deep reinforcement learning control for a radiant heating system,' in Proceedings of the 5th Conference on Systems for Built Environments , 2018, pp. 148-157.

Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots, 'Agile autonomous driving using end-to-end deep imitation learning,' in Robotics: science and systems , 2018.

X. Qi, Y. Luo, G. Wu, K. Boriboonsomsin, and M. Barth, 'Deep reinforcement learning enabled selflearning control for energy e ffi cient driving,' Transportation Research Part C: Emerging Technologies , vol. 99, pp. 67-81, 2019.

B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, 'Deep reinforcement learning for autonomous driving: A survey,' IEEE Transactions on Intelligent Transportation Systems , vol. 23, no. 6, pp. 4909-4926, 2021.

F.-L. Fan, J. Xiong, M. Li, and G. Wang, 'On interpretability of artificial neural networks: A survey,' IEEE Transactions on Radiation and Plasma Medical Sciences , vol. 5, no. 6, pp. 741-760, 2021.

R. V. Yampolskiy, 'Unexplainability and incomprehensibility of AI,' Journal of Artificial Intelligence and Consciousness , vol. 7, no. 02, pp. 277-291, 2020.

S. Dutta, X. Chen, and S. Sankaranarayanan, 'Reachability analysis for neural feedback systems using regressive polynomial rule inference,' in ACM International Conference on Hybrid Systems: Computation and Control (HSCC'19) , 2019, pp. 157-168.

J. Fan, C. Huang, X. Chen, W. Li, and Q. Zhu, 'Reachnn*: A tool for reachability analysis of neuralnetwork controlled systems,' in International Symposium on Automated Technology for Verification and Analysis . Springer, 2020, pp. 537-542.

R. Ivanov, T. Carpenter, J. Weimer, R. Alur, G. Pappas, and I. Lee, 'Verisig 2.0: Verification of neural network controllers using taylor model preconditioning,' in International Conference on Computer Aided Verification (CAV'21) . Springer, 2021, pp. 249-262.

C. Huang, J. Fan, X. Chen, W. Li, and Q. Zhu, 'Polar: A polynomial arithmetic framework for verifying neural-network controlled systems,' in The 20th International Symposium on Automated Technology for Verification and Analysis (ATVA'22) . Springer, 2022, pp. 414-430.

K. Makino and M. Berz, 'Taylor models and other validated functional inclusion methods,' International Journal of Pure and Applied Mathematics , vol. 6, pp. 239-316, 2003.

W. Ruan, X. Huang, and M. Kwiatkowska, 'Reachability analysis of deep neural networks with provable guarantees,' in Proceedings of the 27th International Joint Conference on Artificial Intelligence , 2018, pp. 2651-2659.

S. Dutta, S. Jha, S. Sankaranarayanan, and A. Tiwari, 'Output range analysis for deep feedforward neural networks,' in 10th International Symposium on NASA Formal Methods (NFM'18) . Springer, 2018, pp. 121-138.

L. M. Schmidt, G. Kontes, A. Plinge, and C. Mutschler, 'Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning,' in 2021 IEEE Intelligent Vehicles Symposium (IV) . IEEE, 2021, pp. 171-178.

O. Bastani, Y. Pu, and A. Solar-Lezama, 'Verifiable reinforcement learning via policy extraction,' Advances in Neural Information Processing Systems (NeurIPS'18) , vol. 31, 2018.

C. Buciluˇ a, R. Caruana, and A. Niculescu-Mizil, 'Model compression,' in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD'06) , 2006, pp. 535-541.

G. Hinton, O. Vinyals, and J. Dean, 'Distilling the knowledge in a neural network,' arXiv preprint arXiv:1503.02531 , 2015.

T. Brázdil, K. Chatterjee, M. Chmelik, V. Forejt, J. Kˇ retínsk` y, M. Kwiatkowska, D. Parker, and M. Ujma, 'Verification of markov decision processes using learning algorithms,' in 12th International Symposium on Automated Technology for Verification and Analysis (ATVA'14) . Springer, 2014, pp. 98-114.

D. Abel, D. E. Hershkowitz, and M. L. Littman, 'Near optimal behavior via approximate state abstraction,' in International Conference on Machine Learning (ICML'16) , vol. 48, 2016, pp. 2915-2923.

D. Abel, 'A theory of state abstraction for reinforcement learning,' in AAAI Conference on Artificial Intelligence (AAAI'19) , vol. 33, no. 01, 2019, pp. 9876-9877.

P. Jin, J. Tian, D. Zhi, X. Wen, and M. Zhang, 'Trainify: A CEGAR-driven training and verification framework for safe deep reinforcement learning,' in International Conference on Computer Aided Verification (CAV'22) . Springer, 2022, pp. 193-218.

J. Tian, D. Zhi, S. Liu, P. Wang, G. Katz, and M. Zhang, 'Taming reachability analysis of dnn-controlled systems via abstraction-based training,' in 25th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI'23) . Springer, 2023.

R. Alur, C. Courcoubetis, N. Halbwachs, T. A. Henzinger, P.-H. Ho, X. Nicollin, A. Olivero, J. Sifakis, and S. Yovine, 'The algorithmic analysis of hybrid systems,' Theoretical Computer Science , vol. 138, no. 1, pp. 3-34, 1995.

X. Chen, E. Ábrahám, and S. Sankaranarayanan, 'Flow*: An analyzer for non-linear hybrid systems,' in 25th International Conference on Computer Aided Verification (CAV'13) . Springer, 2013, pp. 258-263.

P. Collins, D. Bresolin, L. Geretti, and T. Villa, 'Computing the evolution of hybrid systems using rigorous function calculus,' IFAC Proceedings Volumes , vol. 45, no. 9, pp. 284-290, 2012.

M. Altho ff , 'An introduction to cora 2015.' ARCH@ CPSWeek , vol. 34, pp. 120-151, 2015.

X. Chen, 'Reachability analysis of non-linear hybrid systems using taylor models,' Ph.D. dissertation, Fachgruppe Informatik, RWTH Aachen University, 2015.

C. Huang, J. Fan, W. Li, X. Chen, and Q. Zhu, 'Reachnn: Reachability analysis of neural-network controlled systems,' ACM Transactions on Embedded Computing Systems , vol. 18, no. 5s, pp. 1-22, 2019.

M. Grze´ s and D. Kudenko, 'Multigrid reinforcement learning with reward shaping,' in 18th International Conference Artificial Neural Networks (ICANN'08) . Springer, 2008, pp. 357-366.

H. Zhu, Z. Xiong, S. Magill, and S. Jagannathan, 'An inductive synthesis framework for verifiable reinforcement learning,' in Proceedings of the 40th ACM SIGPLAN conference on Programming Language Design and Implementation (PLDI'19) , 2019, pp. 686-701.

Y. Wang and H. Zhu, 'Verification-guided programmatic controller synthesis,' in 29th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'23) . Springer, 2023, pp. 229-250.

P. Cousot and R. Cousot, 'Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints,' in Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages (POPL'77) , 1977, pp. 238-252.

M. Afzal, A. Asia, A. Chauhan, B. Chimdyalwar, P. Darke, A. Datar, S. Kumar, and R. Venkatesh, 'Veriabs: Verification by abstraction and test generation,' in The 34th IEEE / ACM International Conference on Automated Software Engineering (ASE'19) . IEEE, 2019, pp. 1138-1141.

K. Heo, H. Oh, and H. Yang, 'Resource-aware program analysis via online abstraction coarsening,' in 2019 IEEE / ACM 41st International Conference on Software Engineering (ICSE'19) . IEEE, 2019, pp. 94-104.

Z. Wang, A. Albarghouthi, G. Prakriya, and S. Jha, 'Interval universal approximation for neural networks,' Proceedings of the ACM on Programming Languages (POPL) , vol. 6, no. POPL, pp. 1-29, 2022.

E. Gallestey and P. Hokayem, 'Lecture notes in nonlinear systems and control,' 2019.

T. P. Lillicrap, J. J. Hunt, A. Pritzel et al. , 'Continuous control with deep reinforcement learning,' in International Conference on Learning Representations (ICLR'16) . OpenReview.net, 2016.

S. Fujimoto, H. Hoof, and D. Meger, 'Addressing function approximation error in actor-critic methods,' in International Conference on Machine Learning (ICML'18) . PMLR, 2018, pp. 1587-1596.

D. E. Rumelhart, R. Durbin, R. Golden, and Y. Chauvin, 'Backpropagation: The basic theory,' Backpropagation: Theory, architectures and applications , pp. 1-34, 1995.

S.-i. Amari, 'Backpropagation and stochastic gradient descent method,' Neurocomputing , vol. 5, no. 4-5, pp. 185-196, 1993.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, 'OpenAI Gym,' 2016, arXiv:1606.01540.

P. Abbeel and A. Y. Ng, 'Apprenticeship learning via inverse reinforcement learning,' in Proceedings of the twenty-first International Conference on Machine Learning (ICML'04) , 2004, p. 1.

A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri, 'Programmatically interpretable reinforcement learning,' in International Conference on Machine Learning . PMLR, 2018, pp. 5045-5054.

A. Verma, H. Le, Y. Yue, and S. Chaudhuri, 'Imitation-projected programmatic reinforcement learning,' Advances in Neural Information Processing Systems , vol. 32, 2019.

D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim, 'Learning to synthesize programs as interpretable and generalizable policies,' Advances in Neural Information Processing Systems (NeurIPS'21) , vol. 34, pp. 25 146-25 163, 2021.

H.-D. Tran, X. Yang, D. Manzanas Lopez, P. Musau, L. V. Nguyen et al. , 'NNV: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems,' in International Conference on Computer Aided Verification (CAV'20) . Springer, 2020, pp. 3-17.

H.-D. Tran, S. Bak, W. Xiang, and T. T. Johnson, 'Verification of deep convolutional neural networks using imagestars,' in International Conference on Computer Aided Verification (CAV'20) . Springer, 2020, pp. 18-42.

C. Schilling, M. Forets, and S. Guadalupe, 'Verification of neural-network control systems by integrating taylor models and zonotopes,' in AAAI Conference on Artificial Intelligence (AAAI'22) , 2022, pp. 81698177.

R. Ivanov, T. J. Carpenter, J. Weimer, R. Alur, G. J. Pappas, and I. Lee, 'Verifying the safety of autonomous systems with neural network controllers,' ACM Transactions on Embedded Computing Systems , vol. 20, no. 1, pp. 1-26, 2020.

A Appendix

A.1 Benchmarks Setting

We first present the goal region and unsafe region of each benchmark in Table 3.

As for the training settings, for the 6 regular benchmarks, the target is training a policy to guide the agent to reach the goal region. Therefore, we set a negative reward when the agent is not in the goal region. Once the agent reaches the goal region, it will be awarded a positive reward. In CartPole, the target of training is to prevent the pole from falling over, namely the pole angle -0 . 2 ≤ x 3 ≤ 0 . 2. We modify the original discrete reward function to a continuous reward function -| x 3 | to try to balance the pole to stay upright. As for QUAD, we aim to control the altitude of the quadrotor above 0.6 ( x 3 ≥ 0 . 6). Thus we set a continuous reward function -| x 3 -0 . 8 | for training a policy that drives the quadrotor to ascend above 0.6.

A.2 Comparison on cumulative reward and robustness

In this section, we provide the comparison results on cumulative reward and robustness of B3-B5 and Tora in Figure 7. The solid lines and shadows refer to the average reward and 95% confidence interval, respectively. For these four cases, it is obvious that there are comparable trends in the cumulative rewards of PLDNNs and DNNs during training and under perturbation. Therefore, we can conclude that using PLDNNs will not a ff ect the training e ffi ciency and the robustness of trained systems.

A.3 Comparison on the tightness of over-approximation sets

Tightness results of regular cases. We present the tightness comparison results of B1 and B3-B5 in Figure 8. For these four regular cases, both Polar and Verisig 2.0 can produce tight over-approximation for decision networks, thus all three methods achieve similar tightness results and successfully verify the goal-reach properties.

Multiple evaluations on CartPole and QUAD. Since the trained policies used in the comparison di ff er due to di ff erent decision networks, we conduct more evaluations on these two complex cases. For each case, we train three more decision networks with the same configuration and the corresponding results are shown in Figure 10. For CartPole, Polar did not finish the calculation under three di ff erent networks as depicted in Figure 10(d-f). Additionally, on the basis of the result of dealing with divergent traces as shown in Figure 10(d), we can see that Polar is not suitable for dealing with the DNN-controlled systems with divergent traces. As for QUAD, we record the computation results within 12 hours and obtain the results as shown in Figure 10(j-l). We can observe that only

(a) B3

(b) B4

(c) B5

(d) Tora

(e) B3

(f) B4

(g) B5

(h) Tora

200

1000

400

400

750

0

200

200

01

500

-200

DNN

0

DNN

-200

DNN

250

DNN

PLDNN

PLDNN

PLDNN

PLDNN

0

100

200

-200

0

20

40

60

-400

0

200

400

600

0

0

100

200

Episode

Episode

Episode

Episode

500

500

1000

175

400

400

800

150

300

125

DNN

300

DNN

DNN

600

DNN

PLDNN

PLDNN

200

PLDNN

PLDNN

100

0

25

50

75

100

0

25

50

75

100

0

25

50

75

100

0

50

100

Noise (0.0005)

Noise (0.0002)

Noise (0.0002)

Noise (0.0005)

a few time steps are completed by Polar, while L in C on can accomplish more than 10 steps within about 1000 seconds under multiple evaluations.

(a) B1 by L

in

C

on

(b) B3 by L

in

C

on

(c) B4 by L

in

C

on

(d) B5 by L

in

C

on

(e) B1 by Polar

(f) B3 by Polar

(g) B4 by Polar

(h) B5 by Polar

(i) B1 by Verisig 2.0

(j) B3 by Verisig 2.0

(k) B4 by Verisig 2.0

(l) B5 by Verisig 2.0

0.5

0.6

0.08

0.60

0.3

0.45

0.0

0.04

0.30-

0.0

0.00

-0.5

0.15

-0.3

0.04

0.00

-0.5

0.0

0.5

1.0

0.2

0.4

0.6

0.8

1.0

0.15

0.00

0.15

0.30

-0.4

-0.2

0.0

0.2

0.4

x1

x1

x1

x1

0.6

0.60

0.5

0.08

0.3

0.45

0.0

X

0.04

0.30

0.0

0.00

0.15

-0.5

0.3

0.04

0.00

-0.5

0.0

0.5

1.0

0.2

0.4

0.6

0.8

1.0

0.15

0.00

0.15

0.30

-0.4

0.2

0.0

0.2

0.4

x1

x1

x1

x1

0.5

0.6

0.60

0.08

0.3

0.45

0.04

0.0

0.30

0.0

0.00

0.15

-0.5

-0.3

0.04

-0.5

0.0

0.5

1.0

0.2

0.4

0.6

0.8

1.0

0.15

0.00

0.15

0.30

0.00

0.4

0.2

0.0

0.2

0.4

x1

x1

x1

x1

A.4 Evaluation on Verisig 2.0 with Big Weights

Verisig 2.0 produces large over-approximation error when dealing with neural networks with big weights. To demonstrate this, we initialize the weights of the neural network with larger values (random numbers wl ∼ N ( µ, σ 2 ) with µ = 0 , σ = 0 . 1) and show the experimental results in Figure 9. We observe that the calculated over-approximation sets contain large over-approximation error except for B4. In Tora, Verisig 2.0 fails to calculate the complete reachable sets due to too large overapproximation error. Hence, it is fairly to say that Verisig 2.0 is sensitive to the DNNs with big weights.

(a) B1

(b) B2

(c) B3

(d) B4

(e) B5

(f) Tora

0.5

1.0

0.6

0.5

0.3

0.0

0.0

0.5

0.0

0.5

-1.0

0.3

0.5

0.0

0.5

1.0

1.5 ~

0.5

0.0

0.5

1.0

0.2

0.4

0.6

0.8

1.0

x1

x1

x1

0.60

1.0

0.08

0.45

0.5

0.04

0.30-

0.0

0.00

0.15

-0.5

0.04

0.00

-1.0

0.15

0.00

0.15

0.30

-0.4

0.2

0.0

0.2

0.4

1.0

0.5

0.0

0.5

1.0

x1

x1

x1

Network

⟨

i

⟩

Network

⟨

ii

⟩

Network

⟨

iii

⟩

CP by L

in

C

on

(a)

(b)

(c)

CP by Polar

(d)

(e)

(f)

QUAD by L

in

C

on

(g)

(h)

(i)

QUAD by Polar

(j)

(k)

(l)

0.02

0.02

0.02

0.00

0.00

θ

0.00

-0.02

-0.02

-0.02

0

10203040

50

0

1020304050

0

1020304050

Time step

Time step

Time step

0.06

0.06

0.04

0.02

0.04

θ0.02

θ

θ0.02

0.00

0.00

0.00

-0.02

-0.02

-0.02

0

1020304050

0

1020304050

0

1020304050

Timestep

Timestep

Time step

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6-

0.4

0.4

0.4

0.2

0.2

0.2

0

5

10

15

0

5

10

15

0

5

10

14

Time step

Time step

Time step

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

20.6

0.4

0.4

0.4

0.2

0.2

0.2

0

5

10

15

0

5

10

15

0

5

10

15

Time step

Time step

Time step

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an inverse transform-then-train approach for verifying deep reinforcement learning systems. It encodes a DNN into efficiently verifiable linear control policies and optimizes them via reinforcement learning. The approach is compatible with existing DRL training algorithms and shows that PLDNN-based systems can be more efficiently and tightly verified, with up to 438 times speedup and a significant reduction in overestimation.\n\nSTRENGTHS:\n+ The paper explores an interesting and important direction\n+ Innovative and practical methodology\n\nWEAKNESSES:\n- Some technical parts are unclear\n- The way of presenting contributions of the work is somewhat misleading\n- The limitation of using PLDNN is not discussed\n\nQUESTIONS:\n##\n## Originality\n\nTo efficiently verify deep reinforcement learning systems, this paper proposes an inverse transform-then-train technique. It efficiently converts a DNN into verifiable linear control policies, which it then uses reinforcement learning to improve. The method is compatible with current DRL training algorithms and demonstrates that PLDNN-based systems can be more effectively and tightly verified, with speedups of up to 438 times and a notable decrease in overestimation.\n\n##\n## Importance of contribution\n\nThe idea of using PLDNN to boost the verification of DRL is interesting. However, it seems like it converts the DNN into a different structure before training, and the PLDNN is different from the original DNN. To me, it is more like a synthesis technique, rather call it as verification. Moreover, the authors claim that it is compatible with most existing DRL training algorithms, but in the evaluation, the details of the training algorithms are not shown.\n\n##\n## Soundness\n\nThis paper proposes an inverse transform-then-train approach for verifying deep reinforcement learning systems, overcoming inaccuracies and scalability issues. It uses piece-wise linear decision neural networks (PLDNNs) for efficient verification, reducing overestimation. However, there are some technical parts that are not so clear and need further clarification. \n\nPage 1, Line 21\n> Most existing approaches [7–10] over-approximate both embedded DNNs and non-linear environment dynamics to build verifiable models, which inevitably introduces dual overestimation.\n\nI am not sure what this dual-estimation is. The sentence following seems only talk about the disadvantage of over-approximation.\n\nPage 2, Line 40\n> ..., we propose a novel, inverse transform-then-train approach:\n\nIt seems like the approach cannot verify a trained DRL agent. So it is more like a synthesis technique.\n\nPage 5, Line 197\n> Consequently, we can extract a piecewise linear decision function with this structure of π on each abstract state.\n\nHow the piecewise linear functions are extracted? Could you explain it more detailedly?\n\n##\n## Evaluation\n\nThe authors try to evaluate whether PLDNN-based training offers reduced partitions, comparable rewards, robustness, time overhead, high verification performance, and scalability for large-sized neural networks and complex systems with high-dimensional state space. In Section 5.2, the authors compare Lincon with polar and verisig 2.0. Have you compared the efficiency of these tools on the original DNNs? Since, maybe the architecture of PLDNN is not easy to be handled for Polar and Verisig 2.0.\n\n##\n## Quality of presentation\n\nThe paper is not that easy to follow. I would recommend the authors add more motivations for the design of each single component.\n\n##\n## Comparison with Related Work\n\nThe PLDNN structure is similar to the DNN with abstract states listed below. What are the main difference and advantages of using PLDNN?\n\n- Jin, Peng, et al. \"Trainify: a CEGAR-driven training and verification framework for safe deep reinforcement learning.\"\n\nLIMITATIONS:\nNo, the authors did not discuss the limitation of the work. I think more attention should be given to the disadvantages of using PLDNN, like is it still the same original network?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach for designing neural network policies, in the context of deep reinforcement learning (DRL) with continuous state and action spaces, that are more amenable to verification compared to standard networks. A standard approach to verifying continuous DRL systems is to abstract the system after learning the policy and to then verify the abstracted system. In contrast, the proposed approach first abstracts the state space (such that the state space is discretized) and then learns a linear policy for each abstract state. The coefficients of the linear policy are learnt using a coefficient network that maps each abstract state to its resulting policy coefficients. This mapping itself need not be linear and can be a complex neural network. However, since the state space is discretized and the policy for each abstract state is fixed and linear, tight and efficient verification of the resulting DRL system becomes feasible. Moreover, the state abstraction computation is itself encoded as the first layer of the coefficient network, so that the resulting policy network (referred to as piece-wise linear decision neural networks or PLDNNs) has a standard input (system state) and an output (actions), and can be directly trained using off-the-shelf DRL algorithms. The empirical evaluation shows that the performance of PLDNNs, measured in terms of cumulative reward and robustness of the policy, is comparable to standard policy networks. At the same time, DRL systems with PLDNN policies are much more amenable to verification.\n\nSTRENGTHS:\nEven when the system dynamics are known, formal verification of safety and liveness properties of DRL systems is challenging to scale, specially for continuous state and action spaces. The paper proposes a very interesting idea for architecting policy networks that are easier to verify. I believe that the proposed notion of abstracting the state space before learning the policy will be a fruitful research direction. I also find the observation that a small set of linear policies can perform as well as a complicated non-linear policy to be surprising and interesting in its own right. I should add the disclaimer that I am not an expert on the topic of reinforcement learning and do not have a good sense of the broader literature on the topic.\n\nWEAKNESSES:\nMy main concern is about the \"scalability of verification vs cumulative reward\" tradeoff for DRL systems with high-dimensional state spaces (for instance, when the inputs to the policy network are images). As the state space becomes more complex, I suspect that a finer-grained partition becomes necessary to learn a good policy, directly impacting the scalability of verification. The presented evaluation considers system states with up to 12 dimensions, so it is hard to make an assessment about PLDNNs in settings where inputs are images required hundreds to thousands of dimensions.\nI am also curious about how the proposed approach would be used when the action space is discrete. Would each abstract state be mapped to a fixed action? Some discussion about this setting could be interesting and helpful. Finally, I have minor concerns about the empirical evaluation (described in the Questions section below).\n\nQUESTIONS:\nI have some clarification questions about the evaluation.\n\n1. What were the model architectures used for the results reported in Section 5.1? Are these the same architectures as in Section 5.2?\n\n2. Are the architectures for PLDNN and DNN the same (except for the first layer) for the results in Section 5.1 and Section 5.2? If not, how were the DNN architectures chosen and how do we ensure that it is an apples-to-apples comparison?\n\n3. Is the time reported in Table 1 in seconds? And is it the total training time over all episodes?\n\n4. How is the reward for the robustness experiments calculated, i.e., what is the horizon length? More generally, what is the reward function? The appendix gives some details; it says that for the 6 regular benchmarks \"we set a negative reward when the agent is not in the goal region. Once the agent reaches the goal region, it will be awarded a positive reward\". How and what is the positive reward assigned? Why is the final cumulative reward so different across the 6 regular benchmarks (based on Fig 5 and 7) if their reward functions are so similar?\n\n5. Are the models used for the robustness experiment the same as the ones used for the cumulative reward experiment? If yes, then why do you say that 10 different policies are trained for the former while you conduct five trials for the latter? My understanding is that the models used for the robustness experiment are trained on non-perturbed data but the evaluation is conducted with perturbed data. If this is so, then the same models can and should be used for both sets of experiments.\n\n6. What are precisely the properties of the DRL systems being verified? Based on Table 3 in the appendix, my guess is that for B1-B5, Tora, and QUAD the verified property is a liveness property whereas for CartPole it is a safety property? Can you please confirm? Is the time horizon for properties bounded or unbounded? It would be helpful to include this detail in the paper as well.\n\nSome more general questions:\n\n1. Can you comment on the applicability of this approach in settings where the inputs are high-dimensional images?\n\n2. Can the approach be adapted to settings where the action space is discrete?\n\nLIMITATIONS:\nThe paper does not necessarily discuss the limitations and might benefit from a small discussion about the same.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new neural network architecture, PLDNN, for better verification of DRL trained / neural-network controlled closed-loop controlled systems. PLDNN differs by abstracting the input space with intervals and applying linear mappings in each abstract states. The controller represented by PLDNN can be integrated with different DRL algorithms. The experimental results showed that the PLDNN can retain the performance by training with DRL algorithms directly and achieve tighter reachable sets with better verification efficiency.\n\nSTRENGTHS:\n- The paper is sound and the topic of the paper is of high interest to the research community.\n- The paper does a good job at introducing technical details and makes the paper easy to follow.\n- The empirical study shows great improvement of the proposed method over existing reachability analysis methods for neural-network controlled systems.\n\nWEAKNESSES:\n- My main concern of the paper is lack of analysis on the policy of reducing the partitions. Though the experimental results show strong results of PLDNN, it is not clear how partitioning plays a role in the performance improvement. It could be for some partitioned space in the state space, a linear control policy is almost close to the optimal control policy. Some other partitioned region may need finer partitions to approach the optimal control policy with a combination of linear control policies. However, similar insights or observations are not provided in the paper. Some analysis on how number of partitions evolves during training or a comparison between the linear policy of partition reduction and a baseline, like a fixed number of partition, would be great to have.\n- The paper positioned PLDNN as a new transform-then-train approach. Training with PLDNN may result in some implicit benefits of finding near-optimal policy represented by PLDNN. But PLDNN and the partition schema are still applicable for the train-then-transform process, e.g., distilling a trained DNN to PLDNN. I would recommend authors to provide an empirical comparison between using PLDNN with transform-then-train and train-then-transform processes.\n\nQUESTIONS:\n- Could you provide more details about the linear policy for reducing the partitions?\n- Could you provide more details on how network are trained for benchmarks in the experimental section, e.g., training time? Are all network trained from scratch?\n\nLIMITATIONS:\nSome limitation discussion on training time cost would be great to have.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach towards more easily verifiable DRL agent. Instead of training a neural network and then applying verification tools to it, the paper proposes to partition the input state, train linear policies in each of the partition, and verify the resulting piecewise-linear policy as a hybrid automation in Flow*.\n\nSTRENGTHS:\n- Verifying DRL systems has gained research attention in the past several years due to its supposed application in many safety/cost-critical scenarios. Currently, scalability is a major concern. Therefore, the topic of the paper is of interests to the ML and verification community.\n\n- While most existing work verifies a neural network controller via input splitting + abstract interpretation, the paper proposes an interesting and novel alternative, which is to eagerly partition the input region before training and make sure that a linear policy is trained for each partition. Compared to the train-and-transform approach, the proposed approach guarantees linearity in each partition, which leads to faster verification empirically.\n\n- The proposed method is relatively easy to implement in existing training framework. The engineering trick that inserts hand-crafted neural network layers to make sure input from the same partition will be multiplied with the same weights is clever.\n\nWEAKNESSES:\nConceptually, the method leverages the observation that a relatively small set of linear functions is sufficient to achieve comparable performance to a more complex neural network. This observation is itself rather surprising and deserve a closer study. For the same input region where the PLDNN is linear, is the behavior of the neural network also relatively linear? Are there input region where non-linearity is truly needed and PLDNN's behavior is problematic?\n \nIf low training loss can truly be obtained by a set of linear functions, it seems reasonable to expect that the behavior of the neural network in the same partition is close to linear as well, making abstract interpretation based techniques relatively precise. Following this thought,  the performance gain in verification precision might be a construct of the baseline neural network being unnecessarily large. Have the authors tried whether smaller networks can result in similar performance?\n\nQUESTIONS:\nHow do you determine the number of segments along each dimension?\n\nHow is the system dynamic handled? Is it encoded precisely?\n\nTo my knowledge, Verisig 2.0 also uses Flow*. Is it fair to say that the main difference between LinCon and Verisig 2.0 is that the former verifies a PLDNN while the latter verifies a canonically trained neural network?\n\nLIMITATIONS:\nThe paper does not explicitly discuss its current limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}