SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Boosting Verification of Deep Reinforcement Learning via Piece-wise Linear Decision Neural Networks
Jiaxu Tian  $^{1}$ , Dapeng Zhi  $^{1}$ , Si Liu  $^{2}$ , Peixin Wang*  $^{3}$ , Cheng Chen  $^{1}$ , Min Zhang*  $^{1}$
$^{1}$  Shanghai Key Laboratory of Trustworthy Computing, East China Normal University
2 ETH Zürich
<sup>3</sup> University of Oxford
51215902014@stu.ecnu.edu.cn, zhi.dapeng@163.com, si.liu@inf.ethz.ch,
peixin.wang@cs.ox.ac.uk, {chchen,zhangmin}@sei.ecnu.edu.cn
Formally verifying deep reinforcement learning (DRL) systems suffers from both inaccurate verification results and limited scalability. The major obstacle lies in the large overestimation introduced inherently during training and then transforming the inexplicable decision-making models i.e., deep neural networks (DNNs), into easy-to-verify models. In this paper, we propose an inverse transform-then-train approach, which first encodes a DNN into an equivalent set of efficiently and tightly verifiable linear control policies and then optimizes them via reinforcement learning. We accompany our inverse approach with a novel neural network model called piece-wise linear decision neural networks (PLDNNs), which are compatible with most existing DRL training algorithms with comparable performance against conventional DNNs. Our extensive experiments show that, compared to DNN-based DRL systems, PLDNN-based systems can be more efficiently and tightly verified with up to 438 times speedup and a significant reduction in overestimation. In particular, even a complex 12-dimensional DRL system is efficiently verified with up to 7 times deeper computation steps.
Deep neural networks (DNNs) have been exhibiting appealing advantages in decision-making and control for deep reinforcement learning (DRL) systems [1-4]. Nonetheless, the complexity and inexplicability [5, 6] of DNNs render the formal verification of their hosting systems, quite often even themselves, inaccurate and unscalable. Most existing approaches [7-10] over-approximate both embedded DNNs and non-linear environment dynamics to build verifiable models, which inevitably introduces dual overestimation. In particular, DNN-specific overestimation is extremely unpredictable due to many factors such as the dimension of system states, the complexity of environment dynamics, and the size, weight, and activation function of a neural network. For example, the verification results may deviate significantly even if the DNNs of the same DRL system differ only in their weights (as we also observed; see Appendix A.4). Unsurprisingly, verifying high-dimensional DRL systems would only exacerbate the problems of large overestimation and limited scalability.
Common practice for formally verifying DRL systems is to train and then transform the embedded DNNs into easy-to-verify models where, for any input set, output ranges can enclose the outputs of the over-approximated DNNs [7-10]. Taylor models [11] are widely adopted due to their preservation of input-output dependencies and less overestimation (accumulated in multiple steps) than the range analysis approaches such as interval over-approximation [12, 13]. However, they are still prone to intractable overestimation as the accuracy of verification depends heavily on the weights of DNNs
* Corresponding authors.
whose effects are difficult to quantify. Several other approaches attempt to extract approximated state-action policies, e.g., decision trees [14, 15], from DNNs via model compression [16] and distillation [17] techniques. However, no equivalence guarantee is established between DNNs and the extracted policies. Consequently, verification results are just probably approximately correct [18].
Inspired by recent advances [19-22] in training near-optimal policies even with reduced training state space imposed by aggregated adjacent states, we propose a novel, inverse transform-then-train approach: encoding a DNN into an equivalent set of easy-to-verify linear control policies and then optimizing them by training the DNN using reinforcement learning. We accompany our inverse approach by devising a novel neural network model called piece-wise linear decision neural networks (PLDNNs), which make linear decisions on each abstract state. Unlike conventional DNNs which build a state-action relation for each actual state, a PLDNN defines a linear relationship, called Linear Control Unit (LCU), between actions and actual states associated with the same abstract state. To this end, a PLDNN is essentially a set of LCUs for all abstract states. In contrast to DNNs, LCUs are more explainable and verifiable without any over-approximation. Moreover, PLDNNs are compatible with most existing DRL training algorithms as both share the same input and output layers for the same control task.
We extensively assess PLDNN, along with the state-of-the-art tools, with respect to both performance (in terms of cumulative rewards and system robustness) and verifiability (in terms of overestimation and time cost for the reachability analysis of trained systems) on a collection of benchmarks, including a 12-dimensional control task. Our experimental results show that, compared to the DNN-based systems, the PLDNN-based systems can be verified more precisely, with significantly less overestimation, and more efficiently, with up to 438 times speedup, while achieving comparable performance. Moreover, compared to the state-of-the-art tools, the complex 12-dimensional control task can be trained and verified with up to 7 times deeper computation steps, along with notable tightness improvement.
Main Contributions. Overall, we provide: (i) a novel inverse approach for boosting the formal verification of DRL systems by learning efficiently and directly (without over-approximation) verifiable piece-wise linear policies with comparable performance; (ii) a novel neural network model to realize the learned piece-wise linear policies, which is compatible with most existing DRL algorithms; and (iii) a prototype called LinCon, along with an extensive assessment which demonstrates its tightness in verification results, outperformance over the state-of-the-art tools (up to 438 times speedup), and scalability (up to a 12-dimensional control task).
2 Problem Formulation and Motivation
A DRL system is driven by a DNN-implemented controller  $\pi$ , which is trained for decision-making, and a physical model defined by the ordinary differential equations (ODEs)  $\dot{s}(t) = f(s(t), a(t))$ , with  $s$  the state variables and  $a$  a control action. In what follows, we omit the time variable  $t$ . Typically, for DRL systems, continuous time is discretized, and we have  $a = \pi(s)$  at state  $s$  and assume  $\dot{a} = 0$  during a small time step e.g.,  $\delta$ . At the time point  $k\delta$ ,  $k \in \mathbb{N}$ , the decision network receives the current state  $s_k$  and outputs an action  $a_k = \pi(s_k)$ . The state variables then evolve according to the physical model during the time interval  $[0, \delta]$ . The reachable state  $s_{k+1}$  at  $\delta$  from  $s_k$  is  $s_{k+1} = s_k + \int_0^\delta f(s, a_k) dt$  which is called the successor state of  $s_k$ . Note that the system evolves continuously from  $s_k$  to  $s_{k+1}$ . The intermediate states can be computed by substituting the time elapses for  $\delta$  in the above formula.
Definition 1 (Path of DRL systems). Given a DRL system  $\mathcal{D}$  with an environment dynamics  $f$ , decision network  $\pi$ , time step size  $\delta$  and initial state set  $S_0$ , a path of  $\mathcal{D}$  is a finite or infinite state sequence:  $[s_0, a_0] \xrightarrow[\delta]{f} [s_1, a_1] \xrightarrow[\delta]{f} [s_2, a_2] \xrightarrow[\delta]{f} [s_3, a_3] \xrightarrow[\delta]{f} \cdots$  such that:
1.  $s_0\in S_0$
2.  $s_{i + 1} = s_i + \int_0^\delta f(s,a_i)dt$  and  $a_{i} = \pi (s_{i})$  for  $i = 0,1,2,\ldots$
A DRL system is essentially a DNN-controlled hybrid system. Definition 2 gives a formal definition of regular hybrid systems. The state space of a hybrid system is the Cartesian product of a set  $L$  of discrete locations and state space of  $n$  real-valued variables  $V_{c}$ . At each location  $l\in L$ , the  $n$  continuous variables evolve continuously according to a dynamical law  $\dot{\nu} = f_l(\nu)$ . When the guard condition on the transition between locations  $(l_1,l_2)\in T$  is triggered, the system moves to  $l_{2}$ , and the continuous variables are reset by  $R$ . There are two steps involved in the state transition from  $(l_i,\nu_i)$  to its successor state  $(l_{i + 1},\nu_{i + 1})$ : first, from  $(l_i,\nu_i)$  to its time successor  $(l_i,\varphi_{f_l_i}(\nu_i,t_i))$ , and then, to
$(l_{i + 1},\nu_{i + 1})$  that is the transition successor of  $(l_i,\varphi_{f_l_i}(\nu_i,t_i))$ , where  $\varphi_{f_l_i}$  is the solution of  $f_{l_i}$  with initial condition  $\nu (0) = \nu_{i}$ , mapping the initial state  $\nu_{i}$  to the state  $\varphi_{f_l_i}(\nu_i,t)$  (i.e., the reachable state at time  $t$  from  $\nu_{i}$ ). Accordingly, the state of a hybrid system can be changed in two ways [23]: (i) by a time delay that changes only the value of continuous variables according to the dynamics of the current location defined in  $F$ ; and (ii) by a discrete and instantaneous transition that changes both location and continuous variables according to the rules in  $T$ .
Definition 2 (Hybrid Automata [9]). A hybrid automaton is an 8-tuple  $H = \langle L, \text{Var}, \text{Inv}, F, T, G, R, I_0 \rangle$ , where:
-  $L$  is a finite set of discrete locations;
- Var is a finite set of  $n$  real-valued variables with state space  $V_{c} \subseteq \mathbb{R}^{n}$ ;
-  $\operatorname{Inv} : L \rightarrow  {2}^{V_c}$  is a function assigning to each location an invariant condition;
-  $F: L \to (V_c \to \mathbb{R}^n)$  is a function associating each location  $l$  to a continuous dynamics  $\dot{\nu} = f_l(\nu)$ ;
-  $T \subseteq L \times L$  is a set of transitions between locations;
-  $G: T \to 2^{V_c}$  is a function assigning each transition  $(l_1, l_2) \in T$  a guard condition  $G(l_1, l_2) \subseteq Inv(l_1)$ ;
-  $R: T \to 2^{V_c}$  is a function assigning each transition  $(l_1, l_2) \in T$  a reset  $R(l_1, l_2) \subseteq \text{Inv}(l_2)$ ;
-  $I_0 \subseteq L \times V_c$  is an initial state set.
Definition 3 (Path of Hybrid Automaton [23]). Let  $H = \langle L, \text{Var}, \text{Inv}, F, T, G, R, I_0 \rangle$  be a hybrid automata. A path of  $H$  is a finite or infinite sequence of location and state value pairs, starting from an initial pair  $(l_0, v_0) \in I_0$ , i.e.,  $(l_0, v_0) \xrightarrow{t_0} (l_1, v_1) \xrightarrow{t_1} (l_2, v_2) \xrightarrow{t_2} (l_3, v_3) \xrightarrow{t_3} \cdots$  such that:
1.  $\forall 0\leq t\leq t_i,\varphi_{f_i}(v_i,t)\in Inv(l_i)$
2.  $(l_{i},l_{i + 1})\in T\land \varphi_{f_{l_i}}(v_i,t_i)\in G(l_i,l_{i + 1})\land v_{i + 1}\in R(l_i,l_{i + 1})$
Theorem 1 (Modeling DRL Systems as Hybrid Automata). A DRL system with an environment dynamics  $f$ , decision network  $\pi$ , time step size  $\delta$ , and initial state set  $S_0$ , state space  $S$ , can be equivalently modeled as the following hybrid automaton:
- Var: state variable  $s$ , action  $a$ , and clock variable  $t_c$
-  $I_0$ :  $\{(l_0, (s \in S_0, a = 0, t_c = \delta))\}$
-  $L \colon  \left\{  {l}_{0}\right\}$  - Inv:  $\operatorname{Inv}\left( {l}_{0}\right)  = \left\{  {s \in  S,{t}_{c} \leq  \delta }\right\}$
-  $F$ :  $F(l_0) = \{\dot{s} = f(s, a), \dot{a} = 0, \dot{t}_c = 1\}$
-  $T\colon \{(l_0,l_0)\}$
-  $R\colon R(l_0,l_0) = \{t_c = 0,a = \pi (s)\}$
Proof. We first analyze the path of the modeled hybrid automaton  $H$ . For an arbitrary initial state  $(l_0, [s_0, 0, \delta])$ ,  $s_0 \in S_0$ , since  $t_c = \delta$ , the only transition with guard condition  $\{t_c = \delta\}$  will be triggered and transition to  $(l_0, [s_0, a_0, 0])$  where  $a_0 = \pi(s_0)$ . Then  $(l_0, [s_0, a_0, 0])$  will move to its time successor  $(l_0, [s_1, a_0, \delta])$  where  $s_1 = s_0 + \int_{0}^{\delta} f(s, a_0) dt$ . Next, the transition with guard condition  $\{t_c = \delta\}$  will be made and conduct the reset operation  $t_c = 0$ ,  $a = \pi(s_1)$  in  $R$  to obtain the transition successor  $(l_0, [s_1, a_1, 0])$  in which  $a_1 = \pi(s_1)$ . Repeating the evolution to the time successor and the transition successor yields the following sequence:
$$
(l _ {0}, [ s _ {0}, 0, \delta ]) \xrightarrow [ f ]{0} (l _ {0}, [ s _ {0}, a _ {0}, 0 ]) \xrightarrow [ f ]{\delta} (l _ {0}, [ s _ {1}, a _ {1}, 0 ]) \xrightarrow [ f ]{\delta} (l _ {0}, [ s _ {2}, a _ {2}, 0 ]) \xrightarrow [ f ]{\delta} \dots
$$
According to Definition 1, there exists the same transition relation  $s_{i+1} = s_i + \int_0^\delta f(s, a_i) dt$  with  $a_i = \pi(s_i)$  in  $\mathcal{D}$  and the modeled hybrid automaton  $H$  at each time point  $i\delta$ . Thus, given an arbitrary initial state  $s_0$ , the value of state variable and action of  $\mathcal{D}$  and  $H$  remain the same at each  $\delta$ . Then with the same state-action pair and dynamics  $f$  at each  $\delta$ ,  $\mathcal{D}$  and  $H$  also have the same state-action value during each time interval  $[i\delta, (i+1)\delta)$ .
Figure 1 depicts the hybrid automaton defined in Theorem 1. There exists only one location  $l_0$ . The invariants in  $Inv$  claim that any state belongs to the state space  $S$  and the clock variable  $t_c$  is less than or equal to the time step size  $\delta$ . The flow in  $F$  defines the dynamics  $f$  of the system. The only transition is triggered when  $t_c = \delta$ , updating the action  $a$  and resetting  $t_c$  as defined in  $R$ . The continuous change happens during the time interval  $[i\delta, (i + 1)\delta]$ , with  $i \in \mathbb{N}$ , and the discrete change of actions occurs at each  $\delta$ .
Unfortunately, the hybrid automaton of a DRL system cannot be verified by using existing hybrid automata model checkers such as Flow* [24], Ariadne [25], and CORA [26]. The reason is that
the action  $a$  in  $R$  depends on the uninterpretable DNN  $\pi$  by  $a \coloneqq \pi(s)$ , and  $\dot{s} = f(s, a)$  can not be expressed in a known closed-form, which however is required by regular hybrid automata supported by these tools [27]. Hence, almost all reachability-based verification methods for DRL systems such as Polar [10], Sherlock [7], and ReachNN [28] inevitably over-approximate  $\pi$  using a Taylor model, at the cost of large overestimation and time overhead.
3 Piece-Wise Linear Control Policies
To bypass the crux of over-approximating DNNs, we devise a novel, alternative neural network model which essentially realizes a set of linear control policies. Our approach bases on the common assumption that there exists a near-optimal linear control policy for every small region of the entire state space [29, 30, 21, 31]. Our objective is then to discretize the state space  $S$  of a DRL system and to train a linear control policy for each discretized region. Specifically, given an  $n$ -dimensional DRL system with  $m$ -dimensional control input, we train a DNN which implements a linear control function  $a_{j} = b^{j} + c_{1}^{j}x_{1} + c_{2}^{j}x_{2} + \dots +c_{n}^{j}x_{n}$  for each control dimension  $1\leq j\leq m$  and each discretized region.
Figure 1: Hybrid automaton for a DRL system.
3.1 Abstracting State Space via Abstract Interpretation
Abstract Interpretation [32] is an effective technique for scaling up formal verification of complex systems or programs by reducing the system space while preserving the soundness of verification results. For instance, an infinite state space  $[-2,0]\times [0,2]$  can be abstracted to be an abstract state represented as  $(-2,0,0,2)$ , when all the states in  $[-2,0]\times [0,2]$  share a same property. In general, given a system state space  $S$ , we denote  $S_{\phi}$  as a finite set of abstract states (each abstract state represents a possibly infinite set of actual system states in  $S$ ). Let  $\phi : S\to S_{\phi}$  be an abstraction function that maps each actual state  $s$  in  $S$  to the corresponding abstract state in  $S_{\phi}$ , and  $\phi^{-1}: S_{\phi}\to 2^{S}$  be the inverse concretization function such that  $\phi^{-1}(s_{\phi}) = \{s|s\in S,\phi (s) = s_{\phi}\}$ .
For state space abstraction, we choose the very primitive but effective abstraction approach which abstracts actual system states as intervals. It is known as interval abstract domain and has been well studied for system [33] and program verification [34] and even the approximation of neural networks [35]. Specifically, let  $L_{i}$  and  $U_{i}$  be the lower and upper bounds for the  $i$ -th dimension value of  $S$ . We first define the abstraction granularity as an  $n$ -dimensional vector  $\gamma = (d_{1}, d_{2}, \dots, d_{n})$ . Then the  $i$ -th dimension will be divided evenly into  $(U_{i} - L_{i}) / d_{i}$  intervals which means each abstract state can be represented as a  $2n$ -dimensional vector  $(l_{1}, u_{1}, \dots, l_{n}, u_{n})$ .
Definition 4 (Interval-Based Abstraction Function). Given an  $n$ -dimensional continuous state space  $S$  and an abstract state space  $S_{\phi}$  which discretizes  $S$  based on abstraction granularity  $\gamma$ ,  $\phi : S \to S_{\phi}$  is called an interval-based abstraction function such that, for every actual state  $s = (x_1, \ldots, x_n) \in S$  and abstract state  $s_{\phi} = (l_1, u_1, \ldots, l_n, u_n) \in S_{\phi}$ , we have  $\phi(s) = s_{\phi}$  if and only if  $l_i \leq x_i < u_i$  holds for each dimension  $1 \leq i \leq n$ .
Example 1 (Running Example). Consider a 2-dimensional system in [36] with state space  $[-2,2) \times [-2,2)$ . The dynamics  $f$  is defined by following ordinary differential equations (ODE) i.e.,  $\dot{x}_1 = x_2 - x_1^3$  and  $\dot{x}_2 = a$ . The sign  $a$  means the control action. The objective is to train a DNN for determining action  $a$  based on  $(x_1, x_2)$  so that the agent can move from the initial region  $x_1 \in [0.7,0.9]$  and  $x_2 \in [0.7,0.9]$  to the goal region  $x_1 \in [-0.3,0.1]$  and  $x_2 \in [-0.35,0.05]$  as soon as possible.
Suppose that the abstraction granularity is  $\gamma = (2, 2)$ . The continuous state space  $[-2, 2) \times [-2, 2)$  is then partitioned into four regions, corresponding to four abstract states represented by  $S_{\phi} = \{(-2, 0, -2, 0), (-2, 0, 0, 2), (0, 2, -2, 0), (0, 2, 0, 2)\}$ , respectively.
3.2 Piece-Wise Linear Decision Neural Networks
We devise an alternative DNN model called piece-wise linear decision neural networks (PLDNNs). Unlike conventional DNNs, a PLDNN contains an abstraction layer between the input layer and the first hidden layer. The abstraction layer is used to convert an actual system state into its corresponding abstract state. Then the output of a PLDNN is the control action that is the dot product result of state variables of the actual state and the linear coefficients determined by the corresponding abstract state.
Algorithm 1: The Training Procedure based on the DDPG algorithm
1 Input: State space  $S$ , abstraction granularity  $\gamma$ . Output: A PLDNN  $\pi$   
2  $\phi \gets$  discretize  $S$  according to  $\gamma$ ; // obtain abstraction function, Sec3.1  
3 Initialize actor network  $\pi$  as a PLDNN by encoding  $\phi$  into the coefficient network  $\pi_c$ ; // Sec3.2  
4 Initialize critic network  $Q$ , target networks  $\pi' \gets \pi, Q' \gets Q$ ;  
5 DDPG( $\pi, Q, \pi', Q'$ ); // train  $\pi$  based on DDPG algorithm  
6 return  $\pi$
Figure 2 exemplifies the architecture of the PLDNN  $\pi$  for a two-dimensional DRL system. The decision-making of  $\pi$  is based on a coefficient network  $\pi_{c}$  that outputs the linear coefficients. The second layer of  $\pi_{c}$  is the inserted abstraction layer which consists of the blue neurons and the red neurons.
The output layer of  $\pi_c$  contains  $n + 1$  neurons that output the  $n + 1$  linear coefficients depicted as purple neurons. As for the weights setting between the input layer and the abstraction layer, the weights of the connections between the  $i$ -th neuron in the input layer and the  $(2i - 1)$ -th and  $2i$ -th neurons in the abstraction layer are set to 1 which are represented by blue lines and red lines, respectively. While the weights of other connections denoted by the black dashed lines are set to 0. Under this setting of weights, the inputs to both  $(2i - 1)$ -th and  $2i$ -th neurons in the abstraction layer are  $x_i$ . Moreover, the activation function of the  $(2i - 1)$ -th neuron in
Figure 2: The arch. of the PLDNN for Example 1.
the abstraction layer is set to  $\phi_u$  with the responsibility of computing the upper bound  $u_i$ , and that of the  $2i$ -th neuron is set to  $\phi_l$  for calculating the lower bound  $l_i$ . Specifically, for a continuous state space partitioned by the abstraction granularity  $\gamma = (d_1, \dots, d_n)$ , the activation functions for the  $(2i - 1)$ -th and  $2i$ -th neurons in the abstraction layer can be formulated as follows:
$$
\phi_ {l} ^ {i} \left(x _ {i}\right) = L _ {i} + \left\lfloor \frac {\left(x _ {i} - L _ {i}\right)}{d _ {i}} \right\rfloor d _ {i} \quad \phi_ {u} ^ {i} \left(x _ {i}\right) = L _ {i} + \left\lfloor \frac {\left(x _ {i} - L _ {i}\right)}{d _ {i}} \right\rfloor d _ {i} + d _ {i}
$$
With the above activation functions, the abstraction layer can output the same abstract state  $s_{\phi} = (l_1, u_1, \ldots, l_n, u_n)$  for  $\forall s \in \phi^{-1}(s_{\phi})$ . The abstract state  $s_{\phi}$  is then propagated to the fully connected layers of  $\pi_c$  to generate the linear control coefficients  $(b, c_1, c_2)$  denoted by  $\pi_c(s)$ .
To obtain the final output of action  $a$ , an additional dot product operation between  $\pi_c(s)$  and  $[1, s]$  is performed with the result of the operation as the control action  $a$  where  $[\cdot, \cdot]$  is the concatenation operation. For multiple dimensional control action  $a = (a_1, \dots, a_m)$ , we only need to modify the output dimension of  $\pi_c$  to  $m(n + 1)$ , such that each  $n + 1$  neurons output the linear coefficients of one dimension of  $a$ . More specifically, we can obtain  $a_j$  as follows:
$$
a _ {j} = \pi (s) _ {j} = \pi_ {c} (s) [ (n + 1) (j - 1): (n + 1) j ] \cdot [ 1, s ], 1 \leq j \leq m.
$$
where vector[ start : end] denotes the slicing operation that extracts the elements of vertor from index start up to but not including index end.
With the additional abstraction layer that can output an identical vector into the fully connected layers of  $\pi_c$  for  $\forall s\in \phi^{-1}(s_{\phi})$ , we can ensure that  $\pi_c$  always produces the same coefficients for all actual states located in the same abstract state. Consequently, we can extract a piecewise linear decision function with this structure of  $\pi$  on each abstract state.
3.3 The Training Procedure
Training a PLDNN can be achieved by extending existing deterministic policy gradient algorithms such as Deep Deterministic Policy Gradient (DDPG) [37] and Twin Delayed Deep Deterministic Policy Gradient [38] since the modifications made stay inside neural networks and are invisible to the DRL algorithms. The pseudo code of the training procedure is given in Algorithm 1, where we take the DDPG algorithm as an example. The training procedure starts with defining the abstraction
function  $\phi$  according to  $\gamma$  (Line 2), initializing PLDNN with an abstraction layer based on  $\phi$  (Line 3), and, following [37], initializing the critic network and the two target networks (Line 4). The procedure then invokes the DDPG algorithm with the networks as arguments (Line 5) since the PLDNN has the same input and output as the actor network implemented by DNN. During this procedure, we freeze the parameters between the input and the abstraction layers of  $\pi$ . The parameters in the fully connected layers are trained based on backpropagation [39] and gradient descent optimization [40].
4 Equivalent Policy Extraction and Verification