SlowGuess/NeurIPS_2023_OCR huggingface dataset:
christianms@cs.aau.dk
Emir Demirovic
e.demirovic@tudelft.nl
Anna Lukina
a.lukina@tudelft.nl
Kim Guldstrand Larsen
kgl@cs.aau.dk
Decision trees have gained popularity as interpretable surrogate models for learning-based control policies. However, providing safety guarantees for systems controlled by decision trees is an open challenge. We show that the problem is undecidable even for systems with the simplest dynamics, and PSPACE-complete for finite-horizon properties. The latter can be verified for discrete-time systems via bounded model checking. However, for continuous-time systems, such an approach requires discretization, thereby weakening the guarantees for the original system. This paper presents the first algorithm to directly verify decision-tree controlled systems in continuous time. The key aspect of our method is exploiting the decision-tree structure to propagate a set-based approximation through the decision nodes. We demonstrate the effectiveness of our approach by verifying safety of several decision trees distilled to imitate neural-network policies for nonlinear systems.
Deep reinforcement learning has shown success in deriving control policies for nonlinear systems for which classical optimal control theory provides no solution [Mnih et al., 2015]. Despite impressive performance, there are two key drawbacks: (i) the resulting policy is difficult to interpret, and (ii) there are no guarantees that the system always reaches the desired goal and avoids unsafe states. The interpretability challenge has led to a review of decision trees as well-performing, compact, and transparent surrogates for deep neural networks [Ashok et al., 2020, 2019, David et al., 2015, Alamdari et al., 2020, Vos and Verwer, 2023]. To provide safety guarantees, prior work proposed to encode a discrete, time-bounded system controlled by a decision-tree policy as a set of logical constraints [Bastani et al., 2018]. However, this approach does not apply to continuous-time dynamics.
Example 1. To illustrate the problem, we consider a quadrotor system [Ivanov et al., 2019] in Figure 1(a), where the task is to follow the brown reference trajectory from  $(0,0)$  and stay within the blue dashed safety corridor. Illustrated schematically in Figure 1(b), we may observe a trajectory that exhibits safe behavior for each discrete time point but leaves the safety corridor in between. Since the trajectory returns quickly enough, this violation is missed by a discrete-time analysis.
In this work, we address, for the first time, the problem of verifying safety (reach-avoid) properties of a decision-tree controlled system (DTCS) with (nonlinear) continuous-time dynamics. We propose an algorithm that exploits the structure of the decision-tree policy and propagates sets of reachable states,
(a) Sketch of the quadrrotor system  $(x / y$  plane) with sample trajectories.
Figure 1: A sketch of the quadrotor system and motivation for continuous-time analysis.
(b) Discrete-time analysis may miss property violations of continuous systems. The threshold (dashed blue line) is breached for a short time (red curve), which is not detected at the discrete time points (black dots).
splitting at the decision boundaries. This way we obtain a sound enclosure of the true reachable states even for continuous dynamics, which guarantees safe goal reachability for all behaviors of the controlled system. Furthermore, our approach naturally generalizes to discrete-time systems, for which, compared to the aforementioned approach based on logical encoding [Bastani et al., 2018], our approach provides visual interpretability of the verification process.
To summarize our contributions, we first provide an algorithm to verify a decision-tree policy of a nonlinear continuous-time system. Our general algorithm is parameterized in two procedures for respectively analyzing the dynamical system and the policy. For these procedures, we formulate sufficient conditions for the algorithm to be sound and relatively complete. Second, we describe an instantiation of the algorithm for nonlinear systems based on Taylor models [Berz and Makino, 1998]. In a nutshell, this procedure propagates a set of states through the system dynamics. For analyzing the decision-tree policy, we focus on the common class of axis-aligned predicates (" $x \leq c$ ) and propose an instantiation that exploits this structure. Third, we show that the problem of verifying decision-tree policies is PSPACE-complete even for very simple (namely, state-independent) dynamics. Finally, we demonstrate that our algorithm can verify several reinforcement-learning benchmarks from classical control and the quadrotor from Figure 1, even for unbounded time.
A decision tree is a popular machine-learning model that has recently regained interest due to its high interpretability [Du et al., 2020]. Decision-tree policies can be trained directly from a tabular dataset of state-action pairs [Quinlan, 1996], e.g., using the CART algorithm [Breiman et al., 1984] or dtControl [Ashok et al., 2020]. Since these algorithms perform an equivalence transformation, safety guarantees transfer from the dataset to the decision tree. Ashok et al. [2019] show how to extract a safe-by-design decision tree from a safe policy synthesized with Uppaal Stratego [David et al., 2015] for a priced timed game. However, for nonlinear systems it is generally unclear how to obtain safe-by-design policies. In practice one uses best-effort methods such as reinforcement learning. As these approaches cannot guarantee that the resulting policy is safe, there is need for approaches to verify a given policy after learning.
Decision trees can be trained to imitate another model such as a neural network. For instance, the Viper algorithm incorporates decision-tree learning into the imitation procedure [Bastani et al., 2018]. As this imitation provides no correctness guarantees, a separate analysis is required. The authors encode the bounded-time reachability problem for a discrete-time system in logical constraints. This encoding is the only verification approach for discrete-time DTCS we are aware of. Other verification efforts for decision trees have focused on robustness and adversarial examples in a supervised setting [Urban and Miné, 2021], which are orthogonal problems to our setting of verifying a policy.
For purely dynamical systems (without a control policy), reachability analysis has been studied extensively [Doyen et al., 2018, Althoff et al., 2021]. One prominent approach is based on Taylor models [Berz and Makino, 1998], which we also employ in our implementation. Combining a
dynamical system with a control policy results in a hybrid system [Henzinger, 1996]. Applying tools for general hybrid-system verification to DTCS is not feasible because they do not exploit the specific structure (we discuss this further in Appendix D). The reach-avoid problem is undecidable for nonlinear dynamics (even without a control policy), and most approaches (including ours) consider only time-bounded analysis. However, as we discuss later, under certain conditions, our results still generalize to unbounded time via fixpoint techniques [Giacobbe, 2019, Bacci et al., 2021]. For piecewise-constant dynamics, axis-aligned policies resemble systems that can be efficiently dealt with using interval-based arithmetic constraint solving [Franzle et al., 2007, Franzle and Herde, 2007].
Closely related to DTCS are neural-network controlled systems (NNCS), where the policy is implemented by a neural network. Recently, many reachability approaches have been proposed [Dutta et al., 2019, Tran et al., 2020, Fan et al., 2020, Akintunde et al., 2022, Ivanov et al., 2021, Schilling et al., 2022, Kochdumper et al., 2023], and implementations compete in a yearly competition [Lopez et al., 2022]. The main difference to DTCS is that the control action comes from a continuous domain and the neural network implements a smooth function. Hence such policies typically yield a similar control action for similar states, which benefits set-based reachability analysis. In contrast, given two similar states, a decision tree can yield vastly different control actions. Thus tools for verifying NNCS are not applicable to our problem. NNCS are Turing-complete [Hyötniemi, 1996]. We show the same result for DTCS even with the simplest environment dynamics, which yields an undecidable problem in unbounded time and a PSPACE-complete problem in bounded time.
To summarize, decision-tree policies are highly desirable due to their interpretability. The most successful methods in obtaining such policies use machine learning, which do not guarantee safety of the resulting policy. While previous work managed to verify discrete-time policies, continuous-time policy verification remains an open problem. Our work bridges this gap with the first algorithm to verify decision-tree policies for continuous-time systems.
Outline. We organize the remainder of the paper as follows. First we formalize DTCS and the reach-avoid problem. Then we describe our verification algorithm and discuss the problem complexity. Next we report on experimental results. Finally we conclude and discuss directions for future work.
2.1 Decision-Tree Controlled Systems
Let  $S \subseteq \mathbb{R}^n$  be an  $n$ -dimensional state space and  $\mathcal{U} \subseteq \mathbb{R}^m$  an  $m$ -dimensional action (or input) space. A decision tree  $\mathcal{T}$  over  $S$  and  $\mathcal{U}$  is a binary tree such that each inner node is labeled with a predicate  $p: S \to \mathbb{B}$  (with  $\mathbb{B} = \{\top, \bot\}$ ) and each leaf is labeled with an action  $u \in \mathcal{U}$ . The nodes in the tree are organized in levels, with the root node being at level 1. Let  $root(\mathcal{T})$  denote the root node of  $\mathcal{T}$ , left( $\mathcal{T}$ ) resp. right( $\mathcal{T}$ ) denote the left resp. right sub-tree of  $\mathcal{T}$ , and  $\ell(\mathcal{T})$  denote the label at the root of  $\mathcal{T}$  (i.e., if  $\mathcal{T}$  is a leaf,  $\ell(\mathcal{T})$  is an action and otherwise a predicate). We can interpret  $\mathcal{T}$  as a function from  $\mathcal{S}$  to  $\mathcal{U}$  as follows. Given a state  $x \in S$ , the image under  $\mathcal{T}$ , written  $act(x, \mathcal{T}) \in \mathcal{U}$ , is defined recursively. For a leaf  $\mathcal{T}$ ,  $act(x, \mathcal{T})$  is just  $\ell(\mathcal{T})$ . For a proper tree with root predicate  $p$ ,  $act(x, \mathcal{T})$  is  $act(x, left(\mathcal{T}))$  if  $p(x) = \top$ , and  $act(x, right(\mathcal{T}))$  otherwise. In this paper, we restrict our analysis to predicates of the form  $x_i \leq c$  where  $x_i$  is the  $i$ -th state and  $c \in \mathbb{R}$  is a constant. Geometrically, these predicates are axis-aligned half-spaces. This class of predicates is commonly used, e.g., in the tools Uppaal Stratego [David et al., 2015] (with industrial applications in control of smart homes [Larsen et al., 2016] and traffic lights [Eriksen et al., 2017]) and dtControl [Ashok et al., 2020].
We consider environments modeled by a system of ordinary differential equations (ODEs),  $\dot{x} = f(x,u)$ , where  $x(t) \in S$  is the state vector and  $u(t) \in \mathcal{U}$  is the vector of control actions. Given an initial state  $x(0) = x_0$  and an action  $u_0$ , we assume that the solution to the corresponding initial-value problem at time  $t \geq 0$ , written  $\xi(x_0, u_0, t, f)$ , exists and is unique (e.g., by Lipschitz continuity).
A decision-tree controlled system (DTCS) is a triple  $(f, \mathcal{T}, \tau)$  where  $f$  describes a system of ODEs,  $\mathcal{T}$  is a decision tree, used as policy, and  $\tau \in \mathbb{R}^{+}$  is a control period. Figure 2 shows a conceptual sketch. The DTCS periodically queries the policy for a new control action. At time points  $k\tau$ ,  $k \in \mathbb{N}$ , the current state  $x(k\tau)$  is passed to the policy  $\mathcal{T}$ , which instantaneously yields the new control action  $u_{k}$ ; this action is then used for the next control period  $\tau$  in the system dynamics  $f$ . Formally, given an initial state  $x_{0}$  at  $t = 0$ , we recursively define the sequence of actions  $u_{k} = act(x(k\tau), \mathcal{T})$ ,  $k \in \mathbb{N}$ , and the evolution of the state  $x(t) = \xi(x(k\tau), u_{k}, t - k\tau, f)$  (i.e., a trajectory) for  $t \in (k\tau, (k + 1)\tau]$ .
Figure 2: A decision-tree controlled system  $(f, \mathcal{T}, \tau)$ .
2.2 Reachable States and Reach-Avoid Problem
Given a DTCS and a set of initial states  $\mathcal{X}_0\subseteq S$ , we are interested in the reachable states, either at time  $t$  as  $\mathcal{R}_t = \{x(t)\mid x(0)\in \mathcal{X}_0\}$  or the generalization to time intervals  $\mathcal{R}_{[T_0,T_1]} = \bigcup_{t\in [T_0,T_1]}\mathcal{R}_t$ .
Similarly, a discrete-time DTCS is a pair  $(f,\mathcal{T})$  where  $f$  describes a recurrence  $x_{k + 1} = x_k + f(x_k,u_k)$  and  $\mathcal{T}$  is a decision tree. The reachable states at step  $k$  are  $\mathcal{R}_k = \{x_k\mid x_0\in \mathcal{X}_0\}$  and analogously  $\mathcal{R}_{[K_0,K_1]} = \bigcup_{k\in [K_0,K_1]}\mathcal{R}_k$ . By default, DTCS refers to the continuous-time DTCS introduced above.
The reach-avoid problem for DTCS is, given a DTCS  $(f,\mathcal{T},\tau)$  over  $\mathcal{S}$ , a set of initial states  $\mathcal{X}_0\subseteq \mathcal{S}$ , a bounded number of control cycles  $k_{\mathrm{max}}$ , and two sets  $\mathcal{G},\mathcal{E}\subseteq \mathcal{S}$ , to decide whether all trajectories  $x(t)$  reach the goal set  $\mathcal{G}$  without reaching the error set  $\mathcal{E}$  before time  $T_{\mathrm{max}} = k_{\mathrm{max}}\tau$ , i.e.,  $\exists t^{*}\leq T_{\mathrm{max}}$ .  $x(0)\in \mathcal{X}_0\land x(t^*)\in \mathcal{G}\land \forall t\in [0,t^* ].x(t)\notin \mathcal{E}$ . If we assume that the goal states  $\mathcal{G}$  are absorbing, this is equivalent to checking
$$
\mathcal {R} _ {T _ {\max }} \subseteq \mathcal {G} \wedge \mathcal {R} _ {[ 0, T _ {\max } ]} \cap \mathcal {E} = \emptyset . \tag {1}
$$
Example 2. Consider again the quadrotor model from Figure 1. A full model description is given in Appendix B.1. The reach-avoid problem here consists of the shaded area at the top as the goal set  $\mathcal{G}$ , the set of states outside the red corridor as the error set  $\mathcal{E}$ , and  $k_{\max} = 30$  control cycles.
Determining reachability is already undecidable for uncontrolled nonlinear dynamical systems [Hainry, 2008], and hence also for DTCS with nonlinear dynamics. A DTCS can be seen as a hybrid (mixed discrete-continuous) system, for which reachability is undecidable even with linear dynamics [Doyen et al., 2018]. Due to these complexity barriers, we aim at enclosing, or overapproximating, the reachable states up to time horizon  $T_{\mathrm{max}}$  by computing a set  $\overline{\mathcal{R}} \supseteq \mathcal{R}_{[0,T_{\mathrm{max}}]}$ .
Algorithm 1 Reachability algorithm for DTCS
Input: DTCS  $(f, \mathcal{T}, \tau)$ , initial set  $\mathcal{X}_0$ , time/cycle bounds  $T_{\mathrm{max}} = k_{\mathrm{max}} \cdot \tau$
Output: set of states  $\overline{\mathcal{R}}\supseteq \mathcal{R}_{[0,T_{\max}]}$
$$
\operatorname {p o s t} _ {f} \left(\mathcal {X} _ {u}, u, f, \left[ t _ {0}, t _ {1} \right]\right)
$$
1:  $\overline{\mathcal{R}}\gets \mathcal{X}_0$
2:  $Q \gets \{(\mathcal{X}_0, 0)\}$
3: while  $\rightharpoonup$  isempty  $(Q)$  do
4:  $(\mathcal{X},t_0)\gets \mathrm{pop}(Q)$
5: if  $t_0 \geq T_{\max}$  then
6: continue
7: end if
8:  $t_1 = \min (t_0 + \tau ,T_{\max})$
9: for  $(\mathcal{X}_u,u)\in post_{\mathcal{T}}(\mathcal{X},\mathcal{T})$  do
10:  $(\mathcal{V},\mathcal{Z}_{t_1})\gets$
11:  $\overline{\mathcal{R}}\gets \overline{\mathcal{R}}\cup \mathcal{Y}$
12: push  $(Q,(\mathcal{Z}_{t_1},t_1))$
13: end for
14: end while
15: return  $\overline{\mathcal{R}}$
In this section, we present our approach to reachability analysis for DTCS. We first describe a general high-level algorithm, which resembles standard reachability schemes for hybrid systems. It is, however, tailored to policies over a finite action space  $\mathcal{U}$ . As such, it is applicable to policies beyond decision trees, but for instance not to typical neural-network policies. We then outline conditions under which the algorithm is sound and relatively complete (i.e., does not introduce additional approximation errors). Finally, we instantiate the algorithm specifically for decision-tree policies, which is a novel contribution of this paper.
Algorithm 1 shows our high-level reachability method, which can be used to solve the reach-avoid problem (Section 2.2). The algorithm is parametric in two procedures,  $post_{\mathcal{T}}$  and  $post_f$ , which together compute an enclosure of the reachable states for one control cycle. The queue  $Q$  contains pairs  $(\mathcal{X}, t)$ , where  $\mathcal{X}$  is a set of states that needs to be explored, and  $t$  is a time point. Each iteration of the while loop (line 3) analyzes one control cycle from  $\mathcal{X}$  and  $t$  (unless the time horizon  $T_{\max}$  is reached, line 6). The result is added to the set  $\overline{\mathcal{R}}$  of reachable states in line 11. Next we describe the requirements for the post procedures.
The procedure  $post_{\mathcal{T}}$ , which is a new contribution of this paper, takes a set of states  $\mathcal{X}$  and a decision tree  $\mathcal{T}$  and returns a finite set of pairs  $(\mathcal{Y}_i^u, u)$ , where  $\mathcal{Y}_i^u$  is a set of states and  $u \in \mathcal{U}$  is an action. Since multiple leaves of a decision tree can be associated with the same action  $u$ , we allow multiple sets  $\mathcal{Y}_i^u$  to be associated with  $u$  as well. For each action  $u$ , the union of the sets  $\mathcal{Y}_i^u$  should enclose the set  $act_u(\mathcal{X}, \mathcal{T}) = \{x \in \mathcal{X} \mid act(x, \mathcal{T}) = u\}$ , so we have an index set  $I_u$  (possibly empty) for each action  $u$  such that
$$
\operatorname {a c t} _ {u} (\mathcal {X}, \mathcal {T}) \subseteq \bigcup_ {i \in I _ {u}} \mathcal {Y} _ {i} ^ {u}. \tag {2}
$$
Procedure  $post_{\mathcal{T}}$  wraps the sets  $\mathcal{Y}_i^u$  together with action  $u$ .
$$
\operatorname {p o s t} _ {\mathcal {T}} (\mathcal {X}, \mathcal {T}) = \bigcup_ {u \in \mathcal {U}} \bigcup_ {i \in I _ {u}} \left\{\left(\mathcal {Y} _ {i} ^ {u}, u\right) \right\} \tag {3}
$$
For  $post_{\mathcal{T}}$  to be sound, we connect equations (2) and (3) and untangle the pairing with  $u$  as follows:
$$
\forall u \in \mathcal {U}. \bigcup \left\{x \in \mathcal {Y} _ {i} ^ {u} \mid \left(\mathcal {Y} _ {i} ^ {u}, u\right) \in p o s t _ {\mathcal {T}} (\mathcal {X}, \mathcal {T}) \right\} \supseteq a c t _ {u} (\mathcal {X}, \mathcal {T}) \tag {4}
$$
The procedure  $post_{f}$  receives a set of states  $\mathcal{X}$ , a control action  $u$ , the environment  $f$ , and a time interval  $[t_0, t_1]$ . (Note that we actually only need the duration  $t_1 - t_0$ , which is  $\tau$  most of the time, because we consider time-invariant systems. We only pass the time interval for notational purposes to refer to  $t_1$ .) The goal is to perform a classical time-bounded reachability computation and return two sets  $\mathcal{Y}$  and  $\mathcal{Z}$  such that  $\mathcal{Y}$  encloses the reachable states for the given time interval and  $\mathcal{Z}$  encloses the reachable states at the final time point  $t_1$ . Thus for  $post_{f}$  to be sound, the following must hold:
$$
\operatorname {p o s t} _ {f} (\mathcal {X}, u, f, \left[ t _ {0}, t _ {1} \right]) = (\mathcal {Y}, \mathcal {Z}) \Rightarrow \mathcal {Y} \supseteq \mathcal {R} _ {\left[ t _ {0}, t _ {1} \right]} \wedge \mathcal {Z} \supseteq \mathcal {R} _ {t _ {1}} \tag {5}
$$
Algorithm 1 in combination with equations (4) and (5) computes a sound enclosure. Let us write  $post_{\mathcal{T}}^{*}$  and  $post_{f}^{*}$  for the procedures such that the inclusions in equations (4) and (5) are satisfied with equality. With such procedures, the algorithm even produces the exact result (proven in Appendix C.1).
Theorem 1 (Termination, soundness, relative completeness). Assume Eq. (4) and Eq. (5) are satisfied. (1) Algorithm 1 terminates if all calls to  $\text{post}_{\mathcal{T}}$  and  $\text{post}_f$  terminate. (2) Let  $\supseteq \in \{\supseteq, =\}$ . The result  $\overline{\mathcal{R}}$  of Algorithm 1 encloses  $(\supseteq)$  resp. equals  $(=)$  the reachable states,  $\overline{\mathcal{R}} \supseteq \mathcal{R}_{[0,T_{\max}]}$ , if in all steps  $\text{post}_{\mathcal{T}}(\mathcal{X},\mathcal{T}) \supseteq \text{post}_{\mathcal{T}}^*(\mathcal{X},\mathcal{T})$  and  $\text{post}_f(\mathcal{X}_u,u,f,[t_0,t_1]) \supseteq \text{post}_f^*(\mathcal{X}_u,u,f,[t_0,t_1])$  hold.
3.1 Implementing the Post Procedures
Algorithm 2 Post for the environment
Input: set X, control action u, environ-
ment f, interval [t0,t1]
Output: pair (Y,Z) such that Y ≥
R[t0,t1] and Z ≥ Rt1
1: P(t) ← TM_reach(X,u,f,[t0,t1])
2: Y ← evaluate(P(t),[t0,t1])
3: Z ← evaluate(P(t),t1)
4: return (Y,Z)
For implementing  $post_{f}$ , we use an algorithm based on Taylor models [Berz and Makino, 1998] for reachability analysis of nonlinear dynamical systems. A Taylor model approximates a function as a polynomial together with an interval remainder over a domain [Makino and Berz, 2003], which we interpret as sets of states. For example, the one-dimensional Taylor model with polynomial  $p(x) = x^{2} - x + 1$  and remainder  $[-0.5, 0.5]$  over the domain  $[-1, 1]$  around expansion point 0 encodes the set  $\{p(x) + r \mid x \in [-1, 1], r \in [-0.5, 0.5]\}$ . Taylor models subsume common set representations. Thus we assume that the initial set  $\mathcal{X}_0$  is given as a Taylor model.
Algorithm 2 summarizes the implementation of  $post_{f}$ . Here TM_reach( $\mathcal{X}, u, f, [t_{0}, t_{1}]$ ) (line 1) computes a special Taylor model  $P(t)$  that depends on time  $t$ . To obtain  $\mathcal{Y}$  and  $\mathcal{Z}$ , we evaluate this Taylor model with different values for  $t$ : For  $\mathcal{Y}$  we evaluate with the time interval  $[t_{0}, t_{1}]$ , which results in an enclosure of  $\mathcal{R}_{[t_{0}, t_{1}]}$ . For  $\mathcal{Z}$  we evaluate with the time point  $t_{1}$ , which results in an enclosure of  $\mathcal{R}_{t_{1}}$ . Algorithm 2 can be adapted for discrete-time systems, where the procedure TM_reach is replaced appropriately, line 2 is skipped, and the result is  $(\mathcal{Z}, \mathcal{Z})$ .
Proposition 1. Algorithm 2 implements procedure  $\text{post}_f$  satisfying Eq. (5).
Algorithm 3 instantiates  $post_{\mathcal{T}}$ . Recall that the goal of this procedure is to compute enclosures of  $act_u(\mathcal{X},\mathcal{T})$ , which are the sets of states that result in an action  $u$ . Given a set  $\mathcal{X}$  and a decision tree  $\mathcal{T}$ , the idea is to propagate  $\mathcal{X}$  down the branches that the states  $x \in \mathcal{X}$  would take. While the algorithm is
Figure 3: Example execution of Algorithm 1 for the first control cycle.
agnostic of the predicates in  $\mathcal{T}$  (interpreted as sets of states) on a conceptual level, it is generally difficult to implement the algorithm for arbitrary predicates. In our implementation, we exploit the structure of axis-aligned half-space predicates  $P$ . Let  $\mathrm{box}(\mathcal{X})$  be the interval enclosure of  $\mathcal{X}$ , which is easy to obtain for a Taylor model. In line 9, we check if all states in  $\mathcal{X}$  take the left branch of  $\mathcal{T}$ . We have  $\mathcal{X} \subseteq P$  if and only if  $\mathrm{box}(\mathcal{X}) \subseteq P$ . In line 11 we check if all states in  $\mathcal{X}$  take the right branch of  $\mathcal{T}$ .