Title: Safety Verification of Decision-Tree Policies in Continuous Time
Venue: NeurIPS 2023
Accepted: yes
Keywords: safety verification

decision tree

reinforcement learning

controller

continuous time
Abstract: Decision trees have gained popularity as interpretable surrogate models for learning-based control policies. However, providing safety guarantees for systems controlled by decision trees is an open challenge. We show that the problem is undecidable even for systems with the simplest dynamics, and PSPACE-complete for finite-horizon properties. The latter can be verified for discrete-time systems via bounded model checking. However, for continuous-time systems, such an approach requires discretization, thereby weakening the guarantees for the original system. This paper presents the first algorithm to directly verify decision-tree controlled system in continuous time. The key aspect of our method is exploiting the decision-tree structure to propagate a set-based approximation through the decision nodes. We demonstrate the effectiveness of our approach by verifying safety of several decision trees distilled to imitate neural-network policies for nonlinear systems.

Safety Verification of Decision-Tree Policies in Continuous Time

Abstract

Decision trees have gained popularity as interpretable surrogate models for learningbased control policies. However, providing safety guarantees for systems controlled by decision trees is an open challenge. We show that the problem is undecidable even for systems with the simplest dynamics, and PSPACE -complete for finitehorizon properties. The latter can be verified for discrete-time systems via bounded model checking. However, for continuous-time systems, such an approach requires discretization, thereby weakening the guarantees for the original system. This paper presents the first algorithm to directly verify decision-tree controlled systems in continuous time. The key aspect of our method is exploiting the decisiontree structure to propagate a set-based approximation through the decision nodes. We demonstrate the effectiveness of our approach by verifying safety of several decision trees distilled to imitate neural-network policies for nonlinear systems.

1 Introduction

Deep reinforcement learning has shown success in deriving control policies for nonlinear systems for which classical optimal control theory provides no solution [Mnih et al., 2015]. Despite impressive performance, there are two key drawbacks: (i) the resulting policy is difficult to interpret, and (ii) there are no guarantees that the system always reaches the desired goal and avoids unsafe states. The interpretability challenge has led to a review of decision trees as well-performing, compact, and transparent surrogates for deep neural networks [Ashok et al., 2020, 2019, David et al., 2015, Alamdari et al., 2020, Vos and Verwer, 2023]. To provide safety guarantees, prior work proposed to encode a discrete, time-bounded system controlled by a decision-tree policy as a set of logical constraints [Bastani et al., 2018]. However, this approach does not apply to continuous-time dynamics.

Example 1. To illustrate the problem, we consider a quadrotor system [Ivanov et al., 2019] in Figure 1(a), where the task is to follow the brown reference trajectory from (0 , 0) and stay within the blue dashed safety corridor. Illustrated schematically in Figure 1(b), we may observe a trajectory that exhibits safe behavior for each discrete time point but leaves the safety corridor in between. Since the trajectory returns quickly enough, this violation is missed by a discrete-time analysis.

In this work, we address, for the first time, the problem of verifying safety (reach-avoid) properties of a decision-tree controlled system (DTCS) with (nonlinear) continous-time dynamics. We propose an algorithm that exploits the structure of the decision-tree policy and propagates sets of reachable states,

0.3

0.0

Py

0.3

initial

0.6

goal

reference

-boundary

0.0

0.2

0.4

0.6

0.8

1.0

1.2

(a) Sketch of the quadrotor system ( x / y plane) with sample trajectories.

0

1

2

3

time

(b) Discrete-time analysis may miss property violations of continuous systems. The threshold (dashed blue line) is breached for a short time (red curve), which is not detected at the discrete time points (black dots).

splitting at the decision boundaries. This way we obtain a sound enclosure of the true reachable states even for continuous dynamics, which guarantees safe goal reachability for all behaviors of the controlled system. Furthermore, our approach naturally generalizes to discrete-time systems, for which, compared to the aforementioned approach based on logical encoding [Bastani et al., 2018], our approach provides visual interpretability of the verification process.

To summarize our contributions, we first provide an algorithm to verify a decision-tree policy of a nonlinear continuous-time system. Our general algorithm is parameterized in two procedures for respectively analyzing the dynamical system and the policy. For these procedures, we formulate sufficient conditions for the algorithm to be sound and relatively complete. Second, we describe an instantiation of the algorithm for nonlinear systems based on Taylor models [Berz and Makino, 1998]. In a nutshell, this procedure propagates a set of states through the system dynamics. For analyzing the decision-tree policy, we focus on the common class of axis-aligned predicates (' x ≤ c ') and propose an instantiation that exploits this structure. Third, we show that the problem of verifying decision-tree policies is PSPACE -complete even for very simple (namely, state-independent) dynamics. Finally, we demonstrate that our algorithm can verify several reinforcement-learning benchmarks from classical control and the quadrotor from Figure 1, even for unbounded time.

1.1 Related Work

A decision tree is a popular machine-learning model that has recently regained interest due to its high interpretability [Du et al., 2020]. Decision-tree policies can be trained directly from a tabular dataset of state-action pairs [Quinlan, 1996], e.g., using the CART algorithm [Breiman et al., 1984] or dtControl [Ashok et al., 2020]. Since these algorithms perform an equivalence transformation, safety guarantees transfer from the dataset to the decision tree. Ashok et al. [2019] show how to extract a safe-by-design decision tree from a safe policy synthesized with Uppaal Stratego [David et al., 2015] for a priced timed game. However, for nonlinear systems it is generally unclear how to obtain safe-by-design policies. In practice one uses best-effort methods such as reinforcement learning. As these approaches cannot guarantee that the resulting policy is safe, there is need for approaches to verify a given policy after learning.

Decision trees can be trained to imitate another model such as a neural network. For instance, the Viper algorithm incorporates decision-tree learning into the imitation procedure [Bastani et al., 2018]. As this imitation provides no correctness guarantees, a separate analysis is required. The authors encode the bounded-time reachability problem for a discrete-time system in logical constraints. This encoding is the only verification approach for discrete-time DTCS we are aware of. Other verification efforts for decision trees have focused on robustness and adversarial examples in a supervised setting [Urban and Miné, 2021], which are orthogonal problems to our setting of verifying a policy.

For purely dynamical systems (without a control policy), reachability analysis has been studied extensively [Doyen et al., 2018, Althoff et al., 2021]. One prominent approach is based on Taylor models [Berz and Makino, 1998], which we also employ in our implementation. Combining a

dynamical system with a control policy results in a hybrid system [Henzinger, 1996]. Applying tools for general hybrid-system verification to DTCS is not feasible because they do not exploit the specific structure (we discuss this further in Appendix D). The reach-avoid problem is undecidable for nonlinear dynamics (even without a control policy), and most approaches (including ours) consider only time-bounded analysis. However, as we discuss later, under certain conditions, our results still generalize to unbounded time via fixpoint techniques [Giacobbe, 2019, Bacci et al., 2021]. For piecewise-constant dynamics, axis-aligned policies resemble systems that can be efficiently dealt with using interval-based arithmetic constraint solving [Fränzle et al., 2007, Fränzle and Herde, 2007].

Closely related to DTCS are neural-network controlled systems (NNCS), where the policy is implemented by a neural network. Recently, many reachability approaches have been proposed [Dutta et al., 2019, Tran et al., 2020, Fan et al., 2020, Akintunde et al., 2022, Ivanov et al., 2021, Schilling et al., 2022, Kochdumper et al., 2023], and implementations compete in a yearly competition [Lopez et al., 2022]. The main difference to DTCS is that the control action comes from a continuous domain and the neural network implements a smooth function. Hence such policies typically yield a similar control action for similar states, which benefits set-based reachability analysis. In contrast, given two similar states, a decision tree can yield vastly different control actions. Thus tools for verifying NNCS are not applicable to our problem. NNCS are Turing-complete [Hyötyniemi, 1996]. We show the same result for DTCS even with the simplest environment dynamics, which yields an undecidable problem in unbounded time and a PSPACE -complete problem in bounded time.

To summarize, decision-tree policies are highly desirable due to their interpretability. The most successful methods in obtaining such policies use machine learning, which do not guarantee safety of the resulting policy. While previous work managed to verify discrete-time policies, continuous-time policy verification remains an open problem. Our work bridges this gap with the first algorithm to verify decision-tree policies for continuous-time systems.

Outline. We organize the remainder of the paper as follows. First we formalize DTCS and the reach-avoid problem. Then we describe our verification algorithm and discuss the problem complexity. Next we report on experimental results. Finally we conclude and discuss directions for future work.

2 Preliminaries

2.1 Decision-Tree Controlled Systems

Let S ⊆ R n be an n -dimensional state space and U ⊆ R m an m -dimensional action (or input) space. A decision tree T over S and U is a binary tree such that each inner node is labeled with a predicate p : S → B (with B = {⊤ , ⊥} ) and each leaf is labeled with an action u ∈ U . The nodes in the tree are organized in levels, with the root node being at level 1 . Let root ( T ) denote the root node of T , left ( T ) resp. right ( T ) denote the left resp. right sub-tree of T , and ℓ ( T ) denote the label at the root of T (i.e., if T is a leaf, ℓ ( T ) is an action and otherwise a predicate). We can interpret T as a function from S to U as follows. Given a state x ∈ S , the image under T , written act ( x, T ) ∈ U , is defined recursively. For a leaf T , act ( x, T ) is just ℓ ( T ) . For a proper tree with root predicate p , act ( x, T ) is act ( x, left ( T )) if p ( x ) = ⊤ , and act ( x, right ( T )) otherwise. In this paper, we restrict our analysis to predicates of the form x i ≤ c where x i is the i -th state and c ∈ R is a constant. Geometrically, these predicates are axis-aligned half-spaces. This class of predicates is commonly used, e.g., in the tools Uppaal Stratego [David et al., 2015] (with industrial applications in control of smart homes [Larsen et al., 2016] and traffic lights [Eriksen et al., 2017]) and dtControl [Ashok et al., 2020].

We consider environments modeled by a system of ordinary differential equations (ODEs), ˙ x = f ( x, u ) , where x ( t ) ∈ S is the state vector and u ( t ) ∈ U is the vector of control actions. Given an initial state x (0) = x 0 and an action u 0 , we assume that the solution to the corresponding initial-value problem at time t ≥ 0 , written ξ ( x 0 , u 0 , t, f ) , exists and is unique (e.g., by Lipschitz continuity).

A decision-tree controlled system ( DTCS ) is a triple ( f, T , τ ) where f describes a system of ODEs, T is a decision tree, used as policy, and τ ∈ R + is a control period. Figure 2 shows a conceptual sketch. The DTCS periodically queries the policy for a new control action. At time points kτ , k ∈ N , the current state x ( kτ ) is passed to the policy T , which instantaneously yields the new control action u k ; this action is then used for the next control period τ in the system dynamics f . Formally, given an initial state x 0 at t = 0 , we recursively define the sequence of actions u k = act ( x ( kτ ) , T ) , k ∈ N , and the evolution of the state x ( t ) = ξ ( x ( kτ ) , u k , t -kτ, f ) (i.e., a trajectory) for t ∈ ( kτ, ( k +1) τ ] .

Environment

˙

x

=

f

(

x, u

)

Decision-tree policy

T

x

(

t

)

t

=

kτ

clock

x

(

kτ

)

u

k

=

act

(

x

(

kτ

)

,

T

)

2.2 Reachable States and Reach-Avoid Problem

Given a DTCS and a set of initial states X 0 ⊆ S , we are interested in the reachable states , either at time t as R t = { x ( t ) | x (0) ∈ X 0 } or the generalization to time intervals R [ T 0 ,T 1 ] = ⋃ t ∈ [ T 0 ,T 1 ] R t .

Similarly, a discrete-time DTCS is a pair ( f, T ) where f describes a recurrence x k +1 = x k + f ( x k , u k ) and T is a decision tree. The reachable states at step k are R k = { x k | x 0 ∈ X 0 } and analogously R [ K 0 ,K 1 ] = ⋃ k ∈ [ K 0 ,K 1 ] R k . By default, DTCS refers to the continuous-time DTCS introduced above.

The reach-avoid problem for DTCS is, given a DTCS ( f, T , τ ) over S , a set of initial states X 0 ⊆ S , a bounded number of control cycles k max, and two sets G , E ⊆ S , to decide whether all trajectories x ( t ) reach the goal set G without reaching the error set E before time T max = k max τ , i.e., ∃ t ∗ ≤ T max . x (0) ∈ X 0 ∧ x ( t ∗ ) ∈ G ∧ ∀ t ∈ [0 , t ∗ ] . x ( t ) / ∈ E . If we assume that the goal states G are absorbing, this is equivalent to checking

Example 2. Consider again the quadrotor model from Figure 1. A full model description is given in Appendix B.1. The reach-avoid problem here consists of the shaded area at the top as the goal set G , the set of states outside the red corridor as the error set E , and k max = 30 control cycles.

Determining reachability is already undecidable for uncontrolled nonlinear dynamical systems [Hainry, 2008], and hence also for DTCS with nonlinear dynamics. A DTCS can be seen as a hybrid (mixed discrete-continuous) system, for which reachability is undecidable even with linear dynamics [Doyen et al., 2018]. Due to these complexity barriers, we aim at enclosing , or overapproximating, the reachable states up to time horizon T max by computing a set R⊇R [0 ,T max ] .

3 Approach

Algorithm 1 Reachability algorithm for DTCS

In this section, we present our approach to reachability analysis for DTCS. We first describe a general high-level algorithm, which resembles standard reachability schemes for hybrid systems. It is, however, tailored to policies over a finite action space U . As such, it is applicable to policies beyond decision trees, but for instance not to typical neuralnetwork policies. We then outline conditions under which the algorithm is sound and relatively complete (i.e., does not introduce additional approximation errors). Finally, we instantiate the algorithm specifically for decision-tree policies, which is a novel contribution of this paper.

Algorithm 1 shows our high-level reachability method, which can be used to solve the reach-avoid problem (Section 2.2). The algorithm is parametric in two procedures, post T and post f , which together compute an enclosure of the reachable states for one control cycle. The queue Q contains pairs ( X , t ) , where X is a set of states that needs to be explored, and t is a time point. Each iteration of the while loop (line 3) analyzes one control cycle from X and t (unless the time horizon T max is reached, line 6). The result is added to the set R of reachable states in line 11. Next we describe the requirements for the post procedures.

The procedure post T , which is a new contribution of this paper, takes a set of states X and a decision tree T and returns a finite set of pairs ( Y u i , u ) , where Y u i is a set of states and u ∈ U is an action. Since multiple leaves of a decision tree can be associated with the same action u , we allow multiple sets Y u i to be associated with u as well. For each action u , the union of the sets Y u i should enclose the set act u ( X , T ) = { x ∈ X | act ( x, T ) = u } , so we have an index set I u (possibly empty) for each action u such that

Procedure post T wraps the sets Y u i together with action u .

For post T to be sound, we connect equations (2) and (3) and untangle the pairing with u as follows:

The procedure post f receives a set of states X , a control action u , the environment f , and a time interval [ t 0 , t 1 ] . (Note that we actually only need the duration t 1 -t 0 , which is τ most of the time, because we consider time-invariant systems. We only pass the time interval for notational purposes to refer to t 1 .) The goal is to perform a classical time-bounded reachability computation and return two sets Y and Z such that Y encloses the reachable states for the given time interval and Z encloses the reachable states at the final time point t 1 . Thus for post f to be sound, the following must hold:

Algorithm 1 in combination with equations (4) and (5) computes a sound enclosure. Let us write post ∗ T and post ∗ f for the procedures such that the inclusions in equations (4) and (5) are satisfied with equality. With such procedures, the algorithm even produces the exact result (proven in Appendix C.1).

Theorem 1 (Termination, soundness, relative completeness) . Assume Eq. (4) and Eq. (5) are satisfied. (1) Algorithm 1 terminates if all calls to post T and post f terminate. (2) Let ⊒ ∈ {⊇ , = } . The result R of Algorithm 1 encloses ( ⊇ ) resp. equals ( = ) the reachable states, R ⊒ R [0 ,T max ] , if in all steps post T ( X , T ) ⊒ post ∗ T ( X , T ) and post f ( X u , u, f, [ t 0 , t 1 ]) ⊒ post ∗ f ( X u , u, f, [ t 0 , t 1 ]) hold.

3.1 Implementing the Post Procedures

For implementing post f , we use an algorithm based on Taylor models [Berz and Makino, 1998] for reachability analysis of nonlinear dynamical systems. A Taylor model approximates a function as a polynomial together with an interval remainder over a domain [Makino and Berz, 2003], which we interpret as sets of states. For example, the one-dimensional Taylor model with polynomial p ( x ) = x 2 -x + 1 and remainder [ -0 . 5 , 0 . 5] over the domain [ -1 , 1] around expansion point 0 encodes the set { p ( x ) + r | x ∈ [ -1 , 1] , r ∈ [ -0 . 5 , 0 . 5] } . Taylor models subsume common set representations. Thus we assume that the initial set X 0 is given as a Taylor model.

Algorithm 2 summarizes the implementation of post f . Here TM_reach ( X , u, f, [ t 0 , t 1 ]) (line 1) computes a special Taylor model P ( t ) that depends on time t . To obtain Y and Z , we evaluate this Taylor model with different values for t : For Y we evaluate with the time interval [ t 0 , t 1 ] , which results in an enclosure of R [ t 0 ,t 1 ] . For Z we evaluate with the time point t 1 , which results in an enclosure of R t 1 . Algorithm 2 can be adapted for discrete-time systems, where the procedure TM_reach is replaced appropriately, line 2 is skipped, and the result is ( Z , Z ) .

Proposition 1. Algorithm 2 implements procedure post f satisfying Eq. (5) .

Algorithm 3 instantiates post T . Recall that the goal of this procedure is to compute enclosures of act u ( X , T ) , which are the sets of states that result in an action u . Given a set X and a decision tree T , the idea is to propagate X down the branches that the states x ∈ X would take. While the algorithm is

Xo

1.5

Y

Z

1.2

y

0.9

0.6

0.3

2

3

(a) Enclosures of the reachable states R [0 ,τ ] = Y (blue) and R τ = Z (purple) starting from X 0 (yellow).

2

3

4

(b) Bisection wrt. x ≤ 3 (red) into Z 1 (green) and Z 2 (brown), and interval enclosures (cyan, orange).

agnostic of the predicates in T (interpreted as sets of states) on a conceptual level, it is generally difficult to implement the algorithm for arbitrary predicates. In our implementation, we exploit the structure of axis-aligned half-space predicates P . Let box( X ) be the interval enclosure of X , which is easy to obtain for a Taylor model. In line 9, we check if all states in X take the left branch of T . We have X ⊆ P if and only if box ( X ) ⊆ P . In line 11 we check if all states in X take the right branch of T .

Algorithm 3 Post for the decision tree

We have X ∩ P = ∅ if and only if box ( X ) ∩ P = ∅ . If both conditions fail, X is bisected into Z 1 = { x | x ∈ X ∩ P } and Z 2 = { x | x ∈ X ∩ P C } in line 14. Here P C denotes the complement of P , which is also a half-space. Since the above sets are hard to compute, we replace X by box( X ) in the implementation. Note that box ( X ) ∩ P ⊇ box ( X ∩ P ) in our setting. We discuss this further in Appendix A.1.

Proposition 2. Algorithm 3 implements procedure post T satisfying Eq. (4) . Furthermore, if all bisections are exact, Eq. (4) is satisfied with equality.

Further details relevant for using our method in practice are given in Appendix A.

Example 3. We walk through Algorithm 1 for the first control cycle, illustrated in Figure 3. Figure 3(a) shows how Algorithm 2 computes the pair ( Y , Z ) , consisting of an enclosure R [0 ,τ ] of the reachable states up to time point τ and an enclosure R τ of the reachable states at the last time point. Figure 3(b) shows how Algorithm 3 bisects the set Z along the predicate x ≤ 3 into sets Z 1 and Z 2 . We furthermore illustrate the approximation with interval enclosures. Assuming this is the only predicate of the decision tree, Algorithm 1 would continue the analysis from these two sets in the next iteration.

3.2 Generalization to Unbounded Time via Fixpoints

Next we discuss how to employ a set-based fixpoint check; for details we refer to the literature [Giacobbe, 2019]. Roughly speaking, if we do not encounter new states after an iteration, we have found a fixpoint. Algorithm 1 iteratively adds states to R to enclose the sequence R [ kτ, ( k +1) τ ] , k < k max (Section 2.2). Let us view the exploration of the elements ( X , t ) in the queue Q as a search tree. First, the search in a node can be terminated if the set X is contained in the union of the sets in the other nodes (i.e., if a fixpoint has been found). Second, if this condition holds in all branches, we have computed the reachable states in unbounded time. However, due to the discrete nature of

y

1.5

1.2

0.9

0.6

0.3

box(Z1)

Z1

the policy, the time points at which to check for fixpoints are relevant. A sufficient condition is to only compare the sets at the beginning of each control cycle, i.e., R kτ ⊆ ⋃ j<k R jτ for some k > 0 . However, when dealing with enclosures, this condition would become R kτ ⊆ ⋃ j<k R jτ , which is not sufficient. The set R kτ on the left-hand side must additionally contain the reachable states from all sets R jτ on the right-hand side. Fortunately, our algorithm satisfies this additional condition.

In our implementation, all sets are (unions of) Taylor models, for which inclusion checking is hard. While we can overapproximate the left-hand side, we would have to underapproximate the right-hand side, which is difficult for Taylor models. However, whenever we bisect sets in post T , we obtain box-shaped sets, for which inclusion checks are easy. Thus, in our implementation, we only perform a fixpoint check after a bisection. We shall see in the evaluation that this is often sufficient in practice.

3.3 State-Independent Dynamics and Computational Complexity

We call a control system state-independent if f ( x, u ) = u for all x (for which u must be n -dimensional). This is arguably the simplest possible control system. If the system dynamics are state-independent and X 0 is a polyhedron (i.e., the intersection of linear inequalities), post f can be implemented exactly and yields a polyhedron [Alur et al., 1996]. Given a polyhedron, post T returns a union of polyhedra. Under these assumptions, Algorithm 1 can be implemented to compute the exact reachable states and thus satisfy Theorem 1 with equality. Finally, Algorithm 1 can be turned into a decision procedure for the reach-avoid problem (namely: also output R T max and check Eq. (1)). Nevertheless, state-independent DTCS can encode Turing machines. Hence the reach-avoid problem is undecidable in unbounded time and PSPACE -complete in bounded time (proven in Appendix C.2).

Theorem 2. The reach-avoid problem for both continuous-time and discrete-time DTCS over rational numbers with state-independent dynamics and polyhedral initial, goal, and error sets ( X 0 , G , E ) is PSPACE -complete, and undecidable when considering the unbounded-time version.

Note that Algorithm 1 solves not only the decision problem but computes the full reachable states. Thus it needs to collect all the sets resulting from bisections. If the number of iterations k max is given in binary, the algorithm is doubly exponential, even if we ignore a potential complexity growth in the set representation. We conjecture that this bound is optimal for the full computation. More precisely, in each iteration, a set may be bisected into several (but at most ℓ , where ℓ is the number of leaves of the decision tree) sets. With k max control cycles, the algorithm may end up with ℓ k max sets in the end.

4 Evaluation

We implemented 1 our method in a tool based on JuliaReach [Bogomolov et al., 2019] for the Taylor-model algorithm (TM_reachin Algorithm 2) [Benet et al., 2019] and LazySets [Forets and Schilling, 2021] for the set computations. We demonstrate our approach on the quadrotor system (Figure 1(a)) and three classical control problems from the Gymnasium [Brockman et al., 2016]: cart/pole, acrobot, and mountain/car. In the following, we use our proof-of-concept implementation to verify and visually explain that, within the given time constraints, the quadrotor safely reaches the end of the corridor, the cart manages to stabilize the pole, the acrobot swings to the

goal height, and the car reaches the top of the mountain. It is crucial to maintain high precision during the analysis to avoid divergence. The systems illustrate different aspects: Our approach applies to time-dependent policies (quadrotor control), proves infinite-time stability (Section 3.2) (cart/pole), handles transformations (Appendix A.2) (acrobot), and deals with discrete-time dynamics (mountain/car), while preserving high precision.

0.3

0.0

Py

-0.3

initial

0.6

goal

reference

-boundary

0.0

0.2

0.4

0.6

0.8

1.0

1.2

We obtain decision-tree policies via neural-network distillation (see Table 1 on the depth and number of nodes and actions). For the quadrotor system we collected 500 samples of state-action pairs from a neural-network policy used by Ivanov et al. [2019] and learned a decision tree of depth 10 using behavioral cloning. For the three other systems (cart/pole, acrobot, mountain/car) we first used deep Q-learning to train a two-layer convolutional neural network [Mnih et al., 2013] and then adopted the Viper algorithm [Bastani et al., 2018] to imitation-learn decision trees for these systems. We manually prune the resulting trees of redundant nodes (as Viper produces balanced trees).

All experiments were conducted on a laptop with an i7 1.80 GHz CPU and 32 GB RAM.

4.1 Evaluation on a Decision Tree for Quadrotor Control

We evaluate our verification approach on a decision tree controlling a six-dimensional quadrotor, tasked to follow a piecewise-linear plan (see, Example 1). The actions represent possible combinations of pitch, roll, and thrust acceleration. This is a complex continuous-time system (see Appendix B.1 for full details). The policies for such high-dimensional systems are typically approximated by a learned model, e.g., a neural network [Royo et al., 2019]. We train a decision-tree policy of depth 10 imitating the neural network from [Ivanov et al., 2019].

Figure 4 shows the set of reachable states. Ivanov et al. [2019] verified the neural-network policy, for which they had to split the initial set into 16 subsets to tame the approximation error. Computing the reachable states took between 10 and 59 minutes for each subset. Our method can be applied to the full initial set directly and verifies the system in 6 . 5 minutes.

4.2 Evaluation on Decision Trees for Classic Nonlinear Control Problems

Next we study three classical control systems, for which we trained small, interpretable decision trees (see Table 1). Small decision trees are often sufficient for optimality [V os and Verwer, 2023].

Cart-Pole System. We consider the cart-pole system [Barto et al., 1983]. A description is given in Appendix B.2. The goal is for the pole to remain vertically stable within an angle of ± 0 . 06 ◦ . This system is challenging because of quick alternations in the control action. In Figure 5(a), we show the reachable states in the θ / ω projection together with the decision boundaries (e.g., the policy moves the cart to the left in the green region). The analysis terminates in 40 seconds and proves that the blue dashed line θ = 0 . 06 is not exceeded. Furthermore, the fixpoint check allows to generalize the results to infinite time; thus we can conclude that the policy is able to balance the pole forever.

left

0.5

goal

right

1.0

initial

avoid

0.0

0.5

0.0

0.5

0.5

1.0

0.03

0.00

0.03

0.06

0

20

40

60

80

100

120

time

(a) Cart-pole system. The initial states are shown as yellow borders. The angle safety threshold is shown as a dashed blue. The background (green, cyan) shows the decision boundaries.

15

15

10

10

02

0

0

goal

initial

initial

7.5

5.0

0.0

2.5

7.5

5.0

0.0

2.5

01

01

Acrobot. We consider the acrobot system [Sutton, 1995], which consists of two links connected by a joint, one of which can swing freely. A full description is given in Appendix B.3. The goal is that the free end reaches a desired height, expressed as the condition -cos( θ 1 ) -cos( θ 2 + θ 1 ) > 1 , where θ 1 is the joint angle and θ 2 is relative to the angle of the first link. This system has complex nonlinear dynamics that require high precision. We obtain two decision-tree policies and compare their performance. The system applies a nonlinear transformation to the state before passing it to the policy. We discuss handling transformations in Appendix A.2. Figure 6 illustrates that all trajectories reach the goal regions. The analysis takes 100 (first policy) resp. 112 seconds (second policy).

Mountain-Car System. To demonstrate applicability to discrete-time systems, we consider the mountain-car system, where a car has to reach from one mountain to another [Moore, 1990]. A description is given in Appendix B.4. Figure 5(b) shows that the car always reaches the top of the mountain within at most 132 steps. The analysis terminates in 16 seconds.

4.3 Comparison with State-of-the-Art Reachability Tool for Hybrid Automata

As explained before, a DTCS can be seen as a special case of a hybrid automaton. We encoded the various DTCS from the evaluation as hybrid automata (a description of the encoding is given in Appendix D) and then applied the state-of-the-art reachability tool JuliaReach [Bogomolov et al., 2019] to them. The result was that the tool got stuck as soon as it had to bisect the set of states for the first time, and we had to terminate the tool after several minutes of no progress. For instance, when we evaluated the reachability tool on the cart-pole system, the tool can analyze the first four control periods (out of 25) in 22 seconds, but then gets stuck in the fifth control period when the bisection

starts, and prints errors about numerical instability. This demonstrates that our algorithm is the first feasible solution to the reach-avoid problem for DTCS.

Our algorithm exploits the decision-tree structure to only explore the feasible automaton transitions (most of the time this is just one). Furthermore, given the periodic control policy, we only have to check the transitions at specific time points (an observation we already made in earlier work [Forets et al., 2020]). Moreover, since we focus on axis-aligned predicates, interval enclosures are often sufficient. Finally, often only one branch of the decision tree is relevant, which allows us to ignore the complement branch and avoid unnecessary calculations. Even if several leaves of the decision tree are reached, these may still all be annotated with the same control action. In that case we do not have to bisect the set (which avoids the main source of approximation error).

The other tool views the automaton as a black-box model and thus cannot make use of this structure; instead it needs to perform many intersection operations, which are expensive and typically force to use a more complex set representation. These structural insights make our analysis not only more efficient but also more precise in practice because general hybrid-automata tools would typically approximate these operations.

5 Conclusion

In this paper, we studied the reach-avoid problem for continuous-time and discrete-time dynamical systems controlled by a decision tree. The problem is undecidable for nonlinear systems, and we showed undecidability even for the simplest dynamics as well as PSPACE -completeness in the bounded-time setting. We proposed the first practical algorithm to solve this problem in continuous time. The abstract algorithm is sound and, for simple systems, complete. We implemented the algorithm for nonlinear systems and decision trees with axis-aligned predicates. Our evaluation shows that the algorithm is precise and performant on typical problems. Our approach enriches the verification toolset for machine-learning based systems and opens novel cross-community research challenges. Our approach lends itself to visualization and can serve for further analysis and refinement of decision trees, which themselves are interpretable surrogates for black-box policies.

Coming to the limitations of our approach, we have only considered time-invariant systems. While the algorithms are general, the fixpoint check we used does not apply to time-varying systems. In our experiments, with the exception of the quadrotor, we focused on small decision trees. We note that the size of the decision tree is not necessarily a good measure for complexity. Multiple leaves may share the same action, and some leaves may not be used at all during the execution. The main impact on the verification method, in our experience, is the number of times the decision boundaries are partially crossed by the reachable states (after a time step). Furthermore, the control systems we verified in the evaluation, while nontrivial from a verification perspective, have relatively simple control tasks where a decision tree is not required. We plan to investigate how our method performs on systems with more challenging control tasks, e.g., the cart/pole system starting with the pole hanging downward (which we were not able to verify in a preliminary attempt).

In future work we will also study how the analysis can be improved both in terms of precision and scalability. For precision, we plan to employ set representations that are closed under bisection, e.g., constrained polynomial zonotopes [Kochdumper and Althoff, 2023], which generalize Taylor models. For scalability, we aim to improve the fixpoint check using simulation-based heuristics. Another natural direction is to apply the approach to learning a safe decision-tree policy, for which methods to compute underapproximations in order to refute unsafe models would be useful. Finally, our recent algorithm to synthesize a safety shield for policies of systems with complex hybrid dynamics [Brorholt et al., 2023] only detects discrete-time safety violations; hence we aim to find synergies with the method presented in this paper.

Acknowledgements. We are grateful for the anonymous reviewers' helpful comments to improve the paper and suggesting the swing-up cart/pole experiment as future challenge. Anna Lukina thanks Mustafa Mert Çelikok and Alexandru B˘ abeanu for their timely feedback. This research was partly supported by DIREC - Digital Research Centre Denmark under reference number 9142-0001B and the Villum Investigator Grant S4OS under reference number 37819.

References

E. Akintunde, E. Botoeva, P. Kouvaros, and A. Lomuscio. Formal verification of neural agents in non-deterministic environments. Auton. Agents Multi Agent Syst. , 36(1):6, 2022. URL ❤/a116/a116♣/a115✿ ✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴/a115✶✵✹✺✽✲✵✷✶✲✵✾✺✷✾✲✸ .

A. Alamdari, G. Avni, T. A. Henzinger, and A. Lukina. Formal methods with a touch of magic. In FMCAD , pages 138-147. IEEE, 2020. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✸✹✼✷✼✴✷✵✷✵✴✐/a115❜♥✳ ✾✼✽✲✸✲✽✺✹✹✽✲✵✹✷✲✻❴✷✶ .

Althoff, G. Frehse, and A. Girard. Set propagation techniques for reachability analysis. Annual Review of Control, Robotics, and Autonomous Systems , 4(1):369-395, 2021. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳ ♦/a114❣✴✶✵✳✶✶✹✻✴❛♥♥✉/a114❡✈✲❝♦♥/a116/a114♦❧✲✵✼✶✹✷✵✲✵✽✶✾✹✶ .

Alur, T. A. Henzinger, and P. Ho. Automatic symbolic verification of embedded systems. IEEE Trans. Software Eng. , 22(3):181-201, 1996. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✵✾✴✸✷✳✹✽✾✵✼✾ .

Ashok, J. Kretínský, K. G. Larsen, A. L. Coënt, J. H. Taankvist, and M. Weininger. SOS: safe, optimal and small strategies for hybrid Markov decision processes. In QEST , volume 11785 of LNCS , pages 147-164. Springer, 2019. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✵✸✵✲✸✵✷✽✶✲✽❴✾ .

Ashok, M. Jackermeier, P. Jagtap, J. Kretínský, M. Weininger, and M. Zamani. dtControl: decision tree learning algorithms for controller representation. In HSCC , pages 17:1-17:7. ACM, 2020. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✸✻✺✸✻✺✳✸✸✽✷✷✷✵ .

Bacci, M. Giacobbe, and D. Parker. Verifying reinforcement learning up to infinity. In IJCAI , pages 2154-2160. ijcai.org, 2021. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✷✹✾✻✸✴✐❥❝❛✐✳✷✵✷✶✴✷✾✼ .

G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. , 13(5):834-846, 1983. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✵✾✴❚❙▼❈✳✶✾✽✸✳✻✸✶✸✵✼✼ .

Bastani, Y. Pu, and A. Solar-Lezama. Verifiable reinforcement learning via policy extraction. In NeurIPS , pages 2499-2509, 2018. URL ❤/a116/a116♣/a115✿✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♥❡✉/a114✐♣/a115✳❝❝✴♣❛♣❡/a114✴✷✵✶✽✴ ❤❛/a115❤✴❡✻❞✽✺✹✺❞❛❛✹✷❞✺❝❡❞✶✷✺❛✹❜❢✼✹✼❜✸✻✽✽✲❆❜/a115/a116/a114❛❝/a116✳❤/a116♠❧ .

Benet, M. Forets, D. P. Sanders, and C. Schilling. TaylorModels.jl: Taylor models in Julia and its application to validated solutions of ODEs. In SWIM , 2019. URL ❤/a116/a116♣/a115✿✴✴/a115✇✐♠✷✵✶✾✳ ❡♥/a115/a116❛✲♣❛/a114✐/a115/a116❡❝❤✳❢/a114✴♣❛♣❡/a114/a115✴❜❡♥❡/a116✳♣❞❢ .

Berz and K. Makino. Verified integration of odes and flows using differential algebraic methods on high-order Taylor models. Reliab. Comput. , 4(4):361-369, 1998. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴ ✶✵✳✶✵✷✸✴❆✿✶✵✷✹✹✻✼✼✸✷✻✸✼ .

Bogomolov, M. Forets, G. Frehse, K. Potomkin, and C. Schilling. JuliaReach: a toolbox for set-based reachability. In HSCC , pages 39-44. ACM, 2019. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴ ✸✸✵✷✺✵✹✳✸✸✶✶✽✵✹ .

Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees . Wadsworth, 1984. ISBN 0-534-98053-8.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. CoRR , 2016. URL ❤/a116/a116♣/a115✿✴✴❛/a114①✐✈✳♦/a114❣✴❛❜/a115✴✶✻✵✻✳✵✶✺✹✵ .

H. Brorholt, P. G. Jensen, K. G. Larsen, F. Lorber, and C. Schilling. Shielded reinforcement learning for hybrid systems. In (A)ISoLA , 2023. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✹✽✺✺✵✴❛/a114❳✐✈✳ ✷✸✵✽✳✶✹✹✷✹ .

David, P. G. Jensen, K. G. Larsen, M. Mikucionis, and J. H. Taankvist. Uppaal Stratego. In TACAS , volume 9035 of LNCS , pages 206-211. Springer, 2015. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳ ✶✵✵✼✴✾✼✽✲✸✲✻✻✷✲✹✻✻✽✶✲✵❴✶✻ .

Doyen, G. Frehse, G. J. Pappas, and A. Platzer. Verification of hybrid systems. In Handbook of Model Checking , pages 1047-1110. Springer, 2018. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴ ✾✼✽✲✸✲✸✶✾✲✶✵✺✼✺✲✽❴✸✵ .

Du, N. Liu, and X. Hu. Techniques for interpretable machine learning. Commun. ACM , 63(1): 68-77, 2020. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✸✺✾✼✽✻ .

Dutta, X. Chen, and S. Sankaranarayanan. Reachability analysis for neural feedback systems using regressive polynomial rule inference. In HSCC , pages 157-168. ACM, 2019. URL ❤/a116/a116♣/a115✿ ✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✸✵✷✺✵✹✳✸✸✶✶✽✵✼ .

B. Eriksen, C. Huang, J. Kildebogaard, H. Lahrmann, K. G. Larsen, M. Muniz, and J. H. Taankvist. Uppaal Stratego for intelligent traffic lights. In ITS European Congress , 2017.

Fan, C. Huang, X. Chen, W. Li, and Q. Zhu. ReachNN*: A tool for reachability analysis of neural-network controlled systems. In ATVA , volume 12302 of LNCS , pages 537-542. Springer, 2020. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✵✸✵✲✺✾✶✺✷✲✻❴✸✵ .

Forets and C. Schilling. LazySets.jl: Scalable symbolic-numeric set computations. Proceedings of the JuliaCon Conferences , 1(1):11, 2021. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✷✶✶✵✺✴❥❝♦♥✳✵✵✵✾✼ .

Forets, D. Freire, and C. Schilling. Efficient reachability analysis of parametric linear hybrid systems with time-triggered transitions. In MEMOCODE , pages 137-142. IEEE, 2020. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✵✾✴▼❊▼❖❈❖❉❊✺✶✸✸✽✳✷✵✷✵✳✾✸✶✹✾✾✹ .

Fränzle and C. Herde. Hysat: An efficient proof engine for bounded model checking of hybrid systems. Formal Methods Syst. Des. , 30(3):179-198, 2007. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴ /a115✶✵✼✵✸✲✵✵✻✲✵✵✸✶✲✵ .

Fränzle, C. Herde, T. Teige, S. Ratschan, and T. Schubert. Efficient solving of large non-linear arithmetic constraint systems with complex boolean structure. J. Satisf. Boolean Model. Comput. , 1(3-4):209-236, 2007. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✸✷✸✸✴/a115❛/a116✶✾✵✵✶✷ .

Giacobbe. Automatic time-unbounded reachability analysis of hybrid systems . PhD thesis, IST Austria, 2019.

Hainry. Reachability in linear dynamical systems. In CiE , volume 5028 of LNCS , pages 241-250. Springer, 2008. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✺✹✵✲✻✾✹✵✼✲✻❴✷✽ .

A. Henzinger. The theory of hybrid automata. In LICS , pages 278-292. IEEE Computer Society, 1996. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✵✾✴▲■❈❙✳✶✾✾✻✳✺✻✶✸✹✷ .

Hyötyniemi. Turing machines are recurrent neural networks. Proceedings of step , 96, 1996.

Ivanov, J. Weimer, R. Alur, G. J. Pappas, and I. Lee. Verisig: verifying safety properties of hybrid systems with neural network controllers. In HSCC , pages 169-178. ACM, 2019. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✸✵✷✺✵✹✳✸✸✶✶✽✵✻ .

Ivanov, T. J. Carpenter, J. Weimer, R. Alur, G. J. Pappas, and I. Lee. Verisig 2.0: Verification of neural network controllers using Taylor model preconditioning. In CAV , volume 12759 of LNCS , pages 249-262. Springer, 2021. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✵✸✵✲✽✶✻✽✺✲✽❴✶✶ .

Kochdumper and M. Althoff. Constrained polynomial zonotopes. Acta Informatica , 2023. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴/a115✵✵✷✸✻✲✵✷✸✲✵✵✹✸✼✲✺ .

Kochdumper, C. Schilling, M. Althoff, and S. Bak. Open- and closed-loop neural network verification using polynomial zonotopes. In NFM , volume 13903 of LNCS , pages 16-36. Springer, 2023. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✵✸✶✲✸✸✶✼✵✲✶❴✷ .

G. Larsen, M. Mikucionis, M. Muñiz, J. Srba, and J. H. Taankvist. Online and compositional learning of controllers with application to floor heating. In TACAS , volume 9636 of LNCS , pages 244-259. Springer, 2016. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✻✻✷✲✹✾✻✼✹✲✾❴✶✹ .

M. Lopez, M. Althoff, L. Benet, X. Chen, J. Fan, M. Forets, C. Huang, T. T. Johnson, T. Ladner, W. Li, C. Schilling, and Q. Zhu. ARCH-COMP22 category report: Artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plants. In ARCH , volume 90 of EPiC Series in Computing , pages 142-184. EasyChair, 2022. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳ ♦/a114❣✴✶✵✳✷✾✵✵✼✴✇❢❣/a114 .

Makino and M. Berz. Taylor models and other validated functional inclusion methods. Int. J. Pure Appl. Math , 4(4):379-456, 2003.

Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller. Playing Atari with deep reinforcement learning. CoRR , 2013. URL ❤/a116/a116♣/a115✿✴✴❛/a114①✐✈✳♦/a114❣✴❛❜/a115✴ ✶✸✶✷✳✺✻✵✷ .

Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nat. , 518(7540):529-533, 2015. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✸✽✴♥❛/a116✉/a114❡✶✹✷✸✻ .

W. Moore. Efficient Memory-based Learning for Robot Control . PhD thesis, University of Cambridge, 1990.

Neumaier. The wrapping effect, ellipsoid arithmetic, stability and confidence regions , pages 175-190. Springer, 1993. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✼✵✾✶✲✻✾✶✽✲✻❴✶✹ .

R. Quinlan. Learning decision tree classifiers. ACM Comput. Surv. , 28(1):71-72, 1996. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✷✸✹✸✶✸✳✷✸✹✸✹✻ .

R. Royo, D. Fridovich-Keil, S. L. Herbert, and C. J. Tomlin. A classification-based approach for approximate reachability. In ICRA , pages 7697-7704. IEEE, 2019. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳ ✶✶✵✾✴■❈❘❆✳✷✵✶✾✳✽✼✾✸✾✶✾ .

J. Savitch. Relationships between nondeterministic and deterministic tape complexities. J. Comput. Syst. Sci. , 4(2):177-192, 1970. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴❙✵✵✷✷✲✵✵✵✵✭✼✵✮✽✵✵✵✻✲❳ .

Schilling, M. Forets, and S. Guadalupe. Verification of neural-network control systems by integrating Taylor models and zonotopes. In AAAI , pages 8169-8177. AAAI Press, 2022. URL ❤/a116/a116♣/a115✿✴✴♦❥/a115✳❛❛❛✐✳♦/a114❣✴✐♥❞❡①✳♣❤♣✴❆❆❆■✴❛/a114/a116✐❝❧❡✴✈✐❡✇✴✷✵✼✾✵ .

C. Shepherdson and H. E. Sturgis. Computability of recursive functions. J. ACM , 10(2):217-255, 1963. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✷✶✶✻✵✳✸✷✶✶✼✵ .

S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. Mozer, and M. E. Hasselmo, editors, NeurIPS , pages 1038-1044. MIT Press, 1995. URL ❤/a116/a116♣/a115✿✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♥❡✉/a114✐♣/a115✳❝❝✴♣❛♣❡/a114✴✶✾✾✺✴❤❛/a115❤✴ ✽❢✶❞✹✸✻✷✵❜❝✻❜❜✺✽✵❞❢✻❡✽✵❜✵❞❝✵✺❝✹✽✲❆❜/a115/a116/a114❛❝/a116✳❤/a116♠❧ .

Tran, X. Yang, D. M. Lopez, P. Musau, L. V. Nguyen, W. Xiang, S. Bak, and T. T. Johnson. NNV: the neural network verification tool for deep neural networks and learning-enabled cyberphysical systems. In CAV , volume 12224 of LNCS , pages 3-17. Springer, 2020. URL ❤/a116/a116♣/a115✿ ✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✵✸✵✲✺✸✷✽✽✲✽❴✶ .

Urban and A. Miné. A review of formal methods applied to machine learning. CoRR , 2021. URL ❤/a116/a116♣/a115✿✴✴❛/a114①✐✈✳♦/a114❣✴❛❜/a115✴✷✶✵✹✳✵✷✹✻✻ .

Vos and S. Verwer. Optimal decision tree policies for Markov decision processes. In IJCAI , pages 5457-5465. ijcai.org, 2023. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✷✹✾✻✸✴✐❥❝❛✐✳✷✵✷✸✴✻✵✻ .

Xue, A. Easwaran, N. Cho, and M. Fränzle. Reach-avoid verification for nonlinear systems based on boundary analysis. IEEE Trans. Autom. Control. , 62(7):3518-3523, 2017. URL ❤/a116/a116♣/a115✿ ✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✵✾✴❚❆❈✳✷✵✶✻✳✷✻✶✺✺✾✾ .

A Additional Information on our Method

This section contains some technical remarks on the precision of our method as well as on how to deal with state transformations. The latter is relevant for solving one of the benchmark problems in the evaluation (Section 4).

A.1 Precision of the Algorithm

The precision of the Taylor-model implementation of post f (Algorithm 2) is good in practice most of the time. The main approximation error in Algorithm 1 comes from the bisections in post T (Algorithm 3) and the subsequent approximation, as exemplified in Figure 3(b). This is hard to avoid with nonlinear dynamics, for which bisection of non-convex sets is challenging. With iterative algorithms like ours, the issue with approximation errors is that they accumulate over time, which is known as the wrapping effect [Neumaier, 1993], especially if there are frequent changes of the control actions (i.e., crossings of the decision boundary). To reduce the likelihood of a bisection and accumulation of errors, one can subdivide X 0 into smaller subsets and run an analysis from each of them so that either all or none of the states cross the decision boundary at the same step (for instance, as mentioned in Section 4.1, this idea was used by Ivanov et al. [2019] for the quadrotor model). This can sometimes be achieved more efficiently by only considering the set boundaries [Xue et al., 2017].

A.2 State Transformations Before Evaluating the Decision Tree

So far we have assumed that the decision tree T takes the current state as input, as shown in Figure 2. However, some policies instead receive a transformation of the state. This is particularly useful for decision-tree policies with linear predicates (like ours) because a nonlinear transformation effectively augments them with nonlinear predicates. For instance, the acrobot model (see Appendix B.3 for more information) has the state vector ( θ 1 , θ 2 , ˙ θ 1 , ˙ θ 2 ) , but the input to T is the vector (cos( θ 1 ) , sin( θ 1 ) , cos( θ 2 ) , sin( θ 2 ) , ˙ θ 1 , ˙ θ 2 ) . To account for transformations, we can modify Algorithm 3 as follows. First we apply the transformation to the input set X , which is easy if X is given as a Taylor model. Then we apply Algorithm 3, and finally we have to 'undo' the transformation for the output. We distinguish three cases. In the first case, if the output is a single pair ( X u , u ) (i.e., there was no bisection), we replace it with ( X , u ) . In all other cases, there was a bisection. If the transformation is invertible, as in the above case, we apply the inverse transformation (here: (arccos( x 1 ) , arcsin( x 2 ) , x 5 , x 6 ) , where x is a state of the output set X u ) to each set X u in the output pairs ( X u , u ) . Otherwise, if the transformation is not invertible, we conservatively replace each pair ( X u , u ) with the pair ( X , u ) .

B Additional Information on the Benchmark Systems

Below we provide further information on the systems from the evaluation (Section 4).

B.1 Quadrotor System

A quadrotor is tasked to follow a piecewise-linear reference trajectory via bang-bang control (see Figure 1(a)). The six-dimensional state space consists of two 3D vectors for position and velocity. The action space consists of a 3D vector representing possible combinations of pitch ( θ ), roll ( ϕ ), and thrust acceleration ( α ). Using the gravity constant g = 9 . 81 , the continuous dynamics are:

The policy's task is to minimize the difference between the reference and the actual trajectory. The set of initial states is X 0 = [ -0 . 05 , 0 . 05] 2 ×{ 0 } 4 . The control period is 0 . 2 time units and the number of control cycles is 30 . The decision-tree policy T has depth 10 and is too large to depict here.

B.2 Cart-Pole System

A pole is vertically attached on top of a cart that can be moved left or right along a frictionless track. The goal is to move the cart such that the pole is kept in an upright pose (i.e., the pole angle is small).

The state consists of the cart position p , cart velocity v , pole angle θ , and angular velocity ω . The action space is U = {-1 , 1 } ('left', 'right') with control variable u . The continuous dynamics are:

The set of initial states is the union of two hyperrectangles X 0 = [ -0 . 05 , 0 . 05] 4 and { 0 } 2 × [ -0 . 025 , 0 . 025] × [ -0 . 2 , 0 . 2] . (We added the second hyperrectangle to make the problem more challenging.) The control period is 0 . 02 time units. The decision-tree policy T is shown in Figure 7(a) and works as follows. If the pole is leaning to the left ( θ ≤ -0 . 014 ) or moving in the left direction ( ω ≤ -0 . 201 ), the cart is moved to the left (action -1 in the leaves), and otherwise to the right.

The goal is to keep the pole angle θ within a small range around 0 . Figure 8 shows simulations from the corners of X 0 plus 100 random initial states inside X 0 for 2 time units. From these simulations, we can see that θ seems to stay in the interval [ -0 . 06 , 0 . 06] , but some trajectories come close to the upper bound. In the evaluation (Figure 5(a)) we prove this property for R [0 , 0 . 5] .

The fixpoint check allows to conclude that the policy is able to balance the pole forever. The fixpoint is only found in the θ / ω projection, but this is sufficient because the cart's position and velocity are irrelevant for the decisions of the tree.

B.3 Acrobot System

The acrobot system consists of two links connected by a joint to form a chain. One end is fixed while the joint and the other end can swing freely. There are four state dimensions: angles θ 1 and θ 2 , and their velocities ˙ θ 1 and ˙ θ 2 . The action space is U = { 0 , 1 , 2 } ('left', 'none', 'right') with control variable u . The continuous dynamics of the velocities are:

where

The goal is that the free end reaches a desired height, expressed as the condition -cos( θ 1 ) -cos( θ 2 + θ 1 ) > 1 , where θ 1 is the joint angle and θ 2 is relative to the angle of the first link.

The initial condition X 0 is a union of a point and a hyperrectangle X 0 = ( {-0 . 0005 } × { 0 } ∪ [ -0 . 0705 , -0 . 0695] × [ -0 . 0005 , 0 . 0005]) ×{ 0 } 2 . The control period is 0 . 2 time units and the time

θ

≤ -

0

.

014

-

1

ω

≤ -

0

.

201

-

1

1

(a) A decision tree for the cart-pole system.

v

0

v

≤ -

0

.

004

0

x

0

.

391

2

0

x

≤ -

0

.

37

2

1

(b) A decision tree for the mountain-car system.

0.06

0.00

θ

-0.06

0.0

0.5

1.0

1.5

2.0

time

˙

θ

1

≤ -

0

.

097

˙

θ

2

≤ -

1

.

131

0

2

˙

θ

2

1

.

582

0

2

˙

θ

1

≤ -

0

.

099

˙

θ

2

≤ -

1

.

351

˙

θ

1

≤ -

2

.

661

2

0

2

˙

θ

2

1

.

123

0

2

horizon is 100 time units. For this system we obtained two different decision-tree policies of depth 2 resp. 3 , shown in Figure 9.

The system applies a transformation to the state before being passed to the decision-tree policy. Concretely, the state vector ( θ 1 , θ 2 , ˙ θ 1 , ˙ θ 2 ) is mapped to the six-dimensional vector (cos( θ 1 ) , sin( θ 1 ) , cos( θ 2 ) , sin( θ 2 ) , ˙ θ 1 , ˙ θ 2 ) . (This transformation is technically irrelevant for the two policies we obtained, because they only consider ˙ θ 1 and ˙ θ 2 , which can be read from the transformed vector; but we do not exploit that.) We explain how to modify our algorithm in order to cope with transformations in Appendix A.2.

B.4 Mountain-car system

The mountain-car system consists of a sinusoidal valley between two mountains and a car in between. The goal is to bring the car to the top of the higher mountain on the right. The engine is not strong enough to drive up the mountain by itself, so the car must drive up the left mountain and gain momentum to reach the top of the right mountain.

The control actions are U = { 0 , 1 , 2 } ('left', 'none', 'right') with control variable u . The system state consists of the car's position x and velocity v . With constants F = 0 . 001 and g = 0 . 0025 , the discrete-time dynamics are:

The set of initial states is X 0 = [ -0 . 505 , -0 . 445] × { 0 } . The decision-tree policy T is given in Fig. 7(b). The maximum number of discrete steps is 200 , but as shown in Fig. 5(b), all trajectories reach the goal region within at most 132 steps.

C Proofs of the Theorems

C.1 Proof of Theorem 1

Proof. We show (i) termination and (ii) soundness and relative completeness of Algorithm 1.

(i) The inner loop in line 9 terminates by assumption. The outer loop in line 3 removes one element at time point t 0 from Q and adds only elements at time point t 1 = min( t 0 + τ, T max ) > t 0 to Q (line 8). Elements after time point T max are not processed (line 6).

(ii) We only show the equality case; the inclusion case follows because, in each step, all sets are overapproximated. The algorithm starts with X 0 = R 0 . In the first iteration, by assumption, post T only returns states x 0 ∈ X 0 together with the corresponding action act ( x 0 , T ) . Similarly, post f only returns states reachable within [0 , τ ] . By the end of the first while-loop iteration, we thus obtain R = R [0 ,τ ] , and the queue Q contains pairs ( X u , τ ) such that the union of the sets X u is the set R τ .

Further loop iterations work similarly. While the iteration order of Q (line 4) is not prescribed, all policies lead to the same result. Let us choose the implementation of Q as a LIFO (last-in, first-out) queue. This corresponds to a breadth-first search where on level k +1 of the search tree we have elements ( X i , kτ ) ; the union of these sets X i is again R kτ . The claim follows by induction.

C.2 Proof of Theorem 2

We first prove PSPACE-completeness in bounded time. For the membership, we sketch a nondeterministic algorithm with polynomial space requirements for the complement of the reach-avoid problem, which is sufficient by Savitch's theorem [Savitch, 1970] and because PSPACE is closed under complementation. The algorithm guesses an initial state and simulates the DTCS. In continuous time, the trajectory in one control cycle is a line segment. In each step, the algorithm checks whether (i) the current line segment intersects with the error states E and does not intersect with the goal set G , or (ii) the final state in the last iteration k max is outside the goal set G ; these checks can again be performed efficiently. The number of iterations may be exponential, but we can maintain a counter with logarithmically many bits.

For the hardness, we show a polynomial-time reduction from the word problem for deterministic linear bounded automata (LBA). Here we consider the LBA model with accepting state, total transition function, and the input tape containing the input word plus two bound markers for simplicity.

For better intuition, we provide an example after the proof, which we recommend reading in parallel.

Given is an input word w ∈ { 0 , 1 } n and an LBA A = ( Q,A,δ,q 0 , q acc , # l , # r ) with states Q = { q 0 , q 1 , . . . } , tape alphabet A = { a 0 , a 1 , . . . } ⊇ { 0 , 1 , # l , # r } , transition function δ ⊆ Q × A → Q × A ×{ L, R } (where L and R denote moving to the left resp. right), initial state q 0 , accepting state q acc, and bound markers # l , # r .

We define two bijections N q : Q →{ 0 , . . . , | Q | -1 } and N a : A →{ 0 , . . . , | A | -1 } to associate a natural number with each state and tape symbol such that q acc is associated with | Q | -1 .

We construct a DTCS with n + 4 state variables x 0 , . . . , x n +1 , x q , x p , n + 4 control actions u 0 , . . . , u n +1 , u q , u p , control domain U = Z n +4 (and control period τ = 1 for continuous systems). The environment dynamics f are state-independent, i.e., f ( x, u ) = u .

The decision tree is more complex, and we first establish some intuition. The state variables encode the LBA configuration, where the first n +2 dimensions ( x 0 , . . . , x n +1 ) encode the tape contents (i.e., the value of x i encodes the tape symbol at position i ), x q encodes the current LBA state, and x p encodes the current position on the tape. The execution of one control cycle corresponds to taking one transition. For that we construct the tree from four gadgets: a gadget to select the current LBA state, a gadget to select the current position on the tape, a gadget to select the current tape symbol, and a gadget to apply the transition function to the current configuration.

The first gadget (LBA state selection) is a degenerated tree only expanding to the right with | Q | -1 decision nodes. On the k -th level the predicate is x q ≤ k -1 . The left leaf node at level k +1 is the sub-tree T k -1 corresponding to the second gadget. On the last layer, the right leaf node is T | Q |-1 .

x

q

0

T

0

x

q

1

T

1

T

2

x

p

0

T

k,

0

x

p

1

T

k,

1

T

k,

2

(b) Gadget 2:

Sub-tree

T

k

with three tape positions.

x

ℓ

0

T

k,ℓ,

0

x

ℓ

1

T

k,ℓ,

1

x

ℓ

2

T

k,ℓ,

2

T

k,ℓ,

3

(c) Gadget 3: Sub-tree T k,ℓ with four tape symbols.

The second gadget (position selection) is an analogous tree T k over x p with n +1 decision nodes. On the ℓ -th level the predicate is x p ≤ ℓ -1 . The left leaf node at level ℓ +1 is the sub-tree T k,ℓ -1 corresponding to the third gadget. On the last layer, the right leaf node is T k,n +1 .

The third gadget (symbol selection) is an analogous tree T k,ℓ over x ℓ with | A | -1 decision nodes. On the m -th level the predicate is x ℓ ≤ m -1 . The left leaf node at level m +1 is the sub-tree T k,ℓ,m -1 corresponding to the fourth gadget. On the last layer, the right leaf node is T k,ℓ, | A |-1 .

The fourth gadget (application of the transition function) is a single leave node T k,ℓ,m . The indices encode that the LBA is currently in state N -1 q ( k ) at position ℓ and reads tape symbol N -1 a ( m ) . Let us assume that the transition function δ prescribes to switch to state q ′ , write tape symbol a ′ , and move to the right. Correspondingly, we want to choose the control actions such that in the next step the state of the system is x q = N q ( q ′ ) , x p = ℓ +1 , x ℓ = N a ( a ′ ) , and all other state variables remain unchanged. This is achieved by choosing

̸

(For moving to the left, choose u p = -1 instead.) Observe that the state variables keep track of the LBA configuration and that we have corresponding leaves in T for each transition in δ . For a fixed transition ( q, a, q ′ , a ′ , d ) we actually have multiple leaves; they only differ in the choice of the control action u ℓ (corresponding to the current cell index on the tape).

Finally, let w i be the i -th symbol of w (starting with i = 1 ). The initial state is x q = N q ( q 0 ) , x p = 1 (first symbol of w ), x 0 = N a (# l ) , x n +1 = N a (# r ) , and x i = N a ( w i ) for all other i . The set of goal states G consists of all states where x q = N q ( q acc ) = | Q | -1 . The set of error states E = ∅ is not needed. The step bound is k max = | Q || A | n +2 ( n +2) (encoded in binary).

The above construction is polynomial in the size of the LBA and the word (the decision tree has O ( | Q | + | A | + n ) nodes), and the following equivalence shows that it is a reduction.

w is accepted by A

⇐⇒ the configuration ⟨ q 0 , # l w # r ⟩ leads to a configuration with state q acc

( ∗ ) ⇐⇒ the DTCS reaches a state with x q = N q ( q acc ) within k max steps

⇐⇒ the DTCS satisfies the reach-avoid specification

The critical step is ( ∗ ) , for which we rely on the intuition established above. Note that in the continuous-time case we have to make sure that a goal state is not reached accidentally between two control cycles. This is ensured by letting N q ( q acc ) = | Q | -1 .

This concludes the proof of PSPACE -completeness. Before we continue with the undecidability in unbounded time, we give an example to better illustrate the previous reduction.

Example 4. Consider the LBA with states q 0 , q s , q acc, tape symbols 0 , 1 , # l , # r , and the only explicit transition δ ( q 0 , 1) = ( q acc , 0 , R ) (all other transitions lead to the sink state q s with appropriate direction L/R ). The LBA accepts words starting with a 1 (and changes it to 0 ). We use the mappings

Now consider the input word w = 1 , which means we have the initial configuration ⟨ q 0 , # l 1# r ⟩ . We construct a state-independent DTCS with state variables x = ( x 0 , x 1 , x 2 , x q , x p ) and control actions

The decision tree T is quite large ( 36 leaves), and we only depict its structure in Fig. 10. The first cartoon shows the outer structure. Each leaf node follows the structure in the second cartoon, and each leaf node shown there follows the structure in the third cartoon. Finally, we give one example leaf T 0 , 1 , 1 of the whole tree, which stands for state q 0 , tape position 1 , and reading symbol 1 . Observe that this is the leaf the decision tree chooses in the initial state. The control-action vector in that leaf is ( 0 , 0 -1 , 0 , 2 -0 , 1 ) T . Thus, in the next time step, the new state will be

︸︷︷︸

cell

0

︸ ︷︷ ︸

cell

1

︸︷︷︸

cell

2

︸ ︷︷ ︸

state

︸︷︷︸

position

(2 , 0 , 3 , 2 , 2 ) T , which is a goal state. This concludes the example.

︸ ︷︷ ︸

l

#

0#

r

q

︸︷︷︸

acc position

Now we prove undecidability in unbounded time. For that, we reduce from the termination problem for deterministic two-counter machines (2CMs). We consider the model from [Shepherdson and Sturgis, 1963, Theorem 4.1] with two counters c 1 , c 2 ∈ N and instruction set ■◆❈ ( c z ) (increment counter c z ), ❉❊❈ ( c z ) (decrement counter c z with zero lower bound), ❏❩ ( c z , ℓ ) for z ∈ { 1 , 2 } (jump to location ℓ if counter c z is zero), and ❙❚❖/a80 (terminate). Each instruction I comes with a position k in the program, which we write ' k : I '. The initial value of the counters is c 1 = c 2 = 0 and the program starts at instruction 1 .

Given is a 2CM with n instructions. We construct a DTCS with 3 state variables x 0 , x 1 , x pc , 3 control actions u 1 , u 2 , u pc , control domain U = Z 3 (and control period τ = 1 for continuous systems). The environment dynamics f are state-independent, i.e., f ( x, u ) = u .

We first establish some intuition. The state variables x 1 and x 2 correspond to the counters c 1 and c 2 , and the state variable x pc corresponds to the program counter. After each control cycle, these variables will have the corresponding values in the simulated 2CM. For the decision-tree policy, we construct a gadget for each instruction in the 2CM, and one more gadget to select the current instruction based on the value of x pc .

We start with the latter gadget (instruction selection), which works precisely as most gadgets in the previous proof. It is a degenerated tree only expanding to the right with n -1 decision nodes. On the k -th level, the predicate is x pc ≤ k . The left leaf node at level k +1 is the sub-tree T k corresponding to one of the other gadgets, depending on the k -th instruction. The last layer's right leaf node is T n .

The other gadgets are trees substituted for T k depending on the k -th instruction. We only describe the instructions for the counter c 1 ; for c 2 the construction is completely analogous by swapping 1 and 2 in the indices. For instruction ' k : ■◆❈ ( c 1 ) ' we set T k to the leaf node with u 1 = 1 , u 2 = 0 , u pc = 1 . For instruction ' k : ❉❊❈ ( c 1 ) ' we set T k to the tree with decision node x 1 ≤ 0 and two leaves; the left leaf node is u 1 = 0 , u 2 = 0 , u pc = 1 , and the right leaf node is u 1 = -1 , u 2 = 0 , u pc = 1 . For instruction ' k : ❏❩ ( c 1 , ℓ ) ' we set T k to the tree with decision node x 1 ≤ 0 and two leaves; the left leaf node is u 1 = 0 , u 2 = 0 , u pc = ℓ -k , and the right leaf node is u 1 = 0 , u 2 = 0 , u pc = 1 . For instruction ' k : ❙❚❖/a80 ' we set T k to the leaf node with u 1 = 0 , u 2 = 0 , u pc = -k .

Finally we define the (singleton) set of initial states X 0 = { (0 , 0 , 1) } , the goal region G = { ( x, y, 0) | x, y ∈ R } , and the set of error states E = ∅ (not used).

It is easy to see that each of the gadgets implements the 2CM instructions. Executing the action corresponding to the ❙❚❖/a80 instruction is the only way that x pc can reach the value 0 . Thus the above construction is a reduction.

the 2CM terminates

⇐⇒ the ❙❚❖/a80 instruction is reached from the configuration c 1 = c 2 = 0 , pc = 1

⇐⇒ the DTCS reaches a state with x pc = 0

⇐⇒ the DTCS satisfies the reach-avoid specification

︸︷︷︸

D Modeling Decision-Tree Controlled Systems as Hybrid Automata

Here we expand on the discussion from Section 4.3 and explain the encoding.

Systems with mixed continuous and discrete dynamics can be modeled as a hybrid automaton [Henzinger, 1996]. As such, DTCS can also be encoded as a hybrid automaton. The reach-avoid problem has been studied extensively for various subclasses of hybrid automata [Doyen et al., 2018, Althoff et al., 2021]. One could thus argue that solutions to the reach-avoid problem for DTCS exist. However, when modeling a DTCS as a hybrid automaton, the problem structure gets lost, and thus tools analyzing this hybrid automaton will not scale. Hence we argue that our algorithm is the first feasible solution to the reach-avoid problem for DTCS.

Below we present one possible encoding of a DTCS as a hybrid automaton, for which we assume that the reader is familiar with the terminology of hybrid automata. We encoded the systems from our evaluation (Section 4) accordingly and applied the state-of-the-art reachability tool JuliaReach [Bogomolov et al., 2019] (which also implements an algorithm based on Taylor models [Benet et al., 2019]) to them. The result was that the tool got stuck as soon as it had to bisect the set of states for the first time, and we had to terminate the tool after several minutes of no progress.

We encode a DTCS ( f, T , τ ) as a hybrid automaton as follows. First we introduce a fresh variable t for time. We use one location ℓ u for each control action u ∈ U that occurs in T . The continuous dynamics in ℓ u are given by f ( x, v ) ∧ ˙ t = 1 , where we substitute the value u for the second argument v . Each location ℓ u has an invariant t ≤ τ , restricting time to one control period. For each location there is a transition to every location ℓ u ′ with a guard condition consisting of t = τ and a big disjunction of conjunctions of the predicates along the paths in T to leaves with action u ′ . For instance, for T given in Figure 7(a), the guards on the transitions leading to ℓ -1 are t = τ ∧ ( θ ≤ -0 . 014 ∨ ( θ > -0 . 014 ∧ ω ≤ -0 . 201)) . Finally, each transition has a reset t := 0 .

Such an automaton is challenging for black-box analysis tools because they always have to explore all transitions. For instance, when we evaluated the reachability tool on the cart-pole system, the tool can analyze the first four control periods (out of 25) in 22 seconds, but then gets stuck in the fifth control period when the bisection starts, and prints errors about numerical instability. By knowing the problem structure, our approach only explores the feasible paths of the decision tree (often there is just one), for which it only has to evaluate d predicates, where d is the depth of the tree. Furthermore, given the periodic control policy, we only have to perform these checks at specific time points. For general hybrid automata, this has to be found out via intersections with invariants and guards. These structural insights make our analysis not only more efficient but also more precise in practice because general hybrid-automata tools would typically approximate these operations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel verification algorithm that enables the verification of decision tree policies for continuous-time systems (also applicable to discrete-time systems). This approach ensures a sound and compact representation of reachable sets using Taylor models, which can be efficiently propagated through non-linear dynamics systems. The algorithm leverages the decision tree structure to propagate a set-based approximation of abstract reachable states through decision nodes, allowing for splits at decision boundaries. The effectiveness of this approach is demonstrated through the verification of safety for several decision trees that imitate neural-network policies in classic low-dimensional nonlinear control systems (e.g. cartpole). This method can provide robust safety and reachability guarantees for all system behaviors in these benchmarks.\n\nSTRENGTHS:\n+ The proposed algorithm marks a significant advancement as the first formal verification method for verifying reach-avoid properties in a decision-tree controlled system (DTCS) with continuous-time dynamics, including nonlinear systems. \n+ It achieves this by efficiently propagating a set of states through the system dynamics and tailoring general hybrid system reachability verification specifically for decision trees. The algorithm effectively utilizes axis-aligned predicates and interval-based set approximations to enhance the precision of abstract states, mitigating the wrapping effect. \n+ Moreover, the algorithm showcases its capability to generalize to unbounded time horizons by conducting reachable states fixpoint calculations, providing the potential for inductive safety verification over infinite time horizons. \n+ The theoretical analysis conducted in this paper provides insights into the main algorithm. \n\nThe paper introduces a tool for ensuring safety and reachability guarantees in decision-tree controlled systems, pushing the boundaries of verification techniques for learning-enabled systems in the context of continuous-time dynamics.\n\nWEAKNESSES:\n+ The proposed method consists of two parameterized procedures: one for analyzing dynamical systems and another for decision tree policies. The verification algorithm for dynamical systems is well-established in the related literature. To enhance the clarity of the paper, it would be beneficial to treat the verification procedure as background information in a dedicated section, allowing the paper to primarily focus on its contribution to the analysis of decision tree policy reachability. \n\n+ The evaluation presented in the paper falls short in adequately showcasing the limitations of the verification algorithm in practical settings. To provide a more comprehensive analysis, it would be valuable to explore scenarios where the reachability analysis may potentially diverge. For instance, investigating the scalability of the reachability analyzer to deep decision trees with a significant number of nodes would be helpful. Additionally, conducting an ablation study on decision trees with various shapes may provide insights into the algorithm's behavior.\n\n+ Furthermore, the benchmarks used in the evaluation appear to be simple, and it seems that a linear controller could be trained directly using reinforcement learning for all of these benchmarks and verified using reachability analysis. To strengthen the evaluation, it would be beneficial to include a benchmark scenario where a linear controller fails, thus highlighting the necessity and added value of decision tree controllers that can be additionally verified. For example,  cartpole swinging up could be a suitable benchmark to demonstrate the necessity of decision tree controllers.\n\nQUESTIONS:\n+ Why does the cart-pole benchmark in the paper use a non-standard initial state set? Could the verification algorithm be applied to the cart-pole system using the initial state set provided in the Gym environment?\n\n+ Is the verification algorithm capable of scaling to slightly more complex benchmarks, such as the pendulum and cart-pole swinging up tasks? Can it provide proof for the safety and reachability properties of these systems over an infinite time horizon?\n\nLIMITATIONS:\nThe paper lacks a specific discussion on the scalability of the approach in terms of the size and complexity of control systems and decision trees that can be effectively verified. It would be helpful to address this aspect in the final version of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach for verifying safety (reach-avoid) properties of controlled systems where  the state space of the system is continuous, its dynamics are continuous-time, and the policy/controller is described by a decision tree that chooses actions from a continuous action space. The proposed reachability-based verification algorithm invokes two main sub-procedures $post_f$ and $post_{\\mathcal{T}}$. Given an initial set of states and an action, $post_f$ computes the reachable states induced by the dynamics using standard techniques based on Taylor models. On the other hand, $post_{\\mathcal{T}}$ uses the structure of a decision tree to compute an overapproximation of all possible state-action pairs induced by the policy given an initial set of states. The design of $post_{\\mathcal{T}}$ as well as the overall verification algorithm are the primary contributions of this work. The paper also includes theoretical results establishing the soundness and the relative completeness of the verification algorithm, as well as results establishing the computational hardness of the safety verification problem for systems with decision tree controllers. Moreover, empirical evaluation shows the scalability of the verification algorithm compared to other, more generic techniques.\n\nSTRENGTHS:\nThe paper presents an elegant verification algorithm as well as establishes hardness results for an interesting, well-specified family of controlled systems. The overall verification algorithm is clean and generic, and provides precise specifications that need to be satisfied by the sub-procedures $post_f$ and $post_{\\mathcal{T}}$ for the overall algorithm to be a sound and complete verifier. The key idea for computing reachability through the decision tree policy---namely, using box abstractions for efficient calculation of inclusion checks and bisection operation---is sensible and intuitive, and suggests clear directions for future research. The paper also presents useful insights on extending the analysis to unbounded time settings. The empirical evaluation supports the claim that designing a verification procedure that exploits the structure of Decision Tree Controlled Systems (DTCS) can lead to efficiency wins. I also greatly appreciate the quality of the writing and the formal rigor of the presented ideas (barring a few concerns described below).\n\nWEAKNESSES:\nMy primary concerns have to do with the presentation of the technical material and the comparison with related work.\n\n1. The paper repeatedly suggests that exploiting the special structure of DTCS for verification is one of the key insights of the presented work but a clear explanation of this special structure and how it is exploited is only cursorily explained in Section 4.3 (with a more detailed explanation in the appendix). I am not an expert on the topic and for readers like me, it would help if these structural insights were clearly explained in the initial sections of the paper.\n\n2. I find the presented ideas about generalizing to unbounded time via fixpoints very interesting. However, the text in Section 3.2 seemed handwavy, and I would very much appreciate a more precise explanation of the ideas. For instance, I found the comments on line 226 (\"However, due to the discrete ...\") and on line 230 quite cryptic. I am also confused about the fact that while $post_f$ takes both a set of states and a time interval as input parameters, the fixed point check only considers the input states (i.e., assumes time-invariant dynamics). If the dynamics is time-invariant, why does $post_f$ need $t_0$ and $t_1$ as parameters? Can $t_0$ and $t_1$ be fixed as 0 and $\\tau$?\n\n3. The evaluation would be much stronger if there were a head-to-head comparison of the algorithm in this paper with reachability tools for hybrid automata. Section 4.3 briefly touches upon this, but a direct comparison for all four examples would be quite instructive. Another minor comment is that given the amount of detail in Figures 4,5, and 6, it would help to make them higher resolution.\n\nQUESTIONS:\nMy questions are related to comments above in the Weaknesses section.\n\n1. What structure of decision trees does the algorithm exploit? Is the structure primarily used in the design of Algorithm 3, or does Algorithm 1 also exploit the structure?\n\n2. Repeating the second comment above, the fixed point check seems to assume time-invariant dynamics. Is this correct? If so, why pass $t_0$ and $t_1$ as parameters to $post_f$ instead of simply passing $\\tau$?\n\nLIMITATIONS:\nThe paper does not explicitly discuss the limitations and might benefit from a small discussion about the same. For instance, a discussion about the scalability of the approach and loss of precision due to approximations (related to the discussion in the last para of the conclusion) would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper puts forward a verification method for decision tree-based systems in
continuous time. The method implements a reachability algorithm that computes
over-approximations of the set of reachable states for a sequence of time
intervals until a time horizon is reached. The approximations at each step are
derived using Taylor models.

STRENGTHS:
While the contribution is based on standard reachability-based verification
methods for hybrid and neural network-driven systems, the development of said
methods to verify decision trees is novel. 

Good experimental evaluation which shows improvements over the state-of-the-art
in reachability for hybrid automata. This can in my view further improved  by
including the following:
    1. a discussion of the scalability of the method with respect to a wider
    range of model sizes.
    2. a discussion of the comparative amenability to reachability-based
    verification of decision tree-based and neural network-based systems.

WEAKNESSES:
The paper does not include a discussion on the scalability of the proposed
method.

QUESTIONS:
I have no questions. I thank the authors for the clarity of the text.

LIMITATIONS:
Adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to solve the reach-avoid problem for dynamical systems controlled by a decision tree in continuous time. The authors assert that this is the first paper to solve the problem in the continuous time setting. In the paper, the authors first provide a good overview of decision trees, the use of decision trees as neural network surrogates, and verification of decision tree controlled systems (DTCS). They also explain why tools developed for verification of neural network controlled systems are not directly applicable to DTCS. The authors then explain their approach, which is based on a typical reachability algorithm and determines the set of reachable states for an iteration of the control loop.\n\nSTRENGTHS:\n1. The problem of DTCS verification is interesting and well motivated.\n2. The paper is clearly and concisely written, and the justification of the paper as described is sound. The authors describe in great detail how their algorithms satisfy the equations necessary for the reach-avoid problem in DTCS. The authors also provide evaluations of their approach using multiple benchmark problems.\n\nWEAKNESSES:\n1. The novelty of the approach is not adequately explained. I believe that it is novel, but the authors need to explicitly state which parts of the algorithm are specifically designed for use with DTCS and which are just adapted from the typical reachability algorithm. \n2. Section 3.2, which describes the generalization of the solution to unbounded time problems, could perhaps use more details.\n3. The authors should also include some statement on how common axis-aligned predicates may be found in real-world DTCS. Is this a reasonable assumption?\n\nQUESTIONS:\nPlease see the comments above on weakness.\n\nLIMITATIONS:\nThe paper could use more discussion on its limitations, particularly regarding scalability. The work does not have obvious negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}