SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3 Post for the decision tree
Input: set  $\mathcal{X}$ , decision tree  $\mathcal{T}$   
Output: set of pairs  $(\mathcal{X}_u, u)$   
1:  $S \gets \{\}$   
2:  $Q \gets \{(\mathcal{X}, \text{root}(\mathcal{T}))\}$   
3: while  $\neg$  isempty(Q) do  
4:  $(\mathcal{Y}, \mathcal{T}_{\mathcal{Y}}) \gets \text{pop}(Q)$   
5: if isleaf( $\mathcal{T}_{\mathcal{Y}}$ ) then  
6: push(S,  $(\mathcal{Y}, \ell(\mathcal{T}_{\mathcal{Y}})))$   
7: else  
8:  $P \gets \ell(\mathcal{T}_{\mathcal{Y}})$   
9: if  $\mathcal{Y} \subseteq P$  then  
10: push(Q,  $(\mathcal{Y}, \text{left}(\mathcal{T}_{\mathcal{Y}})))$   
11: else if  $\mathcal{Y} \cap P = \emptyset$  then  
12: push(Q,  $(\mathcal{Y}, \text{right}(\mathcal{T}_{\mathcal{Y}})))$   
13: else  
14:  $(\mathcal{Z}_1, \mathcal{Z}_2) \gets \text{bisect}(\mathcal{Y}, P)$   
15: push(Q,  $(\mathcal{Z}_1, \text{left}(\mathcal{T}_{\mathcal{Y}})))$   
16: push(Q,  $(\mathcal{Z}_2, \text{right}(\mathcal{T}_{\mathcal{Y}})))$   
17: end if  
18: end if  
19: end while  
20: return S
We have  $\mathcal{X} \cap P = \emptyset$  if and only if  $\operatorname{box}(\mathcal{X}) \cap P = \emptyset$ . If both conditions fail,  $\mathcal{X}$  is bisected into  $\mathcal{Z}_1 = \{x \mid x \in \mathcal{X} \cap P\}$  and  $\mathcal{Z}_2 = \{x \mid x \in \mathcal{X} \cap P^C\}$  in line 14. Here  $P^C$  denotes the complement of  $P$ , which is also a half-space. Since the above sets are hard to compute, we replace  $\mathcal{X}$  by  $\operatorname{box}(\mathcal{X})$  in the implementation. Note that  $\operatorname{box}(\mathcal{X}) \cap P \supseteq \operatorname{box}(\mathcal{X} \cap P)$  in our setting. We discuss this further in Appendix A.1.
Proposition 2. Algorithm 3 implements procedure post $_{\mathcal{T}}$  satisfying Eq. (4). Furthermore, if all bisections are exact, Eq. (4) is satisfied with equality.
Further details relevant for using our method in practice are given in Appendix A.
Example 3. We walk through Algorithm 1 for the first control cycle, illustrated in Figure 3. Figure 3(a) shows how Algorithm 2 computes the pair  $(\mathcal{V},\mathcal{Z})$ , consisting of an enclosure  $\overline{\mathcal{R}}_{[0,\tau]}$  of the reachable states up to time point  $\tau$  and an enclosure  $\overline{\mathcal{R}}_{\tau}$  of the reachable states at the last time point. Figure 3(b) shows how Algorithm 3 bisects the set  $\mathcal{Z}$  along the predicate  $x \leq 3$  into sets  $Z_1$  and  $Z_2$ . We furthermore illustrate the approximation with interval enclosures. Assuming this is the only predicate of the decision tree, Algorithm 1 would continue the analysis from these two sets in the next iteration.
3.2 Generalization to Unbounded Time via Fixpoints
Next we discuss how to employ a set-based fixpoint check; for details we refer to the literature [Giacobbe, 2019]. Roughly speaking, if we do not encounter new states after an iteration, we have found a fixpoint. Algorithm 1 iteratively adds states to  $\overline{\mathcal{R}}$  to enclose the sequence  $\mathcal{R}_{[k\tau,(k+1)\tau]}$ ,  $k < k_{\max}$  (Section 2.2). Let us view the exploration of the elements  $(\mathcal{X},t)$  in the queue  $Q$  as a search tree. First, the search in a node can be terminated if the set  $\mathcal{X}$  is contained in the union of the sets in the other nodes (i.e., if a fixpoint has been found). Second, if this condition holds in all branches, we have computed the reachable states in unbounded time. However, due to the discrete nature of
the policy, the time points at which to check for fixpoints are relevant. A sufficient condition is to only compare the sets at the beginning of each control cycle, i.e.,  $\mathcal{R}_{k\tau} \subseteq \bigcup_{j < k} \mathcal{R}_{j\tau}$  for some  $k > 0$ . However, when dealing with enclosures, this condition would become  $\overline{\mathcal{R}}_{k\tau} \subseteq \bigcup_{j < k} \overline{\mathcal{R}}_{j\tau}$ , which is not sufficient. The set  $\overline{\mathcal{R}}_{k\tau}$  on the left-hand side must additionally contain the reachable states from all sets  $\overline{\mathcal{R}}_{j\tau}$  on the right-hand side. Fortunately, our algorithm satisfies this additional condition.
In our implementation, all sets are (unions of) Taylor models, for which inclusion checking is hard. While we can overapproximate the left-hand side, we would have to underapproximate the right-hand side, which is difficult for Taylor models. However, whenever we bisect sets in  $post_{\mathcal{T}}$ , we obtain box-shaped sets, for which inclusion checks are easy. Thus, in our implementation, we only perform a fixpoint check after a bisection. We shall see in the evaluation that this is often sufficient in practice.
3.3 State-Independent Dynamics and Computational Complexity
We call a control system state-independent if  $f(x, u) = u$  for all  $x$  (for which  $u$  must be  $n$ -dimensional). This is arguably the simplest possible control system. If the system dynamics are state-independent and  $\mathcal{X}_0$  is a polyhedron (i.e., the intersection of linear inequalities),  $post_f$  can be implemented exactly and yields a polyhedron [Alur et al., 1996]. Given a polyhedron,  $post_{\mathcal{T}}$  returns a union of polyhedra. Under these assumptions, Algorithm 1 can be implemented to compute the exact reachable states and thus satisfy Theorem 1 with equality. Finally, Algorithm 1 can be turned into a decision procedure for the reach-avoid problem (namely: also output  $\mathcal{R}_{T_{\max}}$  and check Eq. (1)). Nevertheless, state-independent DTCS can encode Turing machines. Hence the reach-avoid problem is undecidable in unbounded time and PSPACE-complete in bounded time (proven in Appendix C.2).
Theorem 2. The reach-avoid problem for both continuous-time and discrete-time DTCS over rational numbers with state-independent dynamics and polyhedral initial, goal, and error sets  $(\mathcal{X}_0,\mathcal{G},\mathcal{E})$  is PSPACE-complete, and undecidable when considering the unbounded-time version.
Note that Algorithm 1 solves not only the decision problem but computes the full reachable states. Thus it needs to collect all the sets resulting from bisections. If the number of iterations  $k_{\mathrm{max}}$  is given in binary, the algorithm is doubly exponential, even if we ignore a potential complexity growth in the set representation. We conjecture that this bound is optimal for the full computation. More precisely, in each iteration, a set may be bisected into several (but at most  $\ell$ , where  $\ell$  is the number of leaves of the decision tree) sets. With  $k_{\mathrm{max}}$  control cycles, the algorithm may end up with  $\ell^{k_{\mathrm{max}}}$  sets in the end.
We implemented our method in a tool based on JuliaReach [Bogomolov et al., 2019] for the Taylor-model algorithm (TM_reachin Algorithm 2) [Benet et al., 2019] and LazySets [Forets and Schilling, 2021] for the set computations. We demonstrate our approach on the quadrotor system (Figure 1(a)) and three classical control problems from the Gymnasium [Brockman et al., 2016]: cart/pole,acrobot, and mountain/car. In the following, we use our proof-of-concept implementation to verify and visually explain that, within the given time constraints, the quadrotor safely reaches the end of the corridor, the cart manages to stabilize the pole, theacrobot swings to the
Table 1: Sizes of the decision-tree policies, and corresponding verification times averaged over 10 runs.
System	Policy T	Verification time
nodes	depth	actions
Quadrotor	177	10	8	84 sec
Cart/Pole	5	2	2	15 sec
Acrobot	7	2	2	101 sec
9	3	2	113 sec
Mountain/Car	9	3	3	7 sec
goal height, and the car reaches the top of the mountain. It is crucial to maintain high precision during the analysis to avoid divergence. The systems illustrate different aspects: Our approach applies to time-dependent policies (quadrotor control), proves infinite-time stability (Section 3.2) (cart/pole), handles transformations (Appendix A.2) (acrobot), and deals with discrete-time dynamics (mountain/car), while preserving high precision.
The implementation and experiments are available at https://github.com/VeriXAI/Safety-Verification-of-Decision-Tree-Policies-in-Continuous-Time.
Figure 4: Reachable states (red) and simulations (colored) for the quadrotor system. The initial states are shown in yellow on the left, the goal region is shown in blue at the top, the reference plan is shown in brown, and the dashed blue lines mark the safety corridor.
We obtain decision-tree policies via neural-network distillation (see Table 1 on the depth and number of nodes and actions). For the quadrotor system we collected 500 samples of state-action pairs from a neural-network policy used by Ivanov et al. [2019] and learned a decision tree of depth 10 using behavioral cloning. For the three other systems (cart/pole,acrobot,mountain/car) we first used deep Q-learning to train a two-layer convolutional neural network [Mnih et al., 2013] and then adopted the Viper algorithm [Bastani et al., 2018] to imitation-learn decision trees for these systems. We manually prune the resulting trees of redundant nodes (as Viper produces balanced trees).
All experiments were conducted on a laptop with an i7 1.80 GHz CPU and 32 GB RAM.
4.1 Evaluation on a Decision Tree for Quadrotor Control
We evaluate our verification approach on a decision tree controlling a six-dimensional quadrotor, tasked to follow a piecewise-linear plan (see, Example 1). The actions represent possible combinations of pitch, roll, and thrust acceleration. This is a complex continuous-time system (see Appendix B.1 for full details). The policies for such high-dimensional systems are typically approximated by a learned model, e.g., a neural network [Royo et al., 2019]. We train a decision-tree policy of depth 10 imitating the neural network from [Ivanov et al., 2019].
Figure 4 shows the set of reachable states. Ivanov et al. [2019] verified the neural-network policy, for which they had to split the initial set into 16 subsets to tame the approximation error. Computing the reachable states took between 10 and 59 minutes for each subset. Our method can be applied to the full initial set directly and verifies the system in 6.5 minutes.
4.2 Evaluation on Decision Trees for Classic Nonlinear Control Problems
Next we study three classical control systems, for which we trained small, interpretable decision trees (see Table 1). Small decision trees are often sufficient for optimality [Vos and Verwer, 2023].
Cart-Pole System. We consider the cart-pole system [Barto et al., 1983]. A description is given in Appendix B.2. The goal is for the pole to remain vertically stable within an angle of  $\pm 0.06^{\circ}$ . This system is challenging because of quick alternations in the control action. In Figure 5(a), we show the reachable states in the  $\theta/\omega$  projection together with the decision boundaries (e.g., the policy moves the cart to the left in the green region). The analysis terminates in 40 seconds and proves that the blue dashed line  $\theta = 0.06$  is not exceeded. Furthermore, the fixpoint check allows to generalize the results to infinite time; thus we can conclude that the policy is able to balance the pole forever.
Figure 5: Reachable states (red) and simulations (black) for the cart-pole and mountain-car systems.
Acrobot. We consider the acrobot system [Sutton, 1995], which consists of two links connected by a joint, one of which can swing freely. A full description is given in Appendix B.3. The goal is that the free end reaches a desired height, expressed as the condition  $-\cos(\theta_1) - \cos(\theta_2 + \theta_1) > 1$ , where  $\theta_1$  is the joint angle and  $\theta_2$  is relative to the angle of the first link. This system has complex nonlinear dynamics that require high precision. We obtain two decision-tree policies and compare their performance. The system applies a nonlinear transformation to the state before passing it to the policy. We discuss handling transformations in Appendix A.2. Figure 6 illustrates that all trajectories reach the goal regions. The analysis takes 100 (first policy) resp. 112 seconds (second policy).
Mountain-Car System. To demonstrate applicability to discrete-time systems, we consider the mountain-car system, where a car has to reach from one mountain to another [Moore, 1990]. A description is given in Appendix B.4. Figure 5(b) shows that the car always reaches the top of the mountain within at most 132 steps. The analysis terminates in 16 seconds.
4.3 Comparison with State-of-the-Art Reachability Tool for Hybrid Automata
As explained before, a DTCS can be seen as a special case of a hybrid automaton. We encoded the various DTCS from the evaluation as hybrid automata (a description of the encoding is given in Appendix D) and then applied the state-of-the-art reachability tool JuliaReach [Bogomolov et al., 2019] to them. The result was that the tool got stuck as soon as it had to bisect the set of states for the first time, and we had to terminate the tool after several minutes of no progress. For instance, when we evaluated the reachability tool on the cart-pole system, the tool can analyze the first four control periods (out of 25) in 22 seconds, but then gets stuck in the fifth control period when the bisection
starts, and prints errors about numerical instability. This demonstrates that our algorithm is the first feasible solution to the reach-avoid problem for DTCS.
Our algorithm exploits the decision-tree structure to only explore the feasible automaton transitions (most of the time this is just one). Furthermore, given the periodic control policy, we only have to check the transitions at specific time points (an observation we already made in earlier work [Forets et al., 2020]). Moreover, since we focus on axis-aligned predicates, interval enclosures are often sufficient. Finally, often only one branch of the decision tree is relevant, which allows us to ignore the complement branch and avoid unnecessary calculations. Even if several leaves of the decision tree are reached, these may still all be annotated with the same control action. In that case we do not have to bisect the set (which avoids the main source of approximation error).
The other tool views the automaton as a black-box model and thus cannot make use of this structure; instead it needs to perform many intersection operations, which are expensive and typically force to use a more complex set representation. These structural insights make our analysis not only more efficient but also more precise in practice because general hybrid-automata tools would typically approximate these operations.
In this paper, we studied the reach-avoid problem for continuous-time and discrete-time dynamical systems controlled by a decision tree. The problem is undecidable for nonlinear systems, and we showed undecidability even for the simplest dynamics as well as PSPACE-completeness in the bounded-time setting. We proposed the first practical algorithm to solve this problem in continuous time. The abstract algorithm is sound and, for simple systems, complete. We implemented the algorithm for nonlinear systems and decision trees with axis-aligned predicates. Our evaluation shows that the algorithm is precise and performant on typical problems. Our approach enriches the verification toolset for machine-learning based systems and opens novel cross-community research challenges. Our approach lends itself to visualization and can serve for further analysis and refinement of decision trees, which themselves are interpretable surrogates for black-box policies.
Coming to the limitations of our approach, we have only considered time-invariant systems. While the algorithms are general, the fixpoint check we used does not apply to time-varying systems. In our experiments, with the exception of the quadrotor, we focused on small decision trees. We note that the size of the decision tree is not necessarily a good measure for complexity. Multiple leaves may share the same action, and some leaves may not be used at all during the execution. The main impact on the verification method, in our experience, is the number of times the decision boundaries are partially crossed by the reachable states (after a time step). Furthermore, the control systems we verified in the evaluation, while nontrivial from a verification perspective, have relatively simple control tasks where a decision tree is not required. We plan to investigate how our method performs on systems with more challenging control tasks, e.g., the cart/pole system starting with the pole hanging downward (which we were not able to verify in a preliminary attempt).
In future work we will also study how the analysis can be improved both in terms of precision and scalability. For precision, we plan to employ set representations that are closed under bisection, e.g., constrained polynomial zonotopes [Kochdumper and Althoff, 2023], which generalize Taylor models. For scalability, we aim to improve the fixpoint check using simulation-based heuristics. Another natural direction is to apply the approach to learning a safe decision-tree policy, for which methods to compute underapproximations in order to refute unsafe models would be useful. Finally, our recent algorithm to synthesize a safety shield for policies of systems with complex hybrid dynamics [Brorholt et al., 2023] only detects discrete-time safety violations; hence we aim to find synergies with the method presented in this paper.
Acknowledgements. We are grateful for the anonymous reviewers' helpful comments to improve the paper and suggesting the swing-up cart/pole experiment as future challenge. Anna Lukina thanks Mustafa Mert Çelikok and Alexandru Băbeanu for their timely feedback. This research was partly supported by DIREC - Digital Research Centre Denmark under reference number 9142-0001B and the Villum Investigator Grant S4OS under reference number 37819.
K. Makino and M. Berz. Taylor models and other validated functional inclusion methods. Int. J. Pure Appl. Math, 4(4):379-456, 2003.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller. Playing Atari with deep reinforcement learning. CoRR, 2013. URL https://arxiv.org/abs/1312.5602.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Ried-miller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533, 2015. URL https://doi.org/10.1038/nature14236.
A. W. Moore. Efficient Memory-based Learning for Robot Control. PhD thesis, University of Cambridge, 1990.
A. Neumaier. The wrapping effect, ellipsoid arithmetic, stability and confidence regions, pages 175-190. Springer, 1993. URL https://doi.org/10.1007/978-3-7091-6918-6_14.
J. R. Quinlan. Learning decision tree classifiers. ACM Comput. Surv., 28(1):71-72, 1996. URL https://doi.org/10.1145/234313.234346.
V. R. Royo, D. Fridovich-Keil, S. L. Herbert, and C. J. Tomlin. A classification-based approach for approximate reachability. In ICRA, pages 7697-7704. IEEE, 2019. URL https://doi.org/10.1109/ICRA.2019.8793919.
W. J. Savitch. Relationships between nondeterministic and deterministic tape complexities. J. Comput. Syst. Sci., 4(2):177-192, 1970. URL https://doi.org/10.1016/S0022-0000(70)80006-X.
C. Schilling, M. Forets, and S. Guadalupe. Verification of neural-network control systems by integrating Taylor models and zonotopes. In AAAI, pages 8169-8177. AAAI Press, 2022. URL https://ojs.aai.org/index.php/AAAI/article/view/20790.
J. C. Shepherdson and H. E. Sturgis. Computability of recursive functions. J. ACM, 10(2):217-255, 1963. URL https://doi.org/10.1145/321160.321170.
R. S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. Mozer, and M. E. Hasselmo, editors, NeurIPS, pages 1038-1044. MIT Press, 1995. URL https://proceedings.neurips.cc/paper/1995/bitnet/8f1d43620bc6bb580df6e80b0dc05c48-AAbstract.html.
H. Tran, X. Yang, D. M. Lopez, P. Musau, L. V. Nguyen, W. Xiang, S. Bak, and T. T. Johnson. NNV: the neural network verification tool for deep neural networks and learning-enabled cyberphysical systems. In CAV, volume 12224 of LNCS, pages 3-17. Springer, 2020. URL https://doi.org/10.1007/978-3-030-53288-8_1.
C. Urban and A. Mine. A review of formal methods applied to machine learning. CoRR, 2021. URL https://arxiv.org/abs/2104.02466.
D. Vos and S. Verwer. Optimal decision tree policies for Markov decision processes. In *IJCAI*, pages 5457-5465. ijcai.org, 2023. URL https://doi.org/10.24963/ijcai.2023/606.
B. Xue, A. Easwaran, N. Cho, and M. Fränzle. Reach-avoid verification for nonlinear systems based on boundary analysis. IEEE Trans. Autom. Control., 62(7):3518-3523, 2017. URL https://doi.org/10.1109/TAC.2016.2615599.
A Additional Information on our Method
This section contains some technical remarks on the precision of our method as well as on how to deal with state transformations. The latter is relevant for solving one of the benchmark problems in the evaluation (Section 4).
A.1 Precision of the Algorithm
The precision of the Taylor-model implementation of  $post_{f}$  (Algorithm 2) is good in practice most of the time. The main approximation error in Algorithm 1 comes from the bisections in  $post_{\mathcal{T}}$  (Algorithm 3) and the subsequent approximation, as exemplified in Figure 3(b). This is hard to avoid with nonlinear dynamics, for which bisection of non-convex sets is challenging. With iterative algorithms like ours, the issue with approximation errors is that they accumulate over time, which is known as the wrapping effect [Neumaier, 1993], especially if there are frequent changes of the control actions (i.e., crossings of the decision boundary). To reduce the likelihood of a bisection and accumulation of errors, one can subdivide  $\mathcal{X}_0$  into smaller subsets and run an analysis from each of them so that either all or none of the states cross the decision boundary at the same step (for instance, as mentioned in Section 4.1, this idea was used by Ivanov et al. [2019] for the quadrotor model). This can sometimes be achieved more efficiently by only considering the set boundaries [Xue et al., 2017].
A.2 State Transformations Before Evaluating the Decision Tree
So far we have assumed that the decision tree  $\mathcal{T}$  takes the current state as input, as shown in Figure 2. However, some policies instead receive a transformation of the state. This is particularly useful for decision-tree policies with linear predicates (like ours) because a nonlinear transformation effectively augments them with nonlinear predicates. For instance, the acrobot model (see Appendix B.3 for more information) has the state vector  $(\theta_{1},\theta_{2},\dot{\theta}_{1},\dot{\theta}_{2})$ , but the input to  $\mathcal{T}$  is the vector  $(\cos (\theta_1),\sin (\theta_1),\cos (\theta_2),\sin (\theta_2),\dot{\theta}_1,\dot{\theta}_2)$ . To account for transformations, we can modify Algorithm 3 as follows. First we apply the transformation to the input set  $\mathcal{X}$ , which is easy if  $\mathcal{X}$  is given as a Taylor model. Then we apply Algorithm 3, and finally we have to "undo" the transformation for the output. We distinguish three cases. In the first case, if the output is a single pair  $(\mathcal{X}_u,u)$  (i.e., there was no bisection), we replace it with  $(\mathcal{X},u)$ . In all other cases, there was a bisection. If the transformation is invertible, as in the above case, we apply the inverse transformation (here:  $(\arccos (x_{1}),\arcsin (x_{2}),x_{5},x_{6})$ , where  $x$  is a state of the output set  $\mathcal{X}_u$ ) to each set  $\mathcal{X}_u$  in the output pairs  $(\mathcal{X}_u,u)$ . Otherwise, if the transformation is not invertible, we conservatively replace each pair  $(\mathcal{X}_u,u)$  with the pair  $(\mathcal{X},u)$ .
B Additional Information on the Benchmark Systems
Below we provide further information on the systems from the evaluation (Section 4).
B.1 Quadrotor System