POLAR: A Polynomial Arithmetic Framework for Verifying Neural-Network Controlled Systems

1 INTRODUCTION . Neural networks have been increasingly used as the central decision makers in a variety of tasks Mnih et al . ( 2015 ) ; Lillicrap et al . ( 2016 ) ; Pan et al . ( 2018 ) . However , the use of neural-network controllers also gives rise to new challenges on verifying the correctness of the resulting closed-loop control systems especially in safety-critical settings . In this paper , we consider the reachability verification problem of neural-network controlled systems ( NNCSs ) . The high-level architecture of a simple NNCS is shown in Figure 1 in which the neural network senses the system state , i.e . the value of x , at discrete time steps , and computes the corresponding control values u for updating the system dynamics which is defined by an ordinary differential equation ( ODE ) over x and u . The boundedtime reachability analysis problem of an NNCS is to compute an ( over-approximated ) reachable set that contains all the trajectories starting from the initial set for a finite number of control steps . Figure 2 shows an illustration of reachable sets for 4 steps , where the orange regions are the reachable set , and two red arrowed curves are two exemplary trajectories starting from the initial set X0 ( blue ) . Reachability analysis of general NNCSs is notoriously difficult due to nonlinearity in both the neural-network controller and the plant , further exacerbated by the coupling of the controller and the plant . Since exact reachability of general nonlinear systems is undecidable Alur & Dill ( 1994 ) , current approaches for reachability analysis of nonlinear dynamical systems aim at computing a tight overapproximation of the reachable sets Dreossi et al . ( 2016 ) ; Lygeros et al . ( 1999 ) ; Yang et al . ( 2016 ) ; Prajna & Jadbabaie ( 2004 ) ; Huang et al . ( 2017a ) ; Frehse et al . ( 2011 ) ; Chen et al . ( 2013 ) ; Althoff ( 2015 ) . Earlier works on NNCS verification drew on techniques for computing the output ranges of neural networks Huang et al . ( 2017b ) ; Katz et al . ( 2017 ) ; Dutta et al . ( 2018 ) ; Wang et al . ( 2018 ) ; Weng et al . ( 2018 ) ; Zhang et al . ( 2018 ) ; Singh et al . ( 2019 ) when integrating with the aforementioned reachability analysis . However , they have been shown to be ineffective for NNCS verification due to the lack of consideration on the interaction between the neural-network controller and the plant dynamics Dutta et al . ( 2019 ) ; Huang et al . ( 2019 ) . In particular , since their primary goal is to bound the output range of the neural network instead of approximating its input-output function , Figure 1 : A typical NNCS model . they can not be used to track dependency across the closed-loop system and across multiple time steps in reachability analysis . Direct end-to-end approximation Dutta et al . ( 2019 ) ; Huang et al . ( 2019 ) ; Fan et al . ( 2020 ) and layer-by-layer propagation Ivanov et al . ( 2019 ; 2021b ; a ) are the two main categories of approaches in the NNCS verification literature . The former directly computes a function approximation of the neural network and suffers from efficiency problems , i.e . they can not handle systems with more than a few dimensions due to the need to sample in the input space . The latter approach tries to exploit the neural network structure and uses Taylor model ( TM ) arithmetic to more efficiently obtain a function approximation by propagating the TMs layer by layer . A Taylor model ( p , I ) consists of a polynomial p for point-wise approximation , and an interval remainder I to bound the approximation error . However , due to the limitations of basic TM arithmetic , these approaches can not handle non-differentiable activation functions and suffer from rapid growth of the interval remainders during propagation which effectively degrades the analysis to an interval analysis . In this paper , we tackle the challenge of dependency tracking by constructing a function overapproximation , specifically a Taylor model ( TM ) approximation , of the neural-network controller using a unified polynomial arithmetic framework ( POLAR ) that enables precise layer-by-layer propagation of TMs for general feed-forward neural networks . POLAR addresses the key challenges of applying basic TM arithmetic through a novel use of univariate Bernstein polynomial interpolation to handle non-differentiable activation functions and symbolic remainders to taper the growth of the interval remainders . For the former , we leverage an efficient sampling-based analysis to provide a sound overapproximation . The latter avoids the so-called wrapping effect Jaulin et al . ( 2001 ) that can lead to rapid growth of the interval remainders in linear mappings . In summary , our work makes the following novel contributions . • We propose a novel polynomial arithmetic framework for bounded-time reachability analysis of NNCSs , keeping track of state-wise dependency across the closed-loop system . Compared to existing propagation-based approaches , our framework has the advantage of being able to handle NN controllers with general and heterogeneous activation functions . • We propose neuron-wise Bernstein polynomial interpolation and show that it can be seamlessly integrated with Taylor model approximations . In addition , we present the first application of symbolic remainders to tightening the overapproximation of neural network behaviors . • We conduct a comprehensive comparison of our approach with state-of-the-art techniques , including an evaluation on the full set of benchmarks published in these works , showing the overwhelming advantage of our proposed approach . 2 PRELIMINARIES . A Neural-Network Controlled System ( NNCS ) is a continuous plant governed by a neural network controller . The plant dynamics is defined by an ODE of the form ẋ = f ( x , u ) wherein the state variables and control inputs are denoted by the vectors x and u respectively . We assume that the function f is at least locally Lipschitz continuous such that its solution w.r.t . an initial state and constant control inputs is unique Meiss ( 2007 ) . We denote the input-output mapping of the neural network controller as κ , and the controller is triggered every δc time which is also called the control stepsize , then a system execution ( trajectory ) is produced in the following way . Starting from an initial state x ( 0 ) , the controller senses the system state at the beginning of every control step t=jδc for j=0 , 1 , . . . , and update the control inputs to vj=κ ( x ( jδc ) ) . The system ’ s dynamics in that control step is governed by the ODE ẋ=f ( x , vj ) . Given an initial state set X0 ⊂ Rn , all executions from a state in it can be formally defined by a flowmap function ϕN : X0 × R≥0 → Rn , such that the system state at any time t ≥ 0 from any initial state x0 ∈ X0 is ϕN ( x0 , t ) . We call a state x′ ∈ Rn reachable if there exists x0 ∈ X0 and t ≥ 0 such that x′ = ϕN ( x0 , t ) . The reachability problem on NNCS is to decide whether a state is reachable in a given NNCS , and it is undecidable due to their higher expressiveness than the two-counter machines on which the reachability problem is undecidable Alur & Dill ( 1994 ) . Many formal verification problems can be reduced to the reachability problem . For example , the safety verification problem can be reduced to the reachability problem to an unsafe state . In the paper , we focus on computing the reachable set for an NNCS over a bounded number K of control steps . Since flowmap ϕN often does not have a closed form due to the nonlinear ODEs , we seek to compute state-wise overapproximations for it over time segments , that is , in each control step [ ( j − 1 ) δc , jδc ] for j = 1 , . . . , K , the reachable set is overapproximated by a group of flowpipes F1 ( x0 , τ ) , . . . , FN ( x0 , τ ) over the N uniformly subdivided time segments of the time interval , such that Fi ( x0 , τ ) is a state-wise overapproximation of ϕN ( x0 , ( j − 1 ) δc + ( i− 1 ) δ + τ ) for τ ∈ [ 0 , δc/N ] , i.e. , Fj ( x0 , τ ) contains the exact reachable state from any initial state x0 in the i-th time segment of the j-th control step . Here , τ is the local time variable which is independent in each flowpipe . A high-level flowpipe construction algorithm is presented as follows , in which X̂0 = X0 and δ = δc/N which is called the flowpipe step or time step . 1 : for j = 1 to K do 2 : Computing the an overapproximation Ûj−1 for the control input range κ ( X̂j−1 ) ; 3 : Computing the flowpipes F1 ( x0 , τ ) , . . . , FN ( x0 , τ ) for the continuous dynamics ẋ = f ( x , u ) , u̇ = 0 from the initial set x ( 0 ) ∈ X̂j−1 , u ( 0 ) ∈ Ûj−1 ; 4 : R ← R∪ { ( F1 ( x0 , τ ) , . . . , FN ( x0 , τ ) } ; 5 : X̂j ← FN ( z , δ ) ; 6 : end for Notice that x ( 0 ) denotes the local initial set for the ODE used in the current control step , while the variables x0 in a flowpipe are the symbolic representation of an initial state in X0 . Intuitively , a flowpipe overapproximates not only the reachable set in a time step , but also the dependency from an initial state to its reachable state at a particular time . In the paper , we use Taylor models to represent the flowpipes . Taylor models are originally proposed to compute higher-order overapproximations for the ranges of continuous functions ( see Berz & Makino ( 1998 ) ) . They can be viewed as a higher-order extension of intervals Moore et al . ( 2009 ) which are sets of real numbers between lower and upper real bounds , e.g. , the interval [ a , b ] wherein a ≤ b represents the set of { x | a ≤ x ≤ b } . A Taylor model ( TM ) is a pair ( p , I ) wherein p is a polynomial of degree k over a finite group of variables x1 , . . . , xn ranging in an interval domain D ⊂ Rn , and I is the remainder interval . The range of a TM is the Minkowski sum of the range of its polynomial and the remainder interval . Thereby we sometimes intuitively denote a TM ( p , I ) by p+ I in the paper . TMs are often used as overapproximations for smooth functions . Given a smooth function f ( x ) with x ∈ D , its order k TM overapproximation , or simply TM , can be obtained as ( pf ( x ) , If ) such that pf is the order k Taylor approximation of f w.r.t . a point in D , and If is a remainder interval which ensures that ∀x ∈ D. ( f ( x ) ∈ pf ( x ) + If ) . Since pf is also over the same variables x , the overestimation in a TM can be estimated only on the width of If , i , e. , if the remainder is zero , then pf defines the same mapping as f . At this point , a TM serves more like an overapproximate function than just a range overapproximation such as intervals or polyhedra . TM arithmetic . TMs are closed under operations such as addition , multiplication , and integration ( see Makino & Berz ( 2003 ) ) . Given functions f , g that are overapproximated by TMs ( pf , If ) and ( pg , Ig ) respectively . A TM for f + g can be computed as ( pf + pg , If + Ig ) , and an order k TM for f · g can be computed as ( pf · pg − rk , If · B ( pg ) + B ( pf ) · Ig + If ·Ig + B ( rk ) ) wherein B ( p ) denotes an interval enclosure of the range of p , and the truncated part rk consists of the terms in pf · pg of degrees > k. Similar to reals and intervals , TMs can also be organized as vectors and matrices to overapproximate the functions whose ranges are multidimensional . For a vector of TMs ( ( p1 , I1 ) , . . . , ( pn , In ) ) T such that p1 , . . . , pn are over the same variables , we collectively denote it by ( p , I ) such that p is the polynomial vector ( p1 , . . . , pn ) T and I is interval vector ( I1 , . . . , In ) T . As an example , given the TMs ( 1− 0.5x2 , [ −0.1 , 0.1 ] ) and ( x+ 0.1x4 , [ −0.2 , 0.2 ] ) over the domain x ∈ [ −1 , 1 ] . The order 4 TM for the sum is ( 1 + x − 0.5x2 + 0.1x4 , [ −0.3 , 0.3 ] ) , and the order 4 TM for the product is ( x− x3 + x4 , [ −0.38 , 0.38 ] ) . Although the technique of Taylor model flowpipe construction can be used to generate state-wise overapproximate flowpipes for ODEs ( see Berz & Makino ( 1998 ) ; Chen ( 2015 ) ) , it is still a challenge that how to compute and represent the set Ûj−1 by a TM which overapproximates the dependency from an initial state inX0 to the actual control input rangeUj−1 = { κ ( ϕN ( x0 , ( j−1 ) δc ) ) |x0 ∈ X0 } at the beginning of that control step . We illustrate this dependency in Figure 3 . Our approach aims at solving this problem .

Summary: The authors attack a significant problem or bounded-time reachability analysis of neural-network controlled systems (NNCSs). The method allows for verification of decision-making procedures applied by trained neural networks to systems with the known continuous model (continuous plants).  The claimed main novelty of the paper is the application of the basic Taylor Model arithmetic in combination with the univariate Bernstein polynomial interpolation to handle non-differentiable NN activation functions.  The algorithm is tested in a set of benchmarks introduced in the other related work.   The main theoretical result states that the TM flow pipes computed using the POLAR approach are state-wise approximations for the reachable sets.