Title: Provably Safe Neural Network Controllers via Differential Dynamic Logic
Venue: NeurIPS 2024
Accepted: yes
Keywords: Cyber-Physical Systems

Neural Network Verification

Infinite-Time Horizon Safety

Differential Dynamic Logic

Safety Verification

neural network based control systems

Safety

Verification
Abstract: While neural networks (NNs) have a large potential as autonomous controllers for Cyber-Physical Systems, verifying the safety of neural network based control systems (NNCSs) poses significant challenges for the practical use of NNs— especially when safety is needed for unbounded time horizons. One reason for this is the intractability of analyzing NNs, ODEs and hybrid systems. To this end, we introduce VerSAILLE (Verifiably Safe AI via Logically Linked Envelopes): The first general approach that allows reusing control theory literature for NNCS verification. By joining forces, we can exploit the efficiency of NN verification tools while retaining the rigor of differential dynamic logic (dL). Based on a provably safe control envelope in dL, we derive a specification for the NN which is proven with NN verification tools. We show that a proof of the NN’s adherence to the specification is then mirrored by a dL proof on the infinite-time safety of the NNCS.

The NN verification properties resulting from hybrid systems typically contain nonlinear arithmetic over formulas with arbitrary logical structure while efficient NN verification tools merely support linear constraints. To overcome this divide, we present Mosaic: An efficient, sound and complete verification approach for polynomial real arithmetic properties on piece-wise linear NNs. Mosaic partitions complex NN verification queries into simple queries and lifts off-the-shelf linear constraint tools to the nonlinear setting in a completeness-preserving manner by combining approximation with exact reasoning for counterexample regions. In our evaluation we demonstrate the versatility of VerSAILLE and Mosaic: We prove infinite-time safety on the classical Vertical Airborne Collision Avoidance NNCS verification benchmark for some scenarios while (exhaustively) enumerating counterexample regions in unsafe scenarios. We also show that our approach significantly outperforms the State-of-the-Art tools in closed-loop NNV

Provably Safe Neural Network Controllers via Differential Dynamic Logic

Abstract

While neural networks (NNs) have a large potential as autonomous controllers for Cyber-Physical Systems, verifying the safety of neural network based control systems (NNCSs) poses significant challenges for the practical use of NNs-especially when safety is needed for unbounded time horizons . One reason for this is the intractability of analyzing NNs, ODEs and hybrid systems. To this end, we introduce VerSAILLE ( Ver ifiably S afe AI via L ogically L inked E nvelopes): The first general approach that allows reusing control theory literature for NNCS verification. By joining forces, we can exploit the efficiency of NN verification tools while retaining the rigor of differential dynamic logic ( dL ). Based on a provably safe control envelope in dL , we derive a specification for the NN which is proven with NN verification tools. We show that a proof of the NN's adherence to the specification is then mirrored by a dL proof on the infinite-time safety of the NNCS.

The NN verification properties resulting from hybrid systems typically contain nonlinear arithmetic over formulas with arbitrary logical structure while efficient NN verification tools merely support linear constraints. To overcome this divide, we present Mosaic: An efficient, sound and complete verification approach for polynomial real arithmetic properties on piece-wise linear NNs. Mosaic partitions complex NN verification queries into simple queries and lifts off-the-shelf linear constraint tools to the nonlinear setting in a completeness-preserving manner by combining approximation with exact reasoning for counterexample regions. In our evaluation we demonstrate the versatility of VerSAILLE and Mosaic: We prove infinite-time safety on the classical Vertical Airborne Collision Avoidance NNCS verification benchmark for some scenarios while (exhaustively) enumerating counterexample regions in unsafe scenarios. We also show that our approach significantly outperforms the State-of-the-Art tools in closed-loop NNV.

1 Introduction

For controllers of Cyber-Physical Systems (CPSs), the use of neural networks (NNs) is both a blessing and a curse. On the one hand, using NNs allows the development of goal-oriented controllers that optimize soft requirements such as passenger comfort, frequency of collision warnings or energy efficiency. On the other hand, guaranteeing that all control decisions chosen by an NN are safe is very difficult due to the complex feedback loop between the subsymbolic reasoning of an NN and the intricate dynamics often encountered in physical systems. How can this curse be alleviated? Neural Network Verification (NNV) techniques all have tried one of three strategies: Open-loop NNV entirely omits the analysis of the physical system and only analyzes input-output properties of the NN [12,23, 43,44,58,59,95,98-100]. Open-loop analyses alone cannot justify the safety of an NNCS, because they ignore its physical, feedback-loop dynamics. Closed-loop NNV performs a time-bounded analysis of the feedback loop between the NN and its physical environment [5,18,34,37,47-49,85, 88,93,94,97]. Unfortunately, a safety guarantee that comes with a time-bound (measured in seconds

dL

model (proven in KeYmaera X)

Envelope safe

ModelPlex

Open-Loop NNV query

NN

g

Approximation &

DPLL(T) generalization

Open-Loop NNV

SMT reasoning

(for completeness)

NNCS safe

NN property holds

Preliminaries

Section 3:

VerSAILLE

Section 4:

provide semantics for

rather than minutes or hours) is often insufficient when it comes to deploying safety-critical NNCSs in the real world. For example, the safety of an adaptive cruise control system must be independent of the trip length. Finally, another line of work explored techniques for learning and then verifying approximations of barrier certificates for infinite-time guarantees [3, 9, 25, 26, 31, 64]. For continuoustime , verification has not been scaled beyond simple linear control functions [25, Appendix] as it requires open-loop NNV w.r.t. nonlinear specifications, which is a notoriously neglected topic [30,31].

As an alternative to the three outlined approaches, we propose to verify NNCSs based on the rigorous mathematical foundations of differential dynamic logic ( dL ). dL is a program logic allowing the proof of infinite-time safety for abstract, nondeterministic control strategies (often called control envelopes). Due to its expressiveness and its powerful proof calculus, dL even allows the derivation of such guarantees for continuous-time systems or systems whose differential equations have no closed-form solution. By grounding our verification approach in dL , we can reuse safety results from the control theory literature for NN verification - especially for cases where characterizations of safe behavior and controllable/invariant regions are known (e.g. airborne collision avoidance [56]). How this knowledge can be reused is a non-trivial question: While dL is an excellent basis for reasoning about symbolic control strategies, the numerical/subsymbolic reasoning of NNs at their scale is far beyond the intended purpose of dL 's proof calculus. Conversely, open/closed-loop NNV tools and barrier certificates lack the infinite-time and exact reasoning available within dL . This work demonstrates how open-loop NNV can be combined with dL reasoning to combine their strengths while canceling out their weaknesses. Consequently, by relying on results from the control theory literature, we prove infinite-time safety guarantees for NNCSs that are not provable through either technique alone.

Overview. This paper alleviates the curse of NNCS safety. As shown in Figure 1, our work integrates the deductive approach of dL with techniques for open-loop NNV. To apply our approach, we assume that an abstract, nondeterministic control envelope has already been verified in dL (via KeYmaera X[38], synthesized via CESAR [57] or from the literature). Based on the dL safety result, VerSAILLE ( Ver ifiably S afe AI via L ogically L inked E nvelopes; Section 3) derives a verification query for openloop NNV by instrumenting ModelPlex [69]. By reflecting the NN through a mirror program in dL , we can then reason about an NNCS in- and outside the dL calculus simultaneously. The verification of an open-loop NNV query generated by VerSAILLE yields a dL proof that the NNCS refines a safe control envelope -implying that the infinite-time safety guarantee carries over to the NNCS.

Due to the inherent nonlinearities of hybrid systems, the generated open-loop NNV queries often contain polynomial arithmetic and the formulas have arbitrary logical structure. Hence for such queries, we also introduce Mosaic-an efficient, sound and complete framework for open-loop NNV tools. The approach lifts complete off-the-shelf open-loop NNV tools for linear constraints to polynomial constraints with arbitrary logical structure. To this end, we combine approximation with a generalization of DPLL(T) that makes the logical decomposition efficiently applicable to NN verification (whereas 'classical' DPLL(T) would become prohibitively inefficient). At the same time, Mosaic retains completeness by generalizing counterexamples into locally affine regions (Section 4). In summary, VerSAILLE provides rigorous semantics and a formal proof of infinite-time safety, while Mosaic makes our approach practically applicable to real-world systems (see also Section 5).

Contribution. Our contribution has three parts. While our implementation (N 3 V) supports NNs most commonly analyzed by open-loop NNV ( ReLU NNs), our theoretical contribution (VerSAILLE) reaches far beyond this and lays the foundations for analyzing a wide range of NNCS architectures:

We present VerSAILLE, the formal foundation that, for the first time, enables a sound proof of infinite-time safety for a concrete NNCS by reusing safety proofs from control-theory literature (in the form of dL models). VerSAILLE supports a large class of feed-forward NNs (any NN with piece-wise Noetherian activation functions; see Section 2).

We introduce Mosaic, a framework for the efficient, sound and complete verification of properties in polynomial real arithmetic on piece-wise linear NNs. Unlike other NN verifiers, Mosaic furthermore supports constraints with arbitrary propositional structure. Mosaic combines approximation techniques, a generalization of DPLL(T), and judicious SMT reasoning to lift sound and complete linear-constraint open-loop NNV tools to efficient, sound and complete polynomial constraint verification. Mosaic can exhaustively characterize unsafe state space regions (useful for retraining or the generation of fallback controllers).

We implement Mosaic for ReLU NNs in the tool N 3 V and demonstrate our approach on three case studies from adaptive cruise control, airborne collision avoidance ACAS X and steering under uncertainty. We show that, unlike N 3 V, State-of-the-Art closed-loop NNV tools cannot provide infinite-time guarantees due to approximation errors.

Running Example. The common NNCS safety benchmark of Adaptive Cruise Control [22, 39, 51] will serve as running example to demonstrate the introduced concepts. Consider an ego-car following a front-car on a 1-D lane as shown in Figure 2. The front-car drives with constant velocity v const while the ego-car (at position p rel behind the front-car) approaches with arbitrary initial (relative) velocity v rel which is adjusted through the ego-car's acceleration a rel. The primary objective is to ensure the ego-car never crashes into the front car (i.e. p rel > 0 ), however there may be secondary objectives

p

′

rel

=

v

rel

v

′

rel

=

a

rel

v

const

(e.g. energy efficiency) which are learned by an NNCS. We demonstrate how a nondeterministic, high-level acceleration strategy (i.e. a safe envelope) can be modeled and verified in dL (Section 2), how VerSAILLE derives NN properties (Section 3) and how such polynomial properties can be verified on a given NN (Section 4). No techniques are specific to the running example, but all are applicable to a wide range of NNCSs-as demonstrated by our evaluation (Section 5).

2 Background

We review dL , NNs and NN verification. FOL R (resp. FOL L R ) is the set of polynomial (resp. linear) real arithmetic first-order logic formulas. FOL N R extends FOL R with Noetherian functions [78] h 1 , . . . , h r . A Noetherian chain is a sequence of real analytic functions h 1 , . . . , h q s.t. all partial derivatives of all h j can be written as a polynomial ∂h j ( y ) ∂y i ( y ) = p ij ( y, h 1 ( y ) , . . . , h q ( y )) . Noetherian functions are representable as a polynomial over functions in a Noetherian chain. Most activation functions used in NNs are Noetherian (see Appendix A.2) Atoms of a formula ζ are denoted as Atom ( ζ ) and its variables as V ( ζ ) .

2.1 Differential Dynamic Logic

Differential dynamic logic ( dL ) [74-76,78] is a first-order multi-modal logic in which the modality is parameterized with a hybrid program describing a (discrete or continuous) state transition (see also Appendix A). Thus, dL formulas are evaluated in a state ν (if ν satisfies a formula ψ we denote this as ν ⊧ ψ ). Hybrid Programs are constructed from the primitives in Table 1 and are first-class citizens of the logic (see example below). dL is tailored to the analysis of (time discrete and time continuous) hybrid systems and supports the analysis of differential equations. Through its invariance reasoning capabilities, dL allows us to prove the infinite-time safety of control envelopes w.r.t. a system's dynamics-even for cases where the dynamics' differential equations have no closed-form solution. There is a large body of research on the verification of real-world control envelopes using dL (e.g.

ACAS X [50]). In dL , the formula [ α ] ψ expresses that ψ is always satisfied after the execution of α and ⟨ α ⟩ ψ that there exists a state satisfying ψ after the execution of α . dL comes with a sound and relatively complete proof calculus [74,76,78] and an interactive theorem prover KeYmaera X [38].

Running Example. We model our running example as a hybrid program in dL with differential equations describing the evolution of p rel , v rel , a rel along with a control envelope , i.e. an abstract acceleration strategy, α ctrl that runs at least every T seconds while the overall system may run for arbitrarily many iterations (modeled by a nondeterministic loop). Given suitable initial conditions (accInit), our objective is to prove the absence of collisions ( p rel > 0 ). This can be achieved by proving Formula (1) where the place-holder α ctrl determines the relative acceleration -B ≤ a rel ≤ A ( A and -B are resp. maximal acceleration/braking).

Our control envelope α ctrl allows braking with -B and an acceleration of 0 or another value if the constraint accCtrl 0 or resp. accCtrl 1 is satisfied (the concrete constraints are found in Appendix D):

The envelope is nondeterministic: While always braking with -B would be safe, an NNCS may learn to balance braking with secondary objectives (e.g. minimal acceleration or not falling behind). A proof for Formula (1) in KeYmaera X uses the loop invariant accInv (see Appendix D). Automation of dL proofs as well as control envelope and invariant synthesis is discussed in the literature [57, 76, 78, 89].

ModelPlex. Many CPS safety properties can be formulated through a dL formula ϕ → [( α ctrl ; α plant ) ∗ ] ψ where ϕ describes initial conditions, and ψ describes the safety criterion to be guaranteed when following the control-plant loop. ModelPlex shielding [69] allows the synthesis of correct-by-construction controller monitor formulas ζ c that ensure an implementation's runtime behavior matches the envelope α ctrl. Interpreting an implementation's action as a state transition and denoting the old state's variables as x i and the new state's variables as x + i , ζ c tells us which combinations of x i and x + i (i.e. which state transitions) are admissible w.r.t. α ctrl (see Definition 4 in Appendix A).

Running Example. We can apply ModelPlex on the proven contract in Formula (1) to synthesize a monitor for α ctrl. For this simple scenario, the resulting controller monitor formula 1 tells us what new acceleration value a + rel may be chosen given the current values of p rel , v rel :

Here, accCtrl + i is the constraint accCtrl i with a rel replaced by a + rel . Given an action of a concrete controller implementation that changes a rel to a + rel , Formula (2) tells us if this action is in accordance with the strategy modeled by α ctrl, i.e. whether we have a proof of safety for the given state transition.

2.2 Neural Network Verification

This work focuses on feed-forward neural networks typically encountered in NNCSs. The behavior of an NN with input dimension I ∈ N and output dimension O ∈ N can be summarized as a function g ∶ R I → R O . The white-box behavior is described by a sequence of L ∈ N hidden layers with dimensions n ( k ) that iteratively transform an input vector x ( 0 ) ∈ R I into an output vector x ( L ) ∈ R O . The computation of layer k is given by x ( k + 1 ) = f ( k ) ( W ( k ) x ( k ) + b ( k ) ) , i.e. an affine transformation (with FOL N R representable numbers) followed by a nonlinear activation function f ( k ) . We distinguish different classes of NNs. To this end, we decompose the activation functions f ( k ) as f ( k ) ( x ) = ∑ s i = 1 1 q i ( x ) f i ( x ) where f i are functions, q i are formulas over n ( k ) variables and 1 q i ( x ) is 1 iff q i ( x ) is true and 0 otherwise. Table 2 summarizes which results are applicable to which NN class. Each class is a subset of the previous class, i.e. our theory (Section 3) is widely applicable while our implementation (Section 5) focuses on the most common NNs. Open-loop NNV tools analyze NNs in order to verify properties on input-output relations. Their common functionality is reflected in the

VNNLIB standard [15,21]. Off-the-shelf tools are limited to linear, normalized queries (Definition 1). To address this challenge, we present a lifting procedure for the verification of generic (i.e. nonlinear and not normalized) open-loop NNV queries over polynomial real arithmetic (Section 4).

Definition 1 (Open-Loop NNV Query) . An open-loop NNV query consists of a formula p ∈ FOL R over free input variables Z = { z 1 , . . . , z I } and output variables x + 1 , . . . , x + O . We call p normalized iff p is a conjunction of some input constraints and a disjunctive normal form over mixed/output constraints, i.e. it has the structure ⋀ j p 1 ,j ( z 1 , . . . , z I )∧⋁ i ≥ 2 ⋀ j p i,j ( z 1 , . . . , z I , x + 1 , . . . , x + O ) , where all p i,j are atomic real arithmetic formulas and all p 1 ,j only contain the free variables from Z . We call a query linear iff p ∈ FOL L R and call it nonlinear otherwise.

3 VerSAILLE: Verifiably Safe AI via Logically Linked Envelopes

We introduce VerSAILLE, our approach for the verification of NNCSs via dL contracts. The key idea of VerSAILLE are nondeterministic mirrors , a mechanism that allows us to reflect a given NN g and reason within and outside of dL simultaneously. This allows us to instrument open-loop NNV techniques to prove an NN specification outside of dL which implies the safety of a corresponding (mirrored) dL model describing the NNCS. Reconsider the ACC example (Section 1) for which we synthesized a controller monitor formula in Section 2. The remaining open question is the following:

If we replace the control envelope α ctrl by a given piece-wise Noetherian NN g , does the resulting system retain the same safety guarantees?

Summary of VerSAILLE The input for VerSAILLE is a proven dL contract. Additionally, one may provide an inductive invariant ζ s to simplify the subsequent state space analysis. Using ModelPlex's synthesis of ζ c , VerSAILLE constructs a nonlinear open-loop NNV query. If we verify this query on an NN g , then the NNCS where we substitute the control envelope by g retains the same safety guarantee.

Running Example Since one can only provide formal guarantees for something one can describe formally, we first need a semantics for what it means to substitute α ctrl by g . To this end, we formalize a given piece-wise Noetherian NN g as a hybrid program α g which we call the nondeterministic mirror of g (see Definition 16 and Lemma 17 in Appendix C). E.g. for ACC, this program must have two free (i.e. read) variables p rel , v rel and one bound (i.e. written) variable a rel and must be designed in such a way that it exactly implements the NN g . Showing safety (see question above) is then equivalent to proving the following dL formula where α plant describes ACC's physical dynamics:

Since NNs do not lend themselves well to interactive analysis, an automatable mechanism to prove formula (3) outside the dL calculus is desirable. As discussed in Section 2, we can prove the safety ( p rel > 0 ) of α ctrl via dL . Thus, if we can show that all behavior of the nondeterministic mirror α g is already modeled by α ctrl, the safety guarantee carries over from the envelope to α g . To show this refinement relation [65,79], we instrument the controller monitor accCtrlFml in Formula (2). We verify that the NN g satisfies the controller monitor formula accCtrlFml (i.e. we show that g 's input-output relation satisfies Formula (2)). If this is the case, α g 's behavior is modeled by our envelope α ctrl. In practice, it is unnecessary that the behavior of α g is modeled by α ctrl everywhere (e.g. we are not interested in states with p rel ≤ 0 ). It suffices to consider all states within the inductive invariant accInv of the envelope's system (Formula (1)) as those are precisely the states for which the guarantee on α ctrl holds. Thus, we can prove Formula (3) by showing that g satisfies the following specification for all inputs:

case

1

case

2

Linear Constraint

Nonlinear Constraint

Linear Approximation

Reachable by

g

Unsafe

spurious CEX

concrete CEX

Output Space

g

(

x

)

CEX region:

input polytope

with

g

locally affine

VerSAILLE also allows to soundly constrain the system to value ranges where the NN has been trained (Lemma 20). Verifying the queries generated by VerSAILLE with a (nonlinear) open-loop NNV tool (Definition 22) then implies safety of the NNCS (full formalism see Appendix C.1):

Theorem 2 (Soundness) . Let g be a piece-wise Noetherian NN. Further, let C ≡ ( ϕ →[( α ctl ; α plant ) ∗ ] ψ ) be a valid contract with controller monitor ζ c ∈ FOL R and inductive invariant ζ s ∈ FOL R . If a sound Nonlinear Neural Network Verifier returns unsat for the query p ≡ ( ζ s ∧¬ ζ c ) on g then ϕ →[( α g ; α plant ) ∗ ] ψ is valid for the nondet. mirror α g .

If g is piece-wise polynomial, Formula (4) is expressible in FOL R and therefore its verification decidable (Lemma 24 which is a special case of [79]). In practice, we can be much more efficient than naively applying real arithmetic theory solvers by relying on open-loop NNV technologies to check the negated property accInv ∧¬ accCtrlFml: If this property is unsatisfiable for a given NN g , then Formula (3) is valid. Off-the-shelf open-loop NNV tools are unable to reason about Formula (4) due to its nonlinearities and the non-normalized formula structure. Thus, our second contribution (Section 4) lifts open-loop NNV tools to the task of verifying nonlinear, non-normalized queries.

4 Mosaic: Nonlinear Open-Loop NN Verification

Since NNCSs usually exhibit nonlinear physical behavior, the verification property ζ s ∧¬ ζ c will be nonlinear as well. ζ s ∧¬ ζ c is also a formula of arbitrary structure and not a normalized open-loop NNV query (Definition 1). This is evident in the verification query for our running example (see Appendix D) or in Figure 3 where the green and blue constraints (left) describe two independent nonlinear input regions ( case 1 and case 2 ) and the red regions (right) correspond to the unsafe regions for case 1 . In contrast to this, off-the-shelf open-loop NNV tools (e.g. nnenum [12] or Marabou [59]) only support the verification of linear, normalized open-loop NNV queries on piece-wise linear NNs. Mosaic is a framework that allows us to lift off-the-shelf open-loop NNV tools for linear, normalized open-loop NNV queries to polynomial queries of arbitrary logical structure. This approach has the notable advantage that Mosaic's capabilities grow as open-loop NNV technology advances while retaining completeness w.r.t. polynomial constraints. An overview of the algorithm is given in Algorithm 1 and Figure 3 which is explained throughout this section with further details in Appendix B.

MOSAIC. To tackle formulas with arbitrary logical structure we could use DNNV [87] which implements a simple expansion algorithm or the standard formulation of DPLL(T). Consider the specification in Figure 4 where the x-axis represents the NN's inputs, the y-axis possible outputs, and the red regions are considered unsafe: DPLL(T), or an expansion algorithm, would enumerate all 7 red regions individually and then invoke an open-loop NNV tool for each. However, this becomes prohibitively inefficient when used for reachability-based open-loop NNV tools: Such tools will compute the reachable regions for A 1 three times, for A 2 three times, and for A 3 two times. Instead, we propose MOSAIC which first only enumerates input regions (e.g. the regions A 1 , A 2 , and A 3 ) and

A

1

A

2

A

3

then enumerates unsafe regions in the output space per input region . The unsafe regions are then aggregated into disjunctive normal form. Hence, for the example in Figure 4, MOSAIC would only yield three normalized queries which are still compatible with the standardized interface for NN

Output

Algorithm 1: Verification of nonlinear queries on piece-wise linear NNs: ENUM internally uses off-the-shelf open-loop NNV tools for the verification of linear normalized queries on NNs.

verification tools [15]. We call this partitioning of the input space a mosaic and, in reminiscence of this analogy, we call the individual queries azulejos ( [A.Tu'le.xo] , see e.g. regions in shades of gray in Figure 3). MOSAIC guarantees that reachability-based open-loop NNV tools do not explore the same input region multiple times (see flatness result in Proposition 10; Appendix B). For our ACAS case study, naive rewriting (as done by e.g. DNNV [87]) may produce up to 39 trillion propositionally feasible queries. In contrast, MOSAIC only produces 19k queries (see Table 7 in Appendix E.2). MOSAIC also separates nonlinear constraints q n (must be checked outside open-loop NNV) from linear constraints q l (can be passed to open-loop NNV).

On the technical side, Mosaic proceeds by executing a SAT solving based DPLL(T) loop until a satisfiable conjunction of input constraints is found. At this stage, we fix the conjunction's linear input constraints (i.e. the azulejo) and an inner loop enumerates conjunctions over mixed/output constraints that are satisfiable in combination with the fixed azulejo. For each such conjunction, we save the conjunction of linear mixed/output constraints. This results in a linear, normalized Open-Loop NNV query (conjunction over input, disjunctive normal from over output). We employ a similar inner loop to enumerate satisfiable conjunctions of nonlinear constraints to later check counterexamples via SMT solving (see Retaining completeness). At each step, we interleave propositional and theory solving to discard conjunctions unsatisfiable in real arithmetic as early as possible.

LINEARIZE. To verify polynomial specifications via off-the-shelf (linear) open-loop NNV, we need to soundly approximate the query's nonlinear constraints. In principle, we could perform this approximation for each azulejo (see MOSAIC) separately. To this end, consider an atomic polynomial constraint p ( x ) ≤ 0 which is part of a query. If there are two azulejos with the constraints p ( x ) ≤ 0 and p ( x ) > 0 , separate approximation would lead to the constraint p ( x ) ≤ 0 and p ( x ) > 0 (with p / p resp. over/under-approximations of p ). This would duplicate the exploration of the area in between p and p and can lead to an exponential blowup for many approximations. Instead, we use a global piece-wise approximation via OVERT [88] (orange lines around the blue nonlinear constraint on Figure 3) and integrate the approximate constraints into the original query via implications (e.g. we add p ( x ) ≤ 0 → p ( x ) ≤ 0 ). These additional linear constraints are then automatically enumerated via MOSAIC (see azulejos for case 1 in Figure 3). The linearization happens for input as well as output constraints. When passing the open-loop NNV query to the off-the-shelf tool, we soundly omit the nonlinear constraints, thus only leaving behind linear constraints q l (green and orange in Figure 3).

Retaining completeness. Without further efforts, MOSAIC and LINEARIZE together yield a sound algorithm to check nonlinear, not normalized open-loop NNV queries, but not a complete one: A reachability analysis via open-loop NNV for a given azulejo may produce spurious counterexamples that are an artifact of the linearization (see Figure 3) The key insight to achieve completeness is the observation that for piece-wise linear NNs any concrete counterexample generated via openloop NNV corresponds to a counterexample region ι (yellow polytope on the left of Figure 3) on which the NN reduces to an affine transformation ω (obtained by fixing piece-wise functions to the linear segment of the concrete counterexample, i.e. we fix the value of the 1 functions; see also Section 2.2). This insight can be used in two ways: First, we can use it to enumerate all counterexample regions ι (by adjusting the tool's internal enumeration and/or via the algorithm ENUM; see Appendix B.3). Secondly, for a given counterexample region, we can check whether there exists a concrete counterexample to the nonlinear specification via SMT solving (in Algorithm 1 this is performed by FILTER; see Appendix B.3). By exploiting the affine transformation ω , our SMT encoding for nonlinear constraints has at most I free variables (for I input dimensions of the NN) and is therefore significantly more tractable than an encoding of the entire NN in SMT.

Using the components outlined above (for details see Appendix B), we prove the soundness and completeness of Mosaic. Completeness turns out to also be of practical relevance as approximation alone would have failed to verify the DNC NN of the ACAS benchmark discussed in Section 5.

Theorem 3 (Soundness and Completeness) . Let g be a piece-wise linear NN, p be a real arithmetic formula and R variable ranges for all in- and output variables of g . Algorithm 1 returns unsafe iff there exists an input z ∈ R I such that ( z, g ( z )) is in the range R and p ( z, g ( z )) is satisfied.

In Section 5 we build upon nnenum [10,12] which can enumerate all counterexample regions of an NN with N ReLU activations in time O( 2 N ) , and upon cylindrical algebraic decomposition (CAD [28]) with complexity O( 2 2 V ) for V variables 2 . Assuming M atomic formulas in the open-loop NNV query and I input dimensions this yields a worst-case runtime of O( 2 M + N + 2 I ) . This is an exponential improvement over naive O( 2 M + 2 N + I ) CAD encodings as N >> I . In practice the performance is even better; usually, MOSAIC explores fewer queries, nnenum returns less counterexample regions and SMT solving tends to perform very well for the small input dimensions I of NNCS control.

5 Evaluation

We implemented Mosaic for ReLU NNs in a new Julia [16] tool called N 3 V based on the software packages nnenum [10,12], PicoSAT [17,19] and Z3 [32,52]. We provide wall-clock times on an AMDRyzen 7 PRO 5850U CPU (N 3 V is sequential; nnenum uses multithreading). The evaluation presented in this section focuses on vertical airborne collision avoidance (VCAS), with other experiments in Appendix E. Airborne Collision Avoidance Systems try to recognize plane trajectories that might lead to a Near Mid-Air Collision (NMAC) with other aircrafts and advise the pilot to avoid such collisions. NMACs are defined as two planes (ownship and intruder) flying closer than 500 ft horizontally or 100 ft vertically. Currently, the Federal Aviation Administration (FAA) develops a new airborne collision avoidance system called Airborne Collision Avoidance System X (ACAS X) [71]. Prior work by Jeannin et al. [50] showed a nondeterministic, provably safe dL envelope for airborne collision avoidance. While the original proposal for ACAS X [45,60] was shown to be unsafe [50], the correctness of a VCAS NN implementation [53,55] was proven [54] and disproven [56] in special cases. The proposed NNs contain 6 hidden layers with 45 neurons each and produce one of 9 collision avoidance advisories ( Strengthen Climb to at least 2500 ft/min (SCL2500) to Strengthen Descent to at least 2500ft/min (SDES2500); see Table 9 in Appendix F for a list of allowed advisories). The objective of the NNs is to ensure safety while minimizing the number of alerts sent to the pilot. We present an exhaustive analysis of the VCAS NNCS for level flight intruders.

We provide additional experimental results in Appendix E. First, we demonstrate the feasibility of our approach for the running example of ACC. Depending on NN size and chosen linearization, our approach can verify or exhaustively enumerate counterexamples for the NNCS in 47 to 300 seconds. For a case study on Zeppelin steering under (uniformly sampled) wind perturbations, we adapt a differential hybrid games formalization [77] to analyze an NN controller trained by us. Here, we encountered a controller that showed

Table 3: Verification of ACAS NNs for level flight: Previous advisory (=Prev. Adv.), runtime; number of counterexample (=CE) regions and time to the discovery of the first CE.

very positive empirical performance while being provably unsafe in large parts of the input space: While performing very well on average, the control policy was vulnerable to unlikely wind perturbations - an issue we only found through our verification. For ACC, we also perform a comparison to other techniques: While Closed-Loop techniques are useful for the analysis of bounded-time safety, their efficiency greatly depends on the system's dynamics and the considered input space. Our

infinite-time horizon approach can be more efficient than Closed-Loop techniques as it evades the necessity to analyze the system's dynamics along with the NN (see Table 6). Usually, it is desirable to show infinite-time safety on the entire (controllable) state space. However, the approximation errors incurred via prior closed-loop NNV techniques prohibit this as they will either ignore states inside the controllable region or allow unsafe actions pushing the system outside its controllable region. Conversely, SMT-based techniques do not have these approximation issues, but cannot scale to NNs of the size analyzed in this work. We also provide a conceptual comparison demonstrating the efficiency of the Mosaic procedure for normalized query generation over DNNV's expansion-based algorithm (see Table 7), naive SMT solving (Table 8) and Genin et al.'s tailored ACAS approach [42].

Verification Results for VCAS. We use the nondeterministic control envelope and loop invariant by Jeannin et al. [50, Thm. 1] to analyze safety for intruders in level flight (i.e. intruder vertical velocity is 0). For the same reasons as prior work [50], we ignore Clear-of-Conflict advisories. The open-loop NNV queries obtained via VerSAILLE had up to 112 distinct atoms and trees up to depth 9. To determine the maximal output neuron, open-loop NNV queries initially contain atoms sorting the NN's outputs x 1 , . . . , x n (e.g., x 1 ≤ x 2 ). To avoid enumerating all permutations, we perform symmetry elimination via an atomic predicate encoding that some output i is maximal. We analyzed the full range of possible NN inputs for intruders in level flight (relative height ∣ h ∣ ≤ 8000 ft, ownship velocity ∣ v ∣ ≤ 100 ft/s and time to NMAC 6 s ≤ τ ≤ 40 s). Our results are in Table 3: safe implies that the NN's advisories (other than Clear-of-Conflict) in this scenario never lead to a collision when starting within the invariant. The safety for DNC was only verifiable through SMT filtering (approximation yielded spurious counterexamples). This underscores the importance of Mosaic's completeness and implies that [56] is insufficient to prove safety. A non-exhaustive analysis for non-level flight yielded counterexamples even for DNC/DND (see Appendix G). For unsafe level flight scenarios, we exhaustively characterize unsafe regions. This characterization goes far beyond characterizations in prior work [56] that were generated using manual approximation and resulted in (non-exhaustive) point-wise characterizations. Figure 5 shows a concrete avoidable NMAC (more examples are in Appendix G).

Scalability. N 3 V provides guarantees for an NN's full input space. Hence, N 3 V is not directly comparable to, e.g., the scalability of local robustness verifiers, that, while sometimes scaling to hundred thousands of ReLU s [20], only analyze tiny fractions of the input space [92, Sec. 6]. Contemporary work on global properties outside NNCSs has been scaled to 100 ReLU nodes [8]. Related work on infinite-time NNCSs (see also Section 6) has not been scaled beyond 30 nodes in similar settings [25, 64]. The largest NNs verified by us so far had 270 ReLU s, indicating N 3 V is at the frontier of State-of-the-Art scalability for global properties. Some NNCS applications (e.g. ACAS [55]) turn to NNs to efficiently encode complex strategies in mid-scale NNs. N 3 V scales to NNs of this kind.

6 Related Work

Shielding. Justified Speculative Control [39] is closely related in its use of dL . However, we verify the NNCS a priori instead of treating ML models as a black box and a posteriori using runtime enforcement techniques [61,69]. Shielding can also be used to train models such that they (probabilistically) conform to a shield/monitor [6]. Training methodologies are beyond the scope of this paper.

Barrier Certificates. An orthogonal direction of research explores learning Neural Barrier/Lyapunov Functions to prove safety properties [30]. Although initially used for 'pure' dynamical and hybrid systems (without NNs) [1,2,4,70,73,83,101,104], the methods have since been extended to NNCS with discrete [9,26,64] and continuous [25,31,81] time behavior. While the former works

can only approximate continuous time behavior, the latter techniques use off-the-shelf SMT solvers (see also Appendix E.2) for certificate verification which severely limits scalability. While some works ignore verification entirely [31, 81], the remaining work only considered linear single-layer NNs [25, Appendix] verified with dReal (see SMT comparison in Appendix E.2). Dawson et al. [31] note 'scalable verification for learned certificate functions remains an open problem'. Using N 3 V as an alternative to SMT for certificate verification is future work. VerSAILLE evades the necessity to learn barrier certificates by reusing established control-theory literature.

Open-Loop NNV. VEHICLE [29] integrates open-loop NNV with Agda. However, VEHICLE only allows importing normalized, linear properties which limits applicability to CPS verification in realistic settings. Open-loop NNV tools [12,23,35,43,44,58,59,95,98-100] do not consider the physical environment and thus cannot guarantee the safety of an NNCS. Even in cases where such methods allow nonlinear behavior in activation functions, they do not admit the verification of arbitrary polynomial constraints over the input and output space. Most methodologies could be integrated into Mosaic's framework, i.e. we can lift complete off-the-shelf open-loop NNV tools to verify polynomial constraints with arbitrary structure. DNNV [87] proposed an approach for open-loop NNV query normalization using a simple expansion algorithm. DNNV has the same limitations as all open-loop NNV tools (no NNCS analysis; no nonlinear constraints) and is less efficient than Mosaic w.r.t. NN reachability analysis (see Section 4). Pre-image computation [62,68,103] computes input regions producing a fixed NN prediction. For efficiency, our work constrains the input space with invariants and value ranges-this efficiency would be lost by a backward computation alone.

Related Techniques. Unlike [13, 80], we support arbitrary polynomial constraints and retain completeness. Moreover, these works do not support arbitrary logical structure and represent openloop NNV techniques unable to analyze NNCSs. Some prior work used techniques for constructing counterexample regions [33,102] but only for individual datapoints-neither to compute exhaustive characterizations nor to regain completeness for incomplete verifiers. Unlike classical DPLL(T) [40], MOSAIC is tailored to theory-solving w.r.t. reachability analyzers. To this end, MOSAIC groups output regions with the same input constraints which deduplicates work (see Section 4).

Closed-Loop NNV. Closed-loop NNV tools [5,18,34,37,47-49,85,88,93,94,97] only consider a fixed time horizon and thus cannot guarantee infinite-time horizon safety (see Appendix E.2). Unlike [11,42,56], our approach is automated and applicable to any CPS expressible in dL (not just ACAS X; see case studies in Appendix E). Other approaches verify simplified control outputs [42]; rely on hand-crafted approximations while lacking exhaustive counterexamples characterizations [56]; or require quantization effectively analyzing only a surrogate system instead [11].

7 Conclusion and Future Work

This work presents VerSAILLE, the first technique exploiting dL contracts to prove safety of NNCSs with piece-wise Noetherian NNs. VerSAILLE requires open-loop NNV tools capable of verifying nonnormalized polynomial properties that did not exist. Thus, with Mosaic we present an efficient, sound and complete approach for the verification of such properties on piece-wise linear NNs. We implemented Mosaic for ReLU NNs in the tool N 3 Vand demonstrate the applicability and scalability of our approach on multiple case studies (Section 5 and Appendix E). The application to NNCSs by Julian et al. [53, 55] shows that our approach scales even to intricate, high-stakes applications such as airborne collision avoidance. Our results underscore the categorical difference of our approach to closed-loop NNV techniques. Overall, we demonstrate an efficient and generally applicable approach that opens the door for developing of goal-oriented and infinite-time horizon safe NNCSs in the real world.

Future Work. We believe there is potential for engineering and algorithmic improvements that could further improve the performance of N 3 V. Our implementation could also be extended to other piece-wise linear activation functions. We would also like to explore the efficiency of N 3 V for non-polynomial specifications w.r.t. suitable SMT solvers. Mosaic may be of interest even beyond NNCS verification in VerSAILLE. Since, e.g., neural barrier certificate verification also requires nonlinear open-loop NNV, Mosaic could be equally applicable in this context. Finally, it would be interesting to apply our approach to further case studies. To this end, the bottle-neck is currently the limited availability of NNCS which are safe w.r.t. an infinite-time horizon.

References

A. Abate, D. Ahmed, A. Edwards, M. Giacobbe, and A. Peruffo. FOSSIL: a software tool for the formal synthesis of Lyapunov functions and barrier certificates using neural networks. In S. Bogomolov and R. M. Jungers, editors, HSCC '21: 24th ACM International Conference on Hybrid Systems: Computation and Control, Nashville, Tennessee, May 19-21, 2021 , pages 24:1-24:11. ACM, 2021.

A. Abate, D. Ahmed, M. Giacobbe, and A. Peruffo. Formal synthesis of Lyapunov neural networks. IEEE Control. Syst. Lett. , 5(3):773-778, 2021.

A. Abate, A. Edwards, and M. Giacobbe. Neural abstractions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022.

D. Ahmed, A. Peruffo, and A. Abate. Automated and sound synthesis of Lyapunov functions with SMT solvers. In A. Biere and D. Parker, editors, Tools and Algorithms for the Construction and Analysis of Systems - 26th International Conference, TACAS 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings, Part I , volume 12078 of LNCS , pages 97-114. Springer, 2020.

M. E. Akintunde, E. Botoeva, P. Kouvaros, and A. Lomuscio. Formal verification of neural agents in non-deterministic environments. Auton. Agents Multi Agent Syst. , 36(1):6, 2022.

M. Alshiekh, R. Bloem, R. Ehlers, B. K¨ onighofer, S. Niekum, and U. Topcu. Safe reinforcement learning via shielding. In S. A. McIlraith and K. Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI 18, New Orleans, Louisiana, USA, February 2-7, 2018 , pages 2669-2678. AAAI Press, 2018.

M. Althoff. An introduction to CORA 2015. In G. Frehse and M. Althoff, editors, ARCH1415. 1st and 2nd International Workshop on Applied veRification for Continuous and Hybrid Systems , volume 34 of EPiC Series in Computing , pages 120-151. EasyChair, 2015.

A. Athavale, E. Bartocci, M. Christakis, M. Maffei, D. Nickovic, and G. Weissenbacher. Verifying global two-safety properties in neural networks with confidence. In A. Gurfinkel and V. Ganesh, editors, Computer Aided Verification - 36th International Conference, CAV 2024, Montreal, QC, Canada, July 24-27, 2024, Proceedings, Part II , volume 14682 of LNCS , pages 329-351. Springer, 2024.

E. Bacci, M. Giacobbe, and D. Parker. Verifying reinforcement learning up to infinity. In Z. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 21542160. ijcai.org, 2021.

S. Bak. nnenum: Verification of ReLU neural networks with optimized abstraction refinement. In A. Dutle, M. M. Moscato, L. Titolo, C. A. Mu˜ noz, and I. Perez, editors, NASA Formal Methods - 13th International Symposium, NFM 2021, Virtual Event, May 24-28, 2021, Proceedings , volume 12673 of LNCS , pages 19-36. Springer, 2021.

S. Bak and H. Tran. Neural network compression of ACAS Xu early prototype is unsafe: Closed-loop verification through quantized state backreachability. In J. V. Deshmukh, K. Havelund, and I. Perez, editors, NASA Formal Methods - 14th International Symposium, NFM 2022, Pasadena, CA, USA, May 24-27, 2022, Proceedings , volume 13260 of LNCS , pages 280-298. Springer, 2022.

S. Bak, H. Tran, K. Hobbs, and T. T. Johnson. Improved geometric path enumeration for verifying ReLU neural networks. In S. K. Lahiri and C. Wang, editors, Computer Aided

Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part I , volume 12224 of LNCS , pages 66-96. Springer, 2020.

M. Balunovic, M. Baader, G. Singh, T. Gehr, and M. T. Vechev. Certifying geometric robustness of neural networks. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 15287-15297, 2019.

H. Barbosa, C. W. Barrett, M. Brain, G. Kremer, H. Lachnitt, M. Mann, A. Mohamed, M. Mohamed, A. Niemetz, A. N¨ otzli, A. Ozdemir, M. Preiner, A. Reynolds, Y. Sheng, C. Tinelli, and Y. Zohar. cvc5: A versatile and industrial-strength SMT solver. In D. Fisman and G. Rosu, editors, Tools and Algorithms for the Construction and Analysis of Systems - 28th International Conference, TACAS 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Munich, Germany, April 2-7, 2022, Proceedings, Part I , volume 13243 of LNCS , pages 415-442. Springer, 2022.

C. Barrett, G. Katz, D. Guidotti, L. Pulina, N. Narodytska, and A. Tacchella. The VNNLIB standard for benchmarks (draft). Technical report, AIMS-LAB, University of Genoa, 2021.

J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah. Julia: A fresh approach to numerical computing. SIAM Review , 59(1):65-98, 9 2017.

A. Biere. PicoSAT essentials. J. Satisf. Boolean Model. Comput. , 4(2-4):75-97, 2008.

S. Bogomolov, M. Forets, G. Frehse, K. Potomkin, and C. Schilling. JuliaReach: a toolbox for set-based reachability. In N. Ozay and P. Prabhakar, editors, Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2019, Montreal, QC, Canada, April 16-18, 2019 , pages 39-44. ACM, 2019.

J. Bolewski, C. Lucibello, and M. Bouton. PicoSAT.jl. GitHub, 2024.

C. Brix, S. Bak, C. Liu, and T. T. Johnson. The fourth international verification of neural networks competition (VNN-COMP 2023): Summary and results. CoRR , abs/2312.16760, 2023.

C. Brix, M. N. M¨ uller, S. Bak, T. T. Johnson, and C. Liu. First three years of the international verification of neural networks competition (VNN-COMP). Int. J. Softw. Tools Technol. Transf. , 25(3):329-339, 2023.

M. Brosowsky, F. Keck, J. Ketterer, S. Isele, D. Slieter, and M. Z¨ ollner. Safe deep reinforcement learning for adaptive cruise control by imposing state-specific safe sets. In 2021 IEEE Intelligent Vehicles Symposium (IV) , pages 488-495, 2021.

R. Bunel, J. Lu, I. Turkaslan, P. H. S. Torr, P. Kohli, and M. P. Kumar. Branch and bound for piecewise linear neural network verification. J. Mach. Learn. Res. , 21:42:1-42:39, 2020.

S. Cai, B. Li, and X. Zhang. Local search for SMT on linear integer arithmetic. In S. Shoham and Y. Vizel, editors, Computer Aided Verification - 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II , volume 13372 of LNCS , pages 227-248. Springer, 2022.

Y. Chang, N. Roohi, and S. Gao. Neural Lyapunov control. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 32403249, 2019.

S. Chen, M. Fazlyab, M. Morari, G. J. Pappas, and V. M. Preciado. Learning Lyapunov functions for hybrid systems. In 55th Annual Conference on Information Sciences and Systems, CISS 2021, Baltimore, MD, USA, March 24-26, 2021 , page 1. IEEE, 2021.

A. Cimatti, A. Griggio, B. J. Schaafsma, and R. Sebastiani. The mathsat5 SMT solver. In N. Piterman and S. A. Smolka, editors, Tools and Algorithms for the Construction and Analysis of Systems - 19th International Conference, TACAS 2013, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2013, Rome, Italy, March 16-24, 2013. Proceedings , volume 7795 of LNCS , pages 93-107. Springer, 2013.

G. E. Collins. Quantifier elimination for real closed fields by cylindrical algebraic decomposition-preliminary report. SIGSAM Bull. , 8(3):80-90, 1974.

M. L. Daggitt, W. Kokke, R. Atkey, L. Arnaboldi, and E. Komendantskaya. Vehicle: Interfacing neural network verifiers with interactive theorem provers. CoRR , abs/2202.05207, 2022.

C. Dawson, S. Gao, and C. Fan. Safe control with learned certificates: A survey of neural Lyapunov, barrier, and contraction methods for robotics and control. IEEE Trans. Robotics , 39(3):1749-1767, 2023.

C. Dawson, Z. Qin, S. Gao, and C. Fan. Safe nonlinear control using robust neural Lyapunovbarrier functions. In A. Faust, D. Hsu, and G. Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK , volume 164 of Proceedings of Machine Learning Research , pages 1724-1735. PMLR, 2021.

L. M. de Moura and N. S. Bjørner. Z3: an efficient SMT solver. In C. R. Ramakrishnan and J. Rehof, editors, Tools and Algorithms for the Construction and Analysis of Systems, 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings , volume 4963 of LNCS , pages 337-340. Springer, 2008.

D. I. Dimitrov, G. Singh, T. Gehr, and M. T. Vechev. Provably robust adversarial examples. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

S. Dutta, X. Chen, and S. Sankaranarayanan. Reachability analysis for neural feedback systems using regressive polynomial rule inference. In N. Ozay and P. Prabhakar, editors, Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2019, Montreal, QC, Canada, April 16-18, 2019 , pages 157-168. ACM, 2019.

Y. Y. Elboher, J. Gottschlich, and G. Katz. An abstraction-based framework for neural network verification. In S. K. Lahiri and C. Wang, editors, Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part I , volume 12224 of LNCS , pages 43-65. Springer, 2020.

C. Eleftheriadis, N. Kekatos, P. Katsaros, and S. Tripakis. On neural network equivalence checking using SMT solvers. In S. Bogomolov and D. Parker, editors, Formal Modeling and Analysis of Timed Systems - 20th International Conference, FORMATS 2022, Warsaw, Poland, September 13-15, 2022, Proceedings , volume 13465 of LNCS , pages 237-257. Springer, 2022.

J. Fan, C. Huang, X. Chen, W. Li, and Q. Zhu. ReachNN*: A tool for reachability analysis of neural-network controlled systems. In D. V. Hung and O. Sokolsky, editors, Automated Technology for Verification and Analysis - 18th International Symposium, ATVA 2020, Hanoi, Vietnam, October 19-23, 2020, Proceedings , volume 12302 of LNCS , pages 537-542. Springer, 2020.

N. Fulton, S. Mitsch, J. Quesel, M. V¨ olp, and A. Platzer. KeYmaera X: an axiomatic tactical theorem prover for hybrid systems. In A. P. Felty and A. Middeldorp, editors, Automated Deduction - CADE-25 - 25th International Conference on Automated Deduction, Berlin, Germany, August 1-7, 2015, Proceedings , volume 9195 of LNCS , pages 527-538. Springer, 2015.

N. Fulton and A. Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In S. A. McIlraith and K. Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 6485-6492. AAAI Press, 2018.

H. Ganzinger, G. Hagen, R. Nieuwenhuis, A. Oliveras, and C. Tinelli. DPLL(T): fast decision procedures. In R. Alur and D. A. Peled, editors, Computer Aided Verification, 16th International Conference, CAV 2004, Boston, MA, USA, July 13-17, 2004, Proceedings , volume 3114 of LNCS , pages 175-188. Springer, 2004.

S. Gao, S. Kong, and E. M. Clarke. dReal: An SMT solver for nonlinear theories over the reals. In M. P. Bonacina, editor, Automated Deduction - CADE-24 - 24th International Conference on Automated Deduction, Lake Placid, NY, USA, June 9-14, 2013. Proceedings , volume 7898 of LNCS , pages 208-214. Springer, 2013.

D. Genin, I. Papusha, J. Brul´ e, T. Young, G. E. Mullins, Y. Kouskoulas, R. Wu, and A. C. Schmidt. Formal verification of neural network controllers for collision-free flight. In R. Bloem, R. Dimitrova, C. Fan, and N. Sharygina, editors, Software Verification - 13th International Conference, VSTTE 2021, New Haven, CT, USA, October 18-19, 2021, and 14th International

Workshop, NSV 2021, Los Angeles, CA, USA, July 18-19, 2021, Revised Selected Papers , volume 13124 of LNCS , pages 147-164. Springer, 2021.

P. Henriksen and A. Lomuscio. DEEPSPLIT: an efficient splitting method for neural network verification via indirect effect analysis. In Z. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 2549-2555. ijcai.org, 2021.

P. Henriksen and A. R. Lomuscio. Efficient neural network verification via adaptive refinement and adversarial search. In G. D. Giacomo, A. Catal´ a, B. Dilkina, M. Milano, S. Barro, A. Bugar´ ın, and J. Lang, editors, ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 , volume 325 of Frontiers in Artificial Intelligence and Applications , pages 2513-2520. IOS Press, 2020.

J. E. Holland, M. J. Kochenderfer, and W. A. Olson. Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance. Air Traffic Control Quarterly , 21(3):275-297, 2013.

C. Huang, J. Fan, X. Chen, W. Li, and Q. Zhu. POLAR: A polynomial arithmetic framework for verifying neural-network controlled systems. In A. Bouajjani, L. Hol´ ık, and Z. Wu, editors, Automated Technology for Verification and Analysis - 20th International Symposium, ATVA 2022, Virtual Event, October 25-28, 2022, Proceedings , volume 13505 of LNCS , pages 414-430. Springer, 2022.

C. Huang, J. Fan, W. Li, X. Chen, and Q. Zhu. ReachNN: Reachability analysis of neuralnetwork controlled systems. ACM Trans. Embed. Comput. Syst. , 18(5s):106:1-106:22, 2019.

R. Ivanov, T. J. Carpenter, J. Weimer, R. Alur, G. J. Pappas, and I. Lee. Verifying the safety of autonomous systems with neural network controllers. ACM Trans. Embed. Comput. Syst. , 20(1):7:1-7:26, 2021.

R. Ivanov, T. J. Carpenter, J. Weimer, R. Alur, G. J. Pappas, and I. Lee. Verisig 2.0: Verification of neural network controllers using Taylor model preconditioning. In A. Silva and K. R. M. Leino, editors, Computer Aided Verification - 33rd International Conference, CAV 2021, Virtual Event, July 20-23, 2021, Proceedings, Part I , volume 12759 of LNCS , pages 249-262. Springer, 2021.

J. Jeannin, K. Ghorbal, Y. Kouskoulas, A. C. Schmidt, R. W. Gardner, S. Mitsch, and A. Platzer. A formally verified hybrid system for safe advisories in the next-generation airborne collision avoidance system. Int. J. Softw. Tools Technol. Transf. , 19(6):717-741, 2017.

T. T. Johnson, D. M. Lopez, L. Benet, M. Forets, S. Guadalupe, C. Schilling, R. Ivanov, T. J. Carpenter, J. Weimer, and I. Lee. ARCH-COMP21 category report: Artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plants. In G. Frehse and M. Althoff, editors, 8th International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH21), Brussels, Belgium, July 9, 2021 , volume 80 of EPiC Series in Computing , pages 90-119. EasyChair, 2021.

D. Jovanovic and L. de Moura. Solving non-linear arithmetic. ACM Commun. Comput. Algebra , 46(3/4):104-105, 2012.

K. D. Julian. Safe and efficient aircraft guidance and control using neural networks . PhD thesis, Stanford University, 2020.

K. D. Julian and M. J. Kochenderfer. Guaranteeing safety for neural network-based aircraft collision avoidance systems. In 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC) , pages 1-10, 2019.

K. D. Julian, J. Lopez, J. S. Brush, M. P. Owen, and M. J. Kochenderfer. Policy compression for aircraft collision avoidance systems. In 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC) , pages 1-10, 2016.

K. D. Julian, S. Sharma, J.-B. Jeannin, and M. J. Kochenderfer. Verifying aircraft collision avoidance neural networks through linear approximations of safe regions. In Verification of Neural Networks (VNN 2019) , 2019.

A. Kabra, J. Laurent, S. Mitsch, and A. Platzer. CESAR: Control envelope synthesis via angelic refinements. In B. Finkbeiner and L. Kov´ acs, editors, TACAS , volume 14570 of LNCS , pages 144-164. Springer, 2024.

G. Katz, C. W. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In R. Majumdar and V. Kuncak, editors, Computer Aided Verification - 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I , volume 10426 of LNCS , pages 97-117. Springer, 2017.

G. Katz, D. A. Huang, D. Ibeling, K. Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zeljic, D. L. Dill, M. J. Kochenderfer, and C. W. Barrett. The Marabou framework for verification and analysis of deep neural networks. In I. Dillig and S. Tasiran, editors, Computer Aided Verification - 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I , volume 11561 of LNCS , pages 443-452. Springer, 2019.

M. J. Kochenderfer, J. E. Holland, and J. P. Chryssanthacopoulos. Next generation airborne collision avoidance system. Lincoln Laboratory Journal , 19(1):17-33, 2012.

B. K¨ onighofer, R. Bloem, R. Ehlers, and C. Pek. Correct-by-construction runtime enforcement in AI - A survey. In Principles of Systems Design - Essays Dedicated to Thomas A. Henzinger on the Occasion of His 60th Birthday , pages 650-663, 2022.

S. Kotha, C. Brix, J. Z. Kolter, K. Dvijotham, and H. Zhang. Provably bounding neural network preimages. Advances in Neural Information Processing Systems , 36, 2024.

V. Kunc and J. Kl´ ema. Three decades of activations: A comprehensive survey of 400 activation functions for neural networks, 2024.

M. Lechner, D. Zikelic, K. Chatterjee, and T. A. Henzinger. Infinite time horizon safety of Bayesian neural networks. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 10171-10185, 2021.

S. M. Loos and A. Platzer. Differential refinement logic. In M. Grohe, E. Koskinen, and N. Shankar, editors, Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science, LICS '16, New York, NY, USA, July 5-8, 2016 , pages 505-514. ACM, 2016.

D. M. Lopez, M. Althoff, L. Benet, X. Chen, J. Fan, M. Forets, C. Huang, T. T. Johnson, T. Ladner, W. Li, et al. ARCH-COMP22 category report: Artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plants. In Proceedings of 9th International Workshop on Applied , volume 90, pages 142-184, 2022.

D. M. Lopez, M. Althoff, M. Forets, T. T. Johnson, T. Ladner, and C. Schilling. ARCHCOMP23category report: Artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plants. In G. Frehse and M. Althoff, editors, Proceedings of 10th International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH23), San Antonio, Texas, USA, May 9, 2023 , volume 96 of EPiC Series in Computing , pages 89-125. EasyChair, 2023.

K. Matoba and F. Fleuret. Exact preimages of neural network aircraft collision avoidance systems. In Proceedings of the Machine Learning for Engineering Modeling, Simulation, and Design Workshop at Neural Information Processing Systems , volume 2020, 2020.

S. Mitsch and A. Platzer. ModelPlex: verified runtime validation of verified cyber-physical system models. Formal Methods Syst. Des. , 49(1-2):33-74, 2016.

N. Noroozi, P. Karimaghaee, F. Safaei, and H. Javadi. Generation of Lyapunov functions by neural networks. In Proceedings of the World Congress on Engineering , volume 2008, 2008.

W. A. Olson. Airborne collision avoidance system X. Technical report, Massachusetts Institute of Technology Lexington Lincoln Lab, 2015.

I. Papusha, R. Wu, J. Brul´ e, Y . Kouskoulas, D. Genin, and A. Schmidt. Incorrect by Construction: Fine Tuning Neural Networks for Guaranteed Performance on Finite Sets of Examples. In 3rd Workshop on Formal Methods for ML-Enabled Autonomous Systems (FOMLAS) , pages 1-12, 2020.

A. Peruffo, D. Ahmed, and A. Abate. Automated and formal synthesis of neural barrier certificates for dynamical models. In J. F. Groote and K. G. Larsen, editors, Tools and Algorithms for the Construction and Analysis of Systems - 27th International Conference, TACAS 2021, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021, Luxembourg City, Luxembourg, March 27 - April 1, 2021, Proceedings, Part I , volume 12651 of LNCS , pages 370-388. Springer, 2021.

A. Platzer. Differential dynamic logic for hybrid systems. J. Autom. Reason. , 41(2):143-189, 2008.

A. Platzer. The complete proof theory of hybrid systems. In Proceedings of the 27th Annual IEEE Symposium on Logic in Computer Science, LICS 2012, Dubrovnik, Croatia, June 25-28, 2012 , pages 541-550. IEEE Computer Society, 2012.

A. Platzer. A complete uniform substitution calculus for differential dynamic logic. J. Autom. Reason. , 59(2):219-265, 2017.

A. Platzer. Differential hybrid games. ACM Trans. Comput. Log. , 18(3):19:1-19:44, 2017.

A. Platzer and Y. K. Tan. Differential equation invariance axiomatization. J. ACM , 67(1):6:16:66, 2020.

E. Prebet and A. Platzer. Uniform substitution for differential refinement logic. In C. Benzm¨ uller, M. J. Heule, and R. A. Schmidt, editors, IJCAR , volume 14740 of LNCS , pages 196-215. Springer, 2024.

C. Qin, K. D. Dvijotham, B. O'Donoghue, R. Bunel, R. Stanforth, S. Gowal, J. Uesato, G. Swirszcz, and P. Kohli. Verification of non-linear specifications for neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019.

Z. Qin, K. Zhang, Y. Chen, J. Chen, and C. Fan. Learning safe multi-agent control with decentralized neural barrier certificates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. J. Mach. Learn. Res. , 22:268:1-268:8, 2021.

S. M. Richards, F. Berkenkamp, and A. Krause. The Lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In 2nd Annual Conference on Robot Learning, CoRL 2018, Z¨ urich, Switzerland, 29-31 October 2018, Proceedings , volume 87 of Proceedings of Machine Learning Research , pages 466-476. PMLR, 2018.

M. S¨ alzer and M. Lange. Reachability is NP-complete even for the simplest neural networks. In P. C. Bell, P. Totzke, and I. Potapov, editors, Reachability Problems - 15th International Conference, RP 2021, Liverpool, UK, October 25-27, 2021, Proceedings , volume 13035 of LNCS , pages 149-164. Springer, 2021.

C. Schilling, M. Forets, and S. Guadalupe. Verification of neural-network control systems by integrating Taylor models and zonotopes. CoRR , abs/2112.09197, 2021.

S. Sharma, S. Roy, M. Soos, and K. S. Meel. Ganak: A scalable probabilistic exact model counter. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI) , 8 2019.

D. Shriver, S. G. Elbaum, and M. B. Dwyer. DNNV: A framework for deep neural network verification. In A. Silva and K. R. M. Leino, editors, Computer Aided Verification - 33rd International Conference, CAV 2021, Virtual Event, July 20-23, 2021, Proceedings, Part I , volume 12759 of LNCS , pages 137-150. Springer, 2021.

C. Sidrane, A. Maleki, A. Irfan, and M. J. Kochenderfer. OVERT: an algorithm for safety verification of neural network control policies for nonlinear systems. Journal of Machine Learning Research , 23(117):1-45, 2022.

A. Sogokon, S. Mitsch, Y. K. Tan, K. Cordwell, and A. Platzer. Pegasus: Sound continuous invariant generation. Form. Methods Syst. Des. , 58(1):5-41, 2022. Special issue for selected papers from FM'19.

A. Tarski. A Decision Method for Elementary Algebra and Geometry . University of California Press, Berkeley and Los Angeles, 1951.

S. Teuber, S. Mitsch, and A. Platzer. samysweb/NCubeV: v0.9. 10.5281/zenodo.13922169, Oct. 2024.

H. Tran, S. Bak, W. Xiang, and T. T. Johnson. Verification of deep convolutional neural networks using imagestars. In S. K. Lahiri and C. Wang, editors, Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part I , volume 12224 of LNCS , pages 18-42. Springer, 2020.

H. Tran, D. M. Lopez, P. Musau, X. Yang, L. V. Nguyen, W. Xiang, and T. T. Johnson. Star-based reachability analysis of deep neural networks. In M. H. ter Beek, A. McIver, and J. N. Oliveira, editors, Formal Methods - The Next 30 Years - Third World Congress, FM 2019, Porto, Portugal, October 7-11, 2019, Proceedings , volume 11800 of LNCS , pages 670-686. Springer, 2019.

H. Tran, X. Yang, D. M. Lopez, P. Musau, L. V. Nguyen, W. Xiang, S. Bak, and T. T. Johnson. NNV: the neural network verification tool for deep neural networks and learning-enabled cyberphysical systems. In S. K. Lahiri and C. Wang, editors, Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part I , volume 12224 of LNCS , pages 3-17. Springer, 2020.

S. Wang, H. Zhang, K. Xu, X. Lin, S. Jana, C. Hsieh, and J. Z. Kolter. Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 29909-29921, 2021.

Wolfram Research Inc. Mathematica, Version 14.0. Champaign, IL, 2024.

W. Xiang, H. Tran, X. Yang, and T. T. Johnson. Reachable set estimation for neural network control systems: A simulation-guided approach. IEEE Trans. Neural Networks Learn. Syst. , 32(5):1821-1830, 2021.

K. Xu, Z. Shi, H. Zhang, Y. Wang, K. Chang, M. Huang, B. Kailkhura, X. Lin, and C. Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

K. Xu, H. Zhang, S. Wang, Y. Wang, S. Jana, X. Lin, and C. Hsieh. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

H. Zhang, T. Weng, P. Chen, C. Hsieh, and L. Daniel. Efficient neural network robustness certification with general activation functions. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr´ eal, Canada , pages 4944-4953, 2018.

H. Zhang, J. Wu, Y. Vorobeychik, and A. Clark. Exact verification of ReLU neural control barrier functions. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023 , 2023.

X. Zhang, A. Solar-Lezama, and R. Singh. Interpreting neural network judgments via minimal, stable, and symbolic corrections. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr´ eal, Canada , pages 4879-4890, 2018.

X. Zhang, B. Wang, and M. Kwiatkowska. Provable preimage under-approximation for neural networks. In B. Finkbeiner and L. Kov´ acs, editors, Tools and Algorithms for the Construction and Analysis of Systems , pages 3-23, Cham, 2024. Springer Nature Switzerland.

Q. Zhao, X. Chen, Y. Zhang, M. Sha, Z. Yang, W. Lin, E. Tang, Q. Chen, and X. Li. Synthesizing ReLU neural networks with two hidden layers as barrier certificates for hybrid systems. In S. Bogomolov and R. M. Jungers, editors, HSCC '21: 24th ACM International Conference on Hybrid Systems: Computation and Control, Nashville, Tennessee, May 19-21, 2021 , pages 17:1-17:11. ACM, 2021.

A Additional Background

This section provides additional background on Differential Dynamic Logic and the status of various activation functions w.r.t. our classification in Table 2.

A.1 Differential Dynamic Logic

Differential dynamic logic ( dL ) [74-76,78] can analyze models of hybrid systems that are described through hybrid programs . The syntax of hybrid programs with Noetherian functions is defined by the following grammar, where the term e and formula Q are over real arithmetic with Noetherian functions:

The semantics of hybrid programs are defined by a transition relation over states in the set S , each assigning real values to all variables. For example, the assignment state transition relation is defined as J x ∶= e K = {( ν, ω ) ∈ S 2 ∣ ω = ν ν ( e ) x } where ν ν ( e ) x denotes the state that is equal to ν everywhere except for the value of x , which is modified to ν ( e ) . The other programs in the same order as in (5) describe nondeterministic assignment of x , test of a predicate Q , continuous evolution along the differential equation within domain Q , nondeterministic choice, sequential composition, and nondeterministic repetition. For a given program α , we distinguish between bound variables BV ( α ) and free variables FV ( α ) where bound variables are (potentially) written to, and free variables are read. The formula [ α ] ψ expresses that ψ is always satisfied after the execution of α and ⟨ α ⟩ ψ that there exists a state satisfying ψ after the execution of α . If a state ν satisfies ψ we denote this as ν ⊧ ψ . dL comes with a sound and relatively complete proof calculus [74,76,78] as well as the interactive theorem prover KeYmaera X [38].

ModelPlex. As demonstrated in the example, many safety properties for CPSs can be formulated through a dL formula with a loop in which α ctrl describes the (discrete) software, and α plant describes the (continuous) physical environment:

ϕ describes initial conditions, and ψ describes the safety criterion to be guaranteed when following the control-plant loop. To ensure that the behavior of controllers and plants in practice match all assumptions represented in the contract, ModelPlex shielding [69] synthesizes correct-byconstruction monitors for CPSs. ModelPlex can also synthesize a correct controller monitor formula ζ c (Definition 4). The formula ζ c encodes a relation between two states. If ζ c is satisfied, then the variable change from x i to x + i corresponds to behavior modeled by α ctrl , i.e. the change upholds the guarantee from Formula (6). To reason about this state relation, we say that a state tuple ( ν, ω ) satisfies a formula ζ (denoted as ( ν, ω ) ⊧ ζ ) iff ν ω ( x 1 ) ...ω ( x n ) x + 1 ...x + n ⊧ ζ (i.e. ν with the new state's value ω ( x i ) as the value of x + i for all i , satisfies ζ ).

Definition 4 (Correct Controller Monitor [69]) . A controller monitor formula ζ c with free variables x 1 , x + 1 , . . . , x n , x + n is called correct for the hybrid program controller α ctrl with bound variables x 1 , . . . , x n iff the following dL formula is valid: ζ c →⟨ α ctrl ⟩ ⋀ n i = 1 x i = x + i .

A.2 Classification of Activation Functions

Table 4 is meant to provide a brief overview on the status of various common activation functions w.r.t. the classes from Table 2. The chosen activation functions are representative examples from the comprehensive survey by Kunc et al. [63]. For piece-wise polynomial and Noetherian functions, any activation function represented by a polynomial over the activation functions presented in Table 4 is resp. piece-wise polynomial or Noetherian. For piece-wise linear functions, any linear combination is resp. piece-wise linear. This table does not claim to be comprehensive-many more activation functions can be classified into one of the categories.

B Mosaic

An overview of the algorithm is given in Figure 3: A piece-wise linear NN g is a function which maps from an input space (left) to an output space (right). We consider a part of the input space that

a The analysis of softmax can often be avoided by using the observation that its application does not change the order of outcomes, i.e. for classification-like tasks we can use the maximum before Softmax application instead.

is constrained by linear (orange) and nonlinear constraints (blue). As our query is not normalized, it may talk about multiple parts of the input space, e.g. in our case the two sets labeled with case 1 and case 2 . For any such part of the input space, say case 1 , we have a specification about unsafe parts of the output space which must not be entered (red dashed areas on the right). For classical open-loop NNV the task is then, given a single input polytope, to compute the set of reachable outputs for g and to check whether there exists an output reaching an unsafe output polytope. In our case the task is more complicated, because the input is not a polytope, but an arbitrary polynomial constraint. Moreover, for each polynomial input constraint (case 1 and case 2 in Figure 3) we may have different nonlinear unsafe output sets. In order to retain soundness, we overand underapproximate nonlinear constraints (see turquoise linear approximations around the blue and red nonlinear constraints in Figure 3). Once all nonlinear queries have approximations, we generate a mosaic of the input space where each azulejo (i.e. each input region) has its own normalized open-loop NNV-query (polytope over the input; disjunction of polytopes over the output). We must not only split between the two original cases (case 1 and case 2 ), but also between different segments of the approximating constraints (see the polytopes on the left in four shades of gray). Each normalized query has associated nonlinear constraints that must be satisfied, but cannot be checked via off-the-shelf open-loop NNV tools. Using the normalized, linear open-loop NNV queries we then instrument off-the-shelf tools to check whether any overapproximated unsafe region (turquoise on the right) is reachable. The amber colored regions represent parts of the outputs reachable by g : As can be seen by the two red dots, a reachable point within the overapproximated unsafe region may be a concrete unsafe output, or it may be spuriously unsafe due to the overapproximation. To retain completeness, we need to exhaustively filter spurious counterexamples. This is achieved by generalizing the counterexample to a region around the point in which the behavior of g is equivalent to a single affine transformation. Such counterexample regions always exist due to g 's piece-wise linearity. For example, in Figure 3 g affinely maps the input space triangle (left) to the upper output space triangle (in amber on the right). We can then check for concrete counterexamples in this region w.r.t. the affine transformation using an SMT solver. This avoids the need to encode the entire NN in an SMT formula. By excluding explored regions, we can then enumerate all counterexample regions and thus characterize the unsafe input set.

The approach is outlined in Algorithm 1 and proceeds in four steps, all of which will be presented in detail throughout the sections below: First, LINEARIZE generates approximate linearized versions for all nonlinear atoms of p on a bounded domain and enriches the formula with these constraints ( p o ). Next, MOSAIC generates a mosaic of p o 's input space where each azulejo (i.e. each input space region)

has an associated linear normalized query q l . Each q l is paired with an associated disjunctive normal form of nonlinear constraints q n . The disjunction over all q l ∧ q n is equivalent to the input query p o and the disjunction over all q l overapproximates LINEARIZE's input p . Each of the linear queries q l is processed by ENUM which internally uses an off-the-shelf open-loop NNV tool to enumerate all counterexample regions for a given query. Each counterexample region is defined through a polytope in the input space ι ⊂ R I and an affine mapping to the output space ω ∶ R I → R O that summarizes the NN's local behavior in ι . The procedure FILTER then checks whether a counterexample region is spurious using an SMT solver. This task is easier than searching nonlinear counterexamples directly since the NN's behavior is summarized by the affine mapping ω . Using the definitions from the following subsections, this procedure is sound and complete (see proof on page 28):

B.1 Linearization

The procedure LINEARIZE enriches each nonlinear atom a i of an open-loop NNV query with linear approximations. The approximations are always with respect to a value range R and we use overapproximations a i (for any state ν with ν ⊧ a i ∧ R it holds that ν ⊧ a i ) as well as underapproximations a i (for any state ν with ν ⊧ a i ∧ R it holds that ν ⊧ a i ). Essential to this component is the idea that LINEARIZE produces an equivalent formula: Approximate atoms do not replace, but complement the nonlinear atoms and the generation of concrete linear regions is left to the mosaic step (see Appendix B.2). LINEARIZE is defined as follows:

Definition 5 (Linearization) . LINEARIZE receives an open-loop NNV query p with nonlinear atoms a 1 , . . . , a k and value range R s.t. p → R is valid. It returns a query p ∧⋀ k i = 1 (( a i → a i ) ∧ ( a i → a i )) where a i ∈ FOL L R (resp. a i ∈ FOL L R ) are overapproximations (resp. underapproximations) of a i w.r.t. R .

We use an approximation procedure based on OVERT [88] while further approximating max / min terms within OVERT. This results in a disjunction of linear constraints (see below for details). As highlighted above, LINEARIZE produces equivalent formulas and therefore retains the relations between linear and nonlinear atoms (see proof on page 28):

Lemma 6 (Equivalence of Linearization) . Let p ∈ FOL R be some open-loop NNV query and p o be the result of LINEARIZE ( p ) . Then p is equivalent to p o .

Approximation. For concicenes we present our approximation approach for over-approximations. Our under-approximations are computed in the same manner, however lower and upper bound computation of terms is flipped in this case. We can approach the question of overapproximation construction from a perspective of models: For a given formula ζ , let J ζ K = { ν ∈ S∣ ν ⊧ ζ } be the set of models (i.e. states satisfying ζ ). We then obtain the following Lemma for the relation between overapproximations and model sets:

Lemma 7 (Supersets are Overapproximations) . Assume bounds B on all variables and a formula ζ . Another formula ζ o ∈ FOL L R is a linear overapproximation of ζ iff J B ∧ ζ K ⊆ J B ∧ ζ o K .

This presentation only considers the case of a polynomial constraint θ > 0 . Our approximation procedure begins by computing the relational approximation of θ using the OVERT algorithm [88]. By resolving intermediate variables introduced through OVERT, we obtain an approximation of the form θ pwl ≤ θ ≤ θ pwl where both bounds are piece-wise linear functions (i.e. linear real arithmetic with the addition of max and min operators). It then holds that J B ∧ θ > 0 K ⊆ q B ∧ θ pwl > 0 y . We now distinguish between univariate and multivariate piece-wise linear behavior: For univariate piece-wise linear behavior there is some variable v ∈ V ( θ pwl ) and some coefficient c ∈ R with terms θ 1 , θ 2 such that q B ∧ θ pwl > 0 y = q B ∧ θ 1 > 0 ∧ v > c y ∪ q B ∧ θ 2 > 0 ∧ v ≤ c y . In order to subsume piece-wise linear splits along variable v which are close to c , we construct the following overapproximation for a small ε > 0 resulting in two normalized queries:

For the multivariate cases we approximate the piece-wise linear behavior. In particular, we introduce a (to the best of our knowledge) novel, closed-form upper bound for the linear approximation of max terms:

Lemma 8 (Upper Bound for multivariate max ) . Let f, g ∶ R I + O → R be two linear functions, and let B ⊂ R I + O be a closed interval box, then: Assume x g ∶= arg max x ∈ B g ( x ) -f ( x ) and x f ∶= arg max x ∈ B f ( x ) -g ( x ) where f ( x f )-g ( x f ) and g ( x g )-f ( x f ) are both positive. Further, assume the following assignments with γ ∶= f ( x f ) -f ( x g ) -g ( x f ) + g ( x g ) :

In this case, it holds for all x ∈ B that: µf ( x )+( 1 -µ ) g ( x )+ c ≥ max ( f ( x ) , g ( x )) . In particular, it holds that

Proof. At first, the choices for µ and c may seem arbitrary, however they are actually the solution of the following set of equations:

The choice of µ and c ensures that we obtain a shifted convex mixture of the two linear functions that matches f and g at their points of maximal deviation. We can now prove that this shifted mixture is indeed larger than f or g at any point within B . Let us begin by proving that our bound is larger than g for x ∈ B . In the following each formula implies the validity of the formula above:

g ( x f ) ≤ f ( x f ) is trivially true since x f was specifically chosen this way. We also have to prove that our bound is bigger than f for x ∈ B :

Thus we obtain an upper bound for the function.

By applying OVERT followed by the univariate resolution and multivariate overapproximation up to saturation, we compute an overapproximation and underapproximation for each nonlinear atom. Subsequently, we append these new formulas to the original formula as defined in Definition 5.

Running Example. The turquoise constraints in Figure 3 visualize exemplary linearized constraints. For ACC one nonlinear atom is p rel -v 2 rel 2 B ≥ 0 . The formula accApprox ≡ p rel -100 2 2 B ≥ 0 ∧ v rel > 50 ∨ v rel ≤ 50 ∧ p rel -50 2 2 B ≥ 0 underapproximates the atom for v rel ∈ [ 0 , 100 ] . Wecan thus append the following formula to our query: ( accApprox → p rel -v 2 rel 2 B ≥ 0 ) . For v rel ∈ [ 0 , 100 ] this formula is always satisfied 4 .

B.2 Input Space Mosaics

The MOSAIC procedure takes a central role in the verification of nonlinear, non-normalized open-loop NNV queries. Classically, one uses DPLL(T) to decompose an arbitrary formula into conjunctions then handled by a theory solver. Open-loop NNV's crux is its use of reachability methods which do not lend themselves well to classic DPLL(T): Its usage would result in duplicate explorations of the same input space w.r.t. different output constraints which is inefficient. Therefore, we generalize DPLL(T) [40] through the MOSAIC procedure. The procedure receives a quantifier-free 5 , non-normalized open-loop NNV query and enumerates azulejos of the input space each with an associated normalized linear open-loop NNV query q l (conjunction over input atoms, disjunctive normal form over output atoms) and nonlinear atoms in disjunctive normal form q n . The input space is thus turned into a mosaic and the disjunction over all queries is equivalent to the input query. We can then obtain classical DPLL(T) by marking all atoms as linear input constraints. Our implementation of MOSAIC instruments a SAT solver on the Boolean skeleton of p as well as a real arithmetic SMT solver to restructure a formula in this way.

For a formula ζ ∈ FOL R , let sat-atoms ( ζ ) be the set of set of signed atoms such that for all A ∈ sat-atoms ( ζ ) it holds that A only contains atoms of ζ or its negations ( A ⊆ Atom ( ζ ) ∪ {¬ b ∣ b ∈ Atom ( ζ )} ). Further, we require for sat-atoms ( ζ ) that for any state ν it holds that ν ⊧ ζ iff there exists an A ∈ sat-atoms ( ζ ) such that ν ⊧ ⋀ a ∈ A a . Note that there may exist multiple such sets in which case we can choose an arbitrary one. For example, for ζ ≡ x > 0 ∨¬( y > 0 ) we could get sat-atoms ( ζ ) = {{ x > 0 } , {¬( x > 0 ) , ¬( y > 0 )}} . For a given formula ζ , we will call J ζ the set of input variables. We introduce the following notation for projection of sat-atoms on the set J ζ :

For example, reconsidering the previous example with J ζ = { x } we would get sat-atoms ( ζ )⇂ J ζ = {{ x > 0 } , {¬( x > 0 )}} . For a given open-loop NNV query p o and a given set of input variables J p o , MOSAIC then initially enumerates feasible combinations of linear input atoms (the azulejos) and for each such combination feasible combinations of mixed/output atoms are enumerated. This results in the following set:

Finally, for each q l ∈ S 1 , we can generate all possible combinations of nonlinear atoms S 2 = sat-atoms ( p o ∧ q l ) and generate their disjunction:

We achieve this by enumerating all satisfying assignments for the boolean skeleton of p o (i.e. the formula where all atoms are substituted by boolean variables) using an incremental SAT solver. This initially happens in the same manner as it is done for the classical version of DPLL(T). However, once a model is found, we fix the assignment of linear input-only atoms and enumerate all other satisfying assignments of linear atoms, generating the disjunctions within S 1 . For each conjunction we additionally enumerate possible assignments for the nonlinear atoms. Notably, through the encoding of LINEARIZE the procedure automatically knows which truth-combinations of a nonlinear constraint and its approximations may appear. We additionally provide information on linear dependencies between linear atoms to the SAT solver. All enumeration procedures are interleaved with calls to SMT solvers for linear and polynomial real arithmetic constraints, which check whether a given combination of constraints is indeed also satisfiable in the theory of real arithmetic (i.e. when interpreting the atoms as real arithmetic constraints instead of as boolean variables). In order to discard unsatisfiable solutions more quickly, we make use of unsatisfiability cores and conversely use a cache for satisfiable assignment combinations. We exploit partial models returned by the SAT solver to omit atoms which can (potentially) appear in both polarities for a given combination of constraints.

We show that the decomposition is correct (i.e. the disjunction over all queries is equivalent to the original query, see Proposition 9) and that it is minimal in the sense that the resulting azulejos do not overlap (see Proposition 10) with proofs on pages 28 and 29:

Proposition 9 (Correctness of Mosaic) . Let p be any open-loop NNV query. Let Q ⊂ FOL L R × FOL R be the set returned by MOSAIC ( p ) , then the following formula is valid: p ↔(⋁ ( q l ,q n )∈ Q ( q l ∧ q n ))

Proposition 10 (Flatness of Mosaic) . Let ( i 1 ∧ ⋁ j o 1 ,j ) , ( i 2 ∧ ⋁ j o 2 ,j ) be two linear queries enumerated by MOSAIC then i 1 ∧ i 2 is unsatisfiable.

We could use this approach to decompose a nonlinear formula into a set of normalized linear openloop NNV queries without approximation. MOSAIC then soundly omits all nonlinear constraints. However, this leads to many spurious counterexamples. Therefore, we add linear approximations (Appendix B.1) of atoms which are then automatically part of the conjunctions returned by MOSAIC.

Running Example. In the previous section we extended our query by a linear underapproximation. Our procedure generates an azulejo for the case where p rel -100 2 2 B ≥ 0 ∧ v rel > 50 is satisfied (implying p rel -v 2 rel 2 B ≥ 0 ) and the case where it is not. While the linear approximation is an edge of the mosaic tile, the original atom (the tile's 'painting' describing the precise constraint) would be part of the nonlinear disjunctive normal form. For each azulejo, the output conjunctions of accCtrlFml are enumerated. For OVERT's approximation with N = 1 , our implementation decomposes the ACC query into 20 normalized queries with up to 10 cases in the output constraint disjunction. Without MOSAIC each case would be treated as a separate reachability query leading to significant duplicate work.

Relation to DPLL(T) Abstracting away the real-arithmetic, the MOSAIC algorithm generates tuples of normalized open-loop NNV queries and disjunctive normal forms that are satisfiable w.r.t. a theory solver T . The algorithm itself interleaves SAT-based reasoning about a boolean abstraction (annotated with information on whether an atom is linear and/or an input constraint) and theory solver invocations. We can now consider the case where all atoms (independent of their concrete contents and the theory T ) are annotated as linear input constraints: In this case MOSAIC merely returns a mosaic of this 'input' space where each azulejo corresponds to a conjunction of atoms that is satisfiable w.r.t. to the theory solver T and the disjunction over all those conjunctions is then once again equivalent to the original formula - corresponding to DPLL(T)'s behavior.

B.3 Counterexample Generalization and Enumeration

The innermost component of our algorithm enumerates all counterexample regions (ENUM). To this end, ENUM requires an algorithm which generalizes counterexample points returned by open-loop NNV to regions (GENERALIZE). For each such counterexample region we can then check if there exist concrete violations of the nonlinear constraints (FILTER). We begin by explaining GENERALIZE which converts a counterexample point returned by open-loop NNV into a counterexample region. The key insight for this approach is that a concrete counterexample ( z 0 , x 0 ) returned by an open-loop NNV tool induces a region of points with similar behavior in the NN. A concrete input z 0 induces a fixed activation pattern for all piece-wise linear activations within the NN in a region ι around z 0 . Consider the first layer's activation function f ( 1 ) : f ( 1 ) can be decomposed into linear functions f i and so is a sum of affine transformations A i z 0 + b i which are active iff q i ( z 0 ) is true. We can then describe f ( 1 ) 's local behavior around z 0 as the linear combination of all affine transformations active for z 0 . This sum is itself an affine transformation. By iterating this approach across layers, we obtain a single affine transformation ω describing the NN's behavior in ι . The regions returned by GENERALIZE are then defined as follows:

Definition 11 (Counterexample Region) . For a given open-loop NNV query q and piece-wise linear NN g , let ( z 0 , x 0 ) ∈ R I × R O be a counterexample, i.e. x 0 = g ( z 0 ) and q ( z 0 , x 0 ) holds. The counterexample region for z 0 is the maximal polytope ι ⊂ R I with a linear function ω s.t. z 0 ∈ ι and ω ( z ) = g ( z ) for all z ∈ ι .

Star Sets [12, 93] can compute ( ι, ω ) by steering the Star Set according to the activations of z 0 . As the number of counterexample regions is exponentially bounded by the number of piece-wise linear nodes, we can use GENERALIZE for exhaustive enumeration. This is only a worst-case bound due to the NP-completeness of NN verification [58,84]. In practice, the number of regions is much lower

since many activation functions are linear in all considered states. While a given counterexample region certainly has a point violating the linear query that was given to the open-loop NNV tool, it may be the case that the counterexample is spurious , i.e. it does not violate the nonlinear constraints. However, we can use the concise description of counterexample regions to check whether this is the case: The function ω describes the NN's entire behavior within ι as a single affine transformation and is thus much better suited for SMT-based reasoning. This SMT-based check is performed by FILTER based on the following insight:

Lemma 12 (Counterexample Filter) . Let ( q l , q n ) be a tuple returned by MOSAIC . A counterexample region ( ι, ω ) for q l is a counterexample region for q l ∧ q n iff the formula η ≡ ( q l ( z, x + ) ∧ q n ( z, x + ) ∧ z ∈ ι ∧ x + = ω ( z )) is satisfiable.

See proof on page 29. The size of the formula η only depends on q l , q n , I , and O and, crucially, is independent of the size and architecture of the NN. In practice, even x + can be eliminated (substitute linear terms of ω ( x + ) ).

Based on these insights, the last required component is a mechanism for the exhaustive enumeration of all counterexample regions (denoted as ENUM). There are two options for ENUM: Either we use geometric path enumeration [12,93] to enumerate all counterexample regions (used for the evaluation) or we instrument arbitrary complete off-the-shelf open-loop NNV tools for linear queries through Algorithm 2. We define ENUM as follows:

Definition 13 (Exhaustive Counterexample Generation) . An exhaustive enumeration procedure ENUM receives a linear, normalized open-loop NNV query q and a piece-wise linear NN g and returns a covering E of counterexample regions, i.e. E satisfies { z ∈ R I ∣ q ( z, g ( z ))} ⊆ ⋃ ( ι,ω )∈ E ι.

C Proofs

C.1 Proofs for Section 3

This subsection proves the soundness of the approach outlined in Section 3. This result is achieved by proving that a concrete NNCS refines [65,79] an abstract hybrid program. The approach can either be applied by first proving safety for a suitable dL model or by reusing results from the dL literature (both demonstrated Section 5 and appendix E). In our proofs for Section 3 we show a slightly more general version of the result from Theorem 2 (see Theorem 21). To this end, we formally define a controller description as follows:

Definition 14 (Controller Description) . Let α ctrl be some hybrid program with free variables FV ( α ctrl ) = { z 1 , . . . , z m } and bound variables BV ( α ctrl ) = { x 1 , . . . , x n } , which overlap if a variable is both read and written to. A controller description κ ∈ FOL N R for α ctrl is a formula with free variables FV ( α ctrl ) ∪ { x + ∣ x ∈ BV ( α ctrl )} such that the following formula is valid: ∀ z 1 . . . z m ∃ x + 1 . . . x + n κ .

Based on Controller Descriptions we can then show that such controller descriptions exist for all piece-wise Noetherian NNs:

Lemma 15 (Existence of κ g ) . Let g ∶ R I → R O be a piece-wise Noetherian NN. There exists a controller description κ g ∈ FOL N R with input variables z 1 , . . . , z I and output variables x + 1 , . . . , x + O s.t.

( ν ( x + 1 ) , . . . , ν ( x + O )) = g ( ν ( z 1 ) , . . . , ν ( z I )) iff ν ⊧ κ g , i.e. κ g 's satisfying assignments correspond exactly to g 's in-out relation.

Proof. Previous work showed how to encode piece-wise linear NNs through real arithmetic SMT formulas (see e.g. [36, 72]). Each output dimension of an affine transformation can be directly encoded as a real arithmetic term. For a given output-dimension of a piece-wise Noetherian activation function we have to encode a term ∑ s i = 1 1 q i ( x ) f i ( x ) with Noetherian functions f i and predicates q i over real arithmetic with Noetherian functions. To this end, we can introduce fresh variables v 1 , . . . , v s where we assert the following formula for each v i :

The activation function's result then is the sum ∑ s i = 1 v i . By existentially quantifying all intermediate variables of such encodings, we obtain a real arithmetic formula that only contains input and output variables and satisfies the requirements of the Lemma.

If we assign x + 1 . . . x + O to the values provided by g ( z 1 , . . . , z I ) for a given z 1 . . . z I , the formula κ g is satisfied. Therefore, ∀ z 1 . . . z I ∃ x + 1 . . . x + O κ g . is valid.

When replacing α ctrl by an NN g , free and bound variables of α ctrl must resp. match to input and output variables of g . Based of a description κ g , we then construct a hybrid program that behaves as described by κ g : We now formalize the idea of modeling a given NN g through a hybrid program which behaves identically to g .We show that such nondeterministic mirrors exist for all piece-wise Noetherian NN g :

Definition 16 (Nondeterministic Mirror for κ g ) . Let α ctrl be some hybrid program with bound variables BV ( α ctrl ) = { x 1 , . . . , x n } . For a controller description κ g with variables matching to α ctrl , κ g 's nondeterministic mirror α refl is defined as:

Lemma 17 (Existence of α g ) . For any piece-wise Noetherian NN g ∶ R I → R O there exists a nondeterministic mirror α g that behaves identically to g .

Formally, α g only has free variables z and bound variables x and for any state transition ( ν, µ ) ∈ J α g K : µ ( x ) = g ( ν ( z )) ( x and z vectors of dimension I and O )

Proof. Based on Lemma 15 we can construct a controller description κ g ∈ FOL N R for g which we can turn into a hybrid program through the nondeterministic mirror α refl ( κ g ) .

Similarly to the more general notion of a controller description, Theorem 21 also permits a slightly more general version of a state space restriction instead of an inductive invariant. Formally, this notion is described as a state reachability formula:

Definition 18 (State Reachability Formula) . A state reachability formula ζ s with free variables z 1 , . . . , z m is complete for the hybrid program ( α ctrl ; α plant ) ∗ with free variables z 1 , . . . , z m and initial state ϕ iff the following dL formula is valid where ( ζ s ) z + 1 ...z + m z 1 ...z m represents ζ s with z + i substituted for z i for all 1 ≤ i ≤ m :

There is usually an overlap between free and bound variables, i.e. z 1 , . . . , z m may contain variables later modified by the hybrid program. Our definition requires that for any program starting in a state satisfying ϕ , formula ζ s is satisfied in all terminating states. ζ s thus overapproximates the program's reachable states. In particular, inductive invariants (i.e. for ϕ →[ α ∗ ] ψ a formula ζ s.t. ϕ → ζ and ζ →[ α ] ζ ) are state reachability formulas:

Lemma 19 (Inductive Invariants are State Reachability Formulas) . If ζ is an inductive invariant of ϕ →[( α ctrl ; α plant ) ∗ ] ψ , ζ is a state reachability formula.

Proof. We begin by recalling the requirement for ζ to be a state reachability formula:

Let ζ be an inductive invariant for some contract of the form given above. First, consider that for any state satisfying the left side of the formula above it holds that there exists some k such that ϕ ∧ ⟨( α ctrl ; α plant ) k ⟩ ⋀ m i = 1 z i = z + i is satisfied by the same state (this follows from the semantics of loops in dL ). If we can prove that any such state also satisfies ( ζ s ) z + 1 ,...,z + m z 1 ,...,z m we obtain that ζ is a state reachability formula. We proceed by induction: First, consider k = 0 in this case z i has the same value as z + i for all i . The formula then boils down to ϕ → ζ . This formula is guaranteed to be valid by the first requirement of inductive invariants. Next, we now assume that we already proved that ζ holds for k loop iterations and show it for k + 1 . Since we assume some state u k + 1 that satisfies ϕ ∧ ⟨( α ctrl ; α plant ) k + 1 ⟩ ⋀ m i = 1 z i = z + i , there also has to be some state u k satisfying ϕ ∧ ⟨( α ctrl ; α plant ) k ⟩ ⋀ m i = 1 z i = z + i . However, we already know for u k that it satisfies ( ζ s ) z + 1 ,...,z + m z 1 ,...,z m . Since u k + 1 is reachable from u k through the execution of α ctrl ; α plant we know that u k + 1 satisfies ( ζ s ) z + 1 ,...,z + m z 1 ,...,z m (this corresponds to the property ζ →[ α ] ζ of inductive invariants).

As α g and g mirror each other, we can reason about them interchangeably. Our objective is now to prove that α g is a refinement of α ctrl. To this end, we use the shielding technique ModelPlex [69] to automatically generate a correct-by-construction controller monitor ζ c for α ctrl. The formula ζ c then describes what behavior for α g is acceptable so that α g still represents a refinement of α ctrl. As seen in Section 3, we do not require that α g adheres to ζ c on all states, but only on reachable states. For efficiency we therefore allow limiting the analyzed state space to an inductive invariant ζ s (i.e. for ϕ →[ α ∗ ] ψ a formula ζ s s.t. ϕ → ζ s and ζ s →[ α ] ζ s ). Despite the infinite-time horizon, the practical use of our approach often faces implementations with a limited value range for inputs and outputs (e.g., v rel within the ego-car's physical capabilities). Only by exploiting these ranges, is it possible to prove safety for NNs that were only trained on a particular value range. To this end, we allow specifying value ranges (i.e. intervals) for variables. We define the range formula R ≡ ⋀ v ∈ V ( P ) R ( v ) ≤ v ≤ R ( v ) for lower and upper bounds R and R . Using R , we specialize a contract to the implementation specifics by adding a range check to α plant. The safety results for the original contract can be reused:

Lemma 20 (Range Restriction) . Let ϕ →[( α ctl ; α plant ) ∗ ] ψ be a valid dL formula. Then the formula C 2 ≡ ( ϕ ∧ R →[( α ctrl ; ( α plant ; ? ( R ))) ∗ ] ψ ) with ranges R is valid and R is an invariant for C 2 .

Proof. We use the notation C 1 ≡ ( ϕ →[( α ctl ; α plant ) ∗ ] ψ ) . We begin by showing that the validity of C 1 implies the validity of C 2 . Intuitively, this follows from the fact that the introduced check ? ( R ) only takes away states. Let I be a loop invariant such that ϕ → I , I → ψ and I → [ α ctrl ; α plant ] I (assumed due to the validity of C 1 ). Then clearly, it also holds that ϕ ∧ R → I . Furthermore, I →[ α ctrl ; α plant ; ? ( R )] I can be reduced to I →[ α ctrl ; α plant ] ( R → I ) which we can shown through the monotonicity rule of dL . Since we already know that I → ψ , it follows that C 2 is valid, because I is a loop invariant.

Including R into ζ s allows us to exploit the range limits for the analysis of α g . Our objective is to use open-loop NNV techniques to check whether g (and therefore α g ) satisfies the specification synthesized by ModelPlex. To this end, we use a nonlinear NN verifier to prove safety of our NNCS:

Theorem 21 (Safety Criterion) . Let ζ c and ζ s be controller and state reachability formulas for a valid dL contract C ≡ ( ϕ →[( α ctl ; α plant ) ∗ ] ψ ) . For any controller description κ , if

is valid, then the following dL formula is valid as well:

Let v ∈ S be some arbitrary state. We need to show that any such v satisfies Formula (10):

Proof. Assume the validity of

To this end, assume v ⊧ ϕ , we prove that ψ as well as ζ s is upheld after any number of loop iterations by induction on the number n of loop iterations.

Base Case: n = 0

In this case, the only state we need to consider is v since there were no loop iterations. We know through the validity of C that ϕ → ψ . Thus, v ⊧ ψ . Furthermore, we recall the requirement of a state reachability formula:

By extending v such that all z + i have the same values as the corresponding z i , we get a state that satisfies this formula. Consequently, v ⊧ ζ s .

Inductive Case: n →( n + 1 )

In the induction case, we know that for all

it holds that ˜ v 0 ⊧ ψ and ˜ v 0 ⊧ ζ s .

We must now prove the induction property for any state reachable from ˜ v 0 through execution of the program ( α refl ( κ ) ; α plant ) . For any ˜ v 1 ∈ S such that ( ˜ v 0 , ˜ v 1 ) ∈ J x + 1 ∶= ∗ ; . . . ; x + n ∶= ∗ ; ? ( κ ) ; K (by the definition of κ we know that such a state exists) we know that ˜ v 1 ⊧ κ . According to the coincidence lemma [69, Lemma 3], since ζ s does not concern the x + variables, it is then true that

Through the validity of Formula (9) assumed at the beginning, we then know that it must be the case that ˜ v 1 ⊧ ζ c . More specifically, this means that for any ˜ v 2 ∈ S with

it holds that ( ˜ v 0 , ˜ v 2 ) ⊧ ζ c . By definition this implies that ( ˜ v 0 , ˜ v 2 ) ∈ J α ctrl K .

Through the semantics of program composition in hybrid programs it follows that subsequently for any ˜ v 3 ∈ S with ( ˜ v 0 , ˜ v 3 ) ∈ q α refl ( κ ) ; α plant y it holds that ( ˜ v 0 , ˜ v 3 ) ∈ q α ctrl ; α plant y .

In summary, this means that for any ( ˜ v 0 , ˜ v 2 ) ∈ J α refl ( κ ) K it holds that ( ˜ v 0 , ˜ v 2 ) ∈ J α ctrl K .

We also know that ( v, ˜ v 0 ) ∈ r ( α ctl ; α plant ) ∗ z and that ( ˜ v 0 , ˜ v 3 ) ∈ q α ctrl ; α plant y . Since this implies ( v, ˜ v 3 ) ∈ r ( α ctl ; α plant ) ∗ z , i.e. there is a trace of states from v to ˜ v 3 , and since we already know that v ⊧ ϕ , ( v, ˜ v 3 ) satisfy the right side of Formula (8). Since Formula (8) must be valid we get that ˜ v 3 ⊧ ζ s . Consequently, we know through the validity of C that:

This concludes the induction proof and thereby also the proof of Theorem 21

Definition 22 (Nonlinear Neural Network Verifier) . A nonlinear neural network verifier accepts as input a piece-wise Noetherian NN g and nonlinear open-loop NNV query p with free variables z 1 , . . . , z I , x + 1 , . . . , x + O . The tool must be sound , i.e. if there is a z ∈ R I satisfying p ( x, g ( z )) then the tool must return sat . A tool that always returns unsat if no such z ∈ R I exists is called complete .

Lemma 23 (Soundness w.r.t Controller Descriptions) . Let κ g be a controller description for a piece-wise Noetherian NN g .

Further, let C ≡ ( ϕ →[( α ctl ; α plant ) ∗ ] ψ ) be a contract with controller monitor ζ c ∈ FOL R and inductive invariant ζ s ∈ FOL R where the free and bound variables respectively match g 's inputs and outputs. If a sound Nonlinear Neural Network Verifier returns unsat for the query p ≡ ( ζ s ∧¬ ζ c ) on g then: 1. κ g ∧ ζ s → ζ c is valid; 2. ϕ →[( α refl ( κ g ) ; α plant ) ∗ ] ψ is valid.

Proof. Let all variables be defined as above. We assume that the nonlinear NN verifier did indeed return unsat . By definition this means that there exists no z ∈ R I such that p ( z, g ( z )) = ⊺ . Due to the formalization of κ g (see Lemma 15), this means there exists no z ∈ R I such that ζ s ∧ κ g ∧ ¬ ζ c . Among all states consider now any state v such that v ⊭ ζ s ∧ κ g . In this case v ⊧ ζ s ∧ κ g → ζ c vacuously. Next, consider the other case, i.e. a state v such that v ⊧ ζ s ∧ κ g . In this case it must hold that v ⊭ ¬ ζ c . So v ⊧ ζ c . Therefore, v ⊧ ζ s ∧ κ g → ζ c . This means that Formula (9) is satisfied by all states and, therefore, valid which proves the first claim. Theorem 21 then implies the safety guarantee stated in Formula (10) for κ g which proves the second claim.

While we lay the foundation for analyses on piece-wise Noetherian NNs, a subclass is decidable:

Lemma 24 (Decidability for Polynomial Constraints) . Given a piece-wise polynomial NN g , the problem of verifying ( ζ s ∧¬ ζ c ) ∈ FOL R for g is decidable.

Proof. The problem of verifying ( ζ s ∧¬ ζ c ) ∈ FOL R for g is the same as proving the validity of the formula ζ s ∧ κ g → ζ c (see Lemma 23). For piece-wise polynomial NN this formula is in FOL R and the validity problem is thus decidable.

Proof of 2. The formula ϕ → [( α g ; α plant ) ∗ ] ψ is equivalent to ϕ → [( α refl ( κ g ) ; α plant ) ∗ ] ψ as α g behaves precisely like α refl ( κ g ) . The result immediately follows from Lemma 23

C.2 Proofs for Section 4

Proof of 3. This proof assumes the results from Appendices B.1 to B.3. We begin by proving soundness, i.e. if Algorithm 1 returns safe, then there exists no counterexample. Consider a counterexample region found by ENUM. Lemma 12 tells us that this counterexample can only be concrete if formula ν is satisfied. This is the check performed by FILTER. Thus, Algorithm 1 only skips a counterexample region if it is not concrete. Further, we know through Definition 13 that all counterexamples are returned by the procedure for a given query q l . Further, we know that the disjunction over all q l ∧ q n returned by MOSAIC is equivalent to its input p o (Proposition 9) and thus the disjunction over all q l is an over-approximation thereof. Finally, LINEARIZE returns a formula p o which is equivalent to the input query p (Lemma 6). Therefore, any counterexample of p must also be a counterexample of some q l returned by MOSAIC. Consequently, we iterate over all possible counterexamples and only discard them if they are spurious. Thus, our algorithm is sound.

We now turn to the question of completeness, i.e. we prove that any time Algorithm 1 returns unsafe then there is indeed a concrete counterexample of p . First, remember that Lemma 6 ensures that p and p o are equivalent. Furthermore, Proposition 9 ensures that the disjunction over all q l ∧ q n generated by MOSAIC is equivalent to p o . Assume we found a counterexample. The algorithm will return unsafe iff FILTER returns that the counterexample is concrete. According to Lemma 12 we know that this is only the case if there is indeed a concrete counterexample for q l ∧ q n . Since this counterexample then also satisfies p o (see above), we only return unsafe if FILTER found a concrete counterexample for p . As real arithmetic is decidable and all other procedures in the algorithm terminate as well, this yields a terminating, sound and complete algorithm.

Proof of 6. Let p ∈ FOL R be some nonlinear open-loop NNV query and a 1 , . . . , a k be the nonlinear atoms in p . From the definition of LINEARIZE, we know that p o has the form p ∧ ⋀ k i = 1 (( a i → a i ) ∧ ( a i → a i )) . By definition, it holds for approximations a i , a i that for any state ν with ν ⊧ a i ∧ R it also holds that ν ⊧ a i (resp. for any state ν with ν ⊧ a i ∧ R it also holds that ν ⊧ a i ). Consequently, the formulas R →( a i → a i ) and R →( a i → a i ) are valid for all a i . Let ν be a state such that ν ⊧ p . Then, by definition ν ⊧ R and due to the above mentioned validity it therefore holds that ν ⊧ a i → a i and ν ⊧ a i → a i . Therefore, ν ⊧ p o . Conversely, for any state with ν ⊧ p o it also holds that ν ⊧ p .

Proof of 9. We begin by considering the case where some state ν satisfies ⋁ ( q l ,q n )∈ Q q l ∧ q n . By definition, this means that there exists some ( q ∗ l , q ∗ n ) ∈ Q such that ν ⊧ q l ∧ q n . Through the definition of the set S 1 in Appendix B, we know that q ∗ l contains a conjunction over linear input atoms i ∗ l . Let o ∗ l ∈ sat-atoms ( p ∧ i ∗ l ) be the set of mixed/output atoms such that ν ⊧ ⋀ b ∈ o ∗ b . Further, since ν ⊧ q n ,

we know there also exists an A ∗ ∈ sat-atoms ( p ∧ q l ) such that ν ⊧ ⋀ a ∗ ∈ A ∗ a ∗ . Through the definition of sat-atoms and its projection we then know that A ∗ ∪ i ∗ ∪ o ∗ ∈ sat-atoms ( p ) . Consequently, it must hold that ν ⊧ p .

Consider now the other direction where for some state ν it holds that ν ⊧ p . By definition of sat-atoms, its projection and S 1 we know that there must exist some i ∗ ∈ sat-atoms ( p )⇂ J p such that ν ⊧ i ∗ . Moreover, there must exist an o ∈ sat-atoms ( p ∧ i ∗ ) such that ν ⊧ ⋀ b ∈ o ∗ b . Finally, since ν ⊧ i ∗ ∧ o , there must exist an A ∗ ∈ sat-atoms ( p ∧ i ∗ ∧ o ) such that ν ⊧ ⋀ a ∗ ∈ A ∗ a ∗ Consequently, there exists a ( q l , q n ) ∈ Q such that ν ⊧ q l ∧ q n and therefore ν ⊧ ⋁ ( q l ,q n )∈ Q q l ∧ q n .

Proof of 10. Assume there were two linear queries ( i 1 ∧ ⋁ j o 1 ,j ) and ( i 2 ∧ ⋁ j o 2 ,j ) such that i 1 ∧ i 2 had a model. By definition, each set A ∈ sat-atoms ( p ) must contain each atom of p or its negation. Consider now the projection sat-atoms ( p )⇂ J p from which we obtain all i s (in particular i 1 and i 2 ): Since i 1 and i 2 contain the same set of atoms, it must be the case that for some atom a ∈ i 1 , it holds that ¬ a ∈ i 2 or vice versa (otherwise, the two would be identical). Through the law of the excluded middle, we get that a ∧¬ a is unsatisfiable, and thus i 1 ∧ i 2 is unsatisfiable.

Proof of 12. Assume some ( ι, ω ) is indeed a counterexample region for q l ∧ q n . In this case, we know that there is some z ∈ ι such that with x + = g ( z ) we get q l ( z, x + ) ∧ q n ( z, x + ) . However, by definition of counterexample regions we also know that g ( z ) = ω ( z ) . Therefore, the assignments of z and x + satisfy η . Next, consider the other direction. I.e. we assume we have a satisfying assignment for η . By definition we know that for the given assignment of z it holds that x + = g ( z ) = ω ( z ) . Therefore, z, x + respect the neural network and satisfy q l ∧ q n , which are the two requirements for a counterexample.

D Adaptive Cruise Control

Information on the dL model. The controller α ctrl has three nondeterministic options: it can brake with -B (no constraints), set relative acceleration to a rel = 0 (constraint accCtrl 0 ) or choose any value in the range [-B,A ] (constraint accCtrl 1 ). The constraints for the second and third action are as follows:

We can prove the safety of this control envelope for the following initial condition which is also the loop invariant:

The right-hand side of the invariant/initial condition ensures that the distance is still large enough to avoid a collision through an emergency brake ( a rel = -B ). Based on these foundations, the full specification for the NN generated by VerSAILLE reads as follows:

For our verification, we set T = 0 . 1 (note that this is a bound on the frequency of control decisions, not a time horizon) and A = B = 100 .

E Extended Evaluation

We implemented our procedure in a new tool 6 called N 3 V. Due to the widespread use of ReLU NNs, N 3 V focuses on the verification of generic open-loop NNV queries for such NNs, but could be extended in future work. Our tool is implemented in Julia [16] using nnenum [10,12] for open-loop NNV, PicoSAT [17,19] and Z3 [32,52]. Our evaluation aimed at answering the following questions:

Q1 Can N 3 V verify infinite-time horizon safety or exhaustively enumerate counterexample regions for a given NNCS?

Q2 Does our approach advance the State-of-the-Art?

Q3 Does our approach scale to complex real-world scenarios such as ACAS X?

The case studies comprised continuous and discrete control outputs. (Q3) is answered in the paper's main evaluation (see Section 5; the remaining questions are discussed below. Times are wall-clock times on a 16 core AMD Ryzen 7 PRO 5850U CPU (N 3 V itself is sequential while nnenum uses multithreading).

E.1 Verification of Adaptive Cruise Control

We applied our approach to the previously outlined running example. To this end, we trained two NNs using PPO [82]: ACC contains 2 layers with 64 ReLU nodes each while ACC Large contains 4 layers with 64 ReLU nodes each. Our approach only analyzes the hybrid system once and reuses the formulas for all future verification tasks (e.g. after retraining). We analyzed both NNs and a third one (see below for details) using N 3 V for coarser and tighter approximation settings (using OVERT's setting N ∈ { 1 , 2 , 3 } ) on the value range ( p rel , v rel ) ∈ [ 0 , 100 ]×[-200 , 200 ] . The analyses took 47 to 300 seconds depending on the NN and approximation. The runtimes show mixed results for tighter approximations: While tighter approximations (i.e. a higher N ) sometimes improves performance (e.g. for ACC Large retrained), it can also harm performance (e.g. as seen for ACC Large ). We suspect that this is a combination of two factors. First, finer approximations yield a larger number of queries which may increase the overall overhead. Secondly, our adjustments to OVERT's approximation using approximation of piece-wise linearities (see Appendix B.1) may in some cases worsen the approximation in comparison to a lower N . We leave a more fine-grained analysis of approximation techniques to future work and focus our analysis on approximations with N = 1 . Across all NNs we find that approximation helps: The first row shows performance when omitting the approximated constraints in the open-loop NNV query and uniformly performs worse than an N = 1 approximation. N 3 V finds the NN ACC Large to be unsafe and provides an exhaustive characterization of all input space regions with unsafe actions.

Information on the counterexamples found for ACC Large . Figure 6 shows the input state where the x-axis represents possible values for p rel and the y-axis represents possible values for v rel. The orange line represents the edge of the safe state space, i.e. all values below the orange line are outside the reachable state space of the contract. The red areas represent all parts of the state space where N 3 V found concrete counterexamples for the checked controller monitor formula. Furthermore, the plot contains two lines representing the system's evolution over time when started at certain initial states. In particular, we observe one trajectory leading to a crash due to an erroneous decision in the red area around p rel = 5 , v rel = -25 . This concrete counterexample was found by sampling initial states from the regions provided by N 3 V.

Information on runtimes. The runtimes can be seen in Table 5 where #Filtered corresponds to the number of counterexample regions that were found to be spurious for ACC Large . Comparing linear and N = 1 performance we see that approximation especially helps for larger NNs ( ACC vs ACC Large ). This is the case because the overapproximate constraints can filter out numerous counterexample regions, which would otherwise have to be processed by the FILTER procedure. This effect is less significant for smaller networks where the time for overapproximation construction

-30

Velocity

Unsafe

StateSpaceBound

-60

Successtrajectory(AcC_Large)

Failure trajectory(ACC_Large)

-90

-120

0

25

50

75

100

Position

takes longer in comparison to the NN analysis time and less counterexamples are generated due to the lower number of ReLU nodes.

Further Training. Approx. 3% of ACC Large 's inputs resulted in unsafe actions which demonstrably resulted in car crashes. We performed a second training round on ACC Large where we initialized the system within the counterexample regions for a boosted p ≈ 13% of all runs (choosing the best-performing p ). By iterating this approach twice, we obtained an NN which was safe except for very small relative distances (for ( p rel , v rel ) ∈ [ 0 , 0 . 08 ] × [-2 , 0 . 1 ] ). N 3 V certifies the safety outside this remaining region (see column ACC Large retrained) which can be safeguarded using an emergency braking backup controller. Notably, this an a priori guarantee is for an arbitrarily long trip.

Results on Q1. Our tool N 3 V is capable of verifying and refuting infinite-time horizon safety for a given dL contract. The support for exhaustively enumerating counterexamples can help in guiding the development of safer NNCSs.

E.2 Comparison to Other Techniques

Although Closed-loop NNV tools focus on finite-time horizons, we did compare our approach with the tools from ARCH Comp 2022 [66] (a superset of ARCH Comp 2023 [67]) on ACC. We began by evaluating safety certification on a small subset of the input space of ACC Large ( 0 . 009% of the states verified by N 3 V) for multiple configurations of each tool (see Table 6). Only NNV was capable of showing safety for 0.1 seconds (vs. time unbounded safety) while taking vastly longer for the tiny fraction of the state space.

Comparison to NNV . Weperformed a more extensive comparison to NNV by attempting to prove with NNV that the NNCS has no trajectories leading from within to outside the loop invariant. This would witness infinite-time safety. Due to a lack of support for nonlinear constraints, we approximate the regions. Over-approximating the invariant as an input region trivially produces unsafe trajectories, thus we can only under-approximate. Notably, this immediately upends any soundness or completeness guarantees (it does not consider all possible NN inputs nor all allowed actions). We apply an

Table 6: Comparison of verification tools for NNCSs on the ACC Benchmark: Share of state space analyzed and best results of each tool.

interval-based approximation scheme similar to OVERT (detailed in the subsequent paragraph). This scheme is parameterized by p rel 's step size ( σ ), v rel's distance to the invariant ( ε ) and the step size for approximating the unsafe set ( ρ ). The right configuration of ( σ, ε, ρ ) is highly influential, but equally unclear. For example, with σ = 0 . 25 , ϵ = 5 and ρ = 1 we can 'verify' not only the retrained ACC Large NN for 2 ≤ p rel ≤ 3 , but also the original, unsafe ACC Large despite concrete counterexamples . This is a consequence of a coarse approximation, but also a symptom of a larger problem: Neither overnor under-approximation yields useful results. In particular, discarding inputs close to the invariant's edge equally removes states most prone to unsafe behavior (see Figure 6 in Appendix E.1).

Approximation Scheme employed for NNV. We consider input space stripes of width σ > 0 , i.e. ( p rel , v rel ) ∈ [ p 0 , p 0 + σ ] × [- √ 2 Ap rel + ϵ, TB 2 ] ( v rel is bounded through the minimally allowed velocity and the maximal velocity that can still decrease p rel). While σ determines the granularity of the under-approximation of NN inputs, ϵ > 0 discards velocities too close to the loop invariant which cannot be proven using an underapproximation and must be non-zero to prove any system. For each stripe we compute the smallest reachable position p ∗ and compute a piece-wise linear overapproximation of the negated loop invariant p rel < v 2 rel / ( 2 A ) on the interval [ p ∗ , p 0 ] using an approach conceptually similar to OVERT [88]. We determine the number of pieces through a step size ρ > 0 , i.e. the interval [ p ∗ , p ∗ + ρ ] will have a different line segment than the interval [ p ∗ + ρ, p ∗ + 2 ρ ] . As the negation of the loop invariant (the orange line in Figure 6) is non-convex, we integrated an iterative check for disjunctions of unsafe sets into the verification procedure of NNV .

Comparison with DNNV (Table 7) As DNNV [87] does not support nonlinear properties, a direct comparison to the tool is impossible. However, we improve upon one important feature implemented in DNNV, namely query normalization. By exporting the boolean skeleton generated by MOSAIC, we can use projected model counting [86] to estimate the number of propositionally satisfiable conjunctions over linear constraints. Although the rule based normalization performed by DNNV may produce fewer formulas (this depends on the formula structure and implementation details of the rewriting system), this count provides an upper bound on the number of conjunctions that can be generated for a formula. Without MOSAIC, a rewriting system would first generate a large disjunctive normal form (with at most #Conjunction many elements), then check the feasibility of generated conjunctions and hand feasible conjunctions to an open-loop NNV tool. As indicated by Table 7, such an approach can lead to the number of conjunctions reaching into the trillions which becomes entirely intractable in practice. As can be seen in Table 7, our tool (# Queries) only produces a fraction of the propositionally satisfiable conjunctions (# Conjunctions) and also significantly reduces the number of open-loop NNV queries in comparison to an approach that splits up disjunctions (# Feasible Conjunctions). Note, that DNNV is also required to check generated conjunctions for feasibility, thus, our approach is also efficient in this regard by requiring a comparatively low number of SMT calls. Given the feasiblity of 39 trillion conjunctions, one may wonder whether the propositional structure encoded in the boolean skeleton is of use at all. In this instance, we consider conjunctions over 110 distinct atoms. Indeed, the propositional structure adds value: Without it, we would obtain 2 110 ≈ 10 33 possible conjunctions, i.e. based on the propositional structure we only consider a fraction of approx. 10 -19 of all possible combinations. MOSAIC further reduces this fraction to a degree that is manageable via open-loop NNV.

Comparison with SMT solvers (Table 8) An alternative approach for the verification of non-linear open-loop NNV queries could be encoding the problem using an off-the-shelf SMT solver. In this case, the SMT solver has to check the satisfiability of the nonlinear Formula (9). We can instrument

the Lantern package [42] to encode the NN into a SMT formula. Thus, we performed a comparison on the ACC Large NN as well as the retrained ACC Large NN, i.e. on a satisfiable as well as a non-satisfiable instance. We compared our approach to dReal [41], Mathematica [96], Z3 [32], MathSAT [27] (due to its use of incremental linearization) as well as the first and second place of SMT-Comp 2023 in the QF NRA track: Z3++ [24] and cvc5 [14]. The results of our comparison can be observed in Table 8. The observed timeouts after 12 hours are unsurprising insofar as the work on linear open-loop NNV techniques was partially motivated by the observation that classical SMT solvers struggle with the verification of NNs.

Comparison to the techniques by Genin et al. [42] While the work by Genin et al. [42] represents a case-study with techniques specifically applied to an NN for a simplified airborne collision avoidance setting, some ideas from the example in [42] might in principle generalize to other case studies. Unfortunately, the case-study considered by [42] are not the NNs from Julian et al. [53, 55], but simplified NNs with a single acceleration control output. As the authors did not publish their trained NNs, their exact verification formulas, or their verification runtimes, we instead compare our approach with this line of work on our ACC benchmark. To this end, we approximate the verification property derived in Appendix E.1 using the box approximation techniques described by the authors and use their Lantern Python package to translate the verification tasks into linear arithmetic SMT problems. Using Z3, their technique does not terminate within more than 50 hours on the (unsafe) ACC Large network and thus fails to analyze the NN. This demonstrates significant scalability limitations compared to our approach. Moreover, it is worth pointing out that the authors themselves acknowledge that the technique is incomplete which distinguishes our complete lifting procedure from their approach.

Results on Q2. If closed-loop NNV is a hammer then guaranteeing infinite-time safety is a screw: It is a categorically different problem requiring a different tool. N 3 V provides safety guarantees which go infinitely beyond the guarantees achievable with State-of-the-Art techniques (closed-loop

NNV or otherwise). A direct CAD/SMT encoding of Formula (9) as well as the techniques by Genin et al. [42] are no alternatives due to timeouts ( > 12h) or 'unknown' results (see also Table 8).

E.3 Zeppelin Steering

As a further case study, we considered the task of steering a Zeppelin under uncertainty: The model's goal was to learn avoiding obstacles while flying in a wind field with nondeterministic wind turbulences. This problem has previously been studied with differential hybrid games [77]. The examined scenario serves two purposes: On the one hand, it shows that our approach can reuse safety results from the dL literature which drastically increases its applicability; on the other hand it is a good illustration for why verification (rather than empirical evidence) is so important when deploying NNs in safety-critical fields.

After transferring the differential hybrid games logic contract into a differential dynamic logic contract and proving its safety, we trained a model to avoid obstacles while flying in a wind field with uniformly random turbulences via PPO. After 1.4 million training steps, we obtained an agent that did not crash for an evaluation run of 30,000 time steps. Given these promising results we proceeded to verify the agent's policy assuming a safe - or at least 'almost safe' - flight strategy had been learnt. However, upon verifying the NN's behavior for obstacles of circumfence 40, we found that it produced potentially unsafe actions for large parts of the input space. The reason this unsafety was not observable during empirical evaluation was the choice of uniformly random wind turbulences: The unsafe behavior only appears for specific sequences of turbulences which occur extremely rarely in the empirical setting. This flaw in the training methodology was only found due to the verification. This is where our approach differs from simulation-based evaluation: With an SMT filter timeout of 4 seconds, N 3 V provides an exhaustive characterization of all potentially unsafe regions in 4.1 hours while providing 72 concrete counterexample regions. This is where our approach differs from simulation-based evaluation, as we were able to generate an exhaustive characterization of counterexample regions. In this instance, the tool's bootlenecks were approximation construction and the SMT based counterexample finding. This case study and the stark difference between simulation and verification underscore the importance of rigorous verification of NNs as an addition to empirical evidence in safety-critical areas.

F Allowed advisories for Vertical Airborne Collision Avoidance

Table 9 provides an overview of possible advisories for a Vertical Airborne Collision Avoidance System. The allowed range of vertical velocity and the required minimal acceleration are integrated into the dL model used by VerSAILLE.

G NMACs produced by NN-based ACAS X advisories

Further counterexamples for the advisories of the NNCS can be found in Figures 7 to 11.

Counterexamples to the safety of NNCS advisories for non-level flight of the intruder in the case of a previous advisory Do Not Climb and Do Not Descend can be found in Figures 12 and 13. Note,

10500

NMAC collision region

altitude (ft)

10250

Ownship (current advisory)

Original path of ownship

10000

Intruder

9750

9500

-5.0-2.5

0.0

2.5

5.0

time for intruder / -time for ownship (s)

10500

NMAC collision region

altitude (ft)

10250

Ownship (current advisory)

Original path of ownship

10000

Intruder

9750

9500

-5.0-2.5

0.0

2.5

5.0

time for intruder / -time for ownship (s)

10500

NMAC collision region

altitude (ft)

10250

Ownship (current advisory)

Originai path of ownship

10000

Intruder

9750

9500

-5.0

0-2.5

0.0

2.5

5.0

time for intruder / -time for ownship (s)

that for non-level flight (i.e. both intruder and ownship have a non-zero vertical velocity), there exist two possible interpretations for the advised vertical velocities. These can be interpreted as absolute or relative velocity. For our counterexamples in Figures 12 and 13 we opt for the relative velocity interpretation. This does not affect the analysis for level flight intruders where the interpretations coincide.

10500

NMAC collision region

altitude (ft)

10250

Ownship (current advisory)

Original path of ownship

10000

Intruder

9750

9500

-5.0-2.5

0.0

2.5

5.0

time for intruder / -time for ownship (s)

10500

NMAC collision region

altitude (ft)

10250

Ownship (current advisory)

Original path of ownship

10000

Intruder

9750

9500

-5.0

）-2.5

0.0

2.5

5.0

time for intruder / -time for ownship (s)

11000

NMAC collision region

altitude (ft)

Ownship (current advisory)

10500

Original path of ownship

Intruder

10000

9500

-10

-5

0

5

10

time for intruder / -time for ownship (s)

10500

NMAC collisionregion

三

10000

Ownship (current advisory)

altitude

Original path of ownship

Intruder

9500

9000

-10

-5

0

5

10

time for intruder / -time for ownship (s)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the introduction are supported by the theorems (with proofs), algorithms and evaluations in Sections 3 to 5 (and its appendices).

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The assumptions made by our theoretical results (in particular in Section 3) are explicitly stated and the exponential worst-case runtime of Mosaic is discussed in Section 4.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All formal statements and proofs can be found in Appendix C and are referenced accordingly in the main paper.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide an artifact for the N 3 V tool which comprises all benchmarks mentioned in the paper. The artifact is equipped with scripts for building the tool ( ./build.sh <julia 1.10 path> ), running an example ( ./run example.sh ; runs approx. 130 seconds) or running all N 3 V benchmarks ( ./run experiments.sh ; logs will be saved in ./experiments/<benchmark class>/<benchmark> ). For traceability, the benchmark logs are part of the artifact (will be overwritten by rerunning benchmarks). For the comparisons to other tools in Appendix E.2, we provide logs and documentation of the evaluated configurations (see ./experiments/comparison and ./experiments/dnnv comp ). For all benchmarks, the underlying dL models are provided ( ./experiments/<benchmark class>/<benchmark>/keymaerax ). Where NNs were trained, we also provide the Jupyter notebooks used for training as-is ( ./experiments/<benchmark class>/<benchmark>/training ).

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All code and data can be found on GitHub: https://github.com/ samysweb/NCubeV . We also provide an archived version of our artifact with DOI [91].

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details on the machine used for evaluation; all other details are described in the paper and found in the artifact.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We did not have the computational resources to run the experiments multiple times (this is in particular the case for long-running experiments). More importantly, our results show fundamental, categorical differences between the tools: We repeatedly demonstrate that our tool addresses problems that all other techniques fail to address within a very generous timeout. Thus, a statistical significance test would not be useful in this matter as the results are binary (solves the problem vs. does not solve the problem).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments ran on the same machine (see Section 5)

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The core contribution of our research is a methodology to make NNCS safer. Making NNCS safer can have very positive societal impacts by increasing (warranted) trust in infrastructure and preventing catastrophic failures. We do not see potential issues in any other category mentioned by the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: We see our work as foundational research that could be applied in numerous settings to guarantee safety of NNCS. We see no potential for negative societal impacts directly stemming from this work. Conversely, making NNCS safer can have very positive societal impacts by increasing (warranted) trust in infrastructure and preventing catastrophic failures.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our contribution does not contain any high-risk datasets or models.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Evaluated models from the literature are appropriately cited.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code of our tool N 3 V will be released as open-source with the paper - this includes instructions for verifying models with the tool.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the challenge of verifying the safety of NNCSs for CPS, especially for infinite time horizons. To tackle this problem, the author(s) introduce VerSAILLE, a novel approach that leverages differential dynamic logic to derive specifications for NNs, which are then proven using NN verification tools. \nThis allows proving infinite-time safety of the NNCS via dL.  To bridge the gap between nonlinear arithmetic constraints arising from hybrid systems and linear constraint support in NN verification tools, they also present Mosaic, an efficient, sound, and complete verification method for polynomial real arithmetic properties on piecewise linear NNs. Evaluation shows the effectiveness of VerSAILLE and Mosaic by proving infinite-time safety on benchmarks and enumerating counterexamples in unsafe scenarios.\n\nSTRENGTHS:\n1. The theoretical novelty of this work is sufficient.\n  1.1. The proposed VerSAILLE provides sound proof of infinite-time safety for the NNCSs. \n       This method reuses safety proofs from control-theory literature in the form of dL for the first time and supports a large class of feed-forward NNs.\n  1.2. The framework Mosaic enables the adaptation of existing open-loop NNV tools to handle polynomial queries with arbitrary logical structures while maintaining completeness to polynomial constraints.\n2. The experiments are very thorough, especially the comparison and combination with different methods and tools, which proved the effectiveness of the proposed method.\n3. This paper is well-written and the running example throughout the article makes it easier to follow.\n\nWEAKNESSES:\nMy main concern is the gap between the theoretical contributions and the implementation. As mentioned in lines 71-73, \"the implementation (N3V) supports NNs with Relu\" and \"theoretical contribution (VerSAILLE) reaches far beyond this\". What are the reasons for this gap and what are the difficulties in overcoming it?\n\nQUESTIONS:\n1. How is the scalability of N3V? What is the structure of the largest neural network it can handle?\n2. what is the time overhead of Mosaic i.e., lifting NNV tools for linear, normalized open-loop queries to polynomial queries of arbitrary logical structure?\n\nPlease answer the questions in \"Weaknesses*\" and \"Questions*\"\n\nLIMITATIONS:\nThere are no concerns of negative broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VerSAILLE, a new method using dL contracts to ensure the safety of Neural Network Controlled Systems with piece-wise Noetherian Neural Networks. Mosaic, implemented in N^{3}V for ReLU NNs, efficiently verifies properties across various case studies, including complex applications like airborne collision avoidance. The approach demonstrates scalability and effectiveness compared to traditional closed-loop techniques, offering a promising path for developing safe and goal-oriented NNCSs in practical settings.\n\nSTRENGTHS:\nThe authors have written an good paper with promising contributions which VerSAILLE establishes a formal foundation enabling sound proof of infinite-time safety for NNCSs using dL models from control-theory literature, and  Mosaic introduces an efficient, sound, and complete technique for verifying properties in polynomial real arithmetic on piece-wise linear NNs, enhancing existing open-loop NNV tools. Futhermore, Mosaic supports exhaustive characterization of unsafe state spaces, demonstrated effectively in real-world case studies including adaptive cruise control and airborne collision avoidance ACAS X.\n\nWEAKNESSES:\nThe structure of the paper is well-organized, with tight logic and thorough arguments, and is overall well-written. However, the introduction and related work sections seem unclear and lack readability. It is recommended to simplify the text or organize it into clearer paragraphs.\n\nQUESTIONS:\n(1) The authors propose a VerSAILLE NNCS verification method, which leverages the efficiency of neural network verification tools while maintaining the rigor of differential dynamic logic. However, the method is only applied experimentally to networks with ReLU activation functions, and most related work focuses solely on this single type of network. Have other types of networks been considered, and has the method been tested for effectiveness across different types NNs? \n\n(2) Additionally, they introduce an effective, sound, and complete method, Mosaic, for verifying polynomial real arithmetic properties on piece-wise linear neural networks, extending existing linear constraint tools to nonlinear settings while preserving completeness. How well does this method work for non-polynomial cases?\n\n(3) Furthermore, in experimental evaluations, the authors demonstrate infinite-time safety in certain scenarios, such as the classical Vertical Airborne Collision Avoidance NNCS verification benchmark.  This benchmark appears frequently in related work; are there additional benchmarks available to showcase, and are there more advanced works for a more comprehensive comparison and validation of the proposed approach?\n\nLIMITATIONS:\nThe authors have sufficiently addressed the limitations, explicitly stating the assumptions underlying the theoretical results, particularly in Section 3, and discussing the exponential worst-case runtime of Mosaic in Section 4.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of formal verification of neural-network based control systems. While scalability of the existing methods can still be improved, the authors provide an alternative approach via reusing safety proofs from control theory, open-loop neural-network verification, and differential dynamic logic. The authors are mainly concerned with collision avoidance as proxy for safety. Proposed N3V and Mosaic when compared to closed-loop verification tools showed improvement in terms of runtime.\n\nSTRENGTHS:\nThe work proposes a novel theoretical framework based on a nontrivial combination of existing theoretical results, previously unemployed by the state of the art. The authors also contribute a technique Mosaic based on a novel algorithmic approach to avoiding repeated checks of the same input regions and instead reusing verification results (azulejos).\n\nThe impact of the theoretical framework Versaille extend to the general community of NNCS verification since the core idea of reusing verification proof is generally valuable. The evaluation is extensive enough under the stated assumptions on the activation functions, architecture, input, and safety specification. It would be interesting to see if it extend to other properties and systems. It is a promising step forward to improving verification scalability.\n\nThe paper presents technical and theoretical contributions supported by rigorous proofs and extensive evaluation. The evaluation is quite detailed and the authors comprehensively analyze the results in comparison to the state-of-the-art tools.\n\nOverall, the paper proposes a strong theoretical framework for NNCS verification and sound and complete algorithm for the open-loop NN verification. The paper is self-contained and of high quality.\n\n\n\nThe paper is excellently organized. The authors evidently invested into clear narrative, logical connections, and guidance for the reader.\n\nWEAKNESSES:\nOne recommendation could be to shortly summarize the take-away from the extended evaluation in the main bod of the paper, so that all evaluation questions are answered without necessarily consulting the appendix. For instance, the Zeppelin insights are quite interesting and probably deserve to to be mentioned in the main paper.\n\nMinor:\n- Although \"control envelopes\" may be a well-understood term in control theory, it would be important to briefly introduce it on the concept level the first time it occurs in the introduction (same applies to differential dynamic logic).\n- Illustrative examples and descriptive color names are highly appreciated.\n- aircraft\n- non-Portuguese speakers would not know how to read \"azulejos\", would be good to transcribe.\n- some abbreviations are not introduced, e.g., DNC (although there original work is cited, it would make the paper self contained to include the definitions).\n\nQUESTIONS:\n1. Mosaic provides an exponential improvement in worst-case runtime. The authors mention that their approach performs particularly well for neural-network control systems with \"small\" input dimensions. Do the authors have any insights on how small these have to be to perform well, how well is meant here?\n2. It seems safety is analyzed here as \"collision avoidance\" (as it is the case for all considered benchmarks). Do the results generalize to other notions of safety, e.g., stability (can be expressed via reachability as well)?\n3. For \"unsafe\" level flight scenarios, the author mention using manual approximation. To what extend did that contribute to the resulting times in Table 3?\n4. Why was the tool compared to ARCH Comp 2022 and not the 2023 results?\n5. What is the overhead in preparing system and specifications for N3V and Mosaic? How easily extendable is the implementation?\n\nLIMITATIONS:\nThe addressed problem setting is guarded by multiple assumptions, however, common for the NNCS verification domain and evaluated extensively. The check list says the software is open source, however, the authors promise to make it open source, rather than making it open-source anonymously at the time of submission.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce VerSAILLE (Verifiably Safe AI via Logically Linked Envelopes), a method of verifying neural network based control systems using a provably safe control envelope in dL. VerSAILLE relied on nondeterministic mirrors, which allows authors to reflect a given neural network and reason within and outside dL simultaneously, ultimately having outside dL specifications imply the safety of the mirrored dL model. Authors also introduce Mosaic to leverage off-the-shelf open-loop neural network verifiers on polynomial queries, in addition to their linear ones. Mosaic is both sound and complete, making it a strong fit of critical applications.\n\nSTRENGTHS:\nThe paper is comprehensive in its approach and the authors propose a novel method that seems relatively stand-alone. The paper is organized well and the logic of the method flows well.\n\nWEAKNESSES:\nGenerally, the paper is difficult to read and understand. Readability would be improved from strong definitions and standardized notation.  Paragraphs tend to be very long (notably Related Work), it would be helpful to have more stand alone definitions and methodology. \n\nPage 2, lines 46-53: confusingly written, jumping from point to point without definitions for the terms that are cited. Not sure what the “subsymbolic reasoning of an NN” or “infinite-time control-theory reasoning” are.\n\nPage 2, line 50: “an NN” → “a NN”\n\nEvaluation is conducted only on vertical airborne collision avoidance, and the experiments and parameters are not well defined. On page 8, Table 3, “CE” is not defined.  \n\nThe use of DPLL(T) is not clearly described in the paper.\n\nPOST REBUTTAL:\nThe authors have cleared up their use of DPLL(T) in their response, which now seems to be an interesting contribution of the work.\n\nQUESTIONS:\n- Page 2, line 56: Is it trivial to assume “abstract, nondeterministic control envelope has already been verified in dL”? \n- What are the complexity limitations of Mosaic?\n\nLIMITATIONS:\nNot properly discussed. It would be helpful to understand trade-offs between their methods and other ones.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}