Safe Neurosymbolic Learning with Differentiable Symbolic Execution

We study the problem of learning worst-case-safe parameters for programs that use neural networks as well as symbolic , human-written code . Such neurosymbolic programs arise in many safety-critical domains . However , because they need not be continuous , let alone differentiable , they can not be learned using existing gradient-based approaches to safe learning . Our method , Differentiable Symbolic Execution ( DSE ) , learns such programs by sampling code paths using symbolic execution , constructing gradients of a worst-case “ safety loss ” along these paths , and then backpropagating these gradients through program operations using a generalization of the REINFORCE estimator . We evaluate the method on a mix of synthetic tasks and real-world benchmarks . Our experiments show that DSE significantly outperforms the state-of-the-art DIFFAI method on these tasks . 1 INTRODUCTION . Safety on worst-case inputs has recently emerged as a key challenge in deep learning research . Formal verification of neural networks ( Albarghouthi , 2021 ) is an established response to this challenge . In particular , an exciting body of recent work ( Mirman et al. , 2018 ; Madry et al. , 2017 ; Cohen et al. , 2019 ; Singh et al. , 2018 ) has sought to incorporate formal verification into the training of neural networks . DIFFAI , among the most prominent of such approaches , uses a neural network verifier to construct a differentiable , worst-case safety loss for the learner . This loss is used to regularize a standard data-driven loss , biasing the learner towards parameters that are both performant and safe . A weakness of these methods is that they only consider functional properties ( such as adversarial robustness ) of isolated neural networks . By contrast , in real-world applications , neural networks are often embedded within human-written symbolic code , and correctness requirements apply to the entire neurosymbolic composition . For example , consider a car directed by a neural controller ( Qin & Badgwell , 2000 ) . Safety properties for the car are functions of its trajectories , and these trajectories depend not just on the controller but also the symbolic equations that define the environment . While recent work ( Christakis et al. , 2021 ) has studied the verification of such neurosymbolic programs , there is no prior work on integrating verification and learning for such systems . In this paper , we present the first steps towards such an integration . The fundamental technical challenge here is that while a neural network is differentiable over its parameters , the code surrounding may be non-differentiable or even discontinuous . This puts our problem beyond the scope of existing methods for integrated learning and verification . We overcome this difficulty using a new method , Differentiable Symbolic Execution ( DSE ) , for estimating gradients of worst-case safety losses of nondifferentiable neurosymbolic programs . DSE is based on a generalization of the classic REINFORCE estimator , which backpropagates gradients through non-differentiable operations by approximating integrals with sampling . In our problem , the integral is the aggregate of the safety losses along symbolic control flow paths in the program . To apply REINFORCE-like ideas here , we need to represent and sample paths in a symbolic way . We do so through an adaptation of the classic method of symbolic execution . We evaluate DSE through several case studies in the embedded control and navigation domains . Our baselines include an extended version of DIFFAI , the current state of the art , and an ablation that does not use an explicit safety loss . Our experiments show that DSE significantly outperforms the baselines in finding safe and high-performance model parameters . We also do a series of synthetic experiments that highlight the practical difficulty that DIFFAI-style approaches face in our setting . To summarize , our main contributions are : • We present the first approach to worst-case-safe parameter learning for neural networks embedded within nondifferentiable , symbolic programs . • As part of our learning algorithm , we give a new way to bring together symbolic execution and stochastic gradient estimators that might have applications outside our immediate task . • We present experimental results that indicate the advantages of DSE over the state-of-the-art in verified learning . 2 PROBLEM FORMULATION . Programs . We define programs with embedded neural networks as symbolic transition systems ( STS ) ( Manna & Pnueli , 2012 ) . Formally , a program Fθ is a tuple ( Loc , X , l0 , Init , Safe , Transθ ) . Here , Loc is a finite set of ( control ) locations , X = { x1 , . . . , xm } is a set of real-valued variables , and l0 ∈ Loc is the initial location . Init , a boolean formula over X , is an initial condition for the program . Safe is a map from locations to constraints over X ; intuitively , Safe ( l ) is a safety requirement asserted at location l. Finally , Transθ is a transition relation consisting of transitions ( l , G , Uθ , l ′ ) such that : ( i ) l is the source location and l′ is the destination location ; ( ii ) the guard G is a constraint over X ; and ( iii ) the update Uθ is a vector 〈U1 , θ , . . . , Um , θ〉 , where each Ui , θ is a real-valued expression over X constructed using standard symbolic operators and neural networks with parameters θ . Intuitively , Ui , θ represents the update to the i-th variable . We assume that each Ui , θ is differentiable in θ . Also , we assume that the programs are deterministic . That is , if G and G′ are guards for two distinct transitions from the same source state , then G ∧G′ is unsatisfiable . Programs in higher-level languages can be translated to the STS notation in a standard way . For example , Figure 1 ( left ) shows a simple high-level program . The STS for this program appears in Figure 1 ( right ) . Remarkably , while the program is simple , the state-of-the-art DIFFAI approach to verified learning fails to learn safe parameters for it . Safety Semantics . In classical formal methods , a program is considered to be safe if all of its executions satisfy a logical constraint . However , in learning settings , it helps to know not only whether a program is unsafe , but also the extent to which it is unsafe . Consequently , we define the safety semantics of programs in a quantitative way , in terms of a ( worst-case ) safety loss that quantifies the extent program ’ s safeness . Formally , let a state of Fθ be a pair s = ( l , v ) , where l is a location and v ∈ Rm is an assignment of values to the variables ( i.e. , v ( i ) is the value of xi ) . Such a state is said to be at location l. For boolean constraints B and assignments v to variables in B , let us write B ( v ) if v satisfies B . A state ( l0 , v ) , where Init ( v ) , is an initial state . A state ( l , v ) is safe if ( Safe ( l ) ) ( v ) . Let v ∈ Rm be an assignment to the variables . For a real-valued expression E over X , let E ( v ) be the value of E when xi is substituted by v ( i ) . For an update U = 〈U1 , . . . , Un〉 , we define U ( v ) as the assignment 〈U1 ( v ) , . . . , Un ( v ) 〉 . A length-n trajectory of Fθ is a sequence τ = 〈s0 , . . . , sn〉 , with si = ( li , vi ) , such that : ( i ) s0 is an initial state ; and ( ii ) for each i , there is a transition ( li , G , U , li+1 ) such that G ( vi ) and vi+1 = U ( vi ) . Let us fix a size bound N for trajectories . A trajectory τ = 〈s0 , . . . , sn〉 is maximal if it has length N , or if there is no trajectory τ ′ = 〈s0 , . . . , sn , sn+1〉 with length ≤ N . Because our programs are deterministic , there is a unique maximal trajectory from each s0 . We denote this trajectory by τ ( s0 ) . Let us assume a real-valued loss Unsafe ( s ) that quantifies the unsafeness of each state s. We require Unsafe ( s ) = 0 if s is safe and Unsafe ( s ) > 0 otherwise . We lift this measure to trajectories τ by letting Unsafe ( τ ) = ∑ s appears in τ Unsafe ( s ) . The safety loss C ( θ ) for Fθ is now defined as : C ( θ ) = max s is an initial state Unsafe ( τ ( s ) ) . ( 1 ) Thus , C ( θ ) = 0 if and only if all program trajectories are safe . Problem Statement . Our learning problem formalizes a setting in which we have training data for neural networks inside a program Fθ . While training the networks with respect to this data , we must ensure that the overall program satisfies its safety requirements . To ensure that the parameters of the different neural networks in Fθ are not incorrectly entangled , we assume that only one of these networks , NNθ , has trainable parameters.1 We expect as input a training set of i.i.d . samples from an unknown distribution over the inputs and outputs of NNθ , and a differentiable data loss Q ( θ ) that quantifies the network ’ s fidelity to this training set . Our learning goal is to solve the following constrained optimization problem : min θ Q ( θ ) s.t . C ( θ ) ≤ 0 . ( 2 ) 3 LEARNING ALGORITHM . Our learning approach is based on two ideas . First , we directly apply a recently-developed equivalence between constrained and regularized learning ( Agarwal et al. , 2018 ; Le et al. , 2019 ) to reduce Equation ( 2 ) to a series of unconstrained optimization problems . Second , we use the novel technique of Differentiable Symbolic Execution ( DSE ) to solve these unconstrained problems . At the highest level , we convexify the class of programs Fθ by allowing stochastic combination of programs . That is , we now allow programs of the form Fθ̂ = ∑T t=1 αtFθt , where ∑ i αt = 1 . To execute the program Fθ from a given initial state , one samples a specific program Fθt from the distribution ( α1 , . . . , αt ) and then executes it from that state . Equation ( 2 ) can now be written into the problem maxλ∈R+ minθ̂ Q ( θ̂ ) + λC ( θ̂ ) , which in turn can be solved using a classic algorithm ( Freund & Schapire , 1999 ) for computing equilibria in a twoplayer game . We omit further details about this algorithm because it follows Le et al . ( 2019 ) . ( For more details , see Appendix A.1 ) . A key feature of our high-level algorithm is that it repeatedly solves the optimization problem min θ Q ( θ ) + λC ( θ ) ( 3 ) for fixed values of λ . This problem is challenging because while Q ( θ ) is differentiable in θ , C ( θ ) depends on the entirety of Fθ and may not even be continuous . As we demonstrate in Section 5 , this makes it difficult to apply state-of-the-art gradient-based approaches to worst-case safe learning . DSE , our main contribution , addresses this challenge by estimating gradients ∇θC # ( θ ) of a differentiable approximation C # ( θ ) of C ( θ ) . We present the details of this method in the next section .

Summary: The paper proposes DSE, an approach to optimizing parameters of neurosymbolic programs (programs with mixed neural and symbolic components) while satisfying a safety constraint. To accomplish this, DSE defines a *safety loss*, a nonnegative term in the loss function which is nonzero when the function does not satisfy the safety constraint. This safety loss is constructed by symbolically executing the program, executing discrete transitions probabilistically according to a uniform distribution over concrete states. The paper evaluates DSE on a set of small-scale case studies, showing that in programs with discrete control flow, DSE can often train neural networks that soundly lead to safe behavior, with similar quality of learned results as those of a neurosymbolic training approach which does not take into account safety.