Generalized Universal Approximation for Certified Networks

To certify safety and robustness of neural networks , researchers have successfully applied abstract interpretation , primarily using interval bound propagation . To understand the power of interval bounds , we present the abstract universal approximation ( AUA ) theorem , a generalization of the recent result by Baader et al . ( 2020 ) for ReLU networks to a large class of neural networks . The AUA theorem states that for any continuous function f , there exists a neural network that ( 1 ) approximates f ( universal approximation ) and ( 2 ) whose interval bounds are an arbitrarily close approximation of the set semantics of f . The network may be constructed using any activation function from a rich class of functions—sigmoid , tanh , ReLU , ELU , etc.—making our result quite general . The key implication of the AUA theorem is that there always exists certifiably robust neural networks , which can be constructed using a wide range of activation functions . 1 INTRODUCTION . With wide adoption of neural networks , new safety and security concerns arose . The most prominent property of study has been robustness ( Goodfellow et al. , 2015 ) : small perturbations to the input of a network should not change the prediction . For example , a small change to an image of a stop sign should not cause a classifier to think it is a speed-limit sign . A number of researchers have proposed the use of abstract interpretation ( Cousot & Cousot , 1977 ) techniques to prove robustness of neural networks ( Gehr et al. , 2018 ; Wang et al. , 2018 ; Anderson et al. , 2019 ) and to train robust models ( Mirman et al. , 2018 ; Gowal et al. , 2018 ; Huang et al. , 2019 ; Wong & Kolter , 2018 ; Wong et al. , 2018 ; Balunovic & Vechev , 2020 ) . Suppose we want to verify robustness of a neural network to small changes in the brightness of an image . We can represent a large set of images , with varying brightness , as an element of some abstract domain , and propagate it through the network , effectively executing the network on an intractably large number of images . If all images lead to the same prediction , then we have a proof that the network is robust on the original image . The simplest abstract interpretation technique that leads to practical verification results is interval analysis—also referred to as interval bound propagation . In our example , if each pixel in a monochrome image is a real number r , then the pixel can be represented as an interval [ r− , r+ ] , where denotes the range of brightness we wish to be robust on . Then , the box representing the interval of each pixel is propagated through the network using interval arithmetic operations . The interval domain has been successfully used for certifying properties of neural networks in vision ( Gehr et al. , 2018 ; Gowal et al. , 2018 ) , NLP ( Huang et al. , 2019 ) , as well as cyber-physical systems ( Wang et al. , 2018 ) . Why does the interval domain work for certifying neural networks ? To begin understanding this question , Baader et al . ( 2020 ) demonstrated a surprising connection between the universal approximation theorem and neural-network certification using interval bounds . Their theorem states that not only can neural networks approximate any continuous function f ( universal approximation ) as we have known for decades , but we can find a neural network , using rectified linear unit ( ReLU ) activation functions , whose interval bounds are an arbitrarily close approximation of the set semantics of f , i.e. , the result of applying f to a set of inputs ( e.g. , set of similar images ) . AUA theorem ( semi-formally ) : For a continuous function f : Rm → R that we wish to approximate and error δ > 0 , there is a neural network N that has the following behavior : Let B ⊂ Rm be a box . The red interval ( top ) is the tightest interval that contains all outputs of f when applied to x ∈ B . If we propagate box B through N using interval bounds , we may get the black interval ( bottom ) N # ( B ) , whose lower/upper bounds are up to δ away from the red interval . 6 δ 6 δ minx∈B f ( x ) maxx∈B f ( x ) N # ( B ) Figure 1 : Semi-formal illustration of AUA theorem . ( Right is adapted from Baader et al . ( 2020 ) . ) The theorem of Baader et al . ( 2020 ) is restricted to networks that use rectified linear units ( ReLU ) . In this work , we present a general universal approximation result for certified networks using a rich class of well-behaved activation functions . Specifically , we make the following contributions . Abstract universal approximation ( AUA ) theorem . We prove what we call the abstract universal approximation theorem , or AUA theorem for short : Let f be the function we wish to approximate , and let δ > 0 be the tolerated error . Then , there exists a neural network N , built using any well-behaved activation function , that has the following behavior : For any box of inputs B , we can certify , using interval bounds , that the range of outputs of N is δ close to the range outputs of f . If the box B of inputs is a single point in Euclidean space , the AUA theorem reduces to the universal approximation theorem ; thus , AUA generalizes universal approximation . Fig . 1 further illustrates the AUA theorem . Existence of robust classifiers . While the AUA theorem is purely theoretical , it sheds light on the existence of certifiable neural networks . Suppose there is some ideal robust image classifier f using the ` ∞ norm , which is typically used to define a set of images in the neighborhood of a given image . The classical universal approximation theorem tells us that , for any desired precision , there is a neural network that can approximate f . We prove that the AUA theorem implies us that there exists a neural network for which we can automatically certify robustness using interval bounds while controlling approximation error . In addition , this neural network can be built using almost any activation function in the literature , and more . Squashable functions . We define a rich class of activation functions , which we call squashable functions , for which our abstract universal approximation theorem holds . This class expands the functions defined by Hornik et al . ( 1989 ) for universal approximation and includes popular activation functions , like ReLU , sigmoid , tanh , ELU , and other activations that have been shown to be useful for training robust neural networks ( Xie et al. , 2020 ) . The key feature of squashable activation functions is that they have left and right limits ( or we can use them to construct functions with limits ) . We exploit limits to approximate step ( sign ) functions , and therefore construct step-like approximations of f , while controlling approximation error δ . Proof of AUA theorem . We present a constructive proof of the AUA theorem . Our construction is inspired by and synthesizes a range of results : ( 1 ) the work of Hornik et al . ( 1989 ) on squashing functions for universal approximation , ( 2 ) the work of Csáji ( 2001 ) for using the sign ( step ) function to construct Haar ( wavelet ) functions , and ( 3 ) the work of Baader et al . ( 2020 ) on the specialized AUA theorem for ReLUs . The key idea of Baader et al . ( 2020 ) is to construct an indicator function for box-shaped regions . We observe that squashable functions can approximate the sign function , and therefore approximate such indicator functions , while carefully controlling precision of abstract interpretation . Our proof uses a simpler indicator construction compared to Baader et al . ( 2020 ) , and as a result its analysis is also simpler . 2 RELATED WORK . The classical universal approximation ( UA ) theorem has been established for decades . In contrast to AUA , UA states that a neural network with one single hidden layer can approximate any continuous function on a compact domain . One of the first versions goes back to Cybenko ( 1989 ) ; Hornik et al . ( 1989 ) , who showed that the standard feed-forward neural network with sigmoidal or squashing activations is a universal approximator . The most general version of UA was discovered by Leshno et al . ( 1993 ) , who showed that the feed-forward neural network is a universal approximator if and only if the activation function is non-polynomial . Because AUA implies UA , this means AUA can not hold beyond non-polynomial activation functions . There are also other variants of UA . Some of them study the expressiveness of neural networks with structural constraints , such as restricted width per layer Lu et al . ( 2017 ) ; Kidger & Lyons ( 2019 ) , or specific neural network architectures Lin & Jegelka ( 2018 ) . Another line of work focuses on specific functions that one wants to approximate rather than arbitrary continuous functions , such as Anil et al . ( 2019 ) ; Cohen et al . ( 2019 ) , who study approximation of Lipschitz functions . Neural-network verification has received a lot of attention in recent years . Most techniques are either based on decision procedures , like SMT solvers Ehlers ( 2017 ) ; Katz et al . ( 2017 ) and integer linear programming ( ILP ) solvers Tjeng et al . ( 2019 ) , or abstract interpretation . The former class can often provide sound and complete verification on neural networks with piecewise-linear operations , like ReLU , but is not scalable due to the complexity of the problem and the size of the networks . Abstract-interpretation-based techniques sacrifice completeness for efficient verification . We have considered the simplest non-trivial numerical domain , intervals , that has been shown to produce strong results , both for robustness verification and adversarial training Gehr et al . ( 2018 ) ; Anderson et al . ( 2019 ) ; Huang et al . ( 2019 ) ; Mirman et al . ( 2018 ) ; Wang et al . ( 2018 ) ; Zhang et al . ( 2020 ) . Researchers have considered richer domains Singh et al . ( 2018 ; 2019 ) , like zonotopes Ghorbal et al . ( 2009 ) and forms of polyhedra Cousot & Halbwachs ( 1978 ) . Since such domains are strictly more precise than intervals , the AUA theorem holds for them . 3 FOUNDATIONS AND SQUASHABLE ACTIVATION FUNCTIONS . 3.1 NEURAL NETWORKS AND SQAUSHABLE ACTIVATIONS . A neural network N is a function in Rm → R. We define a network N following a simple grammar , a composition of primitive arithmetic operations and activation functions . Definition 3.1 ( Neural network grammar ) . Let x ∈ Rm be the input to the neural network . A neural network N is defined using the following grammar : N = c | xi | N1 +N2 | c ∗N1 | t ( N1 ) , where c ∈ R , xi is the ith element of x , and t : R→ R is an activation function . We will always fix a single activation function t to be used in the grammar . We now present a general class of activation functions that we will call squashable activation functions . Fig . 2 shows some examples . Definition 3.2 ( Squashable functions ) . t : R→ R is squashable iff ( 1 ) there is a < b ∈ R such that lim x→−∞ t ( x ) = a , lim x→∞ t ( x ) = b , and ∀x < y. t ( x ) 6 t ( y ) ( 1 ) or ( 2 ) we can construct a function t′ that satisfies Eq . ( 1 ) as affine transformations and function compositions of copies of t , i.e. , following the grammar in Def . 3.1 . E.g. , t′ ( x ) = t ( 1− t ( x ) ) . Informally , an activation function is in this class if we can use it to construct a monotonically increasing function that has limits in the left and right directions , −∞ and∞.1 Squashable activation 1In our construction and proof , we do not need the function to be monotonic ; however , in practice , most activation functions are monotonic and abstractly interpreting arbitrary functions is impractical . functions extend the squashing functions used by Hornik et al . ( 1989 ) . All of the activation functions in Fig . 2 are squashable . Fig . 2 ( top ) shows activation functions that satisfy Eq . ( 1 ) , and are therefore squashable . For example , sigmoid and tanh easily satisfy Eq . ( 1 ) : both have limits and are monotonically increasing . What about activation functions like ReLU , ELU , softplus , etc. , shown in Fig . 2 ( bottom ) ? It is easy to see that they do not satisfy Eq . ( 1 ) : none of them have a right limit . However , by point ( 2 ) of Def . 3.2 , given an activation function t , if we can construct a new activation function t′ that satisfies Eq . ( 1 ) , using the operations in the grammar in Def . 3.1 , then t is squashable . We give a general and simple construction that works for all activation functions in Fig . 2 ( bottom ) . Proposition 3.3 . Let t ∈ { ReLU , softplus , smoothReLUa , ELU } . The function t′ ( x ) = t ( 1 − t ( −x ) ) satisfies Eq . ( 1 ) . Therefore , ReLU , softplus , Smooth ReLU , and ELU , are squashable . Example 3.4 . Fig . 3 shows t ( 1 − t ( −x ) ) , for t = ReLU and t = softplus . Both have left/right limits and are monotonic . Thus , they satisfy Eq . ( 1 ) and therefore ReLU and softplus are squashable .

Summary: This paper proposes to extend the techniques of Baader et al. [2020], demonstrating that interval analysis provable ReLU networks are universal approximators, to a larger class of activation functions, which they call squashable functions.    Furthermore, they claim that their proof of this theorem is simpler due to using a bounded depth construction.