Generalized Universal Approximation for Certified Networks

To certify safety and robustness of neural networks , researchers have successfully applied abstract interpretation , primarily using interval bound propagation . To understand the power of interval bounds , we present the abstract universal approximation ( AUA ) theorem , a generalization of the recent result by Baader et al . ( 2020 ) for ReLU networks to a large class of neural networks . The AUA theorem states that for any continuous function f , there exists a neural network that ( 1 ) approximates f ( universal approximation ) and ( 2 ) whose interval bounds are an arbitrarily close approximation of the set semantics of f . The network may be constructed using any activation function from a rich class of functions—sigmoid , tanh , ReLU , ELU , etc.—making our result quite general . The key implication of the AUA theorem is that there always exists certifiably robust neural networks , which can be constructed using a wide range of activation functions . 1 INTRODUCTION . With wide adoption of neural networks , new safety and security concerns arose . The most prominent property of study has been robustness ( Goodfellow et al. , 2015 ) : small perturbations to the input of a network should not change the prediction . For example , a small change to an image of a stop sign should not cause a classifier to think it is a speed-limit sign . A number of researchers have proposed the use of abstract interpretation ( Cousot & Cousot , 1977 ) techniques to prove robustness of neural networks ( Gehr et al. , 2018 ; Wang et al. , 2018 ; Anderson et al. , 2019 ) and to train robust models ( Mirman et al. , 2018 ; Gowal et al. , 2018 ; Huang et al. , 2019 ; Wong & Kolter , 2018 ; Wong et al. , 2018 ; Balunovic & Vechev , 2020 ) . Suppose we want to verify robustness of a neural network to small changes in the brightness of an image . We can represent a large set of images , with varying brightness , as an element of some abstract domain , and propagate it through the network , effectively executing the network on an intractably large number of images . If all images lead to the same prediction , then we have a proof that the network is robust on the original image . The simplest abstract interpretation technique that leads to practical verification results is interval analysis—also referred to as interval bound propagation . In our example , if each pixel in a monochrome image is a real number r , then the pixel can be represented as an interval [ r− , r+ ] , where denotes the range of brightness we wish to be robust on . Then , the box representing the interval of each pixel is propagated through the network using interval arithmetic operations . The interval domain has been successfully used for certifying properties of neural networks in vision ( Gehr et al. , 2018 ; Gowal et al. , 2018 ) , NLP ( Huang et al. , 2019 ) , as well as cyber-physical systems ( Wang et al. , 2018 ) . Why does the interval domain work for certifying neural networks ? To begin understanding this question , Baader et al . ( 2020 ) demonstrated a surprising connection between the universal approximation theorem and neural-network certification using interval bounds . Their theorem states that not only can neural networks approximate any continuous function f ( universal approximation ) as we have known for decades , but we can find a neural network , using rectified linear unit ( ReLU ) activation functions , whose interval bounds are an arbitrarily close approximation of the set semantics of f , i.e. , the result of applying f to a set of inputs ( e.g. , set of similar images ) . AUA theorem ( semi-formally ) : For a continuous function f : Rm → R that we wish to approximate and error δ > 0 , there is a neural network N that has the following behavior : Let B ⊂ Rm be a box . The red interval ( top ) is the tightest interval that contains all outputs of f when applied to x ∈ B . If we propagate box B through N using interval bounds , we may get the black interval ( bottom ) N # ( B ) , whose lower/upper bounds are up to δ away from the red interval . 6 δ 6 δ minx∈B f ( x ) maxx∈B f ( x ) N # ( B ) Figure 1 : Semi-formal illustration of AUA theorem . ( Right is adapted from Baader et al . ( 2020 ) . ) The theorem of Baader et al . ( 2020 ) is restricted to networks that use rectified linear units ( ReLU ) . In this work , we present a general universal approximation result for certified networks using a rich class of well-behaved activation functions . Specifically , we make the following contributions . Abstract universal approximation ( AUA ) theorem . We prove what we call the abstract universal approximation theorem , or AUA theorem for short : Let f be the function we wish to approximate , and let δ > 0 be the tolerated error . Then , there exists a neural network N , built using any well-behaved activation function , that has the following behavior : For any box of inputs B , we can certify , using interval bounds , that the range of outputs of N is δ close to the range outputs of f . If the box B of inputs is a single point in Euclidean space , the AUA theorem reduces to the universal approximation theorem ; thus , AUA generalizes universal approximation . Fig . 1 further illustrates the AUA theorem . Existence of robust classifiers . While the AUA theorem is purely theoretical , it sheds light on the existence of certifiable neural networks . Suppose there is some ideal robust image classifier f using the ` ∞ norm , which is typically used to define a set of images in the neighborhood of a given image . The classical universal approximation theorem tells us that , for any desired precision , there is a neural network that can approximate f . We prove that the AUA theorem implies us that there exists a neural network for which we can automatically certify robustness using interval bounds while controlling approximation error . In addition , this neural network can be built using almost any activation function in the literature , and more . Squashable functions . We define a rich class of activation functions , which we call squashable functions , for which our abstract universal approximation theorem holds . This class expands the functions defined by Hornik et al . ( 1989 ) for universal approximation and includes popular activation functions , like ReLU , sigmoid , tanh , ELU , and other activations that have been shown to be useful for training robust neural networks ( Xie et al. , 2020 ) . The key feature of squashable activation functions is that they have left and right limits ( or we can use them to construct functions with limits ) . We exploit limits to approximate step ( sign ) functions , and therefore construct step-like approximations of f , while controlling approximation error δ . Proof of AUA theorem . We present a constructive proof of the AUA theorem . Our construction is inspired by and synthesizes a range of results : ( 1 ) the work of Hornik et al . ( 1989 ) on squashing functions for universal approximation , ( 2 ) the work of Csáji ( 2001 ) for using the sign ( step ) function to construct Haar ( wavelet ) functions , and ( 3 ) the work of Baader et al . ( 2020 ) on the specialized AUA theorem for ReLUs . The key idea of Baader et al . ( 2020 ) is to construct an indicator function for box-shaped regions . We observe that squashable functions can approximate the sign function , and therefore approximate such indicator functions , while carefully controlling precision of abstract interpretation . Our proof uses a simpler indicator construction compared to Baader et al . ( 2020 ) , and as a result its analysis is also simpler . 2 RELATED WORK . The classical universal approximation ( UA ) theorem has been established for decades . In contrast to AUA , UA states that a neural network with one single hidden layer can approximate any continuous function on a compact domain . One of the first versions goes back to Cybenko ( 1989 ) ; Hornik et al . ( 1989 ) , who showed that the standard feed-forward neural network with sigmoidal or squashing activations is a universal approximator . The most general version of UA was discovered by Leshno et al . ( 1993 ) , who showed that the feed-forward neural network is a universal approximator if and only if the activation function is non-polynomial . Because AUA implies UA , this means AUA can not hold beyond non-polynomial activation functions . There are also other variants of UA . Some of them study the expressiveness of neural networks with structural constraints , such as restricted width per layer Lu et al . ( 2017 ) ; Kidger & Lyons ( 2019 ) , or specific neural network architectures Lin & Jegelka ( 2018 ) . Another line of work focuses on specific functions that one wants to approximate rather than arbitrary continuous functions , such as Anil et al . ( 2019 ) ; Cohen et al . ( 2019 ) , who study approximation of Lipschitz functions . Neural-network verification has received a lot of attention in recent years . Most techniques are either based on decision procedures , like SMT solvers Ehlers ( 2017 ) ; Katz et al . ( 2017 ) and integer linear programming ( ILP ) solvers Tjeng et al . ( 2019 ) , or abstract interpretation . The former class can often provide sound and complete verification on neural networks with piecewise-linear operations , like ReLU , but is not scalable due to the complexity of the problem and the size of the networks . Abstract-interpretation-based techniques sacrifice completeness for efficient verification . We have considered the simplest non-trivial numerical domain , intervals , that has been shown to produce strong results , both for robustness verification and adversarial training Gehr et al . ( 2018 ) ; Anderson et al . ( 2019 ) ; Huang et al . ( 2019 ) ; Mirman et al . ( 2018 ) ; Wang et al . ( 2018 ) ; Zhang et al . ( 2020 ) . Researchers have considered richer domains Singh et al . ( 2018 ; 2019 ) , like zonotopes Ghorbal et al . ( 2009 ) and forms of polyhedra Cousot & Halbwachs ( 1978 ) . Since such domains are strictly more precise than intervals , the AUA theorem holds for them . 3 FOUNDATIONS AND SQUASHABLE ACTIVATION FUNCTIONS . 3.1 NEURAL NETWORKS AND SQAUSHABLE ACTIVATIONS . A neural network N is a function in Rm → R. We define a network N following a simple grammar , a composition of primitive arithmetic operations and activation functions . Definition 3.1 ( Neural network grammar ) . Let x ∈ Rm be the input to the neural network . A neural network N is defined using the following grammar : N = c | xi | N1 +N2 | c ∗N1 | t ( N1 ) , where c ∈ R , xi is the ith element of x , and t : R→ R is an activation function . We will always fix a single activation function t to be used in the grammar . We now present a general class of activation functions that we will call squashable activation functions . Fig . 2 shows some examples . Definition 3.2 ( Squashable functions ) . t : R→ R is squashable iff ( 1 ) there is a < b ∈ R such that lim x→−∞ t ( x ) = a , lim x→∞ t ( x ) = b , and ∀x < y. t ( x ) 6 t ( y ) ( 1 ) or ( 2 ) we can construct a function t′ that satisfies Eq . ( 1 ) as affine transformations and function compositions of copies of t , i.e. , following the grammar in Def . 3.1 . E.g. , t′ ( x ) = t ( 1− t ( x ) ) . Informally , an activation function is in this class if we can use it to construct a monotonically increasing function that has limits in the left and right directions , −∞ and∞.1 Squashable activation 1In our construction and proof , we do not need the function to be monotonic ; however , in practice , most activation functions are monotonic and abstractly interpreting arbitrary functions is impractical . functions extend the squashing functions used by Hornik et al . ( 1989 ) . All of the activation functions in Fig . 2 are squashable . Fig . 2 ( top ) shows activation functions that satisfy Eq . ( 1 ) , and are therefore squashable . For example , sigmoid and tanh easily satisfy Eq . ( 1 ) : both have limits and are monotonically increasing . What about activation functions like ReLU , ELU , softplus , etc. , shown in Fig . 2 ( bottom ) ? It is easy to see that they do not satisfy Eq . ( 1 ) : none of them have a right limit . However , by point ( 2 ) of Def . 3.2 , given an activation function t , if we can construct a new activation function t′ that satisfies Eq . ( 1 ) , using the operations in the grammar in Def . 3.1 , then t is squashable . We give a general and simple construction that works for all activation functions in Fig . 2 ( bottom ) . Proposition 3.3 . Let t ∈ { ReLU , softplus , smoothReLUa , ELU } . The function t′ ( x ) = t ( 1 − t ( −x ) ) satisfies Eq . ( 1 ) . Therefore , ReLU , softplus , Smooth ReLU , and ELU , are squashable . Example 3.4 . Fig . 3 shows t ( 1 − t ( −x ) ) , for t = ReLU and t = softplus . Both have left/right limits and are monotonic . Thus , they satisfy Eq . ( 1 ) and therefore ReLU and softplus are squashable .

Summary: The paper shows an "augmented" universal approximation (UA) result for neural networks that the authors call Abstract UA (AUA for short) and the motivation comes from understanding expressivity and certifiability of NN. Their result holds for NN with a wide variety of activation units and this is the main point of the paper, which directly extends the same result for ReLU networks (Baader et al. 2020).