CONTRIBUTION Approximation Capabilities of Muitilayer Feedforward Networks

-We show that standard multilayer feedfbrward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to LP(lt) performance criteria, for arbitrary finite input environment measures p, provided only that sufficiently many hidden units are available. If the activation function is continuous, bounded and nonconstant, then continuous mappings can be learned uniformly over compact input sets. We also give very general conditions ensuring that networks with sufficiently smooth activation functions are capable of arbitrarily accurate approximation to a_Function and its derivatives. Keywords--Multilayer feedforward networks, Activation function, Universal approximation capabilities, Input environment measure, D'(p) approximation, Uniform approximation, Sobolev spaces, Smooth approximation. 1. I N T R O D U C T I O N The approximation capabilities of neural network architectures have recently been investigated by many authors, including Carroll and Dickinson (1989), Cybenko (1989), Funahashi (1989), Gallant and White (1988), Hecht-Nielsen (1989), Hornik, Stinchcombe, and White (1989, 1990), lrie and Miyake (1988), Lapedes and Farber (1988), Stinchcombe and White (1989, 1990). (This list is by no means complete.) If we think of the network architecture as a rule for computing values at l output units given values at k input units, hence implementing a class of mappings from R k to R ~, we can ask how well arbitrary mappings from R k to R t can be approximated by the network, in particular, if as many hidden units as required for internal representation and computation may be employed. How to measure the accuracy of approximation depends on how we measure closeness between functions, which in turn varies significantly with the specific problem to be dealt with. In many applications, it is necessary to have the network perform simultaneously well on all input samples taken from some compact input set X in R k. In this case, closeness is Requests for reprints should be sent to Kurt Hornik, Institut fur Statistik und Wahrscheinlichkeitstheorie, Technische Universit/~t Wien, Wiedner Hauptstral3e 8-10/107, A-1040 Wien, Austria. measured by the uniform distance between functions on X, that is, P~,.x(f, g) = sup If(x) g(x) i. In other applications, we think of the inputs as random variables and are interested in the average performance where the average is taken with respect to the input environment measure/2, where p(R k) < ~. In this case, closeness is measured by the LP(p) distances Pp.~,(f, g) = [fRk [f(x) -g(x)l" dl~(X)] 'p, 1 -< p < 0o, the most popular choice being p --2, corresponding to mean square error. Of course, there are many more ways of measuring closeness of functions. In particular, in many applications, it is also necessary that the derivatives of the approximating function implemented by the network closely resemble those of the function to be approximated, up to some order. This issue was first taken up in Hornik et al. (1990), who discuss the sources of need of smooth functional approximation in more detail. Typical examples arise in robotics (learning of smooth movements) and signal processing (analysis of chaotic time series); for a recent application to problems of nonparametric inference in statistics and econometrics, see Gallant and White (1989). All papers establishing certain approximation ca-