# A Global Universality of Two-Layer Neural Networks with ReLU Activations

Authors: Naoya Hatano; Masahiro Ikeda; Isao Ishikawa; Yoshihiro Sawano
Journal: Journal of Function Spaces
 (2021)

---

## Abstract

In the present study, we investigate a universality of neural networks, which concerns a density of the set of two-layer neural networks in function spaces. There are many works that handle the convergence over compact sets. In the present paper, we consider a global convergence by introducing a norm suitably, so that our results will be uniform over any compact set.


---

## Body

## 1. Introduction

Neural network is a function that models a neuron system of a biological brain and is defined as alternate compositions of an affine map and a nonlinear map. The nonlinear map in a neural network is called the activation function. The neural networks have been playing a central role in the field of machine learning with a vast number of applications in the real world in the last decade. We refer to [1] and [2] for example.We focus on a two-layer feed-forward neural network with ReLU (rectified linear unit) activation, which is a functionf:ℝ⟶ℝ of the form of fx=∑i=1rciReLUaix+bi for some a1,b1,c1,⋯,ar,br,cr∈ℝ. Here, the function ReLU is called the rectified linear unit defined by
(1)ReLUx≔maxx,0x∈ℝ.The ReLU is one of the most popular activation functions for feed-forward neural networks in practical machine learning tasks for real-world problems.We consider the space of two-layer feedforward neural networks defined by the following linear space(2)X≔SpanReLUa·+b:a≠0,b∈ℝ=∑j=1NλjReLUaj·+bj:N∈ℕ,λj,aj,bjj=1N⊂ℂ×ℝn×ℝ.Then, it is natural to ask ourselves whetherX spans a dense subspace of a function space (topological linear space), which is called universality of X. Historically, the density property of X in the space Cℝ of continuous functions on ℝ is investigated by several authors ([3–5]) since it is important to guarantee the existence of a feed-forward neural network f∈X that well approximates an unknown continuous function. Here, the topology of Cℝ is generated by the seminorms h⟼supx∈Khx, where K ranges over all compact sets in ℝ. Thus, the approximation property of two-layer feed-forward neural networks makes sense only on a local domain.In this study, we prove an approximation property ofX in a global sense. More precisely, we prove the space X is dense in the Banach subspace of Cℝ defined as
(3)Y≔f∈Cℝlimx⟶±∞fx1+xexistsanditisfinite,equipped with the norm
(4)fY≔supx∈ℝfx1+x.Note that any element inY, divided by 1+·, is a continuous function over ℝ¯≔ℝ∪±∞. Our main result in this paper is as follows:Theorem 1.
The linear subspaceX is dense in Y.Our main results claim that any functionf∈X is close to a linear function both at ∞ and at −∞. Near the origin, X approximates any continuous functions.Before we conclude this section, we will offer some words on some existing results. See [6] for the L2-approximation over the real line. Other attempts have been made to grasp the neural network by the use of the Radon transform [7] or by considering some other topologies [5, 8].


## 2. Proof of the Main Theorem

Definition 2.
We define a linear operatorA:f∈Y⟼f/1+·∈BCℝ¯.Lemma 3.
The operatorA:Y⟶BCℝ¯ is an isomorphism from Y to BCℝ¯.A tacit understanding here is that we extendf/1+·, which is initially defined over ℝ, continuously to ℝ¯.Thus, any continuous functional onY is realized by a Borel measure over ℝ¯.Our theorem can recapture the case where the underlying domain is bounded. Indeed, if the domainΩ is contained in −R,R for some R>0, then, we have
(5)fL∞Ω≤1+RfYf∈Y,which will give results by Cybenko [3] and Funahashi [4].Now we start the proof of Theorem1. As Cybenko did in [3], take any measure μ over ℝ¯ such that μ annihilates X. We will show that μ=0. Once this is proved, from the Riesz representation theorem, we conclude that the only linear functional that vanishes on X is zero. Using the Hahn-Banach theorem, we see that X is dense in Y.Remark that(6)max1−x−1,0=ReLUx+ReLUx−2−2ReLUx−1x∈ℝ.Thus, any element inCcℝ can be approximated by a function X in the L∞-norm. Since μ annihilates Ccℝ, it follows that μ is not supported on ℝ. Or equivalently, μ is supported on ±∞. It remains to show that μ±∞=0. Consider
(7)fx=ReLUx−ReLUx−1x∈ℝ.Remark that(8)0=∫ℝ¯fxdμx=μ∞.Likewise, if we test the condition ong=f−·, we obtain μ−∞=0.Thus, we conclude thatX is dense in Y.Remark 4.
The setf,g∪Ccℝ spans a dense subspace in Y, where f and g are functions given in the above proof.



---

*Source: 6637220-2021-09-17.xml*