# ReLU Network with Bounded Width Is a Universal Approximator in View of an Approximate Identity
## 3 Proof of Theorem 1
### General-Dimensional Input  
To generalize our idea on input dimension \(1\) to general input dimension \(n_{in}\), the main difficult part is how to make a ReLU network an approximate identity, for example, a kind of a \(B\)-spline function on \(\mathbb{R}^{2}\). We emphasize that we need a function with the shape of a \(B\)-spline function, not an exact \(B\)-spline function on \(\mathbb{R}^{2}\) (see Figure 1). For \(n=2\), we have  
\[\operatorname{ReLU}(x_{2}+(\operatorname{ReLU}(x_{1}+1)-2 \operatorname{ReLU}(x_{1})+\operatorname{ReLU}(x_{1}-1)))-2\operatorname{ ReLU}(x_{2})\] \[+\operatorname{ReLU}(x_{2}-(\operatorname{ReLU}(x_{1}+1)-2 \operatorname{ReLU}(x_{1})+\operatorname{ReLU}(x_{1}-1)))\] \[=\left\{ \begin{array}{ll}0&\text{if}\quad|x_{1}|+|x_{2}|>1\\ 1-|x_{1}|-|x_{2}|&\text{if}\quad|x_{1}|+|x_{2}|<1.\end{array} \right.\]  
In general,  
**Lemma 1**. : _For \(b\in\mathbb{R}\), let \(h(x_{1},b)=\operatorname{ReLU}(x_{1}+b)-2\operatorname{ReLU}(x_{1})+ \operatorname{ReLU}(x_{1}-b)\). Then, we have_  
\[h(x_{n},h(x_{n-1},h(x_{n-2},\cdots,h(x_{1},1))))=\left\{ \begin{array}{ll}0& \text{if}\quad\sum_{j=1}^{n}|x_{j}|\geq 1,\\ 1-\sum_{j=1}^{n}|x_{j}|&\text{if}\quad\sum_{j=1}^{n}|x_{j}|<1.\end{array} \right.\]  
Figure 1: \(g(\mathbf{x})\) Left: \(1\)-dimension Center: \(2\)-dimension Right: \(3\)-dimension with range. _In addition, we have_  
\[\int\limits_{\mathbb{R}^{n}}h(x_{n},h(x_{n-1},h(x_{n-2},\cdots,h(x_{1},1)))) \mathrm{d}\mathbf{x}=\frac{2^{n}}{(n+1)!}. \tag{1}\]  
We notice that for  
\[g(\mathbf{x})=h(x_{n},h(x_{n-1},h(x_{n-2},\cdots,h(x_{1},1)))),\quad\mathbf{x} \in\mathbb{R}^{n}\]  
(see Figure 1) and \(\frac{(n+1)!}{2^{n}}g_{\lambda}(\mathbf{x})=\frac{(n+1)! }{2^{n}\lambda^{n}}g( \lambda^{-1}\mathbf{x})\), \(\lambda>0\) is an approximate identity: for a continuous function \(f\) on \([-M,M]^{n}\), \(\frac{(n+1)! }{2^{n}}f*g_{\lambda}\) converges to \(f(\mathbf{x})\) uniformly. For any \(\epsilon>0\), there is \(\lambda>0\) such that  
\[\sup_{\mathbf{x}\in[-M,M]^{n}\mid_{in}}\left|\frac{(n+1)!}{2^{n}}f*g_{\lambda} (\mathbf{x})-f(\mathbf{x})\right|<\frac{\epsilon}{2}. \tag{2}\]  
Proof. : We will use mathematical induction on \(n\). We have already shown the case of \(n=1\). Suppose  
\[h(x_{n},h(x_{n-1},\cdots,h(x_{1},1)))=\left\{ \begin{array}{ll}0&\text{if} \quad\sum\limits_{j=1}^{n}|x_{j}|\geq 1,\\ 1-\sum\limits_{j=1}^{n}|x_{j}|&\text{if}\quad\sum\limits_{j=1}^{n}|x_{j}|<1. \end{array} \right.\]  
For simplicity, let \(g(\mathbf{x})=h(x_{n},h(x_{n-1},\cdots,h(x_{1},1)))\), \(\mathbf{x}\in\mathbb{R}^{n}\). If \(\sum_{j=1}^{n+1}|x_{j}|\geq 1\), then \(|x_{n+1}|\geq 1-\sum_{j=1}^{n}|x_{j}|\) and thus \(|x_{n+1}|\geq g(\mathbf{x})\). For \(x_{n+1}\geq 0\),  
\[ \begin{array}{ll}h(x_{n+1},g(\mathbf{x}))&=\text{ReLU}(x_{n+1}+g(\mathbf{x} ))-2\,\text{ReLU}(x_{n+1})+\text{ReLU}(x_{n+1}-g(\mathbf{x}))\\ &=(x_{n+1}+g(\mathbf{x}))-2x_{n+1}+(x_{n+1}-g(\mathbf{x}))\end{array} \]  
is equal to zero. In addition, for \(x_{n+1}<0\), we can show that \(h(x_{n+1},g(\mathbf{x}))\) is also zero. Now, we consider \(\sum_{j=1}^{n+1}|x_{j}|<1\). Then, by hypothesis, \(g(\mathbf{x})=1-\sum_{j=1}^{n}|x_{j}|\). For \(x_{n+1}>0\), we have  
\[ \begin{array}{ll}h(x_{n+1},g(\mathbf{x}))&=\text{ReLU}(x_{n+1}+g(\mathbf{x }))-2\,\text{ReLU}(x_{n+1})+\text{ReLU}(x_{n+1}-g(\mathbf{x}))\\ &=x_{n+1}+1-\sum\limits_{j=1}^{n}|x_{j}|-2x_{n+1}=1-\sum\limits_{j=1}^{n+1}|x_ {j}|,\end{array} \]  
since \(x_{n+1}-1+\sum_{j=1}^{n}|x_{j}|<0\). Similarly, for \(x_{n+1}<0\), we have  
\[ \begin{array}{ll}h(x_{n+1},g(\mathbf{x}))=x_{n+1}+1-\sum\limits_{j=1}^{n}|x _{j}|=1-\sum\limits_{j=1}^{n+1}|x_{j}|.\end{array} \]  
Direct integral calculation follows (1). Again, similar to the one-dimensional case, by mensuration by division, there is \(k\in\mathbb{N}\) such that and thus, with (2), we have for some \(\lambda>0\) and \(k\in\mathbb{N}\),  
\[\left|