Theorem: Universal Approximation Theorem

Let $I_n$ be the $n$-dimensional unit hypercube $\closedint 0 1^n$.
Let $\map C {I_n}$ be the space of continuous functions on $I_n$, equipped with the supremum norm:
:$\ds \norm f = \sup_{x \mathop \in I_n} \size {\map f x}$
Let $\sigma: \R \to \R$ be a continuous sigmoidal function.
That is, $\sigma$ satisfies:
:$\ds \lim_{t \mathop \to -\infty} \map \sigma t = 0$
and:
:$\ds \lim_{t \mathop \to \infty} \map \sigma t = 1$
Let $S$ be the set of functions $G: I_n \to \R$ of the form:
:$\ds \map G x = \sum_{j \mathop = 1}^N \alpha_j \map \sigma {y_j^\intercal x + \theta_j}$
where:
:$N \in \N$ is the number of neurons in the hidden layer
:$y_j \in \R^n$ represents the weights
:$\theta_j \in \R$ represents the biases
:$\alpha_j \in \R$ represents the output weights.
Then the set $S$ is dense in $\map C {I_n}$.
That is, for any $f \in \map C {I_n}$ and any $\epsilon > 0$, there exists a function $G \in S$ such that:
:$\size {\map G x - \map f x} < \epsilon$
for all $x \in I_n$.

Proof:
Let $S$ be the linear subspace of $\map C {I_n}$ spanned by functions of the form $\map \sigma {y^\intercal x + \theta}$.
We aim to show that the closure $\bar S = \map C {I_n}$.
We proceed by Proof by Contradiction.
{{AimForCont}} $\bar S$ is a proper subspace of $\map C {I_n}$, that is $\bar S \subsetneq \map C {I_n}$.
By the Hahn-Banach Theorem, there exists a bounded linear functional $L$ on $\map C {I_n}$ such that:
:$L \ne 0$
and:
:$\map L g = 0$ for all $g \in \bar S$.
By the Riesz Representation Theorem, this bounded linear functional $L$ can be represented by a finite signed regular Borel measure $\mu$ on $I_n$.
Thus, for all $h \in \map C {I_n}$:
:$\ds \map L h = \int_{I_n} \map h x \rd \map \mu x$
Since $L$ annihilates $S$, we have:
:$\ds \int_{I_n} \map \sigma {y^\intercal x + \theta} \rd \map \mu x = 0$
for all $y \in \R^n$ and $\theta \in \R$.
{{improve|As suggested in a previous commit, extract the following into a separate page}}
We now show that this implies $\mu = 0$, a property known as being discriminatory.
Fix $y \in \R^n$ and $\theta \in \R$. Consider the function parameterized by $\lambda > 0$:
:$\ds \map {f_\lambda} x = \map \sigma {\lambda \paren {y^\intercal x + \theta} + \phi}$
where $\phi$ is chosen appropriately.
From the properties of $\sigma$, we have:
:$\ds \lim_{\lambda \mathop \to \infty} \map \sigma {\lambda u} = \begin{cases} 1 & : u > 0 \\ 0 & : u < 0 \end{cases}$
Let $H$ be the half-space defined by $H = \set {x \in I_n : y^\intercal x + \theta > 0}$.
As $\lambda \to \infty$, $f_\lambda$ converges pointwise to the characteristic function (indicator function) $\chi_H$ almost everywhere (specifically, everywhere except the hyperplane $y^\intercal x + \theta = 0$).
Since $\sigma$ is bounded, by the Lebesgue's Dominated Convergence Theorem:
:$\ds \lim_{\lambda \mathop \to \infty} \int_{I_n} \map {f_\lambda} x \rd \map \mu x = \int_{I_n} \map {\chi_H} x \rd \map \mu x = \map \mu H$
Since $\map {f_\lambda} x \in S$, the integral is $0$ by hypothesis. Thus the limit is also $0$.
So:
:$\map \mu H = 0$
for any half-space $H$.
The set of all half-spaces generates the Borel $\sigma$-algebra on $\R^n$.
A measure that is zero on a generating set is the zero measure.
This implies that the only measure orthogonal to all functions of the form $\map \sigma {y^\intercal x + \theta}$ is the zero measure.
Thus we must have $\mu = 0$.
This implies $L = 0$.
This contradicts the statement from the Hahn-Banach Theorem that $L \ne 0$.
Therefore, the assumption that $\bar S \subsetneq \map C {I_n}$ must be false.
So:
:$\bar S = \map C {I_n}$
which means $S$ is dense in $\map C {I_n}$.
{{qed}}