Fitting Small Piece-Wise Linear Neural Network Models to Interpolate Data Sets

The simplest neural network models for functions are compositions of layer functions, where each layer function applies a nonlinear activation function σ, which is not a polynomial, component-wise to an affine linear functions WX + B. The rows of W are referred to as ‘weights’ and B is the ‘bias vector’. The Universal Approximation Theorem for Neural Networks implies that any continuous function on a compact set can be approximated arbitrarily precisely using this class of models. This rich class of functions is proving to be remarkably useful and underpins the explosion of artificial intelligence applications. Neural network model architectures typically use compositions of many layer functions defined in terms of a huge number of parameters. The parameters are typically fit by minimizing a loss function using gradient descent. Mathematically, one is seeking a representation computed by a composition of all but the final layer function that accurately predicts the function values using the final layer function (essentially an affine linear function). However, the use of many layers in combination with gradient descent results in models which are not easy to understand and visualize. This makes it difficult for mathematicians and engineers who are interested in learning more about neural networks and their mathematical underpinnings to acquire an initial concrete understanding. In this paper we focus on explicitly demonstrating the universal approximation property of neural network models for the most commonly used activation function, the Relu function. Relu models provide closed form expressions for a huge class of piece-wise linear functions, a class of functions not typically studied. We exploit the particular nature of Relu activation function to define three small piece-wise linear neural network models which interpolate a given finite set of example training and label data while having a minimal number of linear pieces. We avoid the use of gradient descent to fit the models by proving for these simple models that the interpolation equations for the top layer weight matrix W and bias B have a simple closed form algebraic solution, providing the first layer weights satisfy a mild generic property and the biases are defined in a particular data dependent manner. We also prove that if the weights are further restricted to be norm one and minimize a sequential variation property, the piecewise-line models avoid over-fitting in the sense that they have a minimal number of pieces. We prove that sequential variation is constant on a finite number of sectors and hence easy to minimize and sample. The number of sectors is determined by the number of data points, not the dimension, avoiding the ‘curse of dimensionality’. The minimum sequential variation distribution characterizes the distribution parameters of all of the simplest models, enabling computation of average models. This is not possible when gradient descent is used. One of our models adds an initial layer which uses the above concepts to construct a small binary tree representation, representing the training data in binary coordinates. The models can be simply computed using elementary mathematical operations for small and possibly large data sets. The concepts and models are illustrated by graphing results on small two-dimensional data sets. We hope that the example models will provide some concrete insights into the Relu neural network model for representing functions. No knowledge of neural network models or deep learning methods is assumed.