Polynomial Regression As an Alternative to Neural Nets

Despite the success of neural networks (NNs), there is still a concern among many over their “black box” nature. Why do they work? Yes, we have Universal Approximation Theorems, but these concern statistical consistency, a very weak property, not enough to explain the exceptionally strong performance reports of the method. Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models, with the effective degree of the polynomial growing at each hidden layer. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available. 1 ar X iv :1 80 6. 06 85 0v 2 [ cs .L G ] 2 9 Ju n 20 18 1 The Mystery of NNs Neural networks (NNs), especially in the currently popular form of many-layered deep learning networks (DNNs), have become many analysts’ go-to method for predictive analytics. Indeed, in the popular press, the term artificial intelligence has become virtually synonymous with NNs.1 Yet there is a feeling among many in the community that NNs are “black boxes”; just what is going on inside? Various explanations have been offered for the success of NNs, a prime example being [Shwartz-Ziv and Tishby(2017)]. However, the present paper will present significant new insights. 2 Contributions of This Paper The contribution of the present work will be as follows:2 (a) We will show that, at each layer of an NY, there is a rough correspondence to some fitted ordinary parametric polynomial regression (PR) model; in essence, NNs are a form of PR. We refer to this loose correspondence here as NNAEPR, Neural Nets Are Essentially Polynomial Models. (b) A very important aspect of NNAEPR is that the degree of the approximating polynomial increases with each hidden layer. In other words, our findings should not be interpreted as merely saying that the end result of an NN can be approximated by some polynomial. (c) We exploit NNAEPR to learn about general properties of NNs via our knowledge of the properties of PR. This will turn out to provide new insights into aspects such as the numbers of hidden layers and numbers of units per layer, as well as how convergence problems arise. For example, we use NNAEPR to predict and confirm a multicollinearity property of NNs not previous reported in the literature. (d) Property (a) suggests that in many applications, one might simply fit a polynomial model in the first place, bypassing NNs. This would have the advantage of avoiding the problems of choosing tuning parameters (the polynomial approach has just one, the degree), nonconvergence and so on. 1There are many different variants of NNs, but for the purposes of this paper, we can consider them as a group. 2 Author listing is alphabetical by surname. XC wrote the entire core code for the polyreg package; NM conceived of the main ideas underlying the work, developed the informal mathematical material and wrote support code; BK assembled the brain and kidney cancer data, wrote some of the support code, and provided domain expertise guidance for genetics applications; PM wrote extensive support code, including extending his kerasformula package, and provided specialized expertise on NNs. All authors conducted data experiments.