Training Neural Networks is ER-complete

1 Introduction . Training neural networks is a fundamental problem in machine learning . An ( artificial ) neural network is a brain-inspired computing system . Neural networks are modelled by directed acyclic graphs where the vertices are called neurons ; for an example consider Figure 1 . In this paper , we show that it is ∃R-complete to decide if there exists weights and biases that will result in a cost below a given threshold . Loosely speaking , the complexity class ∃R is defined as problems that are equivalent to deciding whether a system of polynomial equations and inequalities and many real unknowns has a solution , and it is known that NP ⊆ ∃R ⊆ PSPACE , although none of these inclusions are known to be strict . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . It is already well known that training neural networks is NP-hard , so one might wonder what we learn from the ∃R-completeness . There exists a large variety of tools for approaching NP-complete problems : SAT solvers , IP solvers , local search , dynamic programming , divide and conquer , prune and search , tree-width based algorithms , or even brute force approaches . Note that local search refers to a technique where the solution is locally changed combinatorially in a small number of components and thus , local search is not related to gradient descent . In contrast , all research ( we are aware of ) that actively tries to train neural networks uses variants of backpropagation . Many of the fast and reliable off-the-shelve algorithms solving large instances of NP-complete problems optimally are easy to use , fast , need little to no fine-tuning , and are able to give negative certificates . This makes them much better than backpropagation . So one wonders why backpropagation works better for training neural networks than these strong techniques ? Our result offers the following answers . Assuming NP 6= ∃R as widely believed , ∃R-hardness gives a strong argument that techniques working for NP-complete problems will not work for training neural networks . As a consequence , our work offers explanations for the observed good experiences with backpropagation when compared to the classical methods . Neural networks have shown to be superior for solving very complicated tasks , for instance in the context of natural language processing or image recognition . In order to model languages or recognize images , discrete structures ( e.g. , structures avoiding the use of floating points ) are much easier to understand and manipulate and thus in principle preferable over neural networks . This raises the question why we do not use discrete structures like formal grammars in order to describe languages rather than neural networks . So far , we can only reason as follows : Modelling languages using grammars was tried over decades but neural networks outperform formal grammars on all benchmark tasks . Yet , it leaves the question lingering : Have we maybe just not tried hard enough to model image processing and other problems by discrete structures ? As mentioned before , it is widely believed that ∃R-complete problems are strictly more difficult than NP-complete problems . In particular , ∃R-complete problems are more expressible and go beyond discrete structures , as they have an inherently continuous nature . In this sense , the fact that training neural networks is complete for ∃R indicates that neural networks are a more powerful tool than any purely discrete structures , as they are able to model much richer objects . Limitations . While ∃R-completeness gives an interesting perspective on training neural networks , we should also keep the followings facts in mind when interpreting . • In order to prove hardness , we carefully choose the architecture of our neural network to get an instance that is particularly difficult to train . In practice , one would often use a fully connected neural network . It remains an interesting question if this is also ∃R-complete ( see also Section 4 ) . • In practice , one might not care about finding the exact optimum when training a neural network . For instance , we could ask for a solution that gives a training error of at most OPT + ε . Here , OPT is the minimum training error and ε > 0 is a small input parameter . We do not know if this version is also hard . • We show that the problem of exact cost minimization is ∃R-complete . This presupposes that we do all calculations exactly when evaluating the network on an input vector . In practice floating point numbers of 32 bits , or even as little as 8 bits [ 39 ] , are used , which naturally limits the precision of all calculations . Hence , our result does strictly speaking not imply hardness of the problem from a practical perspective . Given these limitations , we think that it is appropriate to regard ∃R-completeness of the training neural network problem as an indication of its practical difficulty , but not as an ultimate truth . Similar limitations hold for many other algorithmic complexity results . 1.1 Definition of the training problem . The source nodes of the graph representing a neural network are called the input neurons and the sinks are called output neurons , and all other neurons are said to be hidden . A network computes in the following way : Each input neuron s receives an input signal ( a real number ) which is sent through all out-going edges to the neurons that s points to . A non-input neuron v receives signals through the incoming edges , and v then processes the signals and transmits a single output signal to all neurons that v points to . The values computed by the output neurons are the result of the computation of the network . A neuron v evaluates the input signals by a so-called activation function ϕv . Each edge has a weight that scales the signal transmitted through the edge . Similarly , each neuron v often has a bias bv that is added to the input signals . Denoting the unweighted input values to a neuron v by x ∈ Rk and the corresponding edge weights by w ∈ Rk , then the output of v is given by ϕv ( 〈w , x〉+ bv ) . During a training process , the network is fed with input values for which the true output values are known . The task is then to adjust the weights and the biases so that the network produces outputs that are close to the ground truth specified by the training data . We formalize this problem in the following definition . Definition 1 ( Training Neural Networks ) . The problem of training a neural network ( NN-TRAINING ) has the following inputs explained above : • A neural network architecture N = ( V , E ) , where some S ⊂ V are input neurons and have in-degree 0 and some T ⊂ V are output neurons and have out-degree 0 , • an activation function ϕv : R −→ R for each neuron v ∈ V \ S , • a cost function c : R2|T | −→ R≥0 , • a threshold δ ≥ 0 , and • a set of data points D ⊂ R|S|+|T | . Here , each data point d ∈ D has the form d = ( x1 , . . . , x|S| ; y1 , . . . , y|T | ) , where x ( d ) = ( x1 , . . . , x|S| ) specifies the values to the input neurons and y ( d ) = ( y1 , . . . , y|T | ) are the associated ground truth output values . If the actual values computed by the network are y′ ( d ) = ( y′1 , . . . , y ′ |T | ) , then the cost is c ( y ( d ) , y′ ( d ) ) . The total cost is then C ( D ) = ∑ d∈D c ( y ( d ) , y′ ( d ) ) . We seek to answer the following question . Do there exist weights and biases ofN such thatC ( D ) ≤ δ ? We say that a cost function c is honest if it satisfies that c ( y ( d ) , y′ ( d ) ) = 0 if and only if y ( d ) = y′ ( d ) . An example of an honest cost function is the popular mean of squared errors : c ( y ( d ) , y′ ( d ) ) = 1 |T | |T |∑ i=1 ( yi − y′i ) 2 . 1.2 The Result . As our main result , we determine the algorithmic complexity of the fundamental problem NNTRAINING . Theorem 2 . NN-TRAINING is ∃R-complete , even if • the neural network has only one layer of hidden neurons and three output neurons , • all neurons use the identity function ϕ ( x ) = x as activation function , • any honest cost function c is used , • each data point is in { 0 , 1 } |S|+|T | , • the threshold δ is 0 , and • there are only three output neurons . Organization We show that NN-TRAINING is contained in ∃R in Section 2 and prove its hardness in Section 3 . We also discusses how our proof could be modified to work with the ReLu activation function . Section 4 contains a discussion and open problems . In the remainder of the introduction , we familiarize the reader with the complexity class ∃R , discuss the practical implications of our result , and give an overview of related complexity results on neural network training . 1.3 The Existential Theory of the Reals . In the problem ETR , we are given a logical expression ( using ∧ and ∨ ) involving polynomial equalities and inequalities with integer coefficients , and the task is to decide if there exist real variables that satisfy the expression . One example is ∃x , y ∈ R : ( x2 + y2 − 6x− 4y + 13 ) · ( y4 − 4y3 + x2 + 6y2 + 2x− 4y + 2 ) = 0∧ ( x ≥ 4 ∨ y > 1 ) . This is a yes instance because ( x , y ) = ( 3 , 2 ) is a solution . Due to deep connections to many related fields , ETR is a fundamental and well-studied problem in Mathematics and Computer Science . Despite its long history , we still lack algorithms that can solve ETR efficiently in theory and practice . The Existential Theory of the Reals , denoted by ∃R , is the complexity class of all decision problems that are equivalent under polynomial time many-one reductions to ETR . Its importance is reflected by the fact that many natural problems are ∃R-complete . Famous examples from discrete geometry are the recognition of geometric structures , such as unit disk graphs [ 29 ] , segment intersection graphs [ 28 ] , stretchability [ 32 , 37 ] , and order type realizability [ 28 ] . Other ∃R-complete problems are related to graph drawing [ 27 ] , statistical inference [ 9 ] , Nash-Equilibria [ 20 , 7 ] , Geometricembeddability of simplicial complexes [ 4 ] geometric packing [ 5 ] , the art gallery problem [ 3 ] , nonnegative matrix factorization [ 36 ] , continuous constraint satisfaction problems [ 31 ] and geometric linkage constructions [ 1 ] . We refer the reader to the lecture notes by [ 28 ] and surveys by [ 35 ] and [ 13 ] for more information on the complexity class ∃R . As mentioned before , establishing ∃R-completeness of a problem gives us a better understanding of the inherent difficulty of finding exact algorithms for it . Problems that are ∃R-complete often require irrational numbers of arbitrarily high algebraic degree or doubly exponential precision to describe valid solutions . These phenomena make it hard to find efficient algorithms . We know that NP ⊆ ∃R ⊆ PSPACE [ 12 ] , and both inclusions are believed to be strict , although this remains an outstanding open question in the field of complexity theory . In a classical view of complexity , we distinguish between problems that we can solve in polynomial time and intractable problems that may not be solvable in polynomial time . Usually , knowing that a problem is NP-hard is argument enough to convince us that we can not solve the problem efficiently . Yet , assuming NP 6= ∃R , ∃R-hardness proves the limitations of NP methods . To give a simple example , NP-complete problems can be solved in a brute-force fashion by exhaustively going through all possible solutions . Although this method is not very sophisticated , it is good enough to solve small sized instances . The same is not possible for ∃R-complete problems due to their continuous nature . As a second example , the difficulty of solving ∃R-complete problems is nicely illustrated by the problem of placing eleven unit squares into a minimum sized square container without overlap . Whether a given square can contain eleven unit squares can be expressed as an ETR-formula of modest size , so if such formulas could be solved efficiently , we would know the answer ( at least to within any desired accuracy ) . However , it is only known that the sidelength is between 2 + 4/ √ 5 ≈ 3.788 and 3.878 [ 21 ] .

Summary: This paper proves that THE "Neural Network Training" problem is complete in a complexity class called ETR that is believed to be strictly larger than the class NP. Thus, it establishes a stronger hardness result than the well-known NP-Hardness for the problem, and shows that the problem is, in fact, probably not NP-Complete.  ETR stands for Existential Theory of the Reals and it can be thought of as a continuous analog of NP. The canonical complete problem for this class is similar to SAT by instead of boolean clauses we have algebraic constraints, and instead of a satisfying boolean assignment to the variables we are looking for an assignment of real values.  The result is proved by: (1) Proving containment in ETR: this is the easy direction and follows from known methods for proving containment in ETR. (2) Proving ETR-Hardness via a reduction from the canonical ETR problem. This is done by simple but clever gadgets that encode algebraic constraints with neural networks and data points.  This result is proved even for very restricted settings of the NN-Training problem, with one hidden layer and standard activation functions and so on.