Training Neural Networks is ER-complete

1 Introduction . Training neural networks is a fundamental problem in machine learning . An ( artificial ) neural network is a brain-inspired computing system . Neural networks are modelled by directed acyclic graphs where the vertices are called neurons ; for an example consider Figure 1 . In this paper , we show that it is ∃R-complete to decide if there exists weights and biases that will result in a cost below a given threshold . Loosely speaking , the complexity class ∃R is defined as problems that are equivalent to deciding whether a system of polynomial equations and inequalities and many real unknowns has a solution , and it is known that NP ⊆ ∃R ⊆ PSPACE , although none of these inclusions are known to be strict . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . It is already well known that training neural networks is NP-hard , so one might wonder what we learn from the ∃R-completeness . There exists a large variety of tools for approaching NP-complete problems : SAT solvers , IP solvers , local search , dynamic programming , divide and conquer , prune and search , tree-width based algorithms , or even brute force approaches . Note that local search refers to a technique where the solution is locally changed combinatorially in a small number of components and thus , local search is not related to gradient descent . In contrast , all research ( we are aware of ) that actively tries to train neural networks uses variants of backpropagation . Many of the fast and reliable off-the-shelve algorithms solving large instances of NP-complete problems optimally are easy to use , fast , need little to no fine-tuning , and are able to give negative certificates . This makes them much better than backpropagation . So one wonders why backpropagation works better for training neural networks than these strong techniques ? Our result offers the following answers . Assuming NP 6= ∃R as widely believed , ∃R-hardness gives a strong argument that techniques working for NP-complete problems will not work for training neural networks . As a consequence , our work offers explanations for the observed good experiences with backpropagation when compared to the classical methods . Neural networks have shown to be superior for solving very complicated tasks , for instance in the context of natural language processing or image recognition . In order to model languages or recognize images , discrete structures ( e.g. , structures avoiding the use of floating points ) are much easier to understand and manipulate and thus in principle preferable over neural networks . This raises the question why we do not use discrete structures like formal grammars in order to describe languages rather than neural networks . So far , we can only reason as follows : Modelling languages using grammars was tried over decades but neural networks outperform formal grammars on all benchmark tasks . Yet , it leaves the question lingering : Have we maybe just not tried hard enough to model image processing and other problems by discrete structures ? As mentioned before , it is widely believed that ∃R-complete problems are strictly more difficult than NP-complete problems . In particular , ∃R-complete problems are more expressible and go beyond discrete structures , as they have an inherently continuous nature . In this sense , the fact that training neural networks is complete for ∃R indicates that neural networks are a more powerful tool than any purely discrete structures , as they are able to model much richer objects . Limitations . While ∃R-completeness gives an interesting perspective on training neural networks , we should also keep the followings facts in mind when interpreting . • In order to prove hardness , we carefully choose the architecture of our neural network to get an instance that is particularly difficult to train . In practice , one would often use a fully connected neural network . It remains an interesting question if this is also ∃R-complete ( see also Section 4 ) . • In practice , one might not care about finding the exact optimum when training a neural network . For instance , we could ask for a solution that gives a training error of at most OPT + ε . Here , OPT is the minimum training error and ε > 0 is a small input parameter . We do not know if this version is also hard . • We show that the problem of exact cost minimization is ∃R-complete . This presupposes that we do all calculations exactly when evaluating the network on an input vector . In practice floating point numbers of 32 bits , or even as little as 8 bits [ 39 ] , are used , which naturally limits the precision of all calculations . Hence , our result does strictly speaking not imply hardness of the problem from a practical perspective . Given these limitations , we think that it is appropriate to regard ∃R-completeness of the training neural network problem as an indication of its practical difficulty , but not as an ultimate truth . Similar limitations hold for many other algorithmic complexity results . 1.1 Definition of the training problem . The source nodes of the graph representing a neural network are called the input neurons and the sinks are called output neurons , and all other neurons are said to be hidden . A network computes in the following way : Each input neuron s receives an input signal ( a real number ) which is sent through all out-going edges to the neurons that s points to . A non-input neuron v receives signals through the incoming edges , and v then processes the signals and transmits a single output signal to all neurons that v points to . The values computed by the output neurons are the result of the computation of the network . A neuron v evaluates the input signals by a so-called activation function ϕv . Each edge has a weight that scales the signal transmitted through the edge . Similarly , each neuron v often has a bias bv that is added to the input signals . Denoting the unweighted input values to a neuron v by x ∈ Rk and the corresponding edge weights by w ∈ Rk , then the output of v is given by ϕv ( 〈w , x〉+ bv ) . During a training process , the network is fed with input values for which the true output values are known . The task is then to adjust the weights and the biases so that the network produces outputs that are close to the ground truth specified by the training data . We formalize this problem in the following definition . Definition 1 ( Training Neural Networks ) . The problem of training a neural network ( NN-TRAINING ) has the following inputs explained above : • A neural network architecture N = ( V , E ) , where some S ⊂ V are input neurons and have in-degree 0 and some T ⊂ V are output neurons and have out-degree 0 , • an activation function ϕv : R −→ R for each neuron v ∈ V \ S , • a cost function c : R2|T | −→ R≥0 , • a threshold δ ≥ 0 , and • a set of data points D ⊂ R|S|+|T | . Here , each data point d ∈ D has the form d = ( x1 , . . . , x|S| ; y1 , . . . , y|T | ) , where x ( d ) = ( x1 , . . . , x|S| ) specifies the values to the input neurons and y ( d ) = ( y1 , . . . , y|T | ) are the associated ground truth output values . If the actual values computed by the network are y′ ( d ) = ( y′1 , . . . , y ′ |T | ) , then the cost is c ( y ( d ) , y′ ( d ) ) . The total cost is then C ( D ) = ∑ d∈D c ( y ( d ) , y′ ( d ) ) . We seek to answer the following question . Do there exist weights and biases ofN such thatC ( D ) ≤ δ ? We say that a cost function c is honest if it satisfies that c ( y ( d ) , y′ ( d ) ) = 0 if and only if y ( d ) = y′ ( d ) . An example of an honest cost function is the popular mean of squared errors : c ( y ( d ) , y′ ( d ) ) = 1 |T | |T |∑ i=1 ( yi − y′i ) 2 . 1.2 The Result . As our main result , we determine the algorithmic complexity of the fundamental problem NNTRAINING . Theorem 2 . NN-TRAINING is ∃R-complete , even if • the neural network has only one layer of hidden neurons and three output neurons , • all neurons use the identity function ϕ ( x ) = x as activation function , • any honest cost function c is used , • each data point is in { 0 , 1 } |S|+|T | , • the threshold δ is 0 , and • there are only three output neurons . Organization We show that NN-TRAINING is contained in ∃R in Section 2 and prove its hardness in Section 3 . We also discusses how our proof could be modified to work with the ReLu activation function . Section 4 contains a discussion and open problems . In the remainder of the introduction , we familiarize the reader with the complexity class ∃R , discuss the practical implications of our result , and give an overview of related complexity results on neural network training . 1.3 The Existential Theory of the Reals . In the problem ETR , we are given a logical expression ( using ∧ and ∨ ) involving polynomial equalities and inequalities with integer coefficients , and the task is to decide if there exist real variables that satisfy the expression . One example is ∃x , y ∈ R : ( x2 + y2 − 6x− 4y + 13 ) · ( y4 − 4y3 + x2 + 6y2 + 2x− 4y + 2 ) = 0∧ ( x ≥ 4 ∨ y > 1 ) . This is a yes instance because ( x , y ) = ( 3 , 2 ) is a solution . Due to deep connections to many related fields , ETR is a fundamental and well-studied problem in Mathematics and Computer Science . Despite its long history , we still lack algorithms that can solve ETR efficiently in theory and practice . The Existential Theory of the Reals , denoted by ∃R , is the complexity class of all decision problems that are equivalent under polynomial time many-one reductions to ETR . Its importance is reflected by the fact that many natural problems are ∃R-complete . Famous examples from discrete geometry are the recognition of geometric structures , such as unit disk graphs [ 29 ] , segment intersection graphs [ 28 ] , stretchability [ 32 , 37 ] , and order type realizability [ 28 ] . Other ∃R-complete problems are related to graph drawing [ 27 ] , statistical inference [ 9 ] , Nash-Equilibria [ 20 , 7 ] , Geometricembeddability of simplicial complexes [ 4 ] geometric packing [ 5 ] , the art gallery problem [ 3 ] , nonnegative matrix factorization [ 36 ] , continuous constraint satisfaction problems [ 31 ] and geometric linkage constructions [ 1 ] . We refer the reader to the lecture notes by [ 28 ] and surveys by [ 35 ] and [ 13 ] for more information on the complexity class ∃R . As mentioned before , establishing ∃R-completeness of a problem gives us a better understanding of the inherent difficulty of finding exact algorithms for it . Problems that are ∃R-complete often require irrational numbers of arbitrarily high algebraic degree or doubly exponential precision to describe valid solutions . These phenomena make it hard to find efficient algorithms . We know that NP ⊆ ∃R ⊆ PSPACE [ 12 ] , and both inclusions are believed to be strict , although this remains an outstanding open question in the field of complexity theory . In a classical view of complexity , we distinguish between problems that we can solve in polynomial time and intractable problems that may not be solvable in polynomial time . Usually , knowing that a problem is NP-hard is argument enough to convince us that we can not solve the problem efficiently . Yet , assuming NP 6= ∃R , ∃R-hardness proves the limitations of NP methods . To give a simple example , NP-complete problems can be solved in a brute-force fashion by exhaustively going through all possible solutions . Although this method is not very sophisticated , it is good enough to solve small sized instances . The same is not possible for ∃R-complete problems due to their continuous nature . As a second example , the difficulty of solving ∃R-complete problems is nicely illustrated by the problem of placing eleven unit squares into a minimum sized square container without overlap . Whether a given square can contain eleven unit squares can be expressed as an ETR-formula of modest size , so if such formulas could be solved efficiently , we would know the answer ( at least to within any desired accuracy ) . However , it is only known that the sidelength is between 2 + 4/ √ 5 ≈ 3.788 and 3.878 [ 21 ] .

Summary: It is known that training neural networks is NP-hard, and this work makes a further attempt by studying the ER-hardness of the problem. The definition of 'neural network training' used in the proof of this paper is: do there exist weights such that the neural net maps every data input to corresponding output? This work proves that training neural networks is ER-complete, which is believed to be harder than NP.