Title: Training Neural Networks is NP-Hard in Fixed Dimension
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computational Complexity

Neural Network

Rectified Linear Unit

Empirical Risk Minimization

Parameterized Complexity
Abstract: We study the parameterized complexity of training two-layer neural networks with respect to the dimension of the input data and the number of hidden neurons, considering ReLU and linear threshold activation functions. Albeit the computational complexity of these problems has been studied numerous times in recent years, several questions are still open. We answer questions by Arora et al. (ICLR 2018) and Khalife and Basu (IPCO 2022) showing that both problems are NP-hard for two dimensions, which excludes any polynomial-time algorithm for constant dimension. We also answer a question by Froese et al. (JAIR 2022) proving W[1]-hardness for four ReLUs (or two linear threshold neurons) with zero training error. Finally, in the ReLU case, we show fixed-parameter tractability for the combined parameter number of dimensions and number of ReLUs if the network is assumed to compute a convex map. Our results settle the complexity status regarding these parameters almost completely.

Training Neural Networks is NP-Hard in Fixed Dimension

Abstract

We study the parameterized complexity of training two-layer neural networks with respect to the dimension of the input data and the number of hidden neurons, considering ReLU and linear threshold activation functions. Albeit the computational complexity of these problems has been studied numerous times in recent years, several questions are still open. We answer questions by Arora et al. [2018, ICLR] and Khalife and Basu [2022, IPCO] showing that both problems are NP-hard for two dimensions, which excludes any polynomial-time algorithm for constant dimension. We also answer a question by Froese et al. [2022, JAIR] proving W[1]-hardness for four ReLUs (or two linear threshold neurons) with zero training error. Finally, in the ReLU case, we show fi xed-parameter tractability for the combined parameter number of dimensions and number of ReLUs if the network is assumed to compute a convex map. Our results settle the complexity status regarding these parameters almost completely.

1 Introduction

Neural networks with recti fi ed linear unit (ReLU) activations are arguably one of the most fundamental models in modern machine learning [Arora et al., 2018, Glorot et al., 2011, LeCun et al., 2015]. To use them as predictors on unseen data, one usually fi rst fi xes an architecture (the graph of the neural network) and then optimizes the weights and biases such that the network performs well on some known training data, with the hope that it will then also generalize well to unseen test data. While the ultimate goal in applications is generalization, empirical risk minimization (that is, optimizing the training error) is an important step in this pipeline and understanding its computational complexity is crucial to advance the theoretical foundations of deep learning.

In this paper, we aim to understand how the choice of different meta-parameters, like the input dimension and the width of the neural network, in fl uences the computational complexity of the training problem. To this end, we focus on two-layer neural networks, which can be seen as the standard building block also for deeper architectures.

b

1

b

2

· · ·

b

k

x

1

x

2

· · ·

x

d

w

1

w

2

w

k

· · ·

a

1

a

2

· · ·

a

k

Formally, a two-layer neural network (see Figure 1) with d input neurons, k hidden ReLU neurons, and a single output neuron computes a map

where w j ∈ R d and a j ∈ { -1 , 1 } are the weights between the layers, b j are the biases at the hidden neurons, and [ x ] + := max(0 , x ) is the recti fi er function. Notice that restricting a j to { -1 , 1 } is without loss of generality because we can normalize by pulling any nonnegative factor into w j and b j . We also study neural networks with linear threshold activation in Section 5.

Given training data x 1 , . . . , x n ∈ R d with labels y 1 , . . . , y n ∈ R , the task of training such a network is to fi nd w j , b j , and a j for each j ∈ [ k ] such that the training error � n i =1 L ( φ ( x i ) , y i ) for a given loss function L : R × R → R ≥ 0 is minimized. Throughout this work, we assume L : R × R → R ≥ 0 to be any loss function with L ( x, y ) = 0 ⇐⇒ x = y . Formally, the decision version of two-layer ReLU neural network training is de fi ned as follows:

2L-RELU-NN-TRAIN( L )

Input:

Data points ( x 1 , y 1 ) , . . . , ( x n , y n ) ∈ R d × R , a number k ∈ N of ReLUs, and a target error γ ∈ R ≥ 0 .

Question:

Are there weights w 1 , . . . , w k ∈ R d , biases b 1 , . . . , b k ∈ R , and coef fi cients a 1 , . . . , a k ∈ { - 1 , 1 } such that

Note that in the over-parameterized case where k ≥ n , the network can exactly fi t any n input points 2 (achieving training error γ = 0 ) [Zhang et al., 2021, Theorem 1]. Thus, we henceforth assume that k < n .

2L-RELU-NN-TRAIN( L ) is known to be NP-hard [Dey et al., 2020, Goel et al., 2021], but all known reductions require the input dimension to be part of the input. The current state-of-the-art exact algorithm for convex loss L is by Arora et al. [2018] and runs in O (2 k n dk poly( L )) time, where L is the input bit-length.

As regards the computational complexity of 2L-RELU-NN-TRAIN( L ), Arora et al. [2018] posed the question(s) whether a running time

'that is polynomial in the data size and/or the number of hidden nodes, assuming that the input dimension is a fi xed constant'

is possible. That is, they asked two questions. The fi rst corresponds to the 'and' statement, which can be phrased as follows:

Question 1 : Is there an algorithm running in ( nk ) f ( d ) poly( L ) time for some function f ?

In other words, the question is whether 2L-RELU-NN-TRAIN( L ) is in the complexity class XP when parameterized by d . The second question corresponding to the 'or' statement can then be interpreted as

Question 2 : Is there an algorithm running in n f ( d ) g ( k, d ) poly( L ) or k f ( d ) g ( n, d ) poly( L ) time for some functions f and g ?

We observe that the second running time is clearly possible since k < n holds by assumption, and hence the algorithm by Arora et al. [2018] runs in g ( n, d ) poly( L ) time. Hence, it remains open whether n f ( d ) g ( k, d ) poly( L ) time is possible, which is equivalent to (uniform) fi xed-parameter tractability with respect to k for every constant d .

Clearly, Question 1 is the stronger statement, that is, a positive answer implies a positive answer to Question 2. Arora et al. [2018] conclude with

'Resolving this dependence on network size would be another step towards clarifying the theoretical complexity of training ReLU DNNs and is a good open question for future research, in our opinion. '

Note that Froese et al. [2022] proved that, for k = 1 , there is no algorithm running in g ( d ) n o ( d ) time unless the Exponential Time Hypothesis fails. Hence, this result already partially answered the two questions above by excluding any algorithm running in n o ( d ) g ( d, k ) poly( L ) time.

Our Contribution. In this paper, we answer Question 1 negatively by showing NP-hardness for d = 2 in Theorem 1, indicating that we cannot get rid of the exponential dependence on the network size in the algorithm by Arora et al. [2018] even if the dimension is fi xed. As regards Question 2, we further exclude (assuming the Exponential Time Hypothesis) any algorithm running in time n o ( d ) g ( d, k ) poly( L ) even for the case γ = 0 and prove W[1]-hardness with respect to d for k = 4 (Theorem 6), which answers an open question by Froese et al. [2022].

We also obtain analogous hardness results if linear threshold activation functions are used instead of ReLUs. As in the ReLU case, it is well-known that training linear threshold networks is NPhard [Blum and Rivest, 1992, Khalife and Basu, 2022]. The running time of the state-of-the-art algorithm due to Khalife and Basu [2022] is polynomial in n for fi xed d and k , but exponential in the latter two parameters. Khalife and Basu [2022] posed an analogous question to Question 1 for linear threshold networks, which we answer negatively in Corollary 7, excluding a polynomial running time even for fi xed dimension. We also show that we cannot expect fi xed-parameter tractability with respect to d even for k = 1 (Corollary 8) and also not for k = 2 and γ = 0 (Corollary 9).

On the positive side, we give an algorithm running in 2 O ( k 2 d ) poly( k, L ) time for ReLU neural networks if γ = 0 and the function computed by the network is assumed to be convex (Theorem 10). Note that this running time yields fi xed-parameter tractability with respect to k for every constant d , and thus answers Question 2 positively for this restricted special case. We would like to emphasize that we do not expect this algorithm to have any practical relevance. We rather see its value in being a puzzle piece for understanding the computational complexity of the training problem.

Implications and Limitations. In the following we provide a brief discussion of the implications and limitations of our results from various perspectives.

Input Dimension. Theorem 1 implies that 2L-RELU-NN-TRAIN( L ) is in fact NP-hard for every fi xed d ≥ 2 . The straight-forward reduction simply pads all the input vectors with d -2 zeros. Similarly, Corollary 7 holds for every fi xed d ≥ 2 .

Target Error. The hardness results Theorems 1 and 6 and Corollaries 7 and 9 also hold for every fi xed γ ≥ 0 . The reduction is straight-forward by introducing a set of incompatible data points which force the network to incur an additional error of γ . For our positive result Theorem 10, however, there is indeed a difference in the complexity between the two cases γ = 0 and γ > 0 . While we show fi xed-parameter tractability for γ = 0 , the same problem is W[1]-hard for γ > 0 , already in the case k = 1 [Froese et al., 2022].

Number of ReLUs. It is not too dif fi cult to see (although it requires some work) that our particular reduction in Theorem 6 can be extended to any k ≥ 4 by introducing more data points far away from the existing data points which enforce the usage of additional ReLUs which then cannot be used to fi t the data points of the actual reduction. Therefore, Theorem 6 holds for every fi xed k ≥ 4 . Similarly, Corollary 8 holds for every fi xed k ≥ 1 and Corollary 9 holds for every fi xed k ≥ 2 .

Other Activation Functions. Our hardness results hold for the piecewise linear ReLU activation function and the piecewise constant linear threshold activation function. Extending them to other piecewise linear or constant activation functions like leaky ReLU or maxout should be straightforward. However, achieving analogous results for smooth activation functions like sigmoids probably requires fundamentally different techniques and is beyond the scope of this paper.

Training vs. Learning. Our results are concerned with the problem of minimizing the training error. While this is inherently different from minimizing the generalization error, there are indeed deep connections between these two problems [Shalev-Shwartz and Ben-David, 2014]. In particular, as pointed out by Goel et al. [2021], hardness of training implies hardness of proper learning if one permits arbitrary data distributions. However, such hardness results can often be overcome by either posing additional assumptions on the data distributions or switching to more general learning paradigms like improper learning [Goel et al., 2017].

Exact vs. Approximate Training. In practice, it arguably often suf fi ces to train a neural network to approximate instead of exact optimality. The results in this paper are concerned with solving the training problem to exact global optimality. However, since Theorems 1 and 6 and Corollaries 7 and 9 already hold for training error γ = 0 , they even rule out the existence of approximation algorithms with any multiplicative factor. We conceive that for appropriate notions of additive approximation (see, e.g., Goel et al. [2021]), our reductions can also be used to show hardness of additive approximation. However, this would signi fi cantly increase the technical complexity of the analysis and is therefore beyond the scope of this paper. We leave it as an open research question to analyze the in fl uence of meta-parameters like input dimension and number of hidden neurons on additive approximation of the training problem.

Related Work. Dey et al. [2020] and Goel et al. [2021] showed NP-hardness of 2L-RELU-NNTRAIN( L ) for k = 1 , but require non-constant dimension. For target error γ = 0 , the problem is NP-hard for every constant k ≥ 2 and polynomial-time solvable for k = 1 [Goel et al., 2021]. Goel et al. [2021] provide further conditional running time lower bounds and (additive) approximation hardness results. Froese et al. [2022] considered the parameterized complexity regarding the input dimension d and proved W[1]-hardness and an ETH-based running time lower bound of n Ω ( d ) for k = 1 .

Boob et al. [2022] studied networks where the output neuron also is a ReLU and proved NP-hardness (and implicitly W[1]-hardness with respect to d ) for k = 2 and γ = 0 . Bertschinger et al. [2022] showed that training 2-layer ReLU networks with two output and two input neurons ( R 2 → R 2 ) is complete for the class ∃ R (existential theory of the reals) and thus likely not contained in NP. This also implies NP-hardness, but note that in contrast to our results, their reduction does not work for one-dimensional outputs. Their result strengthens a previous result by Abrahamsen et al. [2021] who proved ∃ R -completeness for networks with a speci fi c (not fully connected) architecture. Pilanci and Ergen [2020] showed that training 2-layer neural networks can be formulated as a convex program which yields a polynomial-time algorithm for constant dimension d . However, they considered a regularized objective and their result requires the number k of hidden neurons to be very large (possibly equal to the number n of input points) and hence does not contradict our NP-hardness result for d = 2 .

To study the computational complexity of training ReLU networks, a crucial ingredient is to know the set of (continuous and piecewise linear) functions precisely representable with a certain network architecture. This is well-understood for two-layer networks [Arora et al., 2018, Bertschinger et al., 2022, Dereich and Kassing, 2022], but much trickier for deeper networks [Haase et al., 2023, Hertrich and Sering, 2023, Hertrich et al., 2021]. Similar to the study of ReLU networks by Arora et al. [2018], Khalife and Basu [2022] studied the expressiveness and training complexity for linear threshold activation functions. For an extensive survey on intersections of deep learning and polyhedral theory, we refer to Huchette et al. [2023].

2 Preliminaries

Notation. For n ∈ N , we de fi ne [ n ] := { 1 , . . . , n } . For X ⊆ R d , we denote by a ff ( X ) the af fi ne hull of X and by dim( X ) the dimension of a ff ( X ) .

Parameterized Complexity. We assume basic knowledge on computational complexity theory. Parameterized complexity is a multivariate approach to analyze the computational complexity of problems [Downey and Fellows, 2013, Cygan et al., 2015].

An instance ( x, k ) of a parameterized problem L ⊆ Σ ∗ × N is a pair with x ∈ Σ ∗ being a problem instance and k ∈ N being the value of a certain parameter . A parameterized problem L is fi xed-parameter tractable (fpt) (contained in the class FPT) if there exists an algorithm deciding whether ( x, k ) ∈ L in f ( k ) · | x | O (1) time, where f is a function solely depending on k . Note that a parameterized problem in FPT is polynomial-time solvable for every constant parameter value where, importantly, the degree of the polynomial does not depend on the parameter value. The class XP contains all parameterized problems which can be solved in polynomial time for constant parameter values, that is, in time f ( k ) · | x | g ( k ) . It is known that FPT � XP. The class W[1] contains parameterized problems which are widely believed not to be in FPT. That is, a W[1]-hard problem (e.g. CLIQUE parameterized by the size of the sought clique) is not solvable in f ( k ) · | x | O (1) time. It is known that FPT ⊆ W[1] ⊆ XP.

W[1]-hardness is de fi ned via parameterized reductions . A parameterized reduction from L to L � is an algorithm mapping an instance ( x, k ) in f ( k ) · | x | O (1) time to an instance ( x � , k � ) such that k � ≤ g ( k ) for some function g and ( x, k ) ∈ L if and only if ( x � , k � ) ∈ L � .

Exponential Time Hypothesis. The Exponential Time Hypothesis (ETH) [Impagliazzo and Paturi, 2001] states that 3-SAT cannot be solved in subexponential time in the number n of Boolean variables in the input formula, that is, there exists a constant c > 0 such that 3-SAT cannot be solved in O (2 cn ) time.

�

The ETH implies FPT = W[1] [Cygan et al., 2015] (which implies P = NP). In fact, ETH implies that CLIQUE cannot be solved in ρ ( k ) · n o ( k ) time for any function ρ , where k is the size of the sought clique and n is the number of vertices in the graph [Cygan et al., 2015].

�

Geometry of 2-Layer ReLU Networks. For proving our results, it is crucial to understand the geometry of a function φ : R d → R represented by a two-layer ReLU network. Here, we only discuss properties required to understand our results and refer to Arora et al. [2018], Bertschinger et al. [2022], Dereich and Kassing [2022] for additional discussions in this context. Such a function φ is a continuous and piecewise linear function. Each hidden neuron with index j ∈ [ k ] de fi nes a hyperplane w j · x + b j = 0 in R d . These k hyperplanes form a hyperplane arrangement. Inside each cell of this hyperplane arrangement, the function φ is af fi ne. The graph of φ restricted to such a cell is called a (linear) piece of φ .

Consider a hyperplane H from the hyperplane arrangement. Let w be an orthonormal vector of H and let J ⊆ [ k ] be the non-empty subset of indices of neurons which induce precisely H . Note that w j is a scaled version of w for each j ∈ J . Let x ∈ R d be a point on H which does not lie on any other hyperplane in the arrangement. There are exactly two d -dimensional cells in the arrangement containing x : one on each side of H . The difference of the directional derivatives of the corresponding two pieces of φ in the direction of w is exactly � j ∈ J a j � w j � . In particular, this is independent of x and therefore constant along H . If this value is positive, we say that H is a convex hyperplane of φ . If it is negative, we say that H is a concave hyperplane of φ . Note that this matches with φ being convex or concave locally around every point x ∈ H which does not belong to any other hyperplane in the arrangement. Moreover, a point x ∈ R d is called a convex (concave) breakpoint of φ if it lies exclusively on one convex (concave) hyperplane of φ .

One important observation we will heavily use is the following: If we know that φ originates from a 2-layer neural network with k hidden neurons and we know that we need indeed k distinct hyperplanes to separate the pieces of φ , then each hyperplane must be induced by exactly one neuron (and not several). Then the hyperplane corresponding to the j -th neuron is convex if and only if a j > 0 and concave if and only if a j < 0 . For input dimension d = 2 , each of the hyperplanes in the arrangement

is actually a line in R 2 . We call such a line a breakline and de fi ne convex and concave breaklines accordingly.

3 NP-Hardness for Two Dimensions

In this section we prove our main result that 2L-RELU-NN-TRAIN( L ) is NP-hard for two dimensions, thus excluding any running time of the form ( nk ) f ( d ) .

Theorem 1. 2L-RELU-NN-TRAIN( L ) is NP-hard even for d = 2 and γ = 0 .

We give a polynomial-time reduction from the following NP-complete problem [Garey and Johnson, 1979].

POSITIVE ONE-IN-THREE 3-SAT (POITS)

Input:

A Boolean formula F in conjunctive normal form with three positive literals per clause.

Question:

Is there a truth assignment for the variables such that each clause in F has exactly one true literal?

Our construction will be such that the function represented by the neural network is equal to zero everywhere except for a fi nite set of 'stripes', in which the function forms a levee (see De fi nition 2), that is, when looking at a cross section, the function goes up from 0 to 1, stays constant for a while, and goes down from 1 to 0 again. See Figure 2 (right) for a top view of a levee and Figure 3 for a cross section of a levee.

De fi nition 2. A levee with slope s ∈ R (centered at the origin) is the function f s : R 2 → R with

Observation 3. A levee f s is a continuous, piecewise linear function with two convex and two concave breaklines. It can be realized with four ReLUs as follows:

Similar levees have been used by Bertschinger et al. [2022] to prove ∃ R -completeness of neural network training, however, in a conceptually very different way. In their work, levees encode variable values via the slopes of the function on the non-constant regions of the levee. In contrast, in our reduction, we encode discrete choices via rotation of the levees, that is, via the slopes of the breaklines in the two-dimensional input space .

Selection Gadget. We describe a gadget allowing us to model a discrete choice between � many possibilities (levees). An illustration of the selection gadget is given in Figures 2 and 3.

Each of the � different choices corresponds to one of � different slopes s 1 < s 2 < · · · < s � . The gadget consists of a total of 29 + 2 � many data points with labels in { 0 , 1 / 3 , 2 / 3 , 1 } . The precise position of the data points is described in Appendix A.

It is not too dif fi cult to verify that a levee with slope s i , i ∈ [ � ] , fi ts all data points of a selection gadget. We omit the simple but tedious calculations here. More intricately, the following lemma shows that a selection gadget indeed models a discrete choice between exactly � possibilities. The technically involved proof of the lemma is deferred to Appendix A.

Lemma 4. Let f : R 2 → R be a continuous piecewise linear function with only four breaklines that fi ts all the data points of the selection gadget. Then, f = f s i for some i ∈ [ � ] .

Combining Multiple Selection Gadgets. Having constructed and understood a single selection gadget, the next step is to use multiple of these gadgets simultaneously. To this end, we will 'stack multiple selection gadgets upon each other along the x 2 -axis'. The precise construction and a formal proof of the following lemma are contained in Appendix A.

x

2

x

1

y

x

2

Lemma 5 (Informal version of Lemma 11) . Suppose m selection gadgets are placed along the x 2 -axis with suf fi cient distance to each other. Then the unique way of fi tting all these gadgets with a single continuous piecewise linear function f : R 2 → R with at most 4 m breaklines is selecting exactly one levee for each of the m gadgets and summing up the corresponding functions.

Global Construction. We are now ready to describe the overall layout of the reduction.

For a given formula F = C 1 ∧ C 2 . . . ∧ C m with variables v 1 , . . . , v n , we construct data points in R 2 × R such that they can be fi tted exactly with k = 4( m + n ) ReLUs if and only if F is a yes-instance of POITS. See Figure 4 for an illustration. Our construction consists of m + n selection gadgets, namely one for each clause and one for each variable, and 3 m further data points. Each of the m selection gadgets corresponding to a clause determines which literal of this clause we choose to be true. Each of the n selection gadgets corresponding to a variable determines whether this variable is true or false. The 3 m remaining data points ensure that these choices are consistent. More precisely, if the r -th literal of clause C j is v i , then we introduce a data point p j,r with label y = 1 at the intersection of the levee corresponding to the r -th choice in C j and the levee corresponding to setting v j to false. This makes sure that the r -th literal of C j is selected if and only if v i is set to true.

Sketch of the proof of Theorem 1. Our detailed proof in Appendix A starts by showing that the data points p j,r are indeed positioned such that they only interfere with the two intended levees and no other selection gadgets (Lemma 12). Then, for a given instance of POITS, we construct an instance of 2L-RELU-NN-TRAIN( L ) as described above. Making use of our lemmas and Observation 3, we show that the possible solutions of POITS are in one-to-one correspondence to the possible solutions of 2L-RELU-NN-TRAIN( L ), which decompose into a sum of levees. This fi nishes the reduction.

C

1

C

2

C

3

v

1

v

2

v

3

v

4

v

5

C

1

C

2

C

3

v

1

v

2

v

3

v

4

v

5

4 W[1]-Hardness for Four ReLUs

We show that fi xed-parameter tractability with respect to d is unlikely even for target error zero and four ReLUs. In fact, we prove a running time lower bound of n Ω ( d ) based on the ETH.

Theorem 6. 2L-RELU-NN-TRAIN( L ) with k = 4 and γ = 0 is W[1]-hard with respect to d and not solvable in ρ ( d ) n o ( d ) poly( L ) time (where L is the input bit-length) for any function ρ assuming the ETH.

We prove Theorem 6 with a parameterized reduction from the 2-HYPERPLANE SEPARABILITY problem (see Appendix B).

2-HYPERPLANE SEPARABILITY

Input:

Two point sets Q and P in R d .

Question:

Are there two hyperplanes that strictly separate Q and P ?

Here, two hyperplanes strictly separate Q and P if, for every pair ( q , p ) ∈ Q × P , the open line segment qp is intersected by at least one hyperplane and no point from Q ∪ P is contained in any of the two hyperplanes. Giannopoulos et al. [2009] showed that this problem is W[1]-hard with respect to d and not solvable in ρ ( d ) m o ( d ) poly( L ) time assuming the ETH (where m := | Q ∪ P | and L is the instance size). The idea of our reduction is that a separating hyperplane can be realized by two ReLUs approximating a 'step function'. To this end, we add some auxiliary points to Q ∪ P to ensure the separating behaviour of the ReLUs.

5 Hardness Results for Linear Threshold Activations

A nowadays less popular, but more classical activation function than ReLU is the linear threshold function x �→ 1 { x> 0 } . We de fi ne 2L-LT-NN-TRAIN( L ) as the decision version of the training problem for linear threshold functions. It is de fi ned exactly as 2L-RELU-NN-TRAIN( L ) with two modi fi cations: fi rst, the ReLU function [ x ] + is replaced with 1 { x> 0 } , and second, we do not assume a j ∈ { -1 , 1 } anymore because the normalization used in the ReLU case does not apply here. Instead we allow arbitrary a j ∈ R .

As in the ReLU case, the crucial ingredient to study the training complexity of linear threshold networks is their geometry. To this end, observe that every function represented by a 2-layer linear

threshold network is piecewise constant, where the pieces emerge from the hyperplane arrangement de fi ned by the k hyperplanes corresponding to the hidden neurons. Since our reductions for the ReLU case always use two ReLUs to approximate 'step functions' from 0 to 1 and from 1 to 0 , it is easy to adapt the reductions to the linear threshold case.

Corollary 7. 2L-LT-NN-TRAIN( L ) is NP-hard even for d = 2 and γ = 0 .

Proof. We use a reduction analogous to the one in the proof of Theorem 1. Instead of a sum of levees, we use a sum of 'stripes' within which the function attains value 1 . With this idea, it is straight-forward to build selection gadgets and an analogous global construction. Note that the number k of required linear threshold neurons is only k = 2( m + n ) for a POITS instance with m clauses and n variables because each stripe can be realized with two linear threshold neurons instead of the four ReLUs required to build a levee.

For the sake of completeness, we note that also the W[1]-hardness result by Froese et al. [2022] extends to linear threshold functions. To this end, consider the � p -loss � p (ˆ y, y ) = | ˆ y -y | p , where � 0 simply counts the non-zero components of ˆ y -y .

Corollary 8. For each p ∈ [0 , ∞ [ , 2L-LT-NN-TRAIN( � p ) with k = 1 is NP-hard, W[1]-hard with respect to d and not solvable in ρ ( d ) n o ( d ) poly( L ) time (where L is the input bit-length) for any function ρ assuming the ETH.

Proof. Having a careful look into the reduction from MULTICOLORED CLIQUE by Froese et al. [2022], it turns out that the single ReLU neuron used in this reduction can be replaced by a linear threshold neuron without changing the logic of the reduction.

Finally, also Theorem 6 fi nds its analogue for the linear threshold case.

Corollary 9. 2L-LT-NN-TRAIN( L ) with k = 2 and γ = 0 is W[1]-hard with respect to d and not solvable in ρ ( d ) n o ( d ) poly( L ) time (where L is the input bit-length) for any function ρ assuming the ETH.

Proof. The proof is analogous to (even much easier than) the one of Theorem 6. Instead of two ReLUs to realize a step of height one, we can simply use one linear threshold neuron (which is why we obtain hardness already for k = 2 in this case). Note that we do not even need to introduce additional data points, that is, we obtain a much more direct reduction from 2-HYPERPLANE SEPARABILITY.

6 An Algorithm for Exact Fitting in the Convex Case

Contrasting the previous hardness results for 2L-RELU-NN-TRAIN( L ), we now consider the tractable special case where all coef fi cients a j are 1. In this case (called 2L-RELU-NN-TRAIN( L ) + ), the neural network realizes a convex continuous piecewise linear function φ with at most 2 k distinct (af fi ne) pieces. We show fi xed-parameter tractability for the parameter d + k if γ = 0 (see Appendix C). Note that, for γ > 0 , W[1]-hardness with respect to d already holds for k = 1 Froese et al. [2022].

Theorem 10. 2L-RELU-NN-TRAIN( L ) + can be solved in 2 O ( k 2 d ) poly( k, L ) time for γ = 0 , where L is the input bit-length.

The proof idea is that the convexity of φ allows for a branching algorithm assigning the input data points to the 2 k pieces of φ . We note that Theorem 10 analogously holds for the concave case where a j = -1 for all j ∈ [ k ] . However, if positive and negative coef fi cients a j are allowed, then our search tree approach does not work due to non-convexity. Indeed, Theorem 6 implies that this approach cannot work already for k = 4 . It is unclear whether this issue can be resolved for k = 2 or k = 3 .

7 Conclusion

We closed several gaps in the literature regarding the computational complexity of training two-layer neural networks. Our results give some insight into the geometry of functions realized by such

networks and yield a better understanding of their complexity and expressiveness. We thereby settled the border of computational tractability almost completely. The remaining open questions are the following:

Is the problem with d = 2 in FPT when parameterized by k ? This is open for both ReLUs and linear thresholds. Note that W[1]-hardness with respect to k for any constant d would answer Question 2 negatively.

Is the case γ = 0 and k ∈ { 2 , 3 } in FPT with parameter d for ReLUs?

In a broader context, open directions are to further study the computational complexity in appropriate approximate settings, draw further conclusions on generalization, and understand deeper network architectures.

Acknowledgments and Disclosure of Funding

Christoph Hertrich is supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreements ScaleOpt-757481 and ForEFront-615640).

References

Abrahamsen, L. Kleist, and T. Miltzow. Training neural networks is ∃ R -complete. In Advances in Neural Information Processing Systems 35 (NeurIPS '21) , pages 18293-18306, 2021.

Arora, A. Basu, P. Mianjy, and A. Mukherjee. Understanding deep neural networks with recti fi ed linear units. In Proceedings of the 6th International Conference on Learning Representations (ICLR '18) . OpenReview.net, 2018.

Bertschinger, C. Hertrich, P. Jungeblut, T. Miltzow, and S. Weber. Training fully connected neural networks is ∃ R -complete. CoRR , abs/2204.01368, 2022.

L. Blum and R. L. Rivest. Training a 3-node neural network is NP-complete. Neural Networks , 5 (1):117-127, 1992.

Boob, S. S. Dey, and G. Lan. Complexity of training ReLU neural network. Discrete Optimization , 44(Part 1):100620, 2022.

Cygan, F. V. Fomin, L. Kowalik, D. Lokshtanov, D. Marx, M. Pilipczuk, M. Pilipczuk, and S. Saurabh. Parameterized Algorithms . Springer, 2015.

Dereich and S. Kassing. On minimal representations of shallow ReLU networks. Neural Networks , 148:121-128, 2022.

S. Dey, G. Wang, and Y. Xie. Approximation algorithms for training one-node ReLU neural networks. IEEE Transactions on Signal Processing , 68:6696-6706, 2020.

G. Downey and M. R. Fellows. Fundamentals of Parameterized Complexity . Springer, 2013.

Froese, C. Hertrich, and R. Niedermeier. The computational complexity of ReLU network training parameterized by data dimensionality. Journal of Arti fi cial Intelligence Research , 74:1775-1790, 2022.

R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of NPCompleteness . W. H. Freeman and Company, 1979.

Giannopoulos, C. Knauer, and G. Rote. The parameterized complexity of some geometric problems in unbounded dimension. In Parameterized and Exact Computation, 4th International Workshop (IWPEC '09) , volume 5917 of LNCS , pages 198-209. Springer, 2009.

Glorot, A. Bordes, and Y. Bengio. Deep sparse recti fi er neural networks. In Proceedings of the Fourteenth International Conference on Arti fi cial Intelligence and Statistics (AISTATS '11) , volume 15 of JMLR Proceedings , pages 315-323. JMLR.org, 2011.

Goel, V. Kanade, A. R. Klivans, and J. Thaler. Reliably learning the ReLU in polynomial time. In Proceedings of the 30th Conference on Learning Theory (COLT '17) , volume 65 of PMLR , pages 1004-1042. PMLR, 2017.

Goel, A. R. Klivans, P. Manurangsi, and D. Reichman. Tight hardness results for training depth-2 ReLU networks. In 12th Innovations in Theoretical Computer Science Conference (ITCS '21) , volume 185 of LIPIcs , pages 22:1-22:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021.

Haase, C. Hertrich, and G. Loho. Lower bounds on the depth of integral ReLU neural networks via lattice polytopes. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR '23) , 2023.

Hertrich and L. Sering. ReLU neural networks of polynomial size for exact maximum fl ow computation. In The 24th Conference on Integer Programming and Combinatorial Optimization (IPCO '23) , 2023. Accepted for publication.

Hertrich, A. Basu, M. Di Summa, and M. Skutella. Towards lower bounds on the depth of ReLU neural networks. Advances in Neural Information Processing Systems , 34:3336-3348, 2021.

Huchette, G. Muñoz, T. Serra, and C. Tsay. When deep learning meets polyhedral theory: A survey. CoRR , abs/2305.00241, 2023.

Impagliazzo and R. Paturi. On the complexity of k -SAT. Journal of Computer and System Sciences , 62(2):367-375, 2001.

Khalife and A. Basu. Neural networks with linear threshold activations: structure and algorithms. In The 23rd Conference on Integer Programming and Combinatorial Optimization (IPCO '22) , pages 347-360, 2022.

LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Pilanci and T. Ergen. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In Proceedings of the 37th International Conference on Machine Learning (ICML '20) , volume 119 of Proceedings of Machine Learning Research , pages 7695-7705. PMLR, 2020.

Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Peer reviews:
comments: SUMMARY:
The authors analyze the fpt of training NN. Their main result is that finding a global minimum is hard even for fixed dimension.

STRENGTHS:
I like the direction of studying the FPT of training NN. The reductions are new to the best of my knowledge and could find further applications.

WEAKNESSES:
The authors mention generalization a few times. However, the setting considered is completely detached from generalization.
For fixed dimension the sample complexity of the learning problem is fixed and hence can be solved trivially. A more general concern is that low dimension is a rarity in ML applications. In this regard, the number of neurons k seems to be much more interesting. 

"We assume basic knowledge on computational complexity theory." I think this sentence can safely be removed. 

There is a large literature on what can and cannot be done efficiently with respect to learning thresholds that the authors ignore.
One example is:
"Adam Klivans and Pravesh Kothari. Embedding hard learning problems into gaussian space" and various other papers (regarding learning ReLUs) by Kane and Diakonikolas. While the hardness in these is not NP-hardness, they should be mentioned nevertheless.

QUESTIONS:
What is the best running-time dependency in term of k when can expect for fixed dimension? What is the running time of the fastest exponential algorithm for learning shallow ReLUs? Thresholds? Are their improvements for the running time of the algorithm of the Arora et al., paper?

LIMITATIONS:
Proving hardness results for low dimension.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that learning a 2-layered neural network with ReLU activation function, in an underparameterized regime, is NP-hard when the training error is 0. Specifically, they highlight that such learning cannot be accomplished in time complexity dependent on $k^{f(d)}$, where $k$ is the number of neurons in the layer and $d$ is the input dimension. Their main technique is a reduction from the NP-complete problem of POSITIVE ONE-IN-THREE SAT. In their reduction, they strategically position selection gadgets (corresponding to clauses, variables in POITS, and additional data points) along the y-axis, ensuring sufficient spacing between them. This construction can be represented by a piecewise linear function or a combination of ReLUs if and only if it implies a true instance of POITS.\n\nThey also show W[1]-hardness with respect to $d$ for $k = 4$ by reducing the problem to 2 HYPERPLANE-SEPARABILITY. This problem determines whether two given point sets can be strictly separated by two hyperplanes. Additionally, they prove that linear thresholding for 2-layered neural networks is also NP-hard, with a similar reduction as proposed for ReLU activations. Finally, they propose a branching algorithm for learning a 2-layered neural network, which has exponential time complexity with respect to $k$ and $d$.\n\nSTRENGTHS:\nThe authors have effectively motivated the problem and provided a good background for the problem. The part where they describe the geometry of $\\phi$ and define the concept of a levee, accompanied by illustrative figures, was particularly informative and interesting.\n\nWEAKNESSES:\nThe variable $\\ell$ seems to be overloaded, as it is used for both the loss function and the number of possible levees. This can be confusing when it is introduced again in the selection gadget.\n\nQUESTIONS:\n* When you mention \"full dimensional cells,\" are you referring to cells of dimension $d$?\n* In order to prevent the other hyperplanes from becoming convex or concave, does the breakpoint $\\mathbf{x}$ need to lie exclusively on one hyperplane?\n\nLIMITATIONS:\nThe authors discuss various approaches to extend their construction to obtain similar results in higher input dimensions beyond 2, for a fixed training error $\\ge 0$ and for more than 4 ReLUs for the W[1] hardness result. Additionally, they address the possibilities of extending to piecewise linear ReLU and piecewise constant linear thresholding activations and also acknowledge that their techniques might not be applicable to smooth activations. \n\nFinally they address two interesting open questions: 1. Results for the task of minimising generalization error instead of training error and 2. Training neural networks for approximate optimality.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly studies the complexity of training two-layer ReLU networks when they are not over-parameterized. Specifically, the authors prove that training a two-layer ReLU network to zero or arbitrary loss value is NP-hard when the input dimension $d\\geq 2$. This result answers an open question (Question 1) posed by Arora et al. (2018) negatively by proving the NP-hardness. On the other hand, by assuming the exponential time hypothesis, the paper also partially answers a more general question of whether a running time that is polynomial in the data size or the number of hidden nodes (Question 2). The paper also provides a positive answer to Question 2 when the ReLU network is assumed to compute a convex function.\n\nSTRENGTHS:\nThe novelty of the paper is clear. It provides answers to open questions on the complexity of training two-layer ReLU networks. Therefore, the results seem to be significant. The presentation is concise and easy to follow. I enjoy reading the paper.\n\nWEAKNESSES:\nUsually, there are symmetries and redundancies in the dataset. It would be more convincing and realistic if the authors could give some insights on how to leverage these available properties to develop efficient training algorithms. In practice, training a two-layer ReLU network is very efficient. Hence, I feel the settings in this paper seem to be a bit unrealistic. My concern is that it seems hard to apply these results to make a positive impact on our machine learning community. One possible way to address this is to discuss assumptions on different aspects (e.g., architectures) to make training tractable or even efficient. This work focuses on two-layer ReLU networks, but it would be interesting to give insights into extending the results to networks with multiple layers.\n\nQUESTIONS:\n1. Line 35: In the case when $\\gamma$ is strictly positive, are there any extra assumptions imposed on the loss function? The only assumption on the loss function $\\ell(x,y)=0$ iff $x=y$ is fairly general. It would be more convincing if the authors could clarify more about the loss function.\n\n2. Would it yield the same conclusion as Theorem 1 if a regularizer (e.g., weight decay or L1) is added to the loss function ($k<n$)?\n\n3. Would it be possible to consider some assumptions on architectures that give a polynomial-time algorithm for training a two-layer ReLU network?\n\nLIMITATIONS:\nThe authors have addressed some of the limitations of this work in the text. Other limitations in my view are the gaps between the settings of the problem and the training of ReLU networks in practice. For example, one may be able to make some assumptions on architectures, regularizations, and properties of data to derive a polynomial-time training algorithm. However, given that the main purpose of this paper is to answer the open questions, discussions on these aspects may be skipped or minimized.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on two layer neural networks with ReLU or linear threshold activations. The authors show that considering the input dimension (dimension of the data as a constant) equal with 2, there is no polynomial algorithm with respect to the number of data (n) and hidden nodes (k, k<n) that decides whether there exist weights that achieve zero training error. The second result indicates that there is no algorithm with dependence $n^{o(d)}$ on $n$ and arbitrary dependence on $k,d$. Furthermore, if $k=4$ the authors prove $W[1]-$hardness with respect to $d$. Similar results are obtained for the case of threshold activations. Finally, they provide an algorithm, independent of $n$ if the target function is convex.\n\nSTRENGTHS:\nThis paper is in general well-written. It closes some gaps regarding hardness results for deciding whether there exist weights on a neural network that achieve a small or zero training loss. The authors show that even when the target is zero error, the problem is NP-hard for $d=2$ and thus for any $d\\geq 2$ (since we can simply zero pad and thus ignore the rest of the $d-2$ points (Theorem 1). They also prove that for $k=4$ and zero training error the problem is $W[1]$-hard with respect to the parameter $d$. Their results also hold for the case of linear threshold activations. They finally show that for the case that the target functions is convex, meaning the second layer has weights all equal to one, they show that the problem is fixed-parameter tractable (FPT), when the targeted error is exactly zero. It was already known that for the case of non-zero error, positive error the decision problem is $W[1]$-hard.\n\nWEAKNESSES:\nConsidering the last result (theorem 10) I am not sure what it adds to our understanding, it will be rare to encounter a convex problem a providing an algorithm to decide whether it is solvable it's not very useful, since we know that a convex function will have a global minimum and that it can be found through classical methods like gradient descent. The authors have already acknowledged that this algorithm will not have any practical relevance, but I also think its contribution as a theoretical result marginal.\n\nQUESTIONS:\nConsidering Theorem 1, is there some more tight characterization when $d> 2$ and the inputs are not sparse?\n\nLIMITATIONS:\nThe authors have addressed the limitations and this work has no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}