SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vincent Froese
Algorithms and Computational Complexity
Faculty IV, TU Berlin
vincent.froese@tu-berlin.de
We study the parameterized complexity of training two-layer neural networks with respect to the dimension of the input data and the number of hidden neurons, considering ReLU and linear threshold activation functions. Albeit the computational complexity of these problems has been studied numerous times in recent years, several questions are still open. We answer questions by Arora et al. [2018, ICLR] and Khalife and Basu [2022, IPCO] showing that both problems are NP-hard for two dimensions, which excludes any polynomial-time algorithm for constant dimension. We also answer a question by Froese et al. [2022, JAIR] proving W[1]-hardness for four ReLUs (or two linear threshold neurons) with zero training error. Finally, in the ReLU case, we show fixed-parameter tractability for the combined parameter number of dimensions and number of ReLUs if the network is assumed to compute a convex map. Our results settle the complexity status regarding these parameters almost completely.
Neural networks with rectified linear unit (ReLU) activations are arguably one of the most fundamental models in modern machine learning [Arora et al., 2018, Glorot et al., 2011, LeCun et al., 2015]. To use them as predictors on unseen data, one usually first fixes an architecture (the graph of the neural network) and then optimizes the weights and biases such that the network performs well on some known training data, with the hope that it will then also generalize well to unseen test data. While the ultimate goal in applications is generalization, empirical risk minimization (that is, optimizing the training error) is an important step in this pipeline and understanding its computational complexity is crucial to advance the theoretical foundations of deep learning.
In this paper, we aim to understand how the choice of different meta-parameters, like the input dimension and the width of the neural network, influences the computational complexity of the training problem. To this end, we focus on two-layer neural networks, which can be seen as the standard building block also for deeper architectures.
Figure 1: Neural network architecture we study in this paper: After the input layer (left) with  $d$  input neurons, we have one hidden layer with  $k$  ReLU neurons and a single output neuron without additional activation function.
Formally, a two-layer neural network (see Figure 1) with  $d$  input neurons,  $k$  hidden ReLU neurons, and a single output neuron computes a map
$$
\phi \colon \mathbb {R} ^ {d} \to \mathbb {R}, \quad \phi (\mathbf {x}) = \sum_ {j = 1} ^ {k} a _ {j} [ \mathbf {w} _ {j} \cdot \mathbf {x} + b _ {j} ] _ {+},
$$
where  $\mathbf{w}_j\in \mathbb{R}^d$  and  $a_{j}\in \{-1,1\}$  are the weights between the layers,  $b_{j}$  are the biases at the hidden neurons, and  $[x]_{+}\coloneqq \max (0,x)$  is the rectifier function. Notice that restricting  $a_{j}$  to  $\{-1,1\}$  is without loss of generality because we can normalize by pulling any nonnegative factor into  $\mathbf{w}_j$  and  $b_{j}$ . We also study neural networks with linear threshold activation in Section 5.
Given training data  $\mathbf{x}_1, \ldots, \mathbf{x}_n \in \mathbb{R}^d$  with labels  $y_1, \ldots, y_n \in \mathbb{R}$ , the task of training such a network is to find  $\mathbf{w}_j, b_j$ , and  $a_j$  for each  $j \in [k]$  such that the training error  $\sum_{i=1}^{n} \mathcal{L}(\phi(\mathbf{x}_i), y_i)$  for a given loss function  $\mathcal{L} \colon \mathbb{R} \times \mathbb{R} \to \mathbb{R}_{\geq 0}$  is minimized. Throughout this work, we assume  $\mathcal{L} \colon \mathbb{R} \times \mathbb{R} \to \mathbb{R}_{\geq 0}$  to be any loss function with  $\bar{\mathcal{L}}(x, y) = 0 \iff x = y$ . Formally, the decision version of two-layer ReLU neural network training is defined as follows:
2L-RELU-NN-TRAIN(L)
Input: Data points  $(\mathbf{x}_1,y_1),\ldots ,(\mathbf{x}_n,y_n)\in \mathbb{R}^d\times \mathbb{R}$  , a number  $k\in \mathbb{N}$  of ReLUs, and a target error  $\gamma \in \mathbb{R}_{>0}$
Question: Are there weights  $\mathbf{w}_1, \ldots, \mathbf{w}_k \in \mathbb{R}^d$ , biases  $b_1, \ldots, b_k \in \mathbb{R}$ , and coefficients  $a_1, \ldots, a_k \in \{-1, 1\}$  such that
$$
\sum_ {i = 1} ^ {n} \mathcal {L} \left(\sum_ {j = 1} ^ {k} a _ {j} \left[ \mathbf {w} _ {j} \cdot \mathbf {x} _ {i} + b _ {j} \right] _ {+}, y _ {i}\right) \leq \gamma ?
$$
Note that in the over-parameterized case where  $k \geq n$ , the network can exactly fit any  $n$  input points² (achieving training error  $\gamma = 0$ ) [Zhang et al., 2021, Theorem 1]. Thus, we henceforth assume that  $k < n$ .
2L-RELU-NN-TRAIN  $(\mathcal{L})$  is known to be NP-hard [Dey et al., 2020, Goel et al., 2021], but all known reductions require the input dimension to be part of the input. The current state-of-the-art exact algorithm for convex loss  $\mathcal{L}$  is by Arora et al. [2018] and runs in  $O(2^k n^{dk} \mathrm{poly}(L))$  time, where  $L$  is the input bit-length.
As regards the computational complexity of 2L-RELU-NN-TRAIN  $(\mathcal{L})$ , Arora et al. [2018] posed the question(s) whether a running time
"that is polynomial in the data size and/or the number of hidden nodes, assuming that the input dimension is a fixed constant"
is possible. That is, they asked two questions. The first corresponds to the "and" statement, which can be phrased as follows:
Assuming that  $y_{i} = y_{j}$  whenever  $\mathbf{x}_i = \mathbf{x}_j$
Question 1: Is there an algorithm running in  $(nk)^{f(d)}\mathrm{poly}(L)$  time for some function  $f$ ?
In other words, the question is whether 2L-RELU-NN-TRAIN  $(\mathcal{L})$  is in the complexity class XP when parameterized by  $d$ . The second question corresponding to the "or" statement can then be interpreted as
Question 2: Is there an algorithm running in  $n^{f(d)}g(k,d)$  poly(  $L$  ) or  $k^{f(d)}g(n,d)$  poly(  $L$  ) time for some functions  $f$  and  $g$ ?
We observe that the second running time is clearly possible since  $k < n$  holds by assumption, and hence the algorithm by Arora et al. [2018] runs in  $g(n,d)$  poly(  $L$  ) time. Hence, it remains open whether  $n^{f(d)}g(k,d)$  poly(  $L$  ) time is possible, which is equivalent to (uniform) fixed-parameter tractability with respect to  $k$  for every constant  $d$ .
Clearly, Question 1 is the stronger statement, that is, a positive answer implies a positive answer to Question 2. Arora et al. [2018] conclude with
"Resolving this dependence on network size would be another step towards clarifying the theoretical complexity of training ReLU DNNs and is a good open question for future research, in our opinion."
Note that Froese et al. [2022] proved that, for  $k = 1$ , there is no algorithm running in  $g(d)n^{o(d)}$  time unless the Exponential Time Hypothesis fails. Hence, this result already partially answered the two questions above by excluding any algorithm running in  $n^{o(d)}g(d,k)$  poly  $(L)$  time.
Our Contribution. In this paper, we answer Question 1 negatively by showing NP-hardness for  $d = 2$  in Theorem 1, indicating that we cannot get rid of the exponential dependence on the network size in the algorithm by Arora et al. [2018] even if the dimension is fixed. As regards Question 2, we further exclude (assuming the Exponential Time Hypothesis) any algorithm running in time  $n^{o(d)}g(d,k)\mathrm{poly}(L)$  even for the case  $\gamma = 0$  and prove W[1]-hardness with respect to  $d$  for  $k = 4$  (Theorem 6), which answers an open question by Froese et al. [2022].
We also obtain analogous hardness results if linear threshold activation functions are used instead of ReLUs. As in the ReLU case, it is well-known that training linear threshold networks is NP-hard [Blum and Rivest, 1992, Khalife and Basu, 2022]. The running time of the state-of-the-art algorithm due to Khalife and Basu [2022] is polynomial in  $n$  for fixed  $d$  and  $k$ , but exponential in the latter two parameters. Khalife and Basu [2022] posed an analogous question to Question 1 for linear threshold networks, which we answer negatively in Corollary 7, excluding a polynomial running time even for fixed dimension. We also show that we cannot expect fixed-parameter tractability with respect to  $d$  even for  $k = 1$  (Corollary 8) and also not for  $k = 2$  and  $\gamma = 0$  (Corollary 9).
On the positive side, we give an algorithm running in  $2^{O(k^2 d)}\mathrm{poly}(k,L)$  time for ReLU neural networks if  $\gamma = 0$  and the function computed by the network is assumed to be convex (Theorem 10). Note that this running time yields fixed-parameter tractability with respect to  $k$  for every constant  $d$ , and thus answers Question 2 positively for this restricted special case. We would like to emphasize that we do not expect this algorithm to have any practical relevance. We rather see its value in being a puzzle piece for understanding the computational complexity of the training problem.
Implications and Limitations. In the following we provide a brief discussion of the implications and limitations of our results from various perspectives.
Input Dimension. Theorem 1 implies that 2L-RELU-NN-TRAIN  $(\mathcal{L})$  is in fact NP-hard for every fixed  $d \geq 2$ . The straight-forward reduction simply pads all the input vectors with  $d - 2$  zeros. Similarly, Corollary 7 holds for every fixed  $d \geq 2$ .
Target Error. The hardness results Theorems 1 and 6 and Corollaries 7 and 9 also hold for every fixed  $\gamma \geq 0$ . The reduction is straightforward by introducing a set of incompatible data points which force the network to incur an additional error of  $\gamma$ . For our positive result Theorem 10, however, there is indeed a difference in the complexity between the two cases  $\gamma = 0$  and  $\gamma > 0$ . While we show fixed-parameter tractability for  $\gamma = 0$ , the same problem is W[1]-hard for  $\gamma > 0$ , already in the case  $k = 1$  [Froese et al., 2022].
Number of ReLUs. It is not too difficult to see (although it requires some work) that our particular reduction in Theorem 6 can be extended to any  $k \geq 4$  by introducing more data points far away from the existing data points which enforce the usage of additional ReLUs which then cannot be used to fit the data points of the actual reduction. Therefore, Theorem 6 holds for every fixed  $k \geq 4$ . Similarly, Corollary 8 holds for every fixed  $k \geq 1$  and Corollary 9 holds for every fixed  $k \geq 2$ .
Other Activation Functions. Our hardness results hold for the piecewise linear ReLU activation function and the piecewise constant linear threshold activation function. Extending them to other piecewise linear or constant activation functions like leaky ReLU or maxout should be straightforward. However, achieving analogous results for smooth activation functions like sigmoids probably requires fundamentally different techniques and is beyond the scope of this paper.
Training vs. Learning. Our results are concerned with the problem of minimizing the training error. While this is inherently different from minimizing the generalization error, there are indeed deep connections between these two problems [Shalev-Shwartz and Ben-David, 2014]. In particular, as pointed out by Goel et al. [2021], hardness of training implies hardness of proper learning if one permits arbitrary data distributions. However, such hardness results can often be overcome by either posing additional assumptions on the data distributions or switching to more general learning paradigms like improper learning [Goel et al., 2017].
Exact vs. Approximate Training. In practice, it arguably often suffices to train a neural network to approximate instead of exact optimality. The results in this paper are concerned with solving the training problem to exact global optimality. However, since Theorems 1 and 6 and Corollaries 7 and 9 already hold for training error  $\gamma = 0$ , they even rule out the existence of approximation algorithms with any multiplicative factor. We conceive that for appropriate notions of additive approximation (see, e.g., Goel et al. [2021]), our reductions can also be used to show hardness of additive approximation. However, this would significantly increase the technical complexity of the analysis and is therefore beyond the scope of this paper. We leave it as an open research question to analyze the influence of meta-parameters like input dimension and number of hidden neurons on additive approximation of the training problem.
Related Work. Dey et al. [2020] and Goel et al. [2021] showed NP-hardness of 2L-RELU-NN-TRAIN  $(\mathcal{L})$  for  $k = 1$ , but require non-constant dimension. For target error  $\gamma = 0$ , the problem is NP-hard for every constant  $k \geq 2$  and polynomial-time solvable for  $k = 1$  [Goel et al., 2021]. Goel et al. [2021] provide further conditional running time lower bounds and (additive) approximation hardness results. Froese et al. [2022] considered the parameterized complexity regarding the input dimension  $d$  and proved W[1]-hardness and an ETH-based running time lower bound of  $n^{\Omega(d)}$  for  $k = 1$ .
Boob et al. [2022] studied networks where the output neuron also is a ReLU and proved NP-hardness (and implicitly W[1]-hardness with respect to  $d$ ) for  $k = 2$  and  $\gamma = 0$ . Bertschinger et al. [2022] showed that training 2-layer ReLU networks with two output and two input neurons  $(\mathbb{R}^2 \to \mathbb{R}^2)$  is complete for the class  $\exists \mathbb{R}$  (existential theory of the reals) and thus likely not contained in NP. This also implies NP-hardness, but note that in contrast to our results, their reduction does not work for one-dimensional outputs. Their result strengthens a previous result by Abrahamsen et al. [2021] who proved  $\exists \mathbb{R}$ -completeness for networks with a specific (not fully connected) architecture. Pilanci and Ergen [2020] showed that training 2-layer neural networks can be formulated as a convex program which yields a polynomial-time algorithm for constant dimension  $d$ . However, they considered a regularized objective and their result requires the number  $k$  of hidden neurons to be very large (possibly equal to the number  $n$  of input points) and hence does not contradict our NP-hardness result for  $d = 2$ .
To study the computational complexity of training ReLU networks, a crucial ingredient is to know the set of (continuous and piecewise linear) functions precisely representable with a certain network architecture. This is well-understood for two-layer networks [Arora et al., 2018, Bertschinger et al., 2022, Dereich and Kassing, 2022], but much trickier for deeper networks [Haase et al., 2023, Hertrich and Sering, 2023, Hertrich et al., 2021]. Similar to the study of ReLU networks by Arora et al. [2018], Khalife and Basu [2022] studied the expressiveness and training complexity for linear threshold activation functions. For an extensive survey on intersections of deep learning and polyhedral theory, we refer to Huchette et al. [2023].
Notation. For  $n \in \mathbb{N}$ , we define  $[n] := \{1, \dots, n\}$ . For  $X \subseteq \mathbb{R}^d$ , we denote by  $\operatorname{aff}(X)$  the affine hull of  $X$  and by  $\dim(X)$  the dimension of  $\operatorname{aff}(X)$ .
Parameterized Complexity. We assume basic knowledge on computational complexity theory. Parameterized complexity is a multivariate approach to analyze the computational complexity of problems [Downey and Fellows, 2013, Cygan et al., 2015].
An instance  $(x,k)$  of a parameterized problem  $L\subseteq \Sigma^{*}\times \mathbb{N}$  is a pair with  $x\in \Sigma^{*}$  being a problem instance and  $k\in \mathbb{N}$  being the value of a certain parameter. A parameterized problem  $L$  is fixed-parameter tractable (fpt) (contained in the class FPT) if there exists an algorithm deciding whether  $(x,k)\in L$  in  $f(k)\cdot |x|^{O(1)}$  time, where  $f$  is a function solely depending on  $k$ . Note that a parameterized problem in FPT is polynomial-time solvable for every constant parameter value where, importantly, the degree of the polynomial does not depend on the parameter value. The class XP contains all parameterized problems which can be solved in polynomial time for constant parameter values, that is, in time  $f(k)\cdot |x|^{g(k)}$ . It is known that FPT  $\subsetneq$  XP. The class W[1] contains parameterized problems which are widely believed not to be in FPT. That is, a W[1]-hard problem (e.g. CLIQUE parameterized by the size of the sought clique) is not solvable in  $f(k)\cdot |x|^{O(1)}$  time. It is known that FPT  $\subseteq$  W[1]  $\subseteq$  XP.
W[1]-hardness is defined via parameterized reductions. A parameterized reduction from  $L$  to  $L'$  is an algorithm mapping an instance  $(x, k)$  in  $f(k) \cdot |x|^{O(1)}$  time to an instance  $(x', k')$  such that  $k' \leq g(k)$  for some function  $g$  and  $(x, k) \in L$  if and only if  $(x', k') \in L'$ .
Exponential Time Hypothesis. The Exponential Time Hypothesis (ETH) [Impagliazzo and Paturi, 2001] states that 3-SAT cannot be solved in subexponential time in the number  $n$  of Boolean variables in the input formula, that is, there exists a constant  $c > 0$  such that 3-SAT cannot be solved in  $O(2^{cn})$  time.
The ETH implies FPT  $\neq$  W[1] [Cygan et al., 2015] (which implies  $\mathrm{P} \neq \mathrm{NP}$ ). In fact, ETH implies that CLIQUE cannot be solved in  $\rho(k) \cdot n^{o(k)}$  time for any function  $\rho$ , where  $k$  is the size of the sought clique and  $n$  is the number of vertices in the graph [Cygan et al., 2015].
Geometry of 2-Layer ReLU Networks. For proving our results, it is crucial to understand the geometry of a function  $\phi \colon \mathbb{R}^d \to \mathbb{R}$  represented by a two-layer ReLU network. Here, we only discuss properties required to understand our results and refer to Arora et al. [2018], Bertschinger et al. [2022], Dereich and Kassing [2022] for additional discussions in this context. Such a function  $\phi$  is a continuous and piecewise linear function. Each hidden neuron with index  $j \in [k]$  defines a hyperplane  $\mathbf{w}_j \cdot \mathbf{x} + b_j = 0$  in  $\mathbb{R}^d$ . These  $k$  hyperplanes form a hyperplane arrangement. Inside each cell of this hyperplane arrangement, the function  $\phi$  is affine. The graph of  $\phi$  restricted to such a cell is called a (linear) piece of  $\phi$ .
Consider a hyperplane  $H$  from the hyperplane arrangement. Let  $\mathbf{w}$  be an orthonormal vector of  $H$  and let  $J \subseteq [k]$  be the non-empty subset of indices of neurons which induce precisely  $H$ . Note that  $\mathbf{w}_j$  is a scaled version of  $\mathbf{w}$  for each  $j \in J$ . Let  $\mathbf{x} \in \mathbb{R}^d$  be a point on  $H$  which does not lie on any other hyperplane in the arrangement. There are exactly two  $d$ -dimensional cells in the arrangement containing  $\mathbf{x}$ : one on each side of  $H$ . The difference of the directional derivatives of the corresponding two pieces of  $\phi$  in the direction of  $\mathbf{w}$  is exactly  $\sum_{j \in J} a_j \| \mathbf{w}_j \|$ . In particular, this is independent of  $\mathbf{x}$  and therefore constant along  $H$ . If this value is positive, we say that  $H$  is a convex hyperplane of  $\phi$ . If it is negative, we say that  $H$  is a concave hyperplane of  $\phi$ . Note that this matches with  $\phi$  being convex or concave locally around every point  $\mathbf{x} \in H$  which does not belong to any other hyperplane in the arrangement. Moreover, a point  $\mathbf{x} \in \mathbb{R}^d$  is called a convex (concave) breakpoint of  $\phi$  if it lies exclusively on one convex (concave) hyperplane of  $\phi$ .
One important observation we will heavily use is the following: If we know that  $\phi$  originates from a 2-layer neural network with  $k$  hidden neurons and we know that we need indeed  $k$  distinct hyperplanes to separate the pieces of  $\phi$ , then each hyperplane must be induced by exactly one neuron (and not several). Then the hyperplane corresponding to the  $j$ -th neuron is convex if and only if  $a_{j} > 0$  and concave if and only if  $a_{j} < 0$ . For input dimension  $d = 2$ , each of the hyperplanes in the arrangement
is actually a line in  $\mathbb{R}^2$ . We call such a line a breakline and define convex and concave breaklines accordingly.
3 NP-Hardness for Two Dimensions
In this section we prove our main result that 2L-RELU-NN-TRAIN  $(\mathcal{L})$  is NP-hard for two dimensions, thus excluding any running time of the form  $(nk)^{f(d)}$ .
Theorem 1. 2L-RELU-NN-TRAIN  $(\mathcal{L})$  is NP-hard even for  $d = 2$  and  $\gamma = 0$ .
We give a polynomial-time reduction from the following NP-complete problem [Garey and Johnson, 1979].
POSITIVE ONE-IN-THREE 3-SAT (POITS)
Input: A Boolean formula  $F$  in conjunctive normal form with three positive literals per clause.
Question: Is there a truth assignment for the variables such that each clause in  $F$  has exactly one true literal?
Our construction will be such that the function represented by the neural network is equal to zero everywhere except for a finite set of "stripes", in which the function forms a levee (see Definition 2), that is, when looking at a cross section, the function goes up from 0 to 1, stays constant for a while, and goes down from 1 to 0 again. See Figure 2 (right) for a top view of a levee and Figure 3 for a cross section of a levee.
Definition 2. A levee with slope  $s \in \mathbb{R}$  (centered at the origin) is the function  $f_s: \mathbb{R}^2 \to \mathbb{R}$  with
$$
f _ {s} \left(x _ {1}, x _ {2}\right) = \left\{ \begin{array}{l l} 0, & i f | x _ {2} - s x _ {1} | \geq 2, \\ 1, & i f | x _ {2} - s x _ {1} | \leq 1, \\ 2 + x _ {2} - s x _ {1}, & i f x _ {2} - s x _ {1} \in ] - 2, - 1 [, \\ 2 - x _ {2} + s x _ {1}, & i f x _ {2} - s x _ {1} \in ] 1, 2 [. \end{array} \right. \tag {1}
$$
Observation 3. A levee  $f_{s}$  is a continuous, piecewise linear function with two convex and two concave breaklines. It can be realized with four ReLUs as follows:
$$
f _ {s} \left(x _ {1}, x _ {2}\right) = \left[ x _ {2} - s x _ {1} + 2 \right] _ {+} - \left[ x _ {2} - s x _ {1} + 1 \right] _ {+} - \left[ x _ {2} - s x _ {1} - 1 \right] _ {+} + \left[ x _ {2} - s x _ {1} - 2 \right] _ {+}.
$$
Similar levees have been used by Bertschinger et al. [2022] to prove  $\exists \mathbb{R}$ -completeness of neural network training, however, in a conceptually very different way. In their work, levees encode variable values via the slopes of the function on the non-constant regions of the levee. In contrast, in our reduction, we encode discrete choices via rotation of the levees, that is, via the slopes of the breaklines in the two-dimensional input space.
Selection Gadget. We describe a gadget allowing us to model a discrete choice between  $\ell$  many possibilities (levees). An illustration of the selection gadget is given in Figures 2 and 3.
Each of the  $\ell$  different choices corresponds to one of  $\ell$  different slopes  $s_1 < s_2 < \dots < s_\ell$ . The gadget consists of a total of  $29 + 2\ell$  many data points with labels in  $\{0,1/3,2/3,1\}$ . The precise position of the data points is described in Appendix A.
It is not too difficult to verify that a levee with slope  $s_i$ ,  $i \in [\ell]$ , fits all data points of a selection gadget. We omit the simple but tedious calculations here. More intricately, the following lemma shows that a selection gadget indeed models a discrete choice between exactly  $\ell$  possibilities. The technically involved proof of the lemma is deferred to Appendix A.
Lemma 4. Let  $f\colon \mathbb{R}^2\to \mathbb{R}$  be a continuous piecewise linear function with only four breaklines that fits all the data points of the selection gadget. Then,  $f = f_{s_i}$  for some  $i\in [\ell ]$ .