SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Combining Multiple Selection Gadgets. Having constructed and understood a single selection gadget, the next step is to use multiple of these gadgets simultaneously. To this end, we will "stack multiple selection gadgets upon each other along the  $x_{2}$ -axis". The precise construction and a formal proof of the following lemma are contained in Appendix A.
Figure 2: Illustration of the selection gadget with  $\ell = 3$  and  $s_1 = -1$ ,  $s_2 = 0$ ,  $s_3 = 1$ . Both figures show the  $x_{1} - x_{2}$ -plane while the  $y$ -coordinate is indicated via the darkness of the gray color. The left picture shows all data points belonging to the gadget as well as the breaklines of the three possible levees fitting the data points. In addition to these features, the right picture shows a levee with slope  $s_2 = 0$  as one of three possibilities to fit the data points of the gadget.
Figure 3: Cross section of the selection gadget along the  $x_{2}$ -axis. The data points of the gadget force the function  $f$  to attain the shape of a levee.
Lemma 5 (Informal version of Lemma 11). Suppose  $m$  selection gadgets are placed along the  $x_{2}$ -axis with sufficient distance to each other. Then the unique way of fitting all these gadgets with a single continuous piecewise linear function  $f: \mathbb{R}^2 \to \mathbb{R}$  with at most  $4m$  breaklines is selecting exactly one levee for each of the  $m$  gadgets and summing up the corresponding functions.
Global Construction. We are now ready to describe the overall layout of the reduction.
For a given formula  $F = C_{1} \wedge C_{2} \ldots \wedge C_{m}$  with variables  $v_{1}, \ldots, v_{n}$ , we construct data points in  $\mathbb{R}^2 \times \mathbb{R}$  such that they can be fitted exactly with  $k = 4(m + n)$  ReLUs if and only if  $F$  is a yes-instance of POITS. See Figure 4 for an illustration. Our construction consists of  $m + n$  selection gadgets, namely one for each clause and one for each variable, and  $3m$  further data points. Each of the  $m$  selection gadgets corresponding to a clause determines which literal of this clause we choose to be true. Each of the  $n$  selection gadgets corresponding to a variable determines whether this variable is true or false. The  $3m$  remaining data points ensure that these choices are consistent. More precisely, if the  $r$ -th literal of clause  $C_{j}$  is  $v_{i}$ , then we introduce a data point  $\mathbf{p}_{j,r}$  with label  $y = 1$  at the intersection of the levee corresponding to the  $r$ -th choice in  $C_{j}$  and the levee corresponding to setting  $v_{j}$  to false. This makes sure that the  $r$ -th literal of  $C_{j}$  is selected if and only if  $v_{i}$  is set to true.
Sketch of the proof of Theorem 1. Our detailed proof in Appendix A starts by showing that the data points  $\mathbf{p}_{j,r}$  are indeed positioned such that they only interfere with the two intended levees and no other selection gadgets (Lemma 12). Then, for a given instance of POITS, we construct an instance of 2L-RELU-NN-TRAIN  $(\mathcal{L})$  as described above. Making use of our lemmas and Observation 3, we show that the possible solutions of POITS are in one-to-one correspondence to the possible solutions of 2L-RELU-NN-TRAIN  $(\mathcal{L})$ , which decompose into a sum of levees. This finishes the reduction.
Figure 4: Global construction layout for the reduction from POITS to 2L-RELU-NN-TRAIN(L). The figure shows the construction for the instance  $(v_{5} \vee v_{4} \vee v_{3}) \wedge (v_{4} \vee v_{3} \vee v_{2}) \wedge (v_{5} \vee v_{2} \vee v_{1})$ . The vertical dotted line is the  $x_{2}$ -axis along which we place all the selection gadgets. Each gadget is depicted with a black square. Each solid gray line depicts one possible levee. Each gray circle depicts a data point  $\mathbf{p}_{j,r}$  with label one. The picture on the right additionally shows one possible solution to the given instance. Indeed, choosing levees corresponding to the solid black lines selects exactly one levee per selection gadget and exactly one levee passing through each of the nine additional data points. This corresponds to the truth assignment  $v_{1} = v_{3} =$  true and  $v_{2} = v_{4} = v_{5} =$  false.
4 W[1]-Hardness for Four ReLUs
We show that fixed-parameter tractability with respect to  $d$  is unlikely even for target error zero and four ReLUs. In fact, we prove a running time lower bound of  $n^{\Omega(d)}$  based on the ETH.
Theorem 6. 2L-RELU-NN-TRAIN(L) with  $k = 4$  and  $\gamma = 0$  is W[1]-hard with respect to  $d$  and not solvable in  $\rho (d)n^{o(d)}\mathrm{poly}(L)$  time (where  $L$  is the input bit-length) for any function  $\rho$  assuming the ETH.
We prove Theorem 6 with a parameterized reduction from the 2-HYPERPLANE SEPARABILITY problem (see Appendix B).
2-HYPERPLANE SEPARABILITY
Input: Two point sets  $Q$  and  $P$  in  $\mathbb{R}^d$ .
Question: Are there two hyperplanes that strictly separate  $Q$  and  $P$ ?
Here, two hyperplanes strictly separate  $Q$  and  $P$  if, for every pair  $(\mathbf{q},\mathbf{p})\in Q\times P$ , the open line segment  $\mathbf{qp}$  is intersected by at least one hyperplane and no point from  $Q\cup P$  is contained in any of the two hyperplanes. Giannopoulos et al. [2009] showed that this problem is W[1]-hard with respect to  $d$  and not solvable in  $\rho (d)m^{o(d)}\mathrm{poly}(L)$  time assuming the ETH (where  $m\coloneqq |Q\cup P|$  and  $L$  is the instance size). The idea of our reduction is that a separating hyperplane can be realized by two ReLUs approximating a "step function". To this end, we add some auxiliary points to  $Q\cup P$  to ensure the separating behaviour of the ReLUs.
5 Hardness Results for Linear Threshold Activations
A nowadays less popular, but more classical activation function than ReLU is the linear threshold function  $x \mapsto \mathbb{1}_{\{x > 0\}}$ . We define 2L-LT-NN-TRAIN( $\mathcal{L}$ ) as the decision version of the training problem for linear threshold functions. It is defined exactly as 2L-RELU-NN-TRAIN( $\mathcal{L}$ ) with two modifications: first, the ReLU function  $[x]_{+}$  is replaced with  $\mathbb{1}_{\{x > 0\}}$ , and second, we do not assume  $a_{j} \in \{-1,1\}$  anymore because the normalization used in the ReLU case does not apply here. Instead we allow arbitrary  $a_{j} \in \mathbb{R}$ .
As in the ReLU case, the crucial ingredient to study the training complexity of linear threshold networks is their geometry. To this end, observe that every function represented by a 2-layer linear
threshold network is piecewise constant, where the pieces emerge from the hyperplane arrangement defined by the  $k$  hyperplanes corresponding to the hidden neurons. Since our reductions for the ReLU case always use two ReLUs to approximate "step functions" from 0 to 1 and from 1 to 0, it is easy to adapt the reductions to the linear threshold case.
Corollary 7. 2L-LT-NN-TRAIN  $(\mathcal{L})$  is NP-hard even for  $d = 2$  and  $\gamma = 0$ .
Proof. We use a reduction analogous to the one in the proof of Theorem 1. Instead of a sum of levees, we use a sum of "stripes" within which the function attains value 1. With this idea, it is straightforward to build selection gadgets and an analogous global construction. Note that the number  $k$  of required linear threshold neurons is only  $k = 2(m + n)$  for a POITS instance with  $m$  clauses and  $n$  variables because each stripe can be realized with two linear threshold neurons instead of the four ReLUs required to build a levee.
For the sake of completeness, we note that also the W[1]-hardness result by Froese et al. [2022] extends to linear threshold functions. To this end, consider the  $\ell^p$ -loss  $\ell^p (\hat{y},y) = |\hat{y} -y|^p$ , where  $\ell^0$  simply counts the non-zero components of  $\hat{y} -y$ .
Corollary 8. For each  $p \in [0, \infty[$ , 2L-LT-NN-TRAIN( $\ell^p$ ) with  $k = 1$  is NP-hard,  $W[1]$ -hard with respect to  $d$  and not solvable in  $\rho(d)n^{o(d)}\mathrm{poly}(L)$  time (where  $L$  is the input bit-length) for any function  $\rho$  assuming the ETH.
Proof. Having a careful look into the reduction from MULTICOLORED CLIQUE by Froese et al. [2022], it turns out that the single ReLU neuron used in this reduction can be replaced by a linear threshold neuron without changing the logic of the reduction.  $\square$
Finally, also Theorem 6 finds its analogue for the linear threshold case.
Corollary 9. 2L-LT-NN-TRAIN  $(\mathcal{L})$  with  $k = 2$  and  $\gamma = 0$  is W[1]-hard with respect to  $d$  and not solvable in  $\rho (d)n^{o(d)}$  poly  $(L)$  time (where  $L$  is the input bit-length) for any function  $\rho$  assuming the ETH.
Proof. The proof is analogous to (even much easier than) the one of Theorem 6. Instead of two RLUs to realize a step of height one, we can simply use one linear threshold neuron (which is why we obtain hardness already for  $k = 2$  in this case). Note that we do not even need to introduce additional data points, that is, we obtain a much more direct reduction from 2-HYPERPLANE SEPARABILITY.
6 An Algorithm for Exact Fitting in the Convex Case
Contrasting the previous hardness results for 2L-RELU-NN-TRAIN  $(\mathcal{L})$ , we now consider the tractable special case where all coefficients  $a_{j}$  are 1. In this case (called 2L-RELU-NN-TRAIN  $(\mathcal{L})^{+}$ ), the neural network realizes a convex continuous piecewise linear function  $\phi$  with at most  $2^{k}$  distinct (affine) pieces. We show fixed-parameter tractability for the parameter  $d + k$  if  $\gamma = 0$  (see Appendix C). Note that, for  $\gamma > 0$ , W[1]-hardness with respect to  $d$  already holds for  $k = 1$  Froese et al. [2022].
Theorem 10. 2L-RELU-NN-TRAIN  $(\mathcal{L})^{+}$  can be solved in  $2^{O(k^2 d)}$  poly  $(k,L)$  time for  $\gamma = 0$  where  $L$  is the input bit-length.
The proof idea is that the convexity of  $\phi$  allows for a branching algorithm assigning the input data points to the  $2^{k}$  pieces of  $\phi$ . We note that Theorem 10 analogously holds for the concave case where  $a_{j} = -1$  for all  $j \in [k]$ . However, if positive and negative coefficients  $a_{j}$  are allowed, then our search tree approach does not work due to non-convexity. Indeed, Theorem 6 implies that this approach cannot work already for  $k = 4$ . It is unclear whether this issue can be resolved for  $k = 2$  or  $k = 3$ .
We closed several gaps in the literature regarding the computational complexity of training two-layer neural networks. Our results give some insight into the geometry of functions realized by such
networks and yield a better understanding of their complexity and expressiveness. We thereby settled the border of computational tractability almost completely. The remaining open questions are the following:
- Is the problem with  $d = 2$  in FPT when parameterized by  $k$ ? This is open for both ReLUs and linear thresholds. Note that W[1]-hardness with respect to  $k$  for any constant  $d$  would answer Question 2 negatively.
- Is the case  $\gamma = 0$  and  $k \in \{2,3\}$  in FPT with parameter  $d$  for ReLUs?
In a broader context, open directions are to further study the computational complexity in appropriate approximate settings, draw further conclusions on generalization, and understand deeper network architectures.
Christoph Hertrich is supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreements ScaleOpt-757481 and ForEFront-615640).
M. Abrahamsen, L. Kleist, and T. Miltzow. Training neural networks is  $\exists \mathbb{R}$ -complete. In Advances in Neural Information Processing Systems 35 (NeurIPS '21), pages 18293-18306, 2021.
R. Arora, A. Basu, P. Mianjy, and A. Mukherjee. Understanding deep neural networks with rectified linear units. In Proceedings of the 6th International Conference on Learning Representations (ICLR '18). OpenReview.net, 2018.
D. Bertschinger, C. Hertrich, P. Jungeblut, T. Miltzow, and S. Weber. Training fully connected neural networks is  $\exists \mathbb{R}$ -complete. CoRR, abs/2204.01368, 2022.
A. L. Blum and R. L. Rivest. Training a 3-node neural network is NP-complete. Neural Networks, 5 (1):117-127, 1992.
D. Boob, S. S. Dey, and G. Lan. Complexity of training ReLU neural network. Discrete Optimization, 44(Part 1):100620, 2022.
M. Cygan, F. V. Fomin, L. Kowalik, D. Lokshtanov, D. Marx, M. Pilipczuk, M. Pilipczuk, and S. Saurabh. Parameterized Algorithms. Springer, 2015.
S. Dereich and S. Kassing. On minimal representations of shallow ReLU networks. Neural Networks, 148:121-128, 2022.
S. S. Dey, G. Wang, and Y. Xie. Approximation algorithms for training one-node ReLU neural networks. IEEE Transactions on Signal Processing, 68:6696-6706, 2020.
R. G. Downey and M. R. Fellows. Fundamentals of Parameterized Complexity. Springer, 2013.
V. Froese, C. Hertrich, and R. Niedermeier. The computational complexity of ReLU network training parameterized by data dimensionality. Journal of Artificial Intelligence Research, 74:1775-1790, 2022.
M. R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman and Company, 1979.
P. Giannopoulos, C. Knauer, and G. Rote. The parameterized complexity of some geometric problems in unbounded dimension. In Parameterized and Exact Computation, 4th International Workshop (IWPEC '09), volume 5917 of LNCS, pages 198-209. Springer, 2009.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS '11), volume 15 of JMLR Proceedings, pages 315-323. JMLR.org, 2011.
S. Goel, V. Kanade, A. R. Klivans, and J. Thaler. Reliably learning the ReLU in polynomial time. In Proceedings of the 30th Conference on Learning Theory (COLT '17), volume 65 of PMLR, pages 1004-1042. PMLR, 2017.
S. Goel, A. R. Klivans, P. Manurangsi, and D. Reichman. Tight hardness results for training depth-2 ReLU networks. In 12th Innovations in Theoretical Computer Science Conference (ITCS '21), volume 185 of LIPics, pages 22:1-22:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021.
C. Haase, C. Hertrich, and G. Loho. Lower bounds on the depth of integral ReLU neural networks via lattice polytopes. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR '23), 2023.
C. Hertrich and L. Sering. ReLU neural networks of polynomial size for exact maximum flow computation. In The 24th Conference on Integer Programming and Combinatorial Optimization (IPCO '23), 2023. Accepted for publication.
C. Hertrich, A. Basu, M. Di Summa, and M. Skutella. Towards lower bounds on the depth of ReLU neural networks. Advances in Neural Information Processing Systems, 34:3336-3348, 2021.
J. Huchette, G. Muñoz, T. Serra, and C. Tsay. When deep learning meets polyhedral theory: A survey. CoRR, abs/2305.00241, 2023.
R. Impagliazzo and R. Paturi. On the complexity of  $k$ -SAT. Journal of Computer and System Sciences, 62(2):367-375, 2001.
S. Khalife and A. Basu. Neural networks with linear threshold activations: structure and algorithms. In The 23rd Conference on Integer Programming and Combinatorial Optimization (IPCO '22), pages 347-360, 2022.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436-444, 2015.
M. Pilanci and T. Ergen. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In Proceedings of the 37th International Conference on Machine Learning (ICML '20), volume 119 of Proceedings of Machine Learning Research, pages 7695-7705. PMLR, 2020.
S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.