SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 3. Let  $D \subseteq \mathbb{R}^n$  be a finite set of points. Then, the set of partitions  $\Pi = \{(A,B) \mid A \cup B = D \text{ and } A, B \text{ are separated by a hyperplane}\}$  can be enumerated in time  $|D|^{\mathcal{O}(n)}$ .
Remark. A similar result was claimed and used by Arora et al. (2018, Page 14) in their algorithm for dealing with RELU-NNT restricted to a special case of the setting treated in this section. However, the argument presented there seems to be incomplete. In particular, it makes the claim that "the total number of possible hyperplane partitions of a set of size  $D$  in  $\mathbb{R}^n$  is at most  $2\binom{D}{n}$ ", which is imprecise (consider, e.g.,  $|D| = n$ ). Moreover, the book listed as reference for bounding the total number of possible hyperplane partitions does not provide an algorithm for enumerating these efficiently, and enumeration is required for both our result and the result claimed in Arora et al. (2018). A bound on the number of such partitions can be attributed to Harding (1967), but that does not yield efficient enumeration either. If the points are known to lie in general position, efficient enumeration could be carried out via a translation to hyperplane arrangements followed by an application of the results of Edelsbrunner et al. (1986); however, in our setting we cannot guarantee that these lie in general position, necessitating our stand-alone proof of Lemma 3 above (which we believe to be also of general interest).
The considerations preceding Lemma 3 do not immediately translate to neurons that are deeper than in the first layer, since the inputs to these activation functions are obtained by non-linear transformations of  $D$ . Nevertheless, we prove that it is also possible to derive an upper bound for neurons beyond the first hidden layer if we are provided information about the partitions of neurons in previous layers.
Lemma 4. Let  $D$  be a set of data points,  $u$  be a ReLU neuron,  $F$  be the set of all neurons on paths from input nodes to  $u$ , and  $x = |F|$ . Given  $D|_v$  for each  $v \in F$ , we can upper-bound the number of distinct active areas  $D|_u \subseteq D$  over all networks consistent with the given selection of  $\{D|_v \mid v \in F\}$  by  $|D|^{\mathcal{O}(n \cdot 2^x)}$ , and it is possible to enumerate a superset of these in time  $|D|^{\mathcal{O}(n \cdot 2^x)}$ .
We now show how to solve RELU-NNT in polynomial time for each fixed value of  $n$  and  $w$ :
Theorem 5. There is an algorithm that, given an instance  $(G,D)$  of RELU-NNT such that every hidden neuron of the structure has out-degree exactly 1, and the data points are encoded using  $L$  bits overall, computes the global optimal solution in time  $|D|^{\mathcal{O}(n\cdot w\cdot 2^w)}\cdot \mathrm{poly}(L)$ .
Proof Sketch. The algorithm begins by exhaustively branching to determine the weight of each deep edge, which requires us to consider at most  $2^w$  options in view of Lemma 2. Once the weights of the deep edges are fixed, we branch to determine how each hidden and output neuron partitions the training set. For this to work, we need to process layers consecutively—we only branch on the possible partitions for one neuron after we have chosen the partitions for all of its hidden in-neighbors.
For each such neuron, we enumerate all feasible partitions of  $D$  using Lemma 4, and branch over the possibilities. For a given partition, we create a set of inequalities that are satisfied only if each element of  $D$  indeed is affected by  $u$  according to the partition. Once a partition has been chosen for all neurons, the function computed by the resulting neural network  $\mathcal{N}$  for each neuron is fixed save for the weights of edges adjacent to input nodes, and the biases. We will optimize over all these free variables by constructing an instance of the QUADRATIC OPTIMIZATION problem, whereas we optimize  $L_{D}(a,b)$  subject to the set of constraints obtained when partitioning the neurons. The obtained instances of QUADRATIC OPTIMIZATION can be solved in time polynomial in the number of variables and constraints, as well as the bitlengths of the datapoints, e.g., by using the ellipsoid method (Pang, 1983; Kozlov et al., 1979).
Note that the proof of Theorem 5 can be directly extended to also include architectures where some of the hidden or output neurons use a linear activation function instead of ReLU. Indeed, we need not guess a partition for the neuron and hence not construct a constraint in the quadratic program, and furthermore we may treat a linear neuron the same way as an activated ReLU in the following layer. This yields the following theorem, which generalizes the tractability results of both Arora et al. (2018) and Boob et al. (2022).
Corollary 6. There is an algorithm that, given an instance  $(G,D)$  of MIXED-NNT such that every hidden neuron of the structure has out-degree 1, and the data points are encoded using  $L$  bits overall, computes the global optimal solution in time  $|D|^{\mathcal{O}(n\cdot w\cdot 2^w)}\cdot \mathrm{poly}(L)$ .
5 Blowing Up a Neural Network
The tractability results of Arora et al. (2018) on constant-width shallow ReLU-networks raise the natural question of whether it is possible to (efficiently) perform depth reduction on deeper networks, so as to reduce a given (deep) instance for RELU-NNT to a tractable (e.g., shallow) special case. It is widely conjectured that this is not possible in general without sacrificing some quality of the solution (see, e.g., Hertrich (2022), Conjecture 3.1. and Haase et al. (2023); Hertrich et al. (2021)). Among others, it is widely believed that there are functions that are efficiently computable by depth-3 but not by depth-2 ReLU-activated neural networks (Eldan & Shamir, 2016), and similar results were also shown for deeper networks (Cohen et al., 2016; Telgarsky, 2016).
While these results essentially rule out performing depth reduction without a negative impact on the accuracy, our tractability results in Section 4 allow us to approach the question from a different angle. In particular, we show that it is possible to transform an arbitrary constant-size ReLU-activated deep neural network architecture into one of the same depth which is (1) at least as powerful as the original architecture, (2) guaranteed to admit polynomial-time training, and (3) upper-bounded in size by a function of the size of the original architecture<sup>3</sup>.
To explain the construction, we start again with the base case of a single neuron. More precisely, let  $v$  be a single ReLU-activated (hidden) neuron with  $n$  inputs and  $d$  outputs, and say the weights on the input edges are  $a_1, \ldots, a_n$ , the weights on the output edges are  $a_1', \ldots, a_d'$ , and the bias at the hidden neuron is  $b$ . Now consider a new network with a single layer of ReLU-neurons which consists of  $d$  ReLU-activated neurons  $v_1, \ldots, v_d$ , where every  $v_i$  is connected to all inputs with the same weights  $a_1, \ldots, a_n$  as before. Furthermore, each  $v_i$  has a single output edge with weight  $a_i'$ . We call this network the blow-up of  $N$ . Let us first establish that this blow-up has the desired property:
Lemma 7. Let  $N$  be a neural network with a single ReLU-activated hidden neuron as before, and let  $N'$  be its blow-up. Let  $f_{N}, f_{N'}: \mathbb{R}^{n} \to \mathbb{R}^{d}$  be the functions computed by  $N$  and  $N'$ , respectively. Then,  $f_{N} = f_{N'}$  holds.
This procedure can be generalized to blow up an arbitrary ReLU-activated neural network, regardless of the number of its layers.
With a slight abuse of notation, we will also refer to the blow-up of a given network architecture  $G$  (that is, one without weights and biases) as the architecture of the blow-up of an arbitrary neural network with architecture  $G$ , and the process for obtaining the blow-up of an architecture is the very
<sup>3</sup>We remark that such a transformation can be trivially provided for linearly-activated network architectures, since LIN-NNT is polynomial-time solvable on complete networks. We also explicitly note that changing the architecture is not always desirable or even possible.
same as for a weighted network (ignoring any weights). We now turn to establishing the properties of blow-ups with respect to RELU-NNT.
Proposition 8. Let  $G$  be an arbitrary architecture and let  $G'$  be its blow-up. Furthermore, let  $L_{D}$  and  $L_{D}'$  be the loss functions associated with  $G$  and  $G'$ , respectively. Then, for any weights  $a, b$  on  $G$  and the minimizer  $a^{*}, b^{*}$  of  $L_{D}'$  on  $G'$ , we have  $L_{D}'(a^{*}, b^{*}) \leq L_{D}(a, b)$ .
The size of the blow-up of a network with a constant number of hidden neurons behaves as follows.
Proposition 9. Let  $\mathcal{N}$  be a neural network with  $\ell$  layers of  $\lambda \geq 2$  hidden neurons each, and  $o$  output neurons. Then the blow-up of  $\mathcal{N}$  has at most  $b \leq o \cdot \lambda^{\ell + 1}$  hidden neurons, and at most  $o \cdot \lambda^\ell$  neurons in its first hidden layer. If  $\lambda = 1$ , we have  $b \leq o \cdot \ell$ .
Recalling Theorem 5 from Section 4, we now directly obtain:
Theorem 10. There is an algorithm that
- takes as input an instance  $(G, D)$  of RELU-NNT where  $G$  has depth  $\ell$ ,  $n$  input nodes,  $o$  output neurons and at most  $\lambda$  hidden neurons per layer;
computes the blow-up  $G^{\prime}$  of  $G$  and an optimal solution to RELU-NNT on  $(G^{\prime},D)$ , and
- runs in time  $|D|^{\mathcal{O}(n\cdot \tau \cdot 2^{\tau})}$ , where  $\tau = (\ell + \lambda^{\ell + 1}) \cdot o$ .
Observe that the running time in Theorem 10 is polynomial for every architecture of constant size and essentially tight with respect to the lower bound of Froese et al. (2022) for architectures with a single hidden neuron. In combination with Proposition 8, this implies a polynomial-time algorithm which transforms an arbitrary constant-size architecture into a new one which has already been solved to optimality, with no sacrifice in either accuracy or depth.
We remark that the procedure described in this section can be seen as a rough analogue to improper learning in learning theory in that it involves a change of the hypothesis class. In particular, while we prove that our blow-up procedure does not increase the training error, it might increase the generalization error of empirical risk minimization in learning settings. We also point out that a similar procedure was described in an earlier work of Maass, albeit in it was applied in a different context Maass (1997).
6 Training Linear Networks
Untangling of Linear Networks. As discussed in Section 3, the complexity of LIN-NNT is open when restricted to architectures with arbitrarily many input nodes while considering the number of other nodes in the network to be fixed by a constant, and very few classes of architectures are known to admit polynomial-time training for linear activations. Here, we identify a general substructure of the architecture which, when present, guarantees the polynomial-time solvability of LIN-NNT.
Specifically, an untangling of an architecture  $G = (V, E)$  with input nodes  $x_{1}, \ldots, x_{\alpha}$  and output nodes  $y_{1}, \ldots, y_{\beta}$  is a partitioning  $\Pi = (B, C_{1}, \ldots, C_{\alpha}, D_{1}, \ldots, D_{\beta})$  such that:
Each  $G[C_i], 1 \leq i \leq \alpha$  contains  $x_i$  as its only source;
Each  $G[D_j]$ $1\leq j\leq \beta$  contains  $y_{j}$  as its only sink; and
- For each  $x_{i}$  and  $y_{j}$ , if  $y_{j}$  is reachable from  $x_{i}$  in  $G$ , there exist  $s \in C_{i}$  and  $t \in D_{j}$  such that  $st \in E$ .
As the first result in this section, we will show that if one is given a network architecture along with an untangling, it is possible to solve LIN-NNT in polynomial time. Towards this, we observe that since all transformations in such networks are affine, it is possible to assume without loss of generality that non-zero biases are only present on output neurons. Indeed, one can slide the biases along the edges in the network towards the output neurons, and if we multiply them by the weights of edges met along the way, the computed function does not change. As an example, given an edge with weight  $a_1$  from a neuron associated with bias  $b_2 \neq 0$  to a neuron associated with bias  $b_1$ , we can alter the biases by setting  $b_2 \coloneqq 0$  and  $b_1 \coloneqq b_1 + a_1 \cdot b_2$  since  $a_1 \cdot (x + b_2) + b_1 = a_1 \cdot x + (b_1 + a_1 \cdot b_2)$ . Applying this iteratively along all the edges of a network yields:
Observation 11. For each neural network  $\mathcal{N}_{a,b}$  with linear activation functions, there exists a neural network  $\mathcal{N}_{a,b'}$  such that  $b'$  is non-zero only on the output neurons and  $f_{\mathcal{N}_{a,b}} = f_{\mathcal{N}_{a,b'}}$ .
Observation 11 allows us to proceed towards establishing the claimed tractability result.
Theorem 12. LIN-NNT can be solved in time poly  $(|D| + |V|)$  if an untangling of the architecture  $G = (V, E)$  is provided as part of the input.
Computing Untanglings. Given Theorem 12, the natural next question concerns the complexity of actually computing an untangling in a neural network. The aim of this section is to provide a comprehensive answer to that question. As our first result, we prove that deciding whether a given architecture has an untangling is, in general, intractable.
Theorem 13. Deciding whether a given architecture  $G$  has an untangling is NP-hard.
Proof Sketch. We provide a polynomial reduction from the NP-hard DOMINATING SET 3- PARTITIONING problem (Heggernes & Telle, 1998): Given an undirected graph  $H = (V, E)$ , decide whether its vertex set can be partitioned as  $V = A_{1} \cup A_{2} \cup A_{3}$  such that  $A_{1}, A_{2}$  and  $A_{3}$  are all dominating sets of  $H$ . Here, a dominating set of a graph is a subset  $X$  of its vertices such that every vertex  $v$  is either in  $X$  itself, or adjacent to some  $w \in X$ ; we say that such  $w$  dominates  $v$ .
Let  $H = (V, E)$  be an undirected graph. We first describe how to transform  $H$  into an architecture  $G$  such that  $G$  has an untangling if and only if  $H$  can be partitioned into three dominating sets.  $G$  is constructed as a three-layered architecture, that is, with vertex set  $L_1 \cup L_2 \cup L_3$ , where  $L_1 = \{d_1, d_2, d_3\}$ ,  $L_2 = V$ , and  $L_3$  consists of  $|V| + 1$  disjoint copies of  $V$ , that is,  $L_3 = \bigcup_{v \in V} \{v^{(1)}, \ldots, v^{(|V| + 1)}\}$ . Additionally,  $G$  has all possible edges from  $L_1$  to  $L_2$ , and  $v \in L_2$  has an edge towards  $u^{(i)} \in L_3$  in  $G$  if and only if  $u = v$  or  $v$  was adjacent to  $u$  in  $H$ . To complete the proof, it suffices to establish that every untangling in  $G$  implies a partitioning of  $H$  into three dominating sets, and vice-versa that every such partitioning of  $H$  guarantees that there is an untangling in  $G$ .
While negative, the intractability of computing an untangling in a general architecture is far from unexpected. For our next result, we take aim at the restriction of LIN-NNT where the architectures have an arbitrary but fixed bound on the number of hidden neurons—a case which is notable not only due to the fact that the target architectures will typically be much smaller than the training set, but also because its complexity remains open (cf. Section 3). We show that under this restriction, the problem of determining whether the architecture has an untangling is solvable in linear time.
Theorem 14. There is an algorithm which either computes an untangling in a given architecture  $G = (V, E)$  with  $k$  hidden neurons in time  $k^{\mathcal{O}(k)} \cdot |V|$ , or correctly determines that none exists.
Proof Sketch. Let  $W \subseteq V$  be the set of hidden neurons in  $G$ . We begin by exhaustively branching over all of the at most  $k^{\mathcal{O}(k)}$  partitionings of  $W$  into sets  $\Pi' = (B, C_1', \dots, C_\alpha', D_1', \dots, D_\beta')$  (for all choices of  $\alpha$  and  $\beta$ , both of which are trivially upper-bounded by  $k$ ). For each such choice of  $\Pi'$ , the algorithm then proceeds by determining whether it can be extended to an untangling of  $G$  by adding an input node to each  $C_i'$ ,  $1 \leq i \leq \alpha$ , an output neuron to each  $D_j'$ ,  $1 \leq j \leq \beta$ , and keeping all remaining input nodes and output neurons as singletons in the untangling, in a way which satisfies the reachability constraint of an untangling. To this end, we will employ a further set of exhaustive branching rules which will iteratively assign singleton output neurons and input nodes to the sets in  $\Pi'$ . Intuitively, whenever we identify that an input node is missing a connection to an output node (due to at least one of these not being assigned to a part in  $\Pi'$ ), we exhaustively branch over all ways one of these two neurons can be assigned to a part in  $\Pi'$ .
As our final result, in the next Theorem 15 we show that it is also possible to compute untanglings in linear time even on architectures with many hidden neurons, provided that such architectures are "well-structured" in a graph-theoretic sense. In particular, we show that this holds for architectures which are tree-like in the sense of having bounded treewidth: a highly established structural graph parameter which measures how tree-like a graph is (Robertson & Seymour, 1983). Specifically, here we consider the treewidth of the underlying undirected graph, i.e., the simple graph obtained by replacing each directed edge in the architecture by an undirected one.
To put Theorem 15 into context, we remark that the proof is based on an application of Courcelle's well-known Theorem (Courcelle, 1990) and hence a definition of treewidth will not be required for an exhibition of the result or its proof; instead, we refer to the multitude of existing surveys and materials on the topic (Courcelle & Engelfriet, 2012; Bodlaender, 2016). We remark that the result is incomparable to Theorem 14: while it can easily be shown that every architecture with  $k$  hidden
neurons also has treewidth at most  $k$  and hence Theorem 15 can be applied to a strict superclass of architectures, the running time guaranteed here has a nonelementary dependency on the treewidth of the graph (as opposed to the  $k^{\mathcal{O}(k)}$  dependency in Theorem 14).
Theorem 15. Let  $t$  be an arbitrary but fixed integer. Given an architecture  $G = (V, E)$  of treewidth at most  $t$ , it is possible to compute an untangling (or correctly determine that none exists) in time  $f(t) \cdot |V|$  for some computable function  $f$ .
We have examined the theoretical boundaries of computational tractability for training neural networks both for the fundamental cases of ReLU and linear activations, complementing a flurry of recent results establishing increasingly stronger lower bounds for network training. Our results generalize recent algorithms for optimal ReLU-activated network training (Arora et al., 2018; Boob et al., 2022) and are among the first to identify a non-trivial class of linear-activated architectures admitting polynomial-time training.
A natural direction for future work is to examine the extent to which the identified islands of tractability can be generalized before stepping into intractable territory. A long-term goal is to obtain a clear cutoff between intractable and tractable cases of network training. We conclude with three specific questions directly arising from our results:
1. Is LIN-NNT polynomial-time tractable when restricted to constant-treewidth architectures?
2. Is it possible to solve RELU-NNT for constant-size architectures containing hidden neurons with out-degree greater than 1?
3. Can the obtained results be generalized to other loss functions?
The authors confirm that there are no competing interests. We acknowledge support by the Austrian Science Fund (FWF, START Project Y1329). Mathis Rocton further acknowledges support by the European Union's Horizon 2020 research and innovation COFUND programme (LogiCS@TUWien, grant agreement No 101034440).
Abrahamsen, M., Kleist, L., and Miltzow, T. Training neural networks is er-complete. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 18293-18306, 2021. URL https://proceedings.neurips.cc/paper/2021/bit/9813b270ed0288e7c0388f0fd4ec68f5-AAbstract.html.
Arora, R., Basu, A., Mianjy, P., and Mukherjee, A. Understanding deep neural networks with rectified linear units. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=B1J_rgWRW.
Bertschinger, D., Hertrich, C., Jungeblut, P., Miltzow, T., and Weber, S. Training fully connected neural networks is  $\exists$ r-complete. CoRR, abs/2204.01368, 2022. doi: 10.48550/arXiv.2204.01368. URL https://doi.org/10.48550/arXiv.2204.01368.
Bodlaender, H. L. Treewidth of graphs. In Encyclopedia of Algorithms, pp. 2255-2257. 2016. doi: 10.1007/978-1-4939-2864-4\431. URL https://doi.org/10.1007/978-1-4939-2864-4_431.
Boob, D., Dey, S. S., and Lan, G. Complexity of training relu neural network. Discret. Optim., 44 (Part):100620, 2022. doi: 10.1016/j.disopt.2020.100620. URL https://doi.org/10.1016/j.disopt.2020.100620.