SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cornelius Brand
Algorithms & Complexity Group
Vienna University of Technology
Favoritenstraße 9-11, 1040 Vienna, Austria
cbrand@ac.tuwien.ac.at
Robert Ganian
rganian@gmail.com
Mathis Rocton
mrocton@ac.tuwien.ac.at
In spite of the fundamental role of neural networks in contemporary machine learning research, our understanding of the computational complexity of optimally training neural networks remains incomplete even when dealing with the simplest kinds of activation functions. Indeed, while there has been a number of very recent results that establish ever-tagger lower bounds for the problem under linear and ReLU activation functions, less progress has been made towards the identification of novel polynomial-time tractable network architectures. In this article we obtain novel algorithmic upper bounds for training linear- and ReLU-activated neural networks to optimality which push the boundaries of tractability for these problems beyond the previous state of the art. In particular, for ReLU networks we establish the polynomial-time tractability of all architectures where hidden neurons have an out-degree of 1, improving upon the previous algorithm of Arora, Basu, Mianjy and Mukherjee. On the other hand, for networks with linear activation functions we identify the first non-trivial polynomial-time solvable class of networks by obtaining an algorithm that can optimally train network architectures satisfying a novel data throughput condition.
Neural networks are a prominent tool in contemporary machine learning, one which has found ubiquitous applications throughout modern computer science (Goodfellow et al., 2016). A neural network (cf. Figure 1) can be thought of as a directed acyclic network consisting of  $n$  sources (typically called input nodes), and  $w$  remaining nodes, which we partition into output neurons (the sinks) and hidden neurons (all other nodes in the network, typically organized into layers).
Given the prominence of neural networks, it is surprising that—in spite of recent efforts—relatively little is known about complexity-theoretic upper bounds for the funda-
Figure 1: A neural network with two layers of hidden, ReLU-activated neurons, computing a function  $f:\mathbb{R}^4\to \mathbb{R}^3$
mental problem of Neural Network Training (NNT): given a network and a training data set containing  $m$  samples, compute weights and biases that best fit the training data. This contrasts recent advances leading to novel algorithmic upper bounds as well as extensive complexity-theoretic landscapes for many other problems arising in machine learning and neural network research, including, e.g., principal component analysis (Simonov et al., 2019; Dahiya et al., 2021), clustering (Ganian et al., 2022), Bayesian network learning (Ordyniak & Szeider, 2013; Ganian & Korchemna, 2021; Grüttemeier & Komusiewicz, 2022) and matrix completion (Ganian et al., 2018).
Naturally, the complexity of solving NNT to optimality strongly depends on the activation function used in the network as well as on what kind of restrictions are placed on the structure of the network, but—as we will see—even in the best studied and simplest cases we lack an understanding of the frontiers of polynomial-time tractability. The Rectified Linear Unit (ReLU) activation function is a natural first choice to consider for a complexity-theoretic investigation; over the past years, it has become the most popular and widely studied activation function used in neural networks (Gao et al., 2020). Goel, Klivans, Manurangsi and Reichman (2021) recently established that the Neural Network Training problem with ReLU activation functions (ReLU-NNT) is NP-hard, even when restricted to networks with no hidden neurons, meaning that RELU-NNT is computationally intractable when no further restrictions are placed on the network. In fact, an even more recent reduction shows that RELU-NNT is also complete for the complexity class  $\exists \mathsf{R}$  when restricted to complete networks (Bertschinger et al., 2022) with precisely two input nodes and two output neurons, as opposed to networks with arbitrary structure. Moreover, the problem was shown to be hard for the complexity class W[1] (Froese et al., 2022) when parameterized by  $n$  and also intractable for instances of fixed dimension Froese & Hertrich (2023).
While these results seem discouraging at first, an immediate question that arises is whether we can at least efficiently train "small" networks? In particular, what is the complexity of RELU-NNT (as well as other variants of NNT) when restricted to networks where  $n$ ,  $w$ , or both are (fixed but arbitrary) constants? Here we find a stark contrast between the extensive body of research on the complexity of other natural meta-problems² and how little is known for (RELU-)NNT. Indeed, while recent reductions immediately rule out polynomial-time tractability for networks with constant  $w$  (Abrahamsen et al., 2021; Froese et al., 2022) or constant  $n$  (Bertschinger et al., 2022), the complexity of RELU-NNT in the base case where both  $n$  and  $w$  are bounded by a constant (i.e., when training constant-size networks) is a prominent open question in the field. The best partial answer we had so far for this high-profile question lies in the seminal work of Arora, Basu, Mianjy and Mukherjee (2018) (see also the follow-up work of Boob, Dey and Lan (2022)), who developed a polynomial-time algorithm that can solve RELU-NNT for constant-size networks with a single layer of hidden neurons and one output neuron.
Naturally, the study of NNT is also relevant for other popular activation functions such as the Sigmoid or Tanh—however, these seem even less amenable to contemporary algorithmic techniques than ReLUs. On the other hand, there exists a class of fundamental activation functions for which the training problem turns out to be easier than for ReLUs: linear functions. Linear-activated networks have been considered in numerous settings as well as theoretical examples (Abrahamsen et al., 2021; Cowley & Pillow, 2020; Panigrahi et al., 2020) and can in many ways be viewed as the "baseline" choice for considering the complexity of neural network training. Indeed, unlike for RELU-NNT, it is not difficult to show that LIN-NNT (i.e., NNT with linear activation functions) is polynomial-time tractable for any network of bounded (i.e., constant) size, but is known to be  $\exists \mathsf{R}$ -complete for general networks (Abrahamsen et al., 2021) (see also Section 3). However, the complexity of LIN-NNT for general networks with bounded  $w$ —i.e., with constantly many hidden and output neurons—remains open.
Contributions. While several papers have recently provided lower bounds for NNT (Abrahamsen et al., 2021; Froese et al., 2022; Bertschinger et al., 2022), much less progress has been made towards pushing the frontiers of tractability of the problem. In this paper, we remedy this by providing new algorithmic upper bounds that supersede the previous state of the art for RELU-NNT and LIN-NNT.
Formal definitions are provided in the Preliminaries.
<sup>2</sup>Consider, e.g., Lenstra's celebrated theorem for solving Integer Linear Programs with constantly-many variables (Lenstra & Jr., 1983) and the multitude of improvements of that initial result (Kannan, 1987; Frank & Tardos, 1987; Ganian & Ordyniak, 2019; Brand et al., 2021), or Schaefer's famous dichotomy theorem for Boolean CSPs (Schaefer, 1978) and its more recent generalization to bounded-domain CSPs (Bulatov, 2017).
As our first major contribution, we establish the polynomial-time tractability of ReLU-NNT for all constant-size networks satisfying the property that all hidden neurons have out-degree at most 1. We remark that while this condition is trivially satisfied by every neural network that can be handled by the aforementioned algorithms (Arora et al., 2018; Boob et al., 2022), our result may also be applied to much more general networks and in particular can support more than one output neuron and multiple hidden layers. The algorithm underlying our result can also deal with networks combining ReLU and linear activation functions. Moreover, its running time matches the recent algorithmic lower bound of Froese et al. (2022) for single-neuron architectures, meaning that it is essentially optimal in that setting.
One remarkable consequence of our exact algorithm for RELU-NNT is that it allows us to formulate a procedure which can effectively deal with every constant-size network architecture (even those not satisfying the restriction on the out-degree of hidden neurons). In particular, we show that every constant-size ReLU-activated network architecture can be transformed into a new network architecture which is (1) at least as expressive as the original architecture, and (2) can be trained in polynomial time. Crucially, the depth of the new architecture remains the same and its size depends solely on the size of the original architecture.
Next, we turn to the simpler setting of training neural networks with linear activation functions, i.e., LIN-NNT. Recall that here, we are dealing with networks containing a bounded number of hidden/output neurons, but a potentially large number of input nodes. Once we depart from the case of architectures with complete connections between consecutive layers (which can be solved by a direct application of linear regression), training neural networks to optimality on a given data set is known to be hopelessly hard (Abrahamsen et al., 2021) unless the admissible architectures/data are restricted to highly specialized scenarios, such as those with a single output neuron (see Section 3). As our second contribution, we identify a general data throughput condition that allows us to circumvent the aforementioned intractability and guarantee the polynomial-time tractability of LIN-NNT. In particular, we say that a network admits an untangling if a subset of its nodes can be partitioned into connected blocks—one for each output and input node—such that:
1a. each input node is the sole source of an input-block;
1b. each output node is the sole sink of an output-block; and
2. the adjacencies between blocks reflects reachability between inputs and outputs.
Intuitively, an untangling can be seen as a backbone in a network where each block either aggregates information from a source or into a sink. While it may not be obvious from the definition at first glance, not only do single-output (as well as single-input) neural networks admit a trivial untangling, but so do many other network architectures one may consider. To exploit our notion of untanglings, we obtain an algorithm that solves LIN-NNT in polynomial time whenever the network comes with an untangling. We combine this result with a procedure that computes an untangling (or determines that none exists) in linear time on all architectures with a constant number of hidden neurons, and provide a different linear-time procedure to find untanglings on architectures of constant treewidth (Robertson & Seymour, 1983; Bodlaender, 2016).
Graphs. We assume basic knowledge of graph terminology (Diestel, 2012). The graphs considered in this paper are assumed to be directed, that is, a graph is a pair  $G = (V, E)$ , where  $V$  is a set of vertices and  $E$  a set of directed edges. For a vertex  $v$ , the vertices  $u$  such that there is an edge  $(u, v)$  in  $E$  are called the in-neighbors of  $v$ ; correspondingly, the set of in-neighbors is the in-neighborhood of  $v$ . The notions of out-neighbors and out-neighborhood are defined symmetrically. The in-degree and out-degree of a vertex is the size of its in- and out-neighborhood, respectively. For a vertex subset  $X \subseteq V$ , we denote by  $G[X]$  the graph induced on  $X$ .
Neural Networks. The protagonists of this article are neural networks, which are acyclic graphs endowed with additional information. In particular, the architecture of a neural network  $\mathcal{N}$  is a directed acyclic graph  $G = (V,E)$  where the nodes are called neurons. We refer to nodes with no incoming edges as input nodes, while those with no outgoing edges are called output neurons. All other nodes are called hidden neurons. We refer to edges as deep when they are not incident to an input node. Typically, one assumes that the hidden neurons are partitioned into layers defined by their
shortest distance to an input node, where a hidden neuron in layer  $i$  only has in-neighbors in layer  $i - 1$  and out-neighbors in layer  $i + 1$ .
In addition to its neural structure,  $\mathcal{N}$  is defined by assigning a bias  $b_{v} \in \mathbb{R}$  for each non-input neuron  $v$  and a weight  $a_{e} \in \mathbb{R}$  for each edge  $e$ . Furthermore, we fix an activation function  $\sigma: \mathbb{R} \to \mathbb{R}$  for the entire neural network. To formally map the data dimensions onto the input and output nodes in an architecture, we will assume that these nodes come with an implicit predetermined ordering.
A neural network defined as above computes a function  $f_{\mathcal{N}}: \mathbb{R}^n \to \mathbb{R}^d$ , where  $n$  is the number of input nodes, and  $d$  the number of output neurons of  $\mathcal{N}$ . To define  $f_{\mathcal{N}}$ , we say for each neuron  $v$  that it computes a function  $f_v: \mathbb{R}^n \to \mathbb{R}$ , which is inductively defined as follows: The function computed at the  $i$ -th input node is the coordinate projection  $\pi_i: \mathbb{R}^n \to \mathbb{R}$ ,  $(x_1, \ldots, x_n) \mapsto x_i$ . For a non-input neuron, let  $v_1, \ldots, v_t$  be the in-neighborhood of  $v$ , again in some arbitrary, but fixed order, and let  $\mathbf{a}_v$  be the vector of weights  $(a_{vi}v)_{i=1,\ldots,t}$  in the same order. Furthermore, let  $\mathbf{y}_v = (y_1, \ldots, y_t)$  be the values of the functions computed at the neuron  $v_i$ , respectively—that is,  $y_i = f_{v_i}(x_1, \ldots, x_n)$ . Then,  $v$  computes  $\sigma(\mathbf{a}_v^T \cdot \mathbf{y}_v + b_v)$ . The function computed by the neural network  $\mathcal{N}$  itself is then given by  $(f_{o_1}(x_1, \ldots, x_n), \ldots, f_{o_d}(x_1, \ldots, x_n))$ , where  $o_1, \ldots, o_d$  are the output neurons of  $\mathcal{N}$ .
Clearly, the function computed by  $\mathcal{N}$  depends, among other things, on the employed activation function. Here we will consider two specific and well-studied choices for  $\sigma$ , namely ReLU and linear activations. It is well known that, without loss of generality, these can be assumed to be  $\sigma(x) = x$  (i.e., the identity) for linear activations and  $\sigma(x) = \max\{0, x\}$  for ReLU activations (Goodfellow et al., 2016). We will also employ the notation  $x^{+} = \max\{0, x\}$ .
Training Neural Networks. The task of training a neural network is to, given an architecture and activation, compute a set of weights as well as biases, such that the function computed by the resulting neural network approximates in the best possible manner a given set of training data, that is, a given set of samples  $(x,y)$  where  $x$  and  $y$  are vectors of the appropriate dimension for the network architecture.
More formally, the input to the  $\sigma$ -NNT problem is an architecture  $G = (V,E)$  and data points  $D \subseteq \mathbb{R}^n \times \mathbb{R}^d$ , where  $n$  is the number of input and  $d$  the number of output neurons of the architecture. For every setting  $a$  of all edge weights as well as  $b$  of all biases, this fully determines a neural network  $\mathcal{N}_{a,b}$  that in turn computes a function  $f_{a,b} = f_{\mathcal{N}_{a,b}}$ . The task is then to output a weight  $a_e$  for every edge  $e$  in  $G$  as well as a bias  $b_v$  for every neuron such that the function computed by the resulting neural network  $\mathcal{N}_{a,b}$  minimizes, among all such choices of  $a$  and  $b$ , the  $\ell_2^2$ -loss function  $L_D(a,b) = \sum_{(x^*,y^*) \in D} ||y^* - f_{a,b}(x^*)||_2^2$ , where  $||\cdot||_2$  is the standard Euclidean norm.
Note that the problem formalization in particular entails that we do not regard the type of activation function as part of the input, whereas the network architecture is part of the input by default. This either matches or generalizes previous formalizations of the problem (Arora et al., 2018; Abrahamsen et al., 2021; Goel et al., 2021; Froese et al., 2022; Bertschinger et al., 2022). We will denote the problems of training neural networks for linear and ReLU activations as LIN-NNT and RELU-NNT, respectively, and the analogous problem where the network contains both linear and ReLU activation functions as MIXED-NNT. Furthermore, while the definition of  $\sigma$ -NNT implies that a solution is one which minimizes the  $\ell_2^2$ -loss function, we will also sometimes call these "optimal solutions" for emphasis. Last but not least, throughout the article we assume that  $|D| \geq 2$  since the case where  $|D| \leq 1$  is trivial.
3 Synopsis of Known Results
Neural network training is a vast area of research that has received an immense amount of attention both on the theoretical as well as the empirical side. In order to put our results into context, we survey the complexity-theoretic understanding of linear and ReLU neural network training.
Linear Activation. In the simplest case of LIN-NNT where we assume the size of the whole architecture to be upper-bounded by a constant, it is known that the problem can be expressed as a constant-size system of polynomial inequalities which can be solved in polynomial time, e.g., via the seminal work of Tarski (1951).
On the other hand, LIN-NNT is known to be intractable even if the number of output neurons is upper-bounded by a fixed constant; in fact, the problem was shown to be  $\exists R$ -hard even when the number of output neurons is fixed to 3 (Abrahamsen et al., 2021).
One intermediate case between the general and fixed-size fixed-dimension networks is when the network architecture is unrestricted in size but has a special structure, namely, where all edges between layers of nodes exist. In this case, LIN-NNT can be solved in polynomial time by a direct application of multidimensional linear regression (Velu & Reinsel, 2013).
The preceding discussion leaves open a variety of possible directions, such as the complexity of training networks that are not necessarily fully connected, but constrained to have two or even a single output neuron. On the other hand, one may ask what happens if we consider only the number of hidden neurons to be bounded by a constant, without restricting the number of input nodes and output neurons. Our results deal with an intermediate case that captures, among other things, the easy case of single-output networks, as well as some more general networks with a constant number of hidden neurons. In particular, we will formulate a technical condition—whether or not the architecture can be untangled—that will allow us to train such networks with linear activations in polynomial time.
Rectified Linear Activation. Recent results from computational complexity theory provide strong lower bounds on the inherent difficulty of the problem of neural network training with ReLU activation functions, already on highly restrictive architectures. Indeed, Goel et al. (2021) and independently Dey et al. (2020) have proven that, under standard complexity assumptions, even training a single ReLU-activated neuron with  $n$  input nodes cannot be accomplished in time polynomial in  $n$  and the size of the training set  $D$ ; see also the related lower bounds of Bertschinger et al. (2022).
A seminal algorithmic result is that of Arora et al. (2018) for training shallow networks consisting of one layer of hidden ReLU neurons and a single output neuron equipped with a linear activation function. Their algorithm runs in time  $|D|^{\mathcal{O}(w \cdot n)}$ , where  $D$  is the training set,  $n$  the number of input nodes and  $w$  the number of other nodes in the architecture. Boob et al. (2022) showed that the same approach can be used to handle cases where the output neuron is equipped with a ReLU activation function as well. Froese et al. (2022) have shown that this running time dependency on  $n$  in the exponent of  $D$  is asymptotically optimal even for a single hidden neuron, ruling out algorithms running in time, e.g.,  $2^n \cdot D^{o(n)}$  for this basic case.
It is a major open problem to extend the techniques of Arora et al. (2018) to general deep networks—in particular, the complexity of RELU-NNT when restricted to networks whose size is upper-bounded by an arbitrary but fixed constant is wide open. Our work shows that it is nevertheless possible to identify a much more general set of architectures than those covered by the aforementioned results (Arora et al., 2018; Boob et al., 2022) for which RELU-NNT is polynomial-time solvable.
4 Training ReLU Networks
In this section, we establish our tractability results for ReLU-activated neural network training when each hidden neuron has out-degree at most 1. As our first step, we prove that in such instances the weights in the hidden layers can be discretized to only two values.
Lemma 1. For any ReLU network  $\mathcal{N}$  and for any hidden neuron  $u$  with precisely one out-neighbor  $w$ , there is a ReLU network  $\mathcal{N}^*$  with the same architecture such that  $f_{\mathcal{N}} = f_{\mathcal{N}^*}$  and  $a_{uw}^* \in \{-1,1\}$ . Moreover, the only parameters in which  $\mathcal{N}$  and  $\mathcal{N}^*$  differ are the weights of the edges incident to  $u$  and the bias of  $u$ .
Proof Sketch. Let  $(v_{i})_{i}$  be the (non-empty) family of predecessors of  $u$ , and  $w$  its only successor. To obtain  $\mathcal{N}^*$ , we construct a new set of weights and biases at  $u$ , where  $\operatorname{sgn}(x)$  is the sign function:
$$
\begin{array}{l} \forall i, a _ {v _ {i} u} ^ {*} = a _ {v _ {i} u} \cdot | a _ {u w} | \\ b _ {u} ^ {*} = b _ {u} \cdot | a _ {u w} | \\ a _ {u w} ^ {*} = \operatorname {s g n} \left(a _ {u w}\right) \quad (1 \text {i f} a _ {u w} = 0) \\ \end{array}
$$
All other weights in  $\mathcal{N}^*$  remain the same as in  $\mathcal{N}$ . To complete the proof, it suffices to show that  $f_{\mathcal{N}} = f_{\mathcal{N}^*}$  holds by using the fact that the ReLU function commutes with multiplication with non-negative factors.
By iteratively applying Lemma 1, we obtain:
Lemma 2. Let  $\mathcal{N}$  be a ReLU structure such that every hidden neuron has out-degree 1. Then there exists an optimal solution to ReLU-NNT such that all deep edges have weights in  $\{1, -1\}$ .
Lemma 2 will later allow us to reduce our search space for RELU-NNT by only needing to consider two options for the weights of each deep edge. We note that Maass (1997) employed a related idea.
Partitioning by Hyperplanes. The other key argument used to reduce the search space further concerns the way ReLU neurons partition the training set into a "dead area" (i.e., the elements where the linear unit is rectified) and an "active area" (where the unit behaves as a linear function). More precisely, given a data set  $D$  and a ReLU-activated neuron  $u$ , we say that  $u$  partitions  $D$  into an active area  $D|_u$ , which contains all data points for which  $f_u$  outputs a value greater than zero, and a dead area  $\overline{D|}_u = D \setminus D|_u$ , which contains all data points for which  $f_u$  outputs zero.
As a base case, let us consider a neuron  $u$  in the first hidden layer of a neural network and let  $\mathbf{a}_u^T$  be the vector of weights for the edges from the input nodes to  $u$ , where a non-edge would be represented as a weight of 0. As a trivial upper bound, there are in general at most  $2^{|D|}$  ways  $u$  can partition  $D$  into an active and dead area, which provides a trivial upper bound for an algorithm enumerating all such partitions. However, we have that  $f_u(D_i) = (\mathbf{a}_u^T \cdot D_i + b_u)^+$  for each data point  $D_i \in D$  and moreover  $\mathbf{a}_u^T \cdot \mathbf{x} + b_u$  defines a hyperplane in an  $n$ -dimensional space. This allows us to obtain an algorithm with a better running time bound for enumerating the partitions of the point set that is inspired by a result of Megiddo (1988).