The Real Tropical Geometry of Neural Networks for Binary Classification
Authors: Marie-Charlotte Brandenburg
Georg Loho
Guido Montufar
Venue: Accepted by TMLR ()

Abstract: We consider a binary classifier defined as the sign of a tropical rational function, that is, as the difference of two convex piecewise linear functions. In particular, the set of functions represented by a ReLU neural network can be regarded as a subset in the parameter space of tropical rational functions, specifically, it is contained as a semialgebraic set. We initiate the study of two different subdivisions of the parameter space of tropical rational functions with fixed number of terms in the numerator and denominator: a subdivision into semialgebraic sets, on which the combinatorial type of the decision boundary is fixed, and a subdivision into a polyhedral fan, capturing the combinatorics of the partitions of the dataset. The sublevel sets of the $0/1$-loss function arise as subfans of this classification fan, and we show that the level-sets are not necessarily connected. We describe the classification fan i) geometrically, as normal fan of the activation polytope, and ii) combinatorially through a list of properties of associated bipartite graphs, in analogy to covector axioms of oriented matroids and tropical oriented matroids. Our findings extend and refine the connection between neural networks and tropical geometry by observing structures established in real tropical geometry, such as positive tropicalizations of hypersurfaces and tropical semialgebraic sets.

Published in Transactions on Machine Learning Research (09/2024)
The Real Tropical Geometry of Neural Networks for Binary Classification
Marie-Charlotte Brandenburg KTH Royal Institute of Technology
brandenburg@kthh.sm
Georg Loho
georg. Lorg..loooone hannglllokk@mmatfa..ubbeebbbtate
University of Twente & Freic Universität Berlón
Guido Montúfar
montufar@matth.uk ucla. edu
University of California, Los Angeles Maz Planck Institute MiS
Reviewed on OpenReview: https: // openreveee.r.....e..mmee net/ pet//forune forum? id- 1dJW
Abstract
We consider a binary classifier defined as the sign tropical rational function, that is, as the difference of two convex piecewise linear functions. In particular, the of functions represented by ReLU neural network can be regarded as subset in the parameter space of tropical rational functions, specifically, it is contained as semialgebraic set. We initiate the study of two different subdivisions of the parameter space of tropical rational functions with fixed number of terms in the numerator and denominator: a subdivision into semialgebraic sets, on which the combinatorial type of the decision boundary is fixed, and a subdivision into a polyhedral fan, capturing the combinatorics of the partitions the dataset. The sublevel sets of the 0/1-loss function arisc as subfans of this classification fan, and we show that the level-sets are not necessarily connected. We describe the classification fan geometrically, as normal fan of the activation polytope, and ii) combinatorially through list of properties of associated bipartite graphs, in analogy to covector axioms of oriented matroids and tropical oriented matroids. Our findings extend and refine the connection between neural networks and tropical geometry by observing structures established in real tropical geometry, such as positive tropicalizations of hypersurfaces and tropical semialgebraic sets.
1 Introduction
We consider binary classification task with hypotheses given by signs of real-valued functions parameterized by artificial neural networks with piecewise linear activation functions. Given classification task, are interested in the sets of parameters for which the network perfectly classifies the training data, for which it makes a certain number of errors, that is, the level sets of the 0/1-loss function. We seek to understand the combinatorial and methods. One of our aims to enhance the synergies between neural networks and real tropical geometry facilitate progress in both communitics by translating some concepts and results.
structures underlying such a classification task using polyhedral
The combinatorics of the functions represented by neural networks with piecewise linear activations has received significant attention over the years, e.g., in the works of Pascanu et al. (2014): Montúfar et al. (2014h; Telgarsky (2016): Raghu et al. (2017 Serra et al. (2018); Balestricro et al. (2019), or works listed in the overview article of Huchette et al. (2023). A new trend in theoretical considerations of (deep) neural networks with piecewise linear activation functions to study them through the lens of tropical geome- try, mathematical framework which tailored to understand the goometry of piecewise lincar functions. The language of tropical geometry in the context of binary classification already appeared in the work of Charisopoulos & Maragos (2017), and the relation to feedforward neural networks with ReLU activation

---

Published in Transactions on Machine Learning Research (09/2024)
was expanded independently by Charisoponlos & Maragos (2018) and Zhang et al. (2018) and to maxout networks by CChaiioppollss & Maragos| (2018) and [Mnntáfar et al. (2022). The connection arises through the fact that any function represented by such a neural network is continuous piecewise linear (and vice versa, Arora et al. 2018) and can thus be written as the difference of two convex piccewise lincar functions (Hartman 1959: Wang 2004). Such a difference is called a trapical ralional function. The representation as such a difference is not unique, and has been studied for instance by Kripfganz & Schulze (1877) and Melzer (1886); Schlüter & Darup (2020); Tran & Wang (023).
In recent years, there has been increased interest in real tropical geometry, which also encompasses sign information and aims to study tropical varieties inside all orthants (see, Jell et al. 2020 and Rau et al. 2022). This is the perspective that we take in the present article. Concretely, we consider differences - h of two convex (and continuous) piecewise linear functions g, h, where we fix the maximum possible number of linear terms of each, g and h.
The simplest instance of this setup is a linear classifier, which is the special case where g and h are linear functions. Here, results have been rediscovered in many different mathematical communities. The well-known work of Cover (1964) studied the structure of the space of parameters of linear classifiers, which is subdivided by an arrangement of hyperplanes, and described the solution set of the classification task as a polyhedral cone. These structures have been expanded widely by the combinatorics community in the years since. One notable example is the counting formula for chambers in a general hyperplane arrangement by Zaslavsky (1755), and the development of oriented matroids (Björner et al. 1999). which are combinatorial objects that capture the combinatorial structure of the data and their corresponding hyperplane arrangements. Linear classifiers can be modeled by linear networks, which ppramm)paamtt linear functions compositions of linear functions, one for each layer of the network. The space of parameters of linear functions that can be represented by a linear network consists of matrices whose rank is at most equal to the minimum width of any of the network layers, which is an algebraic variety inside the space of matrices. In analogy, in this article we describe the space of parameters of piccewise linear functions that can be represented by a ReLU neural nelwork as a semialgebraic set inside the space of tropical rational functions.
In general, the combinatorial structure of the parameter space refers to its subdivision into regions repre- scnting different dichotomics of the input data. Such subdivisions have been considered in several classic works studying the growth function and Vapnik-Chervonenin
dimension of binary classifiers (Vapnik & Chervonenkis 197I); see Anthony & Bartlett [999 Part 1) and references. These subdivisions naturally interact with the level sets of the training loss. The loss landscape of neural networks has been studied in a series of works, obtaining results on the connectivity of sublevel sets or the existence of paths descending to global minimum. For instance, under some conditions is known that if the data is linearly separable then all local minima are global minima (Gori & Tesi 1992). In general, it has been observed that local minima are not necessarily global minima (Brady et al. 1989 Sontag & Sussmann 1989) and, depending on the loss function, the number of local minima may grow exponentially in the input dimension even for single neurons (Auer et al. 1995). For networks with piecewise linear activations, spurious local minima are common (Safran & Shamir 2018 Liu 2022). This can be rectified under a catalogue of assumptions. In particular, it can be shown that, given an appropriate level of overparametrization, most differentiable local minima are global minima (Soudry & Hoffer 2018 Karhadkar et al.
2024 Various works have studied how any two parameters can be connected through a continuous path which the energy gap remains bounded
2021 We also note a stream of works Wang et al. 2022 Mishkin et al. 2022 Mishkin & Pilanci| 2023 Ergen & Pilanci 2021 Matena & Raffel, 2 which considers the solution sets in terms of duality and the solution set for convex reformulations of the non-convex optimization problem. Further, the symmetries of neural network parametrization maps have been studied for instance by Rolnick & Kording (2020) and Grigsby et al. (2023 and also in relation to the optimization landscape by Simsek et al. (2021).
Freeman & Bruna 2017 Kuditipudi et. al. 2019 Nguyen et. al.
For classification with piecewise linear functions, not only the parameter space exhibits a (not necessarily polyhedral) subdivision, but also the input space exhibits polyhedral subdivision, which is induced by the decision boundary. To this day, understanding the decision boundary of deep neural networks remains a difficult task both in theory and in practice (Humayun et al. 2023). For a ReLU neural network, it is known that the decision regions and the decision boundary are unbounded whenever the width of every layer is
2

---

Published in Transactions on Machine Learning Research (09/2024)
at most equal to the imput dimension d (Beise et al. 2021 Johnson 2019 Nguyen et al. 2018 and if the
network consists of a single hidden layer of width d + then the decision regions can have no more than
one bounded connected component (Grigsby & Lindsey 2022). The connection between decision boundaries
and tropical geometry is straightforward through subcomplexos of tropical hypersurfaces (Zhang et al. 2018]]
Charisopoulos & Maragos, 2018 Alfarra et al.
2023) Notably, this subcomplex is a familiar object in real tropical geometry as the bignod-)positive positiive tropicalization of hypersurface over the
2023 Piwek et. al.
field of real or complex Puiseux series (Virol 2006
Speyer & Williams 2005 Brandeebuugg et al. (023). This
connection to real tropical geometry goes beyond decision boundaries: In parameter space we observe that
the set of solutions of classification task can be formulated as a tropical semialgebraic set, class of sets
which has only recently started to enjoy systematic considerations AAlamigeon et al. 2020 Jell et al. 2020
Contributions. In this article, consider the combinatorics of continuous piecewise linear classifiers. We consider a finite data set D C Rd and classifiers defined as signs of functions g h: IRd + R, where y and h are convex and continuous piecewise linear functions with at most n and m linear pieces, respectively. We initiate the study of two different subdivisions of the paramcter space (dd, n, m) of such classifiers: The first is a subdivision of the parameter space into semialgebraic sets where the combinatorial type of the decision boundary is fixed. The second is polyhedral subdivision, called the classification fan. This obtained from the activation fan, polyhedral fan which captures the activation patterns of the tropical rational function. The different possible dichotomies of the data and the sublevel sets of the 0/1-loss function arise as subfans of the classification fan. We describe the fan i) geometrically, as the normal fan of the activation polylope, and ii) combinatorially through a list of propertics of bipartite graphs associated with the activation patterns in analogy to covectorr axioms of oriented matroids. From this we can show that in the separable case the sublevel sets of the 0/1-loss function are connected for linear classifiers, but are disconnected for general piecewise lincar classifiers. Furthermore, we show that the parameter space of a fixed architecture of ReLU neural networks is a semialgebraic set of bounded degree inside the parameter space of tropical rational functions, and thus intersects both subdivisions of (d, n, m) nontrivially.
This article aims to address multiple audicnces: A data-driven audience with a background in machine learning interested in understanding the underlying combinatorics, and a (discrete) geometric audience with an interest towards applications concerning neural networks. We try to accommodate for the different backgrounds throughout the exposition.
Overview. In Section 2 we begin by recalling known results and concepts describing linear classification. We introduce three different points of view to study natural subdivisions in parameter space: As chambers of a hyperplane arrangement HD (i.e., connected components of the complement of Hp), as cones in the normal fan Ep of a polytope Pp. and as maximal covectors of realizable oriented matroid. The subscquent sections are devoted to generalizing this theory classification by continuous piecewise linear functions. In Section 3 we describe the connection of continuous piecewise linear functions to ReLU neural networks and tropical geometry. While in the lincar case the decision boundary is hyperplane, in Section 4 the decision boundary is piecewise linear. In [Section 5) we introduce the activation polytope and the activation fan, a polytope and its normal fan which are in analogy to the fan En and Pp from the linear case. Each cone is labeled by an activation pattern, and we relate the set of activation patterns to sets of covectors of oriented and tropical oriented matroids. In |Section 6 we consider subdivisions of the paramcter space of continuous piecewise linear functions: We first introduce the classification fan (Section 6.1). Afterwards (Section 6.2) we describe an arrangement of indecision surfaces, which is the natural analog of the hyperplane arrangement HD, and the cells of this arrangement are compatible with the activation and classification fan. We show that the sublevel sets of the 0/1-loss can be viewed as subfans of the classification fan, resulting in a study of the perfect classification fan Section 6.3) and the (sub)-level sets of the 0/1-loss function (Section 6.4).
2 Linear Classifiers
We motivate our studies from classical results about linear classification problems. Many results presented in this section have been rediscovered in different mathematical communities, using the language of classifiers,
3

---

Published in Transactions on Machine Learning Research (09/2024)
hyperplane arrangements realizable oriented matroids. These different viewpoints serve as an inspiration for what follows, in the upcoming sections we will generalize this to theory for classifiers defined by neural networks with ReLU activation and tropical rational functions.
Let == (pIp..... {PI..... PM] C Rd be a finite set data points. A linear binary classifier or simple perceptron is = sgn((s, x) + a), defined by taking the sign of linear function with
function fo: Rd - (-,, fo(x) parameters (), where a E I, IR The parameter space is O(d) {0 (a, s)
R, s E IE3 = Rd+1 The function fo defines affine hyperplane which separates the data points into three classes, E fp() > 0), {p D (p( < 0} and {p D fe(p) 0}. The last of these sets is empty for generic choices of parameters. In this case, fo induces dichotomy C E {-,+) via C; = fo(P:).
We now discuss three different points of view on these dichotomies. We first state the equivalent viewpoints, and define the terms in this statement afterwards.
Theorem 2.1. Let D c Rd be a finite data set. Then
(i) the hyperplane arrangement HD = Uped(1. (1, p)) subdivides the parameter space ((d) into regions according to the represented dichotomies,
(ii) HD induces the normal Jan of the zonotope Po=Epes conv(O, (b)),
(ii)) the dicholomies are the marimal coveclors of realizable oriented malroid.
(t)) = as hyperplane through the origin. A linear hyperplane arrungement H Rd++1 is collection of hyperplanes through the origin, and it subdivides the ambient space into chambers, which are the connected components of Rd+1 H and are open polyhedral cones.
For p E Fdd we consider (1. {(a.s) ((d) (()), (b))
We recall few more definitions from polyhedral geometry. A polyhedron is the intersection of finitely many closed halfspaccs and a polytope a bounded polyhedron. Equivalently, a polytope P S Rd is the conver hull of finitely many points U1,..., Un E RRd ZZiegler 1995 Theorem 1.1), Le. P = conv(vi,..... Un), where
conv(ui......................... vn) s1i + An |4 e + An =1}.
A hyperplane supports P Pff it bounds closed halfspace containing P, and any intersection of P with such supporting hyperplane yields face F of P. A face is a proper face if F ç P and inchusion-maximal proper faces are referred to as facets. Note that also the emply set is a face of P and by convention dim(0) = -1.
The Minkowski sum of polytopes PL..... Ps C Rd is
Pi + PE {aı + I& xz P. for i [k]]
and zonotope is a Minkowski sum of line segments.
A pobyhedral come C C Rd is polyhedron such that Au + n E C for every u.v E C and X, E R20- Equivalently, it is the conical hull of finitely many vectors uI...., Un E Rd Ziegler 1995
Theorem 1.3), i.e.
C = cone(ui,..... = {/r + +... + Mn .i...., pn 2 0}
The lineality space of is the lincar space ((C) -C(-C) and a cone pointed if its lincality space is trivial. The ryys of C its 1-dimensional faces. A polyhedral fan Σ Id is a finite family of nonempty polyhedral cones such that every nonempty face of a cone in Σ is also cone in E, and the intersection of any two cones in Σ is a face of both. An mncusionnmaximall cone of Σ is called a marimal cone. The fan Σ is complete if Uces C' = Id and it is pure if all inclusion-maximall cones have the same dimension.
Any hyperplane arrangement H. uniquely induces a polyhedral fan Σ, whose maximal cones are the Euclidean closures of the chambers of H. A wall of Σ is an interscction of maximal cones that has codimension 1. In particular, C Rd is a complete (and thus pure) polyhedral fan, then the walls are the cones of dimension d 1, and must always exists.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Given any polytope P Rá+l, the (outer) normal cone of face F of P
(P = {y (R* (2.8) = max (x,y) for all 2 F). ZEP
and the normal fan of P is the collection of normal cones over all faces of P. A polytope is zonotope if and only if its normal fan is induced by a hyperplane arrangement H.
The hyperplane arrangement HD = (1. C ((d) uniquely induces a polyhedral fan Dp (d), which is the normal fan of the zonotope PD EpeD conv((O, (p)). Here, 0 (O....,0) denotes the zero vector in Id++1 We can label each cone of the fan DD by a vector of signs,
C. = (sgn((0, 0, (e.)))..... (0, (p )))) = (fo(P))..... folP.) E (-,0,+)
where we can choose any parameter voctor contained in the relative interior of a. We call C2 the (signed) covector of a. The interiors of maximal cones of Ed consist of parameters which define a strict separation of the data, and are labeled with covectors C {-,+] without zero entries (i.e. dichotomies). On the other hand, lower-dimensional cones are non-strict, i.c. there exist some data points p. which lie on the separating hyperplane, and C, = 0 for these data points. Duality between polytopes and normal fans implies that the vertices of PD are in bijection with the dichotomies that the simple perceptron can compute.
Example 2.2. Consider the 1-dimensional dataset D = {-2, 0, 1,2] C RR_ The parameter space ((1) = 2 contains the hyperplane arrangement HD consisting five hyperplanes H-2 = (1, 2)) H = (1.-1) Ho (1,0)* H2 = (1, H2 = (1, 2) as depicted in Figure The induced polyhedral fan Ep consists of 10 maximal (2-dimensional) cones. Any
neighboring maximal cones in this fan separated by one of the 10 1-dimensional walls. The fan Ep is the normal fan of the polytope Po with vertices
(1)) (2s), (2)). (2) (8), (4), (3). (3).
This polytope can be written as the Minkowski sum of line segments
PD cO.v(O, (12)) + conv(0. (-1)) conv(0, (1)) + conv((, (1)) + conv(O, (2))
and is hence zonotope. The vertices of the zonotope dual to the 2-dimensional cones of Ep. As be seen in Figure each maximal cone of Ep is the normal cone of vertex of Pp. The figure also shows the covectors of these maximal cones a, i.e., the dichotomies fo(-)), fo(), (o(1), fo), fo(2)) for 0 E a in the interior of the cones. The value fe(p) expresses on which side of the hyperplane Hp the corresponding cone lies. For example, let C' be the 1-dimensional cone that the wall between the cone with covector (-,,,t;+ +) and (-, t, +, +)). Then the covector of is (-,0, +, +, +), expressing that C contained in the hyperplane H-1-
H1
Ho
H1a
(+mm)))-))
(+++)t+ti))
H H_
(he (++t-----)) (t.+ttt.-)))
b--)
(a) The hyperplane arrangement Hp.
(b) The zonotope Pp.
Figure Illustration of Example 2.2 The left panel shows the parameter space and its subdivision by the hyperplane arrangement Hp for dataset D consisting of five points in R The right panel shows the polytope Po.
As all covectors in this article are signed covectors, we omit the word "signed" throughout this article. The marsmal covectors (or topes) the covectors with inclusion-maximal support, i.e. the dichotomies.
5

---

Published in Transactions on Machine Learning Research (09/2024)
We will use the notation [M] {l..... ................................
M). [I....,M} ; * 0}. The set of all covectors forms an oriented matroid. More formally, an oriented matroid is a pair ([) where C is collection of elements of {-,0, +3|M (called [signed] covectors) satisfying the following axioms (Goodman et al., 2018 Section 6.2.1):
For two covectors C,D, we define the seporation set
S(C, D) = (i [M] C = -D;
(C (Zero) (O,....,0) EC: (C II) (Symmetry) C' = -CEC:
Ca if C 40, D; otherwise;
(C IIT) (Composition) if C, D E C then (Co D) EC, where (Co (CoD),
IV) (Elimination) if C, D E and e S(C, D) then there exists some Z E C such that Z; and 2, = (C' D), vj E [M] D).
Any collection of covectors that ariscs through hyperplanc arrangement is called a realizable oriented matroid. To each oriented matroid, one can associate dual oriented matroid; for realizable oriented matroids this captures the geometry of the vector configuration given by the normal vectors of the hyperplanes, i.e. the dataset D.
In a classification task, we are typically interested in a strict separation of the data, represented by a dichotomy. We now fix target dichotomy C* E (-,+)M, which divides the data into two sets Df {P: E D C = +), = D Df". The target dichotomy C is a covector in the oriented matroid if and only if there exists a hyperplane separating the data into Df", DD"", i.e. the data is linearly separable according to C Equivalently, by Farkas' Lemma, the covector exists if and only if conv(D4") n conv(DD)) = 0. For any parameter vector 0, the 0/1 111-los--fnctton errc- counts the number of mistakes, i.e.
ertc- (6) Ki E [M] sgn(fo(p.)) = H.
Since the 0/1- 0/1-loss function is constant along chambers of HD and (relative interiors of) cones of Ed, we allow ourselves to write errc- (o) for chambers or cones a E Ep. In the notation of separation sets given above, if D is the covector associated to then eric- (a) = |S(Cr, D)). Note that we choose to interpret data points which lic on the classifying hyperplane to be correctly classified. This technical distinction is irrelevant on interiors of maximal cones and allows us to consider polyhedral fans in [Section 6)
The ttth level set is the polyhedral subfan Σ = fo E Σ ertc-(a) = k}, and the sublevel set is 25 - D/. If the data is linearly separable according to C*,, then the set Eg of parameters with 0 error is a maximal cone of Dp On the other hand, if the data is not linearly separable, then the minimun error on the interior of any maximal cone will be larger than 0 and multiple maximal cones may be minima of the 0/1-loss.
Consider target dichotomy Ci (+, (+.-, +,,-,), +). This asks for separation of the input data points (-2,-1, 0, 1,2] by hyperplane in R² such that (, lie the positive side of the hyperplane, and 1- on the negative side. Similarly, Cj (+, asks for separation into (-2, 1, and {3.,0). Clearly the data is not linearly separable in accordance with either of these target dichotomies. Figure 2 shows the value of emrg- on each cone of Hp for both target dichotomies. Notably, for CT the set of minimizers of errys consists of 5 maximal cones, which pairwise intersect only in the origin, and are thus not connected through walls. On the other hand, for C2 the set of minimizers consists of single cone and is a convex set.
Example 2.3. We continue with Example 2.2
In the remainder of this section, we study the connectivity of sublevel sets for linear classifiers. If one seeks to find set of parameters 0 E ((d) which separates the data, then in practice this can be achieved by minimizing a suitable loss function in the parameter space O(d) using an iterative optimization procedure. In this case the search variable is likely to move from chamber to chamber with transitions going through walls of codimension 1. We would like to understand under which circumstances there exists a path along which the loss function is monotonically decrcasing. The following statement is an adaptation of Björner let al. (1999 Proposition 4.2.3) and we give a proof for completeness. Recall that for covectors C, D, the separation set is S(C, D) = (i E [M] C; -D, + 0).
6

---

Published in Transactions on Machine Learning Research (09/2024)
2
3
2
3
2
2
3
2
3
2
3
3
4
(a) Target dichotomy CT.
(b) Target dichotomy C2-
Figure 2: Illustration of Example 2.3 For given set of input data points the figure shows the polyhedral fan Ep in parameter space along with the value of the 0/1-loss for the two target dichotomies CJ (left) and C2 (right). The sets of minimizers are shown in gray.
Proposition 2.4. Let C, D be two maximal covectors corresponding to marimal cones of ED. Then there ewists a sequence of maximal covectors D = =D= ",,D",. D'....,,'" = =C such that the comes corresponding to D'+1 are connected through a wall of codimension for i i O....,k and S(C, > S(C,
Proof. First note that the maximal cones are labelled by covectors without any zero entry. For two such covectors, we have DoC = D, and for any E S(C, D), axiom [C translates transsaate the existence of a covector Z such that Z, = and Z, C2 for all j E [M] \ D(C We prove the statement by induction on the size of S(C, D)- Since C, D can be assumed not to have zero entries, this is the number of entries in which C and D differ. Suppose S(C, D)- {i}. By [C [ there exists a covector Z such that Z,- = 0 and Zj Cj = Dj for all j E [M] \ {i}. Thus, the chambers corresponding to C, D are separated by the hyperplane (1,p:) and Z corresponds to the wall CnD.
Suppose now the statement holds for any D', D" such that |S(D', D*)l < k and let S(C, = k. Fix D). By [CIV) IV] there exists some covector Z such that Z, 0 and Z; = =C, Dj for all j e (M)SS(C,))))
Let A= (ie [M] Z 0} and [ |A| 2 1. Among all such covectors, choose such that is minimal. Thus, there cxists strict subset A' & A and covector Z' such that Z; = 0, Z = C, -- D, for all [M] ((C, D) and Z) / O for all j E [M] \ A'. Recall that Z' corresponds to cone of Σx, which is contained in the hyperplane (L,p,)+ if and only if 2 O. Since Z exists, but no such Z' exists, this implies that the hyperplanes (1. (I.p)..j E A all coincide. In other words, if Z is chosen minimally, then Pj Py for all j.J E A, and hence Z represents wall of Ep. By [C III) we have that also (Z C) and (Z o D) are covectors. Note that (ZoC), Zj- (Zo D)5 # 0 for all j E \4, and ((ZoC), Z, (Z+ = (+,0,- for all i E A. Thus, (Z. C), (Z o D) correspond to adjacent maximal chambers separated by (1, A, and the separating wall corresponds to Z. Since S(C, Z o C) < k and S(Z o D, D) < k, there exist strictly decreasing sequences from C (Z C) and from Z D to D by induction, which together form a sequence from C to D.
Given a sequence O1...., Tk Ep of maximal cones, we say that the sequence forms path in Ep connecting 01 and og if di, 0e41 intersect in wall of codimension 1 for all [k 1]. Proposition 2.4 allows to characterize the set of local and global minima, and bound the error along paths between two minima.
Theorem 2.5. If the data is linearly separable, then the sublevel sets Σ of the 0/1-loss function are connected through walls of codimension 1 for any k 2 0. Conversely, if the data is not linearly sepurable and the set of minimizers of the 0/1-loss function consists of more than one marimal cone, then it is not connected through codimension 1. Moreover, if o. are distinct maximal cones which are local månima of the 0/1-loss, if m errc- (a) denotes the mindmum error and if a, 2,. oi,00 o' forms a path, then errc- is bounded from above along this path by m + 1.
The first two statements of this theorem are well-established in the literature; the (sub-) level sets of the 0/1-loss function are dual to k-facets heavily studied in computational geometry (Wagner 2008 Section 1.2), and to linear programs with violated constraints (Matoušek] 1995). In [Section 6 we will make related statements for the case of continuous piecewise lincar classifiers. Therefore, we give a proof for completeness also for this theorem. Before we prove the statement, we point out that the assertion of the second sentence is not void, and truly necessary, as can be seen from the two target dichotomies given in Example 2.3
7

---

Published in Transactions on Machine Learning Research (09/2024)
Proof. Suppose the data D is linearly separable. Then the set which minimizes errc. is ΣB {C*} and consists of single chamber. In particular, Σg is connected. For any k > 0 and D E x. we have that eIrc. ( S(C*, D) = k, and Proposition impliiss that there is decreasing path from D to C' through walls of codimension 1. Thus, the sublevel sets are connected through walls. For the second statement, suppose that D not separable and the set of minimizers contains at least two chambers. Now, assume for contradiction that it is connected through a wall of codimension 1. That means there are covectors C, D with minimal |S(C", [ - |S(C*, D) and an index i such that the corresponding chambers are separated by the ith hyperplane (1.p:)* / In other words, C₁ -D; and C₁ D, for all j [M], # i. The latter implies (S(C",C) * |S(C", D)], a contradiction. For the last statement, let o = do, 01.............................. 01,0'' 0t+1 be a path in Ep, and let D; be the covector corresponding to o, Then (a,) - S(C, D,). However, since de, 7 intersect in codimension 1, we have that S(D,, Di+1) = 1 and so errz-(a411) S err err (a.) + 1, i.e. the error increases in each cone at most by one. Since (rr) = errō(o)41) this implies that the increase of the error along the path is bounded in terms of the length of the path by ['t1 J. yielding a total bound of m + L+1 on the path.
The bound of m + [ can indeed be attained even as minimum among all paths between a and a'. One example is the 1-dimensional data with coordinates 1, 2, ..., 4k for some k E N and target dichotomy t..nnt t.....t The covectors minimizing the 0/1 llssosss are
times
2k times
& times
and (يح
1: times
3k times
3k times
& times
each making m = k mistakes, and are connected by two distinct paths, each of length l 1+1 1 = 2k + 1, m + -= = 2k. One of the paths contains the covector (+....,+) and the other contains (-,..., -), each
of them making 2k mistakes and thus attaining the bound.
3 Piecewise Linear Classifiers, ReLU Networks, and Tropical Rational Functions
In this section extend the thcoretical framework from linear classifiers to the more general case of classi- fication with continuous piecewise linear functions. An important instance of this are neural networks with piecewise linear activation functions.
Recall that an L-layer foedforward neural network represents a function f Rd. + Rdoa which is obtained as a composition ion f = (L)) up(L) o..l) op(t). For each & E [Z] the preactivation function yl(t;; Rdd-- - Rdd is an affine function ((() ((1)())) w(0)z + cl with weights W()) Rdexd--1 and biases cf [d(dee) of the ehh layer, and din do, dout dr. A ReLU neural network a neural network with ReLU activation function (Rectified Linear Unit) a(0) (11,...,, Ide) = (max(ι1, 0), 0), max(Id, 0)). We denote f(); R. - Rd f(h ((1) o (ptt) o... (1) o (1) and (x) = x. The number L of such compositions is the depth of the network, and the dimension &e E [4] is the width of the geh layer. We do not impose any further restrictions on the weights and biases and so the choices of the depth and width of each layer determine the architect ure of the network. The network consists of all of its activation and preactivation functions.
Any function represented by a ReLU neural network is a continuous piccewise- lincar function. A powerful framework to study piecewise-liness functions is provided by the language of tropical geometry. This is the geometry over the tropical semiring (or max-plus algebra) where we define tropical addition, multiplication, exponentiation and division as
b max(a, b), @ a + b, a ob =. a b= b for a, R.
For vectors x,S E Rd we will write x 2
oos s). A tropical Laurent polynomial is a
function g: _d + IR of the form
9(x)
④
35 = max (a: + (8i, x)::
ie m
i mz]
8

---

Published in Transactions on Machine Learning Research (09/2024)
where == (21,..., zd) is the input variable and a, E R and s; Z [n], are parameters. A tropical signomial follows the same definition, except that we allow s; E R E [n]. For the purposes of this article, this distinction makes no difference, so we use these terms interchangeably. In the same spirit, we define a tropical rational function as a function
f(x) =
a2
:B:
bj ② x
max (a; + (s,,)) rE
max (b, + (t,,x)), JE[m]
JE(m]
which is determined by its parameter vector 0 = (01, (11,5.t...,,1r On: Sn, bı, t1,...,,am bm, fm), where as E R. 8; E Rd for i E [n]] and b; E R, E E IR for j [m]. We denote by (d, n, m) the (n + m) ( 1)-dimensional space of parameters of tropical rational functions with n terms in the numerator and m terms in the denominator, and for a fixed 0 (dd, n, m) we denote by ge @ he the associated tropical rational function. In [Section 6 we show that (dd,n, m) allows a natural polyhedral fan structure in analogy to the fan and hyperplanc arrangement from Seccion 2
A tropical signomial is a convex and continuous piecewise linear function, and vice versa. Every continuous (possibly non-convex) piecewise linear function can be written as the difference of two convex piccowise lincar lunctions, and this difference is a tropical rational function. This establishes the connection to ReLU neural networks.
Theorem 3.1 (Arora et al. 2018 Theorem 2.1 and Zhang al. 2018 Theorem 5.4). A function f: Rd - R is a tropical rational Junclion if and only if ʃ can be represented by feedforward ReLU nelwork. Any tropical rational function Ø h with n terms in the numerator and m terms in the denominator can be represented by a ReLU network with depth at most min([lgg (d + 1)] + 1, max([log,(n)] [log2 + +2).
A similar result has also appeared in work of (Siakkamari et al. (0220). The original formulation of Zhang ct al. (2018) Theorem 5.4) has an additional condition the weights to be integer, however this is merely an artilact of the distinction between tropical Laurent polynomials and signomials. Given the class of tropical rational functions with bounded number of terms in the numerator and denominator, this gives a sufficient condition on the depth of the architecture.
We may also consider the reverse direction: Let ReLU di....,dd d1-1,dz) dz) be the set of piecewise linear functions that can be represented by a fully-connected ReLUU network with dg = d inputs and L layers of sizes di, E N, with dr = 1. Given fixed such architecture, we seek to find n, m N such that for any function f E (eL For lower bounds on n, m. let Kconvex be the maximum number of linear pieces over all convex functions in RdLU dd..,, 1), and Keoneave the maximum number of lincar picces of any concave function. Then necessarily we have n 2 Riconvex and m 2 Keoneave- To obtain upper bounds, we consider a simple decomposi- tion of the functions represented by a ReLU network as differences of convex piecewise linear functions, whose size depends on heonvex and Reoneaaee in each step. For most of the functions in ReLU(d, di,..., d1-11,1 1) the decomposition that we apply is by no means minimal for the individual function. However, this decompo- sition allows us to consider the space of ReLU networks as semialgebraic sets inside nd), We will discuss minimal decompositions at the end of this section.
1) there exists a parameter 0 Odd, n, m) such that f fo-
Theorem 3.2. Let d = do, d ReLU(d, (d, .h....,d,.111. can be represented by a point in O(d, n, m), and there is a semialgebraic subset of ((d, n, m) (described by polynomial inequalities) representing exactly the points in ReLU(d, di.... d 1). This semialgebraic set can be described by polymomial inequalities of degree S LL+1 The n, m can be chosen as n = 2m and (ogz(m) < I 24-1-k I1--1 di.
= - E There exist n.m E N such that each function in
We make the statement of this theorem more precisc. Recall that any vector of parameters 0 E Odd, n, m) defines a tropical rational function ga @ by with at most n monomials in the numerator and at most m monomials in the denominator. Let CPWL RR be the space of all contimuous piecewise linear functions from Rdd to R, and let s: O(d, n, m) - CPWL(RR R) with (0) = 9o Ø hgg Then Theorem 3.2 says that for any d2....d2-1
N there exist values n, m E N such that ReLU di...., dd-1,,1) S (OO n, m)). and there exists a semialgebraic set S C O(d, n, m) such that (S)) = ReLU dr . dr d).
9

---

Published in Transactions on Machine Learning Research (09/2024)
Proof. We show this by induction on L. For L = 1, let W E Rixd, ,c R denote the weights and biases. Then the network represents the function f() = max(Wx + c) This is already convex piecewise linear function (so m + c, c) - is a representation as a difference with n = 2, m 1). We still perform a transformation to illustrate the procedure that we will use for L in the induction step. We write W W+ W_, where W+ E 20 Then one representation of f()) max + c, 0) as difference of convex functions
go - hg = + W_x
for the parameter vector 0 = (c, W+, O, W, W-) -- ( _ (dd,2,), Since W+, W- are nonnegative vectors with disjoint support, we have that contained in the semialgebraic set.
= {(11 02 b1, ti) E 3()+))) 02 = b 0, 82 = 12 s 0,8 2 0, 81,82: = 0 Vi [d],
which is defined by d + 2 linear equations, 2d linear inequalities and d quadratic equations.
Conversely, for any 0 E S, the function (0) is represented by ReLU network as follows. If 0 (a1,81, $1, 0, 82, 0, t1), where 81, 82 are nonnegative and have disjoint support, then defining W = S1 82 E R. glx gives (0) = max(Wx + O), which is represented by the network with layers of sizes do = d, dl = 1, weights W and biases a1-
We proceed by induction L. - I. + 1, and consider ReLU network with L layers. The idea of the induction is as follows. Let W and be the weights and biases of the last layer. Then W W+ W- has canonical decomposition into nonnegative matrices, and the entire function is of the form f(L+1) = max ki=1 W&fa + c, 0), where fi....,fa fu are the tropical rational functions in the Lth layer. Writing f& = Ol hg as a difference of two convex functions, we obtain the decomposition f(l+1) max maxx(Enn (> (W (g)
c 0) = max W gh + W;, hs + W gs + W - (Wr gh + W hn). We now perform this computation in more detail.
(gx- hx)
The first L layers represent collection of dy tropical rational functions, ic. f()) Rdo - Rde,
91 (1) (x) 2 h (x)
(x)
Ids (x) @ n/
(x) (x)
(1) (x)=
C ex EE LE
(x)
Bluk cx thE
9
jet [m
where TR, mA.L. E N. E [dx] and dz R.L. 1ith E R,,,
R.L. , the E Rdb Again, let W = W+ W- with
W+ W_
R. Ixdr ,c E R be the weights and biases of the last layer.
Let
dp Σ
Y Yoonvex
(
9k
(L) (x) + WT (2) (x)),
k=1
d,
Y concave Σ W k Ik
(x) + W (L) (x))
Then f(++11(x) max (Yoonex Yeoncave c) max(X (Yconvex + c, Yconcave) Yeoncave. We now compute
Yeonvex- Y Yeoncave be computed analogously. For any nonnegative number We holds Wa max(a,
10

---

Published in Transactions on Machine Learning Research (09/2024)
max(Waa, Wib), and furthermore max(a, b) + max(c, d) max(a a, d). Thus,
de
convex
W -
1 (x) W (x))
W max m
(a + ( R. R. ; + W
max jElma,L
(1i + (19" (x))
du
WT+ (1s
k.L
+ ( (5S (
k.L -xx,,)) )))
(Wr (15- + (())tx)))
max <E(,).)
max jE(maLL)
de
(t), W (ati + ( st , )) + W (DtiL + ( thiL ,x (x)))
max
ie R
jE ma.z]
In the following, we use the notation
ke [dg]
d [na.el
Je E [m)..
to denote the maximum over all i E [mI,L],..... dde E [ndc. E [m..z)......b E [ma.,.))- Applying again distributivity of tropical multiplication (i.e. that max(a, b) + max(c, d) = max(a + c, + d, b c, df), and bilincarity of (.,.) yields
Σ R=1
W
R.L + ( (the Oi
k.L xx) + W
(b% + ( L. (2)xx)))
max
kEldL] in [ da E [m).2
(1)
du
di
max kEldy, ix E.EnM..L E [
W stilk
k.b is
W t
x
W k. & a
W. (t.c) 1/
and this expression consists of Ih nR I ma.t linear terms. A similar reasoning applies to Yeoncave R. Recall that W is a (1 x dr.) dz.)-matrix, and thus (W di + WW ( R.
In the following, we use the shorthand notations
(L+1) (L+1) GR di min
(L+1)
(L+1)
Ok,m,, to "m.,tii.... + thy mi wt.............................. t).... t.
where is E Mi..del.js E [2 for all k E [dz]. With this, we have expressed f((+1) as a quotient of two tropical polynomials:
[(x+))(x)=
a (L+1) Rikide
+ ( (
(L+1) Ribiidae
x) , aet ti. + ( (1+t)
max kE(d2]
i E m
j.E m.
(1+1) (I+1) bhie ((deede)
max kE[dz]]
ig m j.E (ma
(L+1) (L+1) E R and (L+1) (L+1) for some parameters "kivess Riikei
Rdd where is E [nn, E [ma..]] for any
Skiede Ricc
k [dz]. In other words, we have proven that
dr.
di
d
(((+)(r) (0
n
m
n
m.

---

Published in Transactions on Machine Learning Research (09/2024)
We first discuss the semialgebraic constraints on these parameters. By induction, the parameters of the tropical rational function representation = h kk [dz] are contained in a semialgebraic set 8(4) which can be described by polynomial inequalities of degree at most ++ 1. The above computation
L imposes linear relations between the parameters of f(l+1) in terms of the parameters of f ,k E [dz]. More specifically,
de
dr
$ kj
(L+1)
K.L A Sit
+ W k.
(L+1)
- k=1
(W Oi
k.L
+ W W b + cc,
k=
de
dv
(L+1) Sk,n,, tde.
Wx Sig
L. + W R.L
(L+1) tim
Wi dig
k,L
+ wi b.L
2
k=1
k=1
(I+1)
(L+1)
(L+1)
(L+1)
t k,iidd
2, tig.m,,
Thus the feasible parameters are given as the image of a polynomial map evaluated over a semialgebraic (I+1)
set. In turn, they form a semialgebraic set. Moreover, we have written the variables Skúdedk etc. as a linear
combination of the variables k.L s etc., i.e. as the solution of polynomial of degree in these variables. By induction, cach of the Si " is solution to a system of polynomials of degree L + 1. Substituting these polynomials into a polynomial of degree 1 yields a system of polynomials of degree L ++2 2.
We now discuss the number of monomials in this representation. By induction, we can choose each 7gg.L = 2. and logg(ms..)) < Σk=1 2/-1- =I dr. Therefore
dp Log2(nk.) +
de (m),))
(og((m++1) =
k=1
1-1
L-1
1-1
L-1
< d
2
II i=
di + 1) + dr
>
di)
1=1
k=1
t=
L-1
L-1
L
2 2-11- *
+
dr.
Σ
2
1=k
k=1
i=k
L-1
L
2
di
+
2
d2
km
1m
1-1
1.
L.
Σ
A d2 +
2
2
di
k=1
1=k
k=L
1=
L
Σ 2
24-
di
1=k
and the stated bound follows.
We have just proven ReLU(d, di,..., dr,, 1) C (SS), where S is the semialgebraic set which is implicitly defined through the recursive relations in 2 For the reverse inclusion, let 0 E S. By induction, there exist weights and biases defining a network in ReLU(d, d2) such that a s. ,.L. .t are the parameters of respective tropical rational function representations of some f ((D)ss By definition of there exists solution for the system of linear equations (2) in indeterminates W W. Any such solution gives rise to the weights of the last layer.
R.L.
R.L
Remark 3.3 (S is not unique). The proof of Theorem 3.2 reveals that in the above representation there
are many redundancies which are expressed in the lincar equations defining the semialgebraic set. Morcover, the choice of S is not unique: already for L 1, choosing the representation H' (c. W. 0, O. O, 0) E O(d, 2, 1) yields the semialgebraie set. S' = {(11,81, 02 82, b, t1) d2 by 0,82 t =]. This reflects the fact that
12

---

Published in Transactions on Machine Learning Research (09/2024)
for given continuous piecewise linear function there exist several possible decompositions into difference of convex piecewise linear functions. Either way, since these sets are low-dimensional relative to their ambient space, one may want to project this into a smaller ambient space. However, for any (nontrivial) projection, the degrecs the defining polynomials will in general be higher, and the number of defining incqualities will increase. There no general statement which reasonably bounds the degree and number of the defining polynomials of the projection of a semialgebraic set.
For the number of monomials, we observe that ReLU network with a total of N ReLUs can only represent continuous piecewise linear functions which have at most 2 linear pieces. Several more refined bounds on the number of linear pieces are available depending on the specific network architecture (Montúfar et al. 2014 Montúfar 2017 Serra et al. 2018 Hinz & van de Geer 2020). In particular, [hhang et al. (2018) Theorem 6.3) use similar arguments to the ones used in Theorem 3.2 to obtain a bound on the number of linear regions. We obtain a weaker bound since we focus on the (in)quualssies defining the semialgebraic set. From such bounds on the number of linear regions one can directly obtain upper bounds on the minimal possible n. and by considering decompasitions of continuous piecewise linear functions as differences of
convex piecewise linear functions, such as those discussed by |Kripfgazz & Schulze (1987) and [Schüüter &
Darup (020). Certain upper bounds for functions in two variables have been presented by Tran & Wang 20023 and for general piecewise linear functions by Hertrich et al. 2021 Proposition 4.3).
We close this section by noting that the statement of Theorem 3.2 not exclusive to ReLU networks, but holds generally for networks with piecewise linear activation functions, for appropriate choices of n, m and degree bounds the polynomials. For example, the proof can be adapted to yield a represcntation of fixed network archilectures with maxout units into semialgebraic set in the parameter space of tropical rational functions for some bounded n and m.
4 Decision Boundaries
In Sectionn2 2 we considered linear classifiers which separate the data into two parts by a hyperplane. In this section, we consider the analogous separating sot for contimuous piccewise-linear functions. This leads to a description of the decision boundary through the lens of (real and positive) tropical geometry.
Consider the tropical rational function g Ø h Rd + R, where
(xx = max(a, + (s,, x)) and h(x) = max (b, + (ty.x)) jE(m
sE(n]
are tropical signomials with paramcters a; R. S; Rd. E [m] and b, R, tj CR,, E [m]. The decision boundary of g Ø h is
= (xER)))()
= 0) E Rdd max + (si, x)) = max (by + (ty, (),,,))) kE[n]
jem m
The decision boundary h) splits the input space into two open parts B+ Bt, where either the numerator or the denominator h attains a higher value. A classifier g 2 then separates the data D into two classes D+ S B and D_ 5-
It was noted (Zhang al. 2018 Proposition 6.1) that B(g@ h) is subcomplex of the tropical hypersurface T(g @ h). This observation was also used in the work of Alfarra al. (2023). We will now make this statement more precise. Consider the tropical signomial f(x) = g(x) ④ h(x) = max,e[N] + (s,,))), where N n m, an+j bj. Sntj tj for j E [m]. Each such tropical polynomial divides the input space Rd into N polyhedral regions, one for each it E [N], which are of the form
R₂- = ffx E Rd a. + (8),x) = max (a: + (s., iE[N]
Depending on the coefficients, some these regions might be empty. The collection of regions forms a polyhedral subdivision of the input space Rd.. The collection of (d 1)-dimensional cells this polyhedral subdivision is the tropical typersurface
T() (x 1 the maximum in f(x) is attained in at least two terms).
13

---

Published in Transactions on Machine Learning Research (09/2024)
R3
R1
R3
R4
R4
T()
R1
R2
R1
R2
R2
Ra
S
54
(f)
S4
$2
$2
Figure 3: The three combinatorial types of generic hypersurfaces defined by tropical polynomials in two vari- ables and 4 terms (top), and the dual regular subdivisions of their Newton polygons (bottom), described in Example 4.1
The tropical hypersurface is dual to a regular subdivision of the Newlon polytope Newt(f) = conv(s),...... SN) C Red which can be obtained in the following way: Consider the lifted polytope conv(( (i ...., .,. AN )) c Rd+1, Any facet of the lifted polytope has a unique outer normal vector (up to positive scaling). The upper hull the collection of facets whose normal vector has positive entry in the first coordinate. The projection of the upper hull onto the remaining coordinates forms subdivision of Newt(f), called regular subdivision. The region R.- is the set of linear functionals maximizing the vertex (#:: of the lifted Newton polytope, and the intersection R,- n Rz- is contained in T()) if and only if the pair Sy- form an edge in the regular subdivision. For more detailed expositions on this duality we refer the reader to Maclagan & Sturmfels (0055 Chapter 3.1) and Joswig (2021 Chapter 1.2).
Example 4.1 (Tropical Hypersurfaces and Newton Polytopes). Let N = 4 and d 2, i.e. f(x) ai O x x @ 02 O x O ag @ x @ ① x 0 where a; R and s; R?? The Newton polytope Newt(f) is the convex hull of the 4 points $1, $2, S3, B4 in IR?, and is thus either a triangle or square (except for degenerate cases where Newt(f) is not full dimensional). Choosing generic values 01, a2 03. E IR, we obtain three possible regular subdivisions and their respective dual complexes, as shown in Figure 3
We now extend this duality for understanding tropical rational functions by assigning signs to each region. We have seen that T(g@h) divides the input space into n+m regions. We call the full-dimensional regions R, with i E [n], L.e. those that correspond to terms of the posilêve regions. The regions R,, E [m]] corresponding to terms of h are the negative regions. This terminology stems from the fact that g(p) Ø h(p) 2 0 if p lies in a positive region, and g()) @ h(p) s O if p lies in negative region. These regions are dual to vertices in the regular subdivision of the Newton polytope Newt(g + h), i.c. to those vertices which lie in the upper hull of the lifted Newton polytope. We equip each such vertex with the corresponding sign, obtaining posilive and negative vertices of the regular subdivision. An edge of the subdivision a sign-mired edge if one of its vertices is positive and the other one is negative. Such sign-mixed edge is dual to the intersection of positive and negative region, and thus g ② h 0 along this intersection. We summarize this construction as follows:
Theorem 4.2. The decision boundory Big Ø h) is the (d- (d-1))dimensionul subcomplex of T(g @ h) whose mazimal regions are dual to sin-mizeed edges of the subdivision of the Newton polytope Newt(g (g h).
This statement is alrcady implicit in carlier literature on real and positive tropicalization. If h are tropical polynomials, then the decision boundary Big 2 h) is the tropicalization of the intersection of the positive or- thant with a (family of) hypersurface(s). Such a hypersurface is defined through any polynomial G(x) - such that GG) = g, trop(H) and both G and H have only nonnegative coefficients. These connections between decision boundaries and (tropical) algebraic geometry are, as far as we know, widely unexplored. For more details tropical positivity refer the reader to Speyer & Williams 2005 Viro (2006) and
Gxx) (x)
14

---

Published in Transactions on Machine Learning Research (09/2024)
+
+
Figure 4: All combinatorial possibilities of positive and negative regions of T(g @ h) for = m = 2, up to switching "+" and s-", as explained in Example 4.3, The decision boundary B(g 2 h)) (top) and its dual complex (bottom) are highlighted in blue. The right most column shows configurations of three positive regions and one negative region (n = 3, m= 1). The decision boundary highlighted in orange.
2023 For an introduction to tropicalization of polynomials and hypersurfaces, recommend the texts of Joswig 20021 Chapter 2) and Maclagan & Sturmfels (2055 Chapter 3.1).
Brandenburg et al.
Example 4.3 (Decision Boundaries and Sign-Mixed Subcomplexes). Let d = 2 and let g= + x @02 O 20 and f = by 0 Φ62 @x The polynomial g @ h consists of N = 4 terms, as the one in Example 4.1 Figure 4 shows all nondegenerate combinatorial possibilities of positive and negative regions, where a "4" indicates a region or vertex corresponding to a term of g, and a "_" indicates a region or vertex corresponding a term of h. The decision boundary the 1-dimensional subcomplex of TTg@h) consisting of line segments and rays incident to positive and a negative region. Figure 4 also shows a configuration where g consists of = 3 terms and h is monomial (m = 1). There, the decision boundary separates the space into bounded (negative) cell and its (positive) complement.
The main focus of this article is the parameter space O(d, n, m) of tropical rational functions with n terms in the numerator and m terms in the denominator. It is thus natural to ask about the geometry of the set of parameters such that the decision boundary has a fixed combinatorial type. The decision boundary is a polyhedral complex. The combinatorial type of a polyhedral complex captures the combinatorics of the interscctions and inclusions of its faces. Formally, is defined the isomorphism class of the partially ordered of faces, ordered by inclusion.
By the discussion above, the decision boundary is dual to subcomplex of Newton polytope, the set of parameters which gives fixed combinatorial lype of decision boundaries is subdivided into multiple (but fimitely many) smaller sets, one for each combinatorial type of regular subdivisions of Newton polytopes defined by n + m monomials. Each such smaller set is itself determined by the combinatorial type of the lifted Newton polytope. The set of parameters such that the Newton polytope has a fixed combinatorial type is the realization space of the lifted polytope, and the space of parameters O(d, n, m) can be partitioned into these realization spaces. It known that such a realization space is a semialgebraic set, i.e. finite union and intersection of solution sets to polynomial inequalitics. We thus obtain the following result.
Theorem 4.4. The parameter space @ n. n) of tropical rational functions is partitioned into semialgebraic sets, one for each combinatorial type of regular subdicisions of + points in d-dimennional input space.
However, the Universality Theorem of Realization Spaces certifies that realization spaces of polytopes be arbitrarily complicated (Richter-Gebert & Ziegler 1995). Morcover, observe that this partition into semialgebraic sets is completely independent of the data set D. In Section 5) we will introduce the aclivation fan, which is a polyhedral fan subdividing the parameter space, and its geometry heavily depends on the geometry of the data. The aforementioned semialgebraic sets thus may or may not intersect cones in the activation fan. In other words, these are distinct partitions of the parameter space whose structures are incompatible with another.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Example 4.5 (Realization Space of Decision Boundaries). We consider 1-dimensional input space d= 1, where g(x) max + (), 42 + (82,z)) and h(x) = b + xt1, We describe the semialgebraic nature of the realization space in (d + ()(++))) = 6-dimensional parameter space (11, 2, 1). The Newton polytope Newt(g + h) is a line segment which is the convex hull of 3 points $1, S2, t2 E R (dual to two positive regions and one negative region), and the decision boundary consists of at most 2 distinct points. We want to characterize the set of parameters such that Big 2 h) consists of two distinct points. In terms of the Newton polytope, this is the case when t int(conv(s), s2)) and (b) lies above the lifted line segment conv(($; ) (22)) )), that Newt(g @ h) is subdivided into the two line segments conv(s1, tt) and conv(t1, 82). A subdivision into two line segments arises if and only if there exists a 0 < À < 1 such that As + (1 - A)8₂ t,, and da + (1 A)az < b1 The semialgebraic set in parameter space is the coordinate projection of the set of vectors (01, 81, d2 $2, 43, S3, A) E IR? satisfying the above inequalities, by projecting away the X-coordinate. Concretely, the coordinate projection is the semialgebraic set
S S=((a,, S1, 02 S2 b,, (1) 8
t1 < S2, (t: - 52) (2 - -t1) < (i(s) -22) Ut(a1, 02 82, b t1) $2 t < 02 81) a (s -1) < bs -8))-
which is defined by unions and intersections of linear and quadratic polynomial inequalities. The parameters defining decision boundaries consisting of a single point are O 2, \
5 Activation Fan and Activation Polytope
In Section 2 we have seen that the parameter space of linear classifiers allows for a subdivision induced by a hyperplane arrangement. This subdivision is the normal fan of a zonotope, and its cells correspond to maximal covectors of an oriented matroid. In this section we make a first step towards a generalization of this theory for continuous piccewise lincar functions. Following the idcas from [Section 4 we consider subdivisions of the input space into n m regions, which are induced by tropical hypersurfaces. This allows to introduce the activation fan, polyhedral fan which is the normal fan of the activation polytope. As an analog to covectors, we label the cones in this fan by bipartite graphs, called activation patterns. In the later Section 6 we will then assign signs to obtain the full analog for tropical rational functions.
In Section 5.1 we introduce the concepts and investigate the general combinatorial and geometric structure of these objects. In [Section 5.2] we relate these to known concepts, namely oriented matroids and tropical oriented matroids.
5.1 General Structure
Let D C Ied be a fixed finite data set. In Section 3 (d, n, m) we can associate tropical rational function go @ he, which induces a classification of the data set, Our goal is to understand the underlying combinatorics of this separation in parameter space. In |Section 4 we have seen that the combinatorics of this separation is determined by the combinatorics of the tropical hypersurface T(g ④ h). Here, g é h is tropical signomial with + m terms and T(g ( h) separates the data into n + m classes. We devote this section to the study of the combinatorics of tropical signomials with N = n + m terms within their parameter space. Later, in Section 6 we will extend these considerations to tropical rational functions.
we have seen that to any vector of parameters E
Consider fixed finite data set D (Pi (Pi,..., .Mr} C Rdd and denote a tropical signomial f: Rd R, f(x) maxxe[N] (a, + (s,,)), which is uniquely defined by its parameter vector 0 = (a1, Si,..., SN). We denote by O(d, N) the (d + 1)-dimensional parameter space of tropical signomials with N terms (in numerator and denominator combined). Given a vector of parameters O E Od,,N) we write fo for the corresponding function. For a graph G= (VV(),, E(G)) and a node v E V(G) we denote the neighborhood of by N1; G) = {w E V(G) vw E E(G)). For two graphs G = (V, G" = EV, on the same set of nodes, we write GU for the graph with nodes V and edges EUE'
Definition 5.1 (Activation Pattern of Tropical Signomial). The data point pj E D activates the (i*)th term of the tropical signomial f(x) max (a, + (s,, if f(p)) ar + (s,,x) x). The activation pattern of (J, D) is the bipartite graph G (V( F((G) on nodes V(G) D [N] with edges F(G) (pi
16

---

Published in Transactions on Machine Learning Research (09/2024)
i1
P
f1
13
24
02
01
P2
Pi
Pi
P2
8/2
14
P2
2/
82
11
13
i4
i1
i2
13
P2
14
11
i4 83
1/
1/2
11
(a) Activation pattern.
(b) Subdivisions in input space (top) and dual Newton polytopes (bottom).
Figure 5: The activation pattern G from Example 5.2 partitions T(a) of the input space for three different
values of 0 CD(G) and dual subdivisions of Newton polytopes Newt(fa).
P activates the 8th term of f). The degree of vertex p, deg(p) = NN (p; G)], is the number of terms that it activales. The aclivalion cone I a fixed bipartite graph G = (D LI N B(G)) is the polyhedral cone
Cr(G) cl ({0- (01, Si, a......,,,,,,,... SN) E (),,N) G is the activation pattern of (fa, D)})
in the parameter space of tropical signomials, where cl denotes the Euclidean closure.
In the notation of Section 4. the point activates the (i*) term if and only if p R,,, where R2-- is a maximal region of the subdivision of Ied induced by the tropical hypersurface T()). The activation cone CD( is thus the space of tropical signomials where each data point p lies in a fixed set of regions, namely in regions indexed by Np: G). The activation pattern may be thought of as a multivalued generalization of a covector of an oriented matroid, and the corresponding activation cone serves as an analog of a chamber in the hyperplane arrangement from [Section 2.
Example 5.2 (Activation Patterns). Let D {P1. P2} C R2 with Pı (0, P2 (L.0), let N and G be the activation pattern nodes V(G) DU f i2, i3, i4) with edges E(G) = (P1, 61) (P1, 12), (P2,1s) is)) This graph represents the set of tropical signomials in d= 2 variables with N = terms such that the point P1 lies in the intersection of the regions R. R2 corresponding to the first and second term, and the point Р2 lies in the region R,g- Figure 5) shows some tropical hypersurfaces T(fe) and dual subdivisions of Newt(fe) for f E CD(G). More explicitly, the parameter of the leftmost tropical hypersurface can be chosen as = (0, 1,,1, 0,0,0, 1., 0.5, -2, 2,... Recall from Theorem 4.4 that the parameter space is subdivided into semialgebraic sets, one for each combinatorial type of subdivisions of Newt(fa). This example also shows that the cone CD(G) intersects all such semialgebraic sets nontrivially.
Note that, depending on the geometry of the data D, the cone CD(G) may be emply. On the other hand, every bipartite graph H (Du [N), E(H)) defines polyhedral cone
(D(H)) (0- (a1, $1 ... . SN) sN) (d, at + (s p) S a+- + (sr p) for all p D
and all i [N] such that pi E E(H)).
If CD(HH) # then (D) = Cp(G), where G the smallest activation pattern such that E(H) C E(G). We write G H7 for this smallest graph.
Remark 5.3 (Checking existence through linear programs). In theory it may difficult to determine whether (H) is empty or not. In practice, this can be done via a linear program, where Cp(H) the set of feasible solutions. This program is defined by [E - 1) linear inequalities (where |E(H) S NJD)), and (H) + 0 if and only if the corresponding linear program has at least. solution. The activation
17

---

Published in Transactions on Machine Learning Research (09/2024)
pattern H can be computed by checking the validity of all N(N - 1)[PI inequalities on CD(H). More specifically, H contains the edge i" if aj + (s., S oz- (8+ p) is valid inequality for all parameters in (D(H) for all i E [N].
Definition 5.4 (Activation Fan). The activation fan Ed(N) of finite data set D C R and tropical signomials with N monomials is the set of all activation cones in the parameter space (),
Proposition 5.5. The aclivalion Jan ED(N) is complete polyhedral fan, i.c. collection of polyhedral comes such that the intersection of any two cones is face of both, and that their union covers the entire ambient space.
Proof. Let. G,G' be activation patterns. CD(G) is indeed a polyhedral cone, as it is defined by linear inequalities, and for every E CD(G), À E R one has A0 E CD(G). Its faces are the form Cp(G") where (G) E(G"), and any such activation pattern defines a face of Cp CD(G). For any pair of activation patterns holds CD(G) n CD()") CD(GUG) is face both of Cp(G) and Cp(G"). Therefore, the collection of all activation cones forms polyhedral fan. Every vector of parameters = (a1, 81,.... ,a,, .N,88) Od, N) gives rise to tropical signomial fx(), and 0 CD(H), where H is the activation pattern of (fe, D). Thus, the activation fan is complete.
The activation fan serves as a generalization of the polyhedral fan Ep from Section 2 which is induced by the hyperplane arrangement HD. The fan Dp is the normal fan of a zonotope, iee the Minkowski sum of 1-dimensional simplices, one for each data point in D. We now show an analogous statement for the activation fan Eo(N).
Definition 5.6 (Activation Polytope). point p e Rd defines simplex
(p) conv ((1, p. 0,,00,,,...000000.0.,0 0,0), (0, 0, p...., 0,0), ................))))). C ((d,)
of dimension (N 1), where (0, (0.....0) 0) E Rd For finite data set D C Rd, the activation polytope (D(N) is the Minkowski sum PD = Eped App).
Theorem 5.7. The activation fan (NV the normal fon of the activation polytope Po(N).
Proof. Since (N) is a Minkowski sum, the normal fan of Po(N) is the common refinement of the normal fans of its Minkowski summands A(p), D. For such a simplex, the vertices are of the form
0 I,P, 0....-0) ) for irie [N]. (i (i-1) (N-1)d+1)
We describe the normal cone of a vertex V,- for some it [N]. Since A(p) is a simplex, the normal cone is a simplicial cone, whose facets are orthogonal the directions of edges v. for all E [N] {i*). The normal cone is thus
NP_ 10 O(d, V0, - v.) 2 Vie [N]]
[(11,81,........ SN) a, (s) p) 2 a: + (s., Vie [
[(01,)))......... aN SN)
max (a: + (s.. p)) = a,. (s),
iE[N]
Any maximal cone of the normal fan of (N(N) is of the form C = NPED NP No where (p( [N) with possible repetitions. In other words,
C ((11, si.... ,N, max (a; (s.. (p)(p) + (St+()))P) VDED),
iE[N]
so (D) where the activation pattern is the bipartite graph (D [N], E(G)) with edge set E(G)- (pi p (p) E [N]. NP We have shown that the maximal cones of the normal fan of (N) are contained in ED(N). Since the normal fan is complete fan, this finishes the proof. 7
18

---

Published in Transactions on Machine Learning Research (09/2024)
Remark 5.8 (Linear Classifiers are Special Case). I = then tropical hypersurface is defined through the linear equation (81 82, x - (a1 02) = 0. The activation polytope Pp(2) is then equivalent to the zonotope PD- = DpeD conv((0, 0). ()) C Rd+1 from Section 2. however, this equivalence is not obvious. To make this precise, consider the activation polytope PD(2) DED conv((1, p. 0, 0), (0, S (2((+1) The activation polytope is contained in the (d+ -)ddmennio affine subspace with e; e++d+1 for E [d+1]. Projecting (p(2) onto the first d+1 coordinates induces an isomorphism between the zonotope PD and the activation polytope (2()). An activation pattern is a bipartite graph with nodes 2).. The isomorphism identifies the labeling {1 with the labeling {+--}, recovering the covectors of the oriented matroid.
Proposition 5.9. The dimension of the activation polytope PD(N) C ((d, N) is (N - ()ddim(af((() + +1,, where (D) denotes the smallest offine subspace containing D.
Proof. Recall that dim (P)))) = dim(aff (Po())) Morcover, the lincar space which is parallel to (NN)) is orthogonal to the lineality space L of the normal fan of PD(N), i.e. the largest linear space which contained in each cone of the fan. By construction, the normal fan of Pb(N) is a com- mon refinement of the normal fans of the simplices A(p). E D, and the lineality space & of the normal fan of PD(N) is the intersection of the lineality spaces of the simplices. We thus characterize the lineality space L(p) of the normal fan of simplex (p)) for fixed p E 'D. Again, [(p) is orthog- onal to the linear space parallel to aff() Since A(p) is a (N 1)-dimensional simplex, we have dim(C(p)) = 1)N (A()))(A(())) = (d + 1)N (N - 1) = dN + 1. By construction, ((p) contains
(c;, c....,.....0 C;); i [d 1}, and (V(p), 0, ...0., 0), (0, (p)). ................................ 0),
(0, ...0,...VI...... V(().
where V(p) = (1,p)- We denote E =
e,) i E [d + 1}}) and W(P) = span((V(p)) (p). o...... .).... (0,0, ........ ...O...,V((())) The dimension of the total linear span dim(E) + dim(W (p)) - dim(F En (()))) = (d+1)+ No d == .d 1, so the span equals C(p). The intersection of all these lineality spaces L(p).p P D is spanned by E' and W = -NEED (()) span((V, 0, .......,..... O). (0, o..., V)), where V = - V(p) = span((1} x + We thus obtain for the dimension of the lineality space
dim(C) = dim [b) + - W)
= (d + 1) + (d - dim(aff(D)) - (d dim(af((D))
(d 1) + (N () - (D))
This yields (i)(P = N(d + 1) - dim = (N- 1)ddi + 1).
From the above proof we immediately obtain the following dual result.
Proposition 5.10. The lineality space of the activation fan En(N) has dimension (N+1)+(N-I)(d-afff) and is generated by
V). where e; E Id++1 denotes standard basis vector, and where V = sppan((1 (1fxxf(D))) cd++ a (d-) dimensional vector space.
ce.. e,), i [d+ and .........0.. (V. (V.....))
In the linear case, the cells of the hyperplane arrangement in parameter space are labelled by covectors of the oriented matroid, and the activation patterns naturally generalize covectors. The maximal chambers are labelled by covectors without entries, corresponding to the fact that each data point in input space lies on precisely one side of the dual hyperplane. We now show that the analog holds for activation patterns: Given parameter vector in the interior of a maximal cone in the activation fan, each data point lies in precisely one region of the dual tropical hypersurface in input space.
As in the proof of Theorem 5.7 let vi(p) = (O....,,0, 0, p,1,p,0,... p,O....... 0) denote the 7th vertex of A(p). Since PD(N) is a Minkowski sum, every face (N) can be written uniquely as a sum of faces of A(p), for P E D. In the following, we write F Eped F(P) for a face of PD(N), where F(p) is a face of A(p).
Proposition 5.11. Let Cp(G) E Ep(N), and let F E Po(N) be the dual face. Then F Eped conv (v:(p) i G)) and N(p: (i E [N] ve(p) is verter of F(p)).
19

---

Published in Transactions on Machine Learning Research (09/2024)
Proof. We have that CD(G) PED NFip), where NF(p) is the normal cone of the face F(p). Given
any 0 E NF(P) with associated tropical signomial So, the activated terms of fo(p) are {i E [N] v.(p) is vertex of F. Thus, for any activation cone CD(H)) S Nr(p) with activation pattern H we have that N(p; H) = fi e [M] vs(p) is a vertex of F(P)). Given A E Cp(G) = APEDD NF(p) the edges of the activation pattern are thus Upe l [N], (p)) a verlex F(D)). Dually, this yields
F EpED conv (v:(p) E G)):)).
Since every face has at least one vertex and every vertex exactly the Minkowski sum of vertices we get the following.
Corollary 5.12. IfG is an activation pattern, then deg p) 2 for all p E D. The activation cone CD(G) is of masimal dimension if and only if deg(p) = for all p D.
In the linear classification case, we have seen that any largel dichotomy separates the data into two sets DJ", DD and this dichotomy exists a maximal covector of the oriented matroid if and only if conv n conv(DD = 0. Note that under the identification of covector with a bipartite graph G on nodes Duf+,-b we have that D4" is the set of points which are neighbors of + in G, i.c. D = NN: G), and similarly D = (-- G). We now discuss necessary conditions on the geometry of the data. in the more general case, iee such that an activation pattern or cone may exist in the activation fan.
Theorem 5.13. Let CD(G) E Ed(N) be nonempty cone in the activation fan. Then the data set DD C Rd satisfies the following necessary conditions:
(1) (Converity for maximal cones) If CD(G) is maximal, then for every distinct i,j E [N] one has (N G)) n (onv(N(j;G) = 0.
(di) (Converity for arbilrary cones) Let i E [N] be region and Pi,...,Pr. N(;; G). Then for every DED one has: p conv(pI,......id = P N(;GG)..
(iii) (Regularity) There exists a subdivision of the irepul space into maximal regions Ri...., Ry such that N(i G) S Ri for each i E [N], and a dual subdivision of N points which regular.
Proof. For cones of arbitrary dimension, iii follows from convexity of the regions of T(fo) for any 0 E CD( By Corollary 5.12 the activation patterns of maximal cones satisfy (i; (); 0 for all distinct i, e
[N]. This, together with iii implics ii) Finally, since CD(G) is nonempty, there exists a tropical signomial
So such that G is the activation pattern, and the tropical hypersurface T(Je) induces the subdivision. By Section 4 this subdivision is dual to a regular subdivion of the Newton polytope Newt
We have given necessary conditions for an activation cone to be nonempty by considering the geometry of the data. Recall that the set of covectors of oriented matroids obey the axiom system [C I)
C IV) on
page 6 The following result mimics a system in this spirit to describe the set of activation patterns.
Theorem 5.14. Let G be the set of activation patterns of ED(N). Then G satisfies the following properties.
(A I) (Complete Graph) KNDDEG
(A II) (Symmetry) G GEG = any graph isomorphic to G which arises through relabeling of the nodes i [N] contained in G.
(A III) (Composition) If G. H E then (G a H) E G, where
Gop = G)p:G)
if N(p: H() = 0.
otherwise.
(A IV) (Elimination) If G, H E 9 and p E D then there exists a graph F E 9 with N F) G(p:G) N(p; H).
20

---

Published in Transactions on Machine Learning Research (09/2024)
(A V) (Boundory) Let S S [N] be fized and G be the bipartite graph with edges E(G) {pi p D S). Then G E G.
(A VI) (Comparabilily) For any point p E [D] the comparabilily graph CG., of any two palterns G, H E G is acyclic.
For pair G, H of activation patterns, we consider the comparability graph CG of data point p D. This graph has nodes [N] and contains directed and undirected edges. We draw an edge between j and k if E (p; G) and & E H)) This edge is undirected if j, E N(p, N Hf) and j + k otherwise.
of covectors of oriented matroids. On (A IV) (A V) and A VI) are analogs to axioms (T (T IV) of covectors of tropical oriented matroids, which we will define formally in (Scciin 5.2 For tropical oriented matroids, the composition axiom replaced by the surrounding axiom, which follows from stricter conditions on possible perturbations of tropical hyperplanes.
(A (A II)
A III) and
(A IV] are analogs to axioms [CI I
C IV)
the other hand, (A III)
Proof- [ Tj is realized by 0 (0....,,0) 0), or any point in the lineality space of ED(N). ( holds since the construction of Ex(N) is symmetric in i (M]. (AV) V) will be proven separately in Theorem 5.17
For [A
let x E (D(G) and y E CD(H). These are vectors of parameters of the form x (af. Si .......,,,,,... ( and = (a", sy a"..,,,,sk)) For e > 0 small enough, consider z ey. Then for any p E D such that ((p: G) n N(:: H) = holds
arg aagmax(a + p)) arg max (af (sf p) + +((a + (s,, p))) = arg max(a + (s..)))
KE[N]
(E[N]
KE[N]
If (p: G) (p; H) + 0, then max (a, + (s) = (a + (sf p) + e(a + (s% p)) if and only if both
07- + (s.., p) = max of + (s p) and ax + (s).,)) p) = maxie[N] o. + (s" p). Thus, z E CD o H).
For A IV) let again x E CD(G) and y E CD(H). There are fixed values (a()), ky E R such that for every E N(p: G) and j E (p: G)) holds
max (af (s7 p)) = af. + (s): p) (p),), .E[N]
max + (s) ay. + (S)... (n))). iE[M]
Choose
(aj + p² (p), ................................ x + /r (p), ) such that
(s p) + + (p) = max (o (s p) + +) (p). + (s. + (p)).
For p holds
max (af + (s/,p) + (p)) max (max(a + (s, p) + n (p), (5) + (p)) N
*E(M]
max
max (a + (s) p) + nي (p)) (E[N]
max (a + () + 4 (p))
[EiN]
max(1r((p)) (p) + (p). μ (p) + +r)
and the maximizers are precisely the indices N N(p: G) U N(:
For (A VT) suppose that io, f,,.1,,.3 in: fn41 edges assume that all odges are directed. Let again CD(G), CD(H). An edge from ig to ic+1
E [N] is cycle in CG After contracting all undirected
indicates that we have is E NN; G.) and ie+1 E (p; H). This gives
L + (S p) S of + (s) p), and 0% + (8% p) S 2 + (S p).
and at least one of these two inequalities striet, since the edge is directed. Together this implies
E 1et1 (s p) < a - oy (5, s% p)
Adding these conditions up for all in in yields 0 0.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Theorem 5.14 gives necessary conditions on the set of activation patterns which appear as labels of the activation cones. The following statement shows that not all bipartite graphs will appear as activation patterns, unless the data points are affinely independent and are thus at most d d+I 1 many.
Theorem 5.15. The activation fan [n(N) has at most N marimal cones and 11N cones of arbitrary dimension, where N is the number of monomials and M = [D] is the number of dola points. This bound is attained if and only if the points are affinely independent.
Proof. Recall from Corollary 5.12 that for any activation pattern dog(p) 2 for all p E D. The number of bipartite graphs on M U N without isolated nodes is M is (2 - 1)M, and is thus an upper bound for the number of cones of arbitrary dimension. For cones of maximal dimension, [Coolllary 5.12 implies that the activation patterns satisfy deg(p) = for all p E D, and the number of such bipartite graphs N It thus remains to show that this bound is attained if and only if the points are affinely independent. We first assume that the points are affinely dependent and show that there exists graph which does not occur Par
activation pattern. Let P1...., Pa be affincly dependent, i.c., there are distinct data points Pi.... Pkr
P Pi...., P/,, and Mi, Mj > i, E [k],j E [4] such that there is afline dependency
k
p.P:
PSP
Σ
j=1
Let A = (4
>3- =
Then this gives the point
2im1 Hi
E5-1 My
k
App = ppps E conv(pt..... P&)) conn((pl...... pí),
3=1
and q lies in the relative interiors of both convex hulls. Therefore, Theorem 5.13 iii) implies that such a bipartite graph G which has N(i; = (P and N(j (p'..... .p......,)))p p/} is not an activation pattern of (N)) Conversely, let PI,..., PM be affinely independent, and let G be any bipartite graph such that N(p; G) * 0 for all p E D. Note that since the points are affinely independent we have M S d + 1. This implies that for any E [N] there exists
affine linear functional fi(x) Q. + { x) such that fi(p) = 0 for all p N(i; G) and fi(p) < 0 for all D \ (i; Choose a, c, and
u, for all i E [N] such that N(;; G) # 0, and otherwise choose a; small enough. Then 0 = (a) (a).S).... Si..,, SN) ) is a vector of parameters whose activation pattern is G.
Remark 5.16. The number of maximal cones in the activation fan ED(N) equals the number of vertices of the activation polytope PNN Montúlar et al. 2022) give a sharp upper bound for the number of vertices of generic Minkowski sums. However, the Minkowski summands A(p) are not entirely generic in our case.
5.2 Linear and Tropical Classification within the Activation Fan
In |eection 5.1 we have seen how activation patterns can be thought of as multivalued analogs of covectors of oriented matroids. Specifically; Remark 5.8 describes how N = 2 recovers the linear case. In this section describe how also for N > 2 every activation fan carries family of hyperplane arrangements (and thus oriented matroids) with it, by considering interscctions with affine spaces. We extend our findings to tropical oriented matroids, which are analogs of oriented matroids arising through arrangements of tropical hyperplanes.
Recall from the case of lincar classification, that the dichotomics are given by a hyperplanc arrangement Mp in parameter space, the normal fan of the zonotope Pp.
Theorem 5.17. For any S C [N] the bipartite graph with edges E(G)= = (pi p D S} is an activation pattern of a cone (N). The cone (G) is the normal cone of a face F of the activation polytope PD(N), where F is itself an activation polytope Po([S]). In particular, ranging over all sets with |S| = 2, one gels (?) many faces of the activation pobylope (N(N) which are equal to the zonotope Po-
22

---

Published in Transactions on Machine Learning Research (09/2024)
Proof. For notational convenience, we identify (d + 1)N- dimennsiion point 0 (dd, n, m) with matrix M of size (d+ + 1) x N, where the first column is identified with the first (d + 1) entries of the vector, and so on.
Fix S S [M]. Since D is a finite set of points, there exists a linear functional I Rd+ such that (1.p). > on all e E D. We define the 8h column of the matrix MS as
1 is [
M = 0
Since M corresponds to vector of parameters of a tropical signomial f, it has associated activation pattern G, which we now describe. Recall that N(;; G) = {p E D p activate the 7th term). For p E 'D we have f(p) = ((Ip),) > O, and precisely the terms i E S are active. Thus, E(G) = {pi p E D, E S}, and this is the activation pattern of the cone Cb(G) which contains M in its relative interior.
We now describe the face F$ of the activation polytope whose normal cone is Cp(G). By Proposition 11 we have that fs Eped F()), where the face F()) of A(p) is the (IS - -i-dimensio simplex conv(M E 5). The polytope FS lies inside an affine space of codimension (IS) - 1) (dd + 1). Inside this affine space, we have FS = P)(SS)), which is embedded into @(d, N) by insorting 0 for all columns indexed by [M] \ S, when we view the vertices of Po((S)) as matrices of size (d + 1) x [S| and the vertices of ps as matrices of size (d+ 1) N.
We dualize the above statement to find the hyperplane arrangement HD in the activation fan ED(N). The k- dimensional face Fly identified in the former proof is dual to (N ()))))))-mmmmmioon cone Cit E (DV. More specifically, Cij = (D) where G is the activation pattern such that N(, G) {i.j} for all p D. The #'-dimensional faces of Fiy are dual to (N + 1) - d'ddiennsiin cones of Ed(N) which contain Cig. The collection of these cones is called the slar of Cij. and denoted by star(Ciy)) Let lin(Cig) denote the smallest linear space containing Ctj. Projecting the cones in the star of Ci3 onto the orthogonal space lin (C,))) yiclds the normal fan of the zonotope F.j. We thus obtain the following dual statement:
Theorem 5.18. For each j E [N], let Ciş CD(G) be the activation cone for the pottern G with (i,}) for all p E D. Then the projection of star(Ci;) onto lin(C:))+ is the polyhedral fun induced by the hyperplane arrangement HDD
We now move towards tropical hyperplane arrangements and tropical oriented matroids. Tropical oriented matroids can be viewed as multivalued analogs of oriented matroids, which arise through tropical hyperplane arrangements. We will see that activation patterns can also be viewed as generalization of realizable tropical oriented matroids.
A tropical hyperpiane H--a) is the tropical hypersurface T(f) defined by the linear tropical polynomial fa(x) = Diela 0; @ x, It is an affine translate of the normal fan of a standard simplex with oper -a = -a,,,.... , ad) Rd. Thus, any tropical hyperplane subdivides the ambient space into d maximal regions, one for each term of fa- Given a finite number of tropical hyperplanes H(a')....,H common refinement of all these subdivisions, and label each region R. by tropical coveclor (A,,...,AA, where A; C [d] is the set of indices of the terms of fai which are active on R. We obtain an activation pattern G = ([d) D. E(G)) from a tropical covector by setting N(p: G) = A for P. D {p....., ).)) Any set of tropical covectors which arise through such an arrangement of tropical hyperplanes is called a realizable tropical oriented matroid.
we consider the
Similarly to the linear case, tropical oriented matroids are defined through a set of axioms, and it was shown that sets satisfying these axioms are in bijection with subdivisions of products of simplices (Horn 2016). A set T C {(AI,...,AAl) A; S [N]. E [M] is the set of tropical covectors of a tropical oriented matroid if it satisfies the following axioms (Ardila & Develin
2008 Def. 3.5).
(T I) Boundary: For each j E [N] holds (f)).... (.))....))))..
(T II) Elimination: If A, B T and j E [M] then there exists a type C T with C, = Aj U B, and C& (Ak, Bg, A U B&} for all l E [M]
23

---

Published in Transactions on Machine Learning Research (09/2024)
(T III) Comparability: The comparability graph CG of any two types A and B in T is acyclic.
(T IV) Surrounding: If e T the any refinement also T.
Let A, B be tropical covectors and G, H. be the corresponding activation pattorns. The comparability graph of the tropical covectons CG = UPED Ca where the comparability graph Ca is as defined Theorem 5.14 The refinement of a type A (A1,..., Am) with respect to an ordered partition (Pı,..., P,) of [M] is A (A, (m))) : AM PP (M))) where m(i) the largest index for which ( & 0. Note
that [A mI) A V) (A IV] and A VI] of Theorem 5.14 are generalizations of axioms of tropical oriented
matroids.
Similarly to Theorem 5.18 we can recover realizable tropical oriented matroids by intersection with an affine subspace.
Theorem 5.19. Let N = and consider the d-dimensional affine space
A= {(a1: Udd Sa) E ((d,d) s; - e, for i [d]]
Then Ed(d) n is the collection of cells in the tropical hyperplane arrangement UPED H(-p))
Proof. Let CD(G) E (d) such that Cp nA 0, and consider the coordinate projection a (CD((G)) onto coordinates (1....,aa)) Any parameter 0 E A defines a function fa(x) = Diejd X, By construction, E CD(G) A if and only if a (0) satisfies max4e[]] as + Ps az- P.' for all edges i'p E(G). In other words, is contained in the region it of H(-p) for all such edges, and (CD(GG) is a region in the tropical hyperplane arrangement.
We can extend the above statement to more parameters N > d by requiring that the additional monomials (#: 1 d lie below the lifted polytope conv(( e) (......()))) )). Tn this case they are not visible in any regular subdivision of the dual Newton polytope, and can thus be neglected in the above proof.
The dual statement to Theorem 5.19 identifies a subcomplex of the boundary of the activation polytope Pd(d). This subcomplex corresponds to regular subdivision of products of simplices, and coincides with the boundary complex of the unbounded polyhedron in Develin & Sturmfels 2004 Lemma 10), whose bounded cells determine the tropical convex hull of the points in D.
6 Classification Fan
In SScciio 4 we have seen how we can understand tropical rational functions g Ø h via tropical signomials f by separating the N terms of into n positive and m negative terms. This defines and h respectively, such that f = g O and N = n + m. This procedure subdivides the regions defined by T(f) into positive and negative regions. In terms of the activation patterns, this amounts to a coloring of the nodes [N] as positive or negative nodes. In this section, we consider the classification fan, the fan which arises through the coloring of the nodes. In analogy to the linear case, will consider the level sets and sublevel sets of the 0/1 lolssoss function.
6.1 Dividing the Parameter Space
Let D C Rd be finite set of points, N N and fix n, m Nş1 such that m N. We split the parameter space into parameters of the numerator 9, which we denote a; for i E [n] and parameters of the denominator h, which are given by by an+jj ty = Satj for j E [m]. Given a vector of parameters 0= (a), Si,,,...... .a.....,a,,, Gn, Sn, bi, .ty...,bun bm, (m) E Od, n, m)), this defines a numerator go(x) maxie[n] (a. (a.+ (Ss, x))
and denominator (xxx) = maxjelm[ + (t,. x)), such that fo = go + ha- Given an activation pattern with nodes (G) = D U [N], the split + m N induces a coloring of the nodes [N] into [n] U [m].
Definition 6.1 (Activation Pattern of Tropical Rational Function). The activation pattern of (g Ø h, D)
is the bipartite graph G E(()) on nodes V(G) D ([n] U [m]) with edges E(G) {pi p activates the 4th term of g) Ll (pj | activates the jth term of h).
24

---

Published in Transactions on Machine Learning Research (09/2024)
J2
11
P
11
jn
j2
12
P2
J2
11
Pi
P2
2/
P
P1
12
11
P2
0/
j2
i1
ji
P2 J1
J2
in
j2
j2
J1
j1
12
i2
(b) Decision boundary in input space (top) and dual sign-mixed Newton polytopes (bottom).
(a) Activation pattern.
Figure 6: A coloring of the bipartite graph and the corresponding regions, as described in Example 6.2. The positive regions are labeled by 11, i2 and colored in green, the negative regions are labeled by J1. and colored in red, and the decision boundary highlighted in blue.
Example 6.2 (From Signomials to Rational Functions). We continue with the activation pattern of the tropical signomials f from Example 5.2 Partitioning N 4 into n 2, m 2 yields
(i) a partition of the monomials of f into two signomials g(x) 01 os + a, x 0 and h(x)
bn @ o bg ot
(ii) a partition of the regions defined by T(f) into positive regions i i2 and negative regions J1.
(iii) partition of the vertices of the dual regular subdivision of Newt(f) into positive and negative
vertices.
The positive and negative regions separated by the decision boundary of h. Figure shows all of these partitions for the configuration from Example 5.2
With this partition of parameters, the activation cone of G is
CD(G) =(0 ( max (a: + (st, (k, + (ty, p)) = a. + (s..p) for all pi" e E(G) and (elm)
max (a; + (). (tj, pb) = by- + (ty+.)) p) for all pj" F(G)) iE(m].JE[m]
CO m.) m).
Lemma 6.3. Fir a partition [N] [n] [m] and activation pattern G of tropical rational functions. Then for any 8 E Cp(G) it holds that (ge Ø hp) 2 0 if and only if pi E E(G) for some i E [n], and (go 2 (p) & 0 if and only pj" E E(G) for some j E [m]] The inequality is striet if and only if the set of neighbors of p in G is monochromatic, i.e. ((p:G) C [n] or N(p: S [m].
Proof. If max (a: + (si. p). by + (tz.p)) = a. + (St P) then go(p) 2 (p(p) for all i" E [n] such that pi* E EG) This inequality is strict and only if the maximum not attained in any term of he (p). Similarly, go (p) < hp(p) for all j E [m] such that pj* E E(G).
Definition 6.4 (Classification Fan). The classification fan Ep m) of a finite data set C R and tropical rational functions with n monomials in the numerator and monomials the denominator is the set of all activation cones in parameter space O(d, n, m) of tropical rational functions.
25

---

Published in Transactions on Machine Learning Research (09/2024)
We emphasize that the fan structures of the activation fan Σp(n + m) and classification fan En m) coincide. The only difference is partition of the coordinates in the underlying spaces. However, this distinction is crucial as we now want to distinguish between monomials of the numerator and monomials of the denominator.
6.2 The Space of Dichotomies
In this section, we subdivide the parameter space (dd, n, m) into sets, each corresponding to those parameters which classify the data according to fixed target dichotomy. These (open) sets generalize the chambers of the hyperplane arrangement from [eection 2 We will also describe indecision surfaces analogs of the hyperplanes in the arrangement, which will turn out to be decision boundaries inside the parameter space.
Definition 6.5 (Indecision Surface). The indecision surface of a data point p D is
S()) = {0 E ((d,n, (go 2 hp)(p) =0}
in the parameter space of tropical rational functions.
We will see that the indecision surface plays the role of the hyperplane (1, p)) from the linear case (cf. Section 2 in this more general setup.
Theorem 6.6. The indecision surface S(p) is a sign-mized subfon of the normal Jan of (p)) of dimension (n + (d 1) 1. It divides the parameter space (d, n, m) into two open connected components St (p)
{0 E ((d, n, m) (go @ hp) > 0} and S (p) = {0 E O(d, m)) (go Ø ha)(p)) < <0}.
Proof. First note that the indecision surface is itself the decision boundary of the tropical rational function Dietnl ag O s 2 ① jEjm] by O t, in indeterminates a, S,, by, ty. The Newton polytope of this tropical rational function is the simplex A(p). We use the notation from the proof of Theorem 5.7 and fix a partition [N] = [n] L [m]] as explained in Section 6.1 Theorem 4.2 implics that S(p) is a polyhedral fan, whose maximal cones are dual to edges conv(v,, Vz) where i E [n] is a monomial of g and j E [m] is a monomial of h. The Fuclidean closure of the region (+) the union of all normal cones of vertices v,, i [n] and since A(p) is simplex, this union connected. Similarly, the closure of (pp)) the union of normal fans of vz.j E [m].
Fix a target dichotomy C E {+, MM.. We obtain the space of solutions of a fixed dichotomy as S o%" (d, m) = , (CE(pp)) = (0 E O(d, n, m) sgn (go(pn) @ ho(ps)) = C for all k E [M]]. Corollary 6.7. The orrangement of indecision surfaces S(p) over all p E D divides the parameter space into open sets o% (d,,mm) over all dichotomies C C (+
This arrangement can be viewed as the true analog of the hyperplane arrangement Hp from the lincar case, and oo" (d, n, m) plays the role of an open chamber. In Section 6.3 we will fix target dichotomy C* and study (the closure of) the space Og (d, n. m). We will see that this space is disconnected (cf. Theorem 6.11 but it allows a natural fan structure which is inherited from the classification fan En, m).
The subdivision into these different subfans relates to the growth function and Vapnik- Chervonenkis dimen- sion (Vapnik & Chervonenkis 1971). This is fundamental concept in statistical learning theory which relates the complexity of function class with the generalization ability of any learning algorithm based that function class. Let c(n, m, D) H[ 6% (d, n, m) # 0} denote the number of dichotomies which can be attained by tropical rational functions with + m terms on the data D. The growth function Im : N- + N is given by II, (M) = max(c(D, n, m) D C Rd, |D] = M). The VC-dimension of tropical rational functions is max(M II.. (M) = 2 To compute the growth function, à common strategy is to count the mumber of non-empty regions c(D, n, m) for a given data set D and then maximize this number over the choice of the data set having a fixed cardinality M. We refer the reader to Anthony & Bartlett) (1999) for an overview of these techniques, and Bartlett et al. (2019) for recent bounds on the VC dimension of neural networks with piecewise linear activation functions.
26

---

Published in Transactions on Machine Learning Research (09/2024)
6.3 Classifying Points Correctly
We now fix target dichotomy C* E [+,-) of the data. The dichotomy divides the data points into two
sets DJ = (p: E D C +} and DC = =D D4 An activation pattern G of (g D) is compotible with C* if (p: G) S [n] for all e Df and (p; G) S [m]] for all P DC-. An activation pallern is weekly compatible with C* if G)::)) [n] # for all p E DJ and N(P: n [m] + 0 for all p PED.
Definition 6.8 (Perfect Classification Fan). The perfect classification fan [n, m) is the subfan of the classification fan consisting of those closed cones whose activation patterns are weakly compatible with C* In other words, > (n, m) is the restriction of the classification fan onto the set O8 n) = (0 E (dd, n, m) (0(p) 2 (P) Vp E D4", go(p) I (p) Vp E "D")
The wisdom behind this notation will become clear in Section 6.4, where we consider level-sets of the 0/1-loss. We have chosen to consider activation cones CD(G) as closed cones. As a consequence, we obtain lower- dimensional cones on the boundary of g (d, n, m)), whose activation patterns only weakly compatible with C*, On the other hand, this certifies nice geometric properties of the perfect classification fan.
Proposition 6.9. The perfect classification fan 20 (n, m) is pure, i.e. every inclusion-mazime cone is full-dimensional.
Proof. Any -imen-dimesiionll cone in [D is the intersection of two full-dimensional cones from the activation fan. Let (D(F) be two full-dimensional cones of the activation fan. We show that if CD(G), Cp (H) & ()) then CD(G) n CD(H & (n, Since Cp(G), CD(H) are maximal, Corol- lary 5.12 implics that (G), (H) E E8 m) if and only if G, H are compatible with the target di- chotomy C Therefore, if CD(G) & 2D (n, then there exists some Pá E Df such that N(P G) S [m] or P E DC such that N(p3: G) S [m]. Similarly, Cp(H) & Eg m) implies that there exists some PH E DC such that H) S [m] or Рн E DC such that N(p;; G) S. [n]. Let L = GUH. Then (p(G) n CDH) = =C,(L), and since E(L) 2 E(G) U E(H) we have that ((P: L [m] * 0 or ((p) L) n [m] #0 or N(Pa: n [m] * 0 or N(PH: L) n [n] + 0, i.e. L is not weakly compatible with C
The perfect classification fan can be viewed a subfan of the activation fan, and its full-dimensional cones are thus dual to vertices of the activation polytope. To describe these vertices, recall that Po(n + m) EM APx)). Under the partition of the parameters, each simplex APp)) has vertices v i [m]] and vý.j e
(m], and, similarly to the construction in Section 4 we label each of these vertices with sgn(v?) = + for E [n] and sgn(v)) - for j E [m]. Each vertex v of PD(n + m) can be written uniquely as V Ene Vi(e) for indices ((k) E [n]] L [m] and we equip v with the dichotomy d(v) = (sgu(vil))......sgrr) .sg....ss)
Theorem 6.10. verler v of Po(n + m) is dual to a full-dimensional cone of Eg (n, m)) if and only if d(v) C*.
Proof. A vertex v has d(v) C* if and only if v lies in the intersection of the normal cones of v for all k E [M] and sgn(vā ((k)) = C--- By the proof of Theorem 5.7 this is equivalent to CG(G) being the normal cone of v, where G is compatible with C*.
In the case of linear classifiers, the set of perfect solutions is the single polyhedral cone in the hyperplane arrangement HD which corresponds to the dichotomy C*. In contrast, the perfect classification fan is an consider closed polyhedral cones, they always intersect in the origin.
entire polyhedral fan. Clearly, We thus consider a stronger notion for polyhedral fans. A pure d-dimensional polyhedral fan Σ is strongly connected if for all maximal cones C, D there exists sequence of maximal cones C = Co, CI,...,C Ck, C2+1 D such that dim(C, n C.+1) = d 1. Any fan decomposes into strongly connected components, i... inclusion- maximal subfans of Σ which are strongly connected. The strongly connected components of the perfect classification fan are in bijection with the connected components of O (d, n, m), which is the interior of the support of mQ ().
27

---

Published in Transactions on Machine Learning Research (09/2024)
12
P1
P1
81
1
P1
81
Pi
J1
P2
22
P2
P2
P2
82
P3
P3
J1
P3
j1
Pa
j1
P3
J1
P2
P4
j2
P1
J2
P4
j2
P4
J2
i
j2
P1
11
P1
i1
P1
i1
11
P2
12
P2
12
P2
i2
P2
P3
f1
P3
j1
P3
P3
J1
P4
J2
D1
j2 P4
J2
P1
(a) The point configuration.
(b) Activation patterns of cones in the two connected components.
Figure 7: The 12 cones of correct classification from the example in the proof of Theorem 6.11 strongly connected components.
grouped in
Theorem 6.11. The perfect classification fan is not always strongly connected.
Proof. Consider the data set D (Pr, P2, P3, p4} with p: (0, (1, (1, 1), (2. (3,3), and the target dichotomy is C* (+. The perfect classification fan (2, C 2,2, consists of 8 cones of dimension 12 (the maximal dimension)) which are divided into two strongly connected components: The intersection of any cone from the first connected component and any cone from the second component is solely contained in the lineality space of the fan. The activation patterns of the 8 cones are shown in Figure 7 In this example we can identify two fundamentally different types of cones, assigning P2, Ps either to the same negative index or to two separate negative indices. Then we have discrete symmetries of the parametrization via permutation of the positive indices i1, i2 and permutation of the negative indices J1, The permutation of jı.J2 does not strongly disconnect the set, but the permutation of i, 12 does.
]
In Theorem 5.15 we have seen that the trivial upper bound for the number of maximal cones in the acti- vation fan attained if the data points are affinely independent. We now obtain an analogous result for the classification fan, however, the following statement does not require the entire data set be affinely independent.
Theorem 6.12. Let n. 2 2 and fir a targel dichotomy C*. Then the perfect classification san Eng (n, m) has al most DD ml "n cones of maximal dimension. This bound is allained if and only if DC D are linearly separable and both DC and DC are affinely independent sets.
Proof. Since (n, m) is subfan of the activation fan Ep(n + m), Corollary 5.12 implies that the cones of maximal dimension are labeled by activation patterns G such that P) (i(p)) C [n] for E Df and p) = (i()) C [m] for p E DC". Thus, the number of maximal cones is at most n! m pe Any affine dependency within Dg" or DC forbids at least one such graph. This can be proven by following the argument in the proof of Theorem 5.15 namely that if DG" DC are not linearly separable, then their convex hulls intersect, and Theorem 5.13 implies the existence of a forbidden graph. Thus, the bound is not attained. We now show that the bound can be attained for affincly independent sets DC" and DC", which are linearly separable. Let G be bipartite graph in which every point in D is incident to exactly one edge, connecting every point in Df" with a node in [n] and every point in DC with a node in [m]] Let 1(x) = = (u,x) ) be the equation defining a soparating hyperplane, i.c. 1(p+) > O and I(p=) ) < 0 for all p* E Df", ,p' E DC" Since the points in DC are affincly independent, there exists an affine function s((x d, + (v,, x) for every i E (n), such that [((p)) = for all p+ E N(i;; and f((pt) < for all pt E Df" (). Similarly, there exists an affine function fx) = =ey+ (W,,x) for every j [m] such that f((P) = 0 for all p E EN(j; G)) and f)(p)) < 0 for all p E DC \N(i; G). Since ((p))= < 0 for all p E DC we can choose scalar >> 0 such that for all p E DC" holds -X((p > max f((pT). Similarly, since
1(pt) for all p+ DD" we can choose μ such that for all p+ E DC" holds pl(p+) > maxje(m| fz(+t).
28

---

Published in Transactions on Machine Learning Research (09/2024)
For 1" E |n].p" G)) and jr [m], p E N(;"; G) we obtain
max(A f((p')) M(p") (p*)) = M(D )> >0
KE[n]
max 5e(ml
(-ul(p+) + [)( -ul(p") max (D) <0 jEmm
max (N((p)) + ++(( = M(p + max (fi(p*)) <0 iE|n]
rE(n)
max ]E(m)
up(p + fj(p))) = -- > + fz- (p*) = =1((p )) > 0
We can thus choose a; = Ac + d;, 8; = du vi, by = -ue + ej. tj = = + wj- This defines a vector of parameters with activation pattern G.
We close this section on perfect classification with an observation concerning tropical semialgebraic sets.
Remark 6.13 (2, m) is tropical semialgebraie set). The set mm(n,m) is the set of solutions to
set of tropical polynomial inequalities. Concretely, it is given by the tropical polynomial inequalities Die(n a4 3 s op
+ jmmm by 0t, ty for all p E DL" and (ee(n) dg s; c ) bj @ top for all p E DC",
Tropicalizations of semialgebraic sets have close relations to positive tropicalizations (cf. [Section 4 and their systematic study has been established in the work of Jell et al. (2020).
6.4 Sublevel Sets of the 0/1-Loss Function
Recall that the 0/1-loss function counts the number of mistakes of any function f with respect to the target dichotomy C*, i.c. errc- = ifi E [M] (gg((f((p)) C. If fo = go 2 he is a tropical rational function contained in cone CD (G)), then errc- is constant along (D(G). We thus extend the function errc- to cones and activation patterns. If (G) is maximal, then etty. (G) counts the mumber of edges in G between Df and [m], and between DC and [n].
S Definition 6.14. The hthh level set the polyhedral fan (n, m) = UDDD(G), where the union runs over all
activation patterns G such that erro- (G) k. We define the th sublevel set as DD (n, m) = U E (n, m).
Proposition 6.15. If n m, then for any level set it holds that 2 (n, n) = 20
-I (n,
Proof. Let E (n,n,, n) and define 0' through parameters
of = 8% 5 L,, 6y¹ -0,, tf = for i [m].
Note that the corresponding tropical rational functions satisfy fo(p) = - (er) Thus, fo makes mistake at p if and only if fo classifies p correctly, and vice versa.
implies that the level sets ED (n, m), (b vertices of the activation polytope Po(n + m)) and counting the number of vertices in each part we obtain a sequence of natural numbers. In Algebraic Combinatorics one likes to describe sequences which are symmotric, unimodal or log-coneave. Proposition 6.15) allows us to obtain statement in this spirit:
Theorem 6.10
Ent"m(n (n, m) induce a partition of the
Corollary 6.16. Let l denote the number of maximal cones in the t level set Eb n). Then 4 = lip)- i.e. the sizes of level sets are symmetric.
Recall that Prrposstionn 2.4 shows that in the case of lincar classifiers, a cone of mistakes is always (strongly) connected to a cone fewer mistakes. We now observe that the analog does not necessarily hold in the more general setup of tropical rational functions.
Theorem 6.17. The sublevel sets Σ (n, m) are not always strongly connected, even if the data points are general position. More specifically, let C C Eh (n, be a strongly connected component. It may happen that all full- dilensimal neighbors D En m) of C (which are adjacent through codimension 1) satisfy errc- (TD)-(D)= k.
29

---

Published in Transactions on Machine Learning Research (09/2024)
3
4
7 e
7
8
(a) 0 mistakes
(b) 1 mistake
Figure 8: The point configuration from Theorem 6.17 and two tropical hypersurfaces classifying the points perfectly (left) and according to G (right).
Proof. To be precise, here consider the standard notion of general position: M points in Rd are in general position if no k of them, for k S d+1, are contained in an affine subspace of dimension strictly less than k k-1 We give an example a configuration of points in general position which can be classified perfectly, for which however there exists a cone CG) with errc (Cp(G)) = from which there exist no (weakly) decreasing path any cone with 0 mistakes. Let D (Pi = (-2,3) P2 (3,3), Ps = (0, (0,0), Ps = (-7, -3), Pe (3, -4) and DC = (Pa = (1, P+ = (0, 1), Ps = (-2. 1). = (1), as shown in Figure 8 We computed the 12-dimensional fan E
have (2, (2, U (2, U (2. The perfect classification fan > (2, 2) consists of 16 maximal cones, which divide into 8 strongly connected components, and the first level set consists of 304 cones, which divide into 28 components. The connected component containing (D(G) consists of 20 maximal cones, but none of them is adjacent to any of the 16 cones of ED (2, through codimension 1. More precisely, the connected component which contains CD(G) intersects Σ (2, 2) in dimension 3, which is the dimension of the lineality space of (2, 2). In this sense, the connected component containing CD(G) intersects the perfect classification fan (2, trivially.
using the software SageMath (Sage Developers 2021 By dcfinition,
Corollary 6.18. The 0/1-loss function has local non-global minima.
7 Conclusion
We discussed binary classification by signs parametric piccewise lincar functions from the perspective of real tropical geometry. In this context, we described sets of ReLU networks as semialgebraic sets within of tropical rational functions with fixed number of monomials in the numerator and denominator. We highlighted on the one hand the subdivision of the parameter space by the combinatorial type of the represented functions, specifically the combinatorial type of the decision boundary, and on the other hand given
the subdivision of the parameter space by the different activation patterns that are recorded dataset. These two subdivisions are complementary to each other since a priori the decision boundary is independent of the particular data under consideration. In future work it will be interesting to study how the two interact; concretely, for a particular input dataset one might study the types of decision boundaries that permit certain types of classifications of that particular input datasct. The discussion of activation patterns gives rise to generalization of oriented matroids. We showed how this can be used to obtain results on the structure of the loss landscape.
We hope that this work contributes to the further cnhancement of the synergies between rescarch neural networks and polyhedral geomelry, particularly as approached in real tropical geometry. As we observed, in some cases the same concepts appear in different commmities under different names and leveraging results and perspectives from each side can facilitate advanoes and motivate new interesting future research. We hope that the presented discussion might serve as a springboard for further explorations including: the containment. of neural networks with piecewise linear activations within simple parametrivations of tropical
30

---

Published in Transactions on Machine Learning Research (09/2024)
rational functions; the structure of the subdivisions of the parameter space by the different properties of the represented functions, including the combinatorial type of the decision boundary, well as the interface with other subdivisions arising from a particular dataset, including complexity measures and the combinatorial structure of the loss landscape.
Acknowledgments
We thank Hanna Tseran for various valuable discussions. We are grateful to Günter Rote for clarifying the proof of the composition property for [Theorem 5.14] Arnau Padrol for pointing us to relevant. literature on k-sets and k-faccts, and Stefano Mereta for valuable input on the formulation of Theorem 3.2 We thank anonymous TMLR reviewers for constructive leedback. This project has been supported by DFG grant 464109215 "Combinatorial and implicit approaches to deep learning" within the priority programme SPP 2298 "Theoretical Foundations of Deep Learning". GM has also been supported by NSF DMS-2145630, NSF CCF-2212520, and BMBF in DAAD project 57616814.
Table of notations
(dd)
parameter space for linear classification parameter space for classification by tropical rational function with fixed number of terms ! parameter space of tropical signomials 16
n(d,n,m)
N)
Ho
hyperplane arrangement. Upe polytope whose normal fan is Dp
Pp
Co
sign vector associated to the conc a of a hyperplane arrangement polyhedral fan arising in linear classification induced by the data
Ep
EN(N)
activation fan: polyhedral fan arising classification with tropical rational function with terms
P)(N)
activation polytope: polytopes whose normal fan is (D(N) 18
errc-(0)
loss in classification with parameters A 6
(D(
activation cone: parameter cone with labeling G 17
DD (n,
classification fan: a set of cones in parameter space perfect classification fan: a subfan of the classification fan
25
D (m,
27
References
Motasem Alfarra, Adel Bibi, Hasan Hammoud, Mohamed Gaafar, and Bernard Ghanem. On the decision boundarics of neural networks: A tropical geometry perspective. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):5027 5037, 2023. doi: 10.109//PPAM1.2022.2000
Xavier Allamigeon, Stéphane Gaubert, and Mateusz Skomra. Tropical spectrahedra. Discrete & Computa- lional Geomelry, 63):5550 548, February 2020. doi: 10.1007/88045440000000778
Martin Anthony and Peter Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999. doi: 10. 10.1017/CB097500116444
Federico Ardila and Mike Develin. Tropical hyperplane arrangements and oriented matroids, Mathematische Zeitschrift, 262(4))795 816, August 2008. doi: 10:1007/500000-00000000000
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. In International Conference Learning Representations, 2018. URL https:
J_rgWW
Peter Auer, Mark Herbster, and Manfred K. Warmuth. Exponentially many local min- ima for single ncurons. In Advances Newral Information Processing Systems, volume MIT Press, 1995. URL
31

---

Published in Transactions on Machine Learning Research (09/2024)
Randall Balestriero, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision. In Advances in Newral Information Processing Systems, vol- wne 32. Curran Associates, Inc., 2019. URL 200///11n00011200233223552222222 Papeee.-----pp....... seaper.pdff pdf
Peler L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285- 2019.
Hans-Peter Beise, Steve Dias Da Criz, and Udo Schröder. On decision regions of narrow deep neural networks. Newral Networks, 140:121-129, 2021. doi:
Anders Björner, Michel Las Vergnas, Bernd Sturmfels, Neil White, and Günter M. Ziegler. Oriented Matroids. Cambridge University Press, November 1999. doi: 10.1017/CRR9770011555 15565077.
Martin L. Brady, Raghu Raghavan, and Joseph Slawny, Back propagation fails to separate where perceptrons
succeed. IEFE Transactions on Circuits and Systems, 36(5):665-674, 1989. doi: 10.1109/31.3133
Maric-Charlotee
Brandenburg, Georg Loho, and Rainer Sinn. Tropical positivity and determinantal varicties. Algebruic Combinalorics, 6(4) 1040, August 2023. doi: 10.5802/alco.2860
Vasileios Charisopoulos and Petros Maragos. Morphological perceptrons: Geometry and training algorithms. In Mathematical Morphology and Its Applications to Signal and Image Processing. ISMM 2017., volume 10225 of Lecture Notes in Computer Science. Springer, 2017. doi: 10.1007/978-3-31--574400
Vasileios Charisopoulos and Petros Maragos. A tropical approach to neural networks with piecewisc linear activations, 2018. URL
08749
Thomas M. Cover. Geometricnl and Stalistical Properlies of Linear Threshold Devices. PhD thesis, Stanford University, May 1964. URL
http://tww..edeeoooe/////////ppe pdf
Mike Develin and Bernd Sturmfels. Tropical convexity. Documenta Mathematica, 9:1-27, 2004. ISSN 1431- 0643.
Tolga Ergen and Mert Pilanci. Global optimality beyond two layers: Training deep relu networks via convex programs. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 2993 3003. PMLR, July 2021. URL https:// proceedings mlr. aassss//139/erree2/22
Daniel Froeman and Joan Bruna. Topology and geometry of half-roctifiod network optimization. In 5th International Conference Learning Representations, TCLR 2017, 2017. URL https://openreviee..
net//forum?tddBBKOFWWW..
Jacob E. Goodman, Joseph O'Rourke, and Csaba D. Tóth (ods.). Handbook of Discrete and Computational Geometry. Discrete Mathematics and its Applications. CRC Press, Boca Raton, FL, 3rd edition edition, 2018.
Marco Gori and Alberto Tesi. On the problem of local minima in backpropagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(1):76 86, 1992. doi: 10.1109/34.10700
Elisenda Grigsby and Kathryn Lindsey. On transversality of bent hyperplane arrangements and the topo- logical expressiveness of relu neural networks. SIAM Journal on Applied Algebra and Geomelry, 6(2): 216-201 242, 2022. doi: 211137220M138890.
Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of ReLU networks. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 11731-11760. PMLR, July 2023. URL https::/proceedings.oe httpp:///pooeeenggg..rrrroo///// press/v202/ grigsby23a.c httmll html
32

---

Published in Transactions on Machine Learning Research (09/2024)
Philip Hartman. On functions representable as difference of convex functions. Pacific Journal of Mothe- matics, 9(3):707 1959. doi: (0.2140/pjm.1953)
Christoph Hertrich, Amitabh Basu, Marco Di Summa, and Martin Skutella. Towards lower bounds on the depth of ReLU neural networks. In Advances in Neural Information Processing Systems, 2021. URL
Peter Hinz and Sara de Geer. A framework for the construction of upper bounds on the number of alfine linear regions of ReLU feed-forward neural networks, 2020.
Silke Horn. A topological representation theorem for tropical oriented matroids. Journal of Combinatorial Theory, Series A, 142:77 112, August 2016. doi: 2o..016/.jcta..216......8
Joey Huchette, Gonzalo Muñoz, Thiago Serra, and Calvin Tsay. When deep learning meets polyhedral theory: A survey, 2023.
Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, and Richard G. Baraniuk. Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3789-3798, June 2023. URL https::/openaaceess httperrt.ccccooooete///////////d
SplineCam Exact cesualizatinn aariiiionnnnnnnnaaeeeeeeeeee Chwracterizaaaiinn... of Deep Netippeetworr... Seeepeeetorrceeeeeeettr....... and __d___nd__________e_____________ 2023 Bactl__pper______ html
Philipp Jell, Claus Scheiderer, and Josephine Yu. Real Tropicalization and Analytification of Semialge- braic Sets. International Mathematics Research Notices, 2022(2):928 958, May 2020. doi: 10.1093/1mmy rnaal12.
Jesse Johnson. Deep, skinny neural networks are not universal approximators. In International Conference wttp://oooooriidddyrlsa.a.... on Learning Representations, 2019. URL htt.a//////ooooerrr..//////////a
Michael Jaswig. Essentials of tropical combinatorics, volume 219 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2021. ISBN 978-1-4704-6741-8.
Kedar Karhadkar, Michael Murray, Hanna Tseran, and Guido Montúfar. Mildly overparameterized ReLU networks have favorable loss landscape. Transactions Machine Learning Research, 2024. URL
Anita Kripfganz and R. Schulze. Piecewise affine functions as difference of two convex functions. Opti- mization, 18(1):23-25 29, January 1987. doi: (00000002331887008///8818
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In Advances in Neural In- formation Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.
neurips.cc/papet httpp///ww...aaaooo///////////aa
.dff
Bo Liu. Spurious local minima are common for deep neural networks with piecewise linear activations. IEFE Transactions on Neural Networks and Learning Systems, pp. 1-13, 2022. doi: 10.1109/TNNIS. 2022.3204319.
Diane Maclagan and Bernd Sturmfels. Introduction Tropical Geometry, volume 161 of Groduate Studies in Mathematics. American Mathematical Society, 2015. ISBN 978-0-8218-5198-22 doi: 10.1090/ /t////mmm11// 161.
Michael S Malena and Colin A Raffel A combinatorial perspective on the optimization of shallow relu
networks. In Advances in Newral Information Processing Systems, volume 35, pp. 22187-22199 Cur- ran Associates, Inc., 2022. URL httpps:/pproceeddnggs neurs cctpcccc..rpeeeeee ftt.es//apprr//2/////// 038777/595E4s3arrrroosssrPrrrrre
Jifí Matoušck. On geometric optimization with few violated constraints. Discrete & Computational Geom-
etry, 14(4):365 384, December 1995. doi: (0.1007/b602550771
33

---

Published in Transactions on Machine Learning Research (09/2024)
D. Melzer. On the expressibility of piecewise-linear continuous functions as the difference of two piecewise- linear conver functions, pp. 118-134. Springer Berlin Heidelberg, Berlin, Heidelberg, 1986. doi: 10.1007/ BFb0121142.
Aaron Mishkin and Mert Pilanci. Optimal sets and solution paths of ReLU networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Leurning Re- search, pp. 24888 24924. PMLR, July 2023. URL httpss//proceedddngg mlr www.mlrl.rrrrrr. pttp:pss///222/msssiiii//// html
Aaron Mishkin, Arda Sahiner, and Mert Pilanci. Fast convex optimization for two-layer ReLU networks: Equivalent model classes and cone decompositions. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15770-15816. PMLR, July 2022. URL /tttps://proceedingg. mll. wttp://iiiin......e//e/e/ee/eeee pttss///////m/mmm//2222222. html
Guido Montúfar. Notes on the number of linear regions of deep neural networks. Samp TA 2017 Special Session Mathematics of Deep Learning, 2017.
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of lincar re- gions of deep neural networks. In Advances in Neural Information Processing Systems, volume 27. Cur- ran Associates, Inc., 2014. URL https::/ nwurips.cc/paper..... httpppporr..ileeeeeprr/////////e (191208800f600ea11990111cccllcld
pdf
Guido Montúfar, Yue Ren, and Leon Zhang. Sharp bounds for the number of regions of maxout networks and vertices of Minkowski suns. SIAM Journal on Applied Algebra and Geometry, 6(4):618 649, 2022. doi: 10.1137 /1M144299
Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. Neural networks should be wide enough to learn disconnected decision regions. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3740 3749. PMLR, July 2018. URL https::/prroceedings mlr. www.sspess///e.e. /atasss/vO/aae axres//80//guuyen18bbbb html
Quynh Nguyen, Pierre Bréchet, and Marco Mondelli. When are solutions connected in deep net- works? In Advances in Neural Information Processing Systems, volume 34, pp. ran Associates, Inc., 2021. URL https: http://cceennneeeeeei......eo/ee httpp/reee./apppaaaaaaaaapppp//p filees//apprr/22///iee Paper pdf
Cur-
Razvan Pascanu, Guido Montúlar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. In International Conference on Learning Representa- tions, 2014. URL
Pawel Piwek, Adam Klukowski, and Tianyang Hu, Exact count of boundary pieces of relu classifiers: Towards the proper complexity measure for classification, 2023.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural nolworks In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseorch, pp. 2847 -854.. PMLR, August 2017. URL https://proceeddnng wwwpiiinnssmrr...eee.e a.l..press//7O//rguuul... html
Johannes Rau, Arthur Renaudincau, and Kris Shaw. Real phase structures on matroid fans and matroid orientations. Journal of the London Mathematical Sociely, 106(4))36873377 3710, 2022. doi: 10.1112/jlms. 12671.
Jürgen Richter-Gebert and Günter M. Ziegler. Realization spaces of 4-polytopes are universal. Bulletin of the American Mathematical Society, 32(4):403 412, 1995. doi: 10.1099//02233-0799 1995-99999--09--4--
David Rolnick and Konrad Kording- Reverso-ongincering deep ReLU networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8178 8187. PMLRR July 2020. URL https://proceedingss mlr
34

---

Published in Transactions on Machine Learning Research (09/2024)
Itay Safran and Ohad Shamir. Spurious local minima common in two-layer ReLU neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4433 4441. PMLR, July 2018. URL https://proceeddggs... mml press/
v80/ 280/safran18.herr html
The Sage Developers. SageMath, the Sage Malhemalics https://wwer sagemath. org.
Software System (Version 9.4), 2021.
Nils Schlüter and Moritz Schulze Darup. Novel convex decomposition of piecewise affine functions. In Proc. of the 21st IFAC World Congress, 2020.
Thiago Serra, Christian Tjandraatmadja. and Srikumar Ramalingam. Bounding and counting linear regions deep neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3558-4566. PMLR, July 2018. URL https: //proceedings.
wwwpiiiinnnmmm.l....mmmmmem mmr. www..pees//ee..mmeem. http///80//eerra11/ wwr.eerrrrbbammmmm
Ali Siahkamari, Aditya Gangrade, Brian Kulis, and Venkatesh Saligrama. Piccewise linear regression via difference of convex functions. In Proceedings of the 37th International Conference on Machine Leurning, volume 119 of Proceedings of Machine Learning Research, pp. 8895 8904. PMLR, July 2020. URL https:
//proceedings.cnt mlr. ppwess/vv19/sssseeeeeepeesseeess wttp://s.....reee//////////////e
1aaeii22a.hhhh. html
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape overparameterized neural networks: Sym- metries and invariances. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9722 9732. PMLR, July 2021. URL https://proceediin mlr www.ppree//vee. wttppp/aae//////eep. ptress/v139/simssk2aaa. wtwsssmmmssaaaaaammme
Eduardo Sontag and Héctor J. Sussmann. Backpropagation can give rise to spurious local minima even for networks without hidden layers. Complez Syst., 3, 1989, URL https://wwwww... http://w.woppppppoooeecoeeeeeeee com/
abstracts/v03_101 1tt_vv0000110000000 _
Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks, 2018. URL https:: httt::///pppuuoooee..eeer/rrooop het::forrummiic-ihffmmmmsa
David Speyer and Lauren Williams. The Tropical Totally Positive Grassmannian. Journal of Algebraic Combinatorics, 22(2):189-2114 September 2005. doi:
Matus Telgarsky. benefits of depth in neural networks. In 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pp. 1517-1539. PMLR, June 2016. URL https:
//proceedings. www.iinng.mmm....mmmmmm.. wtw.eeediiinnnm..reeeeeeeeeeeeee wtt../eeeee////////aaaaaaa///// html
Ngoc Mai Tran and Jidong Wang. Minimal representations of tropical rational functions, 2023.
V. N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability 8 Its Applications, 16(2):264- (222)2242280 1971. doi: 10.1137/111600
Oleg Viro. Patchworking real algebraic varieties., November 2006.
Uli Wagner. k-sets and k-facets. In Jacob E Goodman, János Pach, and Richard Pollack (eds.), Surveys on Discrete and Computational Geometry, volume 453, pp. 443 514. American Mathematical Society, 2008. doi: 10.1090/ //oc///o////
Shuming Wang. General constructive representations for contimuous piecewise- linear functions. IEEE Trans- actions on Circuits and Systems I: Regulor Papers, 180):)888-1196 2004. doi: 10.1109/TCST.2004.85555
Yifei Wang, Jonathan Lacotte, and Mert Pilanci. The hidden convex optimization landscape of regularized two-layer ReLU networks: an exact characterization of optimal solutions. In International Conference on Learning Representations, 2022. URL
35

---

Published in Transactions on Machine Learning Research (09/2024)
Thomas Zaslavsky. Facing up arrangements: face-count formulas for partitions of space by hyperplanes. Memoirs of the American Mathemalical Sociely, (issue 1, 154):vii+102, 1975. doi: 2.1090/memo//ii
Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim. Tropical geometry of deep neural networks. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5822-5832. PMLR, July 2018. URL http://pproceeddngg. mlr www.ppresssseeeee hatpa///vv///aag18..18 /t////aaannnnneee/ .tmll
Günter M. Ziegler. Lectures on polytopes. Springer- Vpringer-Verlag, New York, 1995. URL https: http:p///iink..prinnng... com/book/ 10010/97011000000-----11111 8431-1
36


Reviews:
[{"review_id": "629W9whc6X", "reviewer": "Reviewer_62qL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is very well-written and most concepts are introduced intuitively. To my knowledge, the results are novel: certainly the connection between neural networks and tropical geometry has been explored, but I am not aware of any works formally making geometric statements about binary classification. I believe the formalism can serve as a playground for interesting follow up works.\n\nI also found the paper very well referenced. Most of the works I’m familiar with that seemed relevant to me were cited, and many general references and reviews are provided for the interested reader.\n\nFinally, I particularly enjoyed some of the proofs, for example Theorem 3.2 or Theorem 5.14. Both of them were very clearly presented.\n\n**Weaknesses**\n\nClearly the work is very, very dense, and on a topic most people are not familiar with. It is noted that part of the intended audience is machine learning people with interest in combinatorics. However, most people coming from a machine learning background will be completely lost starting at Theorem 2.1. Some visualizations of the terms or further explanations would go a long way. Page 4 would greatly benefit from further visualizations. For example, for a point p in $\\mathbb{R}^2$ the authors could show the hyperplane, in the next subfigure show the hyperplane arrangement (considering a few more points), next show the fan/cones, and then walls. That would greatly help readers create a mental picture of the concepts. Similarly, visualizations of the Newton polytope and its construction can greatly aid understanding.\n\nThere is a relative lack of coherence in the presentation of the paper. At times, sections feel like independent results with no direct connection. For example, it is not exactly clear to me how the two views that the authors study (the subdivision of the parameter set induced by the boundary and the subdivision by the activation patterns) are connected. Another example is practically Section 5.2. It feels disconnected from the rest of the text: here, tropical hyperplanes are considered, in contrast with the rest of the text, but if I’m understanding things correctly, Section 6 switches back to regular hyperplanes. Finally, Section 3 is a bit disconnected. Section 2 presents results in linear classifiers, and then analogues to these results are drawn in Sections 4 and Sections 5, 6. Section 3 does show the correspondence between piecewise linear neural networks and tropical geometry, which is needed. However, considering these correspondences have been shown before (at the level of showcasing them), the section feels unnecessarily long. Then Theorem 3.2 is presented, which feels very disconnected from the rest of the results. This is, in my opinion, an issue, as it is very easy to get lost with the big picture and what the end goal is, with Section 3 being a large contributor to this effect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a binary classification problem from the lens of tropical geometry. In particular, the work draws connections between characterization results on linear classifiers and piecewise linear neural networks using tropical geometry. The authors focus mainly on two directions: in the first, a decision boundary is fixed, which induces a subdivision of the parameter space into semialgebraic sets. In the second one, the emphasis is placed on the polyhedral fan induced by the activation patterns. At a higher level, the authors’ analyses allows them to make formal statements about which parameter sets can lead to perfect classification, but also describe the level-sets, which yields statements about optimality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes**\n\nFirst and foremost, I think the work would greatly benefit from the visualization sequence I mentioned in the Weaknesses section. You could also visualize the Newton polytopes as I again think that will greatly help with readability, but I leave that to your discretion. For Section 5.2, I think again the paper would greatly benefit by the visualization of a tropical hyperplane, as I’m sure most readers are not familiar with those.\n\nAnother suggestion would be to rephrase the abstract. Currently, it is very dense and hard to convey the main contributions. The second and third sentences are hard to parse without context. Along a similar vein, it is a bit unclear exactly how all the results are connected, and essentially what is the impact. I believe that the description alone is interesting, however what are the takeaways and intuitions out of it? It would be very helpful for the work, I believe, to emphasize the contributions and the intuition/impact behind them, especially in the abstract and introduction.\n\n**Comments**\n\nI believe that Charisopoulos and Maragos (2018) also discussed maxout units and if memory serves also had results on the number of linear regions. I’m mentioning this as in the second paragraph of the introduction it is stated that this relation was first proposed by Montufar et al. (2022). Still on the topic of references, the earliest reference I’m aware of for function representations as differences of convex functions is by Hartman, “On functions representable as a difference of convex functions”, 1959. I think that one should be included alongside the other relevant references.\n\nAt the bottom of page 7 it is stated that “Given such a fixed architecture…”. The definition of $\\operatorname{ReLU}(d_i)$ I’m assuming contains the piecewise linear functions when $m, n \\to \\infty$, and that hardly sounds like a fixed architecture. I think the authors agree with that statement, hence the parameter space is defined as $\\Theta(d, n, m)$ in the case of neural networks, unlike the case of the data. I think there are two issues with this: one, I don’t think it’s exactly accurate to treat this infinite set as corresponding to a fixed architecture, and two I think it hurts readability; I think referring to that as a “fixed architecture” would greatly confuse machine learning audiences.\n\nBefore Definition 6.1 it is stated that given an activation pattern $G$, a coloring is induced. However, it seems that the coloring is independent of $G$ (since nodes are colored by $g$ or $h$). Concretely, the complaint is the coloring is independent of the activation patten (which is listed as $G$), and only depends on $V(G)$. The complaint is consistent with Lemma 6.3, which fixed a partition and then an activation pattern.\n\nFinally, there is no conclusion, which seems peculiar. I think adding one would help once more to emphasize the contributions and help readers conceptualize the connections between the results.\n\n**Minor questions**\n\nI had a question about the oriented matroid: it seems that there needs to exist a cone that places all points in the data set on the boundary (C I). It’s not clear to me how this can be true in general configurations. Wouldn’t that require all the points to lie on the hull?\n\nI’m also a bit skeptical of a statement in the proof of 6.17. An example is given of points in a general position. However, I would hardly call these points in general position, as $\\\\{p_2, p_3, p_4, p_5, p_6, p_8\\\\}$ are roughly colinear, and so are $\\\\{p_1, p_4, p_5, p_7, p_9\\\\}$.\n\n**Mistakes**\n\n- Second paragraph of second page “if the data is linearly separable” (data are plural, datum is singular, but understandably archaic)\n- Same paragraph, “For networks with piecewise linear activation” (missing s)\n- In Example 4.1 the definition of $f$ is different from the preceding section and is not a tropical signomial.\n- Typo final line before 6.3 (should say functions).\n- Below figure 5 “Thus, in the bound is not attained”. (remove in)", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZqtM6sXTmh", "reviewer": "Reviewer_CXWW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The two main issues for this submission are: (1) are the results\npresented correct? and (2) are of they of sufficient\ninterest/significance for publication in TMLR?\n\nConcerning (1) I found no errors in the proofs. Since I, unlike quite\na few other people, do not work on the combinatorics of neural\nnetworks, I don't know whether some of these results have appeared\nelsewhere. However, given the authors' evident extensive knowledge of\nthis field, I think it unlikely that any of the paper's purported\ncontributions are re-discoveries. (The authors note some rediscoveries\nfound elsewhere, so they are presumably aware of this 'risk'.)\n\nTheorem 3.2 (the converse of the already-known Theorem 3.1) is a\nuseful, certainly non-obvious (to me) result.  The proof is long and\ncomplex, but perhaps there is no simpler proof. I found Theorem 6.17\nthe most interesting, even though its truth is not too surprising.\n\nThe Introduction section does a nice job of introducing the research\narea with an excellent, extensive set of references. Generally, the\nwriting is excellent and there are virtually no grammatical or\ntypographical errors. The examples that are dotted around the paper\nare useful and welcome. However, the paper ends somewhat abruptly; I\nthink some pointers to future work would be a good idea.\n\nOne of the target audiences for the paper is: \"A data-driven audience\nwith a background in machine learning interested in understanding the\nunderlying combinatorics\". Such a person might have to do a lot of\n'homework' before they could use this paper to gain such an\nunderstanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contributions are a collection of conceptualisations and theorems connecting concepts from tropical geometry to neural networks. See abstract for details. The key idea of this paper is find connections: that between activation patterns and sets of covectors of oriented and tropical oriented matroids is just one example.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suspect many data-driven people would be unfamiliar\neven with the terms such as fan, chambers, zonotopes, etc. (Note that\ne.g the term \"chambers\" is used quite early on, but not defined until\npage 4.) I suggest the authors explicitly point such people to where\nthey can learn the basics.\n\nI encourage the authors to make an argument for theoretical results to\nthose, the majority, working on neural networks who are most concerned\nwith empirical performance. For example, the difference of two convex\npiecewise linear functions is called a tropical rational function. But\nwhat do we gain by so describing a difference of two convex piecewise\nlinear functions? It allows us to exploit known results from tropical\ngeometry: but how, if at all, can use them to get better results when\napplying neural networks. But what one hopes from finding\na connection between neural networks and some area of mathematics is\nto use theorems from that area to find previously unknown properties\nof neural networks which can, at least in the long term, inform the\npractice of neural networks.\n\n\nSMALL POINTS\n\nProp 5.10: \"span\" should be in Roman font.\nProof of Prop 5.11: Odd to start with \"Then\"\nProof of Theorem 6.12:\n\"Thus in the bound is not attained\" -> \"Thus the bound is not attained\"\n\nThe English is neither consistently American nor consistently\nBritish. We have: \"coloring\", \"neighbors\" (US English) but also\n\"labelling\" \"analogues\" (UK English). I personally am not too bothered\nabout this, but some might find it jarring.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ehj1tFuWQD", "reviewer": "Reviewer_jJJV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a technically solid paper that proves many theorems. However, while the paper is well-written, the details are difficult to follow due to its technical nature. Below are more detailed comments, which I hope will be useful.\n\n- Perhaps the paper could have a more suggestive title. The current title does not say more than the paper of Zhang et al. (2018), which is titled \"Tropical Geometry of Neural Networks\". Clearly, the present manuscript builds upon Zhang et al. (2018) and has deeper explorations in many aspects. Personally, I think the paper's content is more about classification and less about ReLU neural networks, so maybe including\"classification\" in the title is a good idea. \n\n- The notion of *polyhedral fan* is not defined. I only know what it means through Wikipedia (https://en.wikipedia.org/wiki/Polyhedral_complex). However, the Wikipedia page does not define *maximal cones* of a polyhedral fan. It would be nice if the authors could add a definition of it, therefore making the paper more self-contained.\n- Does the wall of a polyhedral fan always exist? It seems to be the case, hence it would be great if the author could comment on that when defining it.\n- I found the one-dimensional example given after Theorem 2.3 quite interesting. To make the paper easier to comprehend, it would be great to have similar examples for walls and paths even before introducing Theorem 2.3. Even better: Give such examples any time when introducing a notation and if appropriate.\n- In Theorem 3.2, the bound on $m$ seems to be very large. Could the authors comment on its tightness? On the other hand, how fast the parameter space $\\Theta(d,2m, m)$ grow with respect to $m$? Answering this would give us some idea of whether the bound on $m$ is good or not.\n- The phrase \"stratification\" or \"stratified into\" is not very clear and it is unclear what it means mathematically. Could the authors make it precise? Otherwise it is difficult to understand Theorem 4.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers deep neural networks from the perspective of tropical geometry. Its contributions are mainly theoretical, and the paper derives several theoretical results. For example:\n- Theorem 2.3 characterizes a relationship between linear separability and connectedness of loss sublevel sets through walls and furthermore provides a bound on the error along a given path.\n- Theorem 3.2 concerns whether deep ReLU networks can be represented by tropical rational functions.\n- Theorem 4.2 provides some characterization of the decision boundary via Newton polytopes.\n- Section 5 contains multiple results characterizing some geometric properties of activation fas and activation polytopes.\n- Section 6 contains multiple results for classification fans, and an interesting result is Theorem 6.7, showing that the sublevel sets are not always strongly connected.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above.", "broader_impact_concerns": "The paper is theoretical and it does not have ethical implications to my knowledge.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the results are novel and insightful. The authors have addressed the reviewers' comments, including adding figures and further clarifications where appropriate. I am thus recommending acceptance.
Decision: Accept (poster)