On Space Folds of ReLU Neural Networks
Authors: Michal Lewandowski
Hamid Eghbalzadeh
Bernhard Heinzl
Raphael Pisoni
Bernhard A. Moser
Venue: Accepted by TMLR ()

Abstract: Recent findings suggest that the consecutive layers of ReLU neural networks can be understood geometrically as space folding transformations of the input space, revealing patterns of self-similarity. In this paper, we present the first quantitative analysis of this space folding phenomenon in ReLU neural networks. Our approach focuses on examining how straight paths in the Euclidean input space are mapped to their counterparts in the Hamming activation space. In this process, the convexity of straight lines is generally lost, giving rise to non-convex folding behavior. To quantify this effect, we introduce a novel measure based on range metrics, similar to those used in the study of random walks, and provide the proof for the equivalence of convexity notions between the input and activation spaces. Furthermore, we provide empirical analysis on a geometrical analysis benchmark (CantorNet) as well as an image classification benchmark (MNIST). Our work advances the understanding of the activation space in ReLU neural networks by leveraging the phenomena of geometric folding, providing valuable insights on how these models process input information.

Published in Transactions on Machine Learning Research (02/2025)
On Space Folds of ReLU Neural Networks
Michal Lewandowski
michhallovaaddosski@@ccchaal
Software Competence Center Hagenbery (SCCH)
Hamid Eghbalzadeh Al at Meta
heghbal:@@meta.com
Bernhard Heinzl
berukard.keennaat@ccch...
Software Competence Center Hagenberg (SCCH)
Raphael Pisoni
ruphael.pisoniiSscchuu
Software Competence Center Hagenbery (SCCH)
Bernhard A.Moser
monnuard.mosrr&scchh..
Software Competence Center Hagenberg (SCCH)
Johannes Kepler University of Linz (JKU)
Reviewed on OpenReview: https: // openreview. net/ farmm? id- RFFg
Abstract
Recent findings suggest that the consecutive layers of ReLU neural networks can be under- stood geometrically as space folding transformations of the input space, revealing patterns of self-similarity. In this paper, we present the first quantitative analysis of this space folding phenomenon in ReLU neural networks. Our approach focuses on examining how straight paths in the Euclidean input space mapped to their counterparts in the Hamming acti- vation space. In this process, the convexity of straight lines is generally lost, giving rise to non non-convex folding behavior. To quantify this effect, we introduce a novel measure based on range metrics, similar to those used in the study of random walks, and provide the proof for the equivalence of convexity notions between the imput and activation spaces. Furthermore, we provide empirical analysis on a geometrical analysis benchmark (CantorNet) as well as an image classification benchmark (MNIST). Our work advances the understanding of the activation space in ReLU neural networks by leveraging the phenomena of geometric folding, providing valuable insights on how these models process input information.
1 Introduction
Neural networks are inspired by the biological structure of the brain (Rosenblatt 1958 standing performance across various domains, including computer vision (Krizhevsky et al. 2012) and speech recognition (Maas et al.) 2013). However, despite these impressive results, their underlying mechanisms re- main poorly understood from a mathematical perspective, and current. advances lack solid foundation
They achieve out-
rigorous mathematical analysis (Zhang al. 2017 Neyshabur et al. 2017 Marcus 2018
Sejnowski 2020)
As will discuss in this work, geometric folding provide valuable insights and serve useful tool to expand our understanding of neural networks. The phenomena of geometric folding can be described as the process by which a structure undergoes a transformation from a lincar or planar form into a more compact, layered configuration, where space is elficiently organized through recursive bending or folding patterns. For example, in biological systems, DNA molecules fold into complex yet highly organized shapes fit within the confines of a cell mucleus (Dekker al. (013). Further, also proteins fold into precise, three-dimensional shapes, transforming from linear amino acid chains into complex structures essential for their specifie functions (Crescenzi et al. 1998 Dill et al. 2008 Jumper et al. 2021). Folding has been argued

---

Published in Transactions on Machine Learning Research (02/2025)
to appear in neural networks, as the layering of data representations across the network depth allows for increasingly abstract, compact, and hierarchical information encoding, capturing patterns at multiple scales. It was proposed more than a decade ago, that successive layers of ReLU neural networks can be interpreted as folding operators (Montúfar et al. 2014 Raghu et al. 2017 These folds result in the replication of shapes formed by the network and contribute to understanding how the space is folded, which can help reveal symmetries the decision boundaries that the network learns. [Kuu and Hclias (2022) likened this process to the physical process of paper folding, where the input space is "folded" during learning. However, folding occurred in neural networks is elusive in the contimuous input space. In case of protein folding, this process has been quantified using discrete mathematics sequences of amino acids (Crescenzi et al. 1998). For neural ReLU networks, the activation space offers a possibility for further analyses of this phenomenon. In this paper, we focus on the activation space to investigale symmetries and self-similarity in the learned regions (Balestriero et al., 2024).. The activation space and its related linear regions have also been used also as a mcasure of the network's expressivity (c.g., Montúfar et al. (2014); Raghu et al. (2017): Hanin and Rolnick (2019a)).
Insofar the concept of space folding by neural networks remains largely qualitative, with no prior work attempting to quantify these effects. In this paper, we introduce the first method for mcasuring these transformations using range measures (Weyl 1916 Moser 2012) and discrete mathematics. Our analysis is based on a topological investigation of the activation patterns along a straight path in the activation space. In the Euclidean space the shortest path between two points is a straight line. Walking along such a path without turns monotonically increases the distance to the starting point. However, this observation longer applies in the activation space. During the folding operation, the convexity of the created linear regions (defined in Sec. B may not be preserved and the Hamming distance straight path between two (uon-sdjacent) patterns can decrease. This lack of preservation inspires the introduction of our space folding measure, which measures the deviations from convexity on a straight path in both the input (Euclidean) and the activation (Hamming) spaces. In summary, our contributions in this work are as follows:
We prove the equivalence of convexity notions between the input and activation spaces.
We introduce space folding measure to quantify local deviations from convexity in the activation space of ReLU networks. We provide both local and global versions of our measure, We further provide the exact algorithm for its computation, together with its complexity analysis, and suggest the ways of reducing this complexity through intra-class clustering of samples.
We experimentally investigate the behaviour of our measure on (i) CantorNet, specially constructed synthetic example with an arbitrarily ragged decision surface, and (H) ReLU networks with varying depth and width with constant number of hidden neurons trained on the MNIST benchmark. We then extend the analysis to larger networks and find that, although the folding values do not change much, the ratio of inter-digit paths that exhibit the folding effects increases significantly.
The remainder of the paper is organized as follows: Sec, 2 details the related work; Sec. 3 recalls some basic facts and fixes notation for the rest of the paper; Sec. establishes convexity results, Sec. 3 introduces the space folding measure; Sec. p describes the experimental results and discusses how the measure extends to different types of layers; Sec. 7 outlines the future work; Sec. / provides concluding remarks, the take-away message of our work and possible future directions for our work. In Appendices, Appendix A briefly intro- duces CantorNet (following Lewandowski et al.] (2024)); Appendix B details results obtained on the MNIST dataset; Appendix C provides a preliminary study of the sensitivity of measure to intra-class clustering of samples; Appendix D presents the results obtained for larger networks.
2 Related work
Activation Space. The pioneering study by Makhoul et al. has investigated partitioning the input space with neural networks with 2-hidden-layers, thresholding neurons with the ReLU activation function (without naming it). With two hidden layers, the first layer creates hyperplanes that divide the input space into regions, adjacent if they have Hamming distance of (Makhoul et al. 1991). Connected regions are
2

---

Published in Transactions on Machine Learning Research (02/2025)
defined by a path through adjacent regions. The interest in the mumber of these regions was revived in 2014 by Montúfar et al., with several follow-up works, e.g., (Raghu et al., 2017; Serra et. al. 2018 Xiong et al. 2020 Hanin and Rolnick, 2019a b). The authors provided ever tighter bounds on the number of activation regions, and used them as a proxy for its expressiveness, among others.
Space Folds. The idea of folding the space has been investigated in the computational geometry (Demaine itt al. 2000). Demaine and Demaine surveyed the phenomenon, focusing on the type of object being folded, e.g.: paper, or polyhedra. Bern and Hayes] explored whether given crease pattern can be folded into flat origami (non-crossing polygons in 2D with layers). Later, Bern and Hayes] showed that any compact, orientable, piecewise-lincar 2-manifold with a Euclidean metric can achieve this structure. In Montútar et al. (2014) in Section 2.4, the authors briefly mentioned the folding phenomena, although through the lens of linear regions. They argue that each hidden layer in a neural network acts as a folding operator, recur- sively collapsing input-space regions. This folding depends on the network's weights, biases, and activation functions, resulting in input regions that vary in size and orientation, highlighting the network's 's flexible par- titioning. Tn Phuong and Lampert (2020), in the Appendix A.2 the authors explored the folding operation by ReLU neural networks, but leave the exploration quite early on. In Keup and Helias (2022), the authors argued that it is through the folding operation that the neural networks arrive at their approximation power.
Self-Similarity and Symmetry. Self-similarity and symmetry are related but distinct concepts, often found in nature, mathematics, and physics. Self-similarity means that a structure or pattern looks similar to itself different scales, and is also present in numerical data, e.g., images (Wang et al.) 2020), audio tracks (Foote, 1999) or videos (Alemán-Flores and Álvarez León 2004). Symmetry implies that object or pattern is invariant under certain transtormations, e.g., reflection, rotation, translation. In the context of neural nelworks, in Grigsby et al.] (2023), the authors describe a number of mechanisms through which hidden symmetries can arise. Their experiments indicate that the probability that a network has no hidden symmetrics decrcases towards O as depth increases, while increasing towards as width and input dimension increase. Many fractal shapes, such as the Mandelbrot set (Mandelbrot, 1983) or CantorNet (Lewandowski et al. 2024), exhibit both self-similarity and certain symmetries. Moreover, both concepts relate to the folding operation: invariance under reflection (symmetry) can be equivalently understood as a space fold, and self-similarity can be interpreted as recursive folding scaling operations that replicate the pattern across different levels. In neural network architectures, these principles can manifest through hierarchical structures, where each layer effectively "folds" information from previous layers, producing patterns that may repeat or reflect across layers or nodes (Raghu et al. 2017).
Distance Alteration. Lipschitz constant, a well established concept in mathematical analysis, bounds how much function's output can change in proportion to a change in its input. In context of neural networks, it has been linked to adversarial robustness, e.g.; (Tsuzuku et. al., 2018 Virmaux and Scaman 2018), generalization properties, e.g., (Bonicelli et al., 2022). Cisse et al. showed that the Lipschitz constant of neural network can grow exponentially with its depth. Anil et al. observe that enforcing the Lipschitz property leads to some limitations, and show that norn-constrained ReLU networks are less expressive than unconstrained ones. The exact computation of the Lipschitz constant, even for shallow neural networks (two layers), is NP-hard (Virmaux and Scaman 2018). In Gamba et al. (2023), the authors experimentally studied the (empirical) Lipschitz constant of deep networks undergoing double descent, and highlighted non- monotonic trends strgly correlating with the test error. Finally, |Hanin et al.] prove that the expected length distortion slightly shrinks for ReLU networks with standard random initialization, building the results of |Price and Tanner) While important, none of the aforementioned works touch on the activation space of neural networks, nor do they investigate the monotonicity of a mapped straight line. Our analysis extends beyond the concept of the Lipschitz constant by exploring the convolution of the input space under a neural network.
3 Preliminaries
We define a ReLU neural network N X + y with the total number of N neurons as an alternating composition of the ReLU function (a)) == max(x, 0) applied element-wise on the input æ, and affine functions
3

---

Published in Transactions on Machine Learning Research (02/2025)
Euclidean Input Space
Hamming Activation Space
TT.1
T3
W2 (010)
(110)
π3
h2
+
(111)
(000)
71 72
hi
Figure 1: Illustration of walk straight path in the Euclidean input space and the Hamming activation space. Left: the dotted line represent the shortest path in the Euclidean space. The arrows represent shortest path in the Hamming distance between activation patterns and (note that in the Hamming space the notion of the shortest path becomes ambiguous). Right: The illustration of a shortest path connecting #i and n in the Hamming activation space.
with weights We and biases by at layer k. An input x E X' propagated through N generates non-negative activation values on each neuron. binarization a mapping क g + {0, 1 applied to a vector (UI...., RN, resulting in a binary vector by clipping strictly positive entries of v to 1, and non-positive entries to 0, that is x(u;) is the concatenation of all neurons of all hidden layers, called activation pattern, and it represents element in a binary hypercube HN = {0, 13 where the dimensionality is equal to the nunber N of hidden neurons in network N. A linear region is an element of partition covering the input domain where the network behaves as an affine function (Montúfar et al.] 2014)) (see Fig. 1 left). The Hamming distance, dæ (u, v) = [u, * Us for = 1,...., N,,, measures the difference between u, v E HN.
if u, > 0, and π T(;)) 0 otherwise. In our case, the vector v
4 Convexity
Convexity is a key concept in computational geometry and plays a critical role in various computer engineer- ing applications, such as robotics, computer graphics, and optimization (Boissonnat and Yvince 1998). In Euclidean space, convexily can be defined property of sets that are closed under convex combinations, where the set contains all line segments between any two points within it (Roy and Stell 2003). We extend this notion of convexity to the Hamming space as follows.
Definition 1 (Adapted from Moser et al. (002)) ). A subset S of the Hamming cube H is conver if. for every paär of points y E S, all (observable) points every shortest palh between n, are also in s
Example 1. Consider points = (000) and T4 = (111) (Fig. 1 right). Then the Hamming distance d (TI, = 3. Every shortest path consists of three edges, flipping one "bit" at time, and thus a convex set is the whole Hamminng cube H3.
Example 2. Consider activation patterns कι = (0111), π2 = (0001), = (1011) as shoun in Fig. ssee Appendir A for more details). For a walk through any two of the activation patterns, (1) गi + T2, (2) π2 wa, (3) ग + æa, there are intermediate activation potterns Tinter that we traverse. They are, respectively: (1) Tinter {(0011), (0101)), (2) Tenter = {(0011), (1001)}, (3) Tinter (0011), (1111)}- As none of them is contained in (TA, TT, ne), we conclude that they are non-obseruable the considered domain, (0. 1] (0, Thus, the activation potterns {M1, n2, ns} form conver set in the Hamming cube sense.
In the discrete activation space might happen that some points (L..., binary vectors) are non-observable (e.g., point (011)
the tessellation in Fig. []
4

---

Published in Transactions on Machine Learning Research (02/2025)
Wag W1
Figure 2: Activation patterns т. of rocursion-based representation of CantorNet (sce Appendix A We skip neurons with unchanged values. The colours are used for increased visibility; activation palterns { n2, ma are convex in the Hamming cube sense (see Ex. 2). The darker gray of Tī has been used to visually distinguish from T4 and Te- (Adapted from Lewandowski et al. (2024) with the authors' approval.)
Before introducing the space folding measure, which relies on the notion of convexity, prove the equivalance of convexity notions between the input and activation spaces for hyperplanes that intersect the entire input space. This further justifics the need of deeper layers to observe any space folding effects.
Lemma 1. Consider tessellation of activation regions formed by N hyperplanes hi...., by with activation regions R Rr. C IR" and corresponding activation patterns A= ( mr}. A union R= UreA R of activation regions is conver in IR" if and only if the set A of corresponding activation patterns is conver the Hamming space H
Proof. Convexily in Rd 3 Convexity in Hamming space:
Assune that the set R = UTEA Rg is convex in R". We want to show that the set A is convex in the Hamming space. We start with showing the connectivity of A. Let n,, T, E A be any two activation regions, and choose any points P Rs, and Q E Rx, in respective activation regions. Since R is convex, the line segment [P, Q) lies entirely within R. As move along [P.Q]: activation state changes. Each such crossing corresponds to flipping exactly one bit in the activation pattern (see Fig. 3.. This sequence of bit flips forms a path in the Hamming space from म; to Tj, showing that A is connected. Let us now show the convexity of A. Assume, for contradiction, that A is not convex in the Hamming space. Then, there exists a shortest path in the Hamming space connecting म; and my that leaves A; that is, some activation patterns along 7 are not correspouding to the line segment [P. stays entirely within A, as corresponds activation patterns of points within R. Since [P.Q] is straight line, it corresponds shortest path in the Hamming space). Therefore, there exists a shortest path within A, contradicting the assumption. Hence, A is convex in the Hamming space.
may cross hyperplancs h& where the
However, from connectivity, the path
minimal sequence of bit flips (i.c.,
Convexity in Hamming space = Convexity in R":
Now assume that the set of activation patterns convex the Hamming space. We want show that the union R UrEA R2 convex in R Let P.Q E R be any two points, and denote by Rm, R E their activation patterns. Consider the line segment [P. in R". As we move from to Q, we may cross hyperplanes hgg changing activation patterns. Each crossing of hyperplane h& corresponds to flipping bit in the activation pattern, forming a path in the Hamming space from Tp to TQ, as previously (again, see Fig, 3) Since A is convex in the Hamming space, all shortest paths between Tp and TQ remain within A, and in particular, the sequence of activation patterns along [P, Q) is such a shortest path. Therefore, all activation patterns along [P, Q] are in A. Since every point along [P, Q] has an activation pattern in A, it lies within R. Thus, [P, Q] C R, showing that is convex in R..
5

---

Published in Transactions on Machine Learning Research (02/2025)
Rio
h3
h2
+
R.
RT
p
h1
The shaded gray area illustrates a convex set the Fuclidean space. The hyperplanes h1, h2,, las intersect the entire imput space (it holds for the hyperplanes described by neurons from the first hidden layer of ReLU neural network). A straight line [P, Q] connecting points P and Q crosses hyperplanes hı and ha, resulting in "bit" flip at time.
Figure 3:
Lemma important for two reasons. First, its direct consequence is that the space folding effects are observable only in networks with at least two hidden layers. Moreover, it proposes another angle to see the classic XOR problem @Minkky and Papert, 1969), and why one layer is insufficient for class separation. Second, observe limitations of Lemina 1 We assume that along a walk on shortest path (in the Hamming sense) between any activation patterns corresponding to linear regions in a convex arrangement AA in the Euclidean input space, the Hamming distance of adjacent lincar regions diffors by onc, which only holds in case of hyperplanes that intersect the entire input space, and such hyperplanes are described by the 1st hidden layer of a ReLU neural network (e.g., Raghu et al. (2017)). For deeper layers, there may appear activation regions, which, although neighboring, may have the Hamming distance exceeding one (see Example 2
5 Analysis in the Activation Space
Range Mcasures. Our space folding measure is inspired by the construction range measures of random walks. Consider a walk along a line given by the sequence s = (s of N steps length 8; at 7th step. At each step i the walk can go either up or down. The maximum absolute route amplitude of the walk given by TA == max,5N Ey dependent on the choice of the coordinate system. However, we can remove this dependency by considering (D(s) == max {E== Sk, 0} min (En $x. 0} = maX Ei syle which represents the diameter the walk invariant under translational coordinate transformations (Moser 2014 2017p. Both (s) and (s) are examples of range metrics encountered in the field of random walks in terms of an asymptotic distribution resulting from diffusion process (Jain and Orey
1968). It turns out that
(s) and (s) are norms, || A and ||s|| D. respectively (Alexiewicz) 1948
Note that
||
< ||||| 2||s||A.
stating the norm-equivalence of II-Ila and ||-I|D.
Example 3. A simple example of a range measure is a variance estimotor of sample with n observations, x (Z1. (Z1....,IA)) Rn), defined as Var(x) m(n-1) Em (z: - z)2, where & is the sample's 's arithmetic average.
The Space Folding Measure. Consider straight line connecting two samples X2 in the Euclidean input space realized convex combination (I Ax)x1 + X where X; are equally spaced on [0, 1] (the equal spacing is due to practicality, but is not necessary). Then, consider the mapping of the straight line [xi, X2] to path in the Hamming activation space, with intermediate points (Tı,..., m, E H under a neural network A (see Fig. left). We consider a change in the Hamming distance at each step i
A = de (T1+1, T1) de (1))
(1)

---

Published in Transactions on Machine Learning Research (02/2025)
We then look at the maximum of the cumulative change max; Σi=1 As along the path T,
ri (T))
Σ (d -d (T) iE =e(...,n11y iE(
max
max (T(
(2)
The above expression equals to the maximum Hamming distance the starting point reached along the path. Next, we keep track of the total distance traveled on the hypercube when following the path,
(2 Σ du m411)-
For the measure of space flatness, consider the ratio @(T) = (TT)/r2())) Equivalently, the space folding measure equals
x(T) x- 1 Q(T) = - max iE(I...)
du( mT)
Σ dn T441)
(4)
Lemma Mariaagguuraann guarantees that a straight line in both the input and the activation space is convex, and measures the deviation from convexity along this path, effectively measuring deviation from flatness, hence its name. The higher X is, the more folded the space is along the path r. We say that the space is flat is not folded, and in that sense "folding" opposite to "flatness".
Lemma 2. For every path I the space folding measure satisfies 0 S x(T) S (provided that EE (H T++1) > O, i.., the path I' traverses more than one region).
Proof. We only show the upper bound as the lower is obtained in the similar way. From the triangle inequality for any activation patterns TG, T++1 E HN it holds that du (m,, T+41) S dh (T,,))) + (T T(41). Writing this for every index i E f1...., 1}. n > 2, and summing by sides we obtain
Σ du m:41) &
Σ dn mi) + E du 7441) S 2(n 1) max ( (T) m1).
Recall that Σ, du (T,, T+41) > 0 and divide each side by this sum. It follows that
x(T) < 1 1. 2(n-1) - 1)
To understand the motivation behind the construction of the measure, consider a straight path I in the Fuclidean input space that gets mapped to a straight path in the Hamming space. Tn this case, the range measures T1, 72 increase and their ratio (T))/r22T = 1, thus there no space folding, i.c., x(T) = 0. If a straight path the Euclidean path gets mapped to a curved path in the Hamming activation space (Fig. 4 left), we observe non-zero values of the space folding measure X. The space folding can equal x(T) in the case presented in Fig. 4 right. Consider path r (T1, #2, I1, T2, .... Then, the range measure r (T)
and rT) co, hence xTT) 1. Our measure be made global by considering the supremum over all possible paths I in the Hamming activation space, i.e.,
Φx = sup XT).
(5)
TEX

---

Published in Transactions on Machine Learning Research (02/2025)
dg dæ
I (TT, #2, N1, 11,,,,1-. :22...))
π2
dy [TT1 02) =1
max; (T1,
du (1 Tn)
ri (T) max, (TI, =1
X1 [
r2(1') > du m++1)
0
1 A in (I A)X1 Ax2
x(1) 1
Figure 4: Left: Straight line between X1 and X2 in the Euclidean space. Observe that, when mapped to the Hamming activation space (dotted arrows), the Hamming distance may decrease when following the path, i.e., might happen that ( Tn) < max; dir (T), Right: An extreme case when space folding x(T) 1. Note that it is sufficient that Tı (I) for some c R+, and that the path I' is looped between the same regions, resulting in r2(T)- co. This construction, although theoretically possible, might not be realizable in practice.
The Algorithm. In this paragraph, we present the algorithm for computing the space folding mcasure x introduced earlier, along with its associated computational complexity.
Algorithm 1: Computation of the Space Folding Measure (Eq.
Input:
Two input samples X1, X2, the number of intermediate points n, the total number of hidden neurons N, cost of running the network the inference mode O(C)
Output: Space Folding x1)) as in Eq. 4
Step 1: Lincarly interpolate X1 and X2, sampling points;
// Sampling Complexity: O(n)
Step 2: For each sampled point:
begin
Compute the binarisation;
// Binarization Complexity Per Point: O(C)
// Total Binarization Complexity: O(n-O)
Step 3: Compute the maximal (from the starting point) and total Hamming distances between intermediate points; return Space Folding XT);
// Computation of Range Measures Complexity: O(n N)
// Total Algorithm Complexity: 0 (n (N + c))
In the comments for each line, we have included the computational complexity of the corresponding operation.
The complexity of running the network in inference mode is denoted as O(C). The total computational complexity for every pair of samples from classes C1 and C2 is thus
(n (N C) |C1 (C2)),
(6)
where ICil [C2) denote the cardinalities of the respective classes of points x1. X2. Since N C const, the overall computational cost can be controlled by adjusting or by subsampling within the classes. In the following, we establish an upper bound on the number of intermediate steps n. For a neural network with N hidden neurons, consider two input samples x₁ and X2. The maximum Hamming distance between their corresponding activation patterns, Ø1 and T2, equals ( m2) = N. Consequently, the number of lincar regions between inpuls Xı and X₂ at most N. Since intermediate steps that fall within the same activation region do not affect the space folding measure, the number of steps should not exceed N. We note that evenly spacing intermediate points for the computation of the space folding measure x may be suboptimal. Instead, the focus should be on traversing all linear regions between the two samples. Optimization of the path will be addressed in a future submission.
Secondly, to address the (potentially) high cardinality of classes C₁ and C2, we propose the following ap- proach. Let mi |Ci] and m2 |C2], where m, and my represent the number of clusters into which the samples in classes C₁ and C2 grouped. For each cluster in respective class C C2, select. centroid
8

---

Published in Transactions on Machine Learning Research (02/2025)
(qi))=) and (c and compute the space folding measure X between every pair of centroids, instead of using the original samples. By reducing my and m2, we can significantly lower the computational cost of calculating the measure X. The exact impact of this reduction is left for future work, however we provide preliminary results of impact of such grouping on the space folding values in Appendix
6 Experiments
6.1 Experimental Setup
CantorNet. We start the experimental evaluation of our measure on CantorNet, a hand-designed example inspired by the fractal construction of the Cantor set (Lewandowski al., 2024) (see App. A for the sum- mary). First, note that we may consider subset of layers for the computations of the space folding measure;
33
33
72
TT
Figure 5: Activation patterns म; of the recursion- based representation CantorNet. For the computation of the space folding measure X we can consider a subset of layers; left: highlighted activations in the first layer, right: all layers. For a path T: My >> πTç - и4, the folding x(T) = if we consider only the activations from the first layer, while x(T) 3 if consider all the layers. Colours are used for increased visibility; "white" patterns form convex set in the Hamming cube sense (see Ex. 2 We skip neurons with unchanged values.
however, we risk not detecting any folds. Indeed, for CantorNet of the recursion level k = 1, consider a path P Te > F5 > #4 (see Fig. 5) we consider only the activation patterns the first layer, we obtain x(T) = 0, while if we include all the layers, then x(T) 1
Next, set the recursion depth to k 2 (Fig. 6 the background), and create path I between points (0. 2) (1, 3) (Fig. 6 the arrows). We illustrate the evolution of range measures 11,72 along path I (Fig. 6 dotted and dashed curves, respectively), and the corresponding space folding measure (in blue). In this example, we obtain a higher space folding value than for the recursion level = 1, x(T) ss 0.7.
MNIST. Further, study the behavior of the space folding measure ReLU neural nets trained MNIST (LeCun et al. 1998). We keep the number of hidden neurons constant (equal 60), and we experiment with the depth and the width of the network, trying the following architect 2 x 30, 3 x 20, x 15, 5 x 12,6 x 10, 10 x 6, with the notation (no. layers) (no. neurons) P We then train those networks for 30 epochs on pre-defined random scods (in Python NumPy and Torch), and store their parameters. For the networks that we managed to train high validation accuracy (especially for deeper networks it is highly dependent on the initializatinn, we analyze the relationship between the depth of a ReLU network and the aggregated median of maximas of non-zero space folding across all pairs of digits in the MNIST test set (100 pairs, ~ 1M paths I for each pair), obtaining the Pearson correlation coefficient 0.987 for the random seed (equal 4) where the networks performed the best. Interestingly, for the networks that did not reach satisfactory validation accuracy, the aggregated space folding was much lower than for networks that trained
2 As underpinned by Lemma we not expect folding effects network with hidden layer and 60 neurons and hence we omit it.

---

Published in Transactions on Machine Learning Research (02/2025)
Coefficient
-Axx
+
+
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.8
Hamming Distance de
A1
Space Folding x(F)
diim, 17, 1)
du(mı, ))
Folding x(r)
101
T12
Tu
Activation Pattems Along the Path
Figure 6: Behaviour of dær(m -)) for respective (dotted line) and distance between neighboring patterns dæ (Ta, Ts-1) (dashed line) on path I' constructed between points x1 = (0, ) and X2 = (1, () (indicated recursion level k (see App. A For the illustration purposes, it has been scaled to fill the figure, but its domain is a unit square Note that the Hamming distance du between the initial activation pattern can decrease (dotted line), indicating deviations from convexity, discussed previously. The blue line represents the space folding mcasure xT). (Best viewed in colors.)
by arrows). Background represents CantorNet recursion-based representation
[0,1] 00, 1]. The cumulative maximum (Eq- (2)) is dir (Ii(T mg)
well. It hints that the symmetries that arise with depth help with achieving some generalization ability. See Appendix B for more details.
We further perform similar experiments much larger networks with 2 x 300, x 200, for no. layers no. neurons. We observed that, while the folding values do not vary much, the ratio of paths that feature space folding increases dramatically: from 0.35 + 1 and 0.44 + 0.15 for networks 2 x 30 and 3 x 20, respectively, to 0.97 + 0.04 and 0.99 + 0.02 for networks 2 x 300 and x 200, respectively. For more details see Appendix D
6.2
Results
In Fig. wc have shown an intercsting phenomenon. We first mapped a straight-line walk from the Euclidean input space to the Hamming activation space. During this mapping, while walking along the mapped path, we have sometimes observed a decrease in the Hamming distance with respect to the initial input point, while in the input space, the Euclidean distance is increasing. This indicates that there is replication of the activation pattern along the path in the activation space, which call folding. This is important, it allows us to understand how neural networks transform and compress input data, revealing the intrinsic geometric properties of the network's activation space. In the next section, we provide more discussions and hypothesis for these results.
6.3 Discussion
Beyond MLPs. In the previous section, we have provided results on space folding by ReLU neural net- works. So far, we have used all the hidden layers; however, a subset of hidden layers could also be utilized. We now provide additional remarks on the measure's utility when applied to different types layers:
Residual layers: Our study extends straightforwarddy to networks with skip connections (e.g.. ResNet (He et al. 2016)).
Normalization layers: Normalization layers modify the cocfficients of the weight matrices, which consequently alters the overall structure of the tessellation. However, our measure remains applicable before. The exact impact of normalization is left for future study.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Attention layers: While our measure may not be directly applicable to attention layers in its current form, it has been shown that ReLU can be used in self-attention mechanisms. Thus, our measure could potentially be applied to such layers (Shen et al. 2023
7
Future Work
In this work, we have proposed a novel method to measure deviations from convexity between walk a straight line in the Euclidean imput space and its mapping to the Hamming activation space. Further, we have used the introduced measure to analyze deviations from convexity under syuthetic example, Nen and ReLU neural networks trained on MNIST. Our work highlights the convexity transformation of the imput space by a ReLU neural network. In our experiments, we have tried various pairs of digits (intra- and inter- class) and observed qualitatively similar behavior across tested architectures: the max value of X increases with the depth of the network, if it was trained to high validation accuracy, and decrcases if the achieved accuracy was lower. We thus hypothesize that the maximal value of is associated with the network's generalization capacity, as depth has been shown to be necessary (but. insufficient) for generalization in neural networks by enabling deeper layers to learn hierarchical features (Tellarsky 2015 2016). We summarize our findings in the form of a hypothesis (The question) and its evidence (The answer).
The question:
Given a space folding value x(T) = r E [0, for some path r, what can we learn?
The answer:
We propose see the space folding mcasure X as feature of a neural network. It tells us how twisted the Euclidean input space becomes during the learning process. is upper and lower bounded, thereby it may serve reference point for various models. As we have seen, neural networks with a high degree of self-similarity (CantorNet),, it can reach values close to 0.8, whereas for simple networks trained on MNIST to high validation accuracy the folding values are significantly lower. Moreover, the space folding measure provides insights into how effectively a network encodes self- similaritics: higher values indicate more compact and efficient representations. Conversely, a low value suggests potential room for improvement in the network's architecture for the given problem.
We now outline several directions and discuss them briefly, with the hope that the community will build upon our efforts and advance these directions further. Although we consider this work to be milestone that establishes a novel perspective on space transformation by neural networks, there are several important next steps we identify as future work.
Firstly, addressing the computational cost of our measure an interesting and important direction. As we have already hinted in Soc. one idca for future work is to explore how to optimize the choice of intermediato points the path I such that each intermediate point falls into distinct linear region and the number of intermediate points n equals the number of linear regions between the two samples. A greedy approach would be to use the upper bound N on the number of linear regions between two input points X1, X2 (see the elaboration in Sec. 5) as follows: (i) interpolate N points linearly (step in Alg. [ (ii) compare their activation patterns: if there are duplicates go back to (i) and sample more points, continue until all intermodiate points have distinct activation patterns then break. promising approach scems to be that proposed by Gamba et al. (2022), where the authors describe method for discovery of linear regions wet points X1 and x1 in the direction d X2 X1. We remark that more efficient approaches may exist but we leave this exploration for future work. Furthermore, we will investigate reducing the computational cost of computing the measure X by clustering samples within the classes C₁ and Cz. For preliminary experiments on the measure's sensitivity to that grouping see Appendix C
Another possible next step, would be to investigate the folding effects between an image from a selected class and its adversarial perturbation across various attacks, as well as for different types of data augmentation techniques. It might also be worth examining empirically the space folding effects various setups such
11

---

Published in Transactions on Machine Learning Research (02/2025)
networks trained with random labels, starting with random labels, then progressing to 10% random labels, 20%, and up to 100%. Some natural follow-up questions that might arise of these experiments are to understand: What are the differences, What about different learning rates and optimization techniques, and How does the mcasure change when using a fraction of hidden ncurons compared to all hidden neurons.
Another important and interesting direction to take on, would be extend this study to various neural networks types N (e.g,, binary neural networks (Courbariaux al. 2015 Rastegari al. 2016 Conti al.
2018)), and transformer-like architectures with ReLUs (Shen et al.
2023
Mirzadeh et al. 2023).
Lastly, in the context of reinforcement learning, it has been shown that linear regions evolve differently depending on the policy used (Cohan et al. 2022). A natural extension of this work would be to investigate how our measure evolves under different policies.
8 Conclusions
We have proposed novel method to measure deviations from convexity between a walk on a straight line in the Euclidean input space and its mapping to the Hamming activation space. Further, we have used the introduced measure to analyze deviations from convexity under a synthetic example, CantorNet, and ReLU neural networks trained on MNIST. Our work highlights the convexity transformation of the input space by ReLU neural network.
Acknowledgments
SCCH and JKU's research was carried out under the Austrian COMET program (project S3AI with FFG no. 872172), which is funded by the Austrian ministries BMK, BMDW, and the province of Upper Austria. We thank the anonymous reviewers for their constructive comments, which improved the quality of this work.
References
Alemán-Flores, M. and Álvarez Loón, L. (2004). Video segmentation through multiscale texture analysis. In Campilho, A. and Kamel, M., editors, Image Analysis and Recognition, ICIAR 2004, Lecture Notes in Computer Science, volume 3212, pages 239-346. Springer, Berlin, Heidelberg.
Alexiewicz, A. (1948). Linear functionals on denjoy-integrable functions. Collog. Malh., 1:289 293.
Anil, C., Lucas, J., and Grosse, R. (2019). Sorting out Lipschitz function approximation. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 291-301. PMLR.
Balestricro, R., Humayun, A. I,, and Baraniuk, R. (2024). On the geometry of deep learning. arXiv preprint arXiv:2408.0468 04889.
Bern, M. and Hayes, B. (1996). The complexity of flat origami. In Proceedings of the Seventh Annual ACM- SIAM Symposium on Discrele Algorithmss, pages 175 183, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics.
Bern, M. and Hayes, B. (2008). Origami embedding of piecewise-linear linear two-manifolds. In Laber, E. S., Bornstein, C., Nogueira, L. T., and Faria, L., editors, LATIN 2008: Theoretical Informatics, pages 617 629, Berlin, Heidelberg. Springer Berlin Heidelberg.
Boissonnat, J.-D. and Yvinee, M. (1998). Algorithmic Geometry. Cambridge University Press.
Bonicelli, L., Boschini, M., Porrello, A., Spampinato, C., and Calderara, S. (2022). On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Newral Information Processing Systems, 35:31886- 099988066-111.
Cantor, G. (1883). Über unendliche, lineare punktmaamigfhligkkitenn Math. Annalen, 21(4):545 591.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N. (2017). Parseval networks: Improving robustness to adversarial examples. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 553. PMLR.
Cohan, S., Kim, N. H., Rolnick, D., and van de Panne, M. (2022). Understanding the evolution of lincar regions in deep reinforcement learning. In Advances in Newral Information Processing Systems.
Conti, F., Schiavone, P. D., and Benini, L. (2018). Xnor neural engine: hardware accelcrator ip for 21.6 li/op binary neural network inference. IEEE Transactions on Compuler- Aided Design of Inlegrated Circuits and Systems, (7(11):29400 2951.
Courbariaux, M., Bengio, Y., and David, J.-P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, NIPS'15, page 11331, Cambridge, MA, USA. MIT Press.
Crescenzi, P., Goldman, D., Papadimitrion, C.,, Piccolboni, A., and Yannakakis, M. (1998). On the plexity of protein folding. Journal of Computational Biology, 5(3):423 465.
Dekker, J., Marti-Renome M. A., and Mirny, L. A. (2013). Exploring the three diree-dimensiona organization of genomes: interpreting chromatin interaction data. Nolure Reviews Genelics, 14(6):390 403.
Demaine, E. D. and Demaine, M. L. (2005). A survey of folding and unfolding in computational geometry. Goodman, J. E.,, Pach, J., and Welzl, E., editors, Combinatorial and Computational Geometry, volume 52 of Mathematical Sciences Research Institute Publications, pages 111-211. Cambridge University Press, Cambridge, UK.
Demaine, D., Demaine, M. I.., and Lubiw, A. (2000). Folding and cutting paper. In Akiyama, J., Kano, M., and Urabe, M., editors, Discrete and Computational Geometry, pages 104 118, Berlin, Heidelberg. Springer Berlin Heidelberg.
Dill, K. A., Ozkan, S. B., Shell, M. S., and Weikl, T. R. (2008). The protein folding problem. Annual Review of Biophysics.
Foote, J. (1999). Visualizing music and audio using self-similarity. In Proceedings of the Seventh ACM International Conference on Multimedia (Parl 1), pages 77 80.
Gamba, M., Azizpour, H., and Björkman, M. (2023). On the lipschitz constant of deep networks and double descent. arXie preprint orXiv:2301. 12309.
Gamba, M., Chmiclewski-Ann Anders, A., Sullivan, J., Azizpour, H., and Bjorkman, M. (2022). Are all linear regions created equal? In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 6573-6590. PMLR.
Grigsby, E., Lindsey, K., and Rolnick, D. (2023). Hidden symmetries of ReLU networks. In International Conference on Machine Learning, PMLR.
Hanin, B., Jeong, R., and Rolnick, D. (2021). Deep relu networks preserve expected length. In International Conference Learning Representations.
Hanin, B. and Rolnick, D. (2019a). Complexity of linear regions in deep networks. In International Conference on Machine Learning, pages 2596-2604. PMLR.
Hanin, and Rolnick, D. (2019b). Deep relu networks have surprisingly few activation patterns. In NeurIPS.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Jain, N. and Orcy, S. (1968). On the range of random walk. Israel J. Math., 6:373-380.
Jumper, J. al. (2021). Highly accurate protein structure prediction with alphafold. Nature.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Keup, C. and Helias, M. (2022). Origami in dimensions: How feed-forward networks manufacture linear separability. Xiv preprind or 220: 11955.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NeurIPS.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278 2324.
Lewandowski, M., Eghbal-zadeh, H., and A.Moser, B. (2024). Cantornet: A sandbox for lesting topologi- cal and geometrical measures. In NeurIPS Workshop On Symmetry and Geometry in Neural Networks, Proceedings of Machine Learning Research.
Maas, A. L., Hanmun, A. Y., and Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing.
Makhoul, J., El-Jaroudi, A., and Schwartz, R. (1991). Partitioning capabilities of two-layer neural networks. IEEE Transactions on Signal Processing, (9(6):1433 1440.
Makhoul, J., Schwartz, R., and El-Jaroudi, A. (1989). Classification capabilities of two-layer neural nets. In International Conference on Acoustics, Speech, and Signal Processing, pages 635-638 vol.1.
Mandelbrot, B. B. (1983). The Fractal Geometry of Nature. Macmillan.
Marcus, G. (2018). Deep learning: A critical appraisal. arXiv.
Minsky, M. and Papert, S. (1969). Perceplrons: An Introduction to Compulational Geometry. MIT Press.
Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. (2023). Relu strikes back: Exploiting activation sparsity in large language models. orXio preprint arXiv:2310.0456,,
Montúfar, G. F., Pascamu, R., Cho, K., and Bengio, Y. (2014). On the number of linear regions of deep neural networks. In NeurIPS, volume 27.
Moser, B. A. (2012). Geometric characterization of weyl's discrepancy norm in terms of its n-dimensional unit balls. Discrete Comput. Geom., 48(4):793-S0) 806.
Moser, B. A. (2014), The range a simple random walk on 2: An elementary combinatorial approach. Electron. J. Comb., 21.
Moser, B. A. (2017). Similarity recovery from threshold-based sampling under general conditions. IEEE Transactions on Signal Processing, 65(17):4645-467) 4654..
Moser, B. A., Lewandowski, M., Kargaran, S., Zellinger, W., Biggio, B., and Koutschan, C. (2022). Tessellation- ciaiioonniiitttinnggoo relu neural networks. IJCAL.
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. Advances in neural information processing systems, 30.
Phuong, M. and Lampert, C. H. (2020). Functional vs. parametric equivalence of relu networks. In Interna- tional Conference Learning Representations.
Price, I. and Tanner, J. (2021). Trajectory growth lower bounds for random sparse deep relu networks. 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1004 1009.
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J. (2017). On the expressive power of deep neural networks. TCML.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). Xnor-net: Imagenet classification using binary convolutional neural networks. In Computer Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 525 542. Springer.
2016: 14th European Conference,
Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386 408.
Roy, A. J. and Stell, J. G. (2003). Convexity in discrete space. In Kuhn, W., Worboys, M. F., and Timpl, S., editors, Spatial Information Theory, Foundations of Geographic Information Science, pages 253 269, Berlin, Heidelberg. Springer Berlin Heidelberg.
Sejnowski, T. J. (2020). The unrcasonable effectiveness of deep learning in artificial intelligence. Proceedings of the National Academy of Sciences, 117(43):30033 30038.
Serra, T., Tjandraatmadja, C., and Ramalingam, S. (2018). Bounding and counting linear regions of deep neural networks. In International Conference on Machine Learning, pages 4558 4566.
Shen, K., Guo, J., Tan, X., Tang, S., Wang, R., and Bian, J. (2023). A study on relu and softmax in transformer. arXiv preprint 2tc:2302.06461
Telgarsky, M. (2015). Representation benefits of deep feedforward networks. ar preprint arXiv:1509.0811
Telgarsky, M. (2016). Benefits of depth in neural networks. In Feldman, V., Rakhlin, A., and Shamir, O., editors, 29th Anmual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pages 1517-1533 1539, Columbia University, New York, New York, USA. PMLR.
Tsuzuku, Y., Sato, I., and Sugiyama, M. (2018). Sopschitz-marghh training: Scalable certification of pertur- bation invariance for deep neural networks. Advances in Neural Information Processing Systems, 31.
Virmaux, A. and Scaman, K. (2018). Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems, volume 31.
Wang, X., Wang, K., and Lian, S. (2020). A survey on face data augmentation for the training of deep neural networks. Neural Computing and Applications, 32(19):15503-15531 15531.
Weyl, H. (1916). Uber die gleichverteilung von zahlen. Eins. Malhemeeeetteehce Annalen, 77:331 352.
Xiong, H., Huang, L., Yu, M., Liu, L., Zhu, F., and Shao, L. (2020). On the number of linear regions of convolutional neural networks. In International Conference Machine Learning. PMLR.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In ICLR.
A CantorNet
CantorNet Lewandowski et al. 2024) is synthetic example inspired by the triadic construction of the Cantor set [Cantor 1883). It features two representations opposite in terms of their Kolmogorov complexities, one linear in the recursion depth k, and one exponential. It is defined through the function A [0, + [0, 1] + max(-3x + 1, 3rr - 2}, as the generating function which is then nested as (x) ==
A A(A(0) (z)), (x) == A(x). Based on the generating function, the docision manifold R₃ is defined as:
Rs : (Z. E [0, 112 : (x() 1)/2}.
(7)
The decision surface of Rs (Eq. and biases defined as
7 equals to the O-preimage of a ReLU net Xx( 10, 1J2 + IR with weights
--3 0
1
10 0 0
W1 =
3 0
b1 =
-2
W2 =
(8)
0
15

---

Published in Transactions on Machine Learning Research (02/2025)
and the final layer Wr. (-2 ,r. (-2) For recursion depth k, we define N/A
N (x) o (x) + br.
(9)
where (x)+ == (g(((((((())))) o is the ReLU function, and
(x) = a W200 W + b1).
(10)
B Heatmaps
In this section, we present additional results for the median of maxima of non-zero space folding values and the corresponding Median Absolute Deviation (MAD), defined as:
MAD median Σ (x(TT)
Ix(T)-median(()]] median(x))
(11)
across different pairs of digits. We proceeded as follows:
Algorithm 2: Computation of Aggregated Folding Measures for Digit Pairs
Input: Images of digits from classes C₁ and C2
Output: Median of non-zero maxima of space folding values across digit pairs + MAD (Eq. 11
Step 1: For each pair (x16, X2j) where X E Ci and X2j C2, compute x(I') using Alg,
Step 2: Among [C1| |C2) measures, select those with non-zero values, forming the set X(T)+
Step 3: For each x(T)+, compute the maximum value along P. Collect maximas into the set max X+-
Step 4: For each digit pair compute the modian of max X+ - MAD (Eq. (11).
Each heatmap corresponds lo a network with a different depth. We observe an increase in the median space folding values with the depth of the neural network. In the main paper, we reported the Pearson correlation coefficient between the depth and the aggregated median space folding X for ReLU networks trained on the random seed (equal 4) for which the networks achieved high validation accuracy, even for deeper layers. Initially, we worked with seeds (0, 2, 2,,44, but only for seeds {3,4) did the network achieve the desired accuracy. For depths up to 5 layers, the validation accuracy easily exceeds 0.95, while for 6, and 10 layers, it drops, with the deepest network achieving around 0.85 of validation accuracy.
C Sensitivity to Grouping
In this section, we present a preliminary study on the sensitivity of our measure to clustered data as compared to working directly with the original samples. We study the effect of clustering (i) within one class of digits, (di) within both classes of digits. Our analysis focuses on 16 pairs of digit classes from the MNIST test dataset (we work with digits from classes {0, 6,9) x {0, 3, 6, 9)). We then vary the number of clusters (k) into which the digits are grouped, experimenting with & E {1, 2, 5, 10, 20, 50, 100). We use k-means in the (flattened) pixel space of the images. After clustering, the set of centroids is extracted and used as input to the procedure outlined in Algorithm repeat this analysis for all 16 digit combinations, resulting in 16 values for each level of clustering (x-axis), for which we report the mean and standard deviation values, and add the reference value of space folding for comparison purposes. We report the results for 2 different neural networks, distinguished by colors (Fig B)
2 This process yields a single folding value for each digit pair. We
Qualitatively, results indicate that the behavior of the measure is not strongly affected even with little as 5 clusters per group. Although these preliminary findings require validation additional datasets, they suggest that computational complexity, as defined in Eq. ( could be significantly reduced without compromising accuracy.
Analysis for Deeper Networks
We performed a preliminary study on the folding effects for deeper networks with architectures {2 300, 3 x 200} for the of layers no. of neurons, respectively. As previously, train those networks to
16

---

Published in Transactions on Machine Learning Research (02/2025)
Median + MAD for Digit Poirs for Noural Architecture 2x30
Modian + .MAD for Digit Pairs for Neural Architecture 3x20
0.6
0.6
3:30
8:300
6:15
12:3)
130%
1204
0.5
0.5
637
8.4
1000
1304
0.94
K.901
0.30
8.58
3:30
1903
1304
0.4
0.4
Digit
14
0.33
8.22
0.3
6.5
0-22
0.53
0.27
Value
0.14
0.3
8.53
1924
3.34
0.18
8.23
016
0.3
0.3
0.15
0.14
0.34
6:15
0.2
10-
0.15
6.33
0.15
6.34
0.13
0.2
6.22
8.54
1100
8.22
1224
3.57
0.001
3.1
6.77
0.30
12011
8.50
12.00
8.39
120%
0.1
0.1
0.33
8.2
0.2
6:34
8.15
6:15
6.29
0.11
6.27
6.18
4.22
0.18
1202
0.0
0.0
0
1
2
9
2
3
Digit
Digit
Median MAD for Digit Palirs for Noural Architecture 4x115
Median MAD for Digit Pairs for Noural Architecture 5x12
0.6
0.6
037
8:22
025
8:30
0.18
1317)
0.39
0.5
(11%
6:38
8.52
0:3
8-23
533
6-29
(371
0.5
0.011
6.411
3.35
8.37
0.4
m
6.27
9.88
8.22
0.4
Digit
0.2
8.50
2.20
115%
31111
1317
0.37
0.15
0.22
0.3
Value
0.33
0.22
8.29
1347)
01
130%
0.3
6:18
8.30
0.17
6.15
0.17
6.17
3.15
6:15
9.10
6.17
0.2
0.18
6.29
0.55
6.15
4.25
0.18
0.25
0.15
6:22
0.16
80
0:38
8.0
028
8.00
3.30
8.6
0.3
0.1
80
0.5
035
817
635
8.2
0.2
6.2
838
0.1
3.30
1807
0.23
422
104m
4.22
0.0
0.0
0
1
2
3
8
9
1
2
3
9
Digit
Digit
Median MAD Digit Pairs for Neural Architecture 6x10
Median MAD Digit Pairs Neural Architecture 10x6
0.6
0.6
8.45
A33
6-37
9.53
0.22
0.5
0/K
6:21
5.51
8:20
0.5
6:38
8:31
035
6:29
6.23
1277
6:37
3.75
0.35
0.4
12/4
6:22
3.22
CLUB
1214
0.4
Digit
0.3
value
0.3
5
0.2
8.26
0.29
6.5%
823
6.35
6.39
AM1
C-N
0-7%
4.1%
0.1%
6:25
1347)
4.25
1324
01
0.2
JOCK
0.2
628
8.6
0.35
6.5
12.5
3.30
1305
6:11
1/0
623
0.1
AM1
E45
CAR
5.55
1042
X.
0.1
0.0
0.0
0
1
2
3
9
1
2
3
Digit
Digit
Figure 7: Median of maximas of non-zero space folding + median absolute deviation for every digit pair in MNIST for a selected ReLU neural network trained with random seed 4. The space folding values increase with depth (darker blue values). (Best viewed in colors.)
17

---

Published in Transactions on Machine Learning Research (02/2025)
Impact Clustering thin one Class on X
Impact of Clustering Within on
0.40
0.40
Ref. for 3x20
Ref. x for 5x12
Mean 4: Stt xi
0.30
3x20
Mcan Stdd YIF
0.30
0.43
5x12
0.23
0.15
0.30
0.30
No. clusters
No. clusters
Figure 8: Preliminary results illustrate the robustness of the space folding mcasure X under clustering. Left: Only one digit class C₁ is clustered, while the other class C2 remains intact. The measure remains stable across varying numbers of clusters k. Right: Both digit classes are clustered. Although no folding effects arise for k = 1,2, introducing k = clusters per class already produces consistent folding effects, and these persist at higher clustering levels. Dashed lines indicate the reference values for the space folding measure computed using all the pairs of digits of respective classes.
Median MAD Dight Pains for Neural Architecture 2x300
Median max MAD for Dight Pairs for Neural Architacture 3x200
0:11
0.13
0.18 10.00
0.18
0.22
(0.04)
(0.04]
(0.94)
(0.04)
[0.05]
(0.02
[0.05]
10,041
10.00]
0.14
0.14
0.13
0:32
0:188
0.22
0.15
0.38
[0.05)
(0.04)
00.05)
10.041
(6,04)
(0.04)
10.04)
(9.05)
(5.00)
10.00)
00.00
30.051
Dgt
Figure 9: We investigate the folding effects for larger networks. Interestingly, we do not observe much difference for the folding values. However, we see a strong increase in the number of paths that feature folding effects.
convergence at MNIST, and store their parameters. We then investigate the space folding values similarly before. Interestingly, we do not observe much difference in the space folding values computed in Alg 2 when compared to values obtained shallower architectures (see Fig. gh. Although the folding values
themselves show little difference compared to shallower networks, the proportion of paths exhibiting folding effects increases substantially. For the smaller architectures x 30 and 3 x 20 the mean ratio (# std) of
paths with folding effects is 0.35 0.1 and 0.44 0.15, respectively, while for deeper architectures x 30 and 20 those values are 0.97 + 0.04 and 0.99 0.02, respectively.
18


Reviews:
[{"review_id": "oZ5Jxuu7DR", "reviewer": "Reviewer_ho1r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper investigates an interesting property of ReLU networks and supports its findings with empirical analysis.\n\n2. Practical implications of the proposed measure are discussed, particularly in Section 7.\n\n\nWeaknesses:\n\nThe paper's organization could be improved. While many parts of the paper have been updated, the introduction and abstract do not reflect these revisions. These sections should be updated to align with the rest of the paper and provide a clearer overview of the contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a concept called the \"space folding measure\" to analyze how ReLU networks transfer convexity between the Euclidean input space and the Hamming activation space. The authors show that straight-line paths in the input space correspond to non-convex trajectories in the activation space and propose range-based metrics to quantify these deviations. Their empirical analysis includes experiments with a synthetic \"CantorNet\" architecture as well as networks trained on the MNIST dataset to validate their findings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see 'weaknesses'.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c8idGlFy8y", "reviewer": "Reviewer_jq1r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not an expert for this task. P.S. before I sent my review I noticed that the authors added a revision which already addressed some of the issues that I was observing.\n\nStrengths: \n\nThe question is interesting from a theoretical point of view. The paper is generally well-written, and contexts are provided to help the readers understand the topic.\n\nWeaknesses:\n\nI find definition 1 and convexity and example 2 confusing. Definition 1 states that a subset is convex if all points on every shortest path between every two points are also in the subset. In example 2, it was stated that there are intermediate patterns, which I think lies on shortest path(s), not in the subset; yet the subset still forms a convex set, and the author justified that by stating that no intermediate points are contained in the complement of the subset. Does this contradict with the definition? Or am I understanding it incorrectly?\n\nThe proposed measure needs to be computed between every pair of input samples, thus leading to relatively high computational cost. Is it in principle possible to calculate the measure based on a batch of pairs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of how ReLU neural networks fold input spaces, which is beneficial for users to understand the working mechanism of the networks. The paper proposes a novel measure building on the range measure which can measure the fold-ness of the resulting space and demonstrate a connection between the convexity of the input space and the activation space. Empirical evidence was provided on both CantorNet and MNIST image classification benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors reported the computational complexity of their algorithm, which, to me, is not that high when C_{1} and C_{2} are relatively small. In the experiment section, for MNIST classification the employed fully connected neural network has fewer parameters than one might typically expect. While this is not critical, it would be interesting to observe what happens with a larger number of neurons, e.g. three layers with 200 units per hidden layer; or would this be too computationally challenging?", "broader_impact_concerns": "The work is largely theoretical, thus not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mh98Doh1Ob", "reviewer": "Reviewer_sXC9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. (Good first impressions) While the paper operates with complex mathematical concepts, it was immediately obvious to me, a practitioner with a decent mathematical apparatus, that the authors have made a commendable effort in presenting the concepts and laying out the idea well. I appreciate it, as I would be struggling to review the paper if the majority of the concepts were just assumed to be known by the reader. I also appreciate the visualisations (Figs. 1, 3, 4). I do retain my reviewer's right to request some polishing/clarifications (cf. Requested changes), but I have seen papers that could present an equivalently hard topic with 0 images and worse flow.\n1. (New metric) As the authors discuss in the related work section, the idea of viewing activations as bits/computation as space foldings/etc. is not new. However, I am also not aware of past work that quantifies the folding and if no other reviewer flags such a quantitative measure already existing, this would be a strong point for acceptance. It is not my subfield of research, but some googling got me nowhere close to a similar metric.\n\n## Weaknesses\n**The fact I listed only two strengths does not necessarily imply the paper does not have any others, or that it is bound to be rejected.**\n\n(Allow me to put my practitioner’s hat on for most of these)\n\n1. (Assumptions of the metric and generality) While I do not see why not, I also do not see the reason the paper focuses only on MLPs with ReLUs? After all, sections 4 and 5 mention ReLU only twice in total – once in figure 3, regarding the fact that hyperplanes intersect the entire input space (which the authors say it “holds for the hyperplanes [...] from the first hidden layer”) and once at the very end of \\S4 (same statement). Hence, I have to ask the question, would the metric be of any use, if we have:\n     * normalisation layers – I presume not, as they may break the straightness of the lines? (unsure)\n     * residual connections – I cannot see why not\n     * attention layers – except for the attention coefficients' calculations, the rest of the dataflow are affine transformations. \n\n   I believe the assumptions behind this metric (and reasons to focus on MLPs) should be emphasised in \\S3 and 4. While I know it is common for theory and especially interpretability to lag behind practical advances, if this study is not easily extensible to other architectures, I would consider it as a ground for rejection.\n\n1. (Measure algorithm, computational complexity and robustness) The authors currently state that “computing the folding measure for large architectures may seem infeasible due to the potentially high number of linear regions”. However, the authors never stated the exact algorithm to calculate the measure or its time complexity. From my understanding, the steps, **for two points** are:\n   * Interpolate them linealry. Sample $n$ points on this interpolation. $O(n)$\n   * For each point, compute the binarisation. $O(n*Cost)$ where $Cost$ is the complexity to run the network in inference mode.\n   * Calculate the maximal distance from the starting point and the sum of distances between intermediate points. As we are using Hamming distance, for $N$ neurons this is $O(N)$ for one measure. We need $O(n)$ such measures, so $O(n*N)$ in total.\n\n   If my algorithm is correct, this is $O(n*(N+Cost))$ for a *pair of samples*. Measuring the distance, as done in the experiments, for one **pair of classes** $C\\_1$ and $C\\_2$ is $O(n*(N+Cost)*|C_1|\\*|C_2|)$.\n\n   Obviously, this will not easily scale. As we cannot change the $(N+Cost)$ factor, we need subsampling. This can be achieved by varying $n$ or subsampling from the classes. I, however, have zero idea and, unfortunately, zero evidence of how robust the proposed measure is to varying those hyperparameters.\n1. (Few experiments) Currently, the measure is only evaluated on two datasets, and for MNIST, only on one pair of classes. Consider, e.g. making a heatmap for all pairs of classes? E.g. a board with 5 10x10 (or even 5x5) heatmaps for each architecture, all sharing the same legend (so as to show that $\\chi$ is indeed greater for all pairs for the 6x10 architecture). Feel free to consider other datasets too.\n1. (What can we learn) The authors say that $\\chi$ should be seen “as a feature of a neural network”. I have two problems with this, one practical and one philosophical:\n   * The measure is also based on the data. Thus, it is rather a feature of the network and the data as a pair.\n   * What leverage do I get, from calculating it? Do I get to understand what the “underlying logic” of the model is? Or can it tell me how to improve the architecture? Yes, I know it measures the amount of twisting/folding the NN will do to an input, but adding some discussion of future directions, apart from the fact you will investigate the space folding measure of more pretrained models, would likely improve your paper on _”Would some individuals in TMLR's audience be interested in the findings of this paper?”_ aspect of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the widely assumed property that ReLU neural networks rely on space folding transformations when learning. Compared to past works, this work is _the first to quantify the amount of space folding that occurs_. This quantitative measure is built by viewing the activation patterns as bits and quantifying the convexity ratio in the activation space when using Hamming distance as a distance function. Along a (shortest) path $\\Gamma$ the space folding $\\chi(\\Gamma)$ is based on the ratio of two range metrics – the first measuring the maximum distance from the starting point; the second measuring the total distance travelled.\n\nExperiments are obtained on synthetic models (CantorNet) and real world models (ones trained on MNIST). CantorNet is a network solving the decision problem of a point belonging to the Cantor set (right?) which is known to require high folding, and the experiment serves to validate our new metric. The MNIST experiment seems to affirm that deeper MLPs have higher folding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N.B. There are a few extra, regarding the clarity of the writing.\n\nN.B.#2, I will use this space to get clarifications.\n\n1. Example 2, Convexity. How did you get the intermediate states? E.g. for 0001->1011, a shortest path, flipping one bit at a time is: 0001->0011->1011 or 0001->1001->1011. Where did 1111 come from? Also, in this example why does the indexing start from 4 ($\\pi_4$)? And shouldn’t these be added to the convex subset? (0011 is on a shortest path from two points in S, but is not in S, making S not be a convex subset)\n1. What do the colours in Figure 2 represent? Please describe the figure. While I acknowledge that Lewandowski et al. have provided you with it, the paper is not publicly available yet. (Do send a link if it is).\n1. [Nit] Convexity proof (=> direction). $R_{\\pi_1}$ should be $R_P$? (similar for the other) Double-check and fix it if so.\n1. [Nit?] Section 5, “The space folding measure”. $\\Gamma$ suddenly gained a subscript $n$, but it doesn’t have one when introduced? Why? And is the subscript even necessary for the proof.\n1. First experiment and Figure 5. Again, someone unfamiliar with Cantor set and CantorNet, like me, would find it very challenging to understand what is going on. What I understood (also from Wiki) is that:\n   * The neural network has fixed weights and requires no training. \n   * The “dataset” describes a fractal and the higher the recursion depth, the more complex (in terms of amount of folding) the neural network has to become.\n\n   However, I do not know how was $\\chi$ measured (did you sample random points on the input line?). Please, give a little more intuition about the task and let the reader know how the value was calculated. Also, the value of 0.5 is different from the value of 0.6+ in Figure 6, which confuses me more.\n1. Figure 6 “Background represents CantorNet recursion-based representation at recursion level k = 2”. I find it quite hard to wrap my head around the sentence and the gray vs white background of the figure. Is the background related in any way to the x and y axis? Or, rather, just the x axis? (I presume the latter, but add some more clarification in the caption)\n1. Figure 7. The standard deviations seem high (even looking at figure 8). To what quantitative extent, across the dataset, does the following correlation hold: “Deeper architectures have higher $\\chi(\\Gamma)$”. Given the use of word “significant”, I’d expect some correlation coefficients, at the very least. You can obtain those by:\n    * considering different classes\n    * retraining the networks with different random seeds\n\n   I suggest you do both.\n1. Address weaknesses.\n   * state assumptions\n   * add algorithm/complexity and a robustness figure (e.g. do your current observations change, if you linearly interpolate of steps 0.01 and 0.2)\n   * (I will not get fixated on that one, but:) Consider adding one more dataset.\n   * add future work discussion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The Authors have addressed all of the major issues highlighted by the Reviewers, and all Reviewers are in favour of accepting the work to TMLR. Well done!
Decision: Accept (poster)