Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
Authors: Nico Pelleriti
Max Zimmer
Elias Samuel Wirth
Sebastian Pokutta
Venue: ICML 2025 poster (2025)
Keywords: neural network
manifold hypothesis
vanishing ideal
polynomials
latent space
computational algebra
Primary area: deep_learning->other_representation_learning
TL;DR: We replace the deep network tail with a compact polynomial layer that’s fast, accurate, and often more generalizable.

Abstract: Deep neural networks have reshaped modern machine learning by learning powerful latent representations that often align with the manifold hypothesis: high-dimensional data lie on lower-dimensional manifolds. In this paper, we establish a connection between manifold learning and computational algebra by demonstrating how vanishing ideals can characterize the latent manifolds of deep networks. To that end, we propose a new neural architecture that (i) truncates a pretrained network at an intermediate layer, (ii) approximates each class manifold via polynomial generators of the vanishing ideal, and (iii) transforms the resulting latent space into linearly separable features through a single polynomial layer. The resulting models have significantly fewer layers than their pretrained baselines, while maintaining comparable accuracy, achieving higher throughput, and utilizing fewer parameters. Furthermore, drawing on spectral complexity analysis, we derive sharper theoretical guarantees for generalization, showing that our approach can in principle offer tighter bounds than standard deep networks. Numerical experiments confirm the effectiveness and efficiency of the proposed approach.

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
Nico Pelleriti 12
Max Zimmer 12
Elias Wirth
Sebastian Pokutta 12
Abstract
Baseline NN
Truncated NN
Deep neural networks have reshaped modern ma- chine learning by learning powerful latent rep- resentations that often align with the manifold hypothesis: high-dimensional data lie on lower- dimensional manifolds. this paper, we estab- lish a connection between manifold learning and computational algebra by demonstrating how van- ishing ideals can characterize the latent manifolds of deep networks. To that end, we propose a new neural architecture that (i) truncates a pretrained network at an intermediate layer, (ii) approxi- mates each class manifold via polynomial gen- erators of the vanishing ideal, and (iii) transforms the resulting latent space into linearly separable features through a single polynomial layer. The resulting models have significantly fewer layers than their pretrained baselines, while maintaining comparable accuracy, achieving higher through- put, and utilizing fewer parameters. Furthermore, drawing on spectral complexity analysis, we de- rive sharper theoretical guarantees for generaliza- tion, showing that our approach can in principle offer tighter bounds than standard deep networks. Numerical experiments confirm the effectiveness and efficiency of the proposed approach.
toy
dog
Figure 1. Comparison between a baseline NN (left) and truncated NN (right). The baseline NN output is interpreted as class proba- bilitics. The truncated NN output is interpreted using the manifold hypothesis, suggesting that classes lie distinet manifolds. We characterize these manifolds using a set of polynomials, capturing their underlying structure. The depicted manifold visualization of a Calabi Yau manifold, licensed under the Creative Commons Attribution Alike 2.5 Generic license (Lunch, 2007).
1. Introduction
Deep Neural Networks (NNs) have revolutionized various fields by learning powerful feature representations from data (Shaheen et al., 2016; Alzubaidi et al., 2021; Shi al., 2022). key theoretical foundation of their success is the manifold hypothesis (Cayton, 2008; Fefferman et al., 2016; Brahma et al., 2016), which suggests that high-dimensional real- world data lies on -iver-dimensional manifolds, making many learning tasks tractable.
Such manifolds can often described as the zero set of system of polynomial equations (Nash, 1952)'. Formally, this system gives rise to algebraic variety, which is de- fined as
Z = (ZER". (z( 0 for all i E (1,..., [}],
for some set of polynomials pı.... Pk- Given sample Z (zi...., Zm C Z drawn from this variety, we can in turn estimate the polynomials .......PP Py that characterize Z by computing the vanishing ideal of this sample, i.e., the
'Department for Al in Society, Science, and Technology, Zuse Institute Berlin, Germany *Institute of Mathematics, Technische Universität Berlin, Germany, Correspondence Nico Pelleriti <pelleriti Ditirritizzbbde>>.. zib.de>, Max Zimmer <zimmer@ziin zib.de>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
While not every manifold is an algebraic variety, every com- pact smooth manifold is diffeomorphic real algehraic variety by the Nash- Nash-Tognol theorem (Nash, 1952).

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
set of all polynomials that vanish over Z, defined as
layer, but instead of just adding linear classifier, we con- struct generators of the per-elass vanishing ideals to capture the nonlinear structure of the latent manifolds. VI-Nets leverage these latent representations at layer L' to yield a feature transformation that renders linearly-inseparable data linearly -heaall-separable, which is then used for simple dis- crimination and classification. Notably, while discarding all layers beyond layer L', VI-Nets still achieve competitive performance while using significantly fewer parameters. We summarize our key contributions:
I(Z) - (o P: : p(z,) = 0 for all 2, E Z),
where P is the set of all r--ariaat polynomials. While the vanishing ideal is infinite, there exist finitely many polyno- mials that generate it (Cox et al., 2010). To construct such a set of generators, one typically employs vanishing ideal algorithms (Fassino, 2010; Heldt et al., 2009; Livni et al., 2013; Limbeck, 2014; Zhang, 2018; Wirth & Pokutta, 2022): For given set of points Z. these algorithms construct the finite set of generators. thereby characterizing I(Z). These sets of polynomials have been leveraged in downstream tasks, such as hand posture recognition, principal variety analysis for nonlinear data modeling and solution selection using genetic programming (Iraji & Chitsaz, 2017; Kera & Iba, 2016; Wang et al., 2018).
1. Characterizing Manifolds in NN Latent Spaces. Van- ishing ideal algorithms require low-dimensional input points and often produce an excessive number of generators, each with numerous monomial terms needing individual evalua- tion. This makes their application to the high-dimensional latent space of NNs infeasible. Furthermore, the complete set of generators often fails accurately represent the alge- braic varieties of the latent space. We address these issues by: a) introducing dimensionality reduction and data rescal- ing techniques to enable the application of vanishing ideal algorithms in NN latent spaces; b) leveraging and adapt- ing stochastic formulations of vanishing ideal algorithms to derive relatively sparse polynomials that can be efficiently evaluated; e) proposing pruning methods to significantly reduce the number of generators, preserving only those that meaningfully capture the underlying manifold.
In this work, we aim to use vanishing ideal algorithms to construct polynomials that describe the latent space of an NN, which serves as a compressed, structured rep- resentation of the input data within the NN. Prior work (Brahma et al., 2016) has shown that this latent space often behaves like a manifold, where the network disentangles complex, high-dimensional manifolds into simpler, lower- dimensional ones.
A key observation, and central motivation for our work, is that if the latent space is manifold, it can be algebraically characterized as the zero set of certain polynomials. Further, if the data classes correspond to disjoint algebraic varieties, then the generators of their vanishing ideals can be used as discriminative features to separate classes in the latent space, making them linearly separable (Livni al., 2013).
2. Building Efficient VI-Nets. We propose VI-Nets, novel NN architecture that truncates a pretrained NN and replaces its final layers with a single polynomial layer. This layer embeds the generators of each class's vanishing ideal, transforming the linearly-inseprrable but polynomially- separable features of the latent space into Ahearly-separabee ones. Unlike methods that approximate manifolds through indirect means like simplicial complexes (Khrulkov & Os- eledets, 2018) or k-nearest neighbor graphs (Tsitsulin et al., 2020). our approach directly captures their algebraic struc- ture through polynomial roots.
In consequence, computing generators of the vanishing ide- als such latent representations is highly desirable. Un- fortunately however, current vanishing ideal algorithms are incapable of processing nigh-dimensional data, such as im- ages or even their representations in the latent space of NNs. While recent advancements (Wirth & Pokutta 2022) have made it feasible to compute approximate generators for data with dimensionalities up 67, beyond these scales the com- putational cost grows prohibitively. Already at this scale, vanishing ideal algorithms can produce millions of polyno- mials, making subsequent evaluation computationally infea- sible. Hence, identifying a compact and meaningful subset of generators that effectively characterizes the underlying manifold remains an open challenge.
3. Providing Theoretical Learning Guarantees. We prove learning guarantees for VI-Nets by leveraging the spectral complexity framework (Bartlett et al., 2017). Specifically, we exploit the sparsity of the constructed gen- erators to derive better generalization error bounds when comparing to the corresponding pretrained NN.
Addressing these challenges, we introduce set of tech- niques that make vanishing ideal computations feasible for high-dimensional latent spaces. Building on these ad- vancements, we propose novel neural network architec- ture, which we term Vanishing Ideal Networks (VI-Nets). Similar to linear probing (Alain & Bengio, 2016), our ap- proach truncates a pretrained network at an intermediate
4. Training VI-Nets. We perform extensive experiments that showcase that VI-Nets can achieve performance com- petitive to the pretrained baseline NNs while using much fewer parameters. In particular, we analyze the trade-off between model accuracy and parameter count, when trun- cating more and more layers from the pretrained NN.
2

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
To the best of our knowledge, leveraging tools from Com- putational Algebra in the context of Deep Learning is relatively underexplored field. By demonstrating the feasi- bility of using vanishing ideal computations to characterize data manifolds in NN latent spaces, we aim to encourage further research in this direction.
generators can be computed using variety of algorithms (Livni et al., 2013; Limbeck, 2014; Wirth & Pokutta, 2022).
2.2. Class Separation in the Latent Space
A key insight is that data points from different classes in the latent space, while not linearly separable, can be dis- tinguished through polynomial separability. This concept is central to VI-Nets, which we will define later. Consider a pretrained NN & that we modify by truncating a certain number of its final layers, resulting in & (see Figure 1). Let X represent input data points for class & E [K], such as those from a validation dataset. Correspondingly. Zt de- notes the latent space representations after truncation, i.e., Z = ((x*). For each class k, we anticipate that the set of output activations z resides on a manifold that can be upproximated using generators of the vanishing ideal of zt.
2. Algebraic Characterization of Data Manifolds
In this section, we present our methodology for approximat- ing data manifolds using generators of the vanishing ideal, specifically addressing the challenges of applying vanishing ideal algorithms to NN latent spaces. Our approach con- structs class-specific polynomial sets that characterize the underlying manifold structure, which we then leverage to build efficient polynomial layers. We begin by formally stating our core problem:
Suppose now that for each class k, we are given ny gener- ators pf....; Ph, of the vanishing ideal of Z², Then, we define the feature transformation for latent space sample R" as:
Problem 2.1. For each input class k [K], let Z {zf. ..i.....22 = S I"" represent the latent space activations at some layer L,' of an NN for samples belonging to class k. The goal is to compute subset of generators of the approximate vanishing ideal of z that effectively describe the underlying manifold U²
g (IF(())),.... ... Ip (z)l) E IR*
Aggregating these feature transformations for all classes yields the mapping:
2.1. Preliminaries
Let I, k, m, n. q, K E N. Vectors and matrices are denoted in bold. Define T and P as the sets of monomials and polynomials, respectively. A polynomial g E P is expressed as g Ei=1 Cit, with coefficients с = (cI...., Cy) and monomials 4 E T for i [g]. We focus on the latent representations of NNs, specifically the output activations of a specific layer T'. These vectors, denoted as a E IR.,, are derived from passing a sample x through an NN Φ. We focus on NNs for classification tasks. For notational simplicity, functions are often applied to sets, e.g., ((X)= = fØ(x) : E X} for set X. Superscript indices denote class membership, while subscript indices denote specific elements in set, e.g., x xx, x are three distinct input samples (interpreted as vectors) of class kt
G(z) (o')).. (z) E R R
(POLY)
Typically, the latent space subsets z are not linearly separa- ble. While additional non-linear layers might achieve linear separability, this is not assured at intermediate layers. By definition of the vanishing ideal, the generators pf...., ph for class k vanish on 2 but most likely not on Z' for i & k. Thus, for a point z E Zt, G(z) is zero at coordinates for class k generators and positive elsewhere, enabling a lin- ear classifier to separate the classes. For an input x, the mapping
X WG(O(x))
allows linear classification, where W E the weight matrix mapping to the K class logits.
Given some set Z = {zi,...., 3m} C RR", we follow Cox et al. (2010) and define the vanishing ideal as follows. Definition 2.2. Given sample Z, the vanishing ideal I(Z) C P is defined
2.3. Practical Data Manifold Approximation
Approximating class data manifolds with algebraic varieties provides a structured method to capture data relationships. As we have shown, this approach can further be used con- struct classifier from truncated NN. However, real-world data is noisy, making it impractical to find polynomials that vanish exactly on the data. Furthermore, we argue that com- puting the exact vanishing ideal of a sample Z is not only computationally intractable but also lacks generalization properties. Since is finite, itself forms a variety (e.g. as
I(Z)= {g P:: g(z,) = 0 for all i E [m]]- Conversely, given set of polynomials F C PP the corre- sponding algebraic variety VFF) S IR" is defined as
V(F) = (z E Rd g(z) = 0 for all F).
Despite I(Z) being an infinite set, there exist finitely many polynomials that generate (Cox et al., 2010). Further, such
3

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
the roots of an appropriate polynomial), and by the ideal- variety correspondence (Cox et al., 2010), V(I(Z)) = Z. Thus, the algebraic variety defined by the vanishing ideal's generators includes only the points in Z, offering no general- ization beyond the sample. However, for class k; our aim is to approximate the underlying manifold U from which z is drawn, not just to describe Z In general, recovering U exactly from a finite sample is impossible without structural information, as U could range from being as minimal as Z to as broad as RR..
mate vanishing ideal generates the entire polynomial ring. To meaningfully approximate the underlying manifold Uj. for each class k E [K], we select a subset of the constructed generators of the approximate vanishing ideal with some tolerance vbb > 0, based on two key criteria: a) low complex- ity and b) discriminative power. We detail these approaches below and provide theoretical justification in Section 4 from the perspective of statistical learning theory.
Low Complexity: Finding coefficient-sparse generators. The number of monomial terms increases exponentially with the polynomial's maximal degree, making evaluation inefficient. A polynomial's 's complexity is determined by the sparsity of its coefficient vector c; greater sparsity implies fewer non-zero coefficients and thus fewer monomial terms to compute. To achieve this, we use the meta- aega--algor Ora- cle Approximate Vanishing Ideal Algorithm (OAVI) (Wirth et al., 2023; Wirth & Pokutta, 2022) for vanishing ideal generation, adopting the Frank- algorithm (Frank & Wolfe, 1956; Jaggi, 2013) as its oracle to produce sparse coefficients. We also employ the Approximate Buchherger- Möller Algorithm (ABM) (Limbeck, 2014), which similarly yields sparse generators (Wirth & Pokutta, 2022), and limit the maximal degree avoid overfitting. While both OAVI and ABM yield coefficient-sparee polynomials, they can still generate many that do not meaningfully approximate UK an issue addressed next.
Hence, research has mainly focused on constructing approx- imate generators for the vanishing ideal, i.e., polynomials that vi-approximately vanish over the data allowing a small error tolerance vb 2 0 (Heldt et al., 2009; Livni al., 2013; Limbeck, 2014; Wirth & Pokutta, 2022). We define 4 approximately vanishing polynomials as follows:
Definition 2.3. For & 2 O, a polynomial g = - 444i is el-approximately vanishing over Z if MSE(g. Z) = 1 -E (2,) I &b and its leading coefficient equals 1. m
Setting the leading coefficient (i.e., the highest degree mono- mial's coefficient) to 1 is essential for defining approxi- mately vanishing polynomials. Without this constraint, any polynomial g E P can be rescaled to approximately van- ish for all z E Z. causing the spurious vanishing problem (Kera& & Hasegawa, 2019), where polynomials fail to capture meaningful data structure. Unfortunately, the ideal gener- ated by all -i-approximately vanishing polynomials, known as the approximate vanishing ideal, also fails to provide the desired structure (Heldt et al., 2009), as it typically becomes the unit ideal, encompassing the entire polynomial ring.
Finding distinctive generators. We ensure selecting polynomials that effectively differen- tiate U from other class regions by eliminating those that also vanish for other classes. Our experiments demonstrate that most generators capture the dataset's global structure rather than class-specific features, we introduce prun- ing score to rank each generator based on its non-vanishing ability on other classes. Let Z* = {zf,.... 2 m ) repre- sent the latent representations for class k. For generator p associated with class j. its score o (p) is defined as
Discriminative Power:
To summarize: In contrast to the exact vanishing ideal, which is too restrictive and only captures the points in Z, the approximate vanishing ideal generates the entire polyno- mial ring and does not depend on Z at all. Thus, we focus on identifying& subset G of its constructed generators, ap- proximating the underlying variety and therefore manifold effectively by VCG).
1
(p) = -in
p(Zz))
3. Constructing practical VI-Nets
This score measures the minimal average vanishing of gen- erator p for class j on other classes. Low scores indicate that a generator nearly vanishes on some class k. despite being a generator for class i, showing its lack of distinctiveness. We rank generators using s² (g) and select the top p % for each of the class j. as the manifolds for different classes likely exhibit varying complexity (Brown et al., 2023). The remaining N generators are the most discrininative and are used to form the polynomial layer, while others are dis- carded. In Section 5, we validate our method, demonstrating that a subset of generators suffices for classification.
We introduced the core concept of VI-Nets, using gener- ators of the vanishing ideal to approximate per-class data manifolds. Now, we address the identified challenges: We start with our approach for finding polynomial generators that meaningfully describe the underlying manifolds, fol- lowed by adaptations of cxisting vanishing ideal algorithms to being able to compute the generators in the first place.
3.1. Selecting Meaningful Polynomials
We derived that computing the exact vanishing ideal is too restrictive, only capturing points in Z, while the approxi-
Evaluating many sparse polynomials can still be compu-
4

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
thereby controlling the growth of monomial terms. Third, we employ lower-precision arithmetic (e.g., float16) in the vanishing ideal computations and a tanh-based rescaling (Hampel et al., 2011) (cf. Appendix C.4) of latent features to stabilize numerical computations. Although these choices can slightly reduce initial accuracy, they preserve the crucial monomial structure, which we later refine by reiraining co- efficients via gradient descent. This pipeline thus retains the principal strengths of OAVI and ABM to discover sparse, class-specific polynomials- suring improved scalability in the setting of latent spaces within neural networks.
23
Ph
Figure 2. Illustration of our method for obtaining parameter- efficient representations of the most discriminatory vanishing ideal generators. The process hegins by pruning columns corresponding to polynomials with the lowest pruning scores, such as p2 and in this example. Exploiting the sparsity of the generated polyno- mials, rows with only zero entries indicating monomials that do result, only the essential monomials (y, xz and v² in this case) need to be evalu- ated during the final computation. Experiments confirm that this approach effectively reduces the number of evaluated monomials (cf. Figure in the appendix).
the ability en-
not contribute- do not need to be evaluated.
3.3. The VI-Net Pipeline
Algorithm describes the VI-Net pipeline. It starts by training or using a pretrained network Φ, truncating it at layer L', and extracting features Z for each class A E [K]. Generators .......,',, pig of the vanishing ideal of z are computed using stochastic variants of OAVI and ABM, followed by pruning. The pruned generators are aggregated to construct the transformation G from (POLY). Finally, the coefficients of the polynomials in G and the weights Wr of the appended linear layer are jointly retrained to optimize classification performance.
tationally intensive due to the lack of sparsity overlap in their coefficient vectors: different polynomials might have entirely different monomials. Our two-fold approach ad- dresses this: First, we use OAVI or ABM to generate sparse coefficient vectors. Second, we prune the set of generators to enhance discriminative power, significantly reducing the number of polynomials and indirectly also the number of monomial terms that need to be evaluated, allowing us to drastically shrink the matrix of coefficients. The remaining coefficients are then fine-tuned via gradient descent to re- cover any lost accuracy. This process, visualized in Figure 2, focuses on identifying a subset of generators that effectively approximate the data manifold.
Algorithm 1 VI-Net Pipeline
1: Input: Dataset X = {x" Jr,s NN 6, intermediate layer L', pruning function PRUNE, vanishing ideal algorithm VANISH.
Output: Trained VI-Net H1
3: Φ <- TRAIN (x
4: Φ2 + TRUNCATE(O), L')
5: for class K E [K] do
3.2. Adapting Vanishing Ideal Algorithms to the Latent Space
6: Z/2 +- (x*)
7: pf....,pth + VANISH(Z*)
end for
In the previous section, our pruning approach assumed ac- cess to the full set of generators for each class's 's vanishing ideal. While ABM and OAVI (both fully described in Ap- pendix C) are known for constructing sparse polynomials (Wirth & Pokutta, 2022), directly applying these methods to nigh-dimensional latent spaces with large sample sizes is computationally infeasible. Recent advances demonstrate their effectiveness on et al., 2023), but making them tractable in large-scale set- tings requires further adaptation.
for class Ks E [K] do
10: pf Pf...., p/ t- PRUNE((A) ...... .................. ....)))))
11: g² +
12: end for
13: G (g) .g',....g** R²), Wr INIT(K,n) +..,,
14: WF + TRAIN (WP.G(Z)
datasets (Wirth
15: Φ.. < WW Go
We propose three key adaptations. First, we adopt stochastic variants of OAVI or ABM, enforcing vanishing on random subsets of each class' rather than on the entire class dataset. This significantly reduces computational overhead and mem- ory usage. Second, we perform dimensionality reduction (e.g., via PCA) in the latent space, effectively reducing the dimensionality of input features of OAVI and ABM and
4. Theoretical Analysis
We establish learning guarantees for VI-Nets, leveraging the sparsity of the coefficient matrix obtained through pruning (cf. Figure 2). We demonstrate that VI-Nets can achieve lower spectral complexity (Bartlett et al., 2017) than baseline NNs, resulting in a tighter bound on the generalization error.
5

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
4.1. Spectral Complexity of VI-Net
R R$ for E N. Then, G can be represented as the Two layer NN (Wo W M.. where WM RRxx and Wc E R such that E O(S).
Following Bartlett et al. (2017), we formally introduce the spectral complexity of an NN. To that end, let [[]], and
[[]],a W,, w.w.pp....... I| ..m ..m.lp))l lllll denote the spec- tral norm and the (p, q)-norm of matrix W, respectively.
The entire VI-Net consists of the L' remaining truncated layers of the NN, the matrices W M. We from Lemma 4.4 and final, linear layer WF. The spectral complexity of Φ hence be computed analogously to Definition 4.1, albeit with I,' + 3 layers instead of L.
Definition 4.1 (Spectral Complexity). Let & be an NN of depth L with pr-Liischitz activation functions {o:)/= and weight matrices {W.)en i.., is given as the function or W --- Wr.- 0...00 W. The spectral complexity RS of Φ is defined as
4.2. Learning Guarantees for VI-Nets
RØ = [ P w.,ll )
(
I 2/3
3/2
2
We now analyze the spectral properties of matrices W M and We to establish an upper bound on the spectral complexity of the VI-Net, enabling the spectral complexity compari- son with the baseline NN. Consider an NN & with L lay- ers, defined by weight matrices { (W,) W and p:-Lipschitz activation funetions {a as per Definition 4.1. Let ΦF o o W 1/ L W // -.... 01 o W repre- sent the truncated NN of depth L' < L. The corresponding encoded VI-Net कi = We
|W 2/3
In the following, we will show that we can encode VI- Nets in the format of Definition 4.1, which allows us to provide learning guarantees, since by Bartlett et al. (2017, Theorem 1.) the generalization error of é is proportional to its spectral complexity Rs. To that end, we require the following assumption:
WM ΦI has T,' + layers and follows the format of Definition 4.1, incorporat- ing the polynomial layer G from (POLY), with G satisfying Assumption 4.2.
Let the polynomial layer G : R" - R be as in (POLY), where N := n + +.. + πk. Let further S SEN N be the number of monomial terms needed to evaluate G'add d E N the highest degree of any polynomial in G. We assume that there exists 7 > 0 such that ||e] ||ı S F i.e., the coefficient vectors ei of each polynomial p are bounded in the & -nom by T.
Assumption 4.2.
To simplify notation, we denote the weight matrices of Φv as (W....., W144)) with corresponding Lips- chitz constants (fI....,PP.43) Note that W, W for E (1....,I') L'} and (Wi+11, W142,, W1+3)) (WM, M, We, WF), analogously for the Lipschitz con- stants, however noting that for the new layers, we have (pr+1. P2.2, PL) = (d, 1, 1).
This assumption is justified as bounded coefficient vectors can be ensured either directly by the vanishing ideal algo- rithm (e.g., OAVI, where - is hyperparameter) or indirectly through our pruning and fine-tuning approach, which ad- justs and sparsifies the polynomials post hoc. Next, we introduce an activation function, that allows us encode the polynomial layer in the format of Definition 4.1.
For the remainder, we assume that the output of the trun- cated network कι is already restricted to the hypercuhe [-1, 1| In practice, this is achieved by appending a final tanh activation, which also enhances numerical stability (see Appendix C.4).
Definition 4.3 (InEx-activation function). Let d e N as in Assumption 4.2. Abbreviating ja == Σ - (1), the InEx- activation function of degree d is defined as
Theorem 4.5. Let the networks é, du and ΦU as above. Then, the spectral complexities of and dy can be related as R2 R2, where K is given by
R
R
Z
1 d!
Σ (2)) j=
The following lemma leverages that the InEx-activation function, based on the inclusion-exclusion principle, com- putes monomial terms by selecting the appropriate variables using the rows in the matrix W The vector a (Wx) thus represents the S monomial terms needed for evaluating G. These terms are then multiplied by the coefficient matrix We to complete the evaluation.
I/43 11 Ds
W:
WT
W
i= 1/41
2,1
II Pi
/|Well2
|
||
2.1
1/11
Lemma 4.4. Let and S as in Assumption 4.2 and define the stacked -xx-ciiivattion function a = (a ...., de (d)):
theee further exist X1, A2 > O, such that |wwto|
S A
6

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
and [ W/++3
I A2
W Ww+3//
then it holds that
efficiency. Our experiments use pretrained baseline neural network, leveraging only its latent outputs, and apply ap- proximate vanishing ideal computations to these features. The primary goal is to determine if the network's 's final lay- ers can be replaced with a polynomial layer from vanishing ideal generators, without retraining the entire model.
2,1
L'43
W.
S dTAiNN
i= 1-11
E/43
Σ i=/+1
211
S 2
S
+N + +
Specifically, we (i) demonstrate that VI-Net achieves com- petitive accuracy, and (ii) examine the impact of network truncation and polynomial pruning accuracy, parameter efficiency, and throughput (measured in images per sec- ond), Inspired by recent findings that deeper layers in large language models are often less effective than earlier ones (Gromov et al., 2025), we intend to replace deeper layers with polynomial layer for a more compact yet equally expressive model.
W
This result provides a theoretical way to set the hyperpa- rameters of our VI-Net. Given access to the weights of the baseline network, we can compute the spectral norms of its layers using power iteration or singular value decompo- sition. These norms, together with the theorem's bounds, allow us to determine appropriate values for the degree d of the polynomials, the number of generators N, the coef- ficient bound r and the number of monomials S to ensure the VI-Net has lower spectral complexity.
5.1. General Experimental Setup
All experiments use PyTorch (Paszke et al., 2019). We eval- uate classification on CIFAR-10/-100 (Krizhevsky, 2009) with ResNet models (He et al., 2015). VI-Net replaces the final layers of a pretrained network with polynomial genera- tors (Algorithm 1), where we use the following parameters: vanishing tolerance vú 0.1 (cf. Definition 2.3), maximal polynomial degree d = 5, and feature reduction to 128 di- mensions using PCA. Fine-tuning of the linear classifier and coefficient matrix uses SGD (learning rate 0.05, momentum 0.9) with standard data augmentations.
As a corollary, we derive a bound on the generaliza- tion error of a VI-Net, leveraging the spectral complexity (Theorem 4.5) and the margin-based generalization frame- work of Bartlett et al. (2017). The margin measures the gap between the network's s output logit for the correct label and the closest output for any incorrect label, i.c., dz (x)y - mmxx4y (x)
Corollary 4.6. Let Φu be a VI-Net of width w and let RØy, be given as in Theorem 4.5. Then, for (x. y). (I1.81)..... (Im: Vm) drawn i.i.d. from any proba- bälit distribution over Rd [K, we have with probability at least 1 - å over ((I;, V/ ) every margin y >> O satisfies
5.2. Impact of Truncation
We first analyze the impact of removing varying fractions of the final convolutional layers accuracy and the number of constructed generators for CIFAR-100 100, with CIFAR- 10 re- sults deferred to Appendix A.2.1. We compare three setups, all truncating the network at the same layer but differing in the layers they append and retrain: i) VI-Net, which ap- pends the polynomial layer followed by a linear classifier, ii) Linear Head, which appends only a linear classifier to the truncated network, and iii) Random Monomials, which appends polynomial layer with the same number of mono- mials and an identically shaped coefficient matrix as VI-Net, but with randomly sampled monomials instead of those found by the vanishing ideal algorithms.
P arg max (x)x + 1
||xx1 Ri In(w) + m
In ln(1/0)
S (rr:) +0
m
where
1 m
Σ1 Φz (2:)v C,(dv)= m
max é (x+) 5>
II the indicator function, and X = -x Im) Rdxm
The effect of truncation on performance. Figure shows how truncating ResNet-34 layers affects CIFAR-100 test accuracy. Removing layers and adding only a linear layer reduces network performance due non-linearly sep- arable latent features. In contrast, VI-Net maintains com- petitive accuracy even after removing half of the layers. It consistently outperforms the random monomials layer, highlighting that VI-Net's 's strength lies not just in the non- linearity of the monomials but in their discriminative power
5. Numerical Experiments
We assess the capability of VI-Net to replace entire con- volutional and linear layers in standard CNNs, aiming to maintain or enhance accuracy while improving parameter
Ouur code https://github.. cwt. approximat inpaaaiinggnnnnn ing www.eerrraaeeooeeaeeeeeeee folds
is available at
7

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
when determined by the vanishing ideal algorithms. At the start of the residual blocks, VI-Net and the monomials layer show similar accuracy, likely because more monomials are generated in this region (cf. Appendix A.2.2).
Table I. Flapsed time, number of polynomials and monomials gen- erated by the vanishing ideal algorithm vs. percentage of removed Layers: compare effect removing different fractions the final layers on the vanishing ideal algorithm.
CIFAR-10 (ResNet-18)
75.00%
% of Layers Removed
Monomials
Polynomials
Time (s)
70.00%
64.5%
8,386
53,132
6,468.36
53.3%
4,462
14,297
245.09
Accuracy
65.00%
46.7%
1561
4,346
11.73
26.7%
167
1,319
7.31
60.00%
6.7%
130
1,280
6,94
CIFAR-100 100 (ResNet-34)
Head Random Monomials
% of Layers Removed
Monomials
Polynomials
Time (8)
50,00%%
Vi-Net
75.8%
7,625
188,743
3368.76 76
Baseline (Test Accuracy: 74.8%)
50.1%
5,196
53,675
82.46
45.00%
43.9%
1,990
30,004
18.38
0%
10%
20%
30%
40%
50%
60%
70%
25.1%
196
12,978
7.46
Percentage of Layers Removed
7.1%
237
12,997
8,56
Figure 3. CIFAR-IO0 100 (ResNet-34); Test accuracy vs. percentage removed convolutional layers. As more layers are removed, classifier with only a linear head after truncation experiences sharp performance drop, whereas VI-Net maintains accuracy.
ble 2 shows that VI-Nets often achieve both higher through- put and competitive performance.
Table 2. Performance Comparison for VI-Net. Total Parameters
the sum of Baseline Parameters (truncated ResNet) and Polyno- mial Parameters (generated expansions), measured in millions (M). Throughput is measured the test split (batch size 256), averaged across random seeds with standard deviation indicated.
The effect of truncation on the generators. Table shows the number of monomials and polynomials ABM constructed after removing different fractions of layers in ResNet-18 and ResNet-34. Truncating earlier layers in- creases the intrinsic dimensionality (Ansuini al., 2019) of data manifolds, resulting in more monomial terms (cf. Appendix A.2.8). Despite significant truncation, the total number of generated terms remains well below the theo- retical limit for degree-5 monomials in 128 dimensions, indicating that the constructed generators remain sparse.
CIFAR-10 (ResNet-18)
Model
Total
Base
Poly
Acc (%)
Throughput (img/s
&
1.66M
L.8M
88.62
20506
VI-NetlS-Sman
2.84M
2.18M
0.66M
92.66
79307 = 15576
4.35M
3.96M
0.39M
92.92
71851
13987
VL.Net88
9.20M
881M
0.39M
93.02
62086
11291
ResNet18
11.24M
11.24M
92.89
66533 : 12577
CIFAR-100 (ResNet-34)
Model
Total
Base
Poly
Acc (%)
Throughput (imp/s)
5.3. Pruning the generators
VI.Ne134- Tiny
3.52M
0.67M
71.94
42064 7247
V1.Net34-Smm
5.88M
5.21M
0.67M
74.03
39611 + 6502
VI-Ne34-Mediam
14,60M
14,20M
0.40M
74.66
29285 :: 5791
Figure 4 demonstrates that pruning a large fraction of poly- nomials can significantly degrade accuracy, especially in early layers. On the other hand, moderate pruning largely preserves performance. Figure further confirms the valid- ity of the approach outlined in Figure 2: pruning polynomi- als suffices to reduce the number of monomials significantly.
VI-No14-Large
19.32M
18.92M
0.40M
74.78
27862
5358
ResNe134
20.35M
20.35M
74.75
28253
4: 4290
6. Conclusion and Limitations
We establish a connection between manifold learning and computational algebra, demonstrating how vanishing ideal algorithms can be used accurately discriminate different classes in the latent spaces of neural networks, despite said spaces being of much higher dimensionality than what is typically considered in the context of vanishing ideal algo- rithms. By proposing VI-Net, we showed that the found generators of the respective vanishing ideals can be used to construct polynomial layer that, when appended to a the truncated smaller network, can match the performance of
5.4. The parameter-efficenny of VI-Nets
We show that VI-Net achieves parameter efficiency by prun- ing up moderate number of polynomials. We create four variants by truncating increasing portions of the trail- ing layers, substituting them with the polynomial expan- sions: VI-Net-Tiny (retains roughly 30% of base layers), VI-Net-Smmal Small (roughly 50%). -eeNet-Medium (roughly 80%), and VL-Net-Large (removes only the last layer). Ta-
8

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
53, March 2021. ISSN 2196-1115. doi: 10.1186/
Polynomials pruned
$40537-0021-0444488 URL https: htt///dio////////
75.0%
100%
70.0%
Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. Intrinsic dimension of data representations in deep neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buce F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processigg Systems, volume 32. Curran Associates, Inc., 2019. URL https::/prooceddngs. neuriis сc /c//Speer_filessppperr////////// c: c pdf.
Accuracy
65.0%
50%
60.0%
55.0%
0%
0%
20%
40%
60%
Percentage Layers Removed
Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. Spectrally-normalizeed margin bounds for neural networks. In Guyon, I., Luxburg, U. V., Bengio, S. Wallach, H., Fergus, R., Vishwanathan, S., and Gamett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.,
Figure 4. CIFAR-100 (ResNet-34): Test accuracy vs. percentage of removed layers and the pruning fraction of polynomials. We compare the performance of different pruning ratios for removing different fractions of layers.
the full network with higher parameter-efficinng
cc cc/sapet_filee/ppper/////// http://rooceddnns..eeeer.rs.ssss 02223774001994555533333ee8e8 pdf.
2017. URL https::/proceed.gaa
However, limitations remain. If the latent-space class mani- folds overlap significantly or have high intrinsic dimension- ality, the polynomials may fail capture them accurately. Exploring nigher-dimensional latent spaces, such as those in transformer models, could be a promising direction for future work.
Brahma, P. P., Wu, D., and She, Y. Why Deep Learn-
ing Works: A Manifold Disentanglement Perspective. IEEE Transactions on Neural Networks and Learn- ing Systems, 279100:1997-2088 October 2016. ISSN 2162-237X, 2162-2388, doi: 10. 10.1109//NNS..015.
Impact Statement
2496947. URL http: document//348669/ Number: 10.
http:///eeeeeeeooreeeeee/ /eeeeepllrreeeeeeeeeeeeeeeeeeeee iee...rgeere. org/
This paper presents work that aims to advance the field of Machine Learning by introducing scalable and efficient methods for analyzing latent spaces in neural networks ing vanishing ideals. While this research primarily focuses on foundational advancements with potential applications in interpretability, efficiency, and generalization, we acknowl- edge that any progress in Machine Learning can have both positive and negative societal impacts. For this work, we do not identify specific ethical concerns or societal con- sequences that warrant further discussion. However, encourage the community to consider the broader implica- tions of deploying these methods in various domains.
Brown, B. C., Caterini, A. L., Ross, B. L., Cresswell, G. Verifying the union of
J. C., and manifolds hypothesis for image data. In The Eleventh International Conference on Learning Representations, 2023. URL httph:ttp///cc..... /ttp:////eeeeeeeeeeeemeeemeeeeee net net/forum? forum? id=Rvee9CAX4EL
Cayton, L. Algorithms for manifold learning. Technical re- port, UC San Diego: Department of Computer Science & Engineering, 2008. URL https: http///schhllassehcc org/ ht//uc/iii etteetee//eemmmm 8969r8tt
References
Cox, D. A., Little, J., and O'eea, D. Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra. Springer Publish- ing Company, Incorporated, 3rd edition, 2010. ISBN 1441922571.
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes, 2016. URL https:// arxiv. org. abs /alsc./1lc.... 016444
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., and Farhan, L. Review of deep learn- ing: concepts, CNN architectures, challenges, appli- cations, future directions. Journal of Big Data, 8(1):
Fassino, C. Almost vanishing polynomials for sets
limited precision points. Journal of Symbolic Computation, 45(1):19-37, 2010. ISSN 0747- 7171. doi:

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
002. UBL.httpes//ww.eeiinnnnedrreeeee com/ sci sct.aancee//tttcee///////7777700
doi: URL httpps:/www.ecieeceediiie acteno//atticcee/////22222000000
Fefferman, C., Mitter, S., and Narayanan, H. Testing the manifold hypothesis. J. Amer. Math. Soc., 29(4))833 1049, February 2016.
Kera, H. and Hasegawa, Y. Spurious vanishing problem in
approximate vanishing ideal, 2019.
Frank, M. and Wolfe, P. An algorithm for quadratic program- ming. Naval Research Logistics Quarterly, 3(1-2):95- 110, 1956. doi:
Kera, H. and Iba, H. Vanishing ideal genetic programming.
In 2016 IEEE Congress on Evolutionary Computation (CEC), pp- 5018-5025. IEEE Press, 2016. doi: 10.1 CEC.2016.7748325, URL https: org...oooooo 1109/CEC. 7748325.
URL
www.y. com/
doi /aoi/bb//1i /10.s//10.. 1002 nav. 38003000331
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview weriiiiwwee//eeeeee htt:://oommiiinnnneeeeeeeee
Khnulkov, V. and Oseledets, I. Geometry score: A method for comparing generative adversarial networks, 2018.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Hampel, F., Ronchetti, E., Rousseeuw, P., and Stahel, W. Robust Statisties: The Approach Based on Inftuence Func- tions. Wiley Series in Probability and Statistics. Wiley, 2011. ISBN 9781 118150689. URL https: http:tts: /books. google. de wwo.:/ooosssooooo ?ic XK.rreeex......
Levina, E. and Bickel, P. Maximum likelihood esti- mation of intrinsic dimension. In Saul, L.., Weiss, Y., and Bottou, L. (eds.), Advances in Neural Infor- mation Processing Systems, volume 17. MIT Press, 2004. URL
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015.
pdf.
Heldt, D., Kreuzer, M., Pokutta, S., and Poulisse, H. Approximate computation of zero-dimensional polynomial ideals. Journal of Symbolic Com- putation, 1461)11566-1591, November 2009.
Limbeck, J. Computation of Approximate Border Bases and
Applications. PhD Thesis, Universität Passau, 2014.
Livni, R., Lehavi, D., Schein, S., Nachliely. H., Shalev- Shwartz, S., and Globerson, A. Vanishing Component Analysis. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Ma- chine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 597-605, Atlanta, Georgia, USA, June 2013. PMLR. URL https: https/////coccccosss.... mir. mi.sprres/v28/ hwtp//iiiiiiiiii html. Issue: 1.
ISSN 07477171. doi: 10.101661jsc:200.111000
URL httpp::/1iinknghhab elvviirr.................ee....e com/ setraeve/p1///0777711100033 Number: 11.
Iraji, and Chitsaz, H. Principal Variety Analysis. In Levine, S., Vanhoucke, V., and Goldberg, K. (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learn- ing Research, pp. 97-108. PMLR, November 2017. URL https: //ppro/eeednggss.ceeee mlr. press/v78/ iraji17a. html.
Lunch.
Calabi-yau manifold visualization. https://en. www.wwiiiiii....mm ort.wkki////w//////. http/wiki/Fiieee Calabi png, 2007. Licensed under the Creative Commons Attribution-Share Alike 2.5 Generic license. You are free to share and adapt the work with proper attribution and under the same license.
File:
Jaggi, M. Revisiting Frank-Wolfe Projection-free sparse convex optimization. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Con-
MacKay, D. J. C. and Ghahramani, Z. Comments on 'max- imum likelihood estimation of intrinsic dimension' by e. levina and p. bickel (2004). Technical report, The Inference Group, Cavendish Laboratory, University of Cambridge, 2005. URL http: / /www. inference org. uk/mackay. Technical report.
ference on Machine Learning, volume 28 of Pro- ceedings of Machine Learning Research, pp. 427- 435, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. mlm. www.....r.../...mmm..m..m..m.... wwwpprressse.ppppp
URL wwtphhtsss///pceeeeeeeeee jaggil3. hatm.
Kehrein, A. and Kreuzer, M. Computing bor- der bases. Journal of Pure and Applied Alge-
Nash, J. Real algebraic manifolds. Annals of Mathematics, 56(3):405 421, 1952. ISSN 0003486X. URL http: / //www. jator. htt.:/rrrrrggstab//////ee..
bra, 205(2):279-29) 295, 2006. ISSN 0022-4049.
10

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.,, Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library, 2019. URL https:/ aettps://arxie //axxvivvv././///// http://awwpwp/a///a/////////o//t abs 1912.2. 01703.
Zimmer, M., Spiegel, C., and Pokutta, S. Compression- aware Training of Neural Networks using Frank-
Wolfe, pp. 137-168. De Gruyter, Berlin, Boston, 2025. ISBN 9783111376776.
doi: doi:10.1515/ 9783111376776-010, URL https::///oooo org/10..00.
1515/9783311337776--11
Pokutta, S., Spiegel, C., and Zimmer, M. Deep neural network training with frank-wolfe, 2020. URL https: //arxiv. org/ abs/ 2ba/20100 072433
07243.
Shaheen, F,, Verma, B., and Asafuddoula, M. Impact of automatic feature extraction in deep learning architecture. In 2016 International Conference on Digital Image Com- puting: Techniques and Applicattions (DICTA), pp. 1-8, 2016. doi: 10. 10.1100/DICCAA.016..77999
Shi, Z., Wei, J., and Liang, Y. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features, 2022.
Tsitsulin, A., Munkhoeva, M., Mottin, D., Karras, P., Bron- stein, A., Oseledets, I., and Müller, E. The shape of data: Intrinsic distance for data distributions, 2020.
Wang, I.., Hu, I.,, Gu, J., Wu, Y., Hu, Z., He, K., and Hopcrofi, J. Towards Understanding Learning Representations: To What Extent Do Different Neu- ral Networks Learn the Same Representation, Novem- ber 2018. URL http:// / arxiv. wwt.voorg/a//bb/bbobo/oo 11750. 1707 [cs, stat].
Wirth, E., Kera, H., and Pokutta, S. Approximate vanish- ing ideal computations at scale. In Proceedings of the International Conference on Learning Representations, 2023.
Wirth, E. S. and Pokutta, S. Conditional gradients for the approximately vanishing ideal. In Valls-Valls, G., Ruiz, F. J. R., and Valera, (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 2191-2209. PMLR, 28-30 Mar 2022. URL httpttss: htaptt:::/pp/oceeeeee /ttp///eeop/onses/e/eee//e/c/s.s mlr press/ v151/wirth22a.hi html.
Zhang, X. Improvement on the vanishing compo- nent analysis by grouping strategy. EURASIP Journal Wireless Communications and Net- working, 2018(1):111, December 2018. ISSN
1687-1499, doi: 10.1186611338--18--11133
URL http:://xeea--urrssppppooaaaa
sppinsercpen.cco//ttttilee//....
s13638-018-1112- Number: Number:

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A. Appendix: Additional details, experiments and figures
A.I. Implementation Details
For the parametcr-cfficien VI-Net implementation, we prune to fixed number of polynomials and monomial terms. Each variant includes a PCA layer reducing to 128 components, encoded monomials (512 x 128), polynomial coefficients (1280 512 12800 512), and final linear layer.
Table 3. Name of the layer at which we truncate the baseline NN.
Variant
ResNet18 (CIFARI0)
ResNet34 (CIFAR100)
Tiny Small
layer2. 1 1.bn2
layer3, 1 1.bn1
layer3. 1 1.bb1
layer3. ..aar.3..0 3.bb1
Medium
1ayer4.0 .bbbl
1ayer4.1
Large
1ayer4. .1.bnl
1ayer4.2.bn!
All BesNet18-basser variants use 512 monomials and 1280 polynomials, while ResNet34-based variants use 512 monomials and 12800 polynomials.
12

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2. Additional Experiments
We list additional experiments.
A.2.1. IMPACT OF TRUNCATION ON CIFAR-10
We apply truncations to standard ResNet-18, retaining 512 training images per class for constructing approximate vanishing ideals. We do not prune polynomials in this experiment. Both the linear layer and the coefficients were retrained for 20 epochs.
Figure shows that removing up to 20% of the final convolutional layers does not degrade the performance of the linear head compared to VI-Net, likely because the data is already linearly separable these layers. Additionally, observe that network with randomly generated monomials performs significandly worse than the linear head when removing between 0% and 30% of the final layers. However, becomes beneficial. Notably, VI-Net maintains higher accuracy and consistently outperforms both the linear head and the randomly sampled monomials.
layers removed, the non-linearity introduced by the monomials
90.0%
80.0%
Accuracy
70.0%
Linear
60.0%
Random
VI-Net
50.0%
Baseline (Test Accuracy: 92.9%)
0%
10%
20%
30%
40%
50%
60%
Percentage of Layers Removed
Figure 5. CIFAR-I0 (ResNet-18): Test accuracy vs. percentage of removed convolutional layers. The dashed gray line indicates the fully trained baseline. A linear layer retains accuracy only with minimal truncation, whereas VI-Net maintains high accuracy even with substantial truncation.
13

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.2. PRUNING AND MONOMIAL COUNT
In this experiment, we examine how the number of monomials required evaluate the generators varies with different pruning ratios. Figure 6 shows that pruning polynomials significantly reduces the number of monomials to be evaluated. Notably, when removing between 20% and 40% of the layers, the number of remaining monomials falls below the latent dimension of 128 (after PCA). This explains the increased susceptibility of these layers to accuracy degradation: excessive pruning of polynomials leads to the loss of too many monomial terms, reducing accuracy. Furthermore, this result suggests that ABM efficiently constructs representations in this range, as it retains relatively few monomial terms without requiring explicit pruning.
Polynomials
pruned
100%
103
Monomials
50%
102
10
0%
0%
10%
20%
30%
40%
50%
60%
70%
Percentage of Layers Removed
Figure CIFAR-100 (ResNet-34); Impact of pruning monomial count. Removing large fraction of polynomials reduces the number of remaining monomials, sometimes below the latent dimension of 128. This reduction explains the observed accuracy drop at high pruning ratios, particularly when removing 20% 50% the layers (see Figure
14

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.3. COMPARISON DIFFERENT VANISHING IDEAL ALGORITHMS
We construct VI-Net following the pipeline in Algorithm using different vanishing ideal algorithms. Specifically, compare ABM to OAVI with two oracle variants: Frank- Wolfe and Accelerated Gradient Descent. The baseline network is ResNet34, trained and tested on CIFAR-100.
Figure shows that all vanishing ideal algorithms yield similar performance for the resulting VI-Nets, with ABM slightly outperforming the OAVI variants in intermediate layers. This similarity suggests that the algorithms capture comparable structural propertics of the data. Additionally, since we retrain the coefficients the constructed generators, the primary difference lies in the selection of monomial terms by each algorithm.
75.0%
72.5%
70.0%
Accuracy
67.5%
65.0%
62.5%
VI-Net (ABM)
VI-Net (AGD)
60.0%
VI-Net (CG)
57.5%
Baseline (Test Accuracy: 74.8%)
0%
10%
20%
30%
40%
50%
60%
70%
Percentage of Layers Removed
Figure CIFAR-100 (ResNet-34):: Test accuracy vs. percentage removed convolutional layers. The dashed gray line indicates the fully trained baseline. We compare different VI-Nets, constructing polynomials using ABM, as well as OAVI with Accelcrated Gradient Descent Frank- Wolfe.
15

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.4. EFFECT PRUNING CIFAR-10 (RESNET18)
In this experiment, examine the effect of polynomial pruning at different fractions ResNet18 trained on CIFAR-10 Figure 8 shows that removing up to 40% of all layers has negligible impact accuracy, suggesting that many polynomials not essential for classification and can be discarded. A significant accuracy drop is observed only at extreme pruning levels (> 90%). In earlier layers, pruning reduces performance more noticeably, likely due to the loss of expressivity, which the polynomial layer would otherwise compensate for.
Polynomials pruned
100%
90.0%
Accuracy
85.0%
50%
80.0%
75.0%
0%
0%
10%
20%
30%
40%
50%
60%
Percentage of Layers Removed
Figure CIFAR-I0 (ResNet18); Impact of polynomial pruning on test accuracy. Accuracy remains stable up to moderate pruning levels, but excessive pruning especially in early layers Icads to significant drop due reduced expressivity.
16

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.5. EFFECT PRUNING OAVI-CG BASED VI-NET
We analyze the effect of pruning the generator for OAVI using Conditional Gradients (Frank-Wolfe) as the oracle. The results exhibit similar trend to Figure 4, though the accuracy shows slightly weaker dependence the pruning ratio. This is likely due to the higher number of constructed generators, which provides additional redundancy.
Polynomials pruned
100%
75.0%
70.0%
Accuracy
65.0%
50%
60.0%
55.0%
0%
0%
10%
20%
30%
40%
50%
60%
70%
Percentage of Layers Removed
Figure 9. CIFAR-100 (ResNet34): Impact of polynomial pruning on test accuracy for OAVI-CG. While the general trend mirrors Figure 4, the higher number generators reduces sensitivity pruming.
17

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.6. EFFECT PRUNING OAVI-AGD BASED VI-NET
We analyze the effect of pruning the generator for OAVI using Accelerated Gradient Descent the oracle. The results exhibit similar trend to Figure 4, though the accuracy shows slightly weaker dependence the pruning ratio. This is likely due to the higher number of constructed generators, which provides additional redundancy.
Polynomials pruned
100%
75.0%
70.0%
Accuracy
65.0%
50%
60.0%
55.0%
0%
0%
10%
20%
30%
40%
50%
60%
70%
Percentage of Layers Removed
Figure 10. CIFAR- 100 (ResNet-34): Impact of polynomial pruning test accuracy for OAVI-AGD. The overall trend resembles Figure 4. though the higher number generators reduces sensitivity to pruning.
18

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.7. TRUNCATION Vs. NUMBER OF GENERATORS
We analyze the number of polynomials and monomials constructed by ABM when removing different layers of ResNet34 on C1FAR-100. A clear trend emerges: earlier layers produce more monomials and polynomials, increasing computational cost. However, the start of residual blocks, observe unusually high number of constructed generators, suggesting shift in the algebraic structure of the data at these layers.
Layer vs Monomials and Polynomials
Mononiia's
Polynomials Time
10'
Time (3)
100
10
101
10f
Lyya2
Laya2 lod
Jayaz Aml
Jaye2 Iosl
Jaye2 2iml
Jaye2 2
Joye2 1.ml
Espart lopert 1la2
Emyart fac Olml
Emaad fi!
Lapar3 fael
Lapar 22 fmi
Laport 2 los!
layort lol
layert Iod
layer' lnl
layert lac?
Imvat loyert foil
Lanad Olad
Lavard Lai
Layard Led
Layint ba!
Layink 2bx
Layer
Figure 11. CIFAR-100 100 (ResNet34): Number of constructed monomials and polynomials across network layers. Farlier layers generate more terms, increasing computation time, while residual block houndaries show an unexpected increase generators, indicating structural changes in the data.
19

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
A.2.8. INTRINSIC DIMENSION VS. VANISHING IDEAL
In this experiment, we analyze the intrinsic dimension of the data manifolds different layers of ResNet34 and its effect on the number of constructed monomials. We estimate the intrinsic dimension using the Levina-Bickel estimator (Levina & Bickel, 2004) with the Bauckay-Charammnn extension (MacKay & Ghahramani, 2005).
Figure 12 suggests a strong correlation between intrinsic dimension and the number of constructed monomial terms. This aligns with intuition: a more complex underlying manifold requires a more intricate algebraic variety, necessitating a larger number monomial terms.
Number of Monomials
4000
2000
20
22
Intrinsic Dimension (ID)
Figure 12. CIFAR-100 100 (ResNet34): Correlation between intrinsic dimension and number of monomial terms. Higher intrinsic dimension- ality corresponds to greater number of constructed monomials, reflecting increased algebraic complexity.
20

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
B. Appendix: Theoretical Analysis and Learning Guarantees
B.1. Proofs
We provide the proof for the theoretical results.
B. 1.1.1. InEx-ACTIVATION FUNCTION
The InEx activation function is a theoretical tool that introduce encode VI-Net in the form of Definition 4.1. For illustration, compute several monomials. Example B.1. Consider the degree monomial ay. This can be expressed as
zy = 2 ((x + y) x² 12)
which requires only addition and squaring operations. We generalize this approach to degree 3 monomials using the inclusion-exclusion principle. For example, the monomial zyz can be computed by considering all subsets of the variables {z, y, } and their contributions the cube expansion:
(x + + = + 22² 3r y 1-=3x²22 3yfx + +yy + 32 + 32 y + 6zyz.
To isolate ryz, we subtract terms involving lower-order monomials:
1 ((x + シ + (xc y)² (xr -y (y+ + y 23)
ryz 6
This approach also applies to monomials where variables appear with higher powers, such as z²y. To handle such cases, we replace one variable with a duplicate of another. For instance, setting = = x allows us to rewrite x²y y in terms of the cube expansion above.
The following lemmas formalize Example B.I.
Lemma B.2 (Multiplication with InEx-activation). Let a be given as in Desinition 4.3. Let I1,..., Fd "....,,Bal lR and let
(L1...., I2a) ) be the tuple of all subsets of [d], ordered by cardinality. Then, for z with 2j = =Eie,, 2; it holds
d
a (z) = xt.
Proof. This an application of the inclusion-ecclasion-pppiiipepp Note, that by the multinomial theorem, it holds
Σ Di NE,
n!
k
Σ
k==n
VET
Therefore monomial of degree is in the support of (Eies, x, coefficient does depend on the cardinality of Ij. which allows to apply the inclusion-exclusion principle the sum. Thus, after rescaling by nf only the term [ 24 survives.
if and only all variable indices are in Iy. Further, the
Lemma B.3 (Computing monomials with InEx-activationn. Let Zb E IR and a, 2 be given such that Ek= d. Then, there exists a matrix W with & columns and 2d rows, such that
a (d(Wx) = II
iml
21

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
Proof. First introduce artificial variables
which all to x4. Define the set I ((1, : E [k], [oi]]
which has cardinality d, since Σ; Ox d. Then, we can rewrite this product as I x; = [[(tg))r Zi,g. Let 13 C I be
one of the 2d subsets of I and define w = 2 (Ka))T, ei where e, denotes the i-th basic vector dimensions. Since Figg 2ú for all q S o; it holds that w Σ L(gg)),, Vig. Therefore, if we order the sets I by cardinality, we can apply
Lemma B.2 to obtain for W (W1..... W24)" that a (Wx) [( Fig II 2,
Lemma B.4 (Lipschitz property of InExxactivaation) Let a be given as in Definition 4.3. Then, a is d-Lipschitz on the image of W for x E 1,, from Lemma B.3.
Proof. From Lemma B.3, we know that the mapping x - a (W(x)) is equivalent to the mapping of the corresponding monomial, which is d-Lipschitz on [-1, Next, for each i E [k]. it holds of is row in W, therefore for x, E [ 1]
holds Wx Wx'll2 1 ||x x'12.. Since arbitrary points the image of W can be expressed as Wx with x E [- 1]
obtain
(Wx) - a² (Wx'Il2
[[x - a (WW'll
[x Wx'12
x x-x'12 x2|2
d x x' 1|
x x'
|
d.
Here, the second inequality due to the Lipschitz property of the map < (W()).
1.1.2. ENCODING A VI-NET AS DEFINITION 4.
By the previous section, we know that we can compute monomials employing the InEx activation function and a matrix WA. Mr
Lemma B.5, Assume G is given as in Assumption 4.2. Then, can be encoded as the two layer NN
(x())),
where WM M R*x*, and we stack InExxaccivation functions such that o (a"t,..., ds) R R and We E Nxs such that s O(8).
Proof. Construct the matrix W Ar by employing the matrices W from Lemma B.3 for each monomial to compute and stacking the respective InEx activation functions, such that (a ,...,, de). Thus, the i-th component of the vector a (W mx) will be the i-th monomial of the S polynomials we have to compute to evaluate G. Finally, we construct We by choosing the rows as the coefficient vectors of the polynomials in G. This concludes the proof.
1.3. SPECTRAL PROPERTIES OF VI-NET ENCODING
In this section, we obtain bounds for the norm of the matrices that are used to encode VI-Net in the format of Definition 4. We begin my analysing the properties of W
Lemma B.6 (Spectral properties of W M). Let Assumption 4.2 hold, and let W M be constructed as in Lemma B.2. Then, it holds that:
[TT
M la.1 <2'
WM 22 and
[ M llz
where S is the number of monomials, n is the number of variables and d the highest degree of any one monomial.
Proof. We bound the Frobenius norm of the matrix W as By construction, War has most S 24 rows. For each row, the squared euclidean norm is at most d?? Therefore, it holds
Walle= VS2d de 2
(I)
22

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
As W M la S W Walle, it follows that
W W
(2)
Next, for the 2,1 norm, recall that
I
Σ w
W M.ll2,
where W M. is the i-th column of W M. i.e., the i-th row of W M. Let w denote the row with the largest euclidean norm in Was. Ar. This row is part of a block required for computing a specific degree d monomial I 2 as Lemma B.3. Let j denote the index with the largest entry w, i.e. j E argmax;<(M) aę. For each of the other variable i' there is row of the form ogey + ayej. Thus, there are k such rows, i.e, in the j-th column of W, the term oj occurs k times, for each of the remaining variables. By our choice of it holds
Σ af S (h: 1)a3 + 0 S [ WM,
Therefore, holds that the largest euclidean norm of any row is bounded by the largest euclidean norm of any column. Note that the spectral norm of matrix is bounded from below by the column with the largest euclidean norm,i.e.,
W as l2 2 max [ Waaalll.
(4)
Thus, we conclude that [| M I1.11 22 2 M la and therefore
[ 1a.1 [
$ 2
Lemma B.7 (Norm properties of Wc). Let Wo E NxS have rows bounded in the l norm by T. Then, it holds
We SNS.,
[elk S VNT and
Wellz
where S is the number of monomials and N is the number of polynmials.
Proof. We have
Well S WNllo SVN
(5)
(6)
For the ratio, it holds
1 Vs
Wello I Wellz.
For any vector, its euclidean norm is always bounded by the norm.
Therefore, it holds
N | |W
Combining these results, we obtain
W 1ll. N
Walbo
<
Wella
Welles
V
NVS
23

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
1.1.4. PROOF OF THEOREM 4.5
Finally, we ready prove Theorem 4.5.
Theorem B.8. Let & be an NN with layers in the form Definition 4.1. Let G be a polynomial layer as in (POLY) satisfying Assumption 4.2, and è the corresponding VI-Net based the truncated NN â with L' layers. Then, the spectral complexity of to can be bounded as follows
E.
IIW
Σ
II
1Will
w.
I (M.C.F)
iE{M,C,FF
R
R..
L
I
II
W
i= 1/41
w.
Further, ussume the linear layer satisfies
[| S A1 and
W 11. TWol2
S X2. Then, it holds
II sM(MC
IW.ll2 & 2
|WT]]
$
N + +
|W
(M..FF)
Proof. We employ the absolute value function in G as an activation function, which is 1-L.pschitz and the TnEx activation function, which is d-Lipschitz on its domain, while the last linear layer has no activation function. Using our encoding from Lemma 4. 4.4, we get that the spectral complexity of this encoding is given by
d II
IWill
II 1=1
W
&E(M,C,F)
3/2
I 2/3
Σ iE(I....4' (E(MF..F)
||W 11//3
The spectral complexity of the baseline NN given by the definition as
L
W 121
3/2
R$
w
|
2/3
Therefore holds
WT
d
II
R3
(E(M..,F)
(e(M.C.F)
R
II Pal
Will
1=1+1
I
Multiplication by Rs yields the first result. For the second part, apply Lemma B.6 and Lemma B.7 and obtain
II |W.l22 2ia/5 /Nr X
sc (M,.C..F)
Reordering yields the result. For the sum, we proceed the same way.
24

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
In general, there is no priori bound the norms of the weight matrices, which is why focussed this comparison with the baseline network. We note that prior work (Pokutta et al., 2020; Zimmer et al., 2025) examined ways to train NN with bounded norms.
25

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
C. Vanishing Ideal Algorithms
In this section, we provide an overview of the vanishing ideal algorithms considered in our setting: ABM and OAVI. Both algorithms take a set of points and construct generators of the (approximate) vanishing ideal. They extend noise-free generator construction algorithms, which require exact vanishing, by allowing polynomials vanish up to tolerance v. Structurally, they follow the same principles exact algorithms but introduce robustness to noisy data.
Beyond returning set of generators, denoted as G in this section, these algorithms also provide a set O known as the order ideal. An order ideal is subset of monomial terms that is closed under division; that is, if monomial such as x²y belongs to O then all its divisors (e.g.. 1, æ. y, x²) must also be in O. fundamental concept related to order ideals is the border, which captures monomials that are one step outside the order ideal.
Definition C.1 (Border). Let O S T be an order ideal. Then, the d-border of O is defined as:
0,0 {u Ta te Oga,t + u, and uJ.
fundamental result from computational algebra states that every zero-dimensional ideal (such as the vanishing ideal of a finite set of points) admits generating set known as border basis. A border basis provides a structured way to represent the ideal while ensuring computational efficiency in polynomial interpolation and approximation.
order ideal. Then, G a border basis of } } be a set of generators for zero-dimensional ideal 7, and let O be an
Definition C.2 (Border Basis). Let g = (91,...,,
1. G consists of polynomials of the form
Iu Cu,t u E 80,
where each is a polynomial whose leading term belongs to the border 80.
2. The polynomials in G form a basis for the ideal T.
Intuitively, a border basis expresses each polynomial in the vanishing ideal in terms of monomials from the order ideal, ensuring that computations remain well-structured and stable. For a more detailed overview and the algebraic background we refer to Kehrein Kreuzer (2006).
For this section, note that we think of sample of points X as tuple, rather than set, require that the or- der of points is fixed. Then, for a given monomial u E TT we write the evaluation vector (u(xi),... ..... "......,(x,,,) m(x.)) as u(X). For
a set of monomials, such as O, we write the evaluation matrix O(X), where the entry indexed by and corresponds to the i-th monomial in evaluated at the j-th point.
C.I. Approximate Buchberger Möller Algorithm
The Approximate Buchberger-Möller Algorithm (ABM) constructs an approximate border basis, as described in (Limbeck, 2014). Given a term u; in the border, ABM computes the smallest eigenvalue and corresponding eigenvector of the matrix A, which contains the evaluations of all monomials at every point in the dataset. This process identifies the best-fitting polynomial relations while ensuring numerical stability. For detailed explanation, see Limbeck (2014).
26

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
Algorithm 2 Approximate Buchberger Möller Algorithm (ABM) (Limbeck, 2014)
Input: Dataset X, a vanishing parameter v 2
Output: An approximate O-border basis
1: == { g = d == 1, A, == (I,...,1) 1)² E g=.1
2: while 0₂O = {ui...., ") + do
3: for 1 to do
4:
A (w, (u,
5:
B A" A
6:
À = smallest eigenvalue of B
7:
if Á 2 then
8:
m = O
9:
= Sm...: 8m...., 81) == norm-one eigenvector of B w.r.t.
Assume that O Jo .a.....,01. 01]
11:
8m+ 14 Sm Om 8101
12:
+ g (g)
13:
else
14:
O - OU {
15:
end if end for
d d+1
19:
end while
20:
return (9.0) 0)
C.2. Oracle Approximate Vanishing Ideal Algorithm
The Oracle Approximate Vanishing Ideal (OAVI) algorithm, introduced by Wirth & Pokutta (2022), incorporates constrained convex optimization approach. Instead of computing the eigenvector corresponding to the smallest eigenvalue of AT A, OAVI solves constrained convex optimization problem at line 5 to determine the coefficient vector c. This requires a convex optimization oracle and a constraint region, chosen as the balll radius т. Like ABM, OAVI maintains a record of the evaluated monomials in the order ideal O within matrix A. Wirth & Pokutta (2022) the oracle was (among others) choosen as Conditional Gradient, which known for constructing sparse solutions.
Algorithm Oracle Approximate Vanishing Ideal algorithm (OAVI) (Wirth Pokutta, 2022)
Input: Dataset X, vanishing parameter vb 2 0, coefficient bound r 2 2
Output: The set of generators g C P and the set O C
1: O = 9 = 0, d
2: while 0₂0 = (u....,, 4 * do
3:
for 1,..., do
4:
{ 10 A (X). (X), E 1
5:
c argmin m
Ay + b
6:
g + u, E/=1 1=1 Ctt,
7:
MSF X < then
8:
(
9: else
O- ou (mi)
end if
end for
13: d+ d
14:
end while
27

---

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
C.3. Stochastic Versions of ABM and OAVI
In our experiments, a key bottleneck of the vanishing ideal algorithms was memory, primarily due to the matrix A used in both methods. Given dataset X with samples, the matrix A has dimensions |O|, making storage and computation costly for large
To mitigate this, we employ stochastic approach by subsampling the dataset, using only m' < m samples to construct A. This significantly reduces memory usage while maintaining approximation quality. Additionally, we leverage lower precision arithmetic (c.g., float16) to further reduce the memory footprint without compromising stability.
C.4. Addressing Data-Sensitivity of Vanishing Ideal Algorithms
Vanishing ideal algorithms are highly sensitive to the scale of the data. Recall that v > 0 specifies the extent to which polynomials are required to vanish on the training data (cf. Definition 2.3). This parameter depends on the scale of the data. Further, vanishing ideal algorithms typically require the data to be contained in [-1, 1]". Thus, we have to ensure that both the training data and, during inference, the input of the polynomial layer is properly scaled. To that end, we propose employ variant of tanh-rescaling (Hampel et al., 201 1). Given the training dataset, collect and as the mean and standard deviation of the hidden activations at truncation, respectively. These be easily calculated just by using forward passes and are collected similarly to the running averages in batch normalization. Then, tanh-rescaling layer is applied right after truncation, mapping as
(3-M4
2
11
2 + tanh
This mapping ensures that the data is scaled to reasonable range.
28


Reviews:
[{"review_id": "mHENI7WK4z", "reviewer": "Reviewer_Gw8X", "summary": "Present a method to analyse latent manifolds - object representations at intermediate layers of deep networks - by finding a small set of polynomials which zero out at one object but not at other objects. This effectively trunfaces the deep network and replaces the head with a single, simple, non-linear layer - with each feature stemming from a single polynomial of a single object. Such construction is beneficial twofold: in demonstrating a parameter-efficient network with comparable classification performance as the original network and in characterising the latent manifold. \nThis construction pulls several different tricks to get those \"vanishing\" polynomials - dimension reduction by PCA, sparse ideal learning algorithms pruning approaches to allow for efficient inference, reduced precision implementation and numerical rescaling (if I could count them all).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for acknowledging the contributions of our work.\n\n> Those \"learning guarantees\" are usually vavous bounds, and thus, an improved upper bound on something may or may not lead to improved performance.\n\nWe agree that these bounds are often vacuous. In that sense, we agree that improved upper bounds do not necessarily imply improved performance; thank you for that remark, we will make this more clear in the paper. Our main theoretical contribution is embedding VI-Nets into the established framework of Spectral complexity, with our main theorem providing a theoretical basis for selecting hyperparameters.\n\n> As mentioned, this manuscript makes novel contributions to a well-researched area and thus naturally lack some \"essential references\"\n\nThank you for providing these references. We believe the provide valuable context and will add them to the paper.\n\n> to say something on the structure of the latent manifolds. See one suggestion below, perhaps for a future manuscript.\n\nWe also believe this is an interesting future direction of our work. Having constructed these polynomials, we are now able to use the entire toolbox of computational algebra to analyse our found structures. We do believe our work is already quite dense and leave these useful applications to future work - we will address the individual suggestions below!\n\n> * Using a fixed monomials budget what is the resulting accuracy?\n> * Aiming for a fixed target accuracy (e.g., 90%), how many monomials are needed at each layer?\n\nWe ran experiments on CIFAR100, testing monomial counts from 50, 100, and increasing by 100, noting their maximal degree. We limit ourselves to the following sparsities, as higher sparsity results in reaching less than 70% accuracy.\n\n| % of Layers Removed | Monomial Count | Max Degree among best Monomials |\n|-|-|-|\n| 7.1%|50| 1 |\n| 12.2%|50| 1 |\n| 17.5%|300| 2|\n| 25.1%|500| 2|\n| 30.4%|200| 2|\n| 33.3%|300| 2|\n| 38.4%|600| 2|\n| 43.9%|900| 3|\n\nWe observe that deeper layers need fewer and lower-degree monomials. Deeper layers achieve higher accuracy with a fixed monomial count, but adding more monomials eventually stops improving performance.\n\n| # Monomials | 17.5% layers removed | 33.3% layers removed | 75.6% layers removed|\n|-|-|-|-|\n| 50          | 68.47%  | 59.42%  | 43.42%  |\n| 100| 68.31%  | 59.83%  | 45.63%  |\n| 200| 69.66%  | 65.43%  | 48.63%  |\n| 400| 71.84%  | 72.35%  | 47.80%  |\n| 600| 72.54%  | 72.65%  | 47.26%  |\n| 800| 72.90%  | 72.73%  | 47.46%  |\n| 1000        | 72.96%  | 72.91%  | 48.81%  |\n\n\n> Can you provide some analysis on the effect of the maximal degree (keeping the number of monomials fixed)? Are you certain you need to \"limit the maximal degree to avoid overfitting\"? Assuming the number of monomials is fixed, I'm not sure about that.\n\nGiven that tanh-rescaling ensures that the output lies in the unit hypercube, a high degree polynomial with a fixed number of monomials could overfit/vanish on the dataset. However, fixing the number of monomials also has a regularizing effect (cf. Tables below). Our learning guarantee in Theorem 4.5 includes both the highest degree $d$ and the number of monomials $S$, indicating both are relevant. But we agree that our statement is a bit vague, in practice we chose $d = 5$ as maximal degree mainly because we observed that the algorithms never constructed polynomials of degree higher than $5$.\n\n> Can you demonstrate a clear benefit of the fact that \"the spectral norm of the VI-Net is provably lower than that of the base NN\"?\n\nThe lower spectral norm of the VI-Net means it might generalize better, resulting in a **smaller difference between training and testing errors**. For example, below, we see that the VI-Net has a smaller gap between train and test errors in almost all configurations compared to the base neural network.\n\n| % of Layers Removed | Gap Train vs Test Error     | Δ from Baseline |\n|-|-|-|\n| 75.8%| 1.7843| +0.6643|\n| 50.1%| 0.9639| −0.1561|\n| 43.9%| 0.8502| −0.2698|\n| 25.1%| 0.9005| −0.2195|\n| 7.1%| 1.1024| −0.0176|\n\n**Baseline (CIFAR-100, ResNet-34)** = 1.12\n\nFor the experiment you suggested, one can analyse the impact of the number of monomials $S$ on the generalization gap. Fewer monomials typically lead to a smaller generalization gap.\n\n| # Monomials | 17.5% removed | 33.3% removed | 75.6% removed |\n|-|-|-|-|\n| 50| 20.27%| 11.10%| 0.87% |\n| 100| 20.26%| 12.04%| 0.67% |\n| 200| 26.64%| 19.80%| 4.45% |\n| 400| 27.55%| 25.67%| 11.88%|\n| 600| 27.17%| 26.08%| 18.18%|\n| 800| 26.94%| 26.31%| 24.79%|\n| 1000| 26.91%| 26.27%| 28.68%|\n\n**Baseline (CIFAR-100, ResNet-34)** = 25.25\n\n> Can you clarify which tricks are essential for the results and which just improve your running time slightly?\n\nTanh-rescaling ensures convergence. Stochastic algorithms speed up computations the most, while lower precision arithmetic offers mainly memory benefits.\n\nWe hope to have addressed your concerns. If you have any further questions, please let us know.", "claims_and_evidence": "Most claims seem to be well-supported by evidence, with the numerical experiments using ResNet on CIFAR-10/100 used to demonstrate the applicability to real-world data.\n\nThe theoretical analysis is an exception; while I do not doubt the validity of the analysis (demonstrating reduced \"spectral complexity\"), I have serious doubts about the implications. Those \"learning guarantees\" are usually vavous bounds, and thus, an improved upper bound on something may or may not lead to improved performance. As far as I could see, such improved performance was NOT demonstrated here.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria look good to me. See some suggestions/improvements below.", "theoretical_claims": "No, I do not have the capacity to verify the correctness of the theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This is a novel approach to handling a well-researched area: understanding object representations in deep networks. The novel ideas presented show how to solve many practical problems (finding ideals is seems at first irrelevant on computational grounds).", "essential_references_not_discussed": "As mentioned, this manuscript makes novel contributions to a well-researched area and thus naturally lack some \"essential references\", e.g.,:\n\n * Separability and geometry of object manifolds in deep neural networks. \n * Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth\n * Deep neural networks rival the representation of primate IT cortex for core visual object recognition", "other_strengths_and_weaknesses": "Strengths:\n * The manuscript presents a novel idea of truncating a trained network and creating a new kind of learned, object-focused final layer.\n * A large collection of technical solutions is needed to bring this idea close to real-world applicability.\n * The very interesting results demonstrate both that this endeavour can be successful and that it may be useful to gain insight into the structure of latent representations.\n\nWeaknesses:\n * Not enough is done to build on this analysis to say something on the structure of the latent manifolds. See one suggestion below, perhaps for a future manuscript.", "other_comments_or_suggestions": "The results are constructed such that the number of monomial terms is a product of the algorithm, while the maximal degree and representation dimension are fixed. I think this is a mistake - the correct comparison should have been:\n * Using a fixed monomials budget (e.g., maximal degree of d=5, 128 dimensions, and the best 100 (or 200 or 1000) monomials), what is the resulting accuracy? \n * Aiming for a fixed target accuracy (e.g., 90%), how many monomials are needed at each layer?\n\nThe above analysis would have demonstrated how and if later layers of deep network have superior representations so they need a smaller number of monomials for a given accuracy level or better accuracy for a given monomials budget.\n\nUnrelated, the presentation of \"Percentage of Layers Removed\" seems unnatural to me: I would expect to look at \"percent of layers used\" with increasing accurace.", "questions_for_authors": "* Can you provide some analysis on the effect of the maximal degree (keeping the number of monomials fixed)?\n * Are you certain you need to \"limit the maximal degree to avoid overfitting\"? Assuming the number of monomials is fixed, I'm not sure about that.\n * Can you analyse the accuracy using a fixed number of monomials constructed from different layers, or the number of monomials needed to achieve 90% accuracy?\n* Can you demonstrate a clear benefit of the fact that \"the spectral norm of the VI-Net is provably lower than that of the base NN\"?\n* Can you clarify which tricks (e.g., lower precision aritmatic) are essential for the results and which just improve your running time slightly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9XnyIsAVbP", "reviewer": "Reviewer_cAHx", "summary": "This paper introduces efficient methods for approximating vanishing ideals of class \"manifolds\" in a latent space of a neural network classifier, and uses the resulting approximate vanishing ideals to replace later layers of a truncated classifier neural network with a linear combination of polynomial features, calling the resulting classifiers \"VI Nets\". The paper includes theoretical discussion suggesting that such VI Nets might enjoy improved generalization properties, and demonstrates empirically that for a given transition layer, they preserve more accuracy from the initial classifier network than simply replacing the later layers with a linear head. The paper shows that VI Nets can navigate an accuracy-throughput tradeoff.\n\n# Update after rebuttal\n\nI appreciate the authors' thorough response to my questions and comments. While they addressed a number of technical concerns, I maintain my reject recommendation.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful comments. We have grouped related concerns and clarifications below and revised our paper accordingly.\n\n**Overall Clarifications and Algebraic Geometry Context**\n\nWe used the manifold hypothesis to motivate our work, but we agree that currently, some of the wordings are not entirely mathematically precise. Let us make a few clarifications:\n\n> in some theoretical senses \"almost all\" manifolds fail to be algebraic\n\nIn line 42, we cited [Nash, 1952] to reference Tognoli's Theorem that every *compact smooth* manifold is *diffeomorphic* to a nonsingular algebraic set. We revised the wording to clarify that some manifolds may not be described by polynomials.\n\n> the zero set of the vanishing ideal will be super set of the original manifold.\n\nPlease note that we in lines 162-167 we explicitly state that \"recovering [the underlying manifold] $U_k$ exactly from a finite sample is impossible without structural information, as $U_k$ could range from being as minimal as $Z_k$ to as broad as $\\mathbb{R}^n$\" and provide further context in section 2.3.\n\n> \"while not necessarily linearly separable.\"\n\nWe agree and will modify the statement accordingly. Thanks!\n\n> the last paragraph of section 2.1 effectively assumes that the $Z_k$ are pairwise disjoint\n\nCorrect, we implicitly state this in line 80-82, but will make it more explicit here.\n\n**Experimental Evaluation and Baselines**\n\n*Linear Probing:*\n\nFirst of all, we agree that Linear Probing should be mentioned earlier in the paper and will revise the manuscript to make this more clear. Thank you for pointing out that this is not easily accessible. Secondly, we understand the confusion regarding the accuracy drop in Fig. 3, whose content was poorly communicated from our side. At 0% removed, we removed *0% of convolutional layers*, but still remove the avg.-pooling and linear classifier (lines 332-334, despite badly worded), which may result in a performance drop. We make this more clear to not leave the reader behind to why \"0% removal\" actually does something. Last but not least, we retuned the hyperparams of the linear probing baseline, improving its performance (https://imgur.com/a/SOpcrEJ). In general, the accuracy drop aligns with the paper you mentioned, reporting a 6% performance drop when using a linear head after the last convolution - which is a non-insignificant drop (cf. Figure 4 in arxiv.org/abs/1610.01644). Now, while VI-Net still outperforms, both methods perform similarly when removing the trailing convolutional layers. \n\nWe similarly observe the phenomenon that later layers lead to simpler, sometimes linear, polynomials (cf. Figure 11), aligning with the finding that classes become increasingly linearly separable in later layers; the neural collapse phenomenon is further supported by the good performance of the linear probes. VI-Nets could be seen as a natural extension of linear probing. Could you elaborate on what kind of additional baseline you would like to see?\n\n\n*Further experiments:*\n\nTo broaden our experimental evaluation, we added TinyImageNet experiments (https://imgur.com/a/Beh0DN7). Further, we conducted additional experiments measuring excess risk to support our hypothesis (please cf. our response to reviewer Gw8X). \n\nWe will add the evaluation of ResNet-18 on CIFAR-100 to properly contextualize the throughput improvements, however noting the following: While VI-Net's speedup might not match quantization and hardware optimization, we still observed a notable speedup despite non-optimized hardware, indicating the usefulness of the found generators in describing data manifolds.\n\n**Remarks regarding theory**\n\nWe do agree that not every VI-Net has a lower spectral complexity. Theorem 4.5 allows to choose the hyperparameters of our pipeline such that the bound we have on the newly added terms is lower than what can be measured from the baseline NN (see the answer we provided to reviewer 9he7 on the question on when VI-Net has a lower spectral complexity).\n\nFurther, you are correct that the baselines can be analyzed using the same spectral complexity framework. This enables us to compare the spectral complexities of the baseline NN, VI-Net, and linear probing or a shallow wide NN within a unified framework. We think this is a strength of our work.\n\n**Other remarks**\n\n> Fig 12: the number of generators and the dimension of the variety are anti-correlated.\n\nWe think this is a result of the *approximate* VI algorithms. If the underlying data manifold has lower dimension (e.g. a line), fewer monomials are needed to construct polynomials that approximately approximately vanish on the sample.\n\n* We fixed the typos and will include your suggestions, thanks.\n* Lem. 4.4: The $d_i$ are the degrees of the InEx activation functions. \n* Yes, in Tab. 1 that is the elapsed time to compute the generators. \n\nWe hope to have addressed your concerns, please let us know if further clarification is needed.", "claims_and_evidence": "- \"Such manifolds can often be described as the zero set of a system of polynomial equations\" hinges on what is meant by often. I agree that many standard examples of manifolds are zero sets of polynomials, however in some theoretical senses \"almost all\" manifolds fail to be algebraic. See e.g. https://mathoverflow.net/questions/4895/the-relationship-between-complex-and-algebraic-geomety\n- Similarly this is false: \"A key observation, and central motivation for our work, is that if the latent space is a manifold, it can be algebraically characterized as the zero set of certain polynomials\". In general, the zero set of the vanishing ideal will be a super set of the original manifold. There are similar issues elsewhere, such as \"Unlike methods that approximate manifolds through indirect means like simplicial complexes (Khrulkov & Oseledets, 2018) or k-nearest neighbor graphs (Tsitsulin et al., 2020), our approach directly captures their algebraic structure through polynomial roots\" \n- \"A key insight is that data points from different classes in the latent space, while not linearly separable, can be distinguished through polynomial separability\" should say something like \"while not necessarily linearly separable.\" Since they might be linearly separable, and in some theoretical frameworks asymptotically *are* linearly separable (for example, in the infinite width limit).\n- the last paragraph of section 2.1 effectively assumes that the $Z^k$ are pairwise disjoint. This is probably always true in practice, but it should be stated as an assumption. On the topic of this section, squares would be preferable to absolute values since they are polynomials :).", "methods_and_evaluation_criteria": "I think the limitations section could mention something about experiments only covering ResNets and CIFAR datasets (in particular this restricts to the vision domain). \n\nThe experiments primarily focus on \"end to end testing\" of the entire VI Net package. I could imagine additional experiments that would more directly target the underlying hypotheses (e.g. empirically computing some of the terms appearing in the spectral complexity estimates ...)", "theoretical_claims": "I did not read the proofs in the appendix. Regarding the theoretical claims, while the paper derives formulas that could be used to estimate spectral complexity of VI Nets (and relate their complexity to that of the network from which they were derived), since there are no theorems specifically proving that they have lower spectral complexity under certain assumptions, I think the introduction of the paper should be more cautious about suggesting improved generalization properties of VI Nets.", "experimental_designs_or_analyses": "Significant concern: I do not understand why an accuracy difference is observed in figure 3, even with 0% of the layers removed. In particular, I do not understand why the Linear Head method drops all the way to 65% accuracy, which is quite bad for a resnet34 on CIFAR 100. There are publicly available training scripts that claim to achieve accuracy in the high 70s for resnet34s on CIFAR 100.\n\nRegarding table 2: I think that including a resnet18 on CIFAR100 would be a useful baseline. More broadly, there are many methods which practitioners can use to achieve throughput improvements of the scale being demonstrated by VI nets (up to about 2x), with minimal impact on accuracy (hardware friendly inference frameworks, quantization ...). Additional comparison with/discussion of these would help contextualize table 2.", "relation_to_broader_scientific_literature": "Things that come to mind are: empirical science of representations of data in neural network latent space, and application of computational algebraic, geometry to data science. These are both somewhat broad areas, I don't have specific references in mind.", "essential_references_not_discussed": "Not beyond those linked above and below.", "other_strengths_and_weaknesses": "My primary concerns with this paper:\n\n- there's a variety of work showing that classes become increasingly linearly separable in later and later neural networks layers (https://arxiv.org/abs/1610.01644, https://arxiv.org/abs/2004.06093 to name a couple). \n- In particular, the former paper shows that truncating a neural network at an intermediate layer, and simply immediately learning a linear layer can result in performance comparable to that of the original network, *without* any intermediate steps of dimension reduction, tanh rescaling and bit crushing, vanishing ideal computation ... NOTE: I realize this baseline appears in Fig. 3, and that the proposed method of VI nets is shown to result in higher accuracy, however the fact that linear probes are not mentioned until the penultimate page is an issue for me, since as a reader, I was thinking \"what about linear probes\" on page 1.\n- The latter reference suggests that the process of neural network training is already encouraging class separation, one extreme example of this being \"neural collapse\" (https://arxiv.org/abs/2008.08186, https://proceedings.mlr.press/v145/e22b.html). So another baseline if one wanted a shallow or network that achieves linear class separation would be to try to train a shallow or network to the point of neural collapse. I do not see this baseline in the paper.\n- I could be mistaken, but the baselines suggested above might be amenable to similar spectral complexity analysis (definitely the first one is, since it will share the feature that lots of terms coming from the common first L layers cancel).", "other_comments_or_suggestions": "- Def. 4.2, should it be \"d choose i\"?\n- Lem. 4.4, what are the subscripted $d_i$?\n- \" Specifically, if the product of the spectral norms of the truncated layers is smaller than the bound for the newly added layers ...\" Do we not want the opposite to ensure the VI Net has lower spectral complexity? Regardless, using some notation here would make the sentence easier to parse.\n- it would be helpful to have a companion to figure 3 that incorporates the parameter counts of the various models.\n- In table 1 is elapsed time the time it took to compute (approximate) ideal generators?", "questions_for_authors": "Not beyond those above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L55Eb6Y7V7", "reviewer": "Reviewer_9he7", "summary": "The paper aims to improve parameter efficiency of neural networks by truncating a pretrained neural network, finding the polynomial generators of the vanishing ideal of (a finite sample of) the latent manifold, and transforming the latent space into a linearly separable feature space. The proposed method improves upon existing vanishing ideal algorithms, e.g. OAVI & ABM, by (1) performing PCA on latent space for dimensionality reduction, (2) pruning the found generators via class distinctiveness, and (3) other measures to improve computational efficiency such as stochastic sampling and lower-precision arithmetic. The paper also provides theoretical guarantee that the resulting VI-Net has lower spectral complexity and thus better generalization ability than the untruncated base model. The experiments on the CIFAR dataset demonstrate that VI-Net has comparable prediction accuracy to ResNet baselines while being more efficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we appreciate the thorough feedback. Let us address your comments in detail.\n\n> However, the evaluation is performed only on the CIFAR dataset. It would be better if the authors could also include other commonly used benchmarks for image classification.\n\nIn general, we agree that it would be good to include more benchmarks, albeit this was not the main focus of our work. Scaling vanishing ideal algorithms to larger datasets remains an open problem and is an active area of research [Wirth et al., 2023], our work is a step in this direction, with multiple contributions to that end. However, to directly address your concern, we have run preliminary experiments on TinyImageNet using ResNet-50. We observe that the performance is comparable to the CIFAR-100 and CIFAR-10 experiments, indicating that the method works for larger datasets. We will add these results to the paper and run further experiments on image classification benchmarks for the camera-ready version of the paper. Please find the results here https://imgur.com/a/Beh0DN7. \n\n\n> The only issue is that there seem to be no direct comparison between VI-Net and the base model. For example, under what condition does , i.e. VI-Net has a lower spectral complexity?\n\nThe main motivation for using the spectral complexity framework is exactly that it allows for comparison of generalization bounds of different neural network architectures and does more than simply providing a learning guarantee. We can employ Theorem 4.5 to derive for each layer $L'$ a bound on the norms of the newly introduced parameters that is strictly lower than that of the baseline network. This provides a theoretical justification for the use of VI-Net.\n\nExample: suppose we remove the last 3 layers in a pretrained NN. We can compute the spectral norms of their weight matrices $W_1, W_2, W_3$ and compute the product of these terms, i.e., $\\prod_{i = 1}^3 \\|W_i\\|_2$. \n\nSimilarly, we can compute $\\sum_{i = 1}^3\\|W_i^T\\|_{2,1}^{-2/3}\\|W_i\\|_2^{-2/3}$ explicitly. Next, we can *choose* the number of monomial terms $S$, the number of polynomials $N$, the highest degree $d$, the bound on the norm of the coefficient vectors of the polynomials $\\tau$ (Note that these are all hyperparamters of our algorithm pipeline). \n\nIf we have further bounds $\\lambda_1, \\lambda_2$ (cf. Theorem 4.5) on the norms of the final linear weight matrix (either by explicit constraints or implicit regularization), then we can always ensure that \n- $2^dd\\tau \\lambda_1 \\sqrt{NS} \\leq \\prod_{i = 1}^3 \\|W_i\\|_2$ \n\nand \n\n- $2^{2d/3}S^{2/3}+ N^{2/3}S^{1/3} + \\lambda_2^{2/3} \\leq \\sum_{i = 1}^3\\|W_i^T\\|_{2,1}^{-2/3}\\|W_i\\|_2^{-2/3}$. \n\nThus, the spectral complexity of VI-Net is lower compared to the original baseline NN by Theorem 4.5. \n\n> The codebase in the supplementary material seems not runnable.\n\nWe apologize, this was an oversight on our side while anonymizing the code. We have now ensured that all necessary files are included in the repository, and you should be able to run the code without any problems. If the Area Chair permits, we would be happy to share the complete codebase with you (currently only links to images are allowed).\n\n\n> Also, a dedicated section to related works would be helpful.\n\nWe are fairly confident to have included all relevant works with respect to vanishing ideal algorithms, but we appreciate the suggestion of adding a dedicated related work section. We think this will improve the presentation of our work and will add this section to the revision of the paper as soon as possible. Thank you for your suggestion.\n\n> Can we apply the methodology of VI-Net to regression tasks?\n\nThis is an interesting idea. Indeed, one can apply the standard approach of discretizing the regression problem transforming it into a classification problem (i.e. binning numerical outputs). Do you have any other suggestions? We believe this could be an interesting application and thank the reviewer for this idea. \n\n> Moreover, apart from building new models for prediction as you have done in this paper, can we use vanishing ideal algorithms to analyze the structure of the latent manifold itself and provide some kind of interpretability?\n\nWe do observe a high correlation between the intrinsic dimensionality of the data and the complexity of the constructed generators (cf. Figure 12). This aligns well with the findings in the broader literature and we believe that this provides insights about the manifold, although relatively hard to interpret.\n\n> L201, 202, 304, 285: [...]\n\nThank you, we fixed the typos and notational issues.\n\n> L323: What does $s \\in \\mathcal{O}(S)$  mean?\n\nThe size of the intermediate weight matrix scales linearly with the number of monomials.\n\n> L392: Does elapsed time in Table 1 refer to the running time of the vanishing ideal algorithm?\n\nYes.\n\nThanks again for your review, we hope to have addressed your concerns and questions. If you have any further remarks, please let us know.", "claims_and_evidence": "The main claim of the paper is that, by replacing the final layers of a pretrained neural network with generators of vanishing ideals, VI-Net can achieve comparable prediction performance with fewer parameters. This is clearly supported by experimental evidence, particularly those from Table 2. It should be noted, however, that the improvement in computational efficiency is moderate in order to maintain the prediction accuracy (e.g. ~1.3x throughput on CIFAR-100).", "methods_and_evaluation_criteria": "The proposed method builds upon existing vanishing ideal algorithms such as OAVI and ABM. To improve the computational tractability of these algorithms, the authors first perform a PCA to reduce the latent space dimensionality. Then, the generators found by these algorithms are pruned based on how distinctive they are across samples from different classes. The monomials with zero coefficients in all remaining polynomial generators are also pruned to reduce the number of function evaluations. Overall, the proposed method is straightforward and it makes sense for the purpose of this paper -- to improve parameter efficiency of classification networks.\n\nThe evaluation criteria include the classification accuracy and computational efficiency (in terms of parameter count and inference throughput), which make sense for the proposed method. However, the evaluation is performed only on the CIFAR dataset. It would be better if the authors could also include other commonly used benchmarks for image classification.", "theoretical_claims": "Section 4 of the paper claims that VI-Net can achieve lower spectral complexity and thus better generalization ability than untruncated networks. Theorem 4.5 and Corollary 4.6 provide explicit formulas for the spectral complexity and the generalization error bound of VI-Net. The proof is provided in the Appendix and it looks correct to me. The only issue is that there seem to be no direct comparison between VI-Net and the base model. For example, under what condition does $\\kappa < 1$, i.e. VI-Net has a lower spectral complexity?", "experimental_designs_or_analyses": "While only restricted to one dataset, the experiments are comprehensive and demonstrate the effectiveness of different components of the proposed method. The effects of various factors such as truncation proportion, pruning ratio, etc, are investigated through various controlled experiments.", "relation_to_broader_scientific_literature": "The paper mainly applies existing vanishing ideal algorithms, such as OAVI and ABS, to the NN latent space. It introduces some pre-/post-processing techniques to improve the computational tractability of these algorithms on high-dimensional spaces, but does not modify the vanishing ideal algorithms themselves.", "essential_references_not_discussed": "I am not quite familiar with the literature in vanishing ideal algorithms. I'd say that the paper has provided sufficient references throughout the paper for readers to understand the context, but I'm not sure if the paper missed any related works that should have been discussed. Also, a dedicated section to related works would be helpful.", "other_strengths_and_weaknesses": "The paper is well-written. Important concepts are explained clearly and the flow is smooth and coherent.", "other_comments_or_suggestions": "Typos & potential confusions:\n* L201: class $i$ --> class $j$\n* L202: the $p$ in top $p\\%$ could be confused with generator $p$\n* L304: $\\begin{pmatrix} i \\\\\\\\ d \\end{pmatrix}$ should be the other way around?\n* L323: What does $s \\in \\mathcal O(S)$ mean?\n* L285: $\\tilde \\phi_{L'} \\equiv \\mathbf W_F\\circ \\mathbf W_C\\ \\circ\\ ...$\n* L392: Does elapsed time in Table 1 refer to the running time of the vanishing ideal algorithm?", "questions_for_authors": "* Can we apply the methodology of VI-Net to regression tasks?\n* Moreover, apart from building new models for prediction as you have done in this paper, can we use vanishing ideal algorithms to analyze the structure of the latent manifold itself and provide some kind of interpretability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdHruKJue1", "reviewer": "Reviewer_T8ev", "summary": "The authors propose neural networks motivated by the vanishing ideals, called VI-nets. They also investigate to obtain parameter-efficient representations of the vanishing ideal generators. They investigate the performance of the VI-nets theoretically and empirically.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your feedback and will address your comments in detail.\n\n> In pratical situations, how to set the hyperparameters such as L' and S is not clear for me. Do you have any ideas about this?\n\nThank you for your question. Our paper addresses this from two perspectives, one theoretical and the other one practical. From the theoretical perspective, we can compute for each $L'$ with Theorem 4.5 a \"budget\" in the form of a maximal number of monomials $S$, maximal degree $d$ and the other hyperparameters of our pipeline. This yields a provably lower spectral complexity and thus a better bound. From the practical perspective, our experiments indicate that we can remove about 40-50% of all convolutional layers before we observe a significant drop in performance. With the choice of $S$ you raise another interesting question, which we already addressed in the appendix (cf. Figure 6). There we analyse the monomial count ($S$). We found that it is not advisable to choose $S$ to be significantly lower than the latent dimensionality, otherwise we observe a large performance drop. We hope that this answers your question - if not, please let us know.\n\n\n> In section 4.2, the authors investigate the theoretical analysis of generalization error based on the existing results regarding spectral complexity. However, how the geometric aspect of the proposed method helps alleviating the generalization error is not clear for me. Indeed, the bound of $\\kappa$ dependes on $2^d$, which becomes large if $d$ is large.\n\nYes, you are right, the bound of $\\kappa$ can become very loose if $d$ is large. However, in practice, we have always found it entirely sufficient to use values of $d$ not larger than $5$. To be more precise, we enforce the bound of $d$ as follows: we let the OAVI algorithm run until it terminates, and then we only take terms with maximum degree $d$. Despite setting $d$ to $5$, we never actually removed any terms from the ideal. In our computational setting, enforcing $d=5$ does in consequence not lead to any performance loss.\n\n> Although the authors empirically investigate the accuracy of the proposed method, I think investigating how the proposed method captures the underlying manifold for each class would be interesting. Is it possible to visualize or quantify how the learned polynomials separate the classes of the input data?\n\nWe agree that it would be very interesting to investigate how the learned polynomials capture the underlying manifold. Since visualizations in this high-dimensional setting are generally not feasible, we provide the following alternative to see how two classes in particular are being separated. For this, one can choose for each of the class a vanishing polynomial according to our scoring function, say $p_1$ and $p_2$. Then, a simple way to visualize how $p_1$ and $p_2$ separate classes is by using a scatter plot with $x$-axis the absolute value of $p_1$ and the $y$-axis of $p_2$. We provide such a scatter plot here, showing how a training batch is becoming linearly separable: [https://imgur.com/o3VlqZl](https://imgur.com/o3VlqZl)\n\nAs for a quantification of separability, this was the reason we introduced our scoring function. It essentially measures separability in the following sense: If the score is of the same order as the vanishing hyperparameter $\\psi$, then a polynomial is not only vanishing on its own class but also on other classes. If the score is higher then $\\psi$ then this indicates that the polynomial is not as approximately vanishing on other classes. Therefore the score introduced in line 195-197 in the second column is a direct way to quantify how well the learned polynomials separate the data. \n\n\n\nThank you again for your review, we hope to have addressed your concerns. If you have any further questions, please let us know.", "claims_and_evidence": "The prposed method is based on the observation that low-dimensional manifolds that underlies data is characterized by vanishing ideals. They construct a neural network with polynomial activation functions based on this observation. The methodology seems reasonable.", "methods_and_evaluation_criteria": "In pratical situations, how to set the hyperparameters such as $L'$ and $S$ is not clear for me. Do you have any ideas about this?", "theoretical_claims": "In section 4.2, the authors investigate the theoretical analysis of generalization error based on the existing results regarding spectral complexity. However, how the geometric aspect of the proposed method helps alleviating the generalization error is not clear for me. Indeed, the bound of $\\kappa$ dependes on $2^d$, which becomes large if $d$ is large.", "experimental_designs_or_analyses": "Although the authors empirically investigate the accuracy of the proposed method, I think investigating how the proposed method captures the underlying manifold for each class $k$ would be interesting. Is it possible to visualize or quantify how the learned polynomials separate the classes of the input data?", "relation_to_broader_scientific_literature": "N / A", "essential_references_not_discussed": "N / A", "other_strengths_and_weaknesses": "The authors encode the structure of data manifolds into neural networks using activation functions based on polynomials. The idea is interesting and relevent to the community.", "other_comments_or_suggestions": "N / A", "questions_for_authors": "Please see the questions and concerns in Methods And Evaluation Criteria, Theoretical Claims, and Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The article presents methods for approximating vanishing ideals in latent space of neural nets and uses this to identify classifiers with fewer layers. 

One reviewer recommends strong accept, indicating that the manuscript presents a novel idea and interesting results that may be useful to gain insight. At the same time they indicate that not enough is done to build on this analysis to say something about the structure of the latent manifold and that they do not have the capacity to verify the correctness of the theoretical claims. The discussion indicates that the reviewer agrees with comments from another (critical) reviewer, but that they believe this is an interesting contribution nonetheless. 
One reviewer recommends reject with concerns that persisted after the rebuttal. Concerns pertain to the lack of theoretical results establishing differences in complexity as well as certain claims. Some of the notation and presentation aspects were clarified in the rebuttal. Authors conceded that not every VI Net has lower spectral complexity. 
Another reviewer recommends weak accept indicating that the experiments are comprehensive and demonstrate effectiveness, but also that the article mainly applies existing algorithms. They describe the proposed method as straightforward and sensible for improving parameter efficiency. They maintained their score after the rebuttal. 
Another reviewer recommends weak accept, indicating that the authors investigate generalisation based on existing results regarding spectral complexity. 

I find that the article, albeit not being strong in all regards, contributes neat ideas that could generate further discussions leading to more insights. Given that referees overall lean towards accept, I recommend accept. I ask hat the authors take the reviews and rebuttal discussion carefully into consideration when preparing the final version of the manuscript.
Decision: Accept (poster)