Deep Sturm–Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions
Authors: David Vigouroux
Joseba Dalmau
Louis Béthune
Victor Boutin
Venue: ICML 2025 poster (2025)
Keywords: ODEs
Regularization
Sturm-Liouville theory
function approximation
orthogonal basis functions
eigenvalues problems
Elliptic eigenvalues problem
Parabolic eigenvalues problem
Primary area: general_machine_learning

Abstract: Although Artificial Neural Networks (ANNs) have achieved remarkable success across various tasks, they still suffer from limited generalization. We hypothesize that this limitation arises from the traditional sample-based (0--dimensionnal) regularization used in ANNs. To overcome this, we introduce Deep Sturm-Liouville (DSL), a novel function approximator that enables continuous 1D regularization along field lines in the input space by integrating the Sturm-Liouville Theorem (SLT) into the deep learning framework. DSL defines field lines traversing the input space, along which a Sturm-Liouville problem is solved to generate orthogonal basis functions, enforcing implicit regularization thanks to the desirable properties of SLT. These basis functions are linearly combined to construct the DSL approximator. Both the vector field and basis functions are parameterized by neural networks and learned jointly. We demonstrate that the DSL formulation naturally arises when solving a Rank-1 Parabolic Eigenvalue Problem. DSL is trained efficiently using stochastic gradient descent via implicit differentiation and achieves competitive performance on diverse multivariate datasets, including high-dimensional image datasets such as MNIST and CIFAR-10.

Deep Sturm-Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions
David Vigouroux 12
Joseba Dalmau 123
Louis Bethune 24
Victor Boutin 52
Abstract
(a) Neural ODEs
(b) DSL
Although Artificial Neural Networks (ANNs) have achieved remarkable success across various tasks, they still suffer from limited generalization. We hypothesize that this limitation arises from the traditional sample- larization used in ANNs. To overcome this, we introduce Deep Sturm- Liouvile (DSL), a novel function approximator that enables continuous ID regularization along field lines in the input space by integrating the Sturm- The- orem (SLT) into the deep learning framework. DSL. defines field lines traversing the input space, along which a Sturm-Liouville problem solved to generate orthogonal basis functions, enfore- ing implicit regularization thanks the desirable properties of SLT. These basis functions are lin- early combined to construct the DSL approxima- tor. Both the vector field and basis functions are parameterized by neural networks and learned jointly. We demonstrate that the DSL formulation naturally arises when solving a Rank-1 Parabolic Eigenvalue Problem. DSL is trained efficiently using stochastic gradient descent via implicit dif- ferentiation. DSL. achieves competitive perfor- mance and demonstrate improved sample effi- ciency on diverse multivariate datasets including high-diment nigh-dimensional image datasets such as MNIST and CIFAR-10.
R
(O-dimensionnal) regu-
Figure I. (a) In Neural ODEs, the vector field acts as a continuous- depth neural network. Regularization applied along field line represents a OD regularization (i.e. sample specific). (b) For DSL, the vector field's field lines span the entire input space. Regu- larizing along these lines applics to all points they pass through, making it regularization.
processing. These practical results are also supported by theoretical works the expressivity of neural networks. It
has long been known that any function can be approximated by neural networks (Hornik et al., 1989; Cybenko, 1989) and recent works demonstrate exponential approximation accuracy (Elbrächter et al., 2021).
Despite its remarkable achievements, deep learning still suffers from significant generalization limitations. For ex- amplc, neural networks are vulnerable to adversarial attacks, where small changes to the input can drastically alter pre- dictions Grbossvi-Dezfooor et al., 2015). Another critical issue is domain shift, which occurs when a model trained on specific data distribution fails to generalize to new but related data -such as images captured under different light- ing conditions, viewpoints, or backgrounds- despite their semantic similarity (Tzeng al., 2014; Ganin al., 2016; Ben-David et al., 2010). In addition, Neural Network often struggle to generalize to testing samples when trained with a limited number of samples (Zhang et al., 2021; Arpit et al., 2017; McClellan et al., 2024). These generalization issues have spurred entire fields of active research (Rodriguez et al., 2023: Linsley et al., 2023; Szegedy al., 2013), yet their root causes remain poorly understood. Do they stem from the optimization process, the learning paradigm, the reg- ularization techniques, the network architectures, or other underlying factors?
1. Introduction
Neural networks have become the go-to approach in vari- ous applications, demonstrating their versatility in a wide range of tasks from image recognition to natural language
'IRT Saint Exupery AANIII (Artificial and Natural Intelli- gence Toulouse Institute) 'IMT. INSA Toulouse "Now at Apple CCeCCo (CNRS, Université de Toulouse). Correspondence to: David Vigouroux <david. @ irt saintexupery.co>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we argue that the sample-based approach com- monly used in deep learning to compute or regularize the

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
loss function may be a fundamental limitation. Specifi- cally, conventional sampling- regularization--rfeered to here 0D regularization crete points within the domain space, inherently capturing only a partial view of that space. Our core contribution is to move beyond this traditional approach by introducing 1D regularization, a continuous regularization method applied along trajectories that span the entire domain space. Unlike OD regularization, which focuses isolated points, 1D regularization captures richer structural information by in- tegrating along continuous paths. Our work complements other regularization methods such as weight decay or per- sample regularization (e.g. entropy regularization (Grand- valet & Bengio, 2004) or KL regularization (Higgins et al., 2016)). However, unlike traditional approaches, DSL em- phasizes function smoothness by enforcing regularization along continuous trajectories in the input space.
we describe how introduce ID regularization within this framework. We finish with an experimental evaluation of the DSL method where we demonstrate experimentally the expressivity of the method.
the function at dis-
Our main contributions are:
Introducing a new function approximator, called Deep Sturv-Liouville (DSL), built from a task-dependent or- thogonal function basis in an open domain. We demon- strate experimentally the expressivity of DSL.
Formulating ID regularization within the DSL frame- work to control the solution smoothness through im- plicit and explicit spectral conditioning. Establishing link between Deep LiuvnlLi and a Rank-1 Parabolic Eigenvalue Problem.
2. Related work
To enable 1D regularization, we introduce novel class of function approximators, termed Deep Sturm-Liouville (DSL), which possess unique properties. DSL is built upon an ODE defining a vector field (Feldman & Yeager, 2018), parameterized by a neural network, that guides trajectories across the input space. Along each trajectory- sppcifiically, each field line of the vector field- the function is approx- imated using an orthogonal basis, obtained by solving the Sturm-Liouville (SL) Eigenvalue Problem. The first few ba- sis functions of the SL problem exhibit desirable regularity properties, which naturally impose implicit regularization along the entire field line. Moreover, the DSL framework is flexible enough to integrate explicit constraints along the field line. Unlike other ODE-based frameworks, such as neuralODE (Chen et al., 2018b), our approach leverages field lines that are specifically designed to traverse the entire input space (see Fig. 1). This design enables DSL to effec- tively apply 1D regularization, ensuring a more structured constraint across the input domain.
Neural Ordinary Differential Equations (Neural ODEs), in- troduced by (Chen et al., 2018b), parameterize the continu- ous dynamics of hidden units using an ODE specified by a neural network. Unlike traditional discrete-layer architec- tures, Neural ODEs provide a continuous representation and have been widely applied in fields such as normalizing flows (Tabak & Vanden-Eijnden Eijnden, 2010; Rezende & Mohamed, 2015). In contrast, DSL is not a continuous-depth neural network; instead, it introduces a novel function approxi- mator based on orthogonal bases. The primary difference between DSL and -ontin neural networks lies in their vector fields (see figure 1). In continuous-depth networks, the vector field maps the input space to the state space (S + R),. Therefore in Neural ODE the regular- ization is applied on a per-sample basis (OD). Conversely, DSL's vector field operates directly within the input space (Ω + f2), ensuring that field lines traverse the entire domain. This enables a 1D regularization approach that optimizes the loss along entire trajectories rather than just on indi- vidual samples, improving generalization through implicit regularization. DSL achieves this by controlling the number of sign changes in the basis function.
We demonstrate that this new function approximator is in- trinsically connected to more general mathematical frame- work, the rank-1 Parabolic Figenvalue Problem. This con- nection is particularly significant as it shows that the DSL formulation emerges naturally as a solution to the rank-1 Parabolic Eigenvalue Problem.
Continuous-depth neural network have also gained signif- icant attention in the context of generative modeling, par- ticularly in applications such as normalizing flows intro- duced by Tabak& & Vanden-Eijnder (2010) and popularised by Rezende & Mohamed (2015). These models leverage continuous dynamics to transform probability distributions, thanks to the Liouville Identity, enabling flexible and effi- cient mapping between complex data distributions and sim- pler base distributions. Normalizing flows provide power- ful framework for density estimation, generative sampling, and latent space exploration. Integrating DSL into such a framework requires overcoming significant challenges, as the change-of -fariabl- variable formula used in our method is funda-
First, we introduce the Sturn-Liouville theorem (SLT) in its original ID form. Then, expose the Elliptic Eigen- value Problem which extends SLT to the multidimensional case. This Ellipticc Eigenvalue Problem being hard to solve due to its significant calculation time, we also present a related problem called Parabolic Eigenvalue Problem. In section 4, we introduce the Deep SiurvlLi method, which exploits a link between the Rank-1 Parabolic Eigen- value Problem and the LturilLiouville problem to obtain tractable solution for the high-dimensional case. Then,
2

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
mentally different from those typically employed in Neural Odes. This difference in the change of variable prevents to compute density estimation with our work. DSL should be regarded solely as a function approximator.
aim of this work is create a flexible framework where the orthogonal basis functions are not fixed but they are learnt to adapt to a particular machine learning task.
In this work, to obtain the orthogonal functions u, (z),,, we will study a special case of eigenvalue problem: The Sturm- Liouville Problem (SLP). An eigenvalue problem involves finding scalars (A, eigenvalues) and corresponding vectors (v, eigenvectors) such that for a given linear operator or matrix A, the equation Ar Ae holds, where + 0. In the case of SLP, the eigenvectors are 1D function vectors that form a complete basis. These functions exhibit desirable regularity properties and are themselves parameterized by functions, which provide significant flexibility compared to fixed bases. An extension of this theorem exists for the gen- eral case in N-D: the Elliptic Eigenvalue Problem. However, this problem becomes intractable in high dimensions. To address this, we will study a more tractable solution: the 1-rank Parabolic Problem, which is degenerate case of the Elliptic Eigenvalue Problem.
While Neural ODEs operate with a fixed integration time, (Massaroli et al., 2020) introduces Adaptive- Depth Neural ODEs, where the integration time of the ODE is learned through neural network. In Deep Sturm-Liouville, the integration time also changes for every sample. The integra- tion is obtained by stopping criteria when the trajectory cross the boundary of the domain. The stopping condition of the trajectory is governed by a spatial event condition instead of being determined temporally by neural network.
The simplest approach to learning an orthogonal basis is through the Gram-Schmidt process. However, unlike Gram- Schmidt, which defines orthogonality locally in the neigh- borhood of a sample, DSL enforces orthogonality along each field line. By extension, DSL maintains orthogonality across the entire domain 32, as stated in Theorem 5.1.
Some works solve partial differential equations with neural networks, e.g. for magnetic field estimation (Khan et al., 2019), fluid simulations (Kim et al., 2019), eigenvalue func- tions problems (Kovacs et al., 2022) or PDEs similar to the Elliptic Eigenvalues Problems (Marwah et al., 2023). Inspired by complex natural phenomena which can be simu- lated by PDEs, our work takes the opposite point of view of these works by using Rank-l Parabolic Eigenvalue Problems to propose new function approximator.
3.1. Sturm-Liouville theorem
The Sturm-Liouville theorem (Sturm & Liouville, 1837) has a significant importance on the theory of eigenvalue problems for 1D ordinary differential equations (ODE). For instance, Sturm-Liouville theory (SLT) is employed in quan- tum mechanics analyze the solutions of the Schrödinger equations (Bender & Orszag, 1978), in heat conduction problems (Lützen, 1984) to compute vibrational modes (Wang, 1996). Sturm-Liouville eigenvalue problems offer systematic approach to discerning the characteristic fre- quencies and spatial patterns. This relationship between SLT and physics problems motivates us to explore the po- tential application of this theorem machine learning. wide range of ID complete orthonormal function bases can be reinterpreted within this theory; common bases such as Fourier, Bessel or Chebyshev polynomials can be seen as particular cases of this setting.
3. Notation and Eigenvalues Problems
We define the data open domain Ω C R with targets Y E T We assume that Pxy P(s x IRE)) is the joint distribution of data points and targets, and we consider a dataset D ~ Pxy- We define the predictor F as a pair (0, I.) composed of a vector of orthogonal basis functions spper-parameter,, parametrized by 0 with weight function w² Q + RR, and linear map
u Q + Rd (d
The Sturn-Liouville theorem is formulated as an eigen- value and eigenfunction problem satisfying the boundary conditions of an ODE:
L Rdd +R.
F(I,0,L) L(u (x),
s.t I wa (x) (2) 0 vi #i
Theorem 3.1 (Sturm -Sturi-Liouville Liouville Theorem). For any given functions, p. w [a, b] + Et and q : [a, b] + R of classes C¹ and CD respectively, and real numbers a1, 02, 31, B2. there exist unique sequence (N1)21 (of eigenvalues) and associated eigenfinctions ys [a, + R solving the ODE below, with the given boundary conditions:
Our goal is to minimize the empirical risk associated to a loss L: R2* x &
+ R, defined as
min- G.L
Cly, F(),,,
d
(t:(t) dt
72
dt p(t)
+ q(t))e(t) = (t)
by simultaneously learning the linear operator L and the orthogonal function basis u (z). in a data-dependent fash- ion. To avoid the curse of dimensionality incurred by fixed basis functions such as Fourier, polynomial or wavelet, the
dy:
(1vi(a) + 022 dt (a) = 0
011, 0r2 not both 0,
(3)
(1b((b) + B2 dy (b) 0
B1. B2 not both 0.
dt
3

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
The sequence of eigenfunctions (v) forms an orthonor- mal basis in the Hilbert space (.)( with the inner prod- uct weighted by w:
2003). Thirdly, without assumptions on the form of the matrix A, whose size is the square of the size of input space, the matrix A can be too large for high dimension data.
L w (tt)) 6ey.
To make this computational problem more tractable, we explore a related problem where the rank of the matrix A is equal to to simplify the problem structure. We refer to this new problem as the Rank-1 Parabolic Eigenvalue Problem, defined as:
(4)
where ó the kronecker delta.
The eigenvalues \1 A2, ... are real and ordered so that d1 < X2 ...An < ... Kondratiev (1996) Chapter 5-Theorem 19, the n basis function has exactly n - 1 zeros in the interval ]a, b(. By linearly combining the first n basis functions to construct a function approximator and tuning n, we can control the level of regularity of the function. Herein, we assume Dirichlet's boundary conditions: y/ (a) and (b) O.
Definition 3.3. The Figenvalue Problem (see Eq. 5) is called Rank-1 Parabolic Eigenvalue Problem when the matrix A is positive semi-definite and its rank is equal to 1.
According to Egorov &
In such case, the existence of eigenvalues not guaranteed. However, its rank-1 structure will allow us to solve the Parabolic Eigenvalue Problem along a field line by using the ID
theorem (see Theorem 5.3). This will allow us to combine the Liouvile theorem and deep neural networks, thus giving rise to Deep Sturm-Liouville, a means to compute orthogonal bases in high dimensions without the need solve a high dimensional PDE.
For dimensional data, we parameterize the func- tions P. q and w with neural networks. By solving the associated Sturm-Liouville Problem, we obtain the eigen- functions (t) which form an orthogonal basis. A linear combination of the (t) can be used to predict the value at any E (a, b]. The weights of p. 9, w can be learnt to optimize (2). The main idea behind our work is to extend this procedure to the multidimensional case.
4. Deep SiovilLi Method
Deep Sturm Liouville (DSL) is constructed using a vector field that traverses an open domain Ω C R" Along each field line, a Sturm-Liouville problem is solved to generate orthogonal basis functions. When linearly combined, these basis functions define new function. Machine learning problems formulated as the optimization problem in Eq. 2 can be solved using gradient descent.
3.2. Elliptic Eigenvalue Problem
The Strum-Liouville theorem has its extension in dimension greater than one, more precisely an open set Ω2 thanks to the following Elliptic Eigenvalue Problem (EEP) (Larsson, 2003; Muthukumar, 2014):
Theorem 3.2. For any continuous functions A : Ω + R" x RR", symmetric, positive-defiitt, w w Ω + R2, of classes C², C and C respectively, there exist a unique sequence of eigenvalues X; and associated eigenfunctions us satisfying:
4.1. Deep Sturm-Liouville
Ω + R and
First, we define the field line 7 (t) = 2tt) associated with the sample x, which satisfies the following equation param- eterized by the function a a. Ω + R":
de
V . -A(z)) (u,(x)) + (x)) = -),
di = a()), x00) = x.
(6)
with u(()= =0
Vo E An
(5)
Note that Eq. is similar to the one used in Neural Ordinary Differential Equations (Chen et al., 20) However, in this work, this equation serves different purpose: to project the sample distribution onto the boundaries of the domain Ω. While Neural ODEs have fixed final time. Deep Sturm- Liouville assigns a unique final time for each sample x. For any I E Ω we enforce that the field line (t) passing through x intersects the boundary 0 in two unique points to ensure complete coverage of the domain Ω. That is, There exist two unique times t and t 0 such that (t=) and (1E) an.
[ (z)) (x)dr = 8ig-
This theorem could be useful to learn a basis of functions suited to a particular machine learning task in high dimen- sion by optimizing Eq. 2 through the optimization of the functions A, q and w, typically surrogated by neural net- works.
Solving these equations directly is quite challenging. First, even if recent works study the solutions of partial differential equations in high dimension (Wu et al., 2023), efficiently solving this kind of partial differential equations (PDE) is very costly. Secondly, solving the eigenvalue problem very difficult even if numeric methods exists (Larsson,
To obtain the uniqueness and the existence of and Ef. we assume that Eq. 6 has an unique solution which requires (xx) to be Lipschitz continuous -, that there are no limit cy- cles and that (xx) is nowhere tangent to M.. The existence

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
(b)
(a)
Ω
(44)
80
(f
(0) = σ
*(==) 70
+(())
(tL)
uz
(t) = a(z)
(t) - pp(r (1)) d0,(0)
+ g(> (t)) = (t)
Figure 2.
Deep Sturm-Liovville. - (a) For a given point the field line (t) is defined by equation (6), it is such that >0 = and reaches the two points at the boundary of time and t (b) On the field line 7 (t), the Sturm Liouville Problem 7 solved with the parameter functions p(>"(()) 9(7 (4)) and (()" to obtain orthogonal function basis w. that, combined linearly, form the DSL function approximator along the field line. The prediction at is obtained by taking the value this function at t
of limit cycles is a complex problem for which no general solution is known for n > 2; for n=2 the Bendixson-Duus Dullc theorem describes sufficient conditions to have no limit ey- cles (Burton & Burton, 1983). However, special cases exists where the absence of limit cycles has been demonstrated. For example (Johnston, 2015):
are defined up to a multiplying coefficient. The equation on the u. derivative at t- ensures the uniqueness of the solution. v(x) is learnable; however, in most experiments, a fixed value of uxx) proves be sufficient.
We define the function uy $2 - R by setting u,(x) uf (0). The (x(x) are well defined and form an orthogonal basis along the field line 7" Indeed, for all x Ω2 if 22 - (s), we can show that u." (t) u consequence:
a is a strictly positive continuous function and Ω is convex,
u (t s). As
a a gradient of a function with no singular point and the gradient is not vanishing anywhere in the domain' !
u)(""(s)) = ux(22) = 2² nt (s), vs
The field line (t), defined by Eq. 6, is restricted the tem- poral interval [t_, [5]. Along this segment of the field line, eigenvalue problem is solved to construct a 1D orthogonal basis specific to this part of the domain Ω. The Sturn-Liouvilly Problem is parametrized by the learn- able functions p= + Rt. q Ω + R. w Ω - R and o 02 + Fd. These functions are optimized during the training phase to generate adaptive basis functions tailored to the problem. Formally, the ID orthogonal basis functions uf (4) and their corresponding eigenvalues X are obtained by solving the following eigenvalue problem:
This implies that the integral Eq. can be rewritten as a function of the field-line:
a
| w (()))
0.
Remark 4.2. As demonstrated in Theorem 5.1, the collection of ID bases combines to form a global orthogonal basis over the entire domain J
In the Sturm- Theorem, the functions p. q and w depend only on the variable t. In Deep Strum-Liouville, the key idea is that p. q and w depend on the field line y" (t). The purpose of this dependence is to couple Eq. 6 and Eq. 7 through the variable t. For two samples x² and x², which belong to two different field lines 7" and >nd two different local orthogonal ID basis functions are estimated. (7)
p( (1)) du,(t) di
d
((7"" (Ww()"()))ull
di
du,(1=) (E)).
u u,(t = 0,
(x(t)) o,
Remark 4.1.
The ode and the Dirichlet's conditions of Eq. 7 This equation has some similarities with energy-based models (Hinton, 2002).
Consequently, the function approximator on the whole do- main Ω is composed of ID basis functions which are locally orthogonal.
5

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
Once the eigenvalues are obtained, the prediction at given is computed by solving the ordinary differential equations to derive uz (0) from E (see Eq. 7). The function approxi- mator is then defined through a linear map L : R + RR, where d represents the number of eigenvalues (a tunable parameter of our method), and k denotes the dimensionality of the predictor's output F.
4.3. Algorithm details
Eigenvalues To compute the eigenvalues of the Strum- Liouville problem (see Eq. 3), shooting method (Stoer et al., 2002) is performed. The aim of the shooting method is to optimize the eigenvalue À such that the boundary condi- tion y(b) = 0 is satisfied. In our work, we perform a binary search between the lower and upper bounds of the eigen- values (Breuer & Gottlieb, 1971) on an equivalent problem obtained by the Prüfer Substitution (Prüfer, 1926; Lebovitz, 2019), see appendix A for more details.
F..T. (x) = L (x) with 0 = [a, p. g, w. v).
Algorithm 1 Deep Sturn-Liouville Prediction
The computation of the gradients of Deep Sturm-Liouville w.r.f the weights of the function a, 4 and w is not straightforward due to the estimation of eigen- values the X and the times t and E associated to each prediction. In fact, the computation through the shooting process and the stop conditions are not differentiable. To overcome this, we use the implicit differentiation theorem (Krantz & Parks, 2012) thanks to mapping function captur- ing the optimal conditions of the problem (see more details in appendix B).
Gradients
1: Compute t and t with Eq. 6
2: Find eigenvalues X along the field line (t) in (7)
using (A)
3: Resolve Eq. 7 from t to compute (I(
4: Compute the prediction at x: F (x) L (1)
The optimization problem in Eq. 2 can be rewritten by min- imizing the parametric functions of the Sturm-Liouviile problem:
min L(Y, po. (X)). L
5. Results
In our experiments, the functions a(x), p(r), q(x),, wxx), ez)) are typically parameterized by neural networks.
5.1. Theoritical Results
Deep Stuvm-Liouville is an orthogonal basis on Ωi Let us state the main theorem of Deep Sturn-Liouville:
4.2. Regularizations
Theorem 5.1. The functions uz(x) form an orthogonal basis of functions on the open domain Q with a weight function w" (z) +2++1 R
The central idea of Deep LiuvilLe to introduce a 1D regularization along each field line traversing the input space Ω2 aiming to achieve better generalization compared to the sample-bacc based regularization (i.e. OD) typically used in machine learning. This D regularization is both implicit and explicit:
Ja 10 (x))(()) =0
Implicit regularization: This regularization is a natural consequence of the Sturm-Liouville Theorem. It is obtained by selecting the first few elements of the basis. The first elements of the DSL basis oscillate less than higher-ranked elements of the basis (similar to the Fourier basis functions). In the Sturm-Liouvill framework, the oscillation is defined by the number of times where the basis functions change sign: the th base function changes sign exactly n - times. By selecting the d first elements of the basis function, DSL guarantees an implicit regularization along each field line.
The intuition behind the proof of this theorem is simple. Along the field line y the basis functions (x(x)) are orthog- onal. By applying a Fubini-like (Nicolaescu, 2011 1) result to the integral over the whole domain Ω, we rewrite the integral over Ω as a double integral: over the points in the boundary an of the form ((E)), and along the field line 7f, thus obtaining the orthogonality over the whole domain Ω (see proof in appendix C).
Deep Sturm-Liouville is a relaxation of the Rank-1 Parabolic Eigenvalue Problems: Interestingly, Deep Sturm-Liouvilly originated from studying solutions of the Elliptic Eigenvalue Problem. What stands out is that DSL discovered as natural outcome of addressing this broader mathematical problem, rather than being a clever combination of existing techniques. Deep Sturm-Liouville can be seen as a relaxed version of the Rank- Parabolic Eigenvalues Problem when no global eigenvalues exists. We define a sub-class of Deep Liouvile problems.
Explicit regularization Spectral regularization: To avoid strong variations on the derivatives of the basis along the field line y², the absolute value of the eigenvalues of the Sturm-Liouville Theorem, computed for each field line yd (t), are added in the loss as a regularization term:
min E (LLY, (X)) L.0
(X)) + a " M(XI)
d
(9)
0=
6

---

Deep Sturm-LLouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions
Definition S.2. Deep Sturm-Liouville Problem (see Eq. 7) is uniform if all eigenvalues are independent of x.
5.2.1. EXPERIMENTAL RESULTS
Local bases are tailored each sample: As preliminary evaluation, to verify that DSL learns different local basis for each example the local basis along the field line y (t) is analyzed for several different samples of the Dry Bean Dataset. As observed in figure 3, the local basis is different for the each of the examples represented, illustrating the local expressiveness of Deep Siorvl-iouvilll. As expected by the Sturm Liouvill theory, the boundaries satisfy the Dirichlet conditions and the jth base function crosses the z-axis exactly 1 times.
Theorem 5.3. The Uniform Deep Sturm Dionviiouvvllle Problem can be rewritten as a Dirichlet Rank-I Parabolic Eigenvalue Problem when assuming alx) is the gradient of a function with a(z); > O and by defining Us : :RR x RR= - R.
V (a((z))a (xc) Vu,(=)) (x) - (x)) a ((())
) + (()) y) = (x.0
de a(z).
The idea is to define a direction (1)) in which the PDE could lead to a more tractable ODEs. This could be done by rewriting the positive semi-definite rank-1 1 matrix A(z) as A(x) = a(ar)a'(e (x) and by applying change of vari- ables where all gradients of this new system of variables are orthogonal to a(x) (see proof in appendix D).
Sample
Sample 2
30
600
20
400
10
200
5.2. Experiments
-10
0.5
0.5
To evaluate our work. Deep Sturm- Liouville has been trained on three multivariate datasets: Adult (Becker & Ko- havi, 1996), Dry Bean (UCI Machine Learning Repository) and Bank Marketing (Frank, 2010), as well as the MNIST image dataset (LeCun & Cortes, 2010) and the Cifar10 dataset (Krizhevsky et al., 2009). Due the fundamental differences in the change of variables between continuous neural networks and DSL, no experiments were conducted for density estimation.
Figure 3. Eigenfunctions Dry Bean dataset. For two samples, the first three eigenfunctions. The x-axis represents the time t of the field line (t)).
Impact of explicit and implicit regularization: First, we perform a sensitivity analysis on the Dry Bean dataset (Figure 4) to evaluate the impact of implicit and explicit reg- ularization on the classifier's performance. The results show that accuracy reaches its peak when the number of eigenval- ues servin as a parameter for implicit regularization-is around 10, achieving performance comparable to a MLP. As shown in the graph, DSL. demonstrates robustness to this parameter, with accuracy remaining stable across a wide range of values. Significant accuracy drops are observed only when the spectral parameter is set too low or when the number of eigenvalues becomes excessively low.
Deep Liouvile is implemented on jax (Bradbury et al., 2018) and uses di ffrax (Kidger, 2021) to solve the ODEs involved. The solver dopri8 (Prince & Dor- mand, 1981) is used with a relative tolerance of 1e-6 and an absolute tolerance of 1e-6. Refer to Appendix E for analysis of the impact of solver precision on the error in the mapping function used to estimate the gradient via the implicit function theorem. To obtain a good approximation of the times f_ and f_ to reach the boundary of the domain Ω we perform a binary search, between the time triggered by the stopping condition and the previous time before the event was triggered. There is no need for this binary search procedure to be differentiable thanks to the implicit differen- tiation theorem. To solve the eigenvalue problem, the values of q. p and w are computed along the field line fy to obtain a spline which is dependent only on (avoiding numerous calls the neural networks during the shooting phase). In these experiments, piecewise linear function of 2000 parts is used to approximate these functions along the field line. The binary search is done with a tolerance of 1e-4 for the tabular experiments and 1e-8 for the image dataset. For all experiments, the number of eigenfunctions was fixed to 10.
DSL achieves comparable performance than standard Neural Networks: To demonstrate that Deep Sturm- Liouville can reach comparable performance to a Neural Network, DSL and NN were trained on several classification tasks. For this experiment, Neural Networks have similar architectures. Table demonstrates that DSL achieves com- parable results NN with only 10 eigenfunctions.
DSL achieves better sample efficiency: To evaluate the impact of 1D regularization on generalization, we con- ducted a sample efficiency analysis to determine how ef- fectively DSL can extract generalizable predictions from limited training data (see Fig.5). In this experiment, we trained both DSL, and a standard Neural Networks (NN)
More details on architectures of the neural networks and the optimizer's 's parameters are provided in the appendix E.
7

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
(a)
(b)
90
Accunay (8)
0.8
70
Accrecc
60
0.7
40
0.6
15
10-3
10-4
10
10
cigenfunctions
Regularization Spectral Coefficient
Figure 4. Impact of the implicit and explicit regularization on Dry Bean dataset. Validation accuracy as function of eigenfunctions basis size for the implicit regularization (a) and the spectral regularization coefficient a for explicit regularization equation 9 (b).
DATA SET
DSL (OURS)
NODE
6. Limitations
ADULT
84.28%
84.06%
Despite the promising results, Deep Sturm-Liouvilly suf- fers from several flaws. Scalability, Even if Deep Sturm- Liouville is scalable high dimension, the gradient com- putation can be expensive due the form the problem to solve is not particularly due the implicit differenti- ation theorem because observe that the computation of the gradient on the weights of neural networks are the same order of magnitude than the computation of the jacobian on the eigenvalues and times to boundaries. Prediction compu- tation can also be expensive. Even if the binary search itself is quick, the approximation p. q, w along the field line 7 can be costly despite their smoothness?. Stability, The estimation of the gradient of the ODE could be noisy if the solver is not precise enough. During training, in rare config- urations, the ODE solver can fail to detect the boundary and the training fails.
DRY BEAN
91.14%
91.45%
BANK MARKETING
83.10%
83.77%
MNIST
97.93%
96.8%
CIFAR10
58.38%
58.9%
Table Evaluation. Classification accuracies for DSL (Deep Sturm Liouvilllll and NODE (Neural ODEs). The results on Mnist and Cifar10 are sourced from Massaroli et al. (2020).
on small training sets, with sample sizes ranging from 100 to 800, and compared their test accuracy. Furthermore, as an ablation study, we investigate a modified DSL model in which the vector field a(x) is not utilized. On the Bank dataset, as illustrated in Fig.5, DSL consistently outper- formed traditional neural networks, showcasing its superior ability to generalize from fewer samples. Additional details about the experimental setup are provided Appendix E.
7. Conclusion
80
A mathematical formulation has been developed intro- duce the Sturm-Liouville Theory in the deep learning frame- work. We demonstrate the link between the Deep Strum- Liouville formula and the Rank-1 I Parabolic Eigenvalues problem. A trainable procedure based on implicit differen- tiation was implemented, successfully achieving compara- ble results those of neural networks on tabular datasets, MNI ST and Cifar10. 10. We hope that our work paves the way for novel avenues in function regularization. Future work shall be done to develop the change of variable formula of DSL to obtain generative classifier.
Accuracy (%)
75
70
65
DSL (Our)
60
DSL (Ablation) MLP
200
400
600
800
Number of training samples
Figure 5. Impact of training sample on Accuracy. Test Accuracy on Bank dataset as function of number of training samples.
Smoother functions allow for bigger step-size in numerical
integration.
8

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
Code Availability
Breuer, S. and Goutlieb D. Upper and lower bounds eigenvalues of sturm-liouville systems. Jour- nal of Mathematical Analysis and Applications, 36(3):465 1971. ISSN 0022-247X. doi:
The source code used to implement the models and conduct the experiments presented in this paper is publicly avail- aieel-a-pppers// papers/
able at https: https://github. com/ dee/deelllas peel-aat ai
Seep-sturm-llovvllee
URL
com/
Acknowledgments
Burton, and Burton, Volterra Integral and Differ- ential Equations. Mathematics in science and engi- necring. Academic Press, 1983, ISBN 9780121473808. URL
This work was carried out within the DEEL project, which part of IRT Saint Exupéry and the ANITI cluster. The authors acknowledge the financial support from DEEL's Industrial and Academic Members and the France 2030 pro- gram Grant agreements AFAN--I0-AIRTE and n AANR- 23-IACL-0002.
http//oooooo.olllleoe./oooooooce google. frr fr/booksidd DRvALAEACAAT.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duve-
naud, D. K. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018a. URL https://pprroceclnsss.. neeps. neurips cc/paper /c//aper_fileesppprr//////////
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be
1336ffbblieeee886222223333333333 pdf.
specifically highlighted here.
Chen, R. T. Q., Rubanova, Y., Bettenccourt, J., and Duvenaud, D. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Informa- tion Processing Systems, NIPS' 18. pp. 6572-6583, Red Hook, NY, USA, 2018b. Curran Associates Inc.
References
Arpit, D., Jastrzkebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. closer look at memorization in deep networks. In International conference on machine learning, pp. 233-242. PMLR, 2017.
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of Control, Signals and Systems, 2(4): 314444, Dec 1989. ISSN 1435-568X. doi: 10.10007/BF05551274 URL httht://///oo /ttg://///....///////////// 1007/BF02551277
Becker, B. and Kohavi, R. Adult. UCI Ma- chine Learning Repository, 1996.
DOI:
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. theory of learning from different domains. Machine learning, 79:151-175, 2010.
Egorov, Y. Kondratiev, Spectral Properties of Elliptic Operators, pp. 133-151. Birkhäuser Basel, Basel, 1996. ISBN 178-3-0348-9022-3 doi: 10.1007/ 978-3-0348-9029-888 URL https: / /tw.idiiioo////oooooooooo.o..oo org/10.
Bender, C. M. and Orszag, A. Advanced Mathematical Methods for Scientists and Engineers. McGraw-Hii 1978.
Elbrächter, D., Perekrestenko, D., Grohs, P., and Böleskei, H. Deep neural network approximation theory. IEEE Transactions on Information Theory, invited feature pa- per, 67(5), May 2021. URL http: /wwww nari. ee. ethz ch/ppbs/seteccccc httpppbbs//eeeeeee dttp/deep-tt22222
Bernstein, J., Vahdat, A., Yue, Y., and Liu, M.-Y. On the distance between two neural networks and the stability of learning. In Neural Information Processing Systems, 2020.
Feldman, R. and Yeager. Vector Calculus. URL Booksheelvess/baaluulu////---- Calculus_ Yeager).
2018.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milnee, S., and Zhang, Q. JAX: composable transformations Python+NumPy programs, 2018. URL http://github com/ wwt./ooo/o..aeoccooc gowgle/oeeeeeeeooooooo
Frank, A. Uci machine learning repository. http://archive. ics. uci. edu/ml, 2010.
Phttps:/ htttps://wwrr deel. ai/

---

Deep Sturm-Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. Domain- adversarial training of neural networks. Journal of ma- chine learning research, 17(59):1-35, 2016.
Kovacs, A., Exl, L., Kornell, A., Fischbacher, J., Hovorka, M., Gusenbauer, M., Breth, L., Oezelt, H., Yano, M., Sakuma, N., Kinoshita, A., Shoji, T., Kato, A., and Schrefl, T. Conditional physics informed neural networks. Communications in Nonlinear Science and Numerical Simulation, 104: 106041, 2022. ISSN 1007- 5704. doi: http:://bbioog///.ooooo...mmsssa URL https://www.scieneediieetttt occtenne/artiicllepp/////1117111
Grandvalet, Y. and Bengio, Y. Semi-supervised learn- ing by entropy minimization. In Saul, L.,, Weiss, Y., and Bottou, L. (eds.), Advances in Neural Infor- mation Processing Systems, volume 17. MIT Press, 2004. URL
cc/ppppee__llle/ppppr/////////// Paper. 96220055a66133a55277444222228 pdf.
Krantz, S. G. and Parks, H. R. The implicit function theorem: history. theory, and applica- tions. Springer Science and Business Media, 2012. URL https: seonnksppiingrree com/book/1e 1007/978-1-60 111---1--122--5---
Higgins, Matthey, L., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. In International Conference Learning Representations, 2016. URL pottps://ast semant www.chllr.........o.oowc..ca...e org Ctrp/Corrussi :
header.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Larsson, T. The Elliptic Eigenvalue Problem, pp. 77- 94. Springer Berlin Heidelherg, Berlin, Heidelberg, 2003. ISBN 978-3-540-88706-5,
Hinton, G. E. Training Products of Experts by Mini- mizing Contrastive Divergence. Neural Computation, 14(8):1771-1800, 08 2002. ISSN 0899-7667. doi: 10. 10.11620899770022612229111 URL https: /do..
doi: 10.1007/
188-3-500-88706-35 6. URL https: /tt::://ooiiio/ org/10. 1007/978-3-54 0-0-55440-8---5555
Lebovitz, N. Oscillation theory and the spectra of eigenvalues. Ordinary Differential Equations, http://people. uchicago. edw/lebovit/oodes html, 2019. URL https: /pepppee ..w.e.c................... Lcl/:Lebovitt///deshooo// www.lee.s..cchicc cs.uccccag..
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural Networks, 265):359-366, 1989. ISSN 0893-6080.
doi: http:///oooog////.ooooo009999999 com/
LeCun, Y. and Cortes, MNIST handwritee digit database.
URL https: httt:://wwwwww..seeeeeeeeecccee scleec///atiieee/////33/00000000
http://yammleeum.cc//eeedmmmmmm 2010. URL http:
//yann.lecuu com wxdd/mnisbms///
W. Milnor, S. Characteristic classes. Ann. Math. Studies 76, 1974.
Lee, J. M. Introduction to smooth manifolds. 2000.
Linsley, D., Feng, P., Boissin, T., Ashok, A. K., Fel, T., Olaiya, S., and Serre, T. Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception, 2023.
Johnston, M. D. Limit cycles, der pol oscil- lator, poincar's e-bendixson theorem, 2015. URL https: http:::://oottsttommnnne.ee files. wordpress.
Lützen, J. Sturm and liouville's work on ordinary linear differential equations. the emergence of sturm- liovvlie theory. Archive for History of Exact Sciences, 29(4): 309-376, Dec 1984. ISSN 1432-0657. doi: 1001/ BF00348405. URL https: BF00348405.
Khan, A., Ghorbanian, V. and Lowther, D. Deep learning for magnetic field estimation. IEEE Transactions Magnetics, 55(6):1-4, 2019. doi: 10. 2011 2899304.
Kidger, P. On Neural Differential Equations. PhD thesis, University of Oxford, 2021.
Marwah, T., Lipton. Z. C., Lu, and Risteski, A. Neu- ral network approximations of pdes beyond linearity: A representational perspective. In Krause, A., Brun- skill, E.,, Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 24139-241 PMLR,
Kim, B., C. Azevedo, V., Thuerey, N., Kim, T., Gross, M., and Solenthaler, B. Deep Fluids: A Generative Network for Parameterized Fluid Simulations. Computer Graphics Forum (Proc. Eurographics), 38(2):59-70, 2019.
Kingma, P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14Xiv:1412.660 1412.6980 2014.
10

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
2023. wttptttttttt ngs wwi.gsmlr.pppo......... press// v202/ma http://awwa22a.aaaa.
Sturm, C. and Liouville, J. Extrait d'un mémoire le développement des fonctions séries dont les différents termes sont assujettis à satisfaire à une même équation différentielle linéaire, contenant un paramètre variable. J. Math. Pures Appl. 2, pp. 220-223, 1837. URL http:// www. numdam. org/ item/ JMPA_1837_1_ 1MPA_1037____222_00.. 22_00_2200.p..___
Massaroli, S., Poli, M., Park, J., Yamashita, A., and Asama, H. Dissecting neural odes. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, Pp. 3952-3963. Curran Associates, Inc.,
2020. URL https: http:///ppoceeennn...... neurips. oc oc/pppee_filee/ppper///00//L 09335222275555633496677222382222 pdf.
Szegedy, C.. Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6999, 2013.
Tabak, E. G. and Vanden Vanden-Eijnden, E. Density esti- mation by dual ascent of the log-likelihodd. Com- munications in Mathematical Sciences, 8:217-233, 2010. URL /tttpa://ap..stt stt.s//ccc.ssmtttisccccc...aaa org/ /tt/CoopusD::17333110
McClellan, J., Haghani, N., Winder, J., Huang, F., and Tokekar, P. Boosting sample efficiency and generalization in multi-agent reinforcement learning via equivariance. arXiv preprint urXiv:2410.02581, 2024.
S.-M., Fawzi, A., and Frossard, Deep- fool: A simple and accurate method fool deep neu- ral networks. 2016 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 2574-2582, 2015. URL https: wttppp.ssmmaanncsooll.oeo atg/CorpusID:12367..
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., and Darrell, T. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
UCI Machine Learning Repository. Dry Bean Dataset. UCI Machine Learning Repository, 2020. DOI:
Muthukumar, T. Second order elliptic pde. Course notes, 2014. URL httt:://honee iiteeeiiiti iitk ac. in/~tmk/ courses/ / htmmmmniiio wwtniicouuseee www.iooose//FAaE..... Basi cPDE. pdf.
Wang. Q. Sturm-liouville equation for free vibra- tion of a tube-in-tube tall building. Jourmal of Sound and Vibration, 191(3):349-355, 1996. ISSN 0022-460X. doi: https///bi.orr/11.0000////// 0126. URL h http://wwwwwweecereeeeee.ccc cctenneeaarricclee/a///222220202
Nicolaescu, L. The coarea formula. In seminar notes. Citeseer, 2011.
Prince, P. J. and Dormand, J. R. High order embedded Runge-Kutta formulae. J. Comp. Appl. Math, 7(1):67- 75, 1981.
Wu, H., Hu, T., Luo, H., Wang. J., and Long, M. Solving gigh-dimensional pdes with latent spectral models. In International Conference on Machine Learning, 2023.
Prüfer, H. Neue herleitung der sturm-tionvilleschenn rei- henentwicklung stetiger funktionen. Mathematische An- nalen, 95:499-518, 1926. URL ht http::/// //eddm/ org/ doc/159142.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Rezende, and Mohamed, S. Variational inference with normalizing flows. In Proceedings of the 32nd In- ternational Conference International Conference Machine Leurning- Volume 37, ICML'11, pp. 1530-1538. JMLR.org 2015.
Rodriguez, I. F., Linsley, D., Gopal, Fel, T., Acaro, M. J., Sharma, S., Livingstone, M., and Serre, Harmonizing the visual strategies of compuacoee models with humans yields more performant and interpretable models of primate visual system function. Journal of Vision, 23 (9):5768-5768, 2023.
Stoer, J., Bartels, R., Gautschi, W., Bulirsch, R., and Witz- gall, C. Introduction to Numerical Analysis. Texts in Applied Mathematics. Springer New York, 2002. ISBN 9780387954523. URL https:/ htttpp::/bboks....// /books....... google. fr ET/ o00.

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
A. Eigenvalues Computation
First, we compute the lower and upper bounds the eigenvalues of Sturm-Liouville Problem [X A thanks to (Breuer Gottlieb, 1971):
n²²m²2
max q(() w(t)
X
min, wt)p((t) (Jo p(r) dt)
(10)
n²x²
2 + min g(t)
X
max; w())p(() (IS" dt)
w((t)
Then perform a binary search method to avoid tuning some hyper-parameter such as the learning rate if we had chosen the gradient descent.
Unfortunately, is not possible to perform the binary search directly in the interval [AT A. Indeed, the different intervals for each X; may overlap meaning that there might multiple eigenvalues Xy in the interval [A, A ]. So the binary search guaranteed find the correct eigenvalue Aç.
To resolve this issue and to guarantee the monotonicity of the eigenvalue problem in the interval , the Prüfer Substitution (Prüfer, 1926; Lebovitz, 2019) is used to ensure that there is unique solution for each eigenvalue. The equations (3) are substituted by the following equations thanks to the change of variables:
(t( r(t) sin()(t)),
du;(t) r(t)
cos (o())..
dt
p(t)
If Xn is the Th eigenvalue given by the Sturm Liouville theorem, the equations can be re-expressed as:
d0(1))
= (An + (()))(())))))))))
sin sin)(((()) + cos)()(()) p(t))
dt
dr(t) 1
p(t) - ().w(t) + a(t)) r(t) sin(2
(1I)
dt
((a) = 0, O(b) nt.
The boundary conditions are dependent on the parameter n. relating to the nh eigenvalue, which is not the case with the initial formulation (3). This is what allows us to overcome the overlapping intervals problem.
Let g(A) be the function that maps each À to the value ((b) - no obtained by solving the equation (1I) with the initial boundary condition ((a) 0, for a given A. The function g is strictly increasing function, so that binary search can be applied between [>, A find the À such that g()) 0.
The computation of X is done in similar way by using the shooting method along the field line 7" with binary search and by applying the Prüner substitution on equation (7).
B. Gradient computation over t_, 1hE and X;
To compute the gradient over the times to the boundaries and the eigenvalues, the implicit differentiation theorem used. We define the mapping &..tt. + 1d422 capturing the optimal conditions of the problem:
uu (T)()), m5inz (12),
if $ d,
O.M OA (x)=
i
H
max
= 2.
12

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
Remark B.I. The last conditions materialize the intersection of the field line 70 with 0s for the specific domain ? =00 1f that
in our experiment. It should be defined differently for other convex domains such the sphere.
Remark B.2. Va E Q H (x) = 0.
Following the implicit differentiation theorem (Krantz & Parks, 2012):
Vu, (2) = (o)
Vx i1 uz (1))), H (A) (x).
C. Proof Theorem 5.1
My (2) form an orthogonal basis function on a open Q:
Jo (())) = 0.
Proof. By Sturm Liouvill Theory, we have for all e Q and for all ;
wo (t)) = 0.
We will reformulate the integral over the field line y" and by applying the line integrals change of variable and by (8);
w(2) f. ||(2)]]| (2) = 0.
We define the manifold an_ C IR" which is (n reettttiia,
0._ (1) Vo S).
By integrating over Ω we get:
w=zz) ui())u (z) (v) 0.
[[({)]]
We define:
P:0-00. P(x) (t).
Since P is Lipschitz we apply the co-area formula (Nicolaescu, 2011):
J
w(x) [|(x)]]
We let:
u(x) w(x) ||(x)]
det(Jr(x)))-
Then:
[ (z) = 0.
u (x) form orthogonal basis functions under the weight function v(x) the domain Ω.
13

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
D. Proof Theorem 5.3
Uniform Deep Sturm-Liouville can be rewritten to a Dirichlet Rank-1 Parabolic Eigenvalues Problem when assuming that a(x) the gradient of function with a(x);
Proof. From the equation (5), we will take the special case where:
A(x) (x))
Then we will develop the first component of the equation:
(a( Vu((x)) + (())))(x)) = -xx)(x)u:x()).
(12)
We will introduce the following change of variable (t. y) such that:
v (t. y) (x(x))
vt a(x)
V v = One (I) Vk E n 1]
First, we show that such a change of variable exists:
Let E' one-dimensional real vector bundle over a manifold M. is known (J. W. Milnor, 1974) that E is trivial if and only if it admits a global nowhere-vanishing vanishing section equivalently if its first Stiefel- Stiffel-Whitny Whitney class wi (E) vanishes. In section provides a canonical trivialization E = M R. Furthermore, if we choose local coordinates adapted this trivialization and if there exists smooth function V R - IR such that (V a(v), the Jacobian of the corresponding local map Ta can be arranged so that its first row is precisely a(x). This reflects the fact that the coordinate system is chosen to align the fiber direction (generated by a(x)) with the first coordinate axis. With an additional Riemannian metric on M, the orthogonal complement Bundles lemma (Lee, 2000) then provides an orthogonal decomposition T.MM- = span [a(x)} @ span(a(z)) completing the geometric picture. Consequently. the first row of the Jacobian is exactly axx), aligning the coordinate system with the direction defined by a(x). Combined with the or- thonormality of the remaining coordinates, this ensures that the first row of the Jacobian matrix orthogonal to all other rows.
this case,
Then, we can expand V. (x)
aaxx) 00,(1,y)
Ovr(l, 3)) Oyk
=V. a(x))a(())
Σ
Ong (2)
=V, (Ia())PPa(o)
0vi(l,y)
8 Per(t, 3)
n-1
(t.(t,)) v) Oy
=[(a())((a(a))
((x)) 012
anz (x)
+ |a(x))1 Ou, V. a(x)
Ot
+ +2(Jaa(z) a(x)) a(x) (v,(1,y) 82 (1, =I[a(x)
Øt
01
@u;(t.y) v)
+ (2 a(x)) axx) + 1[aax) a(x))
Øt
14

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
We let:
b(z) (Jza(z)) a(z)) + |a(x))] a(z)
Then because a(r); > O, the equation (12) can be rewritten:
|a(z)]] |
(t.v)
b(xx St
Ouy(t.a) + (())o; y) (xt(
812
(13)
dz
a'~1(2)
(Hadamard inverse of a(x:)
0
(p(z)
du @mi(t.)) 8x a(x)
) + (())o((t. y) (tt.))
81
(14)
de
de
15

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
E. Experiments details
Optimizer The optimizers of opt ax library are adam (Kingma & Ba, 2014) with learning rate 2e-3 for the tabular datasets and fromage (Bernstein et al., 2020) (I==le-2) for the MNIST and Cifarl0 datasets. The losses are the hinge loss for the tabular datatsets and the categorical cross-entropy for the MNIST and Cifarl0 datasets. The number of epochs 40 for tabular dataset, 10 for Mnist and 80 for Cifar10.
For the tabular datasets, the functions q(z), pla) and wxx) are defined by MLP with the features [128, 64, 32, 1] and leaky relu activations. The function v(z) The function a(x) is MLP [128, 64, 32, k] with tanh activations, where & is the dimension of the input size of the data. All weights are initialized with glorot-uniform initializer.
For the MNIST dataset, the functions (x). p(z) and w(x) are defined by a convolutional neural network with [32,641128]
features and kernel (3,3), the features of the MI.P are [32. 32, 16, 1] with tanh activations. The function v((r) 1. The function a(x) is an auto-encoder with [32,64] convolutions features and kernel (3,3) with 32 features and tanh activations. The weights are initialized with orthogonal initializer.
For the CIFARIO dataset, the data are first projected by an encoder e(x: in a latent space of dimension 128. The architecture of the encoder has [32, 64, 128] convolutions with kernel (3.3); the top of the encoder is a MLP [512, 256, 128]. The activation function is leaky-relu. To keep the projected domain within [-0.5, 0.5] 128 while avoiding vanishing gradients, the last activation of the encoder is defined as h(())) = lanh(x (/.0///////////0// The encoder is learn on the side of other neural networks. The functions (ar)), p(n) and wzx) are defined by MLP with [128, 64, 66, 1] features and leaky-relu activations. The function v(z) is a MLP with [128, 64, 66, 10] features and leaky-relu activation. The function az)) is a MLP with [256, 256, 256, 128] features and tanh activations. All weights are initialized with glorot-uniform initializer expect the MLP part of the encoder which are initialize with orthogonal initializer.
Eigenvalues bounds For the MNIST and Cifar10 datasets, ensure that the eigenvalues are not too large at initialization, the output domain of each of the functions a, p, q and was bounded. The choice of the appropriate bounds for each function is guided by the lower and upper bounds in the equations (10). To limit the eigenvalues to belong to the interval ~ |-100, 100 n² 2", the following constraints were implemented:
a(x)
4(x)
w(x)
p(a)
DOMAIN (0.01,
(-10,
10)
(1, 10)
(0.1, 10)
Remark E. The eigenvalues bounds were taken experimentally to let enough range to the variation of the eigenvalues while maintaining a reasonable computation times.
These constraints are enforced by using sigmoid activations the end of the model for a, p and and hyperbolic tangent activation for g.
Eigenvalues regularization. The value of the regularization coefficient of the equation (9) was fixed to 1e-4.
Domain and Data normalization. As defined in the remark (B. 1). the domain Ω2 is defined to be J0, 1 Data are normalized so that they belong to [0.25, 0.75]", thus ensuring that they are included in Ωt and that no example is too close to the boundary a2, where the basis functions equal to due to the Dirichlet conditions.
Data Augmentation. No data augmentation was performed.
Baselines. For MNIST and 100 we report the results from Massaroli et al. (2020). For the tabular datasets, we used an architecture similar to the one used for the function a(z).
16

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
Low sample scenario. We train DSL with strong spectral regularization (ar 10.0) and perform experiments using different random seeds. The best model is selected using validation set of 1,000 samples, and testing is conducted separate test set of 1,000 samples. The model is trained for 200 epochs. "As part of ablation study, to evaluate the role of the vector field in our approach, we apply the following Sturm Noum--Louville formulation:
F(z) = Lu(x, 0.5)
d dt plz, t) du(z, t) + (),xx = ((x)w(( t))
dt
u(z, 0) 0 du(ar, 0) 1 u(x,
dl
Here, p. 9, and are modeled as MLPs. The input each MLP is formed by concatenating the data point æ with the time variable
The following results show the accuracy of the ablation model evaluated on the tabular dataset:
DATA SET
DSL (OURS)
DSL (ABLATION)
NODE
ADULT
84.28%
84.21%
84.06%
DRY BEAN
91.14%
88.61%
91.45%
BANK MARKETING
83.10%
84.63%
83.77%
Solver sensitivity We conduct small experiment (Figure E) to analyze the sensitivity of the solver's precision on the mapping function used estimate the gradient via the implicit function theorem. After achieving certain level of precision, observe that the error in the mapping function becomes linearly proportional the solver's precision...
10
10-3
Error inapping function
10
10
10-7
Solver Presicion
Figure Error on the mapping function on the DryBean datatset
17

---

Deep Sturm-Liouville: From Sample-Based to ID Regularization with Learnable Orthogonal Basis Functions
Sample efficiency by eigenvalues:
82
80
75
Accuracy
74
72
DSL Eigenvalues
Eigenvalues
DSL Eigenvalues
DSL Eigenvalues
66
ML.P
65
100
200
300
400
500
600
700
800
Number of training samples
Figure 7. Impact of training sample on Accuracy Test Accuracy on Bank dataset as function of number of training samples and the number of eigenvalues.
18


Reviews:
[{"review_id": "QCr0M8jXaF", "reviewer": "Reviewer_upBa", "summary": "This paper introduces Deep Sturm-Liouville (DSL), a novel function approximator that integrates the Sturm-Liouville Theorem (SLT) into deep learning to achieve continuous 1D regularization along field lines in the input space. Demonstrates competitive performance and improved sample efficiency on diverse datasets including MNIST and CIFAR-10, showing DSL's effectiveness in practical machine learning tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer upBa for taking the time to review our paper in detail and for recognizing the novelty of our approach. The novelty of our approach was highlighted also by reviewer 2moq, who described our article as \"highly original with no directly related techniques in a published literature\". We appreciate your concern regarding the scalability of our method to large-scale datasets or real-time applications. We fully agree that this limitation may hinder the immediate widespread deployment of our framework in such contexts.\n\nHowever, as the reviewer noted, the core contribution of our work lies in introducing a new and principled approximator framework for regularization, grounded on theoretical results. As such, we believe it should be viewed as a foundational step—laying the groundwork for future improvements and extensions.\n\nImproving scalability is a clear direction for future work. Potential avenues include designing more efficient approximations of the Sturm–Liouville basis, new methods to compute gradients through the eigenvalue process that avoid relying on the implicit differentiation theorem, as well as exploring alternative ODE solvers that are better suited to Sturm–Liouville-type equations. Additionally, we plan to explore new forms of regularization beyond the spectral and implicit ones introduced here. We see this work as the first step in a broader research agenda that combines structure, theory, and learning in a unified framework. We will update the paper to make this limitation clearer—especially in the limitations section—and outline these directions as promising areas for future work.", "claims_and_evidence": "After careful review, I did not find any claims that were obviously problematic or lacked sufficient support. The evidence provided seems comprehensive and convincing. The theoretical analysis is rigorous, and the experimental verification validates the authors' theoretical analysis.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate.", "theoretical_claims": "I did not check all the details of the proof line by line, but on the whole, the theoretical analysis part of the paper has clear logic and reasonable proof structure.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are generally sound and valid.", "relation_to_broader_scientific_literature": "The paper's integration of Sturm-Liouville Theory with deep learning provides a novel mathematical foundation for developing more expressive and generalizable function approximators. This interdisciplinary approach bridges gaps between applied mathematics and machine learning, offering new insights into how to design models that can better capture the underlying structure of data.", "essential_references_not_discussed": "No literature missing.", "other_strengths_and_weaknesses": "Strengths:\nThe paper presents a novel integration of Sturm-Liouville theory with deep learning, creating the Deep Sturm-Liouville (DSL) framework. The introduction of 1D regularization along field lines in the input space addresses a fundamental limitation of traditional sample-based (0D) regularization methods, providing a new dimension for controlling model complexity and improving generalization. The experiment verifies the correctness of the proposed method. The overall structure of the paper is clear and easy for readers to understand.\n\nWeaknesses:\n1、\tThe computational overhead of solving Sturm-Liouville problems might restrict scalability to very large-scale problems or real-time applications, especially in scenarios with abundant training data.\n2、\tThe experimental is limited. The author only verified it on small datasets. There is a lack of verification on the large-scale datasets.", "other_comments_or_suggestions": "No", "questions_for_authors": "1、\tThe computational overhead of solving Sturm-Liouville problems might restrict scalability to very large-scale problems or real-time applications, especially in scenarios with abundant training data.\n2、\tThe experimental is limited. The author only verified it on small datasets. There is a lack of verification on the large-scale datasets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJeBJnWBa0", "reviewer": "Reviewer_2moq", "summary": "In the present contribution authors describe a novel approximation scheme suitable for general mappings $\\mathbb{R}^{m}\\rightarrow \\mathbb{R}^{n}$ where both $m$ and $n$ may be large. The scheme is suggested to be used as an alternative to neural networks.\n\nA simplified description of the proposed mapping $y = f(x),$ where $x\\in\\mathbb{R}^{m},\\,y\\in\\mathbb{R}^{n}$ is as follows\n1. Solve the neural ordinary differential equation (ODE) starting from $x$ both for positive and for negative time, until the trajectory hits the boundary of the hypercube. From that record the whole trajectory $\\gamma(t)$ and boundary points $t_{-}$, $t_{+}$, where the trajectory hit the hypercube.\n2. Use $\\gamma(t)$ to define parameters of one-dimensional Sturm–Liouville problem on the interval $\\left[t_{-},t_{+}\\right]$\n3. Solve this problem for first $k$ eigenvectors $v_{j}(x),\\,j=1,\\dots,k$\n4. Compute $y_{i} = \\sum_{j}w_{ij} v_{j}(x),\\,i=1,\\dots,n$\n\nIn this scheme the following parameters are potentially learnable:\n1. Vector field of neural ODE\n2. Parameters of functions that compute coefficients of Sturm-Liouville problem from $\\gamma(t)$\n3. Number of basis functions used from Sturm-Liouville problem\n4. Weights of linear transformation $w_{ij}$\n\nAuthors experimentally evaluated the proposed scheme and show that it achieves results competitive with other more standard approaches.\n\nIn addition to experimental evaluation authors prove two theoretical results:\n1. The obtained eigenvectors are orthogonal on $\\Omega\\subset \\mathbb{R}^{m}$, i.e., the input space where $x$ is defined and not only on the curve $\\gamma(t)$.\n2. That proposed scheme is related to the Dirichlet rank-1 parabolic eigenvalue problem.\n\n## update after rebuttal\n\nSummarised in https://openreview.net/forum?id=CzSNEvCckO&noteId=6lXg5LQbdT", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank you for your detailed review, for carefully examining the demonstrations and for recognizing the novelty of our approach.\n\nI — Demonstration 2\n\nFirst, we would like to address the mistake you identified in Demonstration 2, specifically in lines 728–732. The reviewer is absolutely correct, and we sincerely thank you for pointing this out. This demonstration is a key element of our paper, and we greatly appreciate your attention to its details. Fortunately, the mistake can be corrected without affecting the main results. Below, we explain how to fix it. As a reminder, the objective of this part of the demonstration is to construct a change of variables such that the first row of the Jacobian matrix of the mapping function $(t, y = F(x))$ is the vector $a(x)$ which is required to be orthogonal to the remaining rows of the Jacobian.\n\n(i) — As the reviewer correctly pointed out, $a(x)$ must be the gradient of a scalar function. We will explicitly add this as a condition in the theorem. This is directly related to lines 251–252 of the paper, where we assume the absence of singular points. \n\n(ii) — The reviewer also rightly pointed out an issue in the construction of the orthogonal basis: the Gram-Schmidt process alone is not enough as the resulting vectors also need to be gradients of scalar functions. To address this, we will leverage tools from differential geometry by interpreting the change of variables as defining a one-dimensional bundle generated by the vector field $a(x)$. We will add the following explanation to the paper:\n\n``Let \\(E\\) be a one-dimensional real vector bundle over a manifold \\(M\\). It is known (see Milnor \\& Stasheff's Characteristic Classes) that \\(E\\) is trivial if and only if it admits a global nowhere‐vanishing section—equivalently if its first Stiefel–Whitney class \\(w_1(E)\\) vanishes. In this case, a nowhere‐vanishing section \\(a\\) provides a canonical trivialization $E \\cong M \\times \\mathbb{R}$. Furthermore, if we choose local coordinates adapted to this trivialization and if there exists a smooth function $V : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ such that $\\nabla V(x)=a(x)$, the Jacobian of the corresponding local map \\(T_x\\) can be arranged so that its first row is precisely \\(a(x)\\). This reflects the fact that the coordinate system is chosen to align the fiber direction (generated by \\(a(x)\\)) with the first coordinate axis. With an additional Riemannian metric on \\(M\\), the orthogonal complement Bundles lemma (see John M. Lee, Introduction to Smooth Manifolds) then provides an orthogonal decomposition $ T_xM = \\operatorname{span}\\{a(x)\\} \\oplus \\operatorname{span}\\{a(x)\\}^\\perp, $ completing the geometric picture. Consequently, the first row of the Jacobian is exactly $a(x)$, aligning the coordinate system with the direction defined by $a(x)$. Combined with the orthonormality of the remaining coordinates, this ensures that the first row of the Jacobian matrix is orthogonal to all other rows.''\n\nII — Ablation study\n\nTo evaluate the role of the vector field in our approach, the reviewer suggested an ablation study that removes its influence. We agree this is a valuable experiment and would like to describe it in more detail. In this ablation, we apply the proposed Sturm–Liouville formulation over the time interval $[0, 1]$ and evaluate the basis functions at $t = 0.5$. The input to each MLP is formed by concatenating the data point $x$ with the time variable $t$. \n\nFor the sample efficiency results and the accuracy performance on tabular data, please find below the updated plot and table that includes the ablation model https://rb.gy/xjx9tk. Our results show that DSL achieves higher sample efficiency and comparable accuracy compared to the ablation model.\n\nIII — Weakness and other questions\n\n1 — Code Availability: The source code is accessible via an anonymous GitHub https://rb.gy/n5iz02. Upon acceptance, the source code will be made publicly available. \n\n2 — Use of Neural Networks in DSL: We acknowledge the dependency of the DSL on neural networks. As a direction for future work, we are exploring ways to design a version of the DSL that does not rely on neural networks. DSL represents a foundational block that we aim to develop further.\n\n3 — 0D vs 1D Regularization: Since this question was also raised by another reviewer, we addressed it in the rebuttal of Reviewer cdFW (1).\n\n4 — DSL Coverage of $\\Omega$: The reviewer is right to point out that, for a general vector field, the trajectories might not cover the entire domain $\\Omega$. To ensure full coverage, the ODE must satisfy certain constraints. These are discussed in the paper between line 215 (second column) and line 253 (first column), where we explain the uniqueness of $\\gamma(t_-)$ and $\\gamma(t_+)$. However, we realize this section may not clearly state that these constraints also guarantee coverage of $\\Omega$. We will add an explicit sentence to highlight this point.", "claims_and_evidence": "Authors made several mild claims:\n1. That the proposed approximation scheme leads to comparable performance to classical approaches.\n2. Theoretical claims on orthogonality of eigenvalues.\n3. Theoretical claims on the relation to parabolic eigenvalue problem.\n\nI believe that experimental claims are supported by results on CIFAR and MNIST.\n\nThe first theoretical claim seems to be supported too. At least I can not point to any problems with the proof.\n\nThe second theoretical claim is not supported. I will provide more details in the appropriate section of the review.", "methods_and_evaluation_criteria": "In general, I find evaluation criteria to be appropriate. However, I believe it would be beneficial for the readers to have access to more information on training time, memory load and other metrics along this line. The method proposed by the authors is quite complicated with many non-standard components that require custom derivation of derivative rules, this makes it hard to estimate computational requirements for using this technique.", "theoretical_claims": "I reviewed proofs for both theoretical claims.\n\nThe first one of the orthogonality of the eigenvalues \"in the bulk\" seems fine (Appendix C).\n\nThe proof for the second theoretical statement is not correct. In Appendix D on lines 728-732 authors assume the coordinate transformation they define is valid. Unfortunately, the assumption of the theorem does not exclude the situation when this transformation is not possible to define.\n\nIn the theorem authors assume $a_i(x)>0$. Next, they define scalar coordinate $t(x)$ by identity $\\nabla_{x} t(x) = a(x)$. To be able to do that, one need $a(x)$ to be conservative field. If this is not assumed one can easily build pathological examples. Consider$$a(x) = \\frac{1}{2}\\begin{pmatrix}(x_1 - x_2)^2\\\\\\\\(x_1 + x_2)^2\\\\\\\\1\\end{pmatrix}.$$Curl of this field reads$$\\nabla_{x}\\times a(x) = \\begin{pmatrix}0\\\\\\\\0\\\\\\\\ 2 x_1\\end{pmatrix}.$$Since $a_i(x) >0$ holds our choice agrees with conditions states in the theorem. However, it is not possible that $a(x) = \\nabla_x t(x)$ for some scalar function $t(x)$ since curl of $a(x)$ is not zero.\n\nThe rest of the coordinates defined by the author also have this property but for vectors $a_{k}(x)$ that are orthogonal to $a(x)$. This means these fields should also be potential, and authors need to carefully explain how they are going to build these additional orthogonal fields. The reference to the Gram-Schmidt process is not sufficient.", "experimental_designs_or_analyses": "I find the design and analysis of the experiment reasonable, besides the fact that no data on memory and computation load is available. Besides that in my view authors did not perform basic ablation study that seems appropriate given the complexity of the method they propose. I describe the suggested ablation below.\n\nThe scheme proposed by authors is to extract trajectory from neural ODE and later use this trajectory to construct a basis from Sturm-Liouville problem. The most obvious ablation to this scheme is to completely remove the step with ODE.\n\nThis can be done with simple modifications to Sturm-Liouville problem$$-\\frac{d}{d\\tau}\\left(p_{\\theta}(x, \\tau)\\frac{d}{d\\tau} u_i(\\tau)\\right) + q_{\\theta}(x, \\tau)u_i(\\tau) = \\lambda_i \\omega(x, \\tau) u_{i}(\\tau),$$where coefficients of Sturm-Liouville problem are neural networks.\n\nSince authors implemented an efficient differentiation through Sturm-Liouville solver, this kind of replacement is a simple modification of their code.\n\nI suggest authors perform this simplifications and report how this will affect accuracy and data efficiency.", "relation_to_broader_scientific_literature": "If one considers MLP, each layer can be regarded as a progressive building of basis. The role of the last layer is to use this learned basis to solve the problem of interest with linear or simple nonlinear (link) function (see, e.g., Bishop CM, Nasrabadi NM. Pattern recognition and machine learning).\n\nPresent contributions suggest an alternative way to construct this data-dependent basis based on the initial-value problem and boundary-value problem.\n\nThe first related approach from the literature in neural ordinary differential equations https://arxiv.org/abs/1806.07366. Neural ODE are most widely used in generative modelling https://arxiv.org/abs/1810.01367, https://arxiv.org/abs/2210.02747. However, special classes of these models can also be applied directly to classification https://openreview.net/forum?id=SAv3nhzNWhw, anomaly detection https://arxiv.org/abs/2302.07253, reduced-order modelling https://www.nature.com/articles/s41598-023-36799-6, segmentation https://arxiv.org/abs/2502.06034, etc.\n\nSince the technique proposed in the present contribution is related to partial-differential equations. This provides a second link to existing literature.\n\nMethods based on PDEs were popular in the image processing problems, e.g., Chan TF, Shen J, Vese L. Variational PDE models in image processing, but they fall out of favour with the adoption of methods based on neural networks. PDE-based approaches also recently made they way into the deep learning, e.g., in https://arxiv.org/abs/2403.15726 authors used reaction-diffusion model, in https://arxiv.org/abs/2502.06034 a wave equation, and in http://proceedings.mlr.press/v107/sun20a/sun20a.pdf a general neural PDE is considered. Still, PDE-based approaches are rare and the method proposed by authors of the present contribution is an interesting exception.", "essential_references_not_discussed": "I believe authors sufficiently discussed related literature. It seems Neural ODE is the most relevant related technique. Authors discuss Neural ODEs and perform numerical experiments explicitly comparing their method with Neural ODEs.", "other_strengths_and_weaknesses": "**Strengths:**\n1. Authors rigorously explain their approach including many details how derivatives are computed for the non-standard components.\n2. To the best of my knowledge the approach is highly original with no directly related techniques in a published literature\n\n**Weaknesses:**\n1. The scheme is very complicated and it is not clear which parts are necessary since ablation is not available\n2. The only advantage seems to be data efficiency that can be likely achieved with standard methods coupled with additional regularisation\n3. The code is not available\n4. Neural networks are still used to parametrise neural ODE and Sturm-Liouville problem, and in combination with minor to no improvement it rises the questions about the significance of the proposed approach", "other_comments_or_suggestions": "I have several minor questions that I list here:\n1. In several places authors mention that their approach is a \"1D\" regularisation and other known approaches are from \"0D\" regularisation, e.g., Lines 57-60. This terminology is not clear to me. Typically, regularisation implies some additional constraints, e.g., that the $l_2$ norm of weights is small, or that activations are normalised to $1$. Why does the approach by authors present a form of regularisation? If one uses some method to map between parts of input space (e.g., https://arxiv.org/abs/2303.16852), forming a trajectory between points, is it a form of regularisation too?\n2. Lines 85-88, right column. \"Conversely, DSL’s vector field operates directly within the input space ($\\Omega\\rightarrow\\Omega$), ensuring that field lines traverse the entire domain.\" Why does the field traverse the entire domain? Since all components authors used are learnable, the trajectories of neural ODE can be arbitrary, not necessarily \"dense\" in $\\Omega$.\n3. Lines 303-306, right column. In the formulation of Theorem 5.1, indices $i$ and $j$ should be distinct which is not explicitly specified.", "questions_for_authors": "Here a briefly summarise my main concerns:\n1. Proof of Theorem 5.3 is not correct (see above).\n2. The code is not available.\n3. No information about computation and memory requirements.\n4. The scheme is too complicated, and no ablation is available.\n\nSo, my main suggestions is to perform ablation study (see suggestions above) and correct the proof of Theorem 5.3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "na"}}, {"review_id": "nPfKBdzhDj", "reviewer": "Reviewer_cdFW", "summary": "This paper proposes a novel method (Deep Sturm-Liouville, DSL) that combines the Sturm-Liouville theorem with neural networks. By solving one-dimensional Sturm-Liouville problems, DSL generates orthogonal basis functions from the resulting eigenfunctions to approximate target functions and employs implicit gradients for network training. The method integrates both intrinsic implicit regularization and manually added explicit regularization. Experiments on the Adult, Dry Bean, Bank Marketing, MNIST, and CIFAR10 datasets demonstrate that DSL achieves performance comparable to traditional neural networks while exhibiting superior sample efficiency.\n\n## update after rebuttal\nI will maintain my score after reading the response from authors", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for carefully reading our paper, providing thoughtful feedback and recognizing the novelty of our approach.\n\nI — In response to the reviewer's main concerns:\n\n1 — 1D Regularization vs. 0D Regularization: By 0D regularization, we refer to regularization applied at discrete data points. For example, a term like $\\sum_i \\lvert \\nabla_x F(x_i) \\rvert$ computes the gradient of the predictor $F(x)$ only at the sampled points $x_i$. Ideally, one would compute a continuous regularization such as $\\int_{\\Omega} \\lvert \\nabla_x F(x) \\rvert  dx$, but this is generally intractable. Our method bridges this gap by computing regularization along one-dimensional trajectories defined by a vector field $a(x)$ for each data point $x$. This 1D regularization provides a more informative approximation of the continuous case by integrating along these paths. Within this framework, we introduce two new types of regularization: spectral regularization and implicit regularization. These are complementary to traditional techniques (such as L1 or L2 regularization), which we do not aim to replace. In future work, our framework could also incorporate other forms of regularization or even modify the loss function itself.\n\n2 — Implicit Regularization: The key idea behind implicit regularization in our framework is to retain only the first $n^\\text{th}$ basis functions. By controlling the number of retained basis functions, we also control their level of oscillation—a direct consequence of the Sturm–Liouville theorem (see lines 173–180). This is analogous to the Fourier basis, where higher-order components exhibit more oscillation than lower-order ones. However, Sturm–Liouville provides a more general formulation. This property allows us to construct smoother function approximators, which have a known theoretical connection to generalization. Moreover, sample efficiency is inherently tied to generalization performance (see [1, 2] ). Our experiments on sample efficiency were specifically designed to assess generalization behavior. By limiting the number of basis functions, we directly control the expressiveness and smoothness of the learned function, thus influencing the regularization effect. \n \nII — Regarding the reviewer's concerns on \"Experimental Design and Analyses\": \n\n1 — Architectural Ambiguity for the Baseline: The reviewer is right, and we will include a dedicated section to clarify this point in the appendix. For MNIST and CIFAR-10, we report the results from Massaroli et al. (2020), and we will explicitly reference this in the appendix. For the tabular datasets, we used an architecture similar to the one used for the function \\( a(x) \\). The source code is accessible via an anonymous GitHub https://rb.gy/n5iz02. Upon acceptance, the source code will be made publicly available to support the reproducibility of our work.\n\n2 — Incomplete Hyperparameter Analysis: We conducted an ablation study in Figure 4(b) of the paper to analyze the impact of the spectral regularization coefficient on training accuracy. However, the coefficient $\\alpha$ was missing from the figure and no reference was made to Equation (9). We will update the figure and its legend to include both the coefficient and the appropriate reference, as their absence is indeed misleading. \n\n3 — Computational Fairness: The reviewer is right to highlight that computational budget is a critical factor for fair comparisons. For MNIST and CIFAR-10, we chose to report the results from Massaroli et al. (2020), a recognized reference. For the tabular datasets, we used similar architectures for both the baseline and the DSL model to ensure a fair comparison in terms of parameter count and model complexity.\n\nLastly, thank you for pointing out several typos in the manuscript—we will correct them accordingly.\n\n[1] Zhang, Chiyuan, et al. \"Understanding deep learning requires rethinking generalization.\" arXiv preprint arXiv:1611.03530 (2016).\n\n[2] Arpit, Devansh, et al. \"A closer look at memorization in deep networks.\" International conference on machine learning. PMLR, 2017.", "claims_and_evidence": "In the abstract and introduction, the authors hypothesize that the generalization challenges of neural networks may stem from \"0D regularization\" (i.e., sample-point-based regularization) and claim that DSL can \"overcome\" this limitation. However:\n\n**Theoretical gaps**: No rigorous proof is provided to show that DSL inherently achieves better regularization or generalization than conventional methods. The paper also fails to explain how 0D regularization negatively impacts neural network generalization.\n\n**Experimental limitations**: The results only demonstrate comparable performance to existing methods (standard neural networks and neural ODEs), with no evidence that DSL actually addresses generalization difficulties.", "methods_and_evaluation_criteria": "The methodological framework of DSL is aligned with the core objectives of this research.", "theoretical_claims": "The theoretical proofs are rigorous.", "experimental_designs_or_analyses": "While the current results still meaningfully validate the feasibility and potential of DSL, the experimental section has certain limitations:\n\n**Architectural ambiguity**: The neural network architectures (e.g., layer configurations, parameter counts) used for baseline comparisons are not clearly specified.\n\n**Incomplete hyperparameter analysis**: While the impact of the number of basis functions is discussed, critical hyperparameters (e.g., the regularization coefficient α in Eq. 9, architectures of the field-line generation network) are not systematically analyzed.\n\n**Computational fairness**: DSL requires additional time to compute basis functions and other intrinsic information. To ensure a fair comparison, the authors should include results for traditional neural networks trained under equivalent computational budgets.", "relation_to_broader_scientific_literature": "Traditional regularization methods (e.g., weight decay, dropout) impose constraints in parameter space or via stochastic perturbations, while sample-level regularization (e.g., adversarial training, Mixup) enforces robustness to local input perturbations. These are inherently \"0D\" approaches, operating on discrete samples or parameters. In contrast, DSL introduces 1D regularization along field lines, extending constraints to continuous paths and implicitly enforcing function smoothness through orthogonal basis functions derived from Sturm-Liouville problems.", "essential_references_not_discussed": "No essential references were omitted.", "other_strengths_and_weaknesses": "Have mentioned above.", "other_comments_or_suggestions": "Typos:\n\nPage 2: \"Liouville\" is misspelled as \"Liouiville.\"\n\nPage 7, Line 333: \"Dirichlet\" is misspelled as \"Dirichelt.\"\n\nFigure references:\n\nFigure 5 (Page 8) is not discussed in the main text. The analysis of sample efficiency in Section 5 should reference Figure 5 instead of Figure 7 in the appendix.\n\nPage 7, Line 360: The reference to \"Figure 5.2.1\" is unclear.", "questions_for_authors": "1. How exactly does the proposed method address the generalization challenges attributed to 0D regularization?\n\n2. What is the mechanism behind the implicit regularization in DSL, and how does it theoretically or empirically outperform traditional neural network regularization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel class of function approximators called Deep Sturm-Liouville, which enables 1D regularization along continuous trajectories. The reviewers generally agree that this paper presents an interesting new approach to function approximation (even if at the cost of additional computational overhead), which is justified both theoretically and empirically. Based on this consensus, I recommend acceptance of the paper.
Decision: Accept (poster)