Generative Feature Training of Thin 2-Layer Networks
Authors: Johannes Hertrich
Sebastian Neumayer
Venue: Accepted by TMLR ()

Abstract: We consider the approximation of functions by 2-layer neural networks with a small number of hidden weights based on the squared loss and small datasets. Due to the highly non-convex energy landscape, gradient-based training often suffers from local minima. As a remedy, we initialize the hidden weights with samples from a learned proposal distribution, which we parameterize as a deep generative model. To train this model, we exploit the fact that with fixed hidden weights, the optimal output  weights solve a linear equation. After learning the generative model, we refine the sampled weights with a gradient-based post-processing in the latent space. Here, we also include a regularization scheme to counteract potential noise. Finally, we demonstrate the effectiveness of our approach by numerical examples.

Published in Transactions on Machine Learning Research (07/2025)
Generative Feature Training of Thin 2- 2-Layer Networks
Johannes Hertrich
Université Paris Douphine PSL
Sebastian Neumayer
Technische Universität Chemnitz
Reviewed on OpenReview: https: // openreview. net/ forum? id= 6d=6oXNNKUBD
Abstract
We consider the approximation of functions by 2-layer neural networks with only few hidden weights based on the squared loss and small datasets. Due to the highly non-convex energy landscape, gradient-based training oflen sulfers from local minima. As a remedy, we initialize the hidden weights with samples from a learned proposal distribution, which we paramotorize as a deop generative model. To train this model, we exploit the fact that with fixed hidden weights, the optimal output weights solve a linear equation. After learning the generative model, we refine set of sampled weights with a gradient-based feature refinement in the latent space. Here, also include a regularization scheme to counteract potential noise. Finally, we demonstrate the effectiveness of our approach by numerical examples.
1 Introduction
We investigate the approximation of real-valued functions f: 10, - R based on samples (rx, (r y=) where I& E [0, 1jd are independently drawn from some distribution Vdata and ys 2 f(zx) are possibly noisy obscrvations of fxx)). To achieve this, we study paramctric architectures fre [0, 1] R of the form
fw,b(z) = Re Σ ((0)((w),))))
(1)
where Re denotes the real part, Φ: R + C is a nonlinear function, and WI, uN E Rd are the features with corresponding weights bi....,bu by E C. If the function & is real-valued, the model ① simplifies to a standard 2-layer neural network architecture without Re and with bj...., by E TR. The more general model (1) also covers other frameworks such as random Fourier features (Rahimi & Recht 2207].. Since the Pareto principle suggests that most real-world systems are driven by a few low-complyxity interactions, we are interested in representations [1] with only few features wr. Such an explicit restriction of N also mitigates overfitting, seen in sparse neural networks, compressed sensing and feature selection.
For fixed & and N, we aim to find (w. b) RdNN x CN such that the fu,b from 1 approximates f well. From theoretical perspective, we can obtain such (w, by minimizing the mean squared error (MSE), namely
(wo b) E arg min ||f - fwr (12
(2)
w.b
In practice, we do not have access to Paata and but only to data points (Ix. M where æg are iid samples from Pdata and y1 are noisy versions of [Ixx). Hence, we replace 22) by the empirical risk minimization
(1ē b) E arg min
Iy
fw.((a)1..
(3)
1=1
However, if M is small, minimizing 3 can lead to overfitting towards the training samples (2t, yk ) and poor generalization. To address this issue, investigate the following principles.

---

Published in Transactions on Machine Learning Research (07/2025)
We use fw, of the form (1) with small N. This amounts to the implicit assumption that f can be sparsely represented using ① Unfortunately, under-parameterized networks (N & M) are difficult
to train with conventional gradient-base based algorithms (Boob et al. 2022; Holzmüller & Steinwart, 2022), see also Table Hence, we require an alternative training strategy.
Often, we have prior information about the regularity of f, i... that f is in some Banach space B with a norm of the form
Ifll /
[Df
(4)
10.111
where D is some differential operator and p.q the space of bounded variation (Ambrosio et al.)
A common example within this framework is
2000); which informally corresponds the choice
D V. q 2 and p 1. In practice, the integral in (4) is often approximated using Monte Carlo
mothods with uniformly distributed samples (Fm )s 1]. If we use (4) as regularizer for fwe..
the generalization error can be analyzed in Barron spaces (Li et al. 2022
Contribution We propose a generative modeling approach to solve (3) To this end, we first observe that the minimization with respect to a linear least squares problem. Hence, we can analytically express the optimal 6 in terms of w, which leads to a reduced problem. Using the implicit function theorem, we compute Vub(w) and hence the gradient of the reduced objective. To facilitate its optimization, we replace the deterministic features with stochastic ones, and optimize over their underlying distribution Pe: instead. We parameterize this distribution as Pu = GogN((, Ia) with a deep network Go: R - Rd. Hence, we coin our approach as generalive Seature training. Further, we propose to add a Monte Carlo approximation of the norm 4 to the reduced objective. With this regularization, we aim to prevent overfitting
2 Related Work
Random Features Random feature models (RFM) first appeared in the context of kernel approximation
[Rahimi & Recht 2007 Liu et al.] 2021), which enables the fast computation of large kernel sums with certain error bounds, see also RRahimi & Recht (0088;; Cortes et al. (2010); Rudi & Rosasco (2017). A similar strategy is pursued by [Huan et al. (2006) under the name extreme learning machines. Sparse RFMs (Yen et. al.) 2014 of the form (1) with only a few active features can be computed based on & basis pursuit (Hashemi et al. 2023). Since this often leads to suboptimal approximation accuracy, later works by Xie ct al. (202)); Saha et al. (2023): Bai et al. (2024) instoad proposed to apply pruning or hard-thresholding algorithms reduce the size of w. Commonly, the features are sampled from Gaussian mixtures with diagonal covariances. Unlike our approach, all these methods begin with a large feature set that has to contain sufficiently many relevant ones. Towards this strong implicit assumption, Potts& & Schmischke [002] Potts & Weidensager 2025) propose to identify the relevant subspaces for the feature proposal based on the ANOVA decomposition. Unfortunately, this only works if the features w itself are sparse (few non-zero entries), see Figure I Sparse features also enable the fast evaluation of the fw.s from [ via the non-equispaced fast Fourier transform (Dutt& & Rokhlin 1993; Potts et al. 22001). For kernel approximations, this can be also achieved with slicing methods (Hertrich) 2024 Hertrich al. 2025 which are again closely related to RFMs
[Rxx et al. 2025).
Adaptive Features Besides our work, there are several attempts to design data-adapted proposal dis- tributions Pa for random features {Li al.] 2019c; Dunbar et al. 2025). Recently, [Blager et al. (2023) proposed to sample the features w in regions where it matters, i.c., based the available gradient infor- mation. While this allows some adaption, the still remain fixed after sampling them (a so-called greedy approach). Towards fully adaptive (Fourier) features w. Li et al. (20195) propose to alternately solve for the optimal b, and to then perform a gradient update for the w. Kammonen el al. (2020) propose to instead update the w based on a Markov Chain Monte Carlo method. Unlike our approach, both methods do not incorporate the gradient information of b into the update process of Pw- It is well known that the surrogate alternating updates may perform poorly in certain cases. Note that learnable features have been also used in the context of positional encoding (Li et al. 2021 and implicit kernel learning (Li et al. 2019a)..
2

---

Published in Transactions on Machine Learning Research (07/2025)
2-Layer ReLU Networks We can interpret 2-layer neural networks adaptive kernel methods (E al, 2019). Moreover, they have essentially the same generalization error as the RFM. Several works investigate the learning of the architecture ! with & = ReLL based on a (modified) version of the empirical risk minimization 3 Based on convex duality, PPllanci & Ergen 2020) derive a semi-definite program to find a global minimizer of (3)- A huge drawback is that this method scales exponentially in the dimension d. Later, several accelerations based on convex optimization algorithms have been proposed (Mishkin et al., 2022 Bai let al. 2023). Following a different approach, Barbu (2023) proposed to use an alternating minimization over the parameters w and b that keeps the activation pattern fixed throughout the training. While this has an improved complexity of (dd) in d, the approach is still restricted to ReLUU-like functions Φ. Moreover, gradient-basee based optimization of the parameters for generative (proposal) network such as ours is empirically known lo scale very well with d. A discussion of the rich literat on global minimization guarantees in the over wer--praaeterized regime (N > M) is not within the scope of a sparse architecture
Bayesian Networks Another approach that samples neural network weights is Bayesian neural networks (BNNs) Neal 2012 Jospin et al., 2022). This allows to capture the uncertainty the weights in over- parameterized architectures. A fundamental difference to our approach and RFMs is that sample the features { independently from the same distribution, while BNNs usually learn a separate one for each uy. Further, BNNs are usually trained by minimizing an evidence lower bound instead of (S). see for example
Graves 2011 Blundell et al. 2015 which is required to prevent collapsing distributions.
3 Generative Feature Learning
Given data points (Ix, yk = with yv = f(x) for some underlying f: [0, 1] - R, we aim to find the optimal features w = (wn) )/ C Rd and weights b E CN such that Su, R J, where Sw.b is defined in (1. Before we give our approach, we discuss two important instances of the nonlinearity 4: RR + C from the literature.
Fourier Features:
The choice P(x) = 2xiz is reasonable if the ground-truta function f can be represented by few Fourier features, c.g., if it is smooth. As discussed in Section 2 the deployed fcatures are commonly selected by randomized pruning algorithms.
2-Layer Neural Network: For 4: R I., we can restrict ourselves to b Common examples are the ReLU ((x) max 0) and the sigmoid o(x) = Ite --= Then, fw.b corresponds to 2-layer neural network (i.e., with one hidden layer). Using the so-called bias trick, we can include bias into T That is, we use padded data-points (za, 1) E IRd+1 such that the last entry of the feature vectors wy E [R++1 can act as bias. Similarly, an output bias can be included.
In the following, we outline our procedure for optimizing the parameters w and b for a general fw.b of the form First, we derive an analytic formula for the optimal weights b in the empirical risk minimization 3 with fixed features w. Then, in the spirit of random Fourier features, we propose to sample the w from a proposal distribution Pw, which we learn based on the generative modeling ansatz Pw = Go,,(( Ta)- As last step, we fine-tune the sampled features = Gz(z) by updating set of sampled latent features z with the Adam optimizer. In order to be able to deal with noisy function values y/ s f(xx), can regularize the approximation fw.b during training. Our complete approach is summarized in Algorithm I
3.1 Computing the Optimal Weights
For fixed (w) any optimal weights b(w) E CN for (3) solve the linear system
A Am b(w) A.
(5)
where = (wx) and A,, = (I((T,, wi) ) In order to stabilize the numerical solution of (5) we deploy Tikhonov regularization with small regularization strength E > O. This resolves the potential rank deficiency of AFA and we compute b(w) as the unique solution
(A An eT)b(w) AT.y-
(6)
3

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm GFT and GFT-r training procedures.
1: Given: data (IR, yk 3k)kem km1, architecture fw.b as in 2: while training Gg do sample N latent 2 ~ 1y and set w Go(=) compute optimal b(w) and V.b(w) based on 6
generator Gø, latent distribution 7)
compute VoL(0) or (o with automatic differentiation perform Adam update for 0
GFT-r then
while refining w do set w Gz()
10:
compute optimal b(w) and Vwb(w) based on (6) compute [=) V [=Freg(=) with automatic differentiation
11:
12:
perform Adam update for z
13: Output: features w and optimal weights b(w)
For -> 0, the solution of (6) converges to the minimal norm solution of 5 A key aspect of our approach is that we can compute (() using the implicit function theorem. This requires solving a linear equation of the form 6 with a different right hand side. For small N, the most efficient approach for solving (6 is to use a LU decomposition, and to reuse the decomposition for the backward pass. This procedure is implemented in many automatic differentiation packages such as Py Toroc,, and no additional coding is required.
By inserting the solution b(w) of 6 into the empirical loss (3),
we obtain the reduced loss
L(w) Σ f(xx) (m.
(7)
Naively, we try to minimize 7 directly via gradient-based method (such Adam with its default parameters) starting at some random initialization w² (wf) C Rd. We refer to this as feature op- timization (F-Opt). However, I(w) is non-convex, and our comparisons in Section 4 reveal that feature optimization frequently gets stuck in local minima. Consequently, a good initialization wo is crucial if we method. In the spirit of random Fourier feat ures, we propose to initialize the as independent identically distributed (iid) samples from proposal distribution Pw- To the best of our knowledge, current random Fourier feature methods all rely on handcrafted P
want to minimize (7) with a gradient-
3.2 Learning the Proposal Distribution
Since the optimal Pw is in general not expressible without knowledge of we aim learn it from the available data (Ix, ) based on generalive model. That is, we take simple latent distribution 1) (such the normal distribution (0, Ia)) and make the parametric ansatz p Gø#n. Here, Gg: Rd + Rd is fully connected neural network with parameters f and # denotes the push-forward of under Go. To optimize the parameters 0 of the distribution = Go we minimize the expectation of the reduced loss 7 with iid features sampled from Go4?,, namely the loss
M
[(0) = Eww((Gan)))) [Lw)] = BeNneN [L(G((2))] = E ON
f(xx) fcueercccc)))) (8)
A=1
where the notation μ @N denotes N N-times the product measure of We minimize the loss by stochastic gradient-based algorithm. That is, in cach step, we sample one realization 89 of the latent features to get an estimate for the expectation in (S) Then, we compute the gradient of the integrand with respect to 0 for this specific 2, and update with our chosen optimizer. In the following, we provide some intuition why this outperforms standard training approaches. In the carly training phase, most of the sampled features w do not fit to the data. Hence, they suffer from vanishing gradients and are updated only slowly. On the other hand, since the stochastic generator Ga leads evaluation of the objective I(w) at many different
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
locations, we quickly gather gradient information for large variety of features. In particular, always taking fresh samples from the iteratively updated proposal distribution Pio helps to get rid of useless features.
3.3 Feature Refinement: Adam in the Latent Space
Once the feature distribution Pv- Gg is learned, we sample a collection 20 (zP) of iid latent features =f 11- By design, the associated features w Go(=") (with Go being applied elementwise to =f; =i.-... =& serve estimate for minimizer of 77. Since these w are only an estimate, refine them similarly described for the plain feature optimization approach from Section 3.1 More precisely, starting in 2 instead of a random initialization, we minimize the function
M F(z) = L(Go(2)) = - (aoeMiGeaa
(9)
where L is the loss function from By noting that VF(2) VGa(s)"V this corresponds to initializing the Adam optimizer for the function L(w) with w" = Go(=", and to additionally precondition it by the Jacobian matrix the generator Gg. the step size chosen appropriately, we expect that the value of F(z) decreases with the iterations. Conceptually, our refinement approach similar many socond-order optimization routines, which also require good initialization for convergence.
3.4 Regularization for Noisy Data
If the number of training points M is small or if the noise on the = f((L) is strong, minimizing the empirical risk (3) can suffer from overfitting (i.e., the usage high-frequency features). To prevent this, we deploy a regularizer of the form 4 Choosing p q= 1 and D V in leads to the following training problem with (anisotropic) total variation regularization (Acar Vogel 1994
Chan & Esedoglu, 2005
/
à E arg min
1yk (w, AR(w), R(w)=
I JV (..b(w) (w),
(10)
(amin Amax]
where >> and amin min 1,.,, M) and Qmax max k 1...., M) the entry-wise
minimum and maximum of the training data. For generative training loss (8) adding the regularizer
from 10 leads to
Lrog(() = (G (T(w) + AR(w)).
(11)
Similarly, we replace the F' from 9 for the feature refinement in the latent space by
Frog(: (2) F(2) + AR(GG(==)-
(12)
If we have specific knowledge about the function f that we intend to approximate, then apply more restrictive regularizers of the form 10).. As discussed in Section 2 several RFMs instead regularize the feature selection by enforcing that the features wy E gd only have a few non-zero entries (sparse features).
Given the nature of our reumerical ezamples, we only discussed f: 10, R with dala points yk E R. The extension of our method to maltivariate. f: [0, 1]d - R is straight forward.
Remark 1.
4 Experiments
We demonstrate the effectiveness our method with three numerical examples. First, we visually inspect the obtained features. Here, we also check if they recover the correct subspaces. Secondly, we benchmark our methods on common functions from approximation theory, i.c., with a known groundtruth. Lastly, target regression some datasets from the UCT database (Kelly al. 2023).
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
4.1 Setup and Comparisons
For all our experiments, we set up the architecture Sw, b in ①1 with N = 100 features (wi) and one of the nonlinearities + introduced in Section B
We deploy ((x)
2 without the bias trick. This corresponds the approximation of the underlying ground truth function by Fourier features.
We deploy ((x) TheT, which corresponds to 2-layer network with sigmoid activation functions.
To improve the expressiveness of the model, we apply the bias trick for both layers.
An ablation for different choices of N is given in Appendix A Further, we choose the generator Go for the proposal distribution Pu: = G0, La) ReLU network with hidden layers and 512 neurons per hidden layer. To pick the regularization strength X, divide the original training data into training (90%) and validation (10%) set. Then, we train Gg for each À E {0) U (1 x 10 h -4....,0 0} and choose the À with the best validation error. To minimize the regularized loss functions freg (GFT, see also (11) and Freg (GFTTr, see also [1)), we run 40000 steps of the Adam optimizer. The remaining hyperparameters are
given in Appendix C We benchmark all our methods from Section 3
F-Opt: In the feature optimization, minimize the L(w) from 7 with gradient- based optimizer starting with features drawn from standard normal distribution. As explained in Section 3.1 we expect that the optimization gets stuck in a local minimum. We verify this claim in our experiments.
GFT: For the generative feature training as proposed in Section 3.2 we minimize the loss L()) from
{ and draw iid features w from the generator Go during evaluation.
GFT-r: For the refined generative feature training, we generate features using GFT and refine them with the procedure from Section 3.3 This requires to minimize the loss F(z) in (9)
For each method, we specify the choice of & as "Fourier" and "sigmoid" activation in the corresponding lables. We compare the obtained results with algorithms from the random Fourier feature literature, and with standard training of neural networks. More precisely, we consider the following comparisons:
Sparse Fourier Features: We compare with the random Fourier feature based methods SHRIMP
[Xic et al.) 2022). HARFE (Saha et al.) 2023), SALSA (Kandasamy Yu 2016) and ANOVA-
boosted random Fourier features (ANOVA-RFF; Potts & Weidensager 2025). We do not rerun the
methods and take the results reported by Xie et al. (2022); Potts Weidensager (2025).
2-Layer Neural Networks: We train the parameters of the 2-layer neural networks fw, b with the Adam optimizer. Here, we use exactly the same architecture, loss function and activation function as for GFT. Additionally, we include results for the ReLU activation function ((x) max(x, 0).
Kernel Ridge Regression: We perform a kernel ridge regression (Cristianini & Shawe- Taylor 2000) with the Gaussian kernel, where the kernel parameter is chosen by the median rule.
Our Pylorch implementation is available online" We run all experiments NVIDIA RTX 4090 GPU, Depending on the specific model, the training takes between 30 seconds and 2 minutes. We include further ablations in Appendix B
4.2 Visualization of Generated Features
First, inspect the learned features in simple setting. To this end, we consider the function g: R R with g(x) = =) Since each summand of depends either on 21 or I2, its Fourier
availlbbe httpp://

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
RFF for 9t, test MSE: 6.45E 05
ANOVA-RFF for 9,, test MSE: 9.66E 01
ANOVA RFF for 9u. test MSE: 8.89E
7.5
7.5
75-
2.5
2.5
25
0.0
-2.5
-2.-
-5.0
-5.0
-5.0
-10.0
-30.8
-100
Neural test MSE: 3.50E 02
Neural for 6,95E
Neural MSE: 4.94E
10.0
10.0
10.0
5.0
5.2
5.0
0.0
0.0
-2.5
-25
-25
-7.5
-73
-75
-10.0
-7.5
-5.0
-25
0.0
25
5.0
75
10.0
-30.0
-10.0
-75
-59
-2.
0.0
25
5.0
7.5
10.0
-10.0
-7.5
-5.0
-25
0.0
2.5
5.0
7.5
10.0
GFT gt. best MSE:
1.23-004
GFT for Sb, MSE:
3.84E
GFT for gb. test MSE:
2.49E-03
10.0
10.0
7.5
7.5
7.5
2.5
2.5
255-
-2.5
-25
5.0
5.0
5.0
-10.0
-10.0
-10.0
-7.5
-3.0
GFT-r test MSE: 6.17E-07
GFT-r for test MSE: 1.03F-06
GFT-r test MSE 1.63E-04
16.0
10.0
10.0
5.0
5.5
5.0
0.0
-2.5
-2.
7.5
75
7.5
-100 -7. -3.0
-25
0.0
23
5.0
100
-10.
-73
-30
-25
00
23
5.
75
100
-000 -7.
-300
-2.
255
5.0
7.
10.0
Figure 1: Location of the generated features for the from of the associated weights by The gray lines indicate the support the Fourier transform of g., Since ANOVA- RFF constrains the fcatures to be on the axes, only works for 91- For the standard neural network training, the features not pushed to the axis. This indicates that the optimization got stuck in local minimum.
13 where the marker size rellects the magnitude

---

Published in Transactions on Machine Learning Research (07/2025)
transform is supported on the coordinate axes. To make the task more challenging, slightly adapt the problem by concatenating 9 with two linear transforms As, which leads to the three test functions
( 1 1
cos( + sin
sin( cos(
) A3
0.3
(i(x)= - (AAi)) with A =
A2 =
(13)
0.3
In all cases, the Fourier transform is supported on union of two subspaces. Now, we learn the features with our GFT and GFT-r method based on 2000 samples that are drawn uniformly from [0, 1]2 and plot them Figure 1 The gray lines indicate the support of the Fourier transforms of 9i, and the size of the markers indicates the magnitude of the associated by- For all functions Ji, the features w sampled with GFT are mostly located the support of the Fourier transform. The very few features that are located outside of can be explained by numerical errors and are mostly removed by the refinement procedure GFT-r. In contrast, for ANOVA-RFF, the w restricted to be located the axes. Consequently, it cannot work for 92 and Is, and the error obtained is large. For gradient-based neural network training, the are not pushed to the axis, indicating that the optimization got stuck at local minimum. For functions where the subspaces are orthogonal, such as 92, this issue was recently addressed in Ba et al. (2024) by learning the associated transform in the feature space.
4.3 Function Approximation
We use the same experimental setup as in (Potts & Weidensager 2025 Table 7.1), that is, the test functions
Polynomial: [i))) = 2 2223 + 21 + 24;
Isigami: f2(z) sin(r1) + sin sin (x:2 + 0.1zI sin(;));
Friedmann-1: fa(x) = 10 sisin + 20(x3 112² + 10x, + 5z5-
The input dimension d is set to 5 or 10 for cach fx- In particular, the fk might not depend on all entries of the input x. For their approximation, we are given samples dx ~ Uo,1j4 noise-less function values (xx). The number of samples M and the dimension are specified for each setting. As test we draw M additional samples from Mu.1p= We deploy our methods as well as standard neural network training to the architecture fw.b. The MSEs on the test are given in Table 1 There, also include ANOVA-random Fourier features, SHRIMP and HARFE for comparison. We always report the MSE for the best choice of p from Potts & Weidensager 2025 Table 7.1. The GFT-r with Fourier activation functions outperforms the other approaches significantly. In particular, both the GFT and GFT-r consistently improve over the gradient-based training of the architecture fw.s- This in line with the analysis of gradient-basod training in recent works (Boob et al. 2022 Holzmillerr & Stcinwart, 2022 As expected, Fourier activation functions are best suited for this task.
M, and the corresponding
So far, we considered functions fo that can be represented as sums, where each summand only depends small number of inputs 24- While this assumption is crucial for the sparse Fourier feature methods from Table [ it is not required for our methods. Therefore, we also benchmark them on the following non- non-decomposable functions and compare the results with standard gradient-based neural network training:
hi(x) = sin i x4) + 1|
ha(x:) / ||x: fell,, where the vector with all entries equal to
hg(x)) Vfacz) 105)n( + 20(x3 ) 10x, + 50g-
The results are given in Table 2 As in the previous case, clear advantage of GFT and GFT-r.
8

---

Published in Transactions on Machine Learning Research (07 (225))
Table 1: Comparison with sparse feature methods for function approximation: We report the MSE over the tost set averaged over 5 runs. The values for ANOVA-RFF, SHRIMP and HARFE are taken from Potts & Weidensager (2025). The deployed À is indicated below each result. The best performance is highlighted.
Mechod
Fumction f1
Function /2
Fanction fs
Method
Aesivatiin
(4. (0,300)
[d. M) (10. (00)
(4. 100)
[d. M) (10. 1000)
() (5. (6.500)
M) (10,
ANOVA-RFF -FFFFF SHIRIMP HARFE
Foutier
1.40 x 10""
1.16 x 10"
2.65 x 10
2.62 x 10"5
1.00 10
9.50 x 10
10-9
10-4
8.20 10-3
10-3
2.00 10-4
3.81 10-1
Fourier
5.82 x 10-1
2.38 10P
1.38 x 10-3
0.65 10-1
3.64 x 10"
3,98 10"
ridge sog
5,90 10-3
10-4
1.1 10-5
10-4
10-2
x 10"
2,36 x 10m4
1,3 x 10-9
5.28 x 10m5
2,23 x 10-d
3.14 10-0
2,96 x 10"
Fourler
(A
(A=1 x 10-1)
(A= 02
(A=1 x 10-1)
(A=1xx 10701
(A=1xx x0=1x1001)
noural
signoid
3.84 x 10m9
(A=
(A=1x 10
5.34 x 10
(x=1x 107*)
2,25 x 10
(A=1 x 10"1)
3.71 x 10
2,56 x 10 (A=0)
2,15 16"
(A x11x0000>) 10-")
4.57 x 1071
1,25 x 10
121 10111 10
1,65 x 10
6.77 10-1
1,55 16"
ReLU
()= 1 x 10
(A=
(A= 03
(x=
(A=1 10"*))
(A 1-xx 10-5)
2,50 10-3
1.08 10P
2,30 x 10-6
1.31 10"
5.93 x.33x100- 10-2 (A=0)
168 10
Fourier
(A= x 10 +)
(A 1 10-")
(x= 10 +)
(A= 1 x 10-1)
(A= 10 10-+)
F-Ope
1,15 10-6
2.77 10
1.43 x 10-5
9.54 10-6
5.45 x 10-1
2.91 10"
sigmoid
(A- 0j
() 0)
(A- 0g
(A D)
(A (A-00 0)
(A 1A-0) 0)
2.72 10-7
5.00 10-7
LG3 10-7
4.09 10-7
5.87 10-5
4.47 10-3
(A= 0]
(A= D)
(A=0)
(A= D)
11=1x 10-")
(A = 1 xx10"))
GFT
3.18 10-*
10-9 (A= D)
4.00 1077
10-?
6.40 10-4
1,18 10-2
signoid
(x= 0)
(x=0)
(A= D)
(A=0) 0)
(x= 1 10-1)
6.05
5.46 10-6
2.02 10
8.15 10-*
10-
10-4 (A= 0)
Fourier
(A= 0)
(A= 0)
(x= 0)
(A= 0)
(A= 0)
GFT-
105 10-9
5.60 10-7
4.97 10-4
L12 10-7
1,50 10-3
9.91 10-1
sigmoid
(A 0j
(A 0)
(x= 0)
(A= 0)
(A 0)
(A 10100000
Table 2: Function approximation: We report the MSE over the test set averaged over 5 runs. The deployed À is indicated below each result. The best performance is highlighted.
Mathod
Function &
Function fy
Pamction bg
Method
Activation
(d,M) (10, 1000)
(d.M) (20, 1000)
(d, ) (5,500)
kernel ridge reg
8.91 x 10-2
3.71 x 10-3
5.55 10-3
6.03 10
1.34 10-2 (A=
2:68 10
(A= x 10-3)
(A 0)
net.
sigrasid
4.17 10-2
5.94 10-3
4.42 10-4
(A = 0)
(A= 1 10-*)
(A = 1 10-1)
5.64 10-1
6.89 10-3
5.56 10-3
ReLU
(A x 10-1)
(x-1x 10-*)
(A - 10-2)
10
5.06
7:24 10
(A= 1 x 10P)
(x=1x 10-1100-1
(A 11x1x 1x110-99 10-9)
F-Opt
10-2
7.08 10-4 (x=0)
2.35 10 (A=0)
sigmoid
(A= 0)
2.62 10
3.54 10-3
Fourier
(A=1 10-3)
(A=1x 10-1)
(A 1 1=100)
GFT
9.36 10-2
1.10 10-2
10-4
sigracid
(A=1 x 10-1)
(A= 1x 10-1)
(A= 1x 10-1)
8.96 10-a
2.57 x 10-a
1.04 10-4
Fourier
(A 0)
(A 1x 10-1000
(A- 0)
GFT-r
sigrold
6.06 x 10-2
x002
20-x10-1
(A=1 1: 10
A=1x10099
1x=1x 10-4)
4.4 Regression on UCI Datasets
Next, we apply our method for regression on several UCI datasets Kelly et al. (2023). For this, we do not have an underlying ground truth function f. Here, we compare our methods with standard gradient-based neural network training, SHRIMP and SALSA. To this end, we use the numerical setup of SHRIMP. For each dataset, the MSE on the test split is given Table 3 Compared and SALSA appear a bit more robust noise and outliers, which frequently oceur in the UCI datasets. This behavior is not surprising, since the enforced sparsity of the features wa for those methods is strong
the other methods, SHRIMP
9

---

Published in Transactions on Machine Learning Research (07 (025))
Table 3: Regression on UCI datasets: We report the MSE on the test datasets averaged over runs. The values for SHRIMP and SALSA are taken from Xie et al. (2022). The deployed À is indicated below cach result. The best performance is highlighted.
Method
Dataset
Propuision
Galaxy
Airfai
CCPP
Telemonit
Skilliraft
Mechod
Activation
(6. Mf) = 200)
(d. = (20, 2000)
(d. (f) = (11, 750)
(d.M) = (59, 2000)
(d. = (19. 1000)
(d,M) = (18, 1700)
SURIMP SALSA
Fourier Fourler
1.02 10-"
5.41 x 10""
2.65 x 10-
6.35 10-
6.00 10-
5.81 10
8.81 10-3
1,35 x 10"4
5.18 x 10-1
6.78 10"
3.47 10"2
6.47 x 10-1
kernel ridge reg
8.00 10-4
2,38 10-4
10
10
10
6.30 10"
9.07 10
1.46 x 10"44
3.11 x x010
6.97 10"2
2.51 x 10"2
6.01 x 10"1
Fourier
10-2)
(A=1x 1x11x1 10-4)
1 1=11x1000 10-1)
(A 10-1)
(A=1x 10-5)
(A 10-1))
aeural nes
signoid
9.21 10
(A= 10
167 x 10
3.31 10
(A=1x100 10-11
(A=1 10-1)
8.01 x 10-2
7,86 x 10
(A=1 10-2)
(A= 10-2)
1,57 x 10"
5,92 x 10-4
1.72 x0-** (A=0)
3.66 10-1
6.73 x 10-2
2.71 x 10-2
2:23 x 10"
ReLU
(A 10-2)
(A=1x 10-1)
(==1 10-1)
10-2)
(A= 10")
6,96 x 10-1
3.51 10"
1,0 10"
9.77 x 10-1
LO0 x 10"
Lol x 10"
Fourier
10-2)
(A 10-1)
(A-1x 10-1)
(A- 10-1)
(A- 10-1)
(A 10")
F-Opt
1.57 10-2
1.11 10
5,92 10-1
7,35 10
7:38 10
5.79 x 10
sigmold
(A x 10-1)
(A 0)
(A 10-9)
(A- x 10"1)
(A-1 10-9)
(A- 10")
8.31 10-7
3.1 10-5 (A 0)
2.34 10
8.06 10
1.05 10"?
5.60 10"1
Fourier
(A 0)
(A= 1 10-11
(A=1 x 10-2)
(A= 1 10-2)
(A= 1 10")
GFT
1.22 10
7.12 10
2,90 10
0.86 x 10"2
1.35 10
9:68 10
signnid
(A = 0)
(A = 0)
(A= 1 10-1)
(A=1 10-1)
(A= 1 10-5)
(A= 1 10-1)
6.97 10-7
5.36 10-#
2.34 10-1
8.04 10
6.48 10-
5.65 10
GFT-z
(A = D)
(A 0)
(A=1x 10-1)
(1=1x 10-2)
(A= 10")
(A=1 11 10")
1.67 10-4
1.85 10-5
2.89 10-1
6.84 10-2
9.39 10-3
8.88 10
sigmoid
(A = 0)
(A 10=00 0)
11=1x 10-1)
(A (A= 11=1x 10-11
(A = 0)
(A 1x 10-1)
implicit regularization. Incorporating similar sparsity constraints into generative training is left for future research. Even without such a regularization, GFT-r achieves the best performance on most datasets. Again, both GFT and GFT- achieve significantly better results than standard training with the Adam optimizer.
5 Discussion
We proposed training procedure for fw.b in 1 with only few hidden neurons In procedure, sample the from a generative model and compute the optimal b by solving linear system. To enhance the results, we apply a feature refinement scheme in the latent space of the generative model and regularize the loss function. Numerical examples have shown that the proposed generative feature training significantly outperforms standard training procodures.
Summary
Outlook Our approach can be extended in several directions. First, we want to train deeper networks in greedy way similar to (Belilovsky et al. 2019). Recently, similar approach was considered in the context of sampled networks by [Bolager et al. (2023). Moreover, we encode a sparse structure the features by replacing the latent distribution N(0, Ia) with lower-dimensional latent model or by considering mixtures of generative models. From a theoretical side, we want to characterize the global minimizers of the functional in and their relations to the Fourier transform of the target function.
Limitations If in I gets large, solving the linear system 6 becomes expensive. However, this corre- sponds to the overparameterized regime where gradient- based methods work well. Moreover, the computation of the optimal b depends batch-dependent, and both the theoretical and practical implications remain open. Instead, we emphasize that one motivation for our method is the treatment. of small data sets, where no minibatching is required. This is actually also one of the main use cases for 2-layer neural networks. Finally, note that GFT currently restricted to the Lo-loss function, which limits the applicability of GFT to non-regression tasks. For other loss functions, bilevel learning methods could be used to compute and differentiate the optimal output layer. This is beyond the scope of this paper, and we leave this point for future work.
all data points. Consequently, if we do minibatching, the output weights b are
10

---

Published in Transactions on Machine Learning Research (07 (025)
Acknowledgments
We would like to thank Daniel Potts, Gabricle Steidl and Laura Weidensager for fruitful discussions. JH acknwoledges funding by the German Research Foundation (DFG) within the Walter Benjamin Programme with project number 530824055 and by the EPSRC programme grant "The Mathematics of Deep Learning" with reference EP AP/VO26259/L
References
Robert Acar and Curtis R Vogel. Analysis bounded variation penalty methods for ill- ill-posed problems. Inverse Problems, 10(6):1217-1225) 1994.
Luigi Ambrosio, Nicola Fusco, and Diego Pallara. Functions of Bounded Variation and Free Discontinuity Problems. Oxford Mathematical Monographs. Oxford University Press, New York, 2000.
Fatima Antarou Ba, Olch Melnyk, Christian Wald, and Gabricle Steidl. Sparse additive function decompo- sitions facing basis transforms. Poundations of Data Science, 6(4):514 552, 2024.
Yatong Bai, Tammay Gautam, and Somayeh Sojoudi. Efficient global optimization of two-layer ReLU net- works: Quadratic-time algorithms and adversarial training. SIAM Journal on Mathematics of Data Sci- ence, 5(2):446-47) 2023.
Yaxuan Bai, Xiaofan Lu, and Linan Zhang. Function approximations via fi-E2 optimization. Journal of Applied & Numerical Oplimization, 6(3):371 389, 2024.
Adrian Barbu. Training two-layer ReLU network analytically. Sensors, 23(8):4072 2023.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to Ima- genet. In International Conference on Machine Learning, pp. 583 593. PMLR, 2019.
Charles Blundell, Julien Cornebise, Koray Kayukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613 1622. PMLR, 2015.
Erik Lien Bolager, Tryna Burak, Chinmay Datar, Qing Sun, and Felix Dietrich. Sampling weights of deep neural networks. In Advances in Neural Information Processing Systems, volume 37, 2023.
Digvijay Boob, Santanu Dey, and Guanghui Lan. Complexity training ReLU neural network. Discrete Optimization, 44:100620, 2022.
Tony F Chan and Selim Esedoglu. Aspects of total variation regularized [ function approximation, SIAM Journal on Applied Mathematics, 6111111817118 2005.
Corinna Cortes, Mehryar Mohri, and Ameet Talwalkar. On the impact of kernel approximation on learning accuracy. In International Conference on Artificial Intelligence and Statistics, pp. 113-120, 2010.
Nello Cristianini and John Shawe-Taylor. An introduction to support vector machines and other kernel-based learning melhods. Cambridge University Press, 2000.
Oliver RA Dunbar, Nicholas Н Nelsen, and Maya Mutic. Hyperparameter optimization for randomized algorithms: a case study on random features. Statistics and Computing, 35(3):1-25 28, 2025.
Alok Dutt and Vladimir Rokhlin. Fast Fourier transforms for nonequispaced data. STAM Journal Scientific Computing, 14(6):1368-1398) 1993.
Weinan E, Chao Ma, and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences, 1775):1407-1425, 2019.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Pro- cessing Systems, volume 24, 2011.
11

---

Published in Transactions on Machine Learning Research (07 (225))
Abolfazl Hashemi, Hayden Schaeffer, Robert Shi, Ufuk Topcu, Giang Tran, and Rachel Ward. Generalization bounds for sparse random feature expansions. Applied and Compulational Hormonic Analysis, 62:310-33 -0330, 2023.
Johannes Hertrich. Fast kernel summation in high dimensions via slicing and Fourier transforms. SIAM Journal on Mathematics of Data Science, 6:1109-1137, 2024.
Johannes Hertrich, Tim Jahn, and Michael Quellmalz. Fast summation of radial kernels via QMC slicing. International Conference on Learning Representations, 2025.
David Holzmüller and Ingo Steinwart. Training two-layer ReLU networks with gradient descent is inconsis- tent. Journal of Machine Learning Research, 23(181):1 82, 2022.
Guang-Bia Bin Huang, Qin- Yu Zhu, and Chee- Kheeng Siew. Extreme learning machine: theory and applications. Neurocomputing, (0(143)4489 501, 2006.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands- on Bayesian neural networks-a -a tutorial for deep learning users. IEEE Computational Intelligence Maga- zine, 17(2):29-45 2022.
Aku Kammonen, Jonas Kiessling, Petr Plecháč, Mattias Sandberg, and Anders Szepessy Adaptive random Fourier features with Metropolis sampling. Foundations of Data Science, 2(3):309-333 332, 2020.
Kirthevasan Kandasamy and Yaoliang Yu. Additive approximations in high dimensional nonparametric regression via the SALSA. In International Conference on Machine Learning, pp. 69-78. PMLR, 2016.
Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI machine learning repository, 2023. URL https://archive wtt.:/iirri.iiiiii.ii.......s..c uci.edu
Chun-Liang Li, Wei-Cheng Chang, Youssef Mroueh, Yiming Yang, and Barnabas Poczos. Implicit kernel learning. In International Conference Artificial Intelligence and Statistics, pp. 2007-2016. PMLR, 2019a.
Lingfeng Li, Xue-Cheng Tai, and Jiang Yang. Generalization error analysis of neural networks with gradient based regularization. Communications in Computational Physics, 32(4):1007-1038, 2022.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable Fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 11-15816-15829) 2021.
Yanjun Li, Kai Zhang, Jun Wang, and Sanjiv Kumar. Learning adaptive random features. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 33, pp. 4229 4236, 2019b.
Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of random Fourier features. In International Conference on Machine Learning, pp.
PMLR, 2019c.
Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel approxima- tion: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7128-711 2021.
Aaron Mishkin, Arda. Sahiner, and Mert Pilanci. Fast convex optimization for two-layer ReLU networks: Equivalent model classes and cone decompositions. In International Conference on Machine Learning, pp. 15770 15816. PMLR, 2022.
Radford M Neal. Bayesion Learning for Newral Networks. Springer Science & Business Media, 2012.
Mert Pilanci and Tolga Ergen. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In International Conference Machine Learning, pp. 7695-7703 PMLR, 2020.
Daniel Potts and Michael Schmischke. Interpretable approximation of high-dimensional data. SIAM Journal on Mathematics of Data Science, 3(4):1301- 11333 2021.
12

---

Published in Transactions on Machine Learning Research (07 (225))
Table 4: Comparison for function approximation with: We report the MSE over the test set averaged over 5 runs. The table contains the same experiments Table below each result. The best performance is highlighted.
1 with N 50 features. The deployed À is indicated
Method
Function fi
Fuaction f2
Function fg
Method
Activation
(d. (5.)
(d. (10,500)
(d.M) 500)
(d. M) (10. 1000)
(d.M)
(5. (0500)
(d.M) (10. 200;
3.35 10-5
9.22 x 10-6
6.97 x 10-6
8.15 x 10-6
2.45 10-1
4.63 x
Fourier
(A-
(A x 10-4)
(A
(A
(A- 10-
(A- 10-2)
neural nes
sigmold
103 x 10-3
(A
1,52 x 10-3
(A
1.19 x 10-3
(A 10-4)
1.45 x 10-5
2.14 10-a
(A-0)
2,27 x 10"
(A-1 x=11x00-9)
4.71 10-4
1.40 10-3
3.00 x 10-4
1,12 10"4
2.08 10-1
1,80 10"
ReI.U
(A
(x=1x 10-1)
(A=0)
(A=1x x=1x100">
1.00 x 10-5
2,52 x 10"
6.1 x 10-5
7 10" (A=0)
6.13 10-3
2.00 10
Fourler
(A= 107*)
(A= 0)
(A=
(1=1x 1000
(1=1x 10"1)
F-Opt
2 x 10-4
8.05 x 10-5
3.58 x 10-3
1.69 x 10-
LO8 10"
3,94 x 10"
sigmoid
(A=0)
(A 10
1x=1x 10"*))
0j
(A=11 10"1))
(A=1x
9.49 x 10
1.19 x 10--
5.71 10-3
1.98 x 10-5
9.17 x 10-2
8,72 x 10-3
Fourier
0)=0)
(A 10-2)
1 10-")
(1=1x x 10-4)
10-")
10-+)
GFT
1.94 x 10-1
6.08 10-3
5.41 x 10-1
9.01 x 10-1
9.81 x 10"
1,09 10-2
sigmool
()
10-*)
(A x 10-")
(A-1 10-*)
(A 1 x 10-2)
(A 10-5)
0)
5.52 10
2.94 10
1.24 10
9.76 10"7
8.72 10 (A=0)
2.02 x 10-4
Fourler
(A= 0)
(A
(A= 0)
(A= 1 x 10-1)
(A 0]
GFT-r
1.94 x 10
6.96 x 10
4.33 x 10
4.78 x 10"1
2,66 10 (A=
282 10 (A= 0)
signoid
(x=11 107*)
(A=
(A=0)
(x=1x10 10-1)
Daniel Potts and Laura Weidensager. ANOVA-boosting for random Fourier features. Applied and Compu- tational Harmonic Analysis, 79:101789, 2025.
Daniel Potts, Gabriele Steidl, and Manfred Tasche. Fast Fourier transforms for nonequispaced data: A tutorial. Modern Sampling Theory: Mathematics and Applications, pp. 147-270, 2001.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Newral Information Processing Systems, volume 20, 2007.
Ali Rahimi and Benjamin Recht. Uniform approximation of functions with random bases. In 46th Annual Allerton Conference on Communication, Control, and Computing, pp. 555 561. IEEE, 2008.
Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. In Advances in Newral Information Processing Systems, volume 30, 2017.
Nicolaj Rux, Michael Quellmalz, and Gabriele Steidl. Slicing of radial functions: dimension walk in the Fourier space. Sampling Theory, Signal Processing, and Dala Amalysis, 23(1):1 40, 2025.
Esha Saha, Hayden Schaeffer, and Giang Tran. HARFF: Hard-ridge random feature expansion. Sampling
Theory, Signal Processing, and Data Analysis, 21(2):27, 2023.
Yuege Xie, Robert Shi, Hayden Schaeffer, and Rachel Ward. SHRIMP: Sparser random feature models via iterative magnitude pruning. In Proceedings of Mathematical and Scientific Machine Learning, volume 190, pp. 303 318. PMLR, 2022.
Ian En-Hsu Yen, Ting- Wei Lin, Shou-De Lin, Pradeep K Ravikumar, and Inderjit S Dhillon. Sparse random feature algorithm coordinate descent. in Hilbert space. Tn Advances in Neural Information Processing Systems, volume 27, 2014.
A Dependence on the Number of Features
We redo the experiments from Section 4.3 for N 50 and N 200. The results given in Table and
13

---

Published in Transactions on Machine Learning Research (07 (225))
Table 5: Comparison for function approximation with: We report the MSE over the test set averaged over runs. The table contains the same experiments as Table 1 with N 200 features. The deployed is indicated below each result. The best performance is highlighted.
Method
Function fi
Firation f2
Function fa
Method
Activation
(d.M) (5.)
(d.Af) (10,500)
(d.M) 500)
(d. (10. 1000)
(d.M) (5,5
(G. (10. 200;
7.22 x 10-4
1.21 x 10-2
1,56 x 10-4
1,36 x 10-4
2:26 10-1
3.95 x 10"
Fourier
(A
(A 10-4)
(A- 10-4)
(A- 10-+)
(A- 10-
(A
neural nes
sigmold
1.54 x 10-3
2.35 x 10-5
1.46 x 10-3
4.91 x 10-3
1.11 10-3
1,79 x 10"
(A
(A
(A 10-*)
(A
(A-1 10-1x1100))
(A-0)
3.03 x 10-4
1,03 x 10-3
1,78 x 10-4
2.02 x 10-4
6:20 x. 10-2
1.84 x 10"
ReI.U
(A 0)=
(A=
(A=0))
(A= x 10-1)
(A=1 10
(A=11 x010""
0.45 x 10-3
1.02 x 10-2
4.47 x 10-5
2.34 x 10-2
1,29 x 10P
2,11 10
Fourler
(A=13 x 10"")
(A=1 10"")
(A=1x 107*))
(A=1* 10"")
(A=1x 1=
(A=1x10) 10))
F-Opt
2.52 x 10-9
5,73 x 10--
1.19 10-5
8,92 x 10-5
7.23 x 10
2.40
sigmoid
(A=0) 0)
(A
(A=0)
(>=0)
(A=0)
(A=0)
0.66 x 10-7
2.33 x 10-7
5.73 x 10
0.53 x 10-7
2.26 10-3
151 10"
Fourier
(A A
(A
(A=0)
()
A=00
(A=1x 10
GFT
2.84 x 10-6
3.52 x 10-6
2.04 x 10-7
3.05 x 10-7
L62 10
L73 x 10-2
sigmold
(A 0)
(A
(A 0)
(A
(A O)
(A 0]
1,57 10-7
6.01 10-*
1.42 x 10
1,66 10-
1.28 x 10"6
1.82 x 10"
Fourler
(A =A=00 0)
(A=
(A=0)
(A=
(A 0)
(A 1 10-1)
GFT-r
signoid
2.04 x 10
2,26 x 10"
4.27 x 10
5.41 x 10
1.84 10
1,60 x 10-2
(A=00 0)
(A=
(A= 0)
(>=0)
(A 0)
(A= 0)
Table 6: Comparison for function approximation with: We report the MSE over the test set averaged 5 runs. The table contains the same experiments as Table for the two ablations in Appendix B The deployed À is indicated below each result.
Mothod
Fumetion fr
Function fz
Function fs
Method
(d. - (5,300)
(4.M) - (10,500)
(4.M) - (5, 500)
(d. - (10, 1000)
(d. (0.)
(d. (10,
1.41 x 10-3
1.06 x 10-8
4.01 x 10-3
3.73 x 10-2
4.89 10"
5,70 10
Fully Sampled
Fourier
() 1 x 10-1)
(A- 1 x 10-1)
() 1 x 10-*)
(A x 10-1)
(A 0]
(A 10-1)
1.03 x 1
7,56 10-9
7,01 10
4.16 10-9
1,38 10"2
7,83 10
Noiay gradients
Fourier
(A=
[A= 1 x 10-2)
(x=0)
(A= 1 x 10-2)
1 x00-0))))
(A 10-1)
B Further Ablations
First, to demonstrate that using the optimal output weights b(w) from Section 3.1 is necessary for GFT, we run the following experiment: Instead of sampling only features my from generative model Pa = Go with Go: Rd Rd and then computing b(w), sample pairs (w,,bi) by) from Pw,b Go R for Gg: Rd42 + Rd x C and the d+ 2 dimensional standard normal distribution 7j We use the Fourier activation and the same setup as for Table [] The results in the first row of Table g are significantly worse than for GFT and GFT-r in Table 0 This is not surprising since now each output weight by only depends on its corresponding feature wy instead of all (w)) as for the optimal b(w). In particular, b(w) cannot be learned by a gradient- based algorithms since in each update the w = (wn) are resampled. Thus, there no persisting correspondence of sampled features with specific output weights.
Second, test whether the neural network line in Table can be improved by introducing gradient noise. This technique can help to escape local minima. To this end, we optimize the same loss function as for the 2-layer neural network in Table [ but use the noisy stochastic gradient descent 0x+1 = 6, - LVL(() + aZ with random Z ~ N0, I), step size P, and noise strength a = 0.1p. The obtained results in the second row of Table 0 do not improve upon the baseline from Table
C Implementation Details
We optimize the loss functions for GFT and for the feature refinement with the Adam optimizer using a learning rate of 10-1 for 40000 steps. The regularization for solving the least squares problem f is
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
set to c 1 10-3. For the neural network optimization, use the Adam optimizer with a learning rate of x 10-3 for 100000 steps. In all cases, we discretize the spatial integral for the regularization term in (10) by 1000 samples. For the kernel ridge regression, we use Gauss kernel with its parameter chosen by the median rule. That is, we set it to the median distance two points in the dataset. The PyTorch implementation
corresponding to our experiments is available https: tuub. httpg/tthu..coooooeeeeeeeeee hwtp:jjeerrrrccgggeerrreeee training.
feature
15


Reviews:
[{"review_id": "kQYq6SZ999", "reviewer": "Reviewer_s5U9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths:\n  * Paper is written clearly and easy to follow, especially the narration of problem reduction, core methodology and presentation of results.\n  * The method proposed is novel, not overcomplicated to understand, and numerically proved to learn thin 2-layer networks efficiently over common optimization issues and standard learning procedures. It also considers noisy data.\n\n* Weaknesses:\n  * Lacks theoretical guarantee on the accuracy of the algorithm, e.g. convergence, generalization error, probably approximately correctness including sample complexity, etc. More specifically, it could be something like error on $w$ estimation being propagated to $b$ estimation error and thus being propagated to $f$ estimation error. The learning of $w$ is through a generative neural net which makes it harder to analyze theoretically, but could we make simplification assumptions on this component or use existing theories to make the analysis feasible?\n  * Method seems to target small $N$ (number of hidden layers) only, but in practice $N$ would not be that small ideally.\n  * See Requested Changes for minor issues and low-level feedback.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a novel, simple and efficient learning paradigm, named Generative Feature Learning, that learns thin 2-layer neural nets with nonlinear activation that has nonconvex MSE loss. It first reduces the problem to only needing to learn layer 1 weights, by finding out the deterministic relation between layer 1 and layer 2 weights through linear regression. Then, it learns layer 1 weights via learning a generative network that generate layer 1 weights. Through numerical experiments, this learning paradigm is proved to overcome the local minima issue of direct loss optimization, and outperform baselines and other standard training paradigms, on elementary function estimation and UCI datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The very general form of the problem (Equation 1) is only presented at the very beginning of the Introduction, which seems unclear in purpose. Maybe consider discussing the applicability and extensibility of the proposed algorithm to this general form of the problem.\n* Typo: in Section 3, ii) 2-Layer Neural Network: $W_lin R^{d+1}$ -> $W_l \\in R^{d+1}$.\n* The part of solving Equation 6 (exact form of Tikhonov regularization) is discussed a bit too briefly. The problem is less than full rank and the specific solution to Equation 6 chosen may need some clarification even though it’s already built in AD packages.\n* Question: What will happen if $z$ is not iid sampled but with some covariance?\n* Nit: Notation $\\bigotimes N$ lacks definition. Is it common to denote iid $N$ times?\n* Nit: UCI database lacks citation.\n* Initialization of $z$ in $F(z)$ in the methodology may need more clarification.\n* Nit: Bolding/Highlighting the best performance in the tables would be nice for a more clear presentation.", "broader_impact_concerns": "N/A to this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j1CBhaUCfP", "reviewer": "Reviewer_kQLY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed approach, Generative Feature Training (GFT), addresses a fundamental challenge in machine learning (i.e., difficulty of optimizing non-convex landscapes), which could have wide-ranging applications / implications in the field.\n2. GFT is, to my knowledge, a novel contribution, dividing the problem into learning an initial proposal distribution and then refining it. \n3. The proposed approach generally outperforms existing sparse feature methods for function approximation, including random feature models and neural networks with small hidden dimension trained with backpropagation. \n\nWeaknesses:\n1. While the paper does a good job providing a wide range of related work, it would be helpful if the authors could more explicitly situate their contribution / ideas relative to the related work, e.g., problems with existing methods that this method addresses / bypasses, key ideas in other works that are shared by this work. The current drat has this discussion to some extent, but a more systematic discussion would help.\n2. It would also be helpful to provide additional motivations and intuitions for various details in the proposed approach. While there is some in the paper, sections 1-3 still seem more procedural than expositional in the current state. Ablations in the experiments would also help in this regard. As another example, figure 1 is quite illustrative in comparing GFT with ANOVA-RFF; and an analogous visual describing how GFT improves over backprop (e.g., potentially exemplar loss curves).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an alternative paradigm for function approximation by 2-layer neural networks with small hidden dimension. Concretely, rather than train all weights in the network simultaneously by backpropagation, the proposed paradigm first casts the optimal output weights as a linear function of given fixed hidden weights and learns a proposal distribution for the hidden weights via gradient-based optimization of a generator network (a feedforward a network applied to sampled latents, allowing for reparametrization gradients). Then, the sampled hidden weights $w$ are updated in a refinement step where both the generator network and sampled latents $z$ (treated as parameters) are updated via backpropagation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EtsZk7Ak1a", "reviewer": "Reviewer_1jSb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is novel and intriguing.\n\nWeaknesses:\n- Typos and unclear notations:\n  - Page 3: $\\in$ became $in$.\n  - Page 3: Does \"AD\" stand for auto-differentiation? This is not explained in the paper.\n  - $L$: $L$ denotes a loss function in Eq.(7). However, $L$ seems to denote something else in Eq.(10). Please make sure that notations are consistent.\n  - GFT-p: According to the paper, \"p\" here stands for \"post-processing\". However, what \"post-processing\" refers to is not clearly explained. Is it the refinement in Section 3.3?\n  - Ground-truth function: Earlier in the paper, the true underlying function we try to approximate used the notation $f$. However, in Section 4.2, the notation became $g$. It should be consistent.\n  - Page 10: \"we computation\".\n\n- Ablation study:\n  - All experiments use $N=100$ features. How does GFT perform when $N$ is changed to some other values? How about traditional back-propagation?\n  - GFT consists of two stages of the training: $\\theta$-optimiation and $w$-finetuning. How much does each stage contribute to the model performance? How does the proposed method perform without the $\\theta$-optimization step?\n\n- All experiments consider small statistical learning datasets. One may argue that these smaller tasks are often handled by models based on convex optimization, such as kernel regression. Therefore, I suggest the authors include additional experiments in at least one of the following two directions:\n  - On the smaller datasets used in the current manuscript, compare with some statistical learning tools such as kernel regression.\n  - Neural networks (even smaller ones) are more commonly used in more complex tasks such as image classification/regression. How does GFT perform on those? Classification tasks include MNIST and CIFAR-10. One possible image regression task to consider is aesthetics score prediction (similar to LAION aesthetics prediction [2]).\n\nQuestions:\n- Since the hidden weights are fixed except the feature fine-tuning phase, how is GFT related to Extreme Learning Machine (ELM) [1]?\n- In each GFT plot in Figure 1, there seem to be some dots in the middle of each plot that are not on either axes, and this only occurs to GFT and GFT-p. Is there any thoughts on why it happens?\n- Can GFT be adapted to train deeper neural networks (e.g., three layers)? Can it be used to train non-regression models?\n- The authors mention that \"the computation of the optimal output layer requires to consider all data points at once, such that we cannot use minibatching\". Why is this the case? Wouldn't it be possible to use a random subset of the dataset to perform each optimization step?\n\n[1] Huang et al. Extreme learning machine: a new learning scheme of feedforward neural networks. \\\n[2] Schuhmann et al. Improved Aesthetic Predictor. https://laion.ai/blog/laion-aesthetics", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes GFT, a novel approach to train narrow and shallow (2-layer) neural networks as function approximators. Specifically, the method first learns a distribution that generates random weights for the hidden layer, parameterized by a deep neural network. Then, it fine-tunes the hidden layer weights. Throughout the process, the output layer weights are optimized in closed form. The authors show that GFT achieves smaller errors when used as function approximators on various datasets compared with existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please add some explanations on why we should be interested in optimizing two-layer thin neural networks. Given that we train a weight generator $G_\\theta$ that is a deep neural network in the process of learning $f_{w, b}$, why don't we just use a deep neural network for our desired task? One scenario I can think of is that $f_{w, b}$ needs to be deployed onto an embedded device with extremely limited system resources.\n- The authors mention that \"For all other hyperparameters, we refer to our code.\" I appreciate open-sourcing the code. However, the paper should also be self-contained, so I suggest the authors include the hyperparameters in the appendix.\n- Please address the weaknesses and questions above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper combines a few ideas into a single method: a generative model of the network's weights, implicit differentiation through a least-squares solver. It's reminiscent of work found in Bayesian neural networks, hypernetworks, meta-learning (like MAML), and backpropagation through fixed-point solvers. The particular approach found in this work is limited to a very restricted class of networks: namely 2 layers, a small number of examples, small number of features, and squared error for the loss, however the paper is up-front about these limitations.

The perspective I find most intriguing is the view that the generative model learns a preconditioner for the neural network learning algorithm. It's not obvious that this should be easier than learning the network itself. For example, the benefits could be due to added noise that is lost when we switch from the mini-batch to the full-batch setting, where naive gradient descent can get stuck in spurious local minima.

It's also unclear the degree to which solving the last layer weights is necessary or complementary to the generative network/preconditioning.

I would therefore suggest two ablations:

1. Train a standard neural network in the full-batch setting with gradient noise, using something like stochastic gradient Langevin dynamics (SGLD) (Welling and Teh, 2011)
2. Train the generative network (sampling z), but learning the output weights b normally instead of solving for their optimal value.

And in general, I think exploring the relative difficulty of learning preconditioners for networks vs learning networks to be a fascinating direction. If we think about a classic smoothing technique like weight decay, this smooths the learning process, but introduces bias in the optimal solution. Preconditioning can be seen as smoothing the optimization landscape itself, which leaves the solution intact. It reminds me of the mystery of distillation: training a large network and then training a small network via distillation from the larger network is somehow easier than training the small network directly.

So overall, despite the limitations of the setting, the small-scale experiments, and limited theory, the paper demonstrates an effective approach and gets at a set of deeper questions that should be of great interest to the community.
Decision: Accept (poster)