FACT: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations
Authors: Enric Boix-Adserà
Neil Rohit Mallinar
James B Simon
Mikhail Belkin
Venue: ICLR 2026 Poster (2026)
Keywords: feature learning
deep learning
neural feature ansatz
convergence
theory
Primary area: learning theory
TL;DR: We prove the Features at Convergence Theorem (FACT), which provides a self-consistent formula that neural networks at convergence must satisfy and serves as an alternative to the empirically conjectured Neural Feature Ansatz (NFA).

Abstract: It is a central challenge in deep learning to understand how neural networks learn representations. A leading approach is the Neural Feature Ansatz (NFA) (Radhakrishnan et al., 2024), a conjectured mechanism for how feature learning occurs. Although the NFA is empirically validated, it is an educated guess and lacks a theoretical basis, and thus it is unclear when it might fail, and how to improve it. In this paper, we take a first-principles approach to understanding why this observation holds, and when it does not. We use first-order optimality conditions to derive the Features at Convergence Theorem (FACT), an alternative to the NFA that (a) obtains greater agreement with learned features at convergence, (b) explains why the NFA holds in most settings, and (c) captures essential feature learning phenomena in neural networks such as grokking behavior in modular arithmetic and phase transitions in learning sparse parities, similarly to the NFA. Thus, our results unify theoretical first-order optimality analyses of neural networks with the empirically-driven NFA literature, and provide a principled alternative that provably and empirically holds at convergence.

FACT: A FIRST-PRINCIPLES ALTERNATIVE TO THE NEU- RAL FEATURE ANSATZ FOR HOW NETWORKS LEARN REPRESENTATIONS
Enric Boix-Adsera" UPenn
Neil Mallinar' UCSD
James B. Simon Imbue, UC Berkeley
Mikhail Belkin UCSD
ABSTRACT
It is central challenge in deep learning to understand how neural networks learn representations. A leading approach is the Neural Feature Ansatz (NFA) (Radhakrishnan et al., 2024), conjectured mechanism for how feature learning occurs. Although the NEA is empirically validated, it is an educated guess and lacks theoretical basís, and thus it is unclear when it might fail, and how improve it. In this paper, we take a first-principles approach to understanding why this observation holds, and when does not. We use first-order optimality conditions derive the Features at Convergence Theorem (FACT), an alternative to the NEA that (a) obtains greater agreement with learned features at convergence, (b) explains why the NFA holds in most settings, and (c) captures essential feature learning phenomena in neural networks such as grokking behavior modular arithmetic and phase transitions in learning sparse parities, similarly to the NFA. Thus, our results unify theoretical first-order optimality analyses neural networks with the aapirically-driverr NFA literature, and provide principled alternative that provably and empirically holds at convergence.
INTRODUCTION
A central aim of deep learning theory is to understand how neural networks learn representations. An empirically-driver conjecture that has recently emerged as to the mechanism driving feature learning in neural networks is the Neural Feature Ansatz (NFA) (Radhakrishnan et al., 2024), which states that, after training, a weight layer W in neural network f(x satisfies the proportionality relation
W W
Here the hight-hand-side an empirical expectation over the training data, and the gradient is computed with respect to the input the network (or with respect to the hidden activations if W is not the first layer) - we review more details on the NFA conjecture later, in Section 2.
This conjecture has been validated in practice networks, convolutional networks, and transformers (Radhakrishnan et al., 2024). Furthermore, growing literature has shown that this NFA conjecture captures and explains several intriguing phenomena of neural network training, including grokking of modular arithmetic (Mallinar et al., 2025), learning of hierarchical staircase functions (Zhu al., 2025), and catapult spikes during training (Zhu et al., 2023), Additionally, when used to power adaptive kernel learning algorithm, achieves state-of-the- performance for monitoring models (Beaglehole et al., 2025), for learning tabular datasets (Radhakrishnan et al., 2024), and for low-rank matrix learning (Radhakrishnan et al., 2025).
range of architectures, including fully-connected
Despite its success, the NFA conjecture lacks first-principles backing for why it should necessarily hold during training. Because this conjecture was derived in an empirical fashion, unclear why ought to hold, whether and under which conditions it may fail, and how to improve it. This motivates the main question studied by this paper:
Is there an alternative to the empirically observed rnbsereelly-observeed Neural Feature Ansatz conjecture, which can be derived from forst principles?

---

We answer this question in the affirmative. Our main contribution is demonstrate connection between the aupirically-observee NFA conjecture and the literature studying first-order optimality conditions that must provably hold if the training process converges.
First-order optimality conditions have previously been shown to imply several phenomena in neural network training; the related work in Section 1.1 for references. Thus, results unify two prominent approaches to studying feature learning (the NFA and first-order optimality). In more detail, our contributions are:
(1) We derive a simple alternative to the NEA conjecture based on first-order optimal- ity conditions. We call this the FACT (Features at Convergence Theorem). This is self-consistency formula that neural networks trained with weight decay must satisfy at convergence; see Section 3.
(2) We empirically demonstrate that our first-principles alternative captures neural net- work feature learning phenomena in many of the same ways that the NFA conjecture does. We show that when FACT (instead of NFA) is used to power an adaptive kernel learn- ing algorithm (Radhakrishnan al., 2024), it also reproduces intriguing feature learning behaviors observed in neural networks such as training phase transitions when learning sparse parities (Barak al.. 2022; Abbe et al., 2023), grokking of modular arithmetic (Nanda et al., 2023; Gromov, 2023), and high performance on tabular data matching the sfate-of-the-add (Radhakrishnan et al., 2024); see Section
(3) We provide a derivation for why the NIAA conjecture usually holds based on first-order optimality. By algebraically expanding the FACT relation, and analyzing the terms, we demonstrate that it is qualitatively similar the conjectured NFA relation. We empirically demonstrate that the two relations are proportional in the case of modular arithmetic. This helps put the NEA conjecture on firm theoretical foundation by connecting it to provable first-order optimality conditions, and elucidates the mystery of why it usually holds; see Section 5.
(4)
We construct degenerate training settings in which the NEA conjecture is provably false but where first-order optimality conditions hold true. We formally prove and experimentally observe that in certain settings the NEA predictions can be nearly uncorrelated to the ground truth, while FACT and any other relations based on firstorrdr-optiimaliyy conditions still hold. This indicates that the latter may provide more accurate relation at convergence; see Section 6 as well the discussion Section 7.
1.1 RELATED LITERATURE
Implications of oirstiaider in neural networks First-order optimality conditions of networks at convergence along with results on KKT conditions that arise with exponentially-tann losses at large training times (Soudry et al., 2018; Ji & Telgarsky, 2019; Lyu & Li, 2019; Ji & Telgarsky, 2020) has been used to show implicit bias of deep architectures towards low rank (Gunasekar et al., 2017; Arora et al., 2019b; Galanti et al., 2022), diagonal networks towards sparsity (Woodworth et al., 2020), of convolutional networks towards Fourier-sparsity (Gunasekar et al., 201 and of fully-connected networks towards algebraic structure when learning modular arithmetic (Mohamadi al., 2023; Morwani al., 2023). First-order optimality also has implications to linear regression with bagging (Stewart et al., 2023), understanding adversarial examples (Frei al., 2024), and neural collapse (Han al., 2021; Kothapalli, 2022: Zangrando al., 2024).
Analyses of training dynamics Another recently prevalent approach to understanding feature learning is to study neural network training dynamics tracking weight evolution to understand how features emerge (Olsson et al., 2022; Edelman et al., 2024; Nichani et al., 2024; Cabannes et al., 2023; 2024; Arous et al., 2021; Abbe et al., 2022: 2023; Kumar al., 2023). While insightful, these analyses are technically challenging and are typically limited to synthetic datasets. In this paper, we pursue an alternative approach: we seek conditions on network weights that are satisfied at the conclusion of training, to gain insight into how the trained network represents the learned function. By focusing on the network state at convergence, we can circumvent many of the difficulties associated with analyzing training dynamics, and the insights directly apply beyond simplified synthetic settings.
2

---

multiply by W
h(x) E R
Wh(x) Rdd
fx;
Figure 1: The model only depends on W through multiplication of activations h(x).
Equivariant NFA Another alternative to the NEA, called the "equivariant NEA" (eNFA), was recently proposed in Ziyin et al. (2025) based analysis of the dynamics of noisy SGD, which is invariant to linear transformations in the loss function. This is distinct from the FACT and we also compare in Section 4.
Relation to literature on representation identifiability The focus of this paper is distinct from the literature on representation identifiability and Independent Component Analysis (ICA), see e-g. (Hyvärinen et al., 2024) and references therein. The identifiabiiity literature is primarily concerned with the inverse problem: determining the sufficient conditions (such as -ou-Gaussiy or the presence of auxiliary variables) under which the true latent generative factors can be uniquely recovered from observed data. In contrast, this paper addresses the structure of representations learned by neural networks.
2 TRAINING SETUP AND BACKGROUND
Training setup We consider the standard training setup, with a model f(:0)) : X'' - R trained on sample-wise loss function f: Re y - R on data points (Ii; 1/4 ())) with T,2 regularization parameter À 0 (that is, with non-zero weight decay). The training loss is (0) L())
$1011 r where (()) : Σm= (([(;;; 0), y.). Here X' and y the input and output domains.
Our FACT applies to any weight matrix parameter W E I xd inside a trained model. The only architectural requirement is that the model only depends W via matrix multiplication of internal activations. See Figure 1. Formally, fixing all parameters but W, there are functions g: h such that for all xr,
f(x; =(((()))))
(2.1)
In this notation,' h is the input to the weight matrix, and Wh is the output. Thus, FACT applies to any layer in neural networks that involves matrix multiplications.
For convenience, introduce the notation denote the gradient of the loss and the value of the model with respect to the input of the layer containing the weight matrix W, at the data point z;:
R (Wh, [n=h(,,) E Rdd and Vafe = Ög(Wh,
I(=h(a)) gdxe
0h.
Oh.
Neural Feature Ansatz. In the above notation, the NFA (Radhakrishnan et al., 2024) posits that the neural feature matrix W W proportional to the influence that the different subspaces of the input have on the output, which is captured by the Average Gradient Outer Product (AGOP) matrix. Namely, there is a power & > 0 such that
W W x (AGOP)*, where AGOP = Y2
(NFA)
Equivariant Neural Feature Ansatz. We will also compare to the eNFA proposed in Ziyin et al. (2025), which states
W W eNFA - (v))
(eNFA)
TI
'More precisely, including the dependence on the parameters other than W, what this means is that we partition the parameters [W, [_]., and f(x; = g(Wh(x;; [-w): a; @-w).

---

3 NEURAL FEATURES SATISFY FACT AT CONVERGENCE
In contrast to the empirically-derived NFA and eNFA, we seek to provide a relation derived from first principles. We proceed from the following simple observation: features h(x) are weighted by their influence on the final loss. This is stated in the following theorem.
critical point of the loss, the
Theorem 3.1 (Features at Convergence Theorem). If the parameters of the model are at a critical point of the loss with respect W, then
W W = FACT
1 nd
Σ (V)E)((())) 2m
(FACT)
Proof. The premise of the theorem implies that Vw (x) 0, since W is a subset of the model parameters. By left-multillying by W and using the chain rule, we obtain
0 = =W (V (x))))
-WT(AW VwL(())
1
al(g(h):
) h(xs)
-WT(AW+ +
li=wh(a))
7
Oh
WW W +
(VLL))((2))
The theorem follows by rearranging and dividing by A.
Theorem 3.1 is a straightforward modification of the stationarity conditions. Nevertheless, argue in the remainder of this paper, it is a useful quantity to consider when studying feature learning in neural networks, and it especially fruitful when viewed as first-principles counterpart the NFA. Before proceeding with applications of (FACT), we provide a few remarks and empirical validation.
Remark 3.2 (Symmetrizations of FACT). While W T W is p.s.d., the quantity FACT is only guaranteed to be p.s.d. at critical points of the loss. This means that we can get several other identities at convergence by algebraically modifying the right-hand side in a way that preserves the symmetries of the left-hand side. For instance, since W W = (W WTT, we may conclude that at the critical points of the loss W T FACTT also holds. Similarly, using that W W <WW WW WWT, we
may also conclude that at critical points W W VFACT- FACT also holds.
Remark 3.3 (Empirical validation on real-world data). In Figure 2, we verify FACT on S-layer ReLU MLPs trained until convergence on MNIST (LeCun, 1998) and CIFAR-10 (Krizhevsky et al., 2009) with Mean Squared Error loss and weight decay 10-4. We find that, at convergence, the two sides of the (FACT) relation generally have higher Pearson correlation than those of the (NFA) and (eNFA) relations. For hyperparameter details, see Appendix A.
Remark 3.4 (Backward form). There is also an analogous "backward" version of this equation, (bFACT), derived and empirically validated in Appendix B, that yields information about the left singular vectors of W rather than the right singular vectors. Letting V Vwa& & denote the gradient of the loss with respect to the output of the layer at data point s; we have
WWT = -FACT = nd Σ (WW()
(bFACT)
4 FACT CAPTURES FEATURE LEARNING PHENOMENA IN MANY OF THE SAME WAYS AS THE NFA CONJECTURE
Having validated the FACT, we turn applications. We show that first-principle FACT captures many feature learning phenomena in the same ways that the empirically- driven NFA conjecture has been previously shown do. First, we show that the FACT can be used to design
4

---

MNIST, ReLU MLP, Layers, 1024 Width
CIFAR-10, ReLU MLP, Layers, 3072 Width
1.00
10
0.90
0.85
0.5
0.7
-8- AGOP
I ACOM
0.05
/RGOP
LAGOP
--- FACT
-4- d9S
VRACT MGD
65
VIACT
Lapes.
Figure 2: We train hidden layer ReLU MLPs to interpolation (batch train loss S 10-3) MNIST and CIFAR-IO. We plot Pearson correlation of FACT, AGOP, eNFA (with respect each hidden layer input) to W for that layer. Curves are averaged over 5 independent runs. Both sides of the (FACT) are highly-coreelated at convergence across layers. The power 1/2 for the NFA is suggested by prior work Radhakrishnan et al. (2024); Beaglehole et al. (2023); Mallinar et al. (2025).
learning algorithms that achieve high performance on tabular data based on adapting the recursive feature machine (RFM) algorithm Radhakrishnan et (2024). We also show that this algorithm recovers important feature learning phenomena commonly studied in neural networks, such as phase transitions in sparse parity learning, and grokking of modular arithmetic.
4.1 BACKGROUND: RECURSIVE FEATURE MACHINES
The Recursive Feature Machine (RFM) algorithm (Radhakrishnan et al., 2024) builds upon classical kernel methods (Schölkopf, 2002), which rely on a kernel function K (xx,²') to measure data point similarity (e.g.. Gaussian, Laplace). While kernel methods have been successful, they can be provably less sample-efficien efficieen than alternatives like neural networks that are able to learn features (Abbe et al., 2022: Damian et al., 2022).
To address these limitations, RFM learns a linear transformation W E Rddd and applics a standard kernel K to the transformed data: (w, K K(Wx,, Wx'). This learned W enables RFM to identify salient features, akin to feature learning in a neural network layer (for example, if W range
low rank, its range contains the salient features while the orthogonal complement contains the irrelevant features), Seeking to imitate the feature learning behavior in neural networks, Radhakrishnan et al. (2024) iteratively updates W using a fixed-point iteration satisfy the NFA condition. This is given in Algorithm where the update equation on line 6 is given by
W (AGOP,)*/? where AGOP
Σ
(Vaii 8>0 O. (NFA-RFM update)
Algorithm 1 Recursive Feature Machine (based NEA (Radhakrishnan al., 2024) or FACT
(ours))
1: Input: Training data v). kernel Kw, number of iterations I', ridge-regularization À 2 0
2: Initialize Wo Laxd
3: for t = 0 to T do
4: Run kernel method: at + (Kw,
(K,,X,X) (X,X) nAI)-'y
5: Let f((z) = Kw. (z, X) be the kernel predictor
6: Update Wr, either with (NFA-RFM update) (FACT-RFM update)
7: end for
8: Output: predictor (x)
5

---

FACT-RFM (no geom. averaging)
FACT-RFM (geom. averaging)
Method
NEA-RFM
Kernel regression
Accuracy (%)
85.22
84.99
85.10
83.71
Table I: Average test accuracy over 120 datasets from the UCI corpus Fernandez-Delgude et al. (2014). We compare Laplace kernel regression with adaptively learned Laplace kernels using FACT and NFA, as well as no feature learning.
4.2 FACT-BASED RECURSIVE FEATURE MACHINES
We study RFM with a FACT-based update instead of an NEA-based update. Similarly to the above, let FACT, be the FACT matrix corresponding to iteration We symmetrize in order to ensure that the update is p.s.d. Our FACT-based fixed fixed-point iteration in line 6 of RFM is thus
W1+1 < ((FACT.)(FACT)) ).4
(FACT-RFM update)
We also study variant of this update where average geometrically with the previous iterate to ensure greater stability (which helps for the modular arithmetic task). This geometric averaging variant has the following update
W++1 (FACT,) W.) ).)
(FACT-RFM update")
The exponents in these updates are chosen so that the fixed points these updates coincide with the FACT relation derived for networks at convergence in Theorem 3.1. See Appendix E for more details.
4.3 EXPERIMENTAL RESULTS COMPARING FACT-RFM TO NFA-RFM
We compare FACT-RFM to NFA-RFM across range of settings (tabular datasets, sparse parities, and modular arithmetic).
Tabular datasets. The authors of Radhakrishnan et al. (2024) obtain state-of-the-ar results using NEA-RFM on tabular benchmarks including that of Bernandez-Deladde et al. (2014) which utilizes 121 tabular datasets from the UCI repository. We run their same training and cross- cooss-valddating procedure using FACT-RFM, and report results in Table 1. We find that FACT-RFM obtains roughly the same high accuracy performance as NFA-RFM. Both these feature-learning methods improve over the next-best method found by Radhakrishnan et al. (2024), which kernel regression with the Laplace kernel without any feature learning.
Sparse parities. We train FACT-REM and NFA-RFM on the problem of learning sparse parities and find that both recover low-rank features. The problem of learning sparse parities has attracted attention with respect to feature learning dynamics of neural networks on multi-index models (Edelman et al., 2023;
Abbe et al., 2023).
For training data we sample n points in d-dimensions as x ~ (- +o Ja 1d We experiment with sparsity levels = 2,3, by randomly sampling k < d coordinate indices with which to construct our labels. Labels, y, are obtained from the product of the elements at each of the & coordinates in the corresponding x point and set to be O if the product is negative and if the product is positive. We sample a held-out test set of 1000 points in the same manner.
We use the Mahalanobis Gaussian kernel in both FACT-RFM (with geometric averaging) and NEA- RFM with bandwidth 5 and train for iterations. Our experiments use d 50 and for k: 1.2 we take 500, for A: = 3 we take n 5000, and for A we take = 50000. The results of these experiments are given in Figure We observe that both NFA-RFM and FACT-RFM learn this task and the features learned by both methods are remarkably similar and the support of the sparse parity. Additionally, Figure shows a phase transition learning sparse parities when we take a smaller amount of data 25000, & = 4, which mimies the phase transition when training MLP.
Grokking modular arithmetic. Mallinar et al. (2025) recently showed that NFA-REM exhibits delayed generalization phenomena modular arithmetic tasks, also referred as "grokking". The

---

k- n000
k 3, n5k
k= 4, =50k
Toc: A2c (6)
1.0
2.75
Text AzC
0.75
Aest
0.75
0.5
0.5
0.
terations
NIA-RIM
FACTAUM
Feature Matrices
VAGOP
FACT- 16"
VAGOP
(ACC
VAGOP
FACT M
(AFM Iter 3)
Figure 3: We train FACT-REM and NEA-RFM using the Mahalanobis Gaussian kernel on sparse parity tasks, We train with d = 50, k = 2, 3, 4. The corresponding vAGOP and VFACT M feature matrices are very similar and learn the support of the sparse parity.
FACT Recursive Feature Machine (AFM)
FACT
Hidden Layer ReLU MLP
2.0
05
20
30
60
70
RFM berations
Epochs
Tran
Test
Figure 4: In the lower data regimes of n 25000, h = 4. and d 50, for sparse parity, the FACT-RFM algorithm reproduces phase transitions found in training neural networks.
VAGOP
FACT
NFA RFM
FACT RFM
1.5
1.0
0.7
0.5
0.0-
20
40
300
20
93
300
122
1/5
Figure 5: We train FACT-RFM and NFA-RFM on (r + y) mod 61 for 75 iterations. Both methods achieve 100% test accuracy and exhibit delayed generalization aligned the "grokking" phenomenon. We plot the square root of FACT M and AGOP and find that both methods learn block circulant feature transforms.
7

---

authors find that the square root of AGOP learns block circulant feature transformations on these problems. We train FACT-RFM (with geometric averaging) on the same modular arithmetic tasks and observe the same behavior. Figure 5 shows the square root of AGOP and FACT .M" after achieving 100% test accuracy on modular addition with modulus = 61 when training on 50% of the data and testing the other half. The feature matrices show block circulant structures.
COMPARISON OF NFA AND FACT FOR INNERPPRODUS KERNELS
Having demonstrated that the first-rrinciples FACT obtains many of the same feature learning phenomena as the impirically-coojectuuee NFA, it is natural to ask: is there a direct connection between these two relations? Does the FACT imply the NFA?
Our findings in this section suggest there is such a connection: the updates of NFA-REM are proxies also be viewed as attempting
for the updates of FACT-RFM. Thus, the NEA-RFM algorithm minimize the loss of the kernel method, regularized by the norm of the weights [W]| A similar claim was previously made in Gan Poggio (2024), but the theoretical evidence provided was limited to the dynamics with one sample. Our analysis applies to training with more than one sample.
We restrict our analysis inner-product kernels. The expressions for FACT and AGOP simplify considerably, as stated below. Below, we let a be first-order optimal dual weights for kernel regression with X-ridge regularization computed in the RFM algorithm.
Proposition 5.1 (Comparison of FACT and AGOP for inner-product kernels). Suppose the kernel is an inner-produc kernel of the form Kw (It, x²) = k(x" Md'), where M = W W. Then, we may write the AGOP and the FACT matrices explicitly as:
AGOP Σ
T( xz) Mæ,a, azz, M
f,1=
FACT M =
Σ
(a Mx,) MI,a asj, M
where r(,, M, =
1 n EL_ 1' (xT Ma.) (x),
The proof is deferred to Appendix F. The proposition reveals that the matrix FACT -M is positive semi-definite when the function k is non-increasing (a condition satisfied by common choices like k(t) = exp(() or k(() = 12). This property allows for a simplification of (FACT-RFM update"), ) which can be rewritten as a geometric average between the current feature matrix and the FACT term:
M (FACT,M))
(FACT-REM update' for inner- anorr-product kernels)
This form should be compared with the NFA-RFM update, which also simplifies for inner-product kernels. We write the simplified form of the update below when the power is set to 1/2:
Me +- (AGOP)1/2.
(NFA-RFM update for inner-product kernels)
Notably, Proposition 5.1 also reveals that both updates share the same structural form. The difference lies in the spe- cific factors involved: for the NFA update and k' for the FACT update. Interestingly, both of these factors, k k(xx, Mæ,) and T(z;, M,),, can be interpreted as measures of similar- ity between the data points 2; and Tj. These measures in- crease when the transformed representations Wx; and Way xy closer in the feature space, and decrease otherwise.
VS k' at final iteration of RACT-RFM
Best fit line, A² = 0.987
1.5
1.0-
05
-0.02
3.00
0.02
0.04
5.08
0.08
Consequently, the similarity measures 7 and were ap- proximately equal for most pairs of data points, this would explain the observed similarities in performance between the NEA-RFM and FACT-RFM methods, and account for their general agreement in tracking the feature learning process as occurs in neural networks.
T(K,
M,
Figure 6: Validation of explanation for why AGOP and FACT are sim- ilar when FACT-RFM converges in the modular arithmetic task. Each point corresponds to pair (I,, 2zy) we subsample 1000 points for visu- alization purposes.
8

---

Empirical validation. We empirically validate the above planation, showing that indeed 5 f(,, Ij, .f) is approximately proportional to (x; Mæ,) for FACT-RFM in the challenging setting of arithmetic modulo p = 61 (where demonstrated in Section 4.3 both algorithms converge to similar features).
In Figure 6, we show that best-fit line proportionally relating the two quantities achieves good fit.
NFA AND FACT MAY BE UNCORRELATED IN WORST-CASE SETTINGS
Finally, as counterpoint the analysis in the previous section, we show that when the data distribution is chosen adversarially, NEA and FACT can differ drastically even for shallow, two-layer nonlinear networks. Thus, FACT is perhaps preferable alternative to the NFA.
We craft a dataset to maximimize their disagreement on trained two-layer architecture f(; a, W) (o x) with quadratic activation (t)) f² and parameters a E RRm W E Rmxd and any large enough width m 2 7. For any p E (0, 1) and r E (0, 1), define the data distribution D(p T) over (ar, y) such that x: is drawn from a mixture of distributions: x Unif 1, 2} with probability and such that y = f. (z) = 22122 + 2334 E R. For appropriate choices of the hyperparameters, we show that the NEA prediction can be nearly uncorrelated with weights that minimize the loss, while the FACT provably holds.
and x (1, 1, O, 0) with probability
Theorem 6.1 (Separation between NFA and FACT in two-layer networks). Fix any s > 0. For any e (0, 1), there are hyperparameters Pe: Ae 1) such that any parameters (a, W) minimizing Lx, (0) on data distribution (pe, Te) are uecorl-uneoreelatedd with the (NFA) prediction:
corr corr((AGOr (AGOP) W WT) <,,
where the correlation corr is defined as corr(A, BB) = (A, B) [1 In contrast, (FACT) holds because the weights are at a stationary point.
Proof intuition. At a loss minimizer, the neural network approximates the true function f. because part of the data distribution is drawn from the uniform distribution. Therefore, since the neural network computes quadratic because it has quadratic activations, one can show s re(e1 + ee) + e2) + O For small pe, this
AGOP s EE-D(p, r.) (V2ff)(V matrix has most of its in the first two rows and first columns.
On the other hand, the weight decay in training the neural network means that at convergence the norm of the network weights is min- imized given the function computes. Since the neural network approximates the true function S., in order to minimize the total norm of the weights, W W must have most of its mass on the last two rows and columns when T, is small. This is in contrast to AGOP, since as we have argued that has most of its mass on the first two rows and columns. Thus, the NFA prediction is not met. On the other hand, the FACT prediction is provably met by Theorem 3.1. The formal proof Appendix D.
WTW
FACT
AGOP
Figure 7: The FACT and NEA are uncorrelated at conver- gence on the synthetic dataset.
The construction is empirically validated in Figure 7, which is the result of training a width-10 network for 10 iterations of Adam with learning rate 0.01 on the population loss with 7 0.02, 10-5, A = 10-5, At convergence, FACT achieves 0.994 cosine similarity with W T W, while AGOP achieves < 0.068 cosine similarity.
DISCUSSION
This work pursues a first principles approach to understanding feature learning by deriving a condition that must hold in neural networks at critical points of the train loss. Perhaps the most striking aspect of our results is that FACT is based only on local optimality conditions of the loss. Nevertheless, in Section 4.3 show that when used to drive the RFM algorithm, FACT recovers interesting global behaviors of neural networks: including high-pen geghhrerfoming feature learning for tabular dataset tasks, and grokking and phase transition behaviors arithmetic and sparse parities datasets.

---

The usefulness of FACT is especially surprising since there is reason for FACT to be correlated to neural feature matrices during most training, prior to interpolating the train loss; and indeed FACT
does have low correlation for most epochs (although VFACT FACT has nontrivial correlation), before sharply increasing to near- near-perfect correlation; see Figure 8. This is potential limitation to using FACT to understand the evolution features during training, rather than in the terminal phase. Therefore, it is of interest to theoretically derive quantity with more stable correlation over training.
Layer
Layer
Layer
1.0-
Doarer Whie
0-8-
0.0
D.C-
0.2
325
150
200
350
250
103
150
200
Epech
Tpech
Sport
FPACT-FACT
VeNFA
VAGOP
Figure 8: We train 5 layer ReLU MLPs to interpolation on CIFAR- 10 and plot Pearson correlation vs. epochs comparing FACT, AGOP, eNFA to neural feature matrices for the first three layers of the model. Curves are averaged over five independent runs.
An additional limitation is that there are data distributions, such as sparse parity, where FACT-RFM becomes unstable if continued iterations are performed after convergence, so early stopping is necessary. Understanding this phenomenon may help derive relations that improve over FACT.
Finally, formulation of FACT for neural networks requires non-zero weight decay. This is reasonable assumption for real-world neural network training (LLMs are often pretrained with a reasonably large weight decay factor), but raises othe question of whether it is possible to compute FACT the zero weight-decay limit. In Section 5, we formulate FACT in way that only relies on the kernel dual weights, the learned features, and the data. Therefore it may be possible to compute FACT in neural networks through the network's 's empirical neural tangent kemne (Jacot et al., 2018; Long, 2021), which would allow using FACT without requiring weight decay.
REFERENCES
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-stairease property: necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp. 4782 4887. PMLR, 2022.
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-sadee dynamics. In The Thirty Sixth Annual Conference Learning Theory, pp. 2552-2623. PMLR, 2023.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparamettrizzaton.. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 244-253, PMLR, 10-15 Jul 2018.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. ICLR, 2019a.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019b.
Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent non-convex losses from high- high-dimensional inference. Journal of Machine Learning Research, 22 1106):1-51, 2021.
10

---

Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit, Advances in Neural Information Processing Systems, 1775072577774 2022.
Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Mechanism of feature learning in convolutional neural networks. arXiv preprini arXiv:2309.000 005700 2023.
Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, and Mikhail Belkin. Ag- gregate and conquer: detecting and steering Ilm concepts by combining nonlinear predictors over multiple layers. arXiv preprint urX7:2250..03708, 2025.
Vivien Cabannes, Elvis Dohmatob, and Alberto Bietti. Scaling laws for associative memories. arXiv preprint arXiv:2310.02984, 2023.
Vivien Cabannes, Berfin Simsek, and Alberto Bietti. Learning associative memories with gradient descent. arXiv preprint arXiv:2402. 18724, 2024.
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference Learning Theory, pp- 5413-5452 PMLR, 2022.
Chandler Davis and William Morton Kahan. The rotation of eigenvectors by perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1446 1970.
Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Pareto frontiers in neural feature learning: Data, compute, width, and luck. NeurlPS, 2023.
Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains. arXiv preprint J 1004, 2024.
Manuel Fernandez- Delgaddo, E. Cernadas, S. Barro, and Dinani Amorim. Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research, 2014.
Spencer Frei, Gal Vardi, Peter Bartlett, and Nati Srebro. The double-edged sword of implicit bias: Generalization vs. robustness in relu networks. Advances in Neural Information Processing Systems, 36, 2024.
Tomer Galanti, Zachary Siegel, Aparna Gupte, and Tomaso Poggio. Sgd and weight decay secretly minimize the rank of your neural network. arXiv preprint arXiv:2206.05794 2022.
Yulu Gan and Tomaso Poggio. For hyperbl's agop is greedy approximation gradient descent. Technical report, Center for Brains, Minds and Machines (CBMM), 2024.
Andrey Gromov. Grokking modular arithmetic. arXiv preprint arXiv:2301.02679, 2023. URL http://wwwwgg////.1....o
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018.
XY Han, Vardan Papyan, and David Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. arXiv preprint arXiv:2 106.0000..7. 2021.
Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equatie models: from linear to nonlinear. Annals of the Institute of Statistical Mathe- matics, 76( 16:1:)333,,. 2024.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2018

---

Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent nonseparable data. In Conference on learning theory, pp. 1772-1798. PMLR, 2019.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176-17186, 2020.
Vignesh Kothapalli. Neural collapse: A review modelling principles and generalization. arXiv preprint arXiv:2206.0404] 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Tanishq Kumar, Blake Bordelon, Samuel J Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. arXiv preprint urXiv:2310.06/ 2023.
Yann LeCun. The mnist database of handwritten digits. hllp://yann. lecun. com/exdb/mmiss// 1998.
Philip M Long. Properties of the after kernel. arXiv preprint arXiv:2105.1058; 2021.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint orXiv:1906.055 2019.
Neil Mallinar, Daniel Beaglehole, Libin Zhu, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Emergence in non-neural models: grokking modular arithmetic via average gradient outer product. ICML, 2025.
Pierre Marion and Lénaic Chizat. Deep linear networks for regression are implicitly regularized towards flat minima. NeurIPS, 2024.
Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, and Danica Sutherland. Grokking modular arithmetic can be explained by margin maximization. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023.
Depen Morwani, Benjamin Edelman, Costin-Andrei Oncescu, Rosie Zhao and Sham Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. arXiv preprint arXiv:2311.07563 2023.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
Eshaan Nichani, Alex Damian, and Jason Lee. How transformers learn causal structure with gradient descent. arXiv preprint arXiv: 14735, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 1895, 2022.
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science, 383(6690):146)) 1461-1467, 2024.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Dmitriy Drusvyatskiy. Linear recursive feature machines provably recover low-rank matrices. Proceedings of the National Academy of Sciences, 122(13))e2411125522,, 2025.
Yu Rotfel'd. The singular numbers of the sum of completely continuous operators. In Spectral Theory, pp. 73-78. Springer, 1969.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint, 2014.
Schölkopf. Learning with kernels: support vector machines, regularization, optimization, and beyond, 2002.
12

---

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression classification: Influence of task formulation on neural network features. In International Conference on Artificial Intelligence and Statistics, pp. 1556--1588 PMLR, 2023.
Robert Thompson. Convex and concave functions of singular values of matrix sums. Pacific Journal of Mathematics, 66(1):285-290, 1976.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference Learning Theory, pp. 3635-3673. PMLR, 2020.
Emanuele Zangrando, Piero Deidda, Simone Brugiapaglia, Nicola Guglielmi, and Francesco Tudisco. Neural rank collapse: Weight decay and small within-class variability yield low-rank bias, arXiv preprint MXXv:22402.03911,, 2024.
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, and Mikhail Belkin. Catapults in sgd: spikes in the training loss and their impact generalization through feature learning, arXiv preprint arXiv:2306. 04815, 2023.
Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy, and Maryam Fazel. Iteratively reweighted kernel machines efficiently learn sparse functions. arXiv preprint arXiv:2505.08277, 2025.
Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of deep linear network. NeurIPS, 2022.
Liu Ziyin, Isaac Chuang, Tomer Galanti, and Tomaso Poggio. Formation of representations neural networks. ICLR, 2025.
13

---

A HYPERPARAMETERS IN EMPIRICAL VALIDATION OF FACT AND FURTHER EXPERIMENTS
In the empirical validation of FACT in Figure 2, we train the networks until convergence, which operationalize as the point at which batch train loss Σ 10 3 is achieved. The results are for training 5-layer cully-comeetted networks with MleanSSuurre Error (MSE) loss for 200 epochs using SGD, momentum 0.9, initial learning rate 10-1, cosine decay learning rate schedule, weight decay 10-1 batch size 64, depth 3, and hidden width 1024 for MNIST and 3072 for CIFAR-10, and standard PyTorch initialization.
A.1 VARYING THE NUMBER OF DATA POINTS AND THE WEIGHT DECAY
We experiment in the above setting with varying the number of training data points, as well as the weight decay. Table 2 reports the Pearson correlation to W, W after training for the square root of AGOP, the square root of the eNFA (Ziyin et al., 2025), and the FACT. addition we report the correlation to a symmetrization of FACT given that unlike the other quantities the FACT is only guaranteed to be p.s.d. at exact convergence and we only train to approximate convergence with batch train loss < 10-3, In these experiments, the eNFA and NFA predictions without square root are both less correlated to the W, W1 than their square root counterparts. In all cases except for MNIST 1000 we find that FACT and its symmetrization are better correlated to W, W1 than both AGOP and eNFA and their square roots.
Pearson Convelation to TW after training
W
MNIST
CIFAR- (n 50K)
7 == 1K
n
2K
n
" 60K
wt decay = 10-3
wt decay = 10
FACT
0.890
0.988
0.995
0.999
0.830
0.896
FACT- FACT
0,956
0.998
1.000
1,000
0.929
0.943
AGOP (NFA with 8 1/2)
0.899
0.923
0.924
0.793
0.722
VeNFA
0,903
0.923
0.917
0.903
0.797
0.714
Table 2: We report the correlation of the Neural Feature Matrix W W to corresponding feature matrices interpolation, given by the square root of AGOP, the equivariant NEA, the FACT and symmetrization of FACT. Here W is the first layer of the network. The weight decay for MNIST- trained networks is 10 but experiment with weight decays 10 -3 and 10 for networks. For MNIST, we experiment with changing the training set size as well.
B BACKWARD FORM OF FACT
We provide here an analogous "backward" form of the FACT condition, which applies to W WWT instead of W.
Recall from Section 2 that the neural network depends on W as
f(x) 9(Wh(z),x).
Out of convenience, we introduce notation to denote the gradient of the loss with respect to the output of the layer at data point z;. We write
V Vwn& == (f(g(h,x):00 x):
lā=wh(z,) Rd.
0h
With this notation in hand, the backward form of the FACT, which gives information about the left singular vectors, is: Theorem B.I. If the parameters of the network are at a differentiable, critical point of the loss with respect to W. then
WW = -FACT == nd Σ (Wh(z))
(bFACT)
14

---

MNIST, ReLU MLP, Layers, Width
CIFAR-10, MLP, Layers, Width
6--
-I-
MAST
-T- beliT
02-
Layer
Figure We train hidden layer ReLU MLPss to interpolation on MNIST and CIFAR-10 10. We plot Pearson correlation of the backward versions of FACT, AGOP, eNFA (with respect to pre-activation outputs layer) and compare WW for that layer. Curves are averaged over independent runs.
Layer
Layer
Layer 3
1.0
Pearsnn commattionn to MOV
0.0
0.
Pearsmn comeatinn to
0.-
06-
04-
0.0
00-
100
150
200
30
100
150
200
50
1000
150
200
Epoch
Epoch
Epoch
bFACT
DelIFA
VDAGOP
Figure 10: We train 5 hidden layer ReLU MLPs to interpolation on CIFAR-10. We plot the Pearson correlation of the backward versions of FACT, AGOP, eNFA (with respect to pre-activation outputs of layer) vs. epochs. Curves are averaged over 5 independent runs.
Proof. Left-multipliing by W, we obtain
CV(VCCx())
W(AW + (())
W(WW 1 a n
al(g(h):us)
li-wa(s)
TT
ah
AWWT (Wh(z)))V
Rearranging and dividing by À proves (bFACT).
Again, may symmetrize both sides of the equation and still get valid equations that hold at critical points of the loss: for instance we have WWT bBACT -FACTT.
In the way that we compute bFACT, compute analogous "backward" version of AGOP which given by,
bAGOP Σ (Vwn.))
and consider whether this models the left singular vectors of layer weights as well. The backward eNFA is as computed in (Ziyin et al., 2025). Figures 9 and show the complete set of comparisons for backward versions of FACT, AGOP, eNFA to their respective neural feature matrices compared across both depth and epochs. The hyperparameters and training setup are the same as that described in Appendix A.
15

---

C CASE STUDY FOR DEEP LINEAR NETWORKS
In this appendix, we compare the predictions of FACT and NEA in the toy setting of deep linear networks, which have received significant attention the theoretical literature as a simplified setting for studying training dynamics Arora et al. (2019a); Ziyin et al. (2022); Marion & Chizat (2024); Saxe al. (2014); Arora et al. (2018). A deep linear network f Rd - R is parameterized as
f(x) = WL WL-1 ... W1x
for W1 Rexd W...... WL-1 E RA WL Rexh We the network on data points (0, Ia) and labels given by ground truth linear transformation fx) = W* where W E Rexd
In this setting, Radhakrishnan et al. (2025) show that the exponent s in (NFA) must scale as 1/L in order for the NFA prediction to be correct. Thus, unlike the FACT, the NFA has a tunable hyperparameter that must depend on the particular architecture involved. We rederive this dependence of the exponent on the architecture for completeness.
Informal derivation of NFA power dependence on depth In this setting, the NFA prediction for the first layer can be computed as
AGOP WT WT- WL WL-1 W1,
So, when training has converged and the network is close to fitting the ground truth W", we have
AGOP = (W*)
is known that weight decay biases the solutions of deep linear networks to be "balanced" at convergence Gunasekar et (2017); Arora et (2019b), meaning that the singular values each layer are equal. When the layers are balanced we should therefore heuristically expect that, after training, we have
w,w, = (WW')
because singular values multiply across the layers. Putting the above equations together, at convergence we have
W. Wi (AGOP)
For L, = 2, recover the prescription of using VAGOP suggested by Radhakrishnan al. (2024); Beaglehole et al. (2023); Mallinar et al. (2025). However when L o 2, this is no longer the best power. Our analysis suggests that the AGOP power must be tuned with the depth of the network on the other hand, FACT does not need this tunable parameter.
We empirically validate this Figure 11, with deep linear networks with d 10, 5, 512 and varying the depth L, and sample W E [ exd with independent standard Gaussian entries. The train dataset is of size
5000 where x; (0, Ia). We train to convergence using the Mean Squared Error (MSE) loss for 5000 epochs with SGD, minibatch size 128, learning rate of 5 x 10-3 weight decay of 10-2, and standard Py PyTccch initialization. After training. the singular values of all of the weight matrices are identical after training, indicating balancedness has been achieved.
D PROOF OF THEOREM 6,1, SEPARATING FACT AND NFA FOR TWO-LAYER NETWORKS
We provide the proof of Theorem 6.1, restating the theorem as Theorem D.1 1 for convenience.
Setup Consider trained two-layer architecture f(x; a, W) a (T((Wx) with quadratic activation
(t) = and parameters a E R m W E Rm For any p E (0, 1) and r E 10, define the data distribution D(p,7 r) over (x, y) such that is drawn from mixture of distributions: [ 1, 2 with probability p and x = (1, 1, O, 0) with probability 1 - p. and such that y = f. = T. 2344 E R.
16

---

Correlation of WTW with NFA for different network depths
0.95
Cosine Similariy to WtW
0.85
NFA-- layers
NPA19, layers
NFA layers
NFA layers
0.80
NFA layers
Figure 11: Deep L-layer linear networks trained to convergence synthetic data. AGOP has cosine similarity close to to the NFM (W1 W1), which validates the derivation in Appendix C. For all of these network depths, FACT has cosine similarity 0.999, and there are no tunable hyperparameters that depend on depth.
Theorem D.1 (Separation between NFA and FACT in two-layer networks; restated Theorem 6. 6).. Fix s to be the NFA power. For any (0, 1), there are hyperparameters Pe, d (0, 1) such that any parameters = (a, W) minimizing Lx, (0) on data distribution D. == D(pes Te are
nearly-umcorrelated with the NEA prediction:
corT((AGO)), W WTW) <c. <c<c,
where corr(A, (), = [ is the correlation. On the other hand, the FACT predic- tion holds.
The outline of the proof that the loss-minimizing weights and AGOP are uncorrelated to first show that there set of weights a, W such that the loss Cx. (a, W) is small. This implies that at any minimizer a", W must also have that Lx, (a", W*) small. In turn, this means that the estimated function f(-) f(;a", W*) must be close the true function (xx T2 + Tz Finally, this will let us compare the AGOP to the loss-minimizing weights a, W.
Proof. Set T. = -
1. Construct weights with low loss. Construct W [w1, [1,.... mm
R and
[ū,,- "mm] E Rm by letting Wi = + Wh = W3 Ez, Wh = + eд. Wy eg.
Wg ea, 0i Te/2, 04 1/2, 02 = 0g Tes a5 = 0g -1, and my
and ūj for all
j 2 7. One can check that fa; a, W) fz) for all x. and that || ||a||2 s 13. Therefore
(a, w) & Lx. (a, W) < 169
(D.1)
2. Conclude that / fa. Define f() f(; W). Since f and fa are homogeneous quadratic functions, we may write them as
f(x) Σ Gyaay and f())= Σ CjjTTj.
15i5i54
1Si5i54
Let us show that the coefficients {cij} must be close to the estimated coefficients {dij} using analiti- calculation. Define the distribution Unif[(O, 1, 1,2]41 and the inner product between pair of functions (g, h/, Ex (x))))()))))))) Also define the functions
4.5,
t ==0
1.5,
3, t=
(t)
(t)
6,
t t 1
X (2) =
-3, t =
(. tE (1,2}'
-1.5
f t= 2
1.5, t= 2
17

---

and for any vector of degrees 10, define Xa : 1,0 + by (a IT= x (xs). These functions have been picked so that for any a' E {0, and monomial had (x)
#1 22 2g we have (ha, Xa 1(a = a'). Therefore, for any 1 v i S S 4, there {0, 1, such that
Gj o= = (S+, Xauuu and dg = (f. Xalu-
Therefore, by Maucch-Schwarz, for any 1 S S j < 4 and corresponding a we have
Cij ارنl = 10. -
- f.xa)ul v II. Illllxall S 6*11 111f--fllz fllux
Now we can apply our previous bound in (D.1), which implies that (x,,)) - - C(ā, W) = 169)e and in turn means that
| =
f"
f.(x)) 1699./pe-
So the estimated coefficients { are close the coefficients (cisleg, i.e. for any 1 < sisss 4,
Jaj- - راy S 17000 Ae/Pe := 6e.
(D.2)
Notice that 6, < 17000. 16 < 1/10 for small enough
3a. Estimate the AGOP of f. Since have shown f R the AGOP of the estimated function
be well approximated as follows.
0f 0f
AGOP(j, B(a.))DD D.
Or dr
0f 0f
0f of ]
=(1 Pe)
(==(1,1,000) +..
Ox dr
Or dx
0f of T Dr de
Since |&yl S |cyl de s 11 8, s 11/10 for all i,j it must hold that
||. S 100 for all
E 1,2 so
0s
|AAGOP(}))) D.)
Ox dx 1==(1.1,000 ||r S 1100pe
(D.3)
Finally, Be (==00,,,0,0) 2èner + 282282 + Ć12(e1 + 62) + (êia + d2a)eg + (014 + 024)e4. and
8f
C11 022 = 613 = 614 = 023 = 224 = and 012 = Te, which means that
0f Zx Or
0f Dr 1==(11000)) -Te(EI + +2))lF S 1000
(== -2 - (2) + e12 & 20
(D.4)
Putting the (D.3) and (D.4) together with the triangle inequality, we conclude our estimate of the AGOP
||AGOP(f. D.) (2 + e2) + T2)Tl
|| S 100p. + 1000
(D.5)
3a. Estimate powers of the AGOP of f. Next, let > 0 be the power of the AGOP that we will take. Let \ 2 2 Aa 2 be the eigenvalues of AGOP, with a corresponding set of orthonormal eigenvectors UI, U1,..., E R.. By Weyl's inequality, since +" 2 100p. + 1000 for small enough
A1 > 2T - (100p, + 1000 0.) Z 6
and
A1 e
18

---

and
0 5 A2 100p. 1000 0, S
Additionally, let P1 be the projection to the orthogonal subspace spanned by {(e) + E2)). By Davis-Kahan (in theorem Davis & Kahan (1970),
| 100p. - 1000
S ?
?
Notice that AGOP = Ei= Xooey which we will use later.
4. Estimate the lossmminimizing weights, Now let us estimate the loss-minimizing weights, á, W. The argument here is split into two parts: want to (a) show that W is small in the first and second columns, and (b) show that W is large in the third or fourth column. These two facts combined will be enough show that W T W is close uncorrelated the AGOP.
4a. Show that W Wmm.1 and W 1m. small, Define weights a', W by letting a' à and W W Wimm3 | In other words, we have zeroed out the coefficients of the variables 21 and in the first layer. Then define
(x) = (a')T ((W")x)).
we write f'(x) Eigisiea = 4, notice that c/1 = c22 = d1a = = = 224 = 0 and
that d31 C34, d33 033, and c41 = Ees. Now, let a", W be weights minimizing ||a"||2 + ||
such that
((() = f(;a", w
By the construction in Lemma D.2, we may assume without loss of generality that all but 4 neurons are nonzero: i.e., that añ -... am = and W = W = 0. Now the difference between
the network after the zeroing out and the current network is f(x) f'(x) Eisisisa Eayayay where
cigl |2;j -gp
d5l S +
So by Lemma D.2, this difference can be represented on four neurons with a cost of at most 100(f. + 2/33 Therefore, by editing the weights a", W we can construct weights a" W such that f()) f(a",, w")) and
|a|l2 ||W11 s ||a"|12 |/"' 1|3
S 1|a"|1 + || + 100(T. + 5,)2/3
| [i.m,lll 1l + 100( 5,)2/3, So we can conclude that the norm of the weights in the first and second column small
|| + / ||W
Wi.m, 1.m, 1 21 + || < 100(T. + 8.) Ze e
(D.6)
4b. Show that at least of Wi or W Wim. is large, Finally, let us show that either the third or fourth column of the weights is large.
0.9 S c34 - 6, 5 < Σ a W..
||||
W (W.WW..)
:=1
<
||]|
(Waa)2
Σ Wia)2
19

---

From the construction of the weights W in the first step of this proof, we know that ||ā|
Σ
|a| + W < 13. So ||ā|| I 4. We conclude that
max(I] |I W 1mm.4 II) 2 1/3.
(D.7)
5. Compare AGOP to loss-minimizing weights. Finally, let us compare the NFA approximation (D.5) to the facts proved in (D.6) and (D.7) about the loss- mos-mmnniizzing weights. From (D.5) and (D.6) and [[ 13 and the calculations in step 3b, we conclude that
((AGOP (.)) W (TW) = Σ Aaee WTw)
(IW 1am, 112 + IIW 1.m 2m)))) + P 11 + +1 + A + 2 20 + c
From (D.5) and step 3b we conclude that || (AAGOP(): Pe))" Z (2T?- 100p 1000 /0)= 2 Te 2 26
From (D.7), we conclude that
|| W 11921.
which implies that
corr(AGOP(j) D.), W WTW) (=2c E 80// S 28 +
which can be taken arbitrarily small by sending e
The Lemma that we used in the proof of this theorem is below.
Lemma D.2 (The minimum-nor norm weight solution for network with quadratic activation). Let f(;; a, W) = a (W))) be a neural network with quadratic activation function ot) = and weights W E mg Inm for m 2 d. Then, for any homogeneous quadratic function ((x)
x Qu, where Q = Q the minimum-norm neural network that represents f has cost:
2 Σ 0,(())2/3
-. = + [[]] f W) f(}),
and this can be achieved with a network that has at most d nonzero neurons.
Proof. We can expand the definition of the quadratic network
f(;; a, W) = a (W > x x
a.w.w
For any a, W such that f(; a, W) = f(),, we must have Q Em a.w.p By (a) inequality (2.1) in Thompson (1976) on concave functions of the singular values of sums of matrices (originally proved in Rotfel'd (1969)), we must have
||a||2 |
a + ||ws112
2 2 Σ
waw. 2/33
t=1
m
m
o (a T//33
i=1 1=1
(a)
σ3 (Q)23233
1=1
And notice that given eigendecomposition (A1, v1), ...,, (As, va) of Q. this be achieved by
letting a; sgn(A.))\,,*? and w; //,, for all 1 < < and a; 0 and w; for all + 1 < < m.
20

---

E DERIVATION AND JUSTIFICATION OF FACT-RFM UPDATE
The simplest fixed-poine point iteration scheme would be to apply
W++1 v FACT,,
(E.1)
aiming for the fixed point
WT. W FACT,
However, this scheme cannot be directly implemented because (E.1) is not necessarily well-defined. In particular, FACT not necessary p.s.d. when the network is not at a critical point of the loss, so the square root of FACT in (E.1) is not well defined.
In order to fix it, the most natural solution is to symmetrize FACT and instead run the scheme
W + FFCTT)
since indeed when FACT, W. We we a fixed point with this update.
We experimented with this update, and found good performance with tabular data (this is "no geometric averaging" method reported in Table 1) and parity data, but for the modular arithmetic problem FACT-RFM with this update was unstable and the method often did not converge especially data regimes with low signal.
In order to obtain more stable update, we chose to geometrically average with the previous iterate, as follows:
W (W, WW,, (F)(FACT))))/A
which again has a fixed point when FACT, = W,T W,. This yielded improved performance with modular arithmetic while retaining performance with tabular data and parities. Additionally, discuss in Section 5, we then discovered that this update has an interpretation as being a close relative of the NEA-RFM update when applied inner product kernel machines.
F PROOFS FOR SECTION 5
We first observe that the updates in FACT-RFM can be written in a convenient form in terms of the dual solution a and the derivatives the estimator. This lemma does not depend on the kernel being an inner-product kernel.
Lemma F.1 (Simplified form of FACT for kernel machines). Let (X.y) training data fit by kernel machine with the MSE loss, and let be first-order optimal coefficients for kernel regression with X-ridge regularization. Then the FACT can be equivalently computed as
② Kw (x, la=zdaa a.
FACT Σ
Ox
The proof is by using known first-order optimality conditions for a.
Let us prove the convenient expression in Lemma F.1 for the FACT matrix for kernel machines, which can be used to simplify the implementation of FACT-based RFM.
Proof. We compute the FACT for the estimator f(xx = En= (w,zzz,)a Substituting the definition of FACT and applying the chain rule, this is
FACT
1
i
(((()) v.)) la-a, I.
On f(x) 1b=z; (( u:)))
nd
dr
1
Kw (z. Ja (f(o.))
nX
Ox
where E R denotes the derivative in the first entry. The proof concludes by noting that a; nx (f(x),))) yi) because of the first-order optimality conditions for a, proved below in Lemma F.2.
21

---

Lemma E.2 (Altemative expression for representer coefficients for kernel regression). Let (X, Y) be training data, and let a = (K(X,X) + A Y for some À >> 0. Also ler ((g. y) = 2 yl Then
1
E' (Us. V:).
0i
nx
where ús = K (x4, 2)o, and the derivative El is in the first coordinate.
Proof.
Notice that 8l (Bss y4) ii -y. So
('r,,
[Ka]i, yi
KKK + ndI) y/s
Yi
=-nA(K '
--ndoG-
Remark F.3, A statement of this form relating the representer coefficients the loss derivatives optimality is more generally true beyond the MSE loss, but we do not need here.
Finally, we can prove Proposition 5.1.
Proposition F.4 (Restatement of Proposition 5.1). Suppose the kernel is inner-product kernel of the form Kw(x, (wxxxx)) = k(x T Mx'), where M W W. Then, we may write the AGOP and FACT matrices explicitly
AGOP = Σ
T(E,, Jy, M)Ma, a M
f==1
FACT M Σ
(x Mæ,) ajo M
where T(r, M, == 1.4 W (x) Mx²).
Proof. The expressions can be derived by plugging the expansion f(x) Σラ= 2(x,2;)a For AGOP, we start from its expression Ansatz (NFA), and obtain
22
AGOP
Kw (x Σ
Kw (2,, (1))))
n Σ
Wxx Mr,) k/ (zj (MMMM,aa)(MMoo)))
f
r(r; M,,;) Ma;a M.T
=
For FACT, we start from the expression Lemma F.I:
① Kw (I, )a} an M
FACT FACT-MT
dr
Σ
(xx Maz)aazaj ana M T.
1==1
22

---

G EXPERIMENTAL RESOURCE REQUIREMENTS
The following timings are for one A40 48GB GPU. The tabular data benchmark experiments in Table take under GPU-hour run. The synthetic benchmark task of Figure 7 on which FACT and NFA are uncorrelated takes under GPU-hour to run. The arithmetic experiments in Figure and 6 take under GPU-hour to run. The ReLU MLP experiments on MNIST and CIFAR- 10 in Figures 2, 8, 9, and 10 take under 50 GPU-hours to rum. The sparse parity experiments in Figure and Figure 3 take under GPU-hour to run. The deep linear network experiments in Figure take under 2 GPU-hours to run. Additionally, debugging code and tuning hyperparameters took under 200 GPU-hours to run.
H
LLM USAGE
LLMs were used only as Al coding assistants and help polish some of the writing the paper, and were not used for research ideation.
23


Reviews:
[{"review_id": "VkWvo7HMMT", "reviewer": "Reviewer_NSpx", "summary": "The paper introduces the Features at Convergence Theorem (FACT), a theoretical alternative to the Neural Feature Ansatz, the idea that a layer’s feature covariance correlates with the average gradient outer product. FACT is derived from first-order optimality conditions that must hold at convergence, and thus is more theoretically grounded. Empirical tests show that FACT reproduces key learning behaviours — including grokking, phase transitions, and sparse parity learning — similar to NFA. FACT matches NFA’s predictions in most settings but remains valid where NFA fails.", "questions": "1. FACT’s correlation with (W^\\top W) is low through most of training and spikes only near interpolation (Figure 8). What intermediate quantity (e.g. a the $\\sqrt{\\text{FACT}\\cdot\\text{FACT}^T} $ you note) best tracks feature emergence earlier, and can you motivate it theoretically rather than empirically?\n2.    Since FACT follows from first-order optimality, can you predict when alignment will occur, not just that it must at convergence? Any falsifiable prediction for the timing of grokking/phase transitions beyond endpoint agreement?\n3. Given that Theorem 3.1 is a stationarity rewrite, what’s the new predictive leverage beyond endpoint consistency, for example,  conditions under which FACT will disagree with NFA before convergence in natural (non-adversarial) settings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clarity and simplicity. The core identity (FACT) follows directly from first‑order optimality with L2 weight decay, relating a layer’s feature Gram $W^\\top W$ to an empirical average of loss‑gradients times activations. It’s architecture‑agnostic under the mild requirement that the model depends on a weight matrix multiplication (Equation 2.1 and Theorem 3.1). This gives a simple target that any trained model with weight decay must satisfy.\n- Forward and backward views. Besides the forward statement for $W^T W$, they derive a backward counterpart for $WW^T$ (bFACT), giving traction on both right and left singular spaces of a layer (Remark 3.4).\n- Empirical alignment at convergence across realistic vision setups. In 5‑layer ReLU MLPs on MNIST/CIFAR‑10, Pearson correlation between the two sides of FACT and the learned $W^T W$ rises to $\\sim1$ at convergence (Figure 2), often exceeding NFA and eNFA. This makes FACT a strong descriptor of terminal‑phase features in practice.", "weaknesses": "- FACT's correlation with $W^T W$ is typically low during most of training and spikes only near interpolation (Figure 8). This makes FACT relatively uninformative about how features emerge, arguably the purpose of studying these quantities.\n- Mathematical novelty is modest. The main theorem is essentially the stationarity condition written in feature‑centric form.\nApplies to the terminal phase only.\nKernel analysis is clarifying but incremental. The inner‑product kernel derivations neatly align the two updates $\\tau$ vs $k'$, but the algebra is standard once you adopt the kernelised view. It explains why NFA often works without yielding new predictive dynamics (Section 5).\n- Breadth of architectures/losses is limited. Core validations focus on MLPs with MSE; connections to cross‑entropy (with weight decay), convs/transformers, and decoupled weight decay (AdamW) are not systematically explored. eNFA is included but comparisons remain relatively narrow (Sections 3-4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kjJUk3dWKb", "reviewer": "Reviewer_kUP6", "summary": "This paper derives an alternative to the Neural Feature Ansatz by investigating first-order optimality conditions of a loss function with L2 weight penalty. The alternative, FACT, performs similarly to the NFA in applications, but correlates better to solutions found in neural networks. The paper further shows a synthetic dataset for which NFA predictions fail completely while FACT predictions remain accurate. Overall these results show that FACT is a useful and theoretically principled alternative to NFA.", "questions": "Are there any intuitions for why NFA correlates well with weights throughout learning but FACT does not?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Derives an expression, FACT from first order conditions which is reminiscent of the NFA, thereby providing a theoretical basis for the method (the justification of NFA by contrast being empirical). This is a great contribution.\n\nThe experiments show that FACT behaves similarly to NFA in applications to tabular data, sparse parity, and grokking. These results show that FACT is approximately as useful as NFA.\n\nThe result showing that NFA predictions can break down while FACT predictions remain is particularly nice in highlighting the contribution of the work.\n\nOverall the paper is clear and easy to read, with well made figures.", "weaknesses": "While the proposed FACT method may have a theoretical justification, it does not clearly improve on the NFA method in the applications considered here. The paper could be strengthened by including a case where FACT outperforms NFA on more than a synthetic task. But I do not view this as an essential addition to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G8u3yADJhn", "reviewer": "Reviewer_Z58e", "summary": "The authors propose a principled derivation of first order optimality conditions of neural network matrices at convergence. They show that this explains many of the same phenomena as neural feature ansatz.", "questions": "Please take all questions below as call for action to make your paper more clear:\n\nWhy is eNFA called equivariant? Please explain in paper.\n\nOn the first page, NFA is the gradient with respect to inputs, on page 3 it is w.r.t. to hidden activations. Which of the two is it?\n\n206: how sensitive is this to different values of weight decay?\n\n207: what correlation are you measuring? since NFA and AGOP are only proportional I would expect Spearman. However, for FACT, it is an equality so I would expect Pearson or even variance explained; Also, why look at sqrt? This can be avoided with rank correlation.\n\nfig5: how exactly does NFA or FACT explain grokking? or is this just an observation about behavior of the system? fig 8 shows that FACT only holds at convergence… what do we learn here?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper presents a rigorous and principled theoretical derivation for previously empirically observed (or conjectured) phenomena around neural feature ansatz.\n\nThey prove their key theorem which gives a nice intuition for why this occurs.\n\nThey provide solid experiments that support the theory.\n\nThey have an insightful limitations figure that shows how their theory only holds at convergence (as opposed to NFA)", "weaknesses": "There is a whole field of literature on identifiability and representation learning (see here for a start: https://link.springer.com/article/10.1007/s10463-023-00884-4) that studies the representations learned by neural networks at convergence. I expect the authors to add those references and discuss the relation of their work to that field.\n\nFig2 caption: define what it means to train until 'interpolation'\n\nlines 198-200: unclear argument, the following sentences are just algebraic manipulations\n\nMany of the experimental setups read like: 'we found it works for these hyperparameters' – it would be great to see some ablation studies, e.g., how do think still/no longer work with different learning rate, dimensionality, weight decay etc.\n\n236: 'akin to feature leaning in neural networks' – I think this is a key overstatement (also in the NFA literature), all that is happening here is learned linear filtering on the inputs. This is very different from *nonlinear* feature learning in neural networks. Please tone this down.\n\n262 please write out FACT again below this equation, I think that would show how this is almost tautologically true, that FACT-RFM performs better because it just looks like a gradient step of W w.r.t. to the loss", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q7CXLWVwtU", "reviewer": "Reviewer_f429", "summary": "The authors present FACT, a self-consistent condition for L2-regularized neural networks. This condition provides a more theoretically grounded alternative to the NFA conjecture, establishing a connection to (locally) minimum-norm solutions. The authors demonstrate both empirically and theoretically that FACT aligns more closely with the behavior of trained fully connected networks than existing NFA-based approaches.\n\nOn the empirical side, the authors compare FACT-RFM with NFA-RFM, showing that the two exhibit similar phenomena, such as phase transitions and grokking, across various tasks. Theoretically, they prove a case, under a minimum-norm objective, where the NFA conjecture fails while FACT continues to hold.", "questions": "How different is the minimum-norm solution from the solution obtained by FACT-RFM in practice? Are they generally equivalent, or are there theoretical or empirical cases the authors are aware of where they diverge?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main contribution of this work is to extend the empirically motivated NFA conjecture to a more theoretically grounded framework based on (local) minimum-norm conditions. This perspective helps bridge the gap between the NFA line of research and the broader literature on L2-regularized solutions, highlighting RFM methods as an interesting alternative to gradient-based training.\n\nThe paper is clearly written, and the experiments are illustrative. To the best of my knowledge, the proofs are correct. Overall, this is a solid submission with only minor weaknesses.", "weaknesses": "The paper could benefit from a stronger early narrative explaining why FACT is a meaningful measure — specifically, how its foundation in (local) minimum-norm behavior motivates the framework. Although Sections 5 and 6 provide this perspective more clearly, the earlier parts of the paper introduce it less directly. For instance, establishing the connection earlier would help clarify how the low-rank bias and neural collapse phenomena [1] in Figure 2 relate to L2 regularization [2-3], and likewise how the grokking behavior in Figure 5 connects to L2-driven dynamics [4].\n\nAs a minor comment, the acronym “FACT” has appeared previously in the literature [5]. In addition, the name emphasizes self-consistency at convergence but does not reflect the central role of L2 regularization, which is arguably more important.\n\n 1. X. Y. Han et al., 2022 Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path\n 2. T. Galanti et al., 2024 SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network\n 3. E. Zangrando et al.,2024 Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias \n 4. Lyu et al., 2024 Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking\n 5. Yubin Qin et al., 2023 FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)