Nonparametric Teaching of Attention Learners
Authors: Chen Zhang
Jianghui Wang
Bingyang Cheng
Zhongtao Chen
Wendong XU
Cong Wang
Marco Canini
Francesco Orabona
Yik Chung WU
Ngai Wong
Venue: ICLR 2026 Poster (2026)
Keywords: Nonparametric Teaching
Functional Gradient Descent
Attention Learners
Data Efficiency
Primary area: foundation or frontier models, including LLMs

Abstract: Attention learners, neural networks built on the attention mechanism, e.g., transformers, excel at learning the implicit relationships that relate sequences to their corresponding properties, e.g., mapping a given sequence of tokens to the probability of the next token. However, the learning process tends to be costly. To address this, we present a novel paradigm named **Atte**ntion **N**eural **T**eaching (AtteNT) that reinterprets the learning process through a nonparametric teaching perspective. Specifically, the latter provides a theoretical framework for teaching mappings that are implicitly defined (i.e., nonparametric) via example selection. Such an implicit mapping is embodied through a dense set of sequence-property pairs, with the AtteNT teacher selecting a subset to accelerate convergence in attention learner training. By analytically investigating the role of attention on parameter-based gradient descent during training, and recasting the evolution of attention learners, shaped by parameter updates, through functional gradient descent in nonparametric teaching, we show *for the first time* that teaching attention learners is consistent with teaching importance-adaptive nonparametric learners. These new findings readily commit AtteNT to enhancing learning efficiency of attention learners. Specifically, we observe training time reductions of 13.01% for LLMs and 20.58% for ViTs, spanning both fine-tuning and training-from-scratch regimes. Crucially, these gains are achieved without compromising accuracy; in fact, performance is consistently preserved and often enhanced across a diverse set of downstream tasks.

Published as a conference paper at ICLR 2026
NONPARAMETRIC TEACHING OF ATTENTION LEARNERS
Chen Zhang"
Jianghui Wang**
Bingyang Cheng'
Zhongtao Chen'
Wendong Xu'
Cong Wang
Marco Canini²
Francesco Orabona²
Yik- Yik-Chung Wu!
Ngai Wong'
'Te University of Hong Kong
King Abdullah University of Science and Technology
3 Independent Researcher
czhang @connect. hku.hk jianghui, www.aaasstggggaau.uu..... www.kkaust.edu.....
Project page
ABSTRACT
Attention learners, neural networks built on the attention mechanism, e.g., trans- formers, excel at learning the implicit relationships that relate sequences to their corresponding properties, e.g., mapping a given sequence of tokens to the probabil- ity of the next token. However, the learning process tends to be costly. To address this, we present a novel paradigm named Attention Neural Teaching (AtteNT) that reinterprets the learning process through nonparametric teaching perspective. Specifically, the latter provides a theoretical framework for teaching mappings that are implicitly defined (i.e., nonparametric) via example selection. Such implicit mapping is embodied through a dense set of sequence-properry property pairs, with the AtteNT teacher selecting subset accelerate convergence in attention learner training. By analytically investigating the role of attention on parameter-based gradient descent during training, and recasting the evolution of attention learners, shaped by parameter updates, through functional gradient descent in nonparametric teaching, we show for the first time teaching attention learners is consistent with teaching importance-adaptive nonparametric learners. These new findings readily commit AtteNT to enhancing learning efficiency of attention learners. Specifically, observe training time reductions of 13.01% .011% for LLMs and 20.58% for ViTs, spanning both fine-tuning and training-from-scee from-scratch regimes. Crucially, these gains are achieved without compromising accuracy; in fact, performance is consistently preserved and often enhanced across a diverse set of downstream tasks.
INTRODUCTION
The attention mechanism, inspired by human attention concepts (Ahmad, 1991; Soydaner, 2022). is designed to assess the relative importance of each element in a sequence (Bahdanau et al., 2015; Vaswani al., 2017). By leveraging attention, neural networks can effectively learn the implicit relationships that map sequences their corresponding properties, e.g. mapping sequence of tokens to the probability of the next token. These Attention Neural Networks (ANNs), e,g., transformers (Vaswani et al., 2017: Kong al., 2019), have achieved significant success in wide range of downstream tasks across various fields, including natural language processing (Vaswani et al., 2017; Devlin et al., 2019; Consens et al., 2025), computer vision (Dosovitskiy et al., 2020; Azad et al., 2024; Chen et al., 2024), and multimodal systems (Nagrani et al., 2021; Yang et al., 2024).
However, the process of learning the implicit mappings (i.e., training) be quite costly for ANNs, especially when handling large-scale tasks (Liu et al., 2018b; Beltagy et al., 2019 Gu et al., 2021: Yang et al., 2023). For instance, pretraining language models often requires training on corpora with millions of sentences (Common Crawl, 2007; Li et al., 2024). In the case of video understanding, the
"Equal contribution

---

Published as a conference paper at ICLR 2026
scale can become overwhelmingly large (Bain et al., 2021; Sharma al., 2018; Shu et al., 2025). As a result, reducing training costs and enhancing learning efficiency has become an urgent priority.
Recent research on nonparametric teaching (Zhang al., 2023b;a; 2024a; 2025) presents promising solution to the issue outlined above. Specifically, nonparametric teaching provides a theoretical framework for selecting examples efficiently when the target mapping (i.e., either function model) being taught is nonparametric, i.e, implicitly defined. It builds on the concept of machine teaching (Zhu, 2015; Zhu et al., 2018), which involves designing a training set (dubbed the teaching set) to help the learner quickly converge to the target functions, while relaxing the assumption that the target functions are parametric (Liu al., 2017: 01 2018a), thus enabling the teaching of nonparametric con-closed-fore functions with focus on function space. Unfortunately, these studies are limited to multilayer perceptron- basee learners and do not account for the attention mechanism, making their direct application difficult when the learners are ANNs. Additionally, ANNs are typically updated through gradient descent in parameter space, which contrasts with the functional gradient descent used in nonparametric teaching within function space (Zhang et al., 2023b;a; 2024a; 2025). Hence, it is not immediate to apply nonparametric teaching theory to attention learners.
To this end, we systematically investigate the role of attention on ANN gradient-based training in both parameter and function spaces. Specifically, we analytically examine how attention adaptively assigns different importance to each element in an input sequence during marameter-basee gradient descent in parameter space, and explicitly show that the parameter gradient retains the same form as the input sequence size scales. This importance adaptive update in parameter space drives the evolution of the ANN, which can be expressed using the dynamic Attention Neural Tangent Kernel (ANTK) (Yang, 2019; Hron et al., 2020), and then cast into function space. We prove that this dynamic ANTK converges to the importance-adaptiv canonical kernel used in functional gradient descent, suggesting that the evolution of ANN under parameter gradient descent is consistent with that under functional gradient descent. Therefore, it is natural to interpret the learning process of attention learners through the theoretical framework of nonparametrie teaching: the target mapping represented by dense set of pepueecce-popentt pairs, where each sequence is associated with its target output, and the teacher selects a subset of these pairs to provide to the ANN, ensuring rapid convergence of this attention learner. Consequently, to improve the learning efficiency of ANNs, we propose a novel paradigm called AtteNT, where the teacher applies a counterpart of the greedy teaching algorithm from nonparametric teaching to train attention learner, specifically by selecting the sequence with the greatest discrepancy between their true property values and the ANN outputs. Lastly, we carry out comprehensive experiments to demonstrate the effectiveness of AtteNT across various scenarios, including both natural language processing and computer vision tasks. Our key contributions are as follows:
We propose AtteNT, novel paradigm that interprets attention learner training through the theoreti- cal lens of nonparametric teaching. This facilitates the use of greedy algorithms from nonparametric teaching to effectively improve the learning efficiency of attention learners.
We analytically investigate the role of attention in parameter- baaed gradient descent within parame- ter space, revealing the consistency between the evolution of ANN driven by parameter updates and that under functional gradient descent in nonparametric teaching. We further show that the dynamic ANTK, emerging from gradient descent on the parameters, converges to the aaportancce-daptii canonical kernel of functional gradient descent. These findings bridge nonparametric teaching theory with attention learner training, thereby broadening the application of nonparametric teaching contexts involving attention mechanisms.
We demonstrate the effectiveness of AtteNT through extensive experiments across both natural language processing (NLP) and computer vision (CV) tasks. Our approach reduces Large Language Model (LLM) fine-tuning time by 13.01% % and accelerates Vision Transformer (ViT) training from scratch by 20.58%, thereby providing strong empirical support for our theoretical claims.
2 RELATED WORKS
Attention learners. The effectiveness of the attention mechanism in learning implicit mappings from sequences to relevant properties has spurred a surge in research on attention learners (Bahdanau et al., 2015; Vaswani et al., 2017; Kong al., 2019). This growing interest is particularly evident in the
2

---

Published as a conference paper at ICLR 2026
increasing efforts to apply attention learners across a wide variety of downstream tasks, including natural language processing (Galassi et al., 2020; Jin et al., 2024), computer vision (Dosovitskiy et al., 2020; Hassamin et al., 2024; Zhang et al., 2024b), medicine (Thirunavukarasu et al., 2023; Demszky et al., 2023), and graph-related fields (Veličković et al., 2018; Wu et al., 2024). Various efforts have been made in designing learners for improved mapping learning in vision tasks (Lin et al., 2022; Dosovitskiy et al., 2020; Arnab et al., 2021), as well as for more efficient inference (Kitaev et al., 2020; Katharopoulos et al., 2020; Lu et al., 2021). There have also been ongoing pursuits to enhance learning efficiency, such as sparse training (Frankle & Carbin, 2018; You et al., 2020; Chen et al., 2021c;b; Li et al., 2023), improved initialization (Huang et al., 2020; d' Ascoli et al.. 2021), and data curation (Tang et al., 2023; Zhong et al., 2023; Lin et al., 2024; Li et al., 2024). Differently, we frame attention learner training from a fresh perspective of nonparametric teaching (Zhang et al., 2023b;a), and adopt corresponding variant of the greedy algorithm to enhance the training efficiency of ANNs.
Nonparametric teaching. Machine teaching (Zhu, 2015; Zhu et al., 2018) focuses on designing a teaching set that allows the learner to quickly converge to target model function. It can be seen the reverse of machine learning: while machine learning aims to learn a mapping from a given training set, machine teaching seeks to construct the set based desired mapping. Its effectiveness has been demonstrated across various domains, including crowdsourcing (Singla et al., 2014; Zhou et al., 2018), robustness (Alfeld al., 2017; Ma et al., 2019; Rakhsha et al., 2020), and computer vision (Wang et al., 2021 a; Wang & Vasconcelos, 2021). Nonparametric teaching (Zhang et al., 2023b;a) extends iterative machine teaching (Liu et al., 2017: 2018a) by broadening the parameterized family of target mappings to encompass the more general nonparametric framework. This theoretical framework has proven effective in enhancing the efficiency of multilayer perceptrons for learning implicit functions from signal coordinates to corresponding values (Zhang et al., 2024a; 2026), as well as improving the training efficiency of graph convolutional networks for learning implicit mappings from graphs to their relevant properties (Zhang et al., 2025). Nevertheless, the absence of the attention mechanism in these studies limits their direct applicability to general tasks involving attention learners (Bahdanau et al., 2015; Vaswani et al., 2017). This work systematically investigates the role of attention and highlights the alignment between the evolution of ANN driven by parameter updates and that guided by functional gradient descent in nonparametric teaching. These insights, for the first time, broaden the scope of nonparametric teaching in attention learner training, positioning our AtteNT as a novel approach to improving ANN learning efficiency.
3 BACKGROUND
Notation. 1 Let (ari...., Rs) represent a sequence of length S, where each v, E Rd denotes a d-dimensional feature vector associated with the s-th element, with E Ns (Ns == (1,...,,S) S)). Each Vg is a row vector, expressed as [I = [Te.1....,, T.,aa. The entire collection of feature vectors forms an d feature matrix, denoted Ssxd E S C S (or simply S). The s-th row and the i-th column of this matrix, corresponding to the s-th element and the i-th feature, are denoted by ((s.)) and S (.,)) respectively. Alternatively, these can be written as e,_ S and Se;, where e, is standard basis vector with its i-th entry being 1 and all other entries equal to 0. The bold column vector 1 represents a vector in which all elements are 1. The property of the sequence is represented by y Yr where y is scalar for sequence-leve properties ()' IR) and a vector for element-level properties () C R²)). A set with m items is denoted as {a-)-- If (a.)m C (a+) then (a+ Jm represents a subset of {ai}, containing m items, where the indices are E N,. A diagonal matrix with diagonal entries a1...., am is denoted as (llag(a,...... am), and if all m values are identical, the matrix simplified as diag(a; m).
Let S") : S x S + R denote a symmetric and positive definite sequence kernel (Can- cedda et al., 2003; Király& & Oberhauser, 2019). This kemnel can also be expressed as K(S, S") Ks(S") = K (S), where for simplicity, Ks()) may be abbreviated as Ks. The reproducing kernel Hilbert space (RKHS) H associated with K(S, S") is defined as the closure of the linear span { f(.) Σ. (S, a, E R E S; S}. with the inner product given by (f.g)N = Lb,kk S₁), where g = Σ, k,Ks, (Liu & Wang, 2016; Zhang ct al., 2023b). Rather than assuming the idealized case of a closed-form solution f*, we focus on the more realistic scenario where the realization of I is given (Zhang et al., 2023b;a; 2024a; 2025). Given the target
Thee notation table be found in Appendix A.
3

---

Published as a conference paper at ICLR 2026
mapping f 8 + V, it uniquely maps each sequence S its corresponding output y,, such that Yt = f According to the Fiesz-Fréélie representation theorem (Lax, 2014; Schölkopf & Smola, 2002: Zhang et al., 2023b), the evaluation functional is defined follows:
Definition 1. Let H denote reproducing kernel Hilbert space* equipped with a positive definite sequence kernel Ks E H. where S E S. The evaluation funicionaa Es(.) H. + R. is defined by the reproducing property as
Es(f) = (f,K())x
f H.
(1)
Furthermore, for a functional F H. + I, the Fréchet derivative (Coleman, 2012; Liu, 2017; Zhang al., 2023b) of F is defined as:
Definition 2. (Fréchet derivative RKHS) The Fréchet derivative of a functional F ::H - point f H. represented V F(f(),, is defined implicitly by F(f +g)) o(e) for any E H and E E R. This derivative itself is a function in H.
Attention learners, referring to neural networks that incorporate attention mechanisms, are designed to learn the implicit mapping between input sequences and their associated properties (Vaswani et al., 2017). Specifically, the attention consists of three components: the query matrix Q(S) = SW?, the key matrix K(SS) SWK and the value matrix VSS) SWV where the query and key weight matrices Wº and W are of size d x p. and the value weight matrix WV is of size d x v. For simplicity, this paper primarily focuses on a single-layer, single-head self-attention neural network" (Mahankali et al., 2024; Makkuva et al., 2025), which can be expressed as
((S) Vd
fo(S) = softmax
V(S),
where softmax(-) is applied row-wise.
Nonparametric teaching formulated as a functional minimization over a teaching set, denoted as D {(x' .....//) ,y',.... (x= ))²), where each input E Re represents independent feature vectors, without considering the sequence (Zhang et al., 2023b). The collection of all possible teaching sets is represented by DD:
D arg min M(f X card(D)
s.t. A==(
DED
This formulation involves three key components: M which measures the discrepancy between f and f (e.g., 18. L2 distance in RKHS M(f-, f*) = Ilf - f* card(-), representing the cardinality (or size) of the teaching set D, controlled by regularization constant À > O; and A(D), which denotes the learning algorithm employed by the learners, typically based on empirical risk minimization:
A(D) = arg min
Σ ((f(2'),v)
JEH card(D)
(a* vate)
with convex loss L (w.r.t. f), which optimized using functional gradient descent:"
(DC(f. Of
gal t 5 - Ex
K
(5)
=9(C. (,)), Functional Gradient
where
1,..., V.....Tiss the iteration index, 77 > 0 the learning rate, and Fz = f(x:) denotes the evaluation functional.
'In nonparametric teaching. the extension from scalar-valued to vector-valued functions, relating to element- level properties, is well-established generalization in Zhang et al., 2023a.
"This can be directly extended other attention learners, including those with multi-head attention different types of attention mechanisms (Dong et al., 2021; Kajitsuka Sato, 2024).
'The functional gradient is obtained by applying the functional chain rule (Lemma 5) and the gradient of an evaluation functional (Lemma 6), both of which detailed in Appendix A.2.
4

---

Published as a conference paper at ICLR 2026
4 ATTENT
We begin by investigating the role of attention in parameter-basee gradient descent. Then, by translating the evolution an ANN-driven by importancc-udaatiiv updates in parameter space-into function space, we show that the evolution of the ANN under parameter gradient descent is consistent with that under functional gradient descent. Lastly, we present the greedy AtteNT algorithm, which effectively selects sequences with steeper gradients to enhance the learning efficiency of the ANN.
4.1 IMPORTANCE UPDATE IN THE PARAMETER SPACE
Let the column vector 0 E T denote trainable weights in flattened format, with m representing the total number of parameters in the ANN. Figure illustrates the workflow of the ANN. Given a training set of size N, {S y:) S, E S.Y: V/,, the parameters are updated via gradient descent, as shown be- low:
Fosuu Matris 5
Queyy ?
Key K
Valaz
ge-1 t 0r
Vol(fo B.g.)))
NS
where for (S:) refers to the j-th row of the output foe (S:), corresponding to the j-th ele- ment of the input sequence, and Vi(s.) denotes its associated property value. Since the learning rate 7j small enough, the updates remain mini- mal over multiple iterations, allowing them to be treated as a time derivative and thus expressed as differential equation (Jacot al., 2018; Yang. 2019; Hron et al., 2020):
Oulput
vi
Figure 1: An illustration of the workflow for attention neural network with input sequence S.
a
(f afe: (S))
@C(Jo 0fa (Sw)
fo (S:)
ōt NS
00
The term dfe(S) (with the indexes and omitted for simplicity), which defines the direction for parameter updates, can be more explicitly written as
0So(S) 09
0So(S) 0So(S) OW (..)) OW ow (.0)
0So(S)
OSo(S) 0
0So(S)
ow OWn OW (1.1)
(:.)
(:.p)
w.T.t. weight matrix
w.n.l. the weight matrix
key weight matrix
Here, each term represents the derivative of the output fS(S) w.r.t the weight column vectors. Unlike derivatives for multilayer perceptron-base based learners, where the input is used only once, i.e., the derivative depends on single use of the input, the attention mechanism invokes the input three times Q, K., and V separately, as depicted Figure/ 1. To clearly demonstrate, in an analytical and explicit manner, how these three invocations allow attention to adaptively assign varying different importance to each element in an input sequence within parameter space, we present an example involving the derivative of an ANN with
(3.:) is scalar: meaning that each component of the output for (S)().)
OSe(S) 00
Ofo(S)
0Se(S) OwV aw9
0fa(S)
0fe(S)
ffo(S)
awe
OW K (:.1)
OWK
(.1) (.11)
(s.p)
'Training sequences generally have the same length, corresponding to the maximum length, which is ensured by padding or truncating (Yu al., 2023b; Ding al., 2024). Therefore, this paper focuses sequences of the same length unless noted otherwise. Results for varying lengths can be directly obtained.
5

---

Published as a conference paper at ICLR 2026
where the term
(f((5) awv
has shape of S x d, and is given by
exp (QuL)) /Va)
(S)
S. where exp(.) denotes the element- wise exponential operator. For simplicity, we omit the arguments of ② K,K.V. V. For i E Ny. the term @f((S)
(10)
dWV
1 exp (Qu.)
and afo(S)
ow
aw are
6.4)
(.0)
186
ding (softmmsx Koa (va)) V Kisi CaltuaaaQaaaain V/Va)" (0 TvVi))
(11)
THe Quta
Sxg
8x8
diapp sofeemoo Kina /Va)) Ru.o (ortmnodQQua (Va) softmmed Dusakias (Ni)
(12)
J00
The derivation is provided in Appendix A.3. For the sake of brevity, we focus on the query gradient, with similar results holding for the key and value gradients. As a result of invoking the input three times, Equation 11 reveals that the ANN gradient depends not only on the features of the sequence elements, i.e., S(a.), but also on a scalar wy that is specific to each element.
Specifically, Equation 11 explicitly shows that the gradient row order follows the order of elements in the input sequence, meaning the gradient is equivariant w.r.t. reordering the elements. This is in contrast to the gradient recurrent neural networks (Elman, 1990; Jordan, 1997), where the order of the elements determines the power of the recurrent weights. Moreover, this gradient property is derived during the training stage, yet, interestingly, it aligns with the permutation invariance property of self-attention during inference (Lee al., 2019).
The scalar w, in Equation 11 is computed from Q, K., and V, which reflects the three invocations of input by the attention. It is clear that it is closely associated with the j-th element, meaning it is element-specffic. This scalar is the importance value that attention assigns to each element, leading an importance-adaptive update the parameter space. When all importance values are gradient of the ANN reduces to the derivative of multilayer perceptron without nonlinear activations and with batch input. Additionally, the explicit expressions in Equations 7, 11, and 12 show that the ANN gradient does not depend on the input sequence length (i.e., the number of elements), as this is averaged out. Instead, it depends on the feature dimension. other words, the parameter gradient remains unchanged even if the input sequence length S is scaled.
the
4.2 THE FUNCTIONAL EVOLUTION OF ANN
The importacce-adaptiin update in the parameter space drives the functional evolution of fo E H. This variation in fe. reflecting how fe responds to updates in 8, can be derived using Taylor's theorem as follows:
5(0r+1 f(0') (Vof(0"), 0++1 0²) + 0 0²),
(13)
where f(0t)= = fot. In manner analogous the transformation of parameter updates into their differential form, this can also be expressed in a differential form (Zhang et al., 2024a):
0fe
(0((0')
00
100
(14)
Øt
00t
Øt
(x)
By substituting the specific parameter updates, i.e., Equation 7, into the first-order approximation term (+) of this variation, we obtain
of. Øt
17 NS
ac(fo (SN).
[Kor (l. +
( O0
(15)
Of
Oe (SN)
0t
where the symmetric and positive definite Kø (S..)) = and further discussion, Appendix A.4). Due to the inclusion of nonlinear activation functions in
05 (S.)
(fat(.) 89
(for detailed derivations
80

---

Published as a conference paper at ICLR 2026
f(), the nonlinearity of f() with respect to I results in the remainder a(8 0¹) being nonzero. In a subtle difference, Jacot et al., 2018; Yang, 2019; Hron et al., 2020 apply the chain rule directly, giving less focus to the convexity of L with respect to 0. As result, the first-order approximation is derived as the variation, with Kg being referred to as the Attention Neural Tangent Kernel (ANTK). It has been demonstrated that the ANTK remains constant during training when the ANN width, i.e., d, is assumed to be infinite (Hron et al., 2020). However, in practical applications, the ANN width does not need to infinitely large, prompting us to explore the dynamic ANTK (an example of how the ANTK is computed can be found in Figure 3 in Appendix A.4).
Consider characterizing the variation of fe E from high-level. functional viewpoint (Zhang et al., 2024a; 2025). Using functional gradient descent, it can be written as
f Øt
g)( fo; (S,b))
(16)
where the functional gradient is expressed as
1 Tac(fa (S)). for. = NS Ofo (S.)
(((f).(S)) uN) 0fo
[ (K. -I
(17)
The asymptotic relationship between ANTK and the importance-adaptive canonical kernel (Cancedda et al., 2003; Király Oberhauser, 2019; Zhang et al., 2024a) in the context of functional gradient presented in Theorem 3 below, with the proof provided in Appendix B.I.
Theorem 3. Given convex loss L and a training set {S, ys)]s, E S.y. E VtN. the dynamic ANTK, which derived from performing gradient descent on the parameters of an ANN, converges pointwise to the importance eeportanceaddapiive canonical kernel in the dual functional gradient with respect to the input sequences. Specifically, holds that
lim Kg (S,, :(S.,), Vi NN. t-000
(18)
This suggests that ANTK, which includes adaptive importance information, serves as dynamic substitute for the apportancc-adaptivy canonical kernel in functional gradient descent with sequence inputs, aligning the ANN evolution through parameter gradient descent with that in functional gradient descent (Kuk, 1995; Hron et al., 2020; Geifiman et al., 2020). This functional insight bridges the teaching of attention learners (i.e., ANNs) with that of amportance-adaptii nonparametric learners, while also facilitating further analysis (e.g., convex functional L retains its convexity with respect to fo from a functional perspective, but is typically nonconvex when considering 0). By utilizing the functional insight and applying the canonical kernel (Dou & Liang, 2021) instead of ANTK (which should be considered alongside the remainder). it facilitates deriving sufficient reduction concerning in Proposition 4, with the proof deferred Appendix B.2.
Proposition 4. (Sufficient Loss Reduction) Let the convex loss L be Lipschitz. smooth with a constant T > O, and suppose the importance-adaptive canonical kernel is bounded above by a constant y > If the learning rate ni satisfies n S 1/(27), then a sufficient reduction in L is guaranteed, as demonstrated by
ac (S)) 2
OC
17 2
(19)
Ot
NS
0f (fo(
1=1
This indicates that the variation of L over time is capped by negative value, meaning it decreases by least the magnitude of this upper bound time progresses, ensuring convergence.
4.3 THE ATTENT ALGORITHM
Building on the understanding of how attention adaptively assigns varying importance in parameter- based gradient descent, as well as the consistency between teaching an ANN and a nonparametric learner, we introduce the AtteNT algorithm. This algorithm designed to amplify the steepness of the gradients, thereby improving the learning efficiency of the ANN. By considering the gradient as the sum of projections of ac(f..") onto the basis ( [ (K(S.,) :)) n. the gradient be increased simply 0ie
7

---

Published as a conference paper at ICLR 2026
by maximizing the projection @C (3))))) thus eliminating the need to compute the norm of the (S.)
basis (S,, (Wright, 2015; Zhang et al., 2024a). This suggests that selecting sequences that
either maximize fal v.). correspond the larger components of @C(fa.() can effectively 0fe
(S,)
amplify the gradient, indicating that
(C( yb) (fe(S.)
arg max {S,) (. (S,)
(20)
with Frobenius norm || lls=. From functional viewpoint, for convex loss functional L, the norm of its partial derivative w.r.t. fe. denoted as ||
11 is positively correlated with f As fe
aie
[ gets closer to f", the value of || ft. æl decreases (Boyd Vandenberghe, 2004; Coleman, 2012). This relationship becomes especially prominent when strongly convex with a larger convexity constant (Kakade & Tewari, 2008; Arjevani et al., 2016). Building these insights, the AtteNT algorithm selects sequences by
(5:) =
arg max [[,)((S)) ( (()) ((
(21)
The pseudocode is provided in Algorithm 1.
5 EXPERIMENTS AND RESULTS
To demonstrate the broad effectiveness of the AtteNT Algorithm, we conducted extensive experiments across diverse domains. Our evaluation covered large language models and computer vision models. In addition, we validated performance under multiple training paradigms, including training from scratch, and fine-tuning, consistently achieving strong results.
LLM Scenario. We evaluate AtteNT algorithms across diverse set of natural language gen- eration (NLG) tasks. Specifically, we fine-tune LLaMA 2-7B (Touvron et al., 2023), Mistral- 7B (Jiang et al., 2023), and Gemma-7B (Team et al., 2024) the MetaMathQA dataset (Yu et al., 2023a) to benchmark their mathematical reasoning capabilities on GSM8K (Cobbe et al., 2021) and MATH (Hendryeks et al., 2021). To assess coding proficiency, we further fine-tune these mod- els on CodeFeedback (Zheng et al., 2024b) and evaluate on HumanEval (Chen et al., 2021a) and MBPP (Austin et al., 2021). For conversational ability, we train on WizardLM-Evol- E-vovvr-nnnttuu (Xu et al., 2023) and evaluate on MT-Bench (Zheng et al., 2024a). All experiments are conducted on standardized subsets to ensure comparable training efficiency and are trained for five epochs.
As shown in Table 1, AtteNT consistently outperforms standard fine-tuning across all evaluated models and tasks while reducing computational overhead. Specifically, fine-tuning LLaMA, Mistral, and Gemma with AtteNT yields accuracy gains of 1. 1.39, 2.14, and 2.42 on GSM8K, and 1.59, 2.89, and 0.76 on MATH. On coding benchmarks, AtteNT improves performance by 3.66%, 3.25% and 0.29% on HumanEval, and by 2.08%, 3.25%, and 3.31 % on MBPP. We further report average fine-tuning time per model under identical data volumes and epoch settings. Since runtime variation arises primarily from AtteNT's adaptive data selection, the observed results highlight its efficiency: on average, AtteNT reduces training time by 12.78%, underscoring its advantage in both performance and resource savings.
Table 1: AtteNT NLG tasks. The results averaged over three runs, with standard deviations included. The GSMSK and MATH datasets share math fine-tuned model, while HumanEval and MBPP use code fine-tuned model. MTT-Bench utilizes conversation fine-tuned model. The "Avg. time" represents the average fine-tuning time for the three models.
Model
AtteNT
Avg. Time(()
GSMSK(1)
MATH(1)
HumanEval())
MBPP(()
MTTBencch()
w/o
246+1m
42,9620.12
5.0620.16
18.3520.31
35.65e0.25
4.58:0.01
ILaMA 2-7B
213+2m
43.45:0.55
6,48:0.24
21,80=0.38
37.61=4.42
4.49=0.02
w/o
20412m
69.1310.22
20.06a0.20
43.4240.1 14
58.52a0.13
5.03a0.05
Mistral-7B
180+2m
71.26a0.23
23.12+0.44
46.55e0.25
61.74a0.54
5.32+0.04
w/o
228+22
75. 15.23110.5
30.52=0.48
$3.83
65.69:1.299
5.42=0.04
Gemma-7B
W
201
77.74+0.32
31.40e0.36
54,2620,28
66.28:0..6
5.4420.08
8

---

Published as a conference paper at ICLR 2026
Table AtteNT across various CV downstream tasks. ImageNetS50 uses 50 categories from ImageNet for classification, evaluated by accuracy. NYUv2(S) is semantic segmentation task with mloU as the metric. NYUv2(D) involves depth estimation, evaluated using the ó1 metric, which measures the percentage of pixels with an enror ratio below 1.25 (Doersch & Zisserman, 2017).
Model
AtteNT
Pretraining Time(()
Image 1etS50(1)
NYUv2(S)()) NYUv2(D)(T)
w/o
1234m 980m(-22 980m(-20.58%%
92.2
51.9
52.1
Multi-Modal MAE
92.3
52.6
57.2
Table 3: Ablation study of AtteNT pre-training configurations. Ratio controls how the fraction of selected samples increases over epochs. Interval denotes how often the subset is re-sampled. Selection specifies the sampling strategy: Random (no difficulty prior), Hard (selects only difficult samples), and Soft (Gumbel-Topy Top-k difficulty-awand sampling). The configuration (Incremental, Incremental, Soft) in the red color row is adopted as our final AtteNT setting, as simultancously reduces pre-training time and improves performance on all downstream tasks.
Pre-training
Downstream NYUv2(SX()
Ratio
Interval
Selection
Training time(.)
ImageNetS30(1)
NYUv2(DX()
1234m
92.2
51.9
Cosine
Incremental
Random
966m
88.6
45.3
Cosine
Incremental
Soft
995m
92.1
52.2
58.8
Cosine
Fixed
Soft
1301m
93.2
53.6
61.4
Incremental
Incremental
Soft
980m
92.3
52.6
57.2
Incremental
Fixed
Soft
1319m
92.4
53.7
62.1
Cosine
Incremental
Hard
972m
91.8
49.5
57.3
Cosine
Fixed
Hard
1285m
92.1
53.0
60.8
Incremental
Incremental
Hard
963m
91.4
48.4
57.2
Incremental
Fixed
Hard
1302m
92.5
52.7
59.5
CV Scenario. The Multi-Modal MAE (Bachmann al., 2022) is designed to address a diverse range of downstream tasks by employing three specialized encoders, each dedicated to processing distinct image modality. During pre-training, we explore various selection strategies, including different ratios and intervals, optimize model configuration. The pretraining process is conducted over 800 epochs.
For unsupervised pre-training, utilize ImageNetS50 (Gao et al., 2021) to evaluate the effective- ness of the AttneNT method enhancing the performance downstream tasks under suboptimal conditions. Classification performance is assessed using the validation subset of the original dataset, while semantic segmentation and depth estimation tasks are fine-tuned and evaluated on the NYUv2 dataset (Silberman et al., 2012). Given the absence of large multi-task dataset with aligned task- specific images (Doersch & Zisserman, 2017; Bachmann et al., 2022: Wang et al., 2023), we generate pseudo-labels for ImageNetS50 using Mask2Former (Cheng al., 2022).
As shown in Table 2, the AtteNT strategy results in a significant reduction in training time, saving 20.58% during long-duration training from scratch. Additionally, consistently improves perfor- mance across a wide range of downstream tasks. Notably, the depth estimation task exhibits the largest gain, achieving a 5.1 % improvement. We attribute this improvement to the nature of the depth estimation task, which is independent of image type, thus preventing any disruption in data distribution during the selection process. Our experiments demonstrate the efficacy of AtteNT within the VIT architecture.
The practical performance gains stem directly from the curriculum effect induced by nonparametric teaching (Bengio et al., 2009; Wang al., 202 Zhang et al., 2023b; 2025), which greedily selects the examples that most advance the learner. This naturally creates a curriculum that focuses training on informative, high-gradien examples and avoids gradient dilution from already
ones.
We further present the ablation study results for AttneNT, focusing on the effects of varying data selection strategies and their impact on downstream tasks. Specifically, investigate the influence of dynamic changes in data selection ratios and step sizes, following the strategy proposed in (Zhang al., 2023b). Additionally, examine how different selection strategies affect the performance

---

Published as a conference paper at ICLR 2026
of downstream tasks. The Random strategy involves selecting data without any predefined criteria, while the Hard strategy entails deterministic data selection. The Soft strategy, on the other hand, probability- data selection, derived from loss scores. To implement this, apply the Gumbel- Top-k selection algorithm (Kool et al., 2019) for sampling without replacement. Our results show that the Soft selection strategy achieves the best performance in downstream tasks, significantly improving the model's ss robustness during training. A more detailed study of the sample ratio can be found in Appendix D.I, and additional comparison results are provided in Appendix D.2.
CONCLUDING REMARKS AND FUTURE WORK
This paper introduces AtteNT, novel paradigm that enhances the learning efficiency of attention learners (i.e., ANNs) through nonparametric teaching theory. Specifically, AtteNT reduces the wallclock time required learn the implicit mapping from sequences to relevant properties by 13.01% to 20.58% while consistently preserving and often enhancing the performance across diverse set of downstream tasks. Moreover, AtteNT establishes theoretical connection between the evolution of an ANN via parameter-basee gradient descent and that of a function using functional gradient descent in nonparametric teaching. This connection between nonparametric teaching theory and ANN training expands the potential applications of nonparametric teaching in contexts that involve attention mechanisms.
In future work, it would be interesting to explore other variations of AtteNT for different attention learners, such as graph attention networks (Veličković et al., 2018). Additionally, investigating its robustness under real-worle label noise, building upon recent noise-robust advancements (Wei et al., 2024; Hu et al., 2024), could yield crucial improvements. Another promising direction to examine the practical applications of AtteNT in improving the efficiency of data-driven methods (Henaff, 2020; Touvron et al., 2021; Müller et al., 2022) for attention-related tasks, especially in areas like world models.
REPRODUCIBILITY STATEMENT
We have taken substantial steps promote the reproducibility of our research. Appendix A offers a comprehensive overview of the notation, theoretical background, and key algorithm. All proofs be found in Appendix B. Meanwhile, Appendix C provides comprehensive description of the experimental setup, including training configurations, hyperparam- eter choices, algorithmic details, and dataset preprocessing procedures. Codes are available the following link: LINK.
for theorems and propositions
ACKNOWLEDGMENTS
This work was supported in part by the Theme-based Research Scheme (TRS) project T45-701/22- of the Research Grants Council of Hong Kong, and in part by the AVNET-HKU Emerging Microelectronics and Ubiquitous Systems (EMUS) Lab. For computer time, this research used the resources of the Supercomputing Laboratory at KAUST.
REFERENCES
Subutai Ahmad. VISIT: a neural model of covert visual attention. NeurlPS, 1991.
Scott Alfeld, Xiaojin Zhu, and Paul Barford. Explicit defense actions against test-set attacks. In AAAL, 2017.
Yossi Arjevani, Shai Shalev-Shwartz, and Ohad Shamir. On lower and upper bounds in smooth and strongly convex optimization. The Journal of Machine Learning Research, 17(1):430344333 2016.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. ViViT: video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6836-6846, 2021.
10

---

Published as a conference paper at ICLR 2026
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoe Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 8
Reza Azad, Leon Niggemeier, Michael Hüttemann, Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Yury Velichko, Ulas Bagci, and Dorit Merhof. Beyond self-attentio: Deformable large kernel attention for medical image segmentation. In WACV, 2024.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi- task masked autoencoders. In European Conference on Computer Vision (ECCV), pp. 348-367. Springer, 2022.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 1, 3
Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021. 2
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In EMNLP-IJCNLP, 2019. 1
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, 2009. 9
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Nicola Cancedda, Eric Gaussier, Cyril Goutte. and Jean-Miche Renders. Word- Word-sequence kernels. Journal of machine learning research, 3(Feb):1059-1083 2003. 3,7
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam. Vedant Misra, Evan Morikawa. Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint ar arXin:2107.03374 2021a. 8
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: end-to-end exploration. Advances Neural Information Processing Systems, 34:19974-19998 2021b. 3
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. EarlyBERT: Efficient BERT training via early-bird lottery tickets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 2195522017 2021c.
Yuantao Chen, Runlong Xia, Kai Yang, and Ke Zou, Dnnam: Image inpainting algorithm via deep neural networks and attention mechanism. Applied Soft Computing,
2024.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning (ICML), 2018. 34, 35
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 9, 9,
11

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arX1v:2110.14168, 2021. 8
Rodney Coleman. Calculus normed vector spaces. Springer Science & Business Media, 2012. 4,
8,
Common Crawl. Common crawl, 2007. URL https::/ccomoncraaate org. 1
Micaela E Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian Theis, Alan Moses, and Bo Wang. Transformers and genome language models. Nature Machine Intelligence, pp. 1-17, 2025.
Dorottya Demszky, Diyi Yang, David S Yeager, Christopher Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, Michaela Jones, Danielle Krettek-Cood Cobb, Leslie Lai, Nirel JonesMitchell, Desmond C. Ong. Carol S. Dweck, James J. Gross, and James W. Pennebaker. Using large language models in psychology. Nature Reviews Psychology, 2(11):688-700) 2023. 3
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. Fewer truncations improve language modeling. International Conference on Machine Learning, pp. 1 11030-1 11830-1104.,. PMLR, 2024.
Carl Doersch and Andrew Zisserman. Multi-task sule-supe visual learning. In International Conference on Computer Vision (ICCV), 2017. 9
Yihe Dong, Jean- CaantBap Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pp. 2093-28923 PMLR, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020. 3,
Xialiang Dou and Tengyuan Liang. Training neural networks as learning data-adaptive kernels: Prov- able representation and approximation benefits. Journal of the American Statistical Association, 1166535):500-11500 2021.7
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. ConViT: Improving vision transformers with soft convolutional inductive biases. In International conference on machine learning, pp. 2286-2296. PMLR, 2021. 3
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211 1990.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018. 3
Andrea Galassi, Marco Lippi, and Paolo Torroni. Attention in natural language processing. IEEE transactions on neural networks and learning systems, (2(10):491-43033 2020.
Shang-Hua Gao, Zhong-V Yu Li, Hing-Hsuan Yang, Ming-Ming Cheng. Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. arXiv preprint arXiv:2106.03149, 2021. 9, 32
Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the laplace and neural tangent kernels. In NeurIPS, 2020. 7
Izrail Moiseevitch Gelfand and Richard A Silverman. Calculus of variations. Courier Corporation, 2000. 20
12

---

Published as a conference paper at ICLR 2026
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 2918-2928, 2021. 33
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specifis language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1): 1-23, 2021.
Mohammed Hassanin, Saced Anwar, Ibrahim Radwan, Fahad Shahbaz Khan, and Ajmal Mian. Visual attention methods in deep learning: An in-depth survey. Information Fusion, 102:1024,7, 2024.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
10
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arv: 103.03874, 2021. 8
Jiri Hron. Yasaman Bahri, Jascha and NTK for deep attention networks. In International Conference on Machine Learning, pp. 4376 4386. PMLR, 2020. 7,
and Roman Novak. Infinite attention: NNGP
Yuchen Hu, CHEN CHEN, Chao-Han Huck Yang. Ruizhe Li, Chao Zhang, Pin-Ye Yu Chen, and EngSiong Chng. Large language models are efficient learners of noise-robust speech recognition. In ICLR, 2024. 10
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pp. 646-661. Springer, 2016. 33
Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In International Conference Machine Learning, pp. 4475 4483. PMLR, 2020. 3
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2018. 7, 26
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint 1000v:2310.06825, 2023. 8
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. In ACL (Findings), 2024. 3
Michael I Jordan. Serial order: A parallel distributed processing approach. In Advances in psychology. volume 121, pp.
Elsevier, 1997.
Tokio Kajitsuka and Issei Sato. Are transformers with one layer self-attention using low-rank weight matrices universal approximators? In The Twelfth International Conference on Learning Representations, 2024. 4
Sham M Kakade and Ambuj Tewari. On the generalization ability of online strongly convex programming algorithms. In NewrlPS, 2008. 8
Angelos Katharopoulos and François Fleuret. Not all samples created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525 2535-2534. PMLR, 2018. 34
13

---

Published as a conference paper at ICLR 2026
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers RNNs: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156-5165. PMLR, 2020. 3
Franz Király and Harald Oberhauser. Kernels for sequentially ordered data, Journal of Machine Learning Research, 1: 144,, 2019. 3, 7
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. 3
Qiuqiang Kong, Changsong Yu, Yong Xu, Turab Iqbal, Wenwu Wang, and Mark D Plumbley. Weakly labelled audioset lagging with attention neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(11))1791-100) 2019. 1,2
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In International conference on machine learning, pp.
PMLR, 2019. 10
Anthony YC Kuk. Asymptotically unbiased estimation in generalized linear models with random effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 57(2):395 407, 1995.
Peter D Lax. Functional analysis. John Wiley & Sons, 2014. 4
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- neural networks. In International
former: framework for iitention-based permutation-invant conference on machine learning, pp. 3744-3753, PMLR, 2019. 6
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen. Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruha Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang. Rulin Shao, Sarah M. Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. DataComp-LMM: In search of the next generation of training sets language models. In NeurIPS, volume 37, pp. 12200-11228.. 2024. Datasets and Benchmarks Track. 1,3
Zonglin Li. Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix X. Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2023.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al open, 3: 111-132, 2022.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, yelong shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, and Weizhu Chen. Not all tokens what you need for pretraining. Advances in Neural Information Processing Systems, 37:29029- 2024. 3, 32, 34
Qiang Liu. Stein variational gradient descent gradient flow. In NeurIPS, 2017. 4
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In NeurlPS, 2016.
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative machine teaching. In ICMI., 2017. 2,
Weiyang Liu, Bo Dai, Xingguo Li, Zhen Liu, James Rebg, and Le Song. Towards black-bor box iterative machine teaching. In ICML. 2018a. 2,3
14

---

Published as a conference paper at ICLR 2026
Xin Liu, Qingeai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. Loqme: A large-scale chinese question matching corpus. In Proceedings of the 27th international conference on computational linguistics, pp. 1952-1962, 2018b.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017. 32
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. SOFT: Softmax-free transformer with linear complexity. Advances in Neural
Information Processing Systems, 34:21297-21399, 2021. 3
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. In NewrIPS, 2019. 3
Arvind V Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. The Twelfth International Conference on Learning Representations, 2024. 4
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with Markov: A curious case of single-layer transformers. In The Thirteenth International Conference on Learning Representations, 2025. 4
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arX1v:2404.02948, 2024. 32
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM transactions on graphics (TOG), 41(4): 1-15, 2022. 10
Arsha Nagrani, Shan Yang, Anurag Amab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. In NeurlPS, volume 34, pp. 14200-14213, 2021. 1
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-tine time adversarial attacks against reinforcement learning. In ICML, 2020. 3
Bernhard Schölkopf and Alexander J Smola. Learning with kernels: support vector machines, regularization, oplimization, and beyond. MIT press, 2002. 4
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Longg Pupers), pp. 2556-2565, 2018 2
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-XL: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26160-26169, 2025. 2
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In European Conference on Computer Vision (ECCV), 2012.
Adish Singla, Lija Bogunovic, Gábor Bartók, Amin Karbasi, and Andreas Krause. Near-optimally teaching the crowd to classify. In ICML, 2014. 3
Derya Soydaner. Attention mechanism in neural networks: where it comes and where it goes. Neural Computing and Applications, 34(16): 2022.
Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, and Yifan Liu. Dynamic token pruning in plain vision transformers for semantic segmentation. In Proceedings of the IEFF/CV International Conference on Computer Vision, pp. 777-786, 2023. 3
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An caatucttionfoolowing. LLaMA model. https: / / thub com httmm/aatuuuu lab/ stanford_ pacc,, 2023. 32
15

---

Published as a conference paper at ICLR 2026
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Chool Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Chrrstian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptise Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong. Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy, Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0822 08295, 2024. 8
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8): 1930-1940, 2023. 3
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021. 10
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2300:099000988 2023. 8
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 2,2,,,4
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICIR, 2018. 3, 10
Jianghui Wang, Yang Chen, Xingyu Xie, Cong Fang, and Zhouchen Lin. Task-robust pre-training for worst-case downstream adaptation. Advances in Neural Information Processing Systems, 36: 9458 9488,947,.77 2023. 9
Pei Wang and Nuno Vasconcelos. A machine teaching framework for scalable recognition. In ICCV, 2021. 3
Pei Wang, Kabir Nagrecha, and Nuno Vasconcelos. Gradient-based algorithms for machine teaching.
In CVPR, 202
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555-457 2021b. 9
16

---

Published as a conference paper at ICLR 2026
Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang- Xin Shi, Yu-Feng Li, and Min-Ling Zhang. Vision- language models are strong noisy label detectors. In NeurIPS, 2024. 10
Stephen J Wright. Coordinate descent algorithms. Mathematical programming, 151(1):3-34, 2015.
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang. Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. World Wide Web, 2765);60, 2024. 3
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardI Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2023.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2Seq: Large-scale pretraining of visual language model for dense video captioning. In CVPR, pp. 10714-10726, 2023.
Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems. 32, 2019. 2, 5, 7, 26
Hongyu Yang, Jinjiao Zhang, Liang Zhang, Xiang Cheng, and Ze Hu. MRAN: Multimodal relationship-aware attention network for fake news detection. Computer Standards & Interfaces, 89:103822, 2024.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020. 3
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023a. 8
Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. Bag of tricks for training data extraction from language models. In International Conference on Machine Learning, pp. 40306-40320, PMLR, 2023b.
Chen Zhang, Xiaofeng Cao, Weiyang Liu, Ivor Tsang, and James Kwok. Nonparametric teaching for multiple learners. In NeurlPS, 2023a. 2, 3, 4, 20, 25,27
Chen Zhang, Xiaofeng Cao, Weiyang Liu, Ivor Tsang, and James Kwok. Nonparametric iterative machine teaching. In ICML, 2023b. 2, 3, 4, 9, 20, 33
Chen Zhang, Steven Tin Sui Luo, Jason Chun Lok Li, Yik- Yik-Chung Wu, and Ngai Wong. Nonparametric teaching of implicit neural representations. In ICML, 2024a. 2, 3, 6, 7,8
Chen Zhang, Weixin Bu, Zeyi Ren, Zhengwu Liu, Yik-Chung Wu, and Ngai Wong. Nonparametric teaching for graph property learners. In ICML, 2025. 2, 3, 7,9
Chen Zhang, Wei Zuo, Bingyang Cheng, Yikun Wang, Wei-Bin Kou, Yik-Chung Wu, and Ngai Wong. Ntk-guided implicit neural teaching. In CVPR, 2026.
Shuoxi Zhang, Hanpeng Liu, Stephen Lin, and Kun He. You only need less attention at each stage in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6057-6066. 2024b.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph F.. Gonzalez, and Ion Stoica. Judging auudeee with MT-Bench and chatbot arena. Advances Neural Information Processing Systems, 36, 2024a. 8
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Openrodelntepprteer: Integrating code generation with execution and refinement. arXiv preprint arXiv: /4658, 2024b. 8
17

---

Published as a conference paper at ICLR 2026
Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. Revisiting token dropping strategy in efficient bert pretraining. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10391-10400, 2023. 3
Yao Zhou, Arun Reddy Nelakurthi, and Jingrui He. Unlearn what you have learned: Adaptive crowd teaching with exponentially decayed memory learners. SIGKDD, 2018.
Xiaojin Zhu. Machine teaching: An inverse problem machine learning and an approach toward optimal education. In AAAI, 2015. 2, 3
Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N Rafferty. An overview of machine teaching. arXiv preprint
2018. 2,3
18

---

Appendix
Additional Discussions
20
A.1 Notation Overview
20
A.2 Functional Gradient
20
A.3 The Derivation of importance-adaptive Updates in the Parameter Space.
20
A.4 Attention Neural Tangent Kernel (ANTK)
25
A.5 AtteNT Algorithm...
27
Detailed Proofs
28
B.1 Proof of Theorem 3
28
B.2 Proof of Proposition 4.
28
C Experiment Details
32
C.I LLMs Training Setting
32
C.2 ViT' Training Setting
32
Additional Experiments
34
D.1 Ablation of Sample Ratio
34
D.2 Comparison to Established Methods
34
D.3 Visualizing NTK Analysis
35
E The Use of Large Language Models (LLMs)
36
19

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DISCUSSIONS
A.1 NOTATION OVERVIEW
Table 4: Summary Key Notations.
Notation
Description
Ssxd
Matrix containing all feature vectors from the ordered sequence Øs), with shape d
[ss.jld
T
d-dimensional feature vector for the s-th element, with components *
Short form for [xy]l
(s.)))
The s-th row of S, representing the feature vector for the s-th element
((,))
The i-th column of S, which represents the i-th feature across all ele- ments)
e,
The i-th basis vector, having a value of I at the i-th position and 0 elsewhere
s
Collection of all sequences
y
Property associated with the sequences, which can be scalar vector Space of sequential properties, represented R or TR
y
{aiJm
A set containing items
diag(a [11, am)
Diagonal matrix with diagonal entries ai..... am Diagonal matrix with repeated entries of Set natural numbers from to S
diag (a; m)
Ng = (1....,S) S
S")
A symmetric and positive definite sequence kernel Reproducing kernel Hilbert space (RKHS) defined by K Target mapping from S
H
Property (St) corresponding to the sequence St
A.2 FUNCTIONAL GRADIENT
Zhang al., 2023b;a present the chain rule for functional gradients, which is detailed in Lemma 5 (Gelfand & Silverman, 2000), and utilize the Fréchet derivative to calculate the derivative of the evaluation functional in RKHS, shown in Lemma (Coleman, 2012).
Lemma 5. (Chain rule for functional gradients) For differentiable functions G(F): TR IR that depend on functionals F(f)= m TR, the chain rule is given by
v(G(F()) OG(F(f)) VFF(f). OF(f)
(22)
Lemma 6. The gradient of the evaluation functional at the feature x, denoted as Ex(f) = f(x): - R, is given by Ex = K(x,)), where K(x,x') Rdd R + R
represents a feature-bused bused kernel.
A.3 THE DERIVATION OF IMPORTANCE-ADAPTIVE UPDATES IN THE PARAMETER SPACE.
Before providing the detailed derivation, begin by showing visualizations of general single-head attention learners. Figure 2a depicts a multi-output self-attention learner, Figure 2b presents a multi-output masked self-attention learner, and Figure 2c illustrates multi-output cross-attention leamer. The formulations for the masked self self-attention and cross-attention learners are presented in Equation 23.
fo(S) softmax ((S))(() + +M) V(S) Vd
Masked Self-Attentio::
Cross-Attention:
fo(S, softmax
(QSSKK(S V(S"),
(23)
where M E SxS a a strictly upper triangular matrix, with zeros on and below the diagonal and in every element above the diagonal.
20

---

Published as a conference paper at ICLR 2026
Cupe
(a) Self-attention
(b) Masked self attention.
(c) Cross attention.
Figure 2: An illustration of the workflow for different multi-output attention learners, with input sequence S and
S' (in the case cross-attention
Consider the derivative of an ANN with v 1, meaning that each component of the output foe (S) (S)) (.:)
is a scalar:
0f((S)
(ff(S)) 0f{(S)
0fo(S) 0fo(S)
(fo(S)
(24)
00
OWV dw
ow OWK
Owk
(.11
(:p) (.1)
(.p)
By applying the chain rule, we can compute the derivative of fo(S) with respect to the weight WV in the value matrix V(S).
0fo(S)
i @softmax
Q(S)
Vi
V(S)
OWV
i sootmaa
SW S
SWV
vd
OWV
SWoWKT S
softmax
S
Vd
QKT
softmax
S
Vd
(GG)AT//V exp
S,
(25)
T exp
((1,))CT /vd
where exp(-) denotes the row-wise exponential operator. The case of v 2 2 represents a multi- dimensional extension, which involves more complex notation but can be derived in similar manner.
21

---

Published as a conference paper at ICLR 2026
The derivative of fo( with respect the query weight matrix is more intricate. For i E Np.
(S)(S)
softmax
((S)
vd
V(S)
ow? (:.))
ow?
(::1)
i softmax
Sw s
SWV
vd
ow
Q
(:.)
Swee, T.,K WKT ST
i softmax
SWV
Vd
OWQ (5,5)
W W ((..) W (..)
ST
a softmax
SW
OW (1.1)
(8,)) W w TT (1)) (.))
@softmax
SW
OW4 (6,4)
8 W W.K. TsT
9 Sa. w
W s
sooftmax
Vi
SWV
aw
a S W. W. ST
(..)
(1.6) vid
(26)
(18.) w Vit
(ea) T&T
(6.)) W
w.K (5:3)
TST 9eoftmax
SWV
ow
W.K TS (ts,) W
Let's examine this row by row. For the j-th row (j E Ng) of Equation 26, it expressed as:
St.. W
W T T (ar)) (.1) Vi
Su. 2 WK
ta
(a))
ST
softmax
Vd
SW
ow
Q
Saa W.
W.K
TS
(:.0)
Vd
S W
W
TST
① softmax
(.))
vd
S..
WK
S
SWV
Vd
((A W
WK
ST
Vd
exp
1
Su. W
(.: ST
SG.
1
WK
T (1))
ST
exp
W 6.0
S
SWV
Vd
ad ad-1/22
WK ST
(.))
(..)
= Syg
1
WK T "T diag softmma ( (::)
Dº WK T (:))
ST /Va) sw
Vd
(:3)
WK TS "T (softmax(S we wK T (::8)
(:.)) (6)) (ST/Va) softmax We T
ST (TT/a) swv) (27)
(s3)
22

---

Published as a conference paper at ICLR 2026
The right-hand side of Equation 27 can be rewritten as
dx
1xd dx1
1xd
dxS
G.))
1/2 1/2 (s.8)
d
W
S
diag softmax
S ((ja))
We
(ir))
W T ST (va)
(s.))
s WW
1x1
Sxd dx1
1xd
SxS
1xd
1xd
1xd Sig
1xd
dxs
(.3)) 1x1
1/2
W
ST
softmaxx
W (i
we.
/Va)
softmax
W W (:4)
(1:)
S /Va)
W!
1xd
8x1
183
Sxd dill
1x1
1x8
Sya)* 1xd
d 1/2 ((.))
diag softmax @1.0 KGes)) /Va)
1x1
9x1
1x8
8x8
1x1
1xS
1x8
((.)
softmax 2G0 (aa))) /va)
softmax
2gi Keaa) /Vd) V
1xS
Sx1
1xS
8x8
541 (oottmaxQQuaakinn Quatkin)) T/va)) softmard (i \vR)) y
888
d
Kea) ding softmaax Ko. T T(vo)
(softmaax
(28)
By combining Equation 26, 27 and 28, we derive
(fo(S) ow
(:.4)
128
158
9x5 (sottmaxt (xxtmuxi(e..oF... //Vl)) softmax Qu..y \VVo)
d-122
ding (ootmmas Qus 1/V8)
V
8x8
a-112 Sigg Dia
Kgo (ine Cuan NV.)
(wottmoxl Ois (V2))
softmao (is, /V2)
(Keea dingf /Vai) Keaa
Coottanxxx /Vd))* softmass (e.s) /Vddy)
Skd
= diag Nes ddieg 1/v Ke.a) (softmaxx (2 (vR)) softtmax (Da. ( //VaV,
(
...., Keaa) ddigg(ssttmmaa suftmmsx (untnmxx (Caan
//V///////V (-li)vvKee Kesnn (softanxx (Daann 7/V)) Suttma(( //Va 5/Vd.
(29)
23

---

Published as a conference paper at ICLR 2026
The derivative of fo( with respect to the key weight matrix is derived in manner similar to that of the query weight matrix. For Np.
(S)
Owk (5))
i softmax
((S]A(S)
V(S)
vd
OWK (:.)
i softmax
SW T ST
SW
vd
OWk (:.4)
Sw"(W e:e," ST
ST
i softmax
SW
Vd
OWK (2.4)
((..) W. w.K. TS (1.)) Vi
sootmax
SW
aw (.4)
((5.) O w.k T V/
@softmax
SW
OWCK (.))
(1.) W w TST
91. w
6.0) Gi))
TST
8 soffmax
Vil
SWV
BWA
6.))
(11) va
(30)
(s. W W. TST G.a 6.9
(s. w
6.0)
(..)
T softmax
SWV
(3)
(1.)) (..)
Similarly, let's examine row by row. For the j-th row (j E Ng) of Equation 30, is
Sin. W W. T T
Su 2 WK ST softmax
(or)) (.1)
Vi
Va
SW
OWk (:.))
Saa W
W.K
TS
Vd
S W
W
TST
softmax
(.)
1
we T
vd
Sg.
ST
SWV
Vd
(1i)
((A W
WK
ST
Vd
exp
Su. W
(.: ST
S..
1 T
wQ
ST
exp
1/1/
W 6.0
S
SWV
Vd
(1.))
ad ad-1/22
W
WK ST
(.))
(..)
= Syg
1
we
T T diag softmaa (
0º WK T (:))
ST /Va) sw
Vd
(::)
(:3)
we T TST (softmax(SSu)) we wK T (::1)
(:.)) (6)) (ST/Va) softmax We T
ST (T//a)ssV swv) (31)
(s3)
24

---

Published as a conference paper at ICLR 2026
The right-hand side of Equation 31 can be simplified to:
dxs
1xd
dx1
1xd
dxs
S..) size:
d- w
S
diag soltmax S (
we
W
S (Va)
s wr
(1.8)
1x1
Sxd dxl
Ixd
1xd
Sxs
dxe
dxl
1xd
axs
dxs
dxs
(:8)
softmax (..) W
(.0)
W
8 /va))
softmux(
W (.9)
W
(G))
S /va)
S W.
IX1
1xd
9x1
1x3
1xl
1x8
= SG)
1/2
(1x)
diag softmax (G.) ((.)) /va)
1x1
9x1
1xd
1X8
9xS
1x1
1x8
1x1
1XS
1/2
(1.))
softmax Qu.) (G.)
/va)
softmax Qua Keo)) /vd)
1x1
8x1
1xS
8x1
1x8
128
125
d-12 Sug
Raa
ding (ing(solluussd Rua [D)
(sollmas RuayEaa /Va)) suduure celluns(Qeno /vl)
1xd
(32)
By merging Equation 30, 31 and 32, we get: (S)
ØWK (:.4)
301
1xe
d-122 Sua
Ono ding (otn 1 wa) V
Qua (nottaax(Qp..oec.. /va) nuftmerd On 1/va)
Ddi
1x8 divel sottmmee Disai /VB))
8x5
Siea
Qua Controx(Shenkinn /va)) "nimar((2ssa [V
1y @ne (inusaaattnx(iiissee. /Va)) Qua (soflmax (QaaaKaa) "Vaa)"suttnaall soltinn /Vajo)
5xd
diag
Qua
//aaa Qu.s) Soattmm(QQaaaaaaa /va)) softmax /vajy,
.... Qu. diag (sotttaax(Q /VD))V Qnis) [[oottmax Teaiaa///aa Waa) suftmaxx sultmas( /V8jy
5/Vd.
(33)
From Equations 25, 29, and 33, it can be observed that the ANN gradient for single sequence resembles that for a batch of feature vector inputs, as the gradient can be decomposed for each element. This demonstrates the parallelization-frienee nature of the attention mechanism from gradient perspective.
These results can be directly extended the case where each component of the output for (5) is vector, by considering a multi-dimensional setting (Zhang al., 2023a). The extension multi-heat cases can be done by broadcasting, which involves repeating the derivation in parallel many times as there are heads.
A.4 ATTENTION NEURAL TANGENT KERNEL (ANTK)
By incorporating the parameter evolution (i.e., Equation 7)
d
17
(Sx). HN)
Tafo(())
(34)
N 01
NS (St)
00
25

---

Published as a conference paper at ICLR 2026
Feature Matris
Query
Key KC
Valoe
V
V2
Figure 3: Graphical depiction of the ANTK computation process: Ko(Ss, S'gr)
(f.(8) (5.(S')
(
afa(S) (00)
(
0fe(s kga 4...4
(f) (L.))
0f6 k6 +
(W
03
(4)
aw aw
ow
(d.2)
aw (d.9)
0/o(s') laa ow
aw
(1.)
(1.1)
ow
(d.p)
aw
(d.p)
into the first-order approximation term (+) of Equation 14, derive
(.)
"
(o v1)
aL(for (SN).
fo (S,)
(x)
00
NS
as. (S))
0. (SN)
00
N
27
ac(.fo (S)
(S1).31)
ac(fo (Sv) yNN)
O. (-)
fo (S.)
NS
0fa (S1)
0f (SN)
00
00
N
OC(fa (S1)) (S1),11)
Oc(so (SN),
S (+)
0. (S.)
NS
ffr (S1)
Of (Sw)
00
ag
a Of( (S1), U1)
(SN).
[Kor +0l
(35)
NS
(Si)
OforSSN)
which leads to Equation 15 expressed as
Of
17
@f(ff
OC(fo: (SN). UN)
[K (St. +
(000 100
(36)
Øt
NS (Si)
0. (SN)
Øt
where the symmetric and positive definite Ko (S., =
is called the attention neural tangent kernel (ANTK) (Jacot et al., 2018; Yang, 2019: Hron et al., 2020). Specifically, ANTK for ((,)) and S' (i.a) is a scalar K K(S (c.)). (')))))
(S.) (frr() ag
80
Af (S),. agt
(See(()) Pm Figure 3 illustrates
0g:
the ANTK computation process, where typically, the length of all training sequences standardized to the maximum length. In simple terms, examining model's behavior by focusing on the model itself, rather than its parameters, often involves the use of kernel functions.
The quantity
ffr(() which represents the partial derivative of the ANN with respect to its parameters age
and appears in Kg (S:.-) (S.) (f) is determined by both the network architecture
and the specific parameters 0t, but it is independent of the input sequences. In contrast, the term #f. depends not only on the ANN structure and specific 0 but also on the input sequence 86
S. When the input for (S,) is unspecified, the ANTK simplifies general form K ()))).
26

---

Published as a conference paper at ICLR 2026
However, when specific sequence Sj is provided the input ffr matrix defined as K (S..
05 the ANTK becomes
This formulation aligns with the vector- valued kernel used in functional gradient descent (Zhang et al., 2023a), When the input sequence S; is specified, one argument of Kg= fixed, leading the ANN to update along Kgr (S: with the magnitude of the update determined by (S.) This process reflects the core mechanism of functional gradient descent. In summary, the ANTK and the canonical vector-valued kernel share consistent mathematical framework and exhibit similar effects the evolution of the associated ANN. Additionally, Theorem 3 establishes the asymptotic relationship between the ANTK and the canonical kernel used in functional gradient descent.
(S.) a6
(fa)(S)) 80
A.5 ATTENT ALGORITHM
Algorithm AtteNT Algorithm
Input:
Target mapping f realized by dense set of sequence- pairs, initial ANN foo, the size of selected training set m I N, small constant c and maximal iteration number T
Set for - foo, t
while t v T and II (S,) - (*( do
The teacher selects m teaching sequences:
/* Sequences associated with the m largest (l. - /" (S.))l
{St)m max Illfer (S,) - (S.)),
(S.)m
Provide { {S:)m* to the attention learner
The learner updates for based on received {S.) // Parameter-based gradient descent
Hr *" ss Es.ess)) EØ=1 VoL(fo (
Set t t ++1
end
27

---

Published as a conference paper at ICLR 2026
B DETAILED PROOFS
B.1 PROOF OF THEOREM 3
By examining the evolution of an ANN through changes in its parameters and from high-level perspective within the function space, obtain.
7
fc( (SL).1).
(C(fo.(SN) UN)
[K(S,, -N
NS
(Sa)
DJa:(SN)
OL( (Si),
(SN). UN)
(O a.
1000
=
NS
(S1)
ff (SN)
00 00
+
0t
(37)
Upon reorganizing, derive
7 (L())((S)).
OC(Fo (C()) (SN), yNN 0fo: (SN)
00
[K(S..)) - Kor (S;,
(38)
NS
(Si)
0t
By integrating the parameter evolution
O0
OC "
[ (S (Sx)
Tf (S:)
-1
(39)
0e
NS (Si)
0gt
N
into the remainder, get
(SN))
[K(S.,)) Kg -l
NS
Ofe (51)
(Sx)
[f(ss (S))), (S1),1))
(S uN)
Tafa((s)
(40)
NS
ff (S1)
(SN)
00e
When training an ANN with a convex loss L, which is convex in terms of fo but not necessarily in terms of 0, the following limit holds the vector: lim TaC (Si).n)) aL (Sw).un)
(fp)(S)) ..... 35gt (Sw)
0. Since the right-hand side of this equation is higher-order infinitesimal relative to the left, maintaining this equality results in the conclusion that
lim (K(S.,)) Ko (St, .IN 0.
(41)
This suggests that for each training point, Le., input sequence S E (S,)N, ANTK converges pointwise to the canonical kernel.
B.2 PROOF OF PROPOSITION
Referring to the definition of the Fréchet derivative in Definition 2, the convexity of C implies that
0c
ac
Se
(42)
Ot
0fat1
Ot 74
28

---

Published as a conference paper at ICLR 2026
By computing the Fréchet derivative of ff side, Y, can be expressed as
ac
and the evolution of for, the term the right-hand
r (g
" )
([
[OC( [ (S1),91) (11))))))))))111
aL(So-1 (Sx), UN)
7
[Kslly
N2
0fau: (S1)
(S
(
(SN))
[K
ifo (S1)
Dfo((SN)
H
ac(fo (1)
OC(f (SS))
{[Ks.N (Ksl))
N
0f (Si)
0fe (SN)
OL (S1), y1)
ØC(So (Sx),
0 Of (Si)
(SN)
J (S1),y1)
aL(Jo+1 (Sw) UN)
To (Si). Of (S,)
@C(Jo (S()) (SN),NN)
K
NS
0p (81)
(foti((S)
(SN)
(43)
where K K/ (NSS, and K an NSXNS symmetric, positive definite block matrix with elements (S,, S,) positioned in the i-th row and j-th column block. For convenience, we use simplified ((fo;;
column vector notation
= |0,,0 ((au; ,,,
with 01,0 C(fou; Sw)
01,0 ((fo:: = 2L(. (1,)
The last term in Equation 43 can then be rewritten as
10g, C(for; S.) K 15ge ((for11; S:))
NS
7
(0, C(Sor; (.)) T, K (105, [[[Jort;; S:)) ((or:S)) [0 C(Jo;; 8i)
NS
"
[0,, C(for (S)) K 10 Si)]
NS
"
[0,,, C(fo;; ;.)] K (10 ((fo+t;; [0,,, C(for; ((for: S.)l
NS
")
10,, C(fo;; 8.) N- K [0 [(Sor;; Si)l N
NS
NS (10/.11 (10 C(Jo+:; 1 [ - [(orr S.) - ((o+1;
"
K.
(10 (10 f(Ser-1; S:)], N [0,,, L(fer: 5:)]
(44)
The last term in Equation 44 above can be expanded to
NS (1o ( S()
"
10,, ((oo
((far::)
K
(10 C(for1;; St)] - 10,,, C(fo; (()o 5.))
"
(10 C(fet:: 5i)] [0 (foo n) K (10 ;( S5:):
[0 ((So;
=
NS
7
10 fete C(far (:)) K (10)aces C(far11; - 10/0 C(for: S:) Si)l.N) IN)
NS
7
[0 L(fott:; S.) - 0 L(fa: 8.)]. K [0ror E(fget1; - -0g L(for;
NS
(0/ [0 ((for11;5))]
[a,,, C(fo= [:)o:::))) K (1)/00) ((fo::))) 2
[0,, [(fo; S.)]
NS
10 L(Se; S K [0,,, C(Jor; S:) N
(45)
4NS
Given that K is positive definite, it follows that
77. NS
(D/mn ((fo+1;))) 10 -(J:: :.o::) S.)l: K (1omn (1omnn [[o++1; 1 [0 L(Jo: (()::))))
1
29

---

Published as a conference paper at ICLR 2026
non-negative term. Therefore, by merging Equations 43, 44, and 45, we derive
31/ ANS 10 ((So K [ C(Jo::::))
r
+ NS
7
ffors 11) 0z. [(fo: 8.)) K 10fert ((for11;))) - -000 G(for; S
(46)
Based on the definition of the evaluation functional and the assumption that C is Lipschitz smooth with constant O, the term V in the final part of Equation 46 is bounded above follows:
D1acts (:fo - -, L(for; S.] K [ (foni - -O,, C(fer; (()));S:))
@D((o+))) OL(So: ) K Ess, (aC(Jo+)) OL(Je:)
= E.s. Öfocn
Ofar
Ofocn
Ofa.
S 72 [Es, (foct1 No K [Es, (fatt - fo.)lN
72 (1fo11 for) (1Ks.lt), K -IKKlN (for11 fo-))
17 [0,, L(for; S.)) ([Ks,N (K K. ([Ks.N IKs.IN). [0,, C(for: So)]
NS
NS
(47)
Given that the canonical kernel is bounded above by constant y > have
[K [KssIN.[Ks.ll), 7 (INS, 1s),
and
K S
Y (INS, 1Ns) NS
Therefore, t is bounded above by
7
(10 [(fo: 8.) N INs) (INs: ((fo;
NS
2
Y
, C(for; Sugan)
S(ga))
(48)
NS
Moreover, the last term Equation 47 is also bounded above:
([Ks,IN [Ks K.
(IKs,ly [Ks,)))
17272 10 ((foc;S.)
[O L(for: Si) N
NS
NS
N
N
<
S L( Si(zz))
K
L(Ser Si(ia)) For
NS
NS
1=1
1=1
1=1
j=1
N
122
1
N
S
N
<
0y.1 C(for: Suciaa)
.IN
1
3 C(fe: St(a))
NS
NS
NS
1=
1=1
N
i=1 j=1
2
T22333
Σ Σ
0j C(for S(c::))
(49)
NS
Thus, by combining Equations 46, 47, 48, and 49, we get
r < mm 3 - p2?ppp
S
Syat C(for SeGia)) (1ie))
(50)
NS
=
30

---

Published as a conference paper at ICLR 2026
which means
0C 3
N E
0t - r - m
Σ
Dzor C(fo: S.(ta))
(51)
4
NS
i= j
Hence, if 1 S - 27 follows that
2
dL (Jor (oo ViGa)
Oc
S
7y 2
Sg (co:
1
(52)
at.
NS
NS
0fe: (S.)
[=1
j=1
31

---

Published as a conference paper at ICLR 2026
C EXPERIMENT DETAILS
C.I LLMS TRAINING SETTING
All experiments were conducted on 4 NVIDIA A100 (80GB) GPUs. We employ LoRA fine-tuning following the Alpaca (Taori et al., 2023) and Pizza (Meng et al., 2024) implementation strategy. Specifically, we optimize with A using a batch size of 128, learning rate of 2e-5, cosine annealing scheduling (Loshchilov & Hutter, 2017), and warmup ratio of 0.03, without weight decay, The training objective computes loss only over responses from the selected datasets. We configure LoRA with lora_al aaa_alhhallora_s set lora_ dropout, and insert adapters into all linear layers of the base model. Both the backbone and adapters are trained in Float32 precision.
AtteNT Setting In this experiment, we adopt straightforward variant of AtteNT: the model is trained on the full dataset during the first epoch, after which only the AtteNT selected subset is used in subsequent epochs. Selection is guided by the per-sample loss scores within each epoch, effectively directing the model's attention toward harder examples. Since pretrained models already perform well on most instances, emphasizing more challenging data in later epochs is expected to yield greater fine-tuning benefits. Following prior findings in Rho-1 (Lin et al., 2024), we set the selection ratio to 70%.
Dataset Building The ImageNetS50 dataset is derived from the ImageNet- Ik benchmark, and its construction requires access to a local copy of ImageNet-Ik. Following the official repos- itory (Gao et al., 2021), we generate ImageNerS50 by running data_preparan rapt on. sh with the option -mode=50. Semantic segmentation annotations are obtained using datapreparation_nnee
sh. For depth annotations, we employ the Mask2Former (Cheng et al., 2022) framework, utilizing its released code and pretrained models to generate pseudo-labels. In addition, we directly download the full NYUv2 dataset, where the official semantic segmentation and depth test sets are used for evaluation.
C.2 VITS TRAINING SETTING
We adopt ViT-B (Dosovitskiy et al., 2020) with a 16 x 16 patch size as the backbone for our MAE experiments and evaluate performance on ImageNet-S50. Training is performed using Adam with a base learning rate of 1e-4 and weight decay of 0.05. The learning rate is linearly warmed up for 40 epochs, followed by cosine decay scheduling (Loshchilov & Hutter, 2017). We train with batch size of 2048 on 4 A 1100 GPUs, leveraging automatic mixed precision for efficiency. Data augmentation is limited standard transformations: random cropping with scale sampled from [0.2, 1.0] and aspect ratio from [0.75, 1.33], resizing to 224 x 224, and random horizontal flipping with probability 0.5. A full specification of hyperparameters for pretraining and fine-tuning provided in Tables 5 and 6.
Table 5: Hyperparameters for pre- pre-training Multi- MAE.
Hyperparam
Baseline
AtteNT
Batch Size
2048
2048
Learning Rate
lc-4
lc-4
Min Learning Rate
1c-6
1c-6
Weight Decay
0.05
0.05
Adamw
le-8
le-8
Adamw 91
0.9
0.9
Adamw 88
0.95
0.95
Epoch
800
800
Warm up Epoch
40
40
Learning Rate Schedule
cosine decay 98
cosine decay 98
Non-masked tokens
Input resolution
224x
224x224
Augmentation
RandomResizeCrop
RandomResizeChre
Dropout
0.0
0.0
Patch Size
16
16
Selection
[None] )
|Random, Hard, Soft
32

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for fine-tuning Multi-Modal MAE on various downtasks. The augmentation strategy LSJ is large scale jittering (Ghiasi et al., 2021). We use drop path (Huang et al., 2016) in classification and semantic segmentation tasks.
Hyperparam
ImageNetS50
NYUv2(S)
NYUv2(D)
Epoch
100
2000
Warm up Epoch
5
20
100
Batch Size
1024
1024
2048
Learning Rate
4c-3
lc-4
1c-4
Min Learning Rate
Ic-6
Ic-6
Weight Decay
0.05
0.05
le-4
Adamw $1
0.9
0.9
0.9
Adamw 3g
0.999
0.999
0,999
Layer Decay
0.65
0.75
0.75
Patch Size
16
16
Drop path
0.1
0.1
LR Schedule
cosine decay 224 224
cosine decay
cosine decay 256 256
Input resolution
224 x224
Augmentation
Rand(9, 0.5)
LSJ
LSJ
AtteNT Setting We employ an enhanced AtteNT strategy that dynamically selects training data based on per-sample loss scores. Specifically, data selection in the first epoch is guided by each sample's ' initial loss, and the selection is periodically updated by recomputing loss scores after fixed intervals. This updated subset is then used to initiate the next training stage. Morcover, incorporate a dynamic selection ratio from 20% to 80%, following the adaptive scheme proposed by (Zhang et al., 2023b). As demonstrated in Section 5, this approach achieves favorable trade-off between efficiency and performance.
Dataset Building
All datasets used our experiments are available on HuggingFace:
Mathematical Reasoning: Training meta-matin meta-math/MetaMae MetaMMttMMM evaluation /penai/gsm8k and Sendrycks/MATH
Code Generation: Training on m-a-p/ --pppooeFeeebaccc (restricted to Python samples);
evaluation openai/humaneval and google/mbpp.
Multi-Tumn Dialogue: Training on WizardIM/evol instruct 196k; evaluation on Emsys/mt-boo
33

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
D.I ABLATION OF SAMPLE RATIO
Table Sample Ratio Study ViT models.
Ratio
100
90
80
70
60
50
40
Tasks
ImageNetS50
92.2
91.8
91.4
85.6
78.8
64.5
58.2
NYUv2(S)
51.9
51.1
50.2
46.8
38.2
29.1
21.9
NYUv2(D)
52.1
52.2
51.6
48.3
42.4
36.3
30.5
ImageNet550
90
NYUv2(S)
NYUv2(D)
80
Downstream Metrics
70
60
50
100
90
80
70
60
50
40
Sample Ratio
Figure Downstream Task Performance Sample Ratio.
In this section, investigate the impact of the AtteNT algorithm's 's sample ratio downstream tasks. In Table 7, we compare the results based on the ViT model using different fixed sample ratios.
from Fig 4, there is noticeable drop in performance around the 80% selection ratio. This suggests that, during training, a portion of the data remains relatively unchanged and contributes less to performance when its selection ratio falls below certain threshold (Lin al., 2024; Katharopoulos & Fleuret, 2018). When the ratio of unselected samples during training is lower than this threshold, the model can maintain its performance. Interestingly, small data drop can even act as a form of noise reduction.
As
D.2 COMPARISON TO ESTABLISHED METHODS
Table Performance Comparison of Different Methods.
Methods
Time(J)
ImageNetS50(1))
NYUv2(S)(1)
NYUv2(()(())
AOueNT(Ours)
980m
92.3
52.6
57.2
Class Weight Sampling
1108m
90.4
48.2
52.0
Fixed Weight Sampling
1065m
89.6
49.7
54.6
GradNorm Sampling (Chen et al., 2018)
1112m
91.9
52.4
34

---

Published as a conference paper at ICLR 2026
As shown in Table 8, also performed experiments comparing AtteNT with three simple yet representative sample-sclection baselines to compare with traditional greedy algorithm: The first method, Class- Class-Weight Sampling, assigns sampling weights inversely proportional to the number of samples per class, aiming to encourage the model to treat all classes more equally (sampling weight 11 class frequency). The second, Fixed- Weight Sampling, assigns fixed sampling ratios based on prior beliefs about task difficulty, In our setting, we consider the classification task easier than semantic segmentation and depth estimation, so we reduce the sampling rate for the RGB modality and set fixed sampling weights to :
2: 2 (RGB SemSeg Depth). The third method, GradNorm Sampling (Chen al., 2018), dynamically adjusts the sampling weights for data groups (RGB, SemSeg, Depth) based on their gradient contributions during training. All methods were trained for 800 epochs, with the total sampling budget fixed at 70% for each baseline.
Across all comparisons, AtteNT consistently achieves higher efficiency and stronger predictive performance. These results indicate that AtteNT's gains do not arise from generic greedy sam- pling heuristics, but from its principled nonparametric teaching mechanism, which adapts to model uncertainty and task interactions more effectively than existing selection strategies.
D.3 VISUALIZING NTK ANALYSIS
To empirically confirm that the neural tangent kernel quickly stabilizes in real vision transformer training, we track the NTK on 10 fixed training points during an 800-epoch run of the Multi-Modal MAE backbone:
Figure 5: Frobenius norm of the difference between the empirical NTK at epoch and the canonical kemel. The difference falls sharply within the first 50 epochs and stays near thereafter.
Figure 6: Heatmaps of the 10 10 NTK selected checkpoints. clear pattern is already visible at epoch 139, and from epoch 219 onward the heatmaps are virtually identical and remain unchanged through epoch 799.
These quantitative and qualitative results jointly show that the empirical NTK converges extremely rapidly (within 50-200 epochs) and remains close to the canonical kernel for the rest of training.
1.50
Frebenius Narm of Difference
1.00
0.50
0.00
100
200
300
400
500
600
700
800
Figure 5: Frobenius norm of the difference between the empirical NTK at different training steps and canonical kernel.
35

---

Published as a conference paper at ICLR 2026
Crech 138
Eaech 339
Croch 206
Ezock 379
Epock 400
Erech
Eaach 659
Epech
trech
Figure 6: Evolution of the empirical 10 10 NTK matrix during training. Color represents value
Kg (St, Sj). The matrix stabilizes visually after 200 epochs and shows negligible changes thereafter.
E THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used large language models for language polishing, such grammar and phrasing. And also use AI to assist with code completion. All research ideas, methods, analyses, figures, tables, and conclusions were solely developed by the authors. The authors take full responsibility for all content.
36


Reviews:
[{"review_id": "0rVfauEFXC", "reviewer": "Reviewer_e8RB", "summary": "This paper introduces AtteNT, designed to improve the efficiency of training attention-based models. The authors reinterpret the standard training process through the lens of nonparametric machine teaching, where a teacher strategically selects a subset of training examples to accelerate the learner's convergence. The core theoretical contribution is establishing a formal connection between the parameter-based gradient descent used for attention networks and functional gradient descent in an RKHS, showing that the Attention Neural Tangent Kernel (ANTK) asymptotically converges to a canonical kernel used in nonparametric teaching. This insight justifies a greedy data selection algorithm that prioritizes samples with the highest error.", "questions": "1. There are multiple places that authors mention the convexity of loss. To make it more clear, does loss need to be convex with respect to $f_\\theta$ or with respect to $\\theta$? The first one seems a reasonable assumption while the second one is very limiting.\n\n2. Another clarification question, in Table 3, there is no effect of AtteNT, is that right? Also, it was not clear from the main text what ratio and interval are referring to.\n\n3. A small typo: Eq 17, I believe it should be $K_{\\theta_t}$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The primary strength of this paper is the novel and elegant theoretical bridge it builds between attention learning and nonparametric teaching.\n2. The theoretical insights are translated into a simple, intuitive, and practical algorithm. The idea of focusing on samples with the highest error (i.e., the hardest examples) is a well-known concept, but this paper provides a nice theoretical justification for it.\n3. While the theoretical analysis is limited to a single layer but the authors show that empirically their approach and intuition works on more realistic scenarios of vision and text in scale.", "weaknesses": "1. From a practical point, additional to attention layers, the models that are used also include complexities like residual connections, layer normalization, and multiple non-linear blocks. The paper does not address how the theoretical findings generalize from the simple model to these complex ones.\n\n2. The setup assumes a noise-less case. Eq 21 and its use in Algorithm 1 seem to be sensitive to this assumption. It is not clear or discussed how the analysis for nonparametric teaching performs with the noisy data which is more practical. A greedy data selection based on maximum of error could lead the model to choose these noisy samples, as they would consistently produce high loss. The paper does not discuss the robustness of AtteNT to label noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zmu7HV63mY", "reviewer": "Reviewer_jFUw", "summary": "This paper interprets the parametric attention learning process from a nonparametric teaching perspective, which motivates an AtteNT algorithm that uses sample selection to accelerate the attention learning process. Specifically, the paper shows that the dynamic ANTK in parametric gradient descent converges to the importance-adaptive canonical kernel of functional gradient descent, bridging the parametric attention learning and nonparametric teaching. Theoretical results show that applying the canonical kernel can lead to sufficient loss reduction. The paper thereby proposes an AtteNT algorithm to improve the attention learning efficiency via sample selection. Experiments in both LLM and CV scenarios show the improved performance and reduced training time after applying AtteNT.", "questions": "1. Will the number $m$ of selected samples influence the performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper makes a contribution to accelerate the attention learning, which is an important problem in LLM and CV model learning. \n\n2. The motivation of the AtteNT method has theoretical justification, and the experimental results further validate the effectiveness of the method.\n\n3. The experiments are conducted over multiple LLMs and vision models, ablation study is included.", "weaknesses": "1. Some experimental settings are unclear to me. Since the paper claims the reduction of training time, which metric is used to compute the training time with and without AtteNT? Specifically, how to decide when to terminate the training for each method? Is the sequence selection time included in the model with AtteNT?\n\n2. In Table 1 and Table 2, adding AtteNT also improves the learning performance. Why AtteNT can lead to such performance improvement is not sufficiently explained in the paper. The theoretical analysis shows the connection between the parametric and functional gradient descent, but does not include the extra benefit of using non-parametric teaching. It could be helpful to have some explanation on why adding AtteNT can lead to even better performance than traditional learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqqwyXN3rX", "reviewer": "Reviewer_csfV", "summary": "The authors suggest that attention models can be made more sample efficient by selecting a subset of the training data, that maximises the functional gradient (approximated by the L2 norm between the target and the current estimate). This idea of greedy selection is grounded in non-parametric teaching, where the best possible training set is derived for achieving a target function with a non-parametric method. In this case, the authors connect the NTK of attention to an instance-importance weighted sequence kernel rate. They show that this connection eventually justifies the greedy data selection strategy. They validate their results empirically, showing that this strategy works for a variety of smaller language and vision models while maintaining accuracy.", "questions": "- see weaknesses\n- The Table 3 caption could be completed with a short description of the experiment.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting combination of non-parametric theory with parametric learning.\n- The results of the final method seem convincing for the given examples.\n- Extension of existing method.", "weaknesses": "- I had a quick look at the code and it is not well-documented, one would have to invest some time to understand how to reproduce their results.\n- I find the paper generally difficult to parse, but perhaps this is just my lack of background. In particular, Section 4.1 could be improved by giving some intuition and describing under which circumstances the kernel would not be adaptive in terms of $\\omega_j$, but would require higher-order importance weights.\n- For Theorem 3, how important are your assumptions (large scaling, small learning rate)? Is there a way to verify if or when the kernel limit holds?\n- Similarly, in an experiment, if possible, can you actually verify that you stay in the kernel regime holds (by computing the theoretical kernel and comparing it with the empirical one)? Or does the training diverge from it in practice, and does this correspond to the case where the adaptive selection strategy is useful?\n- For the experiment section, it would be useful if you discussed in how far the architectures also still fall under the setting you discussed previously, or how they diverge from it. Also how the setting compares with finetuning.\n- What is the variability in the accuracy and runtime results (e.g. Table 1) - since you refer to average values, are you already starting several seeds?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3rZxlmYc7", "reviewer": "Reviewer_GNaF", "summary": "This paper addresses the high training cost of attention learners (e.g., Transformers, LLMs, ViTs)—neural networks leveraging the attention mechanism to learn implicit sequence-to-property mappings—by proposing a novel paradigm called Attention Neural Teaching (AtteNT).", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Prior nonparametric teaching research was restricted to multilayer perceptron (MLP)-based learners and failed to account for the attention mechanism—an essential component of modern neural networks (e.g., Transformers, ViTs). This paper removes this critical limitation by systematically investigating the role of attention in parameter and function spaces, proving that attention learners’ training dynamics align with nonparametric teaching principles .\n\n- It introduces the Attention Neural Tangent Kernel (ANTK) and proves (Theorem 3) that dynamic ANTK converges to the \"importance-adaptive canonical kernel\" in nonparametric teaching. This is the first work to establish consistency between parameter-space gradient descent (used by attention models) and functional gradient descent (used by nonparametric teaching), resolving a fundamental mismatch that prevented prior application of nonparametric teaching to attention learners .\n\n-  The AtteNT paradigm is not a trivial adaptation of nonparametric teaching—instead, it leverages attention’s \"importance-adaptive updates\" (element-specific weights derived from Q/K/V interactions) to design a greedy sequence-selection strategy.", "weaknesses": "- Theorem 3 only proves asymptotic convergence (i.e., when training iterations approach infinity), but real-world attention model training uses finite iterations (e.g., LLM fine-tuning typically runs 5–100 epochs, ViT pre-training 800 epochs as in Table 2). The paper does not:\n\n  - Derive a quantitative bound on convergence speed (e.g., how many iterations \\(t_{\\epsilon}\\) are needed for ANTK to be within \\(\\epsilon\\) of the canonical kernel);\n\n  -  Validate via experiments how ANTK evolves over practical iteration counts (e.g., a plot of ANTK similarity to the canonical kernel across 800 ViT training epochs).\n\n- The paper fixes the sample selection ratio (fraction of sequences chosen by AtteNT) at 70% for LLMs (Section C.1) and uses unspecified \"dynamic ratios\" for ViTs (Section C.2), but provides no analysis of how this ratio impacts performance. For example:\n   - Would a 50% ratio further reduce training time but hurt accuracy?\n   - Would an 80% ratio retain accuracy but minimize efficiency gains?\n\n- The paper’s ablation studies (Table 3) only compare AtteNT’s \"Random/Soft/Hard\" selection strategies (all derived from nonparametric teaching) but fail to contrast AtteNT with established sample selection methods for attention models. This makes it impossible to determine if AtteNT’s gains stem from its nonparametric teaching foundation or from generic greedy selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)