Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation
Authors: Yilang Zhang
Abraham Jaeger Mountain
Bingcong Li
Georgios B. Giannakis
Venue: ICLR 2026 Poster (2026)
Keywords: Meta-learning
meta-gradient estimation
bilevel optimization
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Meta-learning offers a principled framework leveraging *task-invariant* priors from related tasks, with which *task-specific* models can be fine-tuned on downstream tasks, even with limited data records. Gradient-based meta-learning (GBML) relies on gradient descent (GD) to adapt the prior to a new task. Albeit effective, these methods incur high computational overhead that scales linearly with the number of GD steps. To enhance efficiency and scalability, existing methods approximate the gradient of prior parameters (meta-gradient) via truncated backpropagation, yet suffer large approximation errors. Targeting accurate approximation, this work puts forth binomial GBML (BinomGBML), which relies on a truncated binomial expansion for meta-gradient estimation. This novel expansion endows more information in the meta-gradient estimation via efficient parallel computation. As a running paradigm applied to model-agnostic meta-learning (MAML), the resultant BinomMAML provably enjoys error bounds that not only improve upon existing approaches, but also decay super-exponentially under mild conditions. Numerical tests corroborate the theoretical analysis and showcase boosted performance with slightly increased computational overhead.

Published as a conference paper at ICLR 2026
BINOMIAL GRADIENT-BASED META-LEARNING FOR ENHANCED META-GRADIES ESTIMATION
Yilang Zhang"), Abraham Jaeger Mountain", Bingcong Li & Georgios B. Giannakis'
'Department of Electrical and Computer Engineering University of Minnesota Minneapolis, MN 55455, USA (2han7453, jaege252, georgios) @umn. edu
*Deparmment of Computer Science ETH Zürich 8092 Zürich, Switzerland (bingeong. 1i @inff www.ff.......mmmmm
ABSTRACT
Meta-learning offers a principled framework leveraging tusk-invrriat priors from related tasks, with which task-specific models can be fine-tuned on downstream tasks, even with limited data records. Gradient-based meta-learning (GBML) relies gradient descent (GD) to adapt the prior methods incur high computational overhead that scales linearly with the number of GD steps. To enhance efficiency and scalability, existing methods approximate the gradient of prior parameters (meta peeetaggadient) via truncated backpropagation, yet suffer large approximation errors. Targeting accurate approximation, this work puts forth binomial GBML (BinomGBML.) which relies on a truncated binomial expansion for peeaaggadieent estimation. This novel expansion endows more in- formation in the meta- gradient estimation via efficient parallel computation. As a running paradigm applied to model-agnose meta-leariing (MAML). the resultant BinomMAML provably enjoys error bounds that not only improve upon existing approaches, but also decay super-expee super-exponentially under mild conditions. Numerical tests corroborate the theoretical analysis and showcase boosted performance with slightly increased computational overhead.
task. Albeit effective, these
INTRODUCTION
Deep learning (DL) has well-documented success in recent years across various domains (Vaswani 2017, He et al. 2016 Brown et al.) 2020). With its merits granted, the effectiveness of DL. relies markedly on high-dimensional models that require massive datasets for training. In applications where data are scarce, acquiring massive training datasets can be prohibitively expensive. Examples of such "data-limited" regimes include medical imaging (Varoquaux Cheplygina 2022). social
sciences |Grimmer et al. 2022). and robot manipulation (Kroemer et al.] (2011).
As opposed to data-hungry DI., a hallmark of human intelligence is the ability to exploit past knowledge swiftly acquire skills through limited experiences and trials. Meta-learning aims to impart humans' knowledge, which can be subsequently transferred aid the learning of new tasks, even when training data are limited. Akin to human learning from past experiences, meta-learning accumulates the sought prior knowledge from a set of related tasks. Model- based approaches, which attempt to encode prior task knowledge in some additional meta-NN, have seen some success, but are prone to poor robustness and require hand-tooled design.
ability into DL by identifying some task-invariant prior
In contrast, gradient-based based meta-lcarning (GBML) uses iterative optimizer, such as gradient descent (GD), to train task-specific models, with prior information embedded in the learnable hyperparameters of the optimizer. By learning efficient optimization hyperparameters,, the model can rapidly adapt to
"qqull contribution

---

Published as a conference paper at ICLR 2026
new tasks in few optimization iterations (Andrychowicz al. 2016). A representative example is model-agnostic meta-learning (MAML) (Finn et al. 2017). which encodes the prior using an initialization shared across tasks and relies GD to adapt the task-specific models. By starting from an informative initial point, the model can easily converge to some local minimum even with limited training data. Building upon MAMI., many GBML variants have been developed to further improve performance (Yoon et al. 2018 Khodak et al. 2019 Nichol 2018 Raghu et al. 2019).
While GBML has gained popularity in various application domains (Zhu et al. 2020; Zang et al. 2020 Chen et al. 202T) the process of learning the prior can incur high computational complexity, In particular, the complexity for calculating the gradient of prior (a.k.a. meta-gradient) is linear with the number of adaptation steps, rendering these algorithms barely scalable. To lower the computational overhead of GBML, meta-gradient estimators have been investigated as an alternative vanilla full backpropagation. For instance, first-order approaches (Finn et al., 2017;; Nichol 2018) do not make explicit use of higher-order information. However, this siver-simliiiication can lead large estima- tion errors, thereby decelerating meta-learning convergence and resulting in degraded performance. For desirable trade-off between accuracy and complexity, truncated backpropagation (Shaban et al.) 2019) is advocated, in the more general context of gradient-basse bilevel optimization, to retain specific portion of second-order information. Another line of work (Rajeswaran et al. 2019 Ravi Beatson 2019 Zhang et al., (023) utilizes implicit differentiation to replace full backpropagation. but the implicit function theorem (Krantz & Parks 2002) involved relies markedly on the stationarity of the solution found by the iterative optimizer.
While first-order and truncated backpropagation lessen the computational load of GBML, we will demonstrate with analysis and empirical observations that they can lead to large approximation errors. With this motivation, we revisit the GBML meta-gradient formulation and develop a novel meta-gradient estimate, binomial gradient-based meta-learning (BinomGBML), which earns its name from the keystone binomial theorem. Our contribution three-fold:
Embracing efficient parallelization, BinomGBML improves upon state-of -ate-offthheaeee estimators by retaining more information to markedly reduce the estimation error.
the-art art meta reta-gradient
Theoretical analyses under three different assumptions reveal BinomMAML's reduced estimation error, which decays anper-exoneentially. Extensive numerical tests on synthetic and real datasets further validate these claims.
As side benefit, when combined with MAML (Finn et al. 2017). BinomMAML addresses the memory scalability of MAML through dynamic management of computational graphs.
Notation. Bold lowercase (uppercase) letters denote vectors (matrices); || and (.-) stand for norm and inner product; and [:], for the i-th entry (column) of vector (matrix).
PRELIMINARIES
2.1 PROBLEM STATEMENT
Meta-learning first acquires task-invariane prior parameterized by 0 E Rº that is common set ==
of related tasks. Let t = I,..., T index these tasks, and := [(X], v²) denote the per-task datasets comprising N, (limited) input-label pairs. For compactness, let matrix X, E RRdxN, be
formed with columns xV and y. R.N. be the corresponding label vector. Each D, is partitioned into training subset Dfr" D, and a validation subset Di := D, Dirm Instead, the meta-training process first extracts a task-invariam prior from these tasks by leveraging their associated data. Then, to measure the quality of the meta- veta-trained prior, an unseen meta-testing task indexed by is provided with limited training data Dtrn, and some test inputs (x") ). As N, can be markedly smaller than required for DL, directly training model over Dim could easily lead overfitting. The meta-testing process transfers the acquired prior to the new task, at which point a DL model can be fine-tuned over the small Diur predict the corresponding test labels {
The meta-training process amounts to a nested empirical risk minimization problem. The inner level (task level) trains the per-task model parameter Ø Rd over D using the task-in inv-iinar prior provided by the outer level. The outer level (meta level) assesses the trained model on Dr and adjusts the prior accordingly. Recall 0 E RD is the shared prior parameter (a.k.a. meta-parameter).
2

---

Published as a conference paper at ICLR 2026
This nested structure is naturally expressed as bilevel objective
min
(0))
(la)
Φ (0) arg min ft (Φ,) R(d;;
(1b)
where (0)) == fum ( Dim) == - log Φ ". is the training loss (negative log- likelihood) evaluating the fit of the task-specific model to Dim, and regularizer R( 0) := log pl Φ 0) represents the negative log-prior probability density function. Using Bayes' rule, one can verify that Φ (0) arg min - log
0) is the maximum posterioni estimator.
While formulation 1 is appealing, the global optimum o; in (Ib) is generally intractable for highly nonlinear NNs. A feasible alternative is rely on a manageable solver approximate d;
Gradient-based meta-learning (GBML) poses the solver as a few cascaded optimization steps. The meta-parameter 0 acts the optimizer hyperparameters shared among all tasks. The first work along this line is termed model-agnostic meta- veta-learning (MAML) (Finn et al. (017). It relies K step GD optimizer as the solver, with task-invariant initialization being the meta-parameter (thus D d). which renders the alternative objective
T
min T s.t.
Σ 1=1 の
(0)
= +
1 Σ
val ( K (0))
(2a)
1=1
Φ. ( Φ (0) - a (o) k k.....,
(2b)
where 0 is constant learning rate. Although 2 contains explicit R, it is pointed out that under second-order approximation (Grant et al. 2018 2b satisfies
K Φ ( (0) arg min p!Tm (Φ.)
11. 0
where A; is determined by a, K, and 2 In other words. MAML's optimization strategy is equivalent to an implicit Gaussian prior p:: 0) = (dd 0, A.), whose mean is the task-invariant initialization.
2.2 COMPUTATIONAL CHALLENGES IN META- LEARNING
This section examines the time and space complexities of GBML. For illustration, the discussion will particularly focus on MAML, while similar analysis can be readily applied to other GBML methods including (Li et al. 2017 Lee & Choi 2018 Rusu et al. 2018 Park & Oliva 2019).
Optimizing 0 in requires computing the gradient
K
K
(C, Vo eyal
b (Ø)
V
(K)
[ a
(Ø,
(3)
where (a) utilizes the chain rule, and (b) is from 2b For compactness, let H denote the Hessian, and g. e ( K) the validation gradient. It follows from that
(Ø)
K-1
(C)
II k=
La
oH g.
(4)
For any vector E Rd, one can compute [La aH by first expanding
and then using the Hessian-vector product (HVP) to efficiently obtain H = V (
As result, O(Kd) time and space are required to compute 4 through backpropagation.
Note that acquiring oe via (2b) also incurs O ((d) time complexity. but the constant hidden inside O is much smaller than (3). Thus, the overall time complexity is dominated by the backpropagation process (Griewank) 1993). Given that GD converges sub-linearly when the gradient is Lipschitz,
3

---

Published as a conference paper at ICLR 2026
requires = (1/ec) iterations for of to reach e-stationary point (Bertsekas, 2016). This implies that a sufficiently large K is required to accurately solve 2), raising concerns about the scalability of MAML its time and space complexities both grow linearly with K.
To address this issue, first-order (FO)MAML (Finn et a 2017) was proposed alongside 'vanilla MAML, providing simpler but less accurate approximation (0) FOMAML discards all second-order information by simply setting H = Oaxd, V k, thus resulting in the (d)) time and space estimate
PrO ()) gg
This "brutal" simplification heavily prioritizes reduction computations at the expense of estimation accuracy, which can degrade model quality and slow meta-training convergence (Sow et al. 2022). different first-order approach, Reptile (Nichol 2018), proposes the meta-update rule <
<T ( 1Ø - 0). with E (0, IJ, rather than performing GD on L()). This method implicitly retains higher-order information, but suffers from sensitivity to training setup in practice and, under favorable conditions, results in comparable performance to FOMAML.
To address the limitations of first-order methods, Truncated MAML (TruncMAML) (Shaban et al. 2019) suggests retaining portion of second-order information by truncating the backpropagation Upon setting H = 0, k = 0, ........A K 1-1, where L E [0, K] is a user-defined truncation constant, the gradient estimate boils down
Thh (0)
L. LL-aH k=K-L
II
(5)
Since (5) involves < K HVP computations, the time and space complexities of TruncMAML both decrease to (Td)) It is seen that TruncMAMI. reduces to FOMAML when J. 0, and recovers vanilla MAMI. when I, K. It is empirically observed, however, that the gradient error of TruncMAML decreases slowly when L small; cf. Figure 3b) More rigorous performance analyses will be provided in Section 3.2 with error bounds illustrated in Figure 2 It will turn out that large L sufficiently close K will needed to ensure an accurate meta-gradient estimate.
Another approach for meta-gradient estimation, termed implicit MAML (iMAML), is to rely on the implicit function theorem (Krantz& & Parks| 2002), which yields closed-form solution to VLr( For instance, it has been shown in (Rajeswaran et al.] (209)) that, with an explicit regularizer R(0;; 0) = ll 0 induced by an isotropic Gaussian prior, the per-task validation gradient Vol (0)) [L + (
(oTT Similarly, as the global optimum oi is intractable, must be approximated with a nearly optimal o in the calculation. Moreover, the inverse Hessian-vector product requires another approximate solver, such conjugate gradient iterations (Rajeswaran et al., 2019). As memory does not grow with the number of solver iterations (denoted L), iMAML has time complexity O(Ld) and memory complexity (dd. However, the iterative solver introduces additional errors in the meta-gradient estimation, making the algorithms numerically less stable than FOMAML and TruncMAML (Rajeswaran al.] 2019 Zhang al.)
2023). Consequently, reverse-mode mode explicit automatic differentiation (i.e., backpropagation) has become more prevalent practice, and this work will primarily focus on it.
3 META-GRADIENT VIA TRUNCATED BINOMIAL EXPANSION
While they reduce computational complexity compared to vanilla MAML, FOMAML and Trunc- MAML can suffer from high estimation errors that impair the neta-learning convergence and performance. Our motivation stems from the observation that both 3 and (5) entail sequential chain of HVP computations that cannot be parallelized. To enhance accuracy of meta- graddegt estimation, our key idea is to incorporate more information by performing parallel computations concurrent with each HVP.
To this end, we will first develop meta- meta-gradient estimator by truncating the binomial expansion of 4 This leads to an approach that we naturally term binomial gradient- gradient-based meta-learning (BinomGBML). In the ensuing Section 3.1, the truncated expansion will be reformulated of L vector operators, each corresponding to several parallel HVP computations. Then. Section (32 will provably show that BinomGBML. not only leads to low estimation error bounds under various
cascade
4

---

Published as a conference paper at ICLR 2026
assumptions, but also diminishes at super-exponential rate L increases, opposed the slow decay of TruncGBML. Consequently, a small L suffices for accurate gradient estimation.
Again, the discussion will be exclusive to MAML for simplicity, while our approach can be broadened to other GBML algorithms, along the lines adopted by FOMAML and TruncMAML extensions. All proofs of this section are deferred to the Appendix.
3.1 META-GRADIENT ESTIMATION VIA BINOMIAL EXPANSION
Our approach is based the binomial expansion of 4 Recall the binomial theorem (Beyer
1984)
K
(1+ 21+
8 Vz E
(6)
This expansion be generalized to multivariate product II (1 2:) and even the matrix product in 44 to yield
K-1
K
II La aHl La Σ
Σ
II (
1=1 oskutck
where we define (0 S k1 t< K} = 10 S 1 < h2 <... < hi K} as combinatorial set containing (*) terms. In 77 Ia corresponds to the over 0 S k t< K and product (-oH respectively align with the binomial factor (*) and the 1-th power 2 in (6) Intuitively, if the learning rate a is sufficiently small, the product (-aH)- = O(a²) can decrease exponentially with This motivates our meta-gradient estimator based on the truncated binomial expansion
term (5)²0 = in ( the summation
(0)) = Ia Σ Σ 1=1 OSkiitKK
II II g
(8)
As the ignored higher-order terms can be small O(a the gradient error is expected to diminish exponentially as truncation L increases. This expansion also amenable to parallelization.
With L = for instance, follows from (8) that B), g - Σk= H. where the K HVPs in the summation
be efficiently performed in parallel thanks their computational independence. However, the double sum in (8) involves total of Σ (f) terms. Directly computing each of them using parallel HVPs can be prohibitive. To improve efficiency, we advocate reducing redundancy
the summation by merging common terms of different matrix products; Appendix A.. for an illustrative example. Using this idea, the next Proposition shows that the truncated binomial expansion can be rewritten as a cascade of matrix operators.
Proposition 3.1 (matrix operator B(). Lei P = Lg, P == II x -L aHt), i= = K, and dummy index ko -1. Define operator B: Rdxd + pdxd such that any matrix M E Rdxd
B = P: - a En
K-11- 1-1-i k1 1-,1 H #z 11 M, = 0, O.....I L 1. For L S K, holds that
L ++ Σ
II ( = B/-lB 1-222 B
BB-BB
(9)
E-1 K 1=1
Proof (sketch). The proof carried out by induction on L. The base case L is readily shown by applying the definition of B, Assuming the cases L = 1, 1...., L' (where L' < K) hold, the sums of 4 can be recursively expanded and rewritten terms of B; to obtain
K-1
Σ Σ 1= u
II
B
B
L +
Σ
(-aH(R44))
B
B.
5

---

Published as a conference paper at ICLR 2026
Notably, with the definition of matrix product P the meta-gradients 4 and ( can be further simplified as (0) P er and ( P! Essentially, each matrix I oH represents mapping Rd + I when multiplying with g and thus can be viewed as vector operator in IR that requires one HVP computation.
we have established the following result. Theorem 3.2 (vector operator ")."). With the notational convention in Proposition 3.1 and given
Building upon these insights and Proposition
vector g E Rd consider operator B3 12d + Ia such that for any vector E R B
K-1-i Pig - 0 Lk. -h H 'v, 'v, 0. 10.......
For 0 S I. S K, it holds that
A Bi ( = B
111, L-2 B
.0 B
g
(10)
Theorem 3.2 reformulates the complicated binomial expansion in ( as series of L vec- tor operators. Note that the vector operator in Theorem [.2 is distinguished by a super- script from the matrix operator in Proposition 3.1 Similar the cascaded matrix-vector multiplication in 4 and (5), these vector operators must be carried out from right to left sequentially. However, each operator in (TO) may contain multiple computationally inde- pendent HVPs that can be readily parallelized on the GPU. The next remark elaborates on this parallelization and analyzes the complexity of the resultant BinomMAML. algorithm.
IB Br L-1
Remark 3.3. In Theorem 3.2 each operator requires computing HVPs with H H where the index ky-i k +...,, - 1-1-i. By the re- cursive relationship of kL- and the definition Ko- 1, it can be deduced that ky ranges from L-i-1 to K-1-i. Therefore, K-L+1 paral-
from A-11-
operator
HVP
u
I+LK --1-
HVP
u/
K-1-1
qer (d)
J+LK 2-1
HVP
u1
221
lel HVPs must be performed per oper- ator, the overall complexity of (10) (Ld) time and ((K-L++) space.
I+LK 3-1
v/
HVP
1.27
(
I+L2-1-1
The Step-by-step pseudocode is pre- sented Algorithm where each iter- ation (that is, computing {u, A IK=L-I
Figure 1: Depiction of BB operator
and {v, 1+1, 1,4 K-1-1
>==--i-)) corresponds one vector operator BR with (u. Jk=L- involving K-L+1
parallel HVPs. An illustrative diagram of the B..." _T-] operator is provided in fig. while a detailed
derivation of Algorithm is provided in appendix A
Complexity considerations bring up two limitations of BinomGBML. First, the parallelization requires (K - + 1) computational cores compared to a single HVP. While modern GPUs are abundant in such resources, applying BinomGBML to systems lacking cores without GPUs can be challenging. Second, implementing parallel computations could incur an extra overhead cost. Fortunately, our numerical tests real datasets suggest this cost is relatively small compared to the HVPs themselves; cf. Figure
Next, we compare the advocated BinomMAML to MAML and its variants outlined in Section 2.2 Remark 3.4. First, it is easily seen that BinomMAML. conforms with FOMAML when I, 0. Moreover, BinomMAMI. with reduced space complexity. This is due to MAML creating the K computation graphs for HVPs when computing oK which are saved in memory. In contrast, BinomMAML creates the HVP computational graphs on the fly, and frees up the associated memory once completed. As a side benefit, thus, BinomMAML addresses the space scalability issue in vanilla MAML. For general L, BinomMAML and TruncMAML have identical time complexity, but the space complexity of BinomMAML. decreases affinely with L.
K incurs the same time complexity as MAMI., yet a markedly

---

Published as a conference paper at ICLR 2026
Algorithm 1 BinomMAML's meta-gradient estimation
Input: training gradients (Vetm ((,))) K-1 validation gradient g
k=0
step size a, truncation L {L, {L,...,A K
Initialize V
g L, ... K
for = 0
L
do
[uf--11 ..... u: K-
V / (6
( v
1.b
V V (V v1 (.
v
1+1,K-1-1
au: R-
V;
for k = K - 1,-1,- - 1k+1
do
vt 111 vi
1+1,+11 au;
end for end for Output: (.) = V,
3.2 ERROR BOUNDS
Having developed the BinomMAML algorithm and elaborated on its computation, this section delves into the estimation errors of FOMAML, TruncMAML, and BinomMAML. The analysis will be performed under three different assumptions that are common and useful in meta-learning, Assumption 3.5. Loss ftrn has H H-pisschitz gradient Vt, i.e., [| (Ø.) Vet (et""((')) = Φill, V Φ E IRd.
H H|
Assumption 3.3 is very mild and widely used not only for meta-learning (Fallah et al. 2020 Wang
ett al. 2022). but also more general machine learning (Jain & Kar 2017 Ji et al. [002]] ), and can
be readily satisfied by wide range of loss functions and NNs (Virmaux & Scaman 2018 Under
Assumption 3.5 three bounds can be established as follows.
Theorem 3.6. With Assumption 3.5 in effect, it holds that
(0) + (.) S [(1 + aH)k -1]11//11 [1, (V), TTh (0)) S [(1 + aH) - -(1 +1+ lll
(Ila)
(11b)
K
(K) (+)
(0) &. (,))
Σ Im InToll
(11c)
With PO eet Bin denoting these bounds, it follows that e-Bin Tx < 50
Theorem 3.6 compares the error bounds of the three meta- peead-gradient estimators, where BinomMAML enjoys the smallest loss due to the additional information injected. Note that all three upper bounds HI,, Vk. Figure 2a shows the three bounds 5, a = 0.25, and H = 1.0. One can observe that the error
sharp, and can be attained upon setting H in Theorem 3.6] across L, with of TruncMAML decreases slowly when L is small. In comparison, the error of BinomMAML diminishes swiftly with L, and small error can be readily achieved even with L = 1.
Next, a stricter yet important assumption will be considered. Assumption 3.7. The loss function 6:Tm is convex Vt.
Although Assumption 3.7 hardly holds for highly non-convex NNs, it is especially useful if the task-specific part of the model linear; e.g., when adapting solely the last layer of an NN (Revaud)
al. 2019 Lee et al. 2019; Oreshkin et al. 2018). This extra assumption on convexity allows us to
establish tighter error bounds as follows.
Theorem 3.8. Under Assumptions 3.5 and 3.7 and with 0 < a S 1/H1 it holds that
[V), c.())l s [1 11- (1-aH)) aH)* Ilg. 11.
(12a)
[V) (TT [1-1111111111111111 (1- <->]]ls/
(12b)
K
(V) (.)))()) < (1+1) (aH)
(cH))+11p ||
(12c)
Moreover, there exists a constant C, = O(K/ L + 1)) such that if < a S H), the upper bound in (2ch decreases super-exponentiall with L.
7

---

Published as a conference paper at ICLR 2026
1.50
Tiure
1.20
0.8
1:00
0.0
upoe
0.00
0.2
0.25
0.9
03
Inuradtan
(a)
(b)
(C)
and (c) 3.10 normalized to Figure 2: Estimation error upper bounds in Theorems (a) 3.6 (b) 3.8
FOMAML error.
In addition to convexity, Theorem 3.8 also assumes the learning rate is not too "aggressive." It known that a sufficient condition for GD to "descend" 0 < a < 2/H, with a = 1/Hf being difficult to estimate in practice, it is common to assume a (11 in general, which is also necessary for the analysis of TruncGBML;; ef. (Shaban et al. 2019). Estimation error bounds (12) are depicted in Figure 2b It is important to note that the bounds for FOMAMI and TruncMAMI. are sharp, yet the BinomMAMI. bound is HLa, vk; and (12b) is reached when Hh = HL, k = ...... K - L and H = Odxd, k = K - L..... K - I. Regardless, the error bound of BinomMAML decreases super-exponentially and outperforms TruncMAML when L 0. Recall that both TruncMAMI. and BinomMAML. are exactly FOMAML with L = 0, so their actual errors are identical in this case.
an oracle choice (Bertsekas) 2016). Since If is unknown
loose. To be specific, (T2)) can be attained by letting H
Aside from global convexity, another assumption worth probing is local strong convexity. Assumption 3.9. The loss ftt is locally h-strongly convex Vt around {Ø+ k min(L, K L}.
K==K- where M S
Assumption is milder than Assumption 3.7 as it presumes strong convexity around only the last M points of the optimization trajectory, which can be sufficiently adjacent to local optimum. Morcover, such an assumption has been widely used for analyzing bilevel optimization; see e.g., (Ghadimi & Wang 2018) (Shaban et al., (2019).
Theorem 3.10. If Assumptions 3.5 and 3.9 hold, and 0 < S 1/H, it then follows that
(V) FFO c(()) < max (( all) KM af) 1, (1- all)*
(13a)
[V), ('))((0) < [11 (1+ +aH) K-M - (1+ (1+
4- [[ arh |g
(13b)
M
K 1
(O), BB C.()) - (aH)
Σ
L )(z -h)
(13c)
1=1
K-M
K
M (auly
(1 ah) M
Σ 1=L+1
Moreover, there exists constant C. = =O(K - T)) such that if0 < a S 11/(C. H), the upper bound 13ch decreases exper-exponentially with L.
Again, the estimation error bound of BinomMAML decays more rapidly and is markedly smaller than TruncMAML in this setup, as corroborated by Figure 2c Since Assumption 3.9 requires M < min LL, K L). the bounds are plotted with M = 1, that L [1.....5]. Other parameters are the same as in Figures 2a and 2b. and h = 0.1.
4 NUMERICAL TESTS
With analytical error bounds established, here we test the empirical performance of BinomMAML.oo on both synthetic and real datasets. Test details can be found in Appendix C
4.1 SYNTHETIC DATA
The first test a few-shot sinusoid regression problem (Finn et al. 2017). where data are sampled from sinusoids with random phases and amplitudes; each phwse- combination defines distinct task. Figure 3 compares the actual errors of different meta- meta-egrddentt estimates calculated on the synthetic sinusoid datasets. Specifically, the left subfigure 3al displays meta-gradient error
8

---

Published as a conference paper at ICLR 2026
Table I: Few-shot classification accuracies on real datasets with early stopping, where + represents sample standard deviation, and the number in parentheses indicates the mean accuracy relative to MAML, highest one marked in bold.
5-way minilmageNet (%)
5-way ticredImageNer (%)
L
Method
1-shot
5-shot
I-shot
5-shot
FOMAML
0.92 (-1.93)
0..00
43.53 0.94 (-3.50)
63.31 1 0.52
0
Reptile
4 (-4.39)
61.07 +: 0.51 (-3.16)
45.06 0.91 (-1.97)
61.27 :: (-3.85)
IMAML
4..47 (-2.03)
6.
0.99 (-2.26)
6..00:00.0
TruncMAML
4 (-1.97)
0.50 (-1.80)
43.67 (-3.36)
64.15 (-0.97)
BinomMAML
4 (-1.00)
0.88 (-1.87)
46.23 (-0.80)
0.. (-0.63)
iMAML
4..5
60.63 (-1.93)
(-3.60)
0.93 (-2.90)
(..0
2
TruncMAML
0.90
63.61 0.48 (-0.62)
0.99 (-1.10)
64.81 0.49 (-0.31)
BinomMAML.
46.23 0.92
0.92 (-0.27)
63,49 + 0.48 (-0.74)
4..20 (-0.83
6 (-0.71)
iMAML
45.03 1.92 (-1.47)
62.77 1: 0.48 (-1.46)
1 (-2.23)
62.38 1 0.50 (-2.74)
3
TruncMAMI.
44.33 + 0.92 (-2.17)
63.67 + 0.48 (-0.56)
45.62
0.98 (-1.41)
63.69 + 0.51 (-1.43)
BinomMAMI.
46.00 + 0.92 (-0.50)
64.17 + 0.47 (-0.06)
46.43 + 0.93 (-0.60)
65.37 0.49 (+0.25)
iMAML
45.43 4 0.92 (-1.07)
62.64 = 0.47 (-1.59)
45,40 + 0,95 (-1.63
61.0 = 0.51 (-3.32)
4
TruncMAML
44.83 1 0.93 (-1.67)
63.44 1 0.48 (-0.79)
44,93 1 0.95 (-2.10)
64.83 1 0.51 (-0.29)
BinomMAML
46.24 1 0.94 (-0.16)
63.86 1 0.48 (-0.37)
46.73 1 0.94 (-0.30)
64.63 1 0,50 (-0.49)
5
MAMI.
46.50 + 0.93
64.23 = 0,50
47.03 + 0.91
65.12 + 0.50
averaged on batch of tasks, where the horizontal axis represents different random batches. With the same truncation L = 4, BinomMAML. consistently outperforms TruncMAML. on varying tasks by an order of 10 to 10 The right subfigure depicts the meta- gret--gradient error across various L values. is observed that BinL significantly reduces the error relative to TruncMAMM. In particular, BinomMAML with becomes negligible when L 2 2. This suggests BinomMAML. can potentially achieve MAML-level performance in certain settings even with a small L. These observations corroborate our analysis in The empirical benefits of BinomMAML are larger than the analytical ones in Fig. 2
exhibits an error comparable to TruncMAML of = 4, and the error
Section 3.2
4.2
REAL DATA
To assess BinomMAML's performance in practice, real datasets are tested here, including minilm- ageNet (Vinyals al., 2016) and magred (Ren et al. 2018). The tests follow the "W-way S-shot" few-shot classification protocol in (Ravi & Larochelle, 2017 Finn et al. (2017). where "way" stands for the number of classes, and "shot" refers to training data per class; that is, (D!mm W.S S. The hyperparameters are in (Finn et al. 2017).
The first test compares the performance of different meta-gradient estimates. To highlight the role of estimation errors, meta-training is early-stopper at 20, 000 iterations. Note that MAML can readily converge in such a setup, while the meta- eeta-gradieet error can slow down convergence when using an estimator. The results are summarized in Table T. where L represents the truncation parameter of BinomMAML. and TruncMAMI. and the number of conjugate gradient iterations for iMAMI. BinomM ML surpasses TruncMAML. in most cases when adopting the same and exhibits comparable performance for other cases, while outperforming iMAML across the board. Of major note is that BinomMAML dominates TruncMAML (+1.3 (+1.3 average) in the 1-shot regime, while the gap narrows (+0.27 average) in the 5-shot regime. This observation suggests TruncMAML benefits from gradient averaging when data are abundant, whereas BinomMAML. remains well-equipped for low-data settings. Moreover, it is seen that the performance gap BinomMAML relative vanilla MAMI. shrinks rapidly with I., as opposed to the slow improvement of TruncMAMI, and a small L. suffices for BinomMAMI. to attain desirable performance, supporting our analytical results.
Trunc
Trunc
Amssage grdkit smot
Binam
Weltegallenn ente
100
10
10
0.0
200
index
Truncation (L)
(a)
(b)
Figure 3: Actual meta- pretiegradient error against (a) different mini-batches of tasks, and (b) truncation L.

---

Published as a conference paper at ICLR 2026
Time (ms/meta-tte)
30
TrunsksANLL BinoRMAAN
RAN/A XAXIL
Space (MB)
400
20
300
10
200
3
5
4
Truncation (L)
Truncation (L)
(a)
(b)
100
FOMANL
RAMIL
GPU utilizaion (%)
BrAmMMMI
3
Truncation (L)
(c)
Figure 4: (a) Time complexity; (b) space complexity; and (c) GPU utilization of GBML algorithms
on minilmageNet.
The next test showcases the actual time, space, and compute consumption of BinomMAMML as measured under the same minilmageNet test setup. Figure 4 displays the clapsed time per meta- training step, peak GPU memory occupation, and GPU compute core utilization across values. should be highlighted that the parallelism, alongside the dynamic creation and release of computation graphs, both incur computational overhead, which varies with L. Consequently, the time and compute of BinomMAML slightly outpaces TruncMAML, and the space complexity
strictly affine in L. Further, as no parallelism is required when L = 0 or L = 5, the time complexity of BinomMAML. matches FOMAML and MAML in these two cases. Additionally, BinomMAML leads markedly reduced memory and compute consumption over vanilla MAML, thanks to the dynamic management of computational graphs; ef. Remark 3.4
Lastly, Figure 5a depicts the change of accuracy and loss during meta-training, where the curves are smoothed for visualization. Compared to TruncMAML, the convergence of BinomMAML aligns better with MAML. This underscores the importance reducing meta-grann veta-gradieet error, and in turn explains BinomMAML's superior performance in Table [
CONCLUSIONS
This work delved into the computational complexity of GBML, and developed reduced-complexity meta-gradient estimator named binomial (Binom)GBML. Inspired by the unique GBML structure in backpropagation, BinomGBML leverages parallelization to maximize the information captured its meta-gradie meta-gradient estimate. BiL was developed as running paradigm, showcasing improved error bounds relative to existing approaches. Experiments on synthetic and real datasets corroborate the derived error bounds, which demonstrate the superiority of BinomMAML.
Mett-trainee trainin accuracy
85
19
17
FOMAML
TruncMAML
30
1.6
BinomMAMLL
FOMAML
25
TruncMALL
BinomMAML
200
400
600
800
1000
200
400
600
800
1000
Nistobooration
Meta-iterations
(a)
(b)
Figure 5: Meta-training (a) accuracy and (b) loss of GBML algorithms minilmageNet
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
As this work represents foundational research in meta-learning, we do not anticipate any socictal or ethical issues beyond the general and broad-reaching impacts of the advancement of machine learning.
REPRODUCIBILITY STATEMENT
We include supplementary material the code needed to reproduce the main results minilmageNNe and migredl Additionally, all proofs of theoretical results are included appendix B
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in newral information processing systems, 29, 2016.
Sebastien M R Arnold, Praateek Mabajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. lear22lcam: A library for meta-learning research. arXiv, 2020. URL http:://aaas
or org abs / 2008. 12284
D Bertsekas. Nonlinear programming. Springer, 2016.
William Beyer. Standard Mathematical Tables. CRC Press, Inc, 1984.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. ArXiv, abs/2110.07814, 2021.
Thomas Cormen, Charles Leiserson, Ronald I. Rivest, and Clifford Stein. Introduction to algorithmss MIT press, 2022.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics, pp. 1082-1092. PMLR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv: Optimization and Control, 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference on Learning Representa- lions, 2018.
Andreas Griewank. Some bounds the complexity of gradients, jacobians, and hessians. In Complexity in numerical optimization, pp. 128-162. World Scientific, 1993.
Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. Text data: A new framework for machine learning and the social sciences. Princeton University Press, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Prateek Jain and Purushottam Kar. Non-convex Optimization for Machine Learning. Now Publishers, Inc., 2017. doi: 10.1561/22000005588
Kaiyi Ji. Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference machine learning, pp. 4882-489) PMLR, 2021.

---

Published as a conference paper at ICLR 2026
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient- based meta- meta-learning methods. In Neural Information Processing Systems, 2019.
Steven George Krantz and Harold R Parks. The implicit function theorem: history, theory, and applications. Springer Science Business Media, 2002.
Oliver Kroemer, Scott Nickum, and George Konidaris. A review robot learning for manipulation: Challenges, representations, and algorithms. Journal of machine learning research, 22(30):1-82, 2021.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10657-10665, 2019.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pp. 2927-2936. PMLR, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09883. 2017.
Nichol. On first-order meta-learning algorithms. arXiv preprint 1 1803.02999, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Eunbyung Park and Junier B Oliva. Meta-curvature. Advances in neural information processing systems, 32, 2019.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.099 09157 2019.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In International Conference on Learning Representations, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International conference on learning representations, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:
2018.
Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. Advances in neural information processing systems, 32, 2019.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. International Conference on Learning Representations, 2018.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- pack-propagation for bilevel optimization. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1723-1732. PMLR, 2019.
Daouda Sow, Kaiyi Ji, and Yingbin Liang. On the convergence theory for hessian-free bilevel algorithms. Advances in Neural Information Processing Systems, 15:4136-414,, 2022.
Gaël Varoquaux and Veronika Cheplygina. Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ digital medicine, 5(1):48, 2022.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2026
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Haoxiang Wang. Yite Wang, Ruoyu Sun, and Bo Li. Global convergence of maml and theory-inspired neural architecture search for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9797-9808, 2022.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
Xinshi Zang, Huaxiu Yao, Guanjie Zheng, Nan Xu, Kai Xu, and Zhenhui. Metalight: Value-based meta-reinforcement learning for traffic signal control. In AAAI Conference on Artificial Intelligence, 2020.
Yilang Zhang, Bingcong Li, Shijian Gao, and Georgios Giannakis. Scalable bayesian meta-learning through generalized implicit gradients. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11298-11306, 2023.
Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, and Guangming Shi. Person alized image aestheties assessment via meta-learning with bilevel gradient optimization. IEEE Transactions on Cybernetics, 52:1798-1811, 2020.
13

---

Published as a conference paper at ICLR 2026
A BINOMMAML ALGORITHM DERIVATION
Proposition A.1 (Proposition 3.1 restated). Let P. = I k -I, - aH;)) i= 1,... K,
P = La. and dummy index Ko -1. Define operator Bj Raxd 4 Rddd such that for any holds for
matrix M E IRdxd, BiM P Lk.. H. M, i O,....L
K-1-i
S S K that
L Σ E 1m1 O
II
B
(14)
Proof. The proposition is proved using mathematical induction on L. As L will change in this proof, will alternatvely denote B as B for notational clarity,
First consider the base case 1. can be readily verified from the definition of B that
L
Σ (-aH Po OSkICK
Σ H
II
(15)
k1=ko+1
Now assume (14) holds for I,..., L' where K, we next prove that this equation also holds
for L = L' + 1. It follows that
L'+1
Ia +
Σ
Σ
II
lml
oghi C... <kicK
L'
II
Σ
Σ
II I-
1=1 05ki
C..ckrK im1
0Sk <... Sk. <K
1'+1
(0)
B
L' I' 1',
1
B
BL
Σ
II
(-oH.²)
Rskis...kkp <ki4 <K i=1
K- L- -1
K K-1'-1
L'+1
(
a
H
B
1/
B
I
Σ
Σ
II (-aH.))
k 0
=1=0
ki Cickas...kkn Kk E=1
K-L'-1
E+11
-P!
a
H
B 1'2222
B
Σ
II
(-aH
L'+1
(
B
L' +1,L'
BL' L' -2
1 +
E
II
(-aH,))
kycke <ku44
x=2
K-1
( 1/+1LL B
B
211+1,1 1/-111111
La+
Σ
(-aH
ker1=k2/t1
=B B
B
44'+
Ia
(16)
where (a) relies the inductive hypothesis for L I', (b) follows from Lemma A.4 and the fact
that Eoskse..<kyus
Σ K-1 Ek, -
Che (c) is from the definition of [ and
(d) is by recursively applying (a) through (c). This demonstrates (44 also holds for L' + 1.
By induction, we conclude that (14 holds for any L -= -,,,..,, K, which completes the proof.
n
Theorem A.2 (Theorem 3.2 restated). Lei P; [[K- K-
Lkkk ll=k-(l (Ia -aHH), i= I,.., K. P La
and kn -1. Given vector g Rdd define operator BR I2d + Rd such that for any vector
E Rd RE: = P(g that
K-1-i E -k kL 'v, i i= O.... L. - 1. It holds for 0 < I < K
B ( B
- BB
1
BR g:
(17)
14

---

Published as a conference paper at ICLR 2026
Proof. For any M Rdxd
it can be verified from the definition of B/ that
K-1-i
K-1-i
P a
Σ
M g
a Pig-
Σ
H H
.M.-M-- B [Mg].
NL
No.
x+1
R
+1
(18)
Therefore, using Proposition
A. results in
Bi
(())
[B
( B
1---2
B Lg)
gt K
[BB
1B (IB
(1006-... ...BB)Ia)
(a)
[BB ... La)g)
B
--
B
x 1.-
B [Tg
B B L-2 B g.
(19)
where (a) is by applying 18). and (b) is from recursion.
Remark A.3. As stated in section each IB corresponds to one iteration of (dd) parallel HVP computation. Likewise, the meta- gradient calculation in MAML and TruncMAML can be also respectively reformulated as chain of K and vector operators.
Lemma A.4. Consider the notation convention in Proposition 3.7
It holds that
K-L-1
BBBB
1--
I
= P
Σ
B 1-2
B
(20)
Proof. By the definition of Bj, we have
B /-
B
1-
B
K-L
Σ
HBB
B 1--2
BiI
K-L+1
K-1
(a) p1-1
aH K- pl-2 a Σ
H pp--3 (P. a Σ
H 1
k
k==k-1
K-L-1 a
H L- L
h=
K-L-1 Σ
aH
K - pp/-2 aH, (HK-Ltt 1+1 3 (P aH 1.H-111
B.B.
B
k=
K-L-1
(6) P a
H
B
(21)
where (a) separates out the k = K L term from the summation, and (b) is because P
aH -I--ippp- -Ia- oH 1111)))P = pi+1 i 0, o.... and the definition Po- Ia-
A.1 CASE STUDY FOR I 2
To give some intuition on the derivation of the BinomMAML. algorithm 2 the BinomMAML. expansion & truncated to L 2 is examined.
K-1 [ o
K-1
K-2
K
I.
HH O
Σ
H
)+ ola²).
k=
k1
k1
The double sum encapsulates all possible combinations of H. H from the product expansion. Expand and rewrite
La
aH
La
aH,
k-
Σ
H
(
Σ
H
15

---

Published as a conference paper at ICLR 2026
R-2
K-1
oH,
Σ
H
gr
Σ
H
ge
(22)
k
=
How can we calculate
Note for each hi, unique sum of HVPs over k2 is required. Denote K - 1.
HVPs u 1.k H
R-2
II [ - H k=0
gg aH
Σ
H
Σ
o.ka
B1
a
k1=0
12
Then denote each unique sum as vl.k ge -a > R-1 then this implies v1.k vl.t11 0.1 x 1,... K - 1. Finally, note g aH, K -K-1__
kk u k= I.... 1. Letting vi,k
gr
gi
v1.K-1
K-1
K-2
II [ - = | km0
Σ mm
H v
The expression is similar to that inside the parentheses above. Indeed, we can define 1.k H.vl 1,4+1 k O..................... K and 2 vIK Σ
k==k K- u l.k' Rewrite
K-1
K-2 Σ
II La aH. k=
ge = v 11
2,0
Which is the final gradient estimation. This procedure boils down to (1) calculate HVPs {u"*
K-1 Jkml
and sums {v1l*)k-] J==1 then (2) calculate HVPs {u1l*) K-2 and sums {v2.k) kk= This illustrates
Jk=0
some intuition behind the operators B8 introduced in theorem 3.2
A.2 COMPUTATION OF BINOMMAML'S VECTOR OPERATORS Now consider the more general BinomMAML truncation where S L S K. It will be shown that computing each vector operator B can be done with K - L L+ parallel HVPs in ((d) time and O(KK 1)d) space. First, denote
v
R&
B
gr
E Rd. 1= 1=1,....
(23a)
1E-1=k-1
k
0,A
gi K
Ak,
V Lkk
E Rdd
1= --1.
(23b)
v.t an the intermediate term resulting from the chain of operators B 1-1 B5 1-1
B
gf Since
g can represent of v depending on the value of the prior index k the second superscript dictates kL-1. Additionally, u," represents an HVP operation whose role is described shortly. With this notation, each Is.. is computed using the set intermediate terms 1 K-1+1 Jk=L-411 and HVPs {u, J -
B ...
{v.
K To see how this leads to the BinomMAML algorithm
first note that the definitions of vi.k 23ab and B. (Theorem
A.2 lead
K-1
J.k V
BE
V.
1-1,4+1
(p!-ig/ gi
Σ
H NE 1-1,b1+1-11)
k
11
k #2-1=k-1
(24a)
K-1
(pl- K a
Σ
1- 1-1,kr-1-
u,
KL11 R c+1
(24b)
1.k The above equations expand B. as expressions with L.k and u, In (24)). each v.A depends on unique subset of the previously calculated terms {v: 1+ 1,k K-i
1= which requires that (v! Jk=L-!
1. K-1
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 BinomMAML's meta-gradient estimation (matrix implementation)
Input: training gradients (Vet (Ø.)) K-1 step size a, and truncation k=0 validation gradient g 1}
(L.....A K
Initialize V [g ,....... g/ E RR -++1)
for / = 0
- do
U = V 0, 1 (ver ( (1))1-1), [V ........
prum
V -1-11) (V'kkkz(+)))
V [
K K-L+1 - TU-TK-L.
end for
Output: A (+) = [V
be saved (rather than summed to find B directly). (2bb) reveals that, though originally unclear, vi is simply a sum of HVPs {ur 1-1 1o-li_jk_-- K-1+1 In fact, (24b) implies that v v/
ou, 1-1,k+1 meaning 4.k K-1
Ik T-- can be computed as sequence computationally trivial additions
{vl vi / ou, ou,- 1-1----11pp K- Ik=L-1 starting from the highest index K The K T.
computationally independent HVPs can be performed parallel with (dd) time and O(K (++)dd
space.
Using this framework, BinomMAML. (77 cascade of T. x TB: operators, be calculated by recursively finding [v/+1,) complexity of BinomMAMI. then (T.d) time and O(KK L++ 1)d) space, meaning the space requirement actually decreases affinely with T.
K-I-1 (v/+1,) I via the HVPs fu, (u K-- 1K=L-1 for O..... 1. The total
The final estimate Bi (()) is found as
& ((()
BRi
L L-2
Bi
BB
K 0 0,k
BR K L
11,11
Vt
K-L
|P! gi a
Σ k1
u 1-111111
v
In other words, [v: F.,k K-L
k== are summed to produce Bi [(0). Finally, it is shown in lemma A.5 that plg v, 1-1,K -I which allows us reuse v, !-1,K-1 as Pig. rather than directly calculate Plg. = (Ia oH, -1)!- g. This recursive operator structure leads to Algorithm requiring ((Ld) time and O((K- - L + 1)d) space.
However, Algorithm can be improved by reformulating the vector additions in the inner for loop as matrix multiplication take advantage of low-level matrix optimizations in libraries such as PyTorch. This can be achieved by stacking (v--1,t+1) the matrices
1 Jk=1-141 and {u, 1.k K-I Jk=E-1 as the columns of
V = [V
K-K
U = Iu] I.
K. K-
Then it follows that
V J. 1,K 1-
I
U RU-K--L11
K-L+1
where TK-L-1 E ((X--+)))(--+)) a strictly lower-triangular matrix filled with I's. This reformulation produces Algorithm 2
Lemma A.5. Consider the notation v
L.ke
23a
It holds for S 1 < L that
P
gg
K
1-1,
(25)
17

---

Published as a conference paper at ICLR 2026
Proof.
As result of lemma
A.4.
we have the two equations
B d-1
B
1-2
B
or
K
P!
ot
k
Σ
H
B
BR. /-2
IB
.0
Bc
(26a)
K-1-1 Σ
B
1-1
1-2
B
gr
P'
H
H
B
B
gr
K
(26b)
RL RR c+1
Subtracting (26b) from 26a). we get
p' P K
aH,
-1
B
1-2
Bai
Bi
K-1-1
Or, by the definition of B that
Pgg. B -- B 112 1-2
Bs
.0
J.K-
gr
B ERROR ANALYSIS
Theorem B.1 (Theorem 3.6 restated). With Assumption 3.5 in effect, it holds that
[V),(0) &FD (.)) [(1 +HH 1|1l."1 ll,
(27a)
( FTh (0)l s [(1 + all) K - (1 + aHd)
(27b)
K
K (oury
[V),,(0) A (0)l <
Σ
|
(27c)
L1=L+1
Moreover, denoting these three upper bounds by FO,
,et Bin, it follows that ei < e FO.
Proof. Using Lemma B.2 yields
(V),(0) FO (0) |Ip gi
1
|p
I [11 [1++ -1]lg
For TruncMAML, have that
K-L-1
[V), - -T (()]| ||p K P.gk 1 || s
II
(Ia (H-oH;) - I || (28)
k=0
[[1 all K-L 1] + a g.
where the last line utilizes Lemma B.2 and that ||P|||
< K-1
H (1 aH)
Next, we prove the upper bound for BinomMAML. Notice from that
(V) 6th C,(0)) <
Pk
La
Σ
Σ
II
hy ) (-oH
|
1=1 OSk &... skkc K 1=1
1
L
Σ
II
(-aH
- la Σ
Σ
g
1= 05k <...kkKK
t=1
oShy <..ck i=1
Σ
I
(-oH
Ilgk
(29)
Σ
Σ
Ha
H
1=L+10cke C... <kick F=1
18

---

Published as a conference paper at ICLR 2026
K
K
(aHY|B
1=++1
K- (Ia- - oH and (b) uses Assumption 3.5 where (a) applies binomial theorem to P.K -Ik-
Lastly, we prove that Bin < < of As (1 att > 1 for 1 < L K, is easy to see that
== (1 + I (1 + + x < [1 aHH) K 1]llg* c.
In addition, we have
- aH)
K Σ
(4)
|
|gs
1=L+1
(4)
K
(alt)'
1. Σ
L
(alt)'
k Σ
(t) (att)'
1=0
1=L+1
4
(aH)'
-Σ
1 (1
(aH))
1=0
1=0
K
(4)
(attly
1
where (a) relies on the binomial theorem, and (b) is due (+) (t) for K L. The proof is
thus completed.
Lemma B.2. With Assumption 3.5 in effect, it holds for -..... K that
P Lal S (1 (1) -
(30)
Proof.
First notice that
- = (Ia aH
ppppp pi la
-PI-1
La) oH, K 'p!-1
K
(0) (P) La) a Σ
H
k=
K-1
H.pk-k
k=k-i
where (a) is by telescoping the second line.
Then, follows from Assumption 3.5 and the definition of P; that
PP
Iall
H K -1
k=K-i
K-1
< a
|
k=K-i
K-1
SaH Σ
K-
k=K-i
= (1+ (1+aH) -1
which completes the proof.
19

---

Published as a conference paper at ICLR 2026
Theorem B.3 (Theorem 3.8 restated), Assume Assumptions 3.5 and 3.7 follows that
hold, and 0 < < S H/H. It
[V,(00) (,) - [1 (1- aH) gr
(31a)
IV), (0) S [1 (1- aa-aH) K-
aHaaaH)--- -210) gt
(31b)
K
(L) - SP 2 L+1 (aH) ++1 |g
(31c)
Moreover, there exists a constant C. = OLK/(L + 1)) such that if < a S 1/(CaH)) the upper bound in 3lc) decreases super-exponentially with L.
Proof. By Assumption 3.7 and 0 < aff S 1, we have 0 S - aHl 3 I, - o 1. It follows that
[Pi|| - I1 - K-; [ aH || I 1, and the smallest eigenvalue Amin (P:) 2 IIl K-i Ami (Ia
K-1
aH (1 HH) x 2 0. As consequence, we obtain 0 3 Ia P = 1 (1 aH) thus giving
(0) (. < [P Lalllg s (1 all a1)
Similarly, we have for TruncMAML that
K-L-1
- c()) s
II
(Ia -HHH - La | l|le || || S [1 (1 "1]l,""
Regarding BinomMAML, using Lemma B.4 gives
VV) -B:0))-BBBC c((0)
Σ
II ( (-aHK gg
sk.. S...kkKKBL
Σ
I a [ a H|| I||l! |||
Loski <..ckisk---ii
(w)
K-L
Σ
(att)
L
where (a) relies on Assumption 3.5 and Pill < 1.
To obtain the desired result (27c). notice that
K-L
-
4
L+1 K-L-2
Σ
K K-I
+
L
L
1=1
(
(L+1) 1
+ (L+
K-T-2 K (
1=1
(6)
(L+2
K-L-2
Σ
(K-1)
L+1
(c)
K
L
L+1
where (a) uses (H) = 1- (+=1), (b) is because (bt)) + (L+1)= = -(bt) for 0 S k S T,, and (c) is
by recursively applying (a) through (b).
Lastly, using Lemma B.5 incurs
[VC, 6 A C,
(aM))aat))1111 < (all
K
eK
) L+1
K
L L+1
The second bound is not tight. As result, there exists Co S eK/(L + 1) to ensure the super- exponential decrease wit L., thus completing the proof.
20

---

Published as a conference paper at ICLR 2026
Lemma B.4. It holds for 1 S L <
K that
K-L
VL,(0) ni (0)
Σ
Σ
I (-aH,- - gi
<k.ck 1-11=1
S.
Proof.
Using binomial theorem as in 29 yields
K
VL,(0) - ())
Σ
Σ
II
(-aH, = M.[Kgg
C...
K
K-1
(
Σ
Σ
IIaaH
(-aH)
(-oHK-l)e
gt
Σ
Σ
[IaaH)) )g
gi
I-
05h <h.cky 1SK-2i=1
1 05ki
K-1
(b)
Σ
(-aH,
Σ
Σ
II
(-aH )g!
05k1 S.. Kky
<k.ck -kisk-2iz1 21==
K-1
Σ
II
(-aH,
(-aH
Σ
Σ
[I(-
(-aHK-1)g
100h 5K-2
1=L+105k
K-1
Σ
Σ
II aHH o
1=b+1 05k <... <h5K-21
K-1 Σ
I
Σ
(-aH
(-oHK-l))) +
Σ
(Ta - aH - ))
Locki <.. <k. -K-2
1=L+ OSki -
L II
E
(-aH (-aH!--)gg gg" MiKK 1](La aH, k gg
(32)
LSCk SK-
where (a) splits the summation into two parts based on whether K = K - 1, (b) rewrites the first
summation by replacing index with + and (c) separates out the L term from the first summation.
Telescoping the series [ in 32 until M, T. 1] leads
VL,(0) (0)
K-1-1
Σ
II
(-
K + M.[L+ 11LK--1-K--4 (-aH:k-))p!-
ge
USKi <k. Sk.
K-1-1
Σ
II
[I-
(-aH((-1) )pt ptt gg +
II
(-
pk -k-1-- gi
US
k=
K-L
L Π(-HH)))
Σ
Σ
(-aH (p K
ge
(33)
[-1
oshg...ck -K-1-
which is the sought result.
Lemma B.5 (Cormen al. 2022) For L = O,..., K, holds
K
ek t.
where e Euler's number.
Theorem B.6 (Theorem) 3.10 restated). Assume Assumptions 3.5 and 3.9 hold, and 0 < a S 1/H. It follows that
(V) FO L.(0)) < max((( aH) K-M ahi 1 (1-aH)**
-11-(1 aH))* l|g (
(34a)
21

---

Published as a conference paper at ICLR 2026
(V) (()) & [1 aH)K-M (1+ aH
(34b)
[VL - (. 5
(aH)
M
K x--)
L ) (1-) + ali
AM
Σ 1=++1
K-M
K
(att))
|1
1
(34c)
Moreover, there exists a constant C. O((K- - L) such that if0 < a S H/(C.H), the upper bound (34c) decreases super-exponentially with L.
1, it holds - H < H, and hence 0 < 1 - aH Ia oH: < 1 ah. In addition, 0 aHl I 1 leads to 0 < o.f 3 L2 oH =
Proof. Assumption 3.9 implies for = K M,....
1 1+ a+l, k = O,..., K M - 1. Therefore, P.K. - 0, and the emoor of FO-MAML has upper bound
K
M
K-1
(0) F c()) & |p Lalllg/11
II
(I. 14-ald aH₂)
II
(I, - oH - Ia
k=
k=K -M
S {max(1++HH)
+ K- ah)M 1,,1- aH) H|&
Moreover, follows from 28 that
K-L-1
VL)
(0)11 <
II
(L (I--HH,) I
||ll
< [1 + +all)-
[](1+ -l11+ ah) |g I|-
Next, we prove the upper bound for our BinomMAMI. Telescoping the series M, K in 32 through M. K - M] leads to
C 7 (0)
(35)
M
Σ
II [
(aHHK-)))! gg M,lK MM[K-M]p!'
gi K
(36)
Thus, its 2--norm be upper bounded via
(0)
(.)))
I
Σ
Σ
II
H
a -1 - "'Ill!"" ||+
oski <... C.......c
-htSK-1-1im
Σ
Σ
PP |ll
1=L+1USk <....... cki S.SK- 11-------
M
K-M
<
((aH)) 1+1
Σ
(1-))-
K -1
ah) +
Σ
K - M
(a
|s (37)
L
1
1=L+1
To this end, we next show that the two terms in (37 both decrease super-exponentially with L.
On one hand, plugging Lemma B.5 into the first term of (37 renders
K (1-ah)-1
"aa an)- (aH) L
re(K - (1- ah)'-1
(aH)L+1
L
(4)
TeK 1) L M
(alt) 1+1
Σ
(1 ah)²
1=1
e(K - 11)
aH
[ (1 (1-al)) a1
h
L
22

---

Published as a conference paper at ICLR 2026
where (a) is because the index 2 1. This suggests there exists C² e(K - 1 ensure its super-exponential diminish.
On the other hand, applying Lemmas B.7 and B.5 to the second term of (37 yields
K-M M
K- .M ) (aft)((1 ah) (1 ah) (aH)
K-M-L
K M
Σ
L
1=1+1
1=1
K K-M-L
Te(K M M-l)
(1 ah) (att)(tt
Σ
(1 (1+aH))
1=1
Te(K M -1) 111 11
K M-L
$ (1 ah) M (aH)Ltt
Σ (1+aH))-)
L
= (1 ah) aHl e(K
- 1)1 -1)1
[(1 + al) k- K-L-M
L
which decreases super-exponentially with some C² S e(K
M 1)/L < e(K 1)/L.
Lemma B.7. Let 7 E TR be a constant. It holds for I, = 0,..., K -
that
K Σ
(1) = g
K-L
K - (
L+1
Σ
(38)
I=++1
Proof. Defining the left-hand side of
38
as SKK), it follows
SIK] =
Σ
K
(a)
K
(K-1)
K
Σ
1K-1)
K
7d
1-1
1
1= 1+
I= L+1
1=
K-1
K K-1) )
1
1=L
|=1+1
(0)
1 L 9
(K-1)
|+|
K-1
Σ
K
+
K-1
Σ
K -
+
1=1.11
1=++
(K-1) 1)
(K-1)
(
((-1))((++1)
++
Σ
1
1=++1
(K (K-1)
+
(7 1)S[K
L
K-L+1
K 1) 1+ (1 ( - + K- 1]
K-L
L (K-1) (1
where (a) is due to (1) (b) changes the index in the first summation from I to 1 + 1, (c) isolates the I = L term from the summation, and (d) leverages the relationship between SKK) and S(K - 1] recursively.
( for L 1 S K-1 and (x) 1 K
(--1)
C NUMERICAL TEST SETUPS
All experiments are implemented on a desktop with an NVIDIA RTX A5000 GPU, and a server with NVIDIA A100 GPUs.
23

---

Published as a conference paper at ICLR 2026
C.I SYNTHETIC DATA TEST
1-dimensional continuous sinusoid regression is performed, where the tasks are to fit the randomly sampled input values 2t [-5,
sinusoids y/ - A, sin(2r, + &e) of varying amplitudes A, [0.1, and phases oe E [0, T]. During task-level training, the model is shown 10 data points. Since the purpose of this test is to view the meta-loss gradient accuracy of TruncMAML and BinomMAMI, no meta-optimization was performed. For comparison, each method uses the same randomly generated dataset, ensuring parity among methods.
In Figure 3a we set K = 5 and I. = 4. In Figure 3b the average errors are taken over 1,000 meta-batches, each containing 10 tasks.
C.2 REAL DATA TEST
Real data tests are carried out on the minilmageNet (Vinyals et al.) 2016) and ticredImageNet(Ren itt al. 2018) datasets, which are both subsets of the ILSVRC-12 geared toward meta-lcarning, The learn2learn Python library (Arnold et al. 2020) was used load the datasets. All images both datasets are 3-channel RGB natural images cropped to 84 x 84 pixels. The datasets are divided as such:
minilmageNet: 64 meta-train classes, 16 meta- vetaavalidation classes, 20 meta-test classes. Each class contains 600 samples. This split was originally proposed in (Ravi & Larochelle 2017).
taared 351 meta-train classes (448,695 images), 97 validation classes (124,261 images), and 160 meta-test (206,209 images). This split was originally proposed in (Ren et al. 2018).
For both datasets, the model architecture and training protocol is that suggested by (Vinyals et al., 2016, Finn et al., 2017). with layers of 3 x 3 convolutions and 32 filters, followed by batch normalization. ReLU non-linearity, and 2 x 2 max- max-pooling. We sample our training and validation sets using the standard W-way -"oshot few-shot learning protocol. Specifically, for each mini- batch of classes, randomly sample W classes and draw S training images and goal validation images, totaling W x S training and W x 5 training and validation images, respectively. We adopt the usual choices of W 5 classes, S = or S images, and Sval 15 images. In addition, K = 5 adaptation steps are adopted. Meta-training are limited to 20, 000 iterations. Other hyperparameters follow from (Finn al. 2017). i.e., adaptation step size a 10 2 meta-training step size 3 10 10-3, and meta batch size B = 4.
For numerical stability, we found beneficial replace the a in with a' = La/ K. This substitution makes the condition = (( + 1)/(K H)) in Theorems 3.8 and 3.10 more easily
satisfied. worth stressing that a' = 0 recovers FOMAML when 0, and reduces to MAML.
with I. K
We also note that, current DL libraries such as PyTorch lack support for the niche but crucial ability to parallelize HVPs. Although the training gradients (V } steps, our current implementation requires calculating them again in the backward pass of (.).
K-1 are obtained the adaptation k=
For a fair comparison, we have excluded the time for the this repeated gradient calculation.
D EXTENSIONS AND FUTURE WORK D.1 HYBRID
ESTIMATOR
Intuitively, in 44. the final training gradient g is less sensitive second-order terms H with small k from the earlier task-level training GD. In truncated backpropagation, this logic leads to the TruncGBML. estimate ( where H Odxds < k < K I.. However, this intuition can also be applied to the BinomGBML. estimate by similarly truncating the early Hessians and performing a binomial expansion on the remaining terms; thus, we arrive
hybrid "binom-trunc" GBML estimate. To this end, the constant C' E (L...., K} } is introduced, which determines the truncation of the computational graph, with L determining the polynomial order of the binomial expansion. The hybrid estimator can be constructed by setting H Oaxd, 0 I k < K C. rendering
BITY L.(0)=
I
Σ
Σ
I[
gi
(39)
I=1 K K-CskattKiee
24

---

Published as a conference paper at ICLR 2026
Table 2: Hybrid binom-trunc estimate performance on minilmageNet and ticredImageNet with with
K 5, L 1, and C' = 4 and all other settings identical to that of tab.
5-way minilmageNet (%)
5-way tieredImageNet (%)
1-shot
5-shot
1-shot
5-shot
Hybrid
45.17 + 0.91
62,41 + 0.49
44.30 + 0.91
63.78 + 0,49
Trunc
44.53 + 0.90
62.43 + 0.50
44.67 + 0.96
65.14 0.51
Binom
45.50 + 0.91
62.36 = 0.48
46.23 + 0.95
64.49 + 0.51
The corresponding estimate can be constructed via alg 1 setting VAtT Od k .,..,, C. Results minilmageNet and tieredImageNet with K 5, L 1, and C are presented in table Overall, the hybrid estimate performs slightly worse than TruncMAML and BinomMAML, which is expected since the constant C' introduces trade-off between computational overhead and estimation accuracy.
D.2 RELATION TO ANIL
As mentioned section 3 akin to other meta-gradient estimates (TruncGBML, iMAML), our approach can be readily combined with ANIL (Raghu et al. 2019). Specifically, ANIL omits the task-specific adaptation of certain model layers, whereas BinonGBML can used to efficiently estimate the meta- gradient of the remaining layers. While both ANIL and MAML have been shown effective on ImageNet-lite datasets, such as those used for evaluation in section it is up to the practitioner choose the appropriate meta-learning algorithm.
25


Reviews:
[{"review_id": "zqbPN5z6XW", "reviewer": "Reviewer_r31s", "summary": "Gradient based meta-learning (GBML) suffers from high computational overhead. While approximations have been proposed to make GBML more tractable, these approximations suffer from large errors. This work proposes BinomMAML via targeted approximations achieved via truncated binomial expansion yields better meta-gradient estimation, in a efficient and scalable manner. The authors present theoretical analysis of their approximation and empirically validate its efficacy in comparisons with a range of competitive baselines.", "questions": "1) I would suggest comparing the current approach to ANIL or NIL on cross domain meta learning settings or other challenging meta learning settings where there is a significant distribution shift between training and testing task distributions [1], table 3 in [2]. \n\n\n[1] https://openaccess.thecvf.com/content/CVPR2023W/VISION/papers/Lee_XDNet_A_Few-Shot_Meta-Learning_Approach_for_Cross-Domain_Visual_Inspection_CVPRW_2023_paper.pdf\n\n[2] https://arxiv.org/pdf/1904.04232", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The truncated binomial expansion along with the  regrouping terms into a few parallelizable computations is quite appealing\n2) The regrouping of terms enables parallel computation, increasing GPU usage and overall efficiency.\n3) They show quantitive improvements over standard approximation methods", "weaknesses": "1) Concerns about benchmark choice. Prior work (e.g., ANIL/NIL [1]) shows that on ImageNet-style few-shot benchmarks, removing or nearly removing the inner loop can match MAML’s performance—likely because train and test tasks are sampled from the same ImageNet distribution, so distribution shift is limited. If inner-loop adaptation offers little measurable benefit in this regime, then demonstrating a “better” meta-gradient approximation there is a weak proxy for approximation quality: the task may not actually require accurate meta-gradients. Consequently, results on ImageNet/TieredImageNet alone are hard to interpret in terms of the value of improved meta-gradient estimation.\n\n2) Lacks discussion of [1].\n\n[1]https://arxiv.org/abs/1909.09157", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLvz9AYp5r", "reviewer": "Reviewer_UEik", "summary": "This paper studies how to obtain accurate estimates of the multi-step inner-optimization gradient in gradient-based meta-learning in both fast and memory efficient manner. In MAML, a representative approach to meta-learning, estimating the posterior over task-invariant parameters requires multiple steps of gradient computation, which becomes a major computational bottleneck. To mitigate this, prior work has used truncation of the number of steps that contribute to the gradient approximation or implicit differentiation, yet these approaches suffer from gradient estimation error and numerical instability. This paper shows that, if one replaces a naive decomposition of the multi-step gradient with a decomposition inspired by the binomial expansion when truncating the number of steps, the truncation error decays exponentially with respect to the inner step size. The proposed method can be algebraically expressed as an operator acting on the gradient vector, which yields a parallelizable algorithm. Across several settings including sufficiently realistic ones, the authors demonstrate both theoretically and empirically that the proposed method attains smaller gradient estimation error than existing methods. In addition, as a secondary contribution, they show in theory and experiments that the proposed method reduces space complexity even relative to vanilla MAML.", "questions": "- The proposal appears to conceptually distinguish BinomGBML from BinomMAML, but the practical difference is not clear to me. Chapter 3 seems to assume the MAML problem setting described in Chapter 2. Is this understanding correct?\n- What is the Hybrid Binom-Trunc Estimator section in Appendix D for?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The problem is very clearly formulated and is important to the community. Gradient-based meta-learning exemplified by MAML is widely studied, and obtaining accurate gradients while controlling computation is obviously one of the central concerns.\n- The paper is well written and easy to follow. In particular, it organizes gradient-based meta-learning from a consistent perspective. The target problem is framed via hierarchical Bayesian modeling with MAML positioned as one solution, and the paper provides a comprehensive review of representative methods for addressing computation time, including FO-MAML, Reptile, Truncated MAML, and implicit MAML. This gives readers a coherent viewpoint and effectively motivates the proposed method.\n- The idea inspired by the binomial expansion solves the problem elegantly with a solid construction. While it does not aim to remove the fundamental growth in computation with the number of steps as implicit MAML does, it is highly practical. The result is a gradient truncation scheme whose error decreases exponentially with the inner step size and that admits parallel computation, which is a noteworthy contribution.\n- The paper provides consistent proofs of upper bounds on gradient estimation error under multiple scenarios, from sufficiently realistic assumptions up to the stronger assumption of convex losses. These theoretical results strongly support the intuition of the method.\n- The theoretical results appear sound, although I could not verify every proofs in full detail.\n- The authors provide numerical validation under realistic settings for gradient estimation accuracy, time complexity, and space complexity. The experiments show that BinomMAML consistently outperforms other methods in gradient accuracy, while the time overhead remains acceptable even compared with TruncMAML. Moreover, although secondary, improved memory efficiency compared with vanilla MAML is also a significant advantage.", "weaknesses": "- Although implicit MAML is discussed as prior work, the paper did not provide any of theoretical or empirical comparisons to it. Since iMAML is undoubtedly one of the representative approaches to addressing MAML’s computational burden, would it be possible to include performance comparisons with such methods, either theoretically or experimentally, to validate the effectiveness of the proposal?\n- Theoretical claims from Proposition 3.1 through Theorem 3.10 are central and valuable contributions. However, there is concern that the proofs of these statements are not discussed in the main text at all. For core results such as Proposition 3.1, Theorem 3.2, and Theorem 3.6, even a high-level proof sketch in the main body would substantially aid reader understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sDZfKJ1frs", "reviewer": "Reviewer_jYLW", "summary": "In this paper, the authors aim to reduce the computational cost of meta-learning while maintaining a low approximation error. Existing approaches typically achieve efficiency gains at the expense of increased approximation bias. To address this trade-off, the authors propose a new framework called BinomGBML. They observe that conventional gradient-based meta-learning methods suffer from limited parallelism in the Hessian–vector product (HVP) computations. To overcome this limitation, the authors develop a meta-gradient estimator based on the binomial expansion, which expresses the meta-gradient as a truncated binomial series. This expansion is further reformulated into a cascade of vector operators, each of which can be executed in parallel. Theoretically, they show that the proposed estimator achieves a lower error bound compared to existing meta-learning methods. Empirically, they demonstrate the efficiency and effectiveness of BinomGBML through numerical experiments on multiple benchmark datasets.", "questions": "1: How can the proposed binomial-expansion based meta-gradient estimator be implemented in practice using PyTorch? A detailed explanation or example would help clarify how the estimator operates and how its components interact in a real training pipeline.\n\n2: How does the runtime performance of the proposed estimator compare with existing baselines such as MAML, iMAML, and other meta-learning methods? Providing empirical comparisons or complexity analysis would help illustrate the computational advantages of the approach.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1: The authors carefully investigate why existing variants of gradient-based meta-learning methods are inefficient. Their theoretical analysis provides valuable insights into the underlying causes of computational bottlenecks, helping the community better understand the limitations of prior approaches.\n\n2: The proposed parallelized meta-gradient estimator is an interesting and original contribution. It introduces a new perspective on improving the efficiency of meta-learning by leveraging the binomial expansion to enable parallel computation of Hessian–vector products.\n\n3: The method demonstrates a well-balanced trade-off between computational efficiency and gradient approximation accuracy, achieving significant speedups without sacrificing performance.", "weaknesses": "1: The method section is somewhat difficult to follow due to the dense mathematical formulation. Including high-level diagrams or flowcharts to illustrate the overall framework of BinomMAML would greatly enhance clarity and accessibility for readers.\n\n2: As the proposed estimator introduces a new way to compute meta-gradients, the paper would benefit from a more detailed discussion on practical implementation — for instance, how to integrate the estimator into common frameworks like PyTorch and how to manage computational overhead in practice.\n\n3:Although the paper emphasizes efficiency as a major advantage, it does not provide clear experimental results or ablation studies quantifying the computational gains. Presenting concrete benchmarks (e.g., runtime, memory usage, GPU utilization) would strengthen the empirical support for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQJQqnFMGv", "reviewer": "Reviewer_4xo1", "summary": "This paper presents a meta-learning method BinomMAML, based on MAML, that uses the binomial expansion of a repeated (I - H_k) application in the meta-gradient to derive a parallelizable execution method and low-error approximation by truncating higher order terms in the expansion.  In contrast to truncating on iteration depth, all iterations' hessians can still be used in the approximation, with truncation happening on the degree of their combinatorial interactions.  This is efficiently computed by parallelizing the computation of the second-order values given the first, etc., so that computing the L'th order approximation requires L serialized sets of parallel computations.", "questions": "Alg. 1 pseudocode I think could be a little better aligned to the notation in the text, in particular making use of the B operators and showing how the computations get these values.  The indexing is very busy in Alg. 1 and difficult to disentangle.\n\nFigures and tables often appear far from their descriptions in the text and sometimes out of order.  It would be easier if this could be organized so the figures are closer to their main discussion.  For example, sec 4.1 on p8 uses Fig 3 on p.9, even though the next section 4.2 looks at the earlier Fig 2.  Fig 3 is also referenced for the first time earlier on p.4, which is OK, but I felt I had to jump around a little too much to find these several times.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* This is an insightful method whose basic derivation is explained well with clear motivation.\n* Extensive theoretical derivations show its error decreases rapidly (possibly above exponential) and compare to similar derivations for MAML and Truncated MAML, showing excellent improvements.  These are verified on the (standard) sinusoid synthetic problem.\n* The method also realizes gains are in the more real-world data settings of mini-Imagenet and tiered-Imagenet (albeit possibly less than one might hope for based on the theory), but still adequate empirical validation", "weaknesses": "* While the theoretical results are excellent, the method's results on the two real-data tasks is more limited and with relatively small gains. (though, the paper touches on this in the discussion and points out its gains are larger in the 1-shot case with reasonable explanation)\n\n* Although I found the explanation of the method's ideas and the core derivation of the combinatorial algorithms and B operators clear, some of the formulas, and Alg. 1 in particular, are very busy and can be difficult to align with the derivation in Eq 8 and Prop 3.1.  Prop. 3.1 also has some indexing I found difficult to interpret in its use of $k_{L-i} = k_{L-1-i}+1$ as a sum subscript (though, I can see it corresponds to the combinatorial index variables 0<= k_1 < ...< k_l < K)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)