Low-Dimension-to-High-Dimension Generalization and Its Implications for Length Generalization
Authors: Yang Chen
Long Yang
Yitao Liang
Zhouchen Lin
Venue: ICML 2025 poster (2025)
Keywords: Length Generalization
Primary area: theory->deep_learning
TL;DR: We study length generalization through the lens of Low-Dimension-to-High-Dimension generalization.

Abstract: Low-Dimension-to-High-Dimension (LDHD) generalization, a subset of Out-of-Distribution (OOD) generalization, involves training on a low-dimensional subspace and testing in a high-dimensional space. Assuming instances are generated from latent variables reflecting problem scale, LDHD generalization captures the inherent scaling challenge of length generalization. We theoretically show that LDHD generalization is unattainable without appropriate inductive bias. Focusing on Boolean functions, we demonstrate that different architectures trained with (S)GD converge to *min-degree interpolators w.r.t. different linearly independent sets*, achieving LDHD generalization only when the target function aligns with this bias. From the perspective of LDHD generalization for length generalization, we explain the success of CoT in restructuring latent space for improved LDHD generalization. We further propose a principle for designing position embeddings to address both LDHD generalization and data format nuisances separately. Following the principle, we introduce RPE-Square, a novel embedding that enhances RPE to better handle data formats.

Lov.Diimensoon----iig--DDeeesss Generaliatinn And Its Implications for Length Generalization
Yang Chen Long Yang
Yitao Liang" 23 Zhouchen Lin 124
Abstract
logic (Abbe et al., 2023; Garcez et al., 2022), and planning (Zhao et al., 2023; Valmeekam et al., 2023). One of the most significant challenges in learning to reason is length gen- eralization (Anil et al., 2022; Zhang et al., 2022b), where models trained on small-scale instances must generalize to large-scale instances. Length generalization crucial because the size of sample spaces often increases exponen- tially with the complexity of reasoning problems, leading to intractable sample complexity and computational costs for models that do not achieve length generalization.
Now-Dimeesioontt-HHilll Hioo-igg--Dimniionnn (LDHD) generalization, a subset of Out-of-Distribution (OOD) generalization, involves training on low-dimensional subspace and testing in a high-dimennional space. Assuming instances are generated from latent variables reflecting problem scale, LDHD generalization captures the inherent scaling challenge of length gener- alization. We theoretically show that LDHD generalization is unattainable without appropriate inductive bias. Focusing on Boolean functions, we demonstrate that different architectures trained with (S)GD converge min-degree interpolators w.r.t. different linearly independent sets, achieving LDHD generalization only when the target function aligns with this bias. From the perspective of LDHD generalization for length generalization, we explain the success of CoT in restructuring latent space for improved LDHD generalization. We further propose principle for designing position embeddings address both LDHD generalization and data format nuisances separately. Following the principle, we introduce RPE-Square, a novel embedding that enhances RPE to better handle data formats.
Numerous works have investigated various techniques for length generalization, including modifications to model ar- chitectures (Shaw et al., 2018: Jelassi al., 2023; Kazemne- jad et al., 2024), transformations in data formats (Lee et al., 2023; Zhou et al., 2023), prompt engineering for Large Lan- guage Models (LLMs) (Wei et al., 2022; Feng et al., 2024). Although some of the above techniques work uniformly well across wide class problems, many are fragile and even ad-hoc, applicable only to specific problems with certain formats (Zhou et al., 2024). This is due the mismatch between the inherent problem scale and the length of the input string. We illustrate it in the next Example 1.
Example 1. Consider the addition learning problem where the input is a string. For an N digit-ppus-- -Ndigit (written as N -addittion) addition of two numbers x and y, where x 2.. u. y yN-1 (xN-1 0 YN-1 0), consider Iwo formats: the Aligned and Reverse Format (ARF), where the instance is represented as "Xo... + ..... Na-1 No-1. ="; and the Un- aligned and Reverse Formal (URF), where the instance is represented as "Xo arg max,(c; + 00,cc {x,y). However, the length of the input strings in neither format faithfully reflects the prob- lem seale: in ARF, the length of the input strings is always invariant; in URF, shorter string may correspond larger scale than a longer string (e.g. 81 =1=11 1234 =" is 4-addition, $2 = 1223 + 123 =" is 3-addition; $1 is of larger scale than 82 but length(s1) < length
... Yo, Fi, Yi E {O.....9)
1. Introduction
Learning to reason has gained significant popularity in the machine learning community due to its impressive perfor- mance in reasoning tasks such as natural language process- ing (OpenAI, 2023a;b), mathematics (Frieder et al., 2023; Jelassi et al., 2023), coding (Zhang et al., 2022a), symbolic
"xg...Xo," -o, yo.. -n,-1 -o The =
'State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University 'Institute for Artificial Intelligence, Peking University 'Beijing Institute for General Artificial Intelligence "Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China. Correspondence Zhouchen Lin <zlin@ www.kcuuuccn... Yitao Liang <yitaol @ pku.edu.cn>>
Example demonstrates the sensitivity of length generaliza- tion to the data format. Developing robust and transferable methods for scalable models requires a formulation that cap- tures the inherent scaling challenge of length generalization and invariant to the data format nuisance.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Low-Dimension-to--ligg Siiiiinnnion-t--iiiiiiinnnoonnnn Dimension Generalization And Its Implications for Length Generalization
X
2-
1-
-1-
-1
-1
X
Train Class
Train Class
Tram Class
Train Class
Train Class
-3
Cass
-3
Class
Class
Test Class
Test Class
Test Class 1
(a) In-Distribution Generalization.
(b) (Typical) OOD Generalization.
(c) LDHD Generalization.
Figure 1. Illustrative comparison of in-distribution generalization, typical OOD generalization, and LDHD generalization. (a) In- distribution generalization assumes identical training and testing distributions. (b) Typical OOD generalization involves shift between training and testing distributions, which remain relatively "close" (e.g.. sharing support or having small distributional distances), (c) LDHD generalization features training distribution restricted a low-dimensional subspace and testing distribution on high- -igensiinnsional space, often vastly different. While LDHD is a type of OOD generalization, its structured shift poses unique challenges, as training data provide no clues about the additional dimensions' contribution to the label.
1.1. Our Main Works
of n-addition In-1...To + In-1... yo can be represented by the latent vector hn = [(x.. yo)..........................y.. (xn-1. 9n-1)] The k-th dimension of hn corresponds to the (k 1)-th digits of the two addenda (i.e., Ik-1 and yk-1) 1. The expan- sion of the state vector in the dimension corresponds the increase in the addendum digits. In the length generaliza- tion of the addition task, the scale shift from No-addition to N-addition can be seen as a generalization from the low-dimensional latent space No to the high-dimensional latent space ΣN, The label, i.e., the sum of the addenda. is determined by the computation of the addition task. The input string is determined by both the latent variable and the data format (e.g., ARF or URF).
Saapenniioont---i------eeeee Generalization Per- spective for Length Generalization. We propose Abstrac- tion that disentangles the problem scale and the data for- mat, which provides a more precise formulation for the analysis of length generalization.
A Abstraction 1 (Data Generation in Length Generalization). The data generation process of an instance of scale with the concept e is as follows:
latent variable sampled from subspace En of dimension n:
2. The label is determined by the concept and the latent variable h, i.e., c(h);
Example 2 demonstrates that each instance of n-addition is generated from an n-dimension latent variable via a data for- mant mapping. Length generalization of addition requires addressing both LDHD generalization in the latent space, which captures the computation of addition as the scale in- creases, and the data format, which may hinder the model in learning the addition operator. In general, from the LDHD generalization perspective. length generalization involves two key aspects: LDHD generalization in the latent space and the nuisance of the data format. LDHD generalization reflects the scale shift between the training and test data, which the inherent challenge of length generalization.
3. The latent variable h is transformed to an input sequence by the data format mapping Φ.
In Abstraction 1, the scale of the instance is captured by the dimension of the latent space from which the instance is sam- pled. The scale shift in length generalization is characterized by Low-Dimension-fe to-iiinn-t--ighhh High- Dimension (LDHD) general- ization in the latent space. In other words, the generalization from "short" instances to "long" instances corresponds to the generalization from the low-dimension latent subspace of the training data the high- high-dimension latent subspace. The computation of the task is represented by the concept e, which is defined on the latent space and independent of the data format. The data format decides how a hidden variable is mapped to an actual input sequence. The next Example 2 illustrates the abstraction in the addition.
No-Free-I Free-Lunch Theorem of LDHD Generalization. The main challenge of LDHD generalization is that the testing space has extra dimensions compared to the training space. As a result, the testing space contains instances with orthog- onal components to the training space. The training samples cannot reveal any information about how these components
Example 2 (Addition). Let = {O...., 9}2. The instance
2

---

www.DDiennsion----llihh--DDnniio Generalization And Its Implications for Length Generalization
contribute the results. For instance, in Example 2, learn- ing with No-addition solely does not tell how (No + 1 digits to N-th digits contribute to the result unless we have the prior knowledge that the addition of each digit shares the same process. Figure 1 shows an intuitive illustration for the challenge of LDHD generalization of linear models on IR, compared to in-distribution generalization and typical OOD generalization settings. When learning a linear de- cision boundary from the low-dimension training data, we need to know the slope of the decision boundary as prior; otherwise, we will fail to achieve LDHD generalization: While all the three solid lines in Figure Ic perfectly sepa- rate the low-dimension training data, only the true decision boundary can achieve LDHD generalization.
additional middle state. This transformation could facilitate LDHD generalization in the latent space for some prob- lems. Additionally, in Section 5.2, we propose principle of position embedding design for length generalization with Transformers: We need to handle both the inherent LDHDD generalization and the nuisances such as the data format in the design of the position embeddings. Following this principle, we propose a novel position embedding named -quare. Our experiments show that the RPE-Square evidently enhances the RPE with the ability to handle the nuisance of the unaligned data format.
The next sections of the paper are organized as follows: In Section 3, we introduce the formal definition and No-Free- Lunch Theorem of L.HHD generalization; in Section 4, we present the results of the inductive bias of different mod- els trained by gradient and how the inductive bias influ- ences LDHD generalization; in Section 5, we show how the LDHD perspective helps to understand and design practical techniques for length generalization.
We formalize the above challenge as No-Free-Lunch Theo- rem (Wolpert & Macready, 1997; Wolpert, 2002) of LDHD Generalization. Theorem 1 shows that no algorithm could achieve LDHD generalization uniformly for all tasks, which necessitates the use of prior knowledge in the learning pro- cess in order to achieve LDHD generalization.
1.2. Notations
Inductive Bias of Models and Algorithms for LDHD Generalization. In practice, the prior knowledge is usually incorporated via the inductive bias of the learning algorithms and the models. To develop LDHD generalizable models, we investigate the inductive bias of different architectures trained by (S)GD. While random feature models are shown to be min-degree interpolators (Abbe et al., 2023), Theo- rem 2 shows that random feature models with projections, where inputs are transformed by projections before fed to the random feature models, converge min-degree interpo- lators w.r.f. different linearly independent sets. Furthermore, we consider Position-Oni Only Linear Attentions with Advice (PLAA), i.e., linear attentions whose attention scores are de- termined only by positions, with additional hints about the scales of the instances. This model can be seen as a simpli- fied abstraction of decoder-only transformers with a special focus on positional relations that are considered crucial for length generalization. Theorems Theorem 3 and Theorem 4 show that PLAA with Absolute Position Embedding (APE) and PLAA with Relative Position Embedding (RPE) con- verge to min-degree interpolators w.r.t. different linearly independent sets. The results illustrate the limitation of PLAA with APE in LDHD generalization and the potential of PLAA with RPE to overcome the limitation.
We use [r:] to represent the set of numbers {I...., .N. We denote the set of all functions from the set X' to the set y as Fx.y. We define Proj(z, V) as the coordinate of the projection of æ onto the space spanned by V with the basis V = [U],..., i... i..., [Proj(æ, V)], = (zz, va) for all = 1, ... We use A" to denote the Kleene closure of the set A, i..., A' = U A We use deg(p) to denote the degree of the polynomial p. We represent the set of N x N upper triangular matrix as UN.
2. Related Work
Length generalization in reasoning problems. Length generalization is key challenge in learning to reason, typi- cally interpreted as the ability to learn with small-scale in- stances of a task and generalize to unseen large instances of the same task (Anil et al., 2022; Zhou et al., 2023). Various reasoning tasks are considered to investigate length gener- alization, including arithmetic (Jelassi et al., 2023; Feng et al., 2024), Boolean logic (Abbe et al., 2023: d' d'Ascoli et al., 2023), symbolic reasoning (Zhang et al., 2022b), etc. Despite the rich literature on length generalization on spe- cific reason tasks, few works have considered challenges and overconditions of length generalization for general prob- lems. The existing works that analyze general length gener- alization mainly focus on the change of the input sequence length (Xiao & Liu, 2023; Ahuja & Mansouri, 2024; Huang et al., 2024). This formulation conflates the inherent scaling challenges with the effects of data format, making it difficult to precisely capture the challenges of length generalization. To address this, our work proposes disentangling length gen- eralization into two distinct aspects: LDHD generalization
Implications of the LDHD Generalization Perspective for Practical Length Generalization Techniques. The LDHD generalization perspective further provides insights into practical length generalization techniques. In Sec- tion 5.1, we discuss the role of Chain-of-Thought (CoT) (Wei et al., 2022) in length generalization. We show that CoT can be seen as change of the hidden space, where each dimension of the latent space is augmented with
3

---

Lov.DiinensionttoTiiigtDDDDi Generalization And Its Implications for Length Generalization
in the latent space, which characterizes the challenges asso- ciated with increasing scale, and the data format nuisance.
istence of universal algorithms for LDHD generalization.
Theorem (No-Free-Lunch Theorem of LDHD Gener- alization). Suppose that the two sets X and y are fi- nite. For some N > No, consider two subsets NNo- XN of X such that NNo & AN S X and dim(XN,) No < NN- dim((x). Let C1,02 E =((== Fxx,y) be two concepts such that ei (x) = -c(()) for all a E NNa- For any C E F and X' S X. define F/ F/(cl I X') == (fEFI(a) = c(x) for all x E.). Let : xxy- + R be the loss function, For any distribution D(Xx) such that supp (D(Xx)) = =xx:
Inductive Bias of Model Architectures and Algorithms. In the situation of learning with overparatermization or in- complete information, proper inductive bias is essential to select the true model from the hypothesis (Neyshabur, 2017: Bartlett al., 2021; Teney et al., 2024). One of the most studied scenarios is the inductive bias of different model architectures under (S)GD. Previous research shows (S)GD combined with different model architectures lead to differ- ent effects of implicit regularization, such as different norms of the learnable parameters Gunasekar et al. (2017); Bartlett et al. (2021) and different complexity characterizations of the models (Razin Cohen, 2020; Razin et al., 2021; Abbe et al., 2023). We specially mention Abbe et al. (2023), which proposes that models trained with (S)GD are biased towards min-degree profile interpolators in the context of Boolean functions, which do not achieve general LDHD general- ization. Our results show that the min-degree profile bias does not hold for all models. We further show that models with different architecture can converge to min-degree pro- file interpolators under different linearly independent sets when trained with (S)GD. This partially explains how model architectures can affect length generalization in reasoning.
E
SEFI(CIlNNN))
= Σ
B ((ap))))l(((((t))((())
SEF/
Theorem necessitates the consideration of structural as- sumptions on the concept class such that a learning algo- rithm could identify the target concept from the hypoth- esis with the imperfect information provided by the low- dimensional training data. For example, the concept class of linear classifiers with fixed weight vector on X R i.e., C = {sgn (wfx + b) wo E T b E R} with the d'- dimensional training sample space X1 = R x (0)=-d' and the d-dimensional testing sample space where d' < d. For any concept e E C, a learning algorithm could not identify the true concept e solely from the training data from Xi and the hypothesis of all linear classifiers. However, if a learning algorithm exploits the structure of the concept class, i.e., by fixing the weight vector to Wi in prior, it can compute the target bias from the training data in X1 and then identify the target concept c.
3. LDHD Generalization
Definition 1 LLow-DmenssionttoHHine Haneiiin--t--igh---ii High-DDmension Gener- alization). Suppose that X' is a sample space and X1, A2 two subspaces such that 21 C X2 C X and dim < (12( Consider concept class C C Fx,y, two distribu- tions D1, D2 where supp(D,)= = X1 and supp(D22= X2, and a learning algorithm A: : (X x y)" + Fx.y. We say pov-dimension-to--ie so-ttigh-dimension generalization of the con- cept class C from D2 to D2 is achieved by the algorithm A with m samples and e error if
We further investigate how model structures can affect LDHD generalization. We show in Section 4 that differ- ent models trained with (S)GD can be seen as min-degree interpolators under different functional bases in the context of Boolean functions, which is a joint inductive bias of the model structures and (S)GD. This insight suggests a prin- ciple for model design to achieve LDHD generalization: ensure the concept class is "low-degree" under the linearly independent set induced by the model structure.
E ET X, [e (fxmee(mmtt)) ((Xx)))))(((+))) S,
where fxm,, is the function learned by the algorithm A from the training samples X labeled by the concept and L: e: x y + IR is the loss function.
Definition 1 extends the Independent Identical Distribution (IID) assumption in PAC learning theory and considers a special shift between the training data and the testing data. This shift particularly challenging because the testing space is of strictly higher dimension than the training space. Generally, it is impossible to fully capture the structure of the training space from the testing data as the training data reveal no information on how components in the orthogonal subspace contribute to the output. Therefore, there is no algorithm that can always guarantee to learn the concept from the training data, Theorem formally states the nonex-
LDHD generalization captures the challenge of scale shift in length generalization. Intuitively, length generalization means that model trained with small-scale instances of a reasoning problem can perform well large-scale instances of the problem. To formalize the intuition, we consider a typical instance being generated from a hidden variable h that represents the "core" of the instance and is transformed to the model input by a data format mapping. The hidden space is of the form E* for some domain Σ. The dimension
4

---

Mow-Diimension-t--liggh.. Dimeiiioniiinninnn Generalization And Its Implications for Length Generalization
of the subspace from which the hidden variable is sampled, arg max; (h E Σ* h E Σ reflects the increase in the scale of the problem. Besides, we define the concept class on the hidden space, depicting that the change of the data format does not change the computation of the concepts. Abstraction 1 describes the pipeline of the data generation in length generalization.
gree profile of the function w.r.t. B, denoted by DegP PopPPp((1 (S)) is a (D+ where D; = LE (, D++1; fb( for i = ...... D cal the lexicographic order of the correpooningg D-tuples Definition 3 (Min-Degree Interpolator w.r.t. Linearly In- dependent Set B). Suppose that B {b....., bR} is linearly independent set of functions in F. Let X" be a subset of the sample space X' = {=1] Denote the set of all interpolators on X' for the concept e by Ex i.e., Gx'c = {9 E F g(æ) = c(x) for all x E {. A function g called the min- interpolator w.r.t. B on Xg for the concept e i g E gx and Dgg S DegP(gg)) for all g' 9x',.-
The order of degree profiles is identi-
Remark 1. Some works observe length generalization on certain tasks without domain-specific priors, using only a small fraction of large-scale data during pretraining. This does not contradict our No-Free-Lunce Theorem, as "no" and "few" large-scale examples are fundamentally different. Prior work (Jelassi et al., 2023) shows that exposure a tiny fraction of long sequences in pre-training, which is called priming, can significantly improve length generalization. From our perspective, this is because the presence of long sequences, even in small quantities, prevents the model from learning an overly simple interpolator that fails to extrapolate. For example, in the addition task, training only on d-gittt numbers may lead the model to ignore digits beyond the third, while including just few 5-digit examples compels it to process later digits, enhancing extrapolation.
4.1. Random Feature Model with Projection
We first consider the random feature model (RFM) and class of its variants, i.e., Random Feature Models with Pro- jection (RFMP; see Definition 4). RFM is widely employed as approximations of practical neural network models in theoretical studies. By comparing the inductive biases in- troduced by RFM and RFMP under various projections, we demonstrate the importance of incorporating prior knowl- edge to achieve LDHD generalization and this prior knowl- edge can be effectively embedded through model design.
4. Main Results
We show theoretically how different models succeed or fail to achieve LDHD generalization as the effect of the inductive bias of the architectures under (S)GD. We focus on Boolean functions. More specifically, in the context of Boolean functions, we have Σ- = {+1}, X ΣN, Xn = Σ" x [1)N-n for n ===11 1...., N. The set of all Boolean functions potentially considered is F = Fx,5. Our analysis can be naturally extended to tasks over finite alphabets by considering their binary representations.
Definition 4 (Random Feature Model with Projection). Sup- pose that V = [v1..... ........ Ur] E RR satisfies VTV I,.. A random feature model with projection w.r.t. V is
K
SRMMP (z; a) = VK
ago (wwk; Proj (zc, V)) + bn),
k=1
where K is the number of random features, o is the ac- tivation function, [a....., axl" is the learnable pa- rameter, and wk L,)), by NN(0, 1/r) for all k = l.....K K.
We consider LDHD generalization from No to N for some No < N. Define I(f) as the minimal set I of indices that the function f can be represented as function of æ1, i.., I(f)= == arg min |7| such that fxx = fx1) for some function f and all x E X. A function f is k-sparse if [[()) k.
The original RFM can be seen as a special instance of RFMP with V = TN- Teehnically, we follow the strongly expres- sive condition (Abbe et al., 2023) for the activation function σ. Abbe et al. (2023) shows that the RFM converges to the min-degree interpolator when initialized at 0 and trained with GD. However, this is not the case for all RFMP models. We show in Theorem 2 that an RFMP model converges to min-degree interpolator w.r.t. linearly independent set determined by the set V.
Before presenting the theoretical results, we introduce two concepts degree profile w.r.t. linearly independent set and min-degree interpolator w.r.t. linearly independent set, which extend the concept degree profile and the concept min-degree interpolator, respectively. We use the two con- cepts to characterize the inductive bias of different model architectures under (S)GD.
Theorem 2. Suppose that V = -[i.....,, v,,] E RRNxr xr satis- fies VTV = I,.. Define the set B(V) of independent func- tions as
Definition 2 (Degree Profile w.r.t. Linearly Independent Set
B(V) = xxx b=)gmm)
B). Suppose that B- = {bi,..., "ih,....,,,) bR} is linearly indepen- dent set of functions in F and D = max dcg(b). Let f TeJ F be a function in the subspace spanned by B, i.c.. f f= R
where
N xx(x) = II
(n
Σ D fb(b;)h; for some (b,) E RR == 1,..., R. The de-
xET n=
5

---

Low-Dimension--t--ligg -oiiiii-ntt--iiggiiiimnnnnooonn Dimension Generalization And Its Implications for Length Generalization
Let at be the learnable parameter the timestep I in the training process where the learnable parameter is initial- ized at ag = 0 and optimized with gradient descent/gradien flow under 22 loss on NNo- Let .No.c.V be the set of all in-
Definition 5 (PLAA). Define the advice function n X + (0, (0,...,,A .N) such that (xx) arg max, fzr, = -1] if there exists & E [N] such that z = -1 and n nzx)) = 0 otherwise.
We additionally define eo := 0. A PLAA model is
terpolators on NNo for the concept c" (xx) = e (Proj(æ, V)) that is ON (1)-sparse. Then we have, as K + oo,t + 00,
(PL A) = T Aen(s);
( as) arg min DogPe B(V)()) gEUNo.O.
where A E Uy is the learnable parameter and en denotes the vector with 1 in the n-th coordinate and O's elsewhere.
When V = IN, the linearly independent set B(V) is the Fourier basis of the Boolean functions, and Theorem 2 im- plies that the RFM converges to the min-degree interpolator. From Theorem 2, we see that an RFMP model with the pro- jection matrix V can achieve LDHD generalization only if the target concept coincides with the min interpolator w.r.t. BVV). Specially, for the RFM, we have:
We further elaborate the intuition behind the PLAA models. In the generation process of decoder-only (lin- ear) Transformer with position embeddings given input s = S1 Sa. the attention is computed by the query at the position n and the keys at the positions i S n. The position of the query is special, advising the length of the input and reflecting the scale of the instance ideally. The PLAA model captures this feature and introduces the no- tation nxx to reflect the dimension of the subspace that belongs to. To further simplify and focus on the position embeddings, we assume that the value of each x; is identical to itself (i.c., we fix the value matrix in the attention to and thus Wy = x;)) since the value head not central to length generalization: the interpolator is expected to learn a suitable value head. We reasonably suppose correct value head to focus on the specific challenge in length generaliza- tion. Additionally, we assume the attention is only related to positions, highlighting the standalone impact of position em- beddings on inductive bias and length generalization. The contribution of the interaction between the position embed- dings is [ei..... e,.'' [nje, = Alj].n for some upper triangle matrix A. When the input is embedded to length N but the query is still made at the position n. the output of the model is x Aen(ar). i.e., the output of the PLAA model. Therefore, the PLAA model is a simplification of decoder- only Transformers focusing on the impact of the position embeddings on length generalization. For a more detailed elaboration on PLAA, see Appendix B.
Corollary 1. For any f E F such that I(f) et. [No]. the min-degree interpolator does not achieve LDHD general- ization from NNo to Xx and thus the RFM initialized at O and trained with GD does not achieve LDHD generalization from NN, to Xx.
Corollary shows that the min-degree interpolator and thus the RFM model can only achieve LDHD generalization for a very restricted set of functions that are only dependent on F[No]. Achieving LDHD generalization with RFMP models requires prior knowledge of the concept class to design the projection. Example 3 illustrates how LDHD generalization is possible for the target function with dependence beyond J[Nal by choosing a proper projection.
Example 3. Consider the target function f()) = 4x² + 302,
No = 1, and N = 2. The min-degree interpolator on XN. is fi (x) = 4m,, which does not achieve LDHD generalization on Xx. In the RFMP model, if we choose
0..
0.6
V =
0.6
00.8
then we have
In Definition 5, directly parameterize the PLAA model directly with the attention matrix. In practice, however, the attention matrix is typically computed by the interac- tion between the position embeddings. Therefore, we con- sider the PLAA models with the Absolute Position Embed- ding (APE) and the Relative Position Embedding (RPE), respectively. See Definitions 6 and 7. Note that we con- sider Generalized RPE (GPRE) in Definition 7 because the RPE can be seen as special instance of the GRPE with u = URRE = (Dr.... DN }. where D₁ is a k-th upper diagonal matrix such that (Dz)ij is 1 if = i + A: - 1 and 0 otherwise. We seck a more general result applicable to all similar parameterization methods to the RPE.
B(V) = (1, 1 + 0.6.2,, 0.6 - 0.822, (0.8z) + --.6x2) - 0.8r2))
The min-degree interpolator w.r.t. the linearly independent set B(V) on NN, is fz(z)= = 4r, + 3r²² fI)) which achieves L.HH) generalization on Xx-
4.2. Position-Only Linear Attention with Advice
In this subsection, we investigate Position-Onie Only Linear At- tention with Advice (PLAA), which can be seen as a simpli- fication of decoder-only Transformers (Definition 5), with a special focus position embeddings that are considered pivot to the length generalization of the Transformers (Shaw et al., 2018; Jelassi et al., 2023).
Definition 6 (PLAA with APE). A PLAA model with APE is
SPLAA (1; P) = x (M o PTP) en(a);
6

---

Low- Lov-Diinensson--e-llighhDDDD Generalization And Its Implications for Length Generalization
RNXN is the upper triangle mask, i.e., (MR) by is if S j and 0 otherwise, and P RdPxN is the learnable parameter of the model.
where Mî
Theorem characterizes that the PLAA with GRPE con- verges to the interpolator that minimizes the degree-profile w.r.t the linearly independent set &
Definition 7 (PLAA with GRPE), For U = (U?,...,UT,, a PLAA moodel with GRPE is
Theorem 4. For the L {UI,.... Ur}, define
{
GR JLAA (1; = I
EUpD en(x):
U.Di
GRPE, PLAA
Σ
(UR)igh (x)
ISISIEN
ISKST
where U. E U ,,...,r are upper triangle matri- ces that satisfy (U;, Ut) = for all I,...,r and (Ug)a (
Let Pt be the learnable parameter at the timestep in the training process where the learnable parameter p initial- ized at po = O and optimized with gradient descent/gradient flow under lz loss NNg- Let Noor" be the set of all in- terpolators on NNo for the concept cxx) SPLAA (x:p"). Then we have, as 1. - DO,
for all # j and S k,l S N, and p Pl, PI.....PPP]' E Th' is the learnable parameter.
GRPE.M
Remark 2. The condition (z (Ug) = 0 for all i * and 1 & k & N in Definition 7 means each element in the position-only attention is characterized by at most one parameter. This condition generalizes the property of RPE that (i S j) is only parameterized by Pj-i-
&GRPE,L JPLAA (x; + arg
min geg 14 No.p"
DegP
Brat (g).
For the PLAA model with APE, Theorem shows that it converges the min-degres interpolator w.r.t. the linearly independent set LLLAA
With the inductive bias of the PLAA with GRPE, Corollary 2 states that LDHD generalization can be achieved if and only if the target concept can be represented by the elements in BPLAA that have dependence on XNo.
Theorem 3. Define the set BRIA as
BRAAA= = (1,1 (x) (x))iSISION
Corollary 2. Consider the concept c(x) -PLLAA
&GRPE. (x:p"):
where
Σ k=
Σ ISISINN
(U
1-2;
i= Jn
(L)
k=j+1
1+ i<j.
k=j+1
2
Under the conditions of Theorem 4, the PLAA with GRPE achieves LDHDD generalization from XNo XN if and only if
Suppose that dp 2 N. Let P((a)) be the learnable pa- rameter at the timestep t in the training process where the learnable parameter is initialized at Po(a) such that PJ (ol)a(() aIN (ox > 0), and optimized with gradi- descent/gradient flow flow under & loss (denoted by L(P)) on XNo. Define (a) == lim Pa()). Let No.A- be the set of all interpolators on XN, for the concept c(x)= = SPIA (2c; )²). I. ((Poc(aa) = 0 for all 0 < a S Or (okg > 0 is some constant) and p = lim (a), then
{k (UR))Na))INo = 0} C { Cs -}.
Remark 3. While the projection in the RFM and the position embeddings in the PLAA may introduce an inductive bias that benefits LDHD generalization, they can reduce the ex- pressiveness of the models. The point align the models with the concept class, incorporating a strong inductive bias while maintaining sufficient expressiveness.
CPPE SLLA (x; P) = arg min gEg
DegP DegPBga((0)) (g).
5. Further Implications
For any function ʃ that can be represented by PLAA model, there exists matrix At E No such that f(x) = EisisisM No A!, A/PPPAA (x) for all x E XNo- Con- sequently, for fALA (z) P) = Eisisien Ag
We discuss further implications of the LDHD generalization perspective for practical length generalization techniques.
5.1. Chain-of-Thought for Length Generalization
we have Ay 0 for all; j 2 No. This implies that the PLAA with APE cannot achieve LDHD generalization for the concept e(x) (I; A*) if A; A O for some j 1212 1 No. which partially explains the limitation of APE for length generalization.
While the Chain-of-Thought (CoT) can lead to more vari- ety in the length of testing samples, it is widely used as an effective technique to improve the length generalization in various reasoning tasks. This seems contradictory if consid- ered in the original space of the input sequence. However, from the LDHD generalization perspective, we can see that CoT intrinsically changes the underlying hidden space by
The PLAA with GRPE can overcome the aforementioned limitation of the PLAA with APE in length generalization.
7

---

Low-Dimeension-t--ligg tiiniisiion-tt--ligiiinnnnnoonnn Generalization And Its Implications for Length Generalization
3.64
Accurasy
D-th
3.th
2.th
34th
BFC
64th
R7E-Square
54th
1952 2006
21000 madee 19l0n sable 1aiteo sodae 175000 20000
Steps
Steps
(a)
(b)
(c)
(d)
Figure 2. Length generalization of Transformer with RPE and RPE- RPE-Square in the unaligned copy and the URF addition tasks. Unaligned
Copy: Transformers with RPE (a) and RPE-Square (b) are trained on lengths 1-5 for 1000 steps and tested on lengths 1-10. While both models generalize in-distribution, only RPE-Square achieves out-of-disrribution generalization for lengths 6-10. URF Addition: Models are trained on URF 4-addition and tested on URF 5-addition. (c) Both the models are trained for 20000 steps. The comparison result shows that the RPE fails while the RPE-Square succeeds in achieving length generalization. (d) RPE-Square trained for 200,000 steps achieves nearly perfect accuracy, with digitwise accuracy shown for each Zi:-
extending each dimension with a "middle" variable and does not lead to the dimensional increase in the hidden space. For example, consider the n-addition without CoT and the n-addition with COT. In the case without COT,, the
query at j and the key at i is
exp (WWo)) E Eisrs exp (( exp (WWozz)
IEREHISISI
instance 2n-1 ... To + Mn-1 10-11-.0 ... yo 2n ... corresponds to the latent state h = ( ......., (In-1 1: Mn-1)) E Σ" for Σ (0....,97 2 In the case with CoT, one step of predicting corresponds to the latent state
x Eigus, exp ((Woz)) ((--))((-1))
where WQ and WK are the weight matrices for the query and the key, respectively. We replace RPE,-; with RPE-Square,, in the Transformer.
hn = [ yo, 20) ..... (Z1-1, V1-1, 2-11, (2,, yl #) ...., (In-1, 10n-1.8n-111 1n n
RPE-Square incorporates prior knowledge of LDHD gen- eralization and unaligned data formats by combining RPE with a mechanism to handle unaligned formats. The posi- tion embedding for a query j and key i is determined by the relative distance of relative distances the differ- ence between the relative distance of j to the position of some token Ii, and to the position of some token rk. pa- rameterized by ((--))((--)) This design, which inspires the name RPE- RPE-Square, uses weighted average over all < S j.1 < k < i, where the weights are derived from the product of attention scores between 2y and 27, and Ty and IĄ- This approach enables automatic learning of special tokens and is particularly suited for tasks involv- ing unaligned data formats, such as URF addition. Further illustration is provided in Appendix D.
for =x (,,0,...,, 9). where is special element indicating undetermined values. CoT does not cause the LDHD generalization challenge but extends the domain E, leading to a more easily learnable target concept.
5.2. Position Embeddings for Length Generalization
Position embeddings are considered closely related to the length generalization in Transformers. Our analysis sug- gests a principle for position embedding design: consider the inherent LDHD generalization and the data format nui- sance separately. To further elaborate, consider the length generalization of the URF addition with COT.. While RPE capture the recursive structure of the addition problem and could lead to LDHD generalization in the latent space (Zhou et al., 2023; Jelassi et al., 2023), it fails to work for the URF addition with CoT, due to the nuisance of the URF.
We compare the length generalization performance of RPE and RPE-Square in two tasks: Unaligned Copy and URF Addition. In the unaligned copy, the input is string whose length is not aligned to a fixed length, and the target is one copy of the input string. An unaligned copy instance of scale like "[BOS] Xo ... Xn-1 Xo x [EOS]". The URF addition is illustrated in Example 1. To examine length generalization, the models are trained only small- scale instances but evaluated on instances of larger scales. More details of the experiments are in Appendix D.2. The experiment results are presented in Figure 2. In the experi- ments, RPE- uqurre, the position embedding derived accord- ing to the LDHD generalization perspective, can achieve
Following the principle, we design novel position em- bedding called RPE-Square to handle the nuisance of the unaligned data format. On the one hand, we keep the RPE structure for the inherent LDHD generalization. On the other hand, we deal with the unalignment by considering the distances to several special tokens (c.g., [BOS], +, and =). These considerations lead to the RPE-Square, in which we compute the relative values between the distances to the special tokens. More concretely, the RPF-Square; for the
8

---

Low.DDieensio--oo-iggg--DDDi Generalization And Its Implications for Length Generalization
length generalization in both tasks, while RPE fails. The Bartlett L.,., Montanari, A., and Rakhlin, A. Deep learning: periment shows that RPE-Square effectively improves RPE in handling the unalignment in the data format.
statistical viewpoint. Acta numerica, 30:87-201, 2021.
d' Ascoli, S., Bengio, S., Susskind, J., and Abbé, E. Bool- former: Symbolic regression of logic functions with trans- formers. arXiv preprint arXiv:2309. 12207, 2023.
6. Conclusion and Discussion
We propose the LDHD generalization perspective for length generalization, which disentangles the problem into as- pects: LDHD generalization in the latent space and handling data format nuisances. From this perspective, we introduce the No-Free-Lunch Theorem of LDHD generalization, high- lighting the necessity of inductive bias for achieving length generalization. Using Boolean functions, we investigate the inductive biases of different model architectures trained with (S)GD and how these inductive biases contribute to LDHD generalization. Our perspective on LDHD generalization further elucidates the role of CoT and leads to the principle of position embedding design in length generalization.
Feng, G., Zhang, B.,
Gu, Y., Ye, H., He, D., and Wang, L. Towards revealing the mystery behind chain thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. Mathematical capabilities of ChatGPT. arXiv preprint arXiv:2301. 13867, 2023.
Garcez, A. d., Bader, S., Bowman, H., Lamb, L. C., de Pen- ning. L.,, Illuminoo, B., Poon, H., and Zaverucha, C. G. Heural-symbolic learning and reasoning: survey and in- terpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342(1):327, 2022.
For future work, while our theory is established with sim- plified models, it crucial investigate inductive bias in more practical and complex models. Additionally, develop- ing a paradigm for position embedding design based on the proposed principles is valuable avenue.
Gunasekar, S., Woodworth, B. E., Bhojanapalli, S., Neyshabur, B., and Srebro, N. Implicit regularization in matrix factorization. Advances in neural information processing systems, 20,300022 2017.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Huang, X., Yang. A., Bhattamishra, S., Sarrof, Y., Krebs, A., Zhou, H., Nakkiran, P., and Hahn, M. A formal framework for understanding length generalization in transformers. arXiv preprint arXiv:
2024.
Acknowledgements
Jelassi, S., d' Ascoli, S., Domingo-Enrich, C., Wu, Y., Li, Y., and Charton, F. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.15400 2023.
Z. Lin and Y. Liang were supported by National Key R&D Program of China 00922D0160300) Z. Lin was also sup- ported by the NSF China (No. 62276004). Y. Liang was additionally supported by CCF Baidu Open Fund.
Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
References
Abbe, E., Bengio, S., Lotfi, A., and Rizk, K. Generalization the unseen, logic reasoning and degree curriculum. In International Conference on Machine Learning, pp. 31-60. PMLR, 2023.
Lee, N., Sreenivasan, K., Lee, J. D., Lee, K., and Papail- iopoulos, D. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023.
Mcl S., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A., et al. Transformers can do arith- metic with the right embeddings. Advances in Neural Information Processing Systems, 37, 2024.
Ahuja, K. and Mansouri, A. On provable length and compo- sitional generalization. arXiv preprint arXiv:2402.04875, 2024.
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization large language models. Advances in Neural Information Pro- cessing Systems, 35:38546-3855 2022.
Neyshabur, B. Implicit regularization in deep learning. arXiv preprint arXiv: 2017.
O'Donnell, R. Analysis of boolean functions. Cambridge University Press, 2014.
9

---

Lov.Diinenssoo-toiiighhhnmin Generalization And Its Implications for Length Generalization
OpenAl. ChatGPT, https::/cchat.opeaa com,
Xiao, C. and Liu, B. Conditions for length general- ization in learning reasoning skills. arXiv preprint arXiv:2311.16173 2023.
2023a.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08874, 2023b.
Zhang, S., Chen, Z., Shen, Y., Ding, M., Tenenbaum, J. B., and Gan, C. Planning with large language models for code generation. In The Eleventh International Conference on Learning Representations, 2022a.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 2021.
Zhang, Y., Backurs, A., Bubeck, S., Eldan, R., Gu- nasekar, S., and Wagner, T. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint S.Kiv:2206.0430/1 2022b.
Razin, and Cohen, N. Implicit regularization deep learning may not be explainable by norms. Advances in neural information processing systems, 33:21174-21187, 2020.
Zhao, Z., Lee, W. S., and Hsu, D. Large language models as commonsense knowledge for large-scale task planning, In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
Razin, N., Maman, A., and Cohen, N. Implicit regulariza- tion in tensor factorization. In International Conference on Machine Learning, pp. 8913- 8924. PMLR, 2021.
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algo- rithms can transformers learn? a study length general- ization. arXiv preprint arXiv:2310.
Shaw, P. Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint 1oX3:0/8030021555 2018.
2023.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocompuling, 568:127063. 2024.
Zhou, Y., Alon, U., Chen, X., Wang, X., Agarwal, R., and Zhou, D. Transformers can achieve length generalization but not robustly. arXiv preprint 40X1v:2402.09371, 2024.
Teney, D., Nicolicioiu, A. M., Hartmann, V., and Abbas- nejad, E. Neural redshift: Random networks are not random functions. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Paltern Recognition, pp. 4786-4796, 2024.
Valmeekam, K., Marquez, M., Sreedharan, S., and Kamb- hampati, S. On the planning abilities of large language models-a critical investigation. Advances in Neural Infor- mation Processing Systems, (6:75993-76003 2023.
Wei. J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Wolf, T., Debut, L., Sanh, V., Chaumond, J.. Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing, 2020.
Wolpert, D. H. The supervised learning no-free-lunde theo- rems. Soft computing and industry: Recent applications, pp. 25 42, 2002.
Wolpert, D. and Macready, W. G. No free lunch theo- rems for optimization. IEEE transactions on evolutionary computation, 1(1):67-82, 1997.
10

---

www.DDieension----lligg--DDnmii Generalization And Its Implications for Length Generalization
A. Background on Boolean Analysis
In this section, we include brief background on Boolean analysis essential to this work. We refer to O' O'Donnell (2014) for further details and comprehensive coverage of Boolean analysis.
Fourier Expansion. A Boolean function f (-1, 13"h R can always be represented by
f(x)
Σ f(T)xr(z),
(I)
TC(M)
where Xr(x)) = xi The polynomial in (1) called the Fourier expansion of the Boolean function f. The number ((T) is the Fourier coefficient of f on T. The set (xT(x))rc(m)) forms basis, named Fourier basis, for the product space w.r. the inner product defined by
(f.)) = -
[())))))))))))))))))) [f(x)g(=))
Degree and Degree Profile. The degree of a Boolean function f' the degree of its Fourier expansion, which is polynomial. The degree profile of f. denoted by DegP a (n + 1)-tuple where DegP.(f) = Erc(a), T| - The order between two degree profiles is lexicographic. The degree profile can be roughly seen as the distribution of the degrees of the monomials in the polynomial. Intuitively, the degree profile reflects the "complexity" of the Boolean function. lower-degree-pront profile Boolean function uses fewer variables combines them more simply than higher-degree- one.
B. Detailed Explanation of PLAA
In this section, we illustrate how PLAA and its variants abstract the effect of position embedding on the LDHD generalization of decoder-only Transformers.
B.1. Construction of PLAA
A typical linear attention at query n can be expressed as
fi (z; Wo, WK, Wv, B)
Σ (Woor)) (WKz.) + Bi,a] Wyz;,
where Wo. WK. Wy are the query, key, value matrices, respectively, and B E RNN is the position bias. To focus on the impact of the position embeddings, we fix Wv = I and consider position-only attention score. Then we have
SPL (w; B)
Σ Bin Σ De.Bentt
iSn
Rewriting as A and restricting A to an upper triangular matrix, we have
fPLA (x; A) = Σ Aen = =T
where A E UN.
In the computation of attention, the query position asymmetric to other positions and can reflect the scale of the instance. To formalize the intuition, we introduce the notation nıx) := arg max N = -1), which represents the last dimension where 2; -1 In the LDHD generalization framework, n(x)) can be interpreted the lowest dimension of the subspaces containing x, reflecting the dimension of æ. We take n(x) as the query position of z. Specifically, when r is all-ones, it is treated as an "empty" sequence. In this case, we fix its output to align with the definition of PLAA. By the above derivation, we obtain Definition 5.
B.2. Construction of PLAA with APE
The expression of the linear attention with APE is slightly different from (2). In practice, APE is commonly added to the token embeddings. Therefore, we omit the position bias and add APE to each x, when computing the attention score (we slightly abuse the notation x; to denote the token embedding), i.e.,
f
xy Wq. WK. Wv, (Di)sE(M)
Σ
WW (s. + PT)] [Wx (z, + p:)) Wvz.

---

www-DDinensoo--t--llihg-----me Generalization And Its Implications for Length Generalization
where p; is the position embedding for the position i.
Similar to the derivation in Appendix B.I, we have
PLAA (2; Wq: Wk)
Σ iSm(n)
Ph(aa W "WKPP
Σ
et m(x)
PTW
T Peçti
iSm(m)
where P is the learnable position embedding matrix such that p; = Pe, for all i e [N]. Without loss of generality, absorb the learnable parameters Wq. Wx into P, obtaining
CAPE PLAA
(z; P) Σ R
æTe PT = =x (M PTP) en(a)
B.3. Construction of PLAA with GRPE
RPE can be seen as reparameterization of the position bias matrix such that the entries on the same diagonals share learnable parameters, i.e.,
B
DaP:
where Dr is the k-th upper diagonal matrix and Pe is the learnable parameter for all 1 k S N.
We generalize RPE so that our model can cover more general position biases. A natural extension is to choose general To normalize, we suppose r. (This condition does not hold for MAPE. We can slightly modify the matrices by replacing
upper triangular matrices, denoted by MGRpE = ... U,}. instead of URPE = (D,....,,y,
(U;, U,) = 1 for all i :....:r.
D& with N+ DR to satisfy the condition.) We further require (U,))aUU)R 0 for all i j and 1 S k, N, i.e., each Brg is characterized by at most one learnable parameter. Using the reparameterizztion
U.P
in the derivation of PLAA, obtain PLAA with GRPE in Definition 7.
C. Proofs
C.1. Proof for Theorem 1
Let D == (Xy | x E XN.) be the conditional distribution given that E AN, and D2 = D (NN 1 $ XN₂) be the conditional distribution given that x & ANo
Σ
|EF/ (cFl(N))
=P E XNo)
Σ
ENND [(((
SEFI (EF((O)XNN))
+ (p & XNo)
Σ
EOND, (((,()) f(x))
JEFi(cllmni
=Polxw)( e (N.)Vi((11 (D(XW)(E & XN, Vo(ci)
where
vi(c):= = Σ
EAND, (e((()),
FEFI(cxNo)
Ve(c)= = Σ
((
SEFI (cxxxo)
12

---

Low-DDinensson--t--iigg.. Diminssoo Generalization And Its Implications for Length Generalization
Similarly, we have
E
BENDIX) ((e()) = (D(xw) E (N,)
+ (D & NN, (2(o))
SEF/ (calNN))
To prove the theorem, it remains to show that Vi(ci) Vi((e) and V2((1) V2(c2).
By the definition of F// (clxxx)) we have
[((a)=
E
BEDD, [((c())),(())
/EF/ (EFI(cilNNo)
Σ
EEND, ((12)((((())
feF/ (
E
EAND, ((22()),
/EF/ /EF/(caNNNO)
Σ
BAND
= Vi(e),
SEF/(calxxo)
where the equality (a) is due to that ci (x) = c2(x) for all NN..
By the no-free-lunch theorem (Wolpert & Macready, 1997; Wolpert, 2002) for XN NNo. V. and Fxw Xx, ys we have
EanD, [c))(
Σ
ExnD, (e()
JE
JENNW
Then we have
Vz(ci)=
Σ
BAND, ((_)))((((x))((1))))
FEF1(allxNo)
Σ
EIND, (())
IEFxy\AN, 3
Σ
E ((()(()),
JEFxyiNNg
Σ
BEND, ((()))(()))),](((())( V3(e2).
(EFl(c))xxO)
C.2. Proof for Theorem 2
Let =(x)= Proj(z, V) and NNo (Proj(æ, V) x E AN.... Let I'o.. be the set of all RFM interpolators on NN, for the concept c(2), i.c.,
NNo.e = (RFM(2; a) (REM(2;
= c(2) for all 2 E 2No).
where
k
fRFM(2; = JK
Σ
(1o(( bb).
According to the proof of Theorem 3.8 in Abbe et al. (2023), the RFM model fRFM(Z; converges to the min-degree interpolator (wrrt. the variable 2) in Viv.,ri.. i.c.
farM(2;))) (4) + arg min DegP Fr (g'), as K + t 00,
o' g'E
13

---

Lov-DDinensio----iliggt--Diini Generaliziion And Its Implications for Length Generalization
where B' = (x
By the definition of z(x) Proj(z, V), we have
V.K J RFMP [I: (I:a) = fREM((; - arg min g' g'ESingi
min DegP ( (g')
arg min gE9No.e.
DegP B(V) (g),
as K + 00,t oc.
C.3. Proof for Corollary 1
Lemma 1. For any f F, the min-degree interpolator f" on NNo satisfies T(f*) [No]-
Proof for Lemma Assume that there exist some f F such that the min-degreee interpolator f of f on XNo does not satisfy I(f) c. [No]-
Let [1x) be the function constructed by fixing all wi, i & [No] to I in f(x). By the construction, we have f(x) = fxx) for all r E NNo and thus f(z) is also an interpolator of f on NNo.
Since I(f) & [No]. we have Df) < Df) This contradicts the assumption that f is the min-degree interpolator.
By Lemma 1, for any F such that T(J) & [No), the min-degreee interpolator f is not identical to f 2' and thus does not achieves LDHD generalization from NN, Xx.
C.4. Proof for Theorem 3
Let (NNo)) be the uniform distribution over XNg. The loss function is
1
L(P) JE E (Xxx)
(JPLAA(Z; P) fFLAa(E;AA))
2
1
(IT (M& PTPP) En(s) =I A'Cn(()) )²
E E (Xxx)
2
1
((ze) [e) o pTP) - -(reT(a)) A-'
A4r) })
E (Xxx)
2
1
( [ (rehh MN pTp) - -(zehie) MEA-)) )
2
E B (XNo)
2
-'p--''p' 2 (M o (()) PTP A') pTP-A-1)
1
-4-))
=E E (xxx)
where (a) follows from the fact that A upper triangular.
Lemma 2. For any matrix Z E NNNN holds that
(( xeT n(a)
1 11Q
E P
2
where
2
2- 3
0
No-1
2-
0
0
2
0
R N
No
2-
0
0
0
0
14

---

Lov--Dinension-----iggg.-- Dimensiion Generalization And Its Implications for Length Generalization
that is,
141
IsisiSM
(QNo)ss =
0:
otherwise.
Proof for Lemma 2. Note that
No
E U (Xxx)
1 ((MM
xel n(o) x))
(MM o oze 2))²
2
NNo
k=1 a: (a) k
No
k-1
2
Σ
Ty Zik Zuk
N
k=1 a: (at =k
No
Dz
2No
D: (1))
i=1
(0)
No
2
2
Z
2No
2
km
No
2
Na
Zio
1
11
21
2
2
k=1
where (a) follows from the fact that {x|
n(x) = kb}l = 2k-1
According to Lemma 2, we have
1 L(P) 1Q o (PTP- A*)
(3)
For any 0 < < S Cg. we have L (Poc (ai) and thus
(M% Poo (a) Poo = Ainol.1Nol
(4)
Let Pt E Rd denote the i-th column of P. According to (3), we have
àL Op: = 0,
for all i = No + N.
N. Therefore, (p:), = (pi) for all and i = No + 1, I....,N. N.
Define
A+ (A))
isisis No. otherwise.
0,
Let pf be the i-th column of Poo (a). We have
Mr o (a) - 1
Σ
(WA) -Ai))
ISiSiSN
N
((x)) p1)))
j=N
Σ
lI 12 ||,
j= i=1
No
Σ
1pp 11
Σ
l 1 |0
j= No 1=11
N=No+1 No+1
15

---

Low-DDinension-t---iggg--- Dimmnsson Generalization And Its Implications for Length Generalization
where (a) follows from (4) and the definition of Á
Note that 1f A S | for all 1 S sisa No and | for all No + -issi N. Then we have
0 0 M o PI (a) 1-12 S SN(N- -N(N- No)a + (N No + 1)N No) a
As a - 0, we have
0 s Ma o pTp - x. 1-1. 50,
which means My o PrP = A'.
It remains show that SPLAA (z; P) is the min-degree interpolator w.r.t. BLA on XN.. Notice that
6PLAA()) (x) ( (xee (re Ess)
bij
Then we have
FAPR SPLAA P);
E ANjo (z) = E
(x).
ISISIE
ISISiENO
For any g = Eisieign Agg E GPLAA we have Arj = Aig for all sisjs
S No. Since A 2 Ar for all
S i Disjs N, we have
DegP BELAA (JAPE (JPLAA P)) I DegP Ba (g)
Hence, SPLAA (z; P) is the min-degree interpolator w.r.t. BPLAA on XNo.
C.5. Proof for Theorem 4
Note that
GRPE.M JPLAA
(xip) = x
Σ pnllia en(ar) = zeT n(z)'
Σ PaUix
k=1
Let DN, be the size of the training set XNo.
Then the loss can be represented as
2 (z)
L(p)
Σ
xeT R(x)*
EpuU )
2DN
TEXNO
Without loss of generality, we use the notation of gradient flow in this proof. Then we have
Σ
(o)) -
c¹ cx())
Σ
(re Un) Uk.
= DNo NENNO
km
Since p(0) have A(0) 0 and thus A(t) E span
Ekm ( (ze) (ze Un
Then the convergence point Á
=ENNo
can be represented Å = EEERNO a(x) Ek= (re Uk) UK. The convergence function f is
f(x) Σ a(x') Σ
(''eT (erry Un) ( (DeTle)
N'ENNo
k=1
Σ
a(x') (al eeerreryy Un) (zehlo)
A= ='
16

---

Lov-Dimension--t--iigh... Dimension Generalization And Its Implications for Length Generalization
Lemma 3. For any upper Iriangle matrix U RNXN we have
(2e (())))))))
E
U,5b(fA((2) (z).
ISiSISN
Proof for Lemma 3. We first prove that (())
n(n) Emn) = PPLAA (x) for any 1 S i i j S N. Notice that (j
4j Zj - = ... ...................... A
(re maa) Bas)
-
otherwise.
Therefore, we have
Tj.
1
Ij+1
1 xN
(re) ( Pig)
= 1zj
2
2
1-zi
k=j-1
N
14th 2
i= j.
1-2,
k=
1tzs 2
1 < j.
PLAA (x).
For any upper triangle matrix U = EiSiSieN Uiz Eij, we have
(re U) = (se) n(z)'
Σ
UnjBot) = E
(())
Eij)=
Σ
ISiSiSN
ISISiSN
ISISiSN
Define bx(x) == (re Uk). By Lemma 3, we have bx(x)) EnsIsiSN(UR))) (UK)) Then the convergence function can be represented
f(x) = Σ
a(z')ba(z')bel = Σ
Paba (x); 1=1
1= x' EENNO
where Pu: ExEx,, a(x') a(z')bu(z').
For any interpolator g(x) = E on XNo:
by (2)) for all XN.. we have Pa and 1h 2 0 p3:
p3::
If by (x) for some x E XNo. we have po ps and thus pi - př To show this, without loss of generality, we suppose
that for 1 < &o < m, there exists some E XN,, such that (x) * 0. and for m + 1 < & I r. bb(x) 0 for all XN,. It suffices to show that the following equation has unique solution:
bi (x1) :
(m))
Pi
c (x1)
(xM)
(x
Pm
(xM)
where MM |N' and XNo = f.I,..., N..). Since p P1.... Pm = Pm is solution, it remains to prove the uniqueness. Let
(1(11)
bm (z1)
Bm
bi (xT)
bm (xr
It suffices to show rank Bm) = m. Note that (mT,Bm)s = Exex bx(x)² > 0 and (BT, Bmlis ExENN, (x))))((x) O. We have rank BT, Bmn) m and thus rank Bm)) m.
17

---

Low-Diieension-oo-iigg--DDD Dimensison Generalization And Its Implications for Length Generalization
Hence, for any interpolator y on NNo. we have
DegP gr (f) S DegP HAA
gURRE (g),
PLAA
or equivalently,
GRPEI PLAA
(z: + arg
min g
DegP se (g), as l - oo.
PLAA
C.6. Proof for Corollary 2
By Theorem 4, the PLAA with GRPE achieves LHDH generalization from Xx, Xy if and only if the target concept c(z) is the min-degred interpolator w.r.t. the linearly independent set PLAA Therefore, it equivalent to prove the target concept c(x) is the min-degree interpolator w.r.t. the linearly independent set EGRPEE if and only if
I (U)))N = 0} S { Ck = 0}.
We denote {kl (N.)ON = -3 by 1C and { c = 0} by K2 in this proof.
We first show the sufficiency. Assume that c(x) is not the min- degree interpolator w.r.. the linearly independent set PPLA on XNg. In other words, there exists interpolator ?(x: Σi=1 EisisieN (5 N',, such that
DegP
(c)
DegP gawe.u (c).
PLAA
FLA
Since both c(x) and ((+) are interpolators on ANo. we have
Ca (UR))
Cx (U(())(N)))1N))>
or equivalently,
Σ RgK.
ca (Uk))No)))
Σ (c((R))NNoLLNNo)
KgK.
Since (U:) 0 for all i and < k, N. we have (Uk))) kerc linearly independent. This implies C = Eか for all l & Ki.
Since Kr K2, we have q and thus ot s o for all ls E K2. Hence, have
Degl' aRRe (c)
Degl parrs. (o),
which contradicts the assumption.
We then prove the necessity. Assume that Kr, K2. Then there exists some kn E Kr, but ho & K2. Define ö(x)
E Čk (Us)s) (2) (x) where
(o,
k = ko
Ck. k f ko-
By the definition of Kc, ē(x) c(x) for all XNo and thus č(ι) is also an interpolator on Xo. Note that Cho 0. By the definition of č(z), we have
DegP
(
DegP
(c),
BRLAA
which contradicts that the target concept c(x) is the min- nen-eegree interpolator w.r.t. the linearly independent set BPLA on XN.-
GRPE,
18

---

Generalization Lov-DDinensoo-----iighhhDDDoaan Senaaaaaa And Its Implications for Length Generalization
D. Experiments
D.I. Detailed Explanation of RPE-Square
The design RPE-Square follows the principle: when devising position embeddings for length generalization, one needs to consider both the LDHD generalization in the latent space and the nuisance of the data format in the input sequence space. This principle derived from the LDHD generalization perspective for length generalization, illustrating the insights of LDHD generalization for practical models.
To further elaborate, show how RPE-Square is constructed in the guidance of the proposed principle to achieve length generalization of the URF addition (with CoT).
LDHD generalization in the latent space. For the addition, the LDHD generalization in the latent space can be handled by RPE.
The data format nuisance of URF. To handle the data format, we need to consider the mapping between a latent variable and its URF string, i.e., how the latent variable h [(ro, Vo, 20)...., (24-11, 1, 1/- ze- (2i,, Mt. *),...> (In 1, 1,, *)] can be recovered from the corresponding URF string "[BOS]xx ... x Yo ................................ Vay-1 2p... 2--1" (to predict z,). The URF mapping be described as follows:
concatenate "[BOS]", the first elements of the dimensions (up to the "highest" nonzero element), "4", the second elements of the dimensions (up to the "highest" nonzero element), a "=", and the third elements of the dimensions (up to the "highest" non-"*" "** element).
According to the URF mapping, we notice that the "position" of an element in the latent variable can be identified by its relative distances to "[BOS]", "4", and "=". (Here, the relative distance from the position i to the position j is - j.). Denote the tuple of the relative distances from some token s to "[BOS]"
" by (m1(s), (12 (2) (s))). Then we have
(e)-1)
ni (8) > 0, (2)) < 03)) <0,
8
1n1(s)-11 m1(s) > 0, (2 > 0, 0,
1ns(s)-12
Ti (5) 0, (2())) 0, 0.
Note that 2 can be determined by #k-1, Th. 1k-1- 1. yks and 2/ (we ignore the boundary cases in this discussion for need to identify the elements
simplicity). To predict the token of some where mi (s) 0, n2( 0, (g)) $1, 82, $3, $4, $5 such that
(s1s)) = ns(s) - 1, (2((1) < 0,
na < 0,
(12) = ny(ss,
(2( < 0,
ny na( ($2) < 0,
n2( = ng(3) 1,
(1(3)) > 0,
nig (83) < 0,
(2( = ng(s),
m1(s1) > 0, na (a( < 0,
na(s)) na(s) 1,
mi(ss) 0, n2(ss)
Therefore, to address the URF data format nuisance, the position embedding can consider the relative distances some
tokens.
D.2. Experiment Details
D.2.1. UNALIGNED COPY
An instance of the unaligned copy task is like
Xx ... Xn-1 Xo Xn-1 e.
xg...X.-1 Xo-1- We use "b" and "e" instead of The model is given the input bXo ... xp-1 and expected output the copy of xo ... "[BOS]" and "[EOS]" for simpler implementation with the GPT-2 tokenizer.
We sample 2000 rellngth instances for each n = 1...., 5 as the training data. In the evaluation, we examine the learned models on instances of length 10. We train GPT-2 with key-only RPE and RPE-Square, respectively. The model trained by AdamW with the cosine scheduler, where the initial learning rate is 0.0005, the weight decay is 1.0, the warmup ratio 0.05, the gradient accumulation step is 2, and the per-device training batch size is 256. We set the training steps to
19

---

Low-Dinnension----ligg---- DiimessoD Generalization And Its Implications for Length Generalization
10000 but early stop step 1000 the model with RPE-Square has achieved nearly perfect length generalization while the model with RPE shows almost length generalization then.
From the perspective of LDHD generalization, the latent variable corresponding to the input bXg .n. .x.... .xxx
[<ro vo)..... ..... (Tk, TR), (Ft-1; ....... (xN
The LDHD generalization in the latent space can be efffectiveyy addressed by RPE. The data format mapping can be handled by considering the relative distance to the tokens "b" and "=". Therefore, RPE-Square is expected to work for the length generalization of the unaligned copy. RPE does not properly deal with the unaligned data format and thus could fail to achieve length generalization in this scenario.
D.2.2. URF ADDITION
For the URF n-addition training data, we first sample the lengths of addends uniformly from {L...., n2 {...... n... n). For two addends of lengths (11, n2),, we then samples from [10 10 1] x [10"2-1, 10 - 1] (If n; then the corresponding sample interval is [0, This is guarantee the addends are length-uniford uniform. For the addition Eng Ema- Zp. the training instance is
-n--1 ....0 + Dng-1-- yo
xxo xm-- yo ... Nng-1 Zo ... 2ny-1 1 Zng
Here, we add spaces between the characters to ensure each is tokenized separately. In our experiments, train with 10000 URF 4-addition samples.
We choose GPT-2 with key-only position embeddings as our model. For the RPE and RPE-Square settings, we augment the GPT-2 model with RPE and RPE-Squart respectively. The implementation is adapted from HuggingFace (Wolf et al., 2020).
We train the models by AdamW, with the initial learning rate 0.0005, the weight decay 1.0, and the cosine scheduler. The warmup ratio is 0.05. The gradient accumulation step is 2. The per-device training batch size is 128. We train the models for 20000 steps and 200000 steps. The experiments are run server with Ubuntu. The models are trained on two NVIDIA GeForce RTX 3090 GPUs.
E. Additional Experiments
E.1. Additional Evaluations of RPE-Square
We present more evaluation results of RPE-Square by comparing against more different position embeddings in additional tasks. Concretely, we consider three extra tasks: ParityCoT, Multiplication (1 * N), and Division N). Across all tasks (including Addition and Copy), we compare APE-Square with five position embeddings: RPE (Shaw et al., 2018), RoPE (Su et al., 2024), NoPE (Kazemnejad et al., 2024), ALiBi (Press et al., 2021), and Abacus (McLeish et al., 2024). We keep the hyperparameter setting for the addition position embeddings in Addition and Copy: the model is trained by AdamW with the cosine scheduler, the initial learning rate is 0.0005, the weight decay is 1.0, the warmup ratio is 0.05, the gradient accumulation step is 2, and the per-device training batch size 256. In the three extra tasks, we choose the initial learning rate as 0.0005 and keep the other hyperparameters the same.
Figures 3-7 presents the experimental results. Both RPE-Square and Abacus achieve non-trivial length generalization performance in the five tasks, and the overall evaluation results of RPE-Square are better than those of Abacus. The other four position embeddings do not achieve good length generalization performance. The reason, from our LDHD perspective, is that RPE-Square and Abacus succeed in handling the unaligned data format, while the others do not.
20

---

Low-Dimension-tt--lin Dimeiiinn Generalization And Its Implications for Length Generalization
10-
RPE-Square
Accuracy
RoPE
NoPE
ALIBI
Abacus
Step
Figure 3. Addition. The models are trained on URF 4-addition and tested on URF 5-addition
Accunay
Accurity
Step
(x100)
Step
(a) RPE-Square
(b) RPE
RoPE
Accuracy
Recuraoy
Rccumoy
10
Step (x100)
Step (x100)
Step (x100)
(d) NoPE
(e) ALiBi
Abacus
Figure 4. Copy. The models are trained on scales 1-5 and tested on scales 1-10.
21

---

Low-Dimension-to--lin Dimeiiiin Generalization And Its Implications for Length Generalization
Accuracy
Accuracy
Step (x200)
(x200)
Step (x200)
(a) APE-Square
(b) RPE
(c) RoPE
Accuracy
Accuracy
Accuracy
(x200)
(x200)
(d) NoPE
(c) ALiBi
(f) Abacus
Figure ParityCoT. The models are trained on scales 1-5 and tested on scales 1-10.
instance
like
"[BOS] Xa y--t [BS)"" where yo To and @ Vc-1 for k 1,...,7 1.
Accuracy
Accuracy
Step (x1000)
Step (x1000)
(x1000)
(a) RPE-Square
(b) RPE
(c) RoPE
Accuracy
Accuracy
Step (x1000)
Step
(x1000)
(d) NoPE
(e) ALiBi
(
Abacus
Figure Multiplication (1 N). The models are trained on scales 1-5 tested on scales 1-10. instance like
"[BOS]x yo... Jn-1 [EOS]".
22

---

Lov-Dimension-tt--ie tiiiiiinnoniiiiggiiiiiooioooo Generalization And Its Implications for Length Generalization
Accuracy
Acturacy
Step (x1000)
Step (x1000)
Step (x1000)
(a) RPE-Square
(b) RPE
(c) RoPE
Step (x1000)
Step [x] 000)
Step (x1000)
(d) NoPE
(c) ALiBi
(f) Abacus
Figure 7. Division (N/1). The models are trained on scales 1-5 and tested on scales 1-10. An instance is like
"[BOS]x] Ya-1... Yo 2a-1 2o [EOS]", where ///z.
23

---

hww-Dimnnsioo----llig-----noee Generalization And Its Implications for Length Generalization
E.2. Additional Application of the Position Embedding Design Principle
In this subsection, we present another application of the position embedding design principle besides RPE-Square. We consider task called AdditionMod10, which computes the sum of the addends modulo 10, in the unaligned format. An instance takes the form
[BOS Xn-1 yo ... Yn 2 ZEOOS)
where = = (xr + y) mod 10. Due to the modulo operation, the output depends solely on the first digits of the addends. This means the positional embedding need to capture the absolute relative distances to specific tokens (namely, "[BOS]" and "+"). The "absolute value" handles the LDHD generalization in the latent space, and "the relative distances to some special tokens" deals with the unaligned data format. Following the design principle used in RPE-Square, we introduce new position embedding called RPE-Absolute encode "the absolute value of the relative distances some special tokens". The expression of RPE-Absolute,, is
exp (V Ri-k-
Σ ISkS
Eierei exp (WWKz)))
where WK, Wo, R are the learnable parameters.
The results shown in Figure 8. We compare GPT-2 with RPE-Absolute against GPT-2 with RPE. The training hyperparameters are identical to those in the experiments of RPE- RPE-Square with the initial learning rate 0.0001. The results demonstrate that RPE-Absolute achieves length generalization, while RPE only generalizes within the training lengths. This finding further supports the effectiveness of our position embedding design principle.
12-
28
Accuracy
Accuracy
2.8
6.1-
10
Step (x200)
Step (x200)
(a) Rbs-lus
(b) RPE
Figure 8. AdditionModlo. The models are trained scales 1-5 and tested on scales 1-10.
24


Reviews:
[{"review_id": "eBFnwscsha", "reviewer": "Reviewer_4S48", "summary": "*Despite my best efforts, I found this paper very hard to parse. I am discounting my confidence to reflect the same.*\n\nThe paper studies Low-Dimension to High-Dimension generalization (LDHD) problem, a special case of OOD generalization. At its core, the paper argues the impossibility of generalizing to high-dimensions without imposing inductive biases or prior knowledge. They then consider implications of their argument on length generalization for unaligned copy and addition tasks, and propose RPE-Square (relative position embedding) that leverages some domain knowledge such as the distance to the special tokens (such as =, +, [BoS], [EoS]). The paper then demonstrated RPE-Square's generalization to high-dimensions. \n\nThe main contributions of the paper are the following. (a) a formalization to disentangle the data format and dimension of the problem, (b) through analysis on binary functions and various positional encodings (relative, absolute and general), the contribution to the attention scores from positional embedding is min-degree interpolator wrt to a linearly independent set, (c) a new positional embedding scheme: RPE-Square.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. Below, we respond to your concerns. \nAll experiments results are in the anonymous link: https://www.dropbox.com/scl/fi/52t23nfzev1lo1sq5dmyj/ICML_2025_5857_Rebuttal.pdf?rlkey=5nde6aampze744klvn0rsb3mp&st=37lg3nzo&dl=0.\n\n1. *... However, the paper did not justify how their choices do not compromise the generality of their analysis.*\n\n(a) We isolate the positional contribution in attention to highlight how position embeddings affect inductive bias and length generalization. This helps us better understand their standalone impact, and the empirical results support this analysis.\n\n(b) We assume an identity value head because it is not central to the question of length generalization. Intuitively, we expect that the interpolator learns the correct value head to perform in-distribution generalization. So when focusing on length generalization, it is reasonable to assume a correct value head and analyze how the model handles larger-scale instances. This simplification helps clarify the role of attention and position embeddings.\n\n(c) While our analysis focuses on binary functions, the core insights naturally extend to tasks over finite alphabets, because we can consider the binary representations. For example, for an alphabet of size $N$, each symbol can be encoded using $log N$ bits, and the analysis for binary functions can be applied similarly. Though not identical, the notion of the \"minimum-degree interpolator over a linearly independent set\" continues to hold in spirit under binary encoding schemes. Thus, the results we derive for binary settings are relevant to broader cases like language modeling.\n\nThank you for pointing out this oversight and we will include the justification in our manuscript.\n\n2. *... It is expected that the paper at least present how their approach extends to other tasks, and what kind of domain knowledge can be exploited for different problems.*\n\nWe conducted additional experiments to address this concern:\n\n(a) We apply RPE-Square to three new tasks: Parity (With CoT), Mulplication (1 * N), and Division (N / 1), and compared its performance against more baselines (RPE, RoPE, NoPE, ALiBi, Abacus). The results (Figures 3 - 5 in the link) show that RPE-Square can also achieve length generalization in these tasks, demonstrating its applicability beyond the initial two tasks.\n\n(b) We also consider a task called AdditionMod10, which requires different domain knowledge from what RPE-Square captures. AdditionMod10 computes the sum of the addends modulo 10. The result depends only on the first digits of the addends (due to the modulo operation), so positional embedding needs to capture the absolute value of the relative distances to some special tokens ([BOS] and \"+\"). Following the same design principle, we consider a new position embedding called RPE-Absolute to encode \"the absolute value of the relative distances to some special tokens\". The experiment result (Figure 6 in the link) shows that RPE-Absolute achieves length generalization in AdditionMod10. This suggests that our position embedding design approach is adaptable to different domain knowledge.\n\n3. *Regarding OOD and LDHD. I believe the paper's analysis do not explain why we observe (surprising) LDHD generalization for several other problems without any special domain knowledge...*\n\n(a) Domain knowledge can be implicitly encoded via architecture design, training algorithms, data distributions, preprocessing strategies, etc. Even if domain knowledge is not injected explicitly, these prior choices reflect inductive biases tailored to the task.\n\n(b) \"...despite <10% representation of non-english data in pretraining\"\n\nIt is important to distinguish low representation from zero representation. Prior work [1] shows that adding even a very small amount of long sequences to the training set can dramatically improve length generalization, which is called priming. From our perspective, this is because the presence of long sequences, even in small quantities, prevents the model from learning an overly simple interpolator that fails to extrapolate. For example, in digit-wise addition, if the training set only includes 3-digit numbers, the model might learn to ignore digits beyond the third. However, including a few 5-digit samples forces the model to also consider later digits, improving its extrapolation ability.\n\n(c) While this work is mainly focused on length generalization, it is an interesting future direction to study how the compositional generalization can be understood from the perspective of LDHD generalization.\n\n[1] Jelassi, S. et al., 2023. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.15400.\n\n4. *The writing of the paper needs much improvement...*\n\nWe will carefully proofread our submission in the revision.", "claims_and_evidence": "The paper's analysis (see (b) from my paper's contribution list above) makes the following choices: (1) isolate and analyse only the positional embedding contribution to the attention score, (b) assume the value head is an identity, (c) restricts only to binary functions.\nHowever, the paper did not justify how their choices do not compromise the generality of their analysis. \n\nTheir proposed positional embedding scheme: RPE-Square relies on domain knowledge of the task in hand. It is expected that the paper at least present how their approach extends to other tasks, and what kind of domain knowledge can be exploited for different problems. \n\nRegarding OOD and LDHD. I believe the paper's analysis do not explain why we observe (surprising) LDHD generalization for several other problems without any special domain knowledge. For example, consider the multilingual abilities of LLMs despite <10% representation of non-english data in pretraining. Given the typically low language representation, it is possible that many (entity, language) combinations are not present in the training data, yet the models demonstrated generalization to such unseen combinations (see [1] for an example). We may see this as LDHD generalization where (entity, language) defines a dimension. \n\n[1] Chen, Yang, Yitao Liang, and Zhouchen Lin. \"Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalization.\" arXiv preprint arXiv:2410.08898 (2024).", "methods_and_evaluation_criteria": "Yes, the paper pertains LDHD generalization and length generalization is a good task. \nThe only evaluation the paper conducted is with addition and copy tasks. The choice of tasks make sense.", "theoretical_claims": "I did not check the correctness of any theorem, and only glanced at the statements.", "experimental_designs_or_analyses": "Yes, their only experiment looks sound.", "relation_to_broader_scientific_literature": "The findings need not surprise a significant class of researchers working on causality or symbolic methods. But the folks who bet on learning purely from signal, large scale training and exploration would point the paper to many phenomena that the paper do not explain like the way I did in the last point of the Claims section.", "essential_references_not_discussed": "NA, please see my response to the next question.", "other_strengths_and_weaknesses": "**Strengths**  \nLDHD generalization is a compelling problem, and studying the SoTA models for LDHD is even more.    \nThe analysis, although on a special class of functions and assumptions, present some interesting insights into generalization. \n\n**Weakness**   \nThe writing of the paper needs much improvement. They should aim to reduce the number of notations introduced, explain in text every equation and symbol before presenting the math expression. Present proof sketches in the main paper. Make a more coherent and less dense read by taking care of the reasoning-related sections and Free-lunch thm section.", "other_comments_or_suggestions": "L92, second column, formant-> format.\nL221, first column. I believe it is $$n=\\arg\\min_k \\\\{h\\in \\Sigma^k\\\\\\}$$", "questions_for_authors": "Please address my questions listed under \"Claims and Evidence\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l2XqbbqL3U", "reviewer": "Reviewer_H9fc", "summary": "This paper introduces the concept of Low-Dimension-to-High-Dimension (LDHD) Generalization to address challenges in out-of-distribution (OOD) generalization, particularly in reasoning tasks where models are trained on low-dimensional subspaces and tested on higher-dimensional spaces. The authors propose that LDHD generalization is fundamental to understanding length generalization, where models extend learned patterns to longer sequences.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable feedback. We'd like to response to your concerns as below.\nAll experiments results are in the anonymous link: https://www.dropbox.com/scl/fi/52t23nfzev1lo1sq5dmyj/ICML_2025_5857_Rebuttal.pdf?rlkey=5nde6aampze744klvn0rsb3mp&st=37lg3nzo&dl=0.\n\n1. *...comparisons with alternative position embeddings designed for length generalization...*\n\nWe conduct additional experiments to compare the mentioned position embeddings (Since Abacus and position coupling are similar apart from some subtle implementation details in our tasks, we only implement Abacus currently.) The results (Figures 1 - 2 in the link) show that RPE-Square achieves the best overall length generalization in the tasks.\n\n2. *Lack of Generalization Tests on Longer Sequences*\n\nWe evaluate the models with RPE-Square on significantly longer sequences in three tasks (Parity (with CoT), Multiplication (1 * N), and Division (N * 1)) The results (Figure 7 in the link) show that the models can extrapolate to significantly longer sequences to some extend and the performance may decreases as the sequence length increases.\n\n3. *Absence of Robustness Analysis: The performance of RPE-Square across various tasks beyond addition and unaligned copy is not explored.\n\nWe apply RPE-Square to three new tasks: Parity (With CoT), Mulplication (1 * N), and Division (N / 1). The results (Figures 3 - 5 in the link) show that Transformers equipped with RPE-Square can achieve length generalization in all the three tasks, showing the robustness of RPE-Square across different tasks.\n\n4. *Experiments comparing with and without CoT seems to be missing.*\n\nWe compare the length generalization of Parity with CoT and without CoT. We train Transformers with RPE-Square. We only achieve length generalization for Parity with CoT (Figure 8 in the link). This is because as CoT enables LDHD generalization via relative distances in the latent space. There are also many empirical results showing CoT can enhance length generalization in previous works, e.g., [1] [2].\n\n[1] Zhou, H., et al., 2023. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028.\n\n[2] Feng, G., et al., 2023. Towards revealing the mystery behind chain of thought: a theoretical perspective. NeurIPS.\n\n5. *I find it hard to make connections between sections and get a clear intuition behind the approach.*\n\n(a) Connections between sections.\n\n**Section 1** introduces the problem of length generalization and proposes LDHD generalization to address the mismatch between input length and problem scale. **Based on the LDHD formulation**,  **Section 3** presents the No-Free-Lunch Theorem for LDHD generalization, motivating the need for inductive bias. **Section 4** consider how different inductive bias can be incorporated by choosing different models, a technical problem to address the LDHD generalization challenge in Section 3. In **Section 5**, we further discuss the implications of the LDHD generalization formulation for practical length generalization. We explain why CoT can promote length generalization. We also propose the position embeddding design principle for length generalization: consider the inherent LDHD generalization and the data format nuisance separately.\n\n(b) Intuition behind the approach.\n\n- The intuition behind LDHD generalization formulation includes: (1) The sample space grows exponentially as the scale increases and the small-scale instances do not tell how large-scale instances can be solved without external information; (2) In language modeling, the sequence length can be affected by the data format and may not faithfully reflect the problem scale.\n\n- The intuition behind RPE-Square: the unaligned formats can be handled by the relative distances to some special tokens. Detailed illustrations are in Appendix D.1.\n\n6. *the definition of \"concept c\" in Abstraction 1 seems missing.*\n\nA \"concept\" refers to a target mapping, as in [1]. We'll clarify this in the revision.\n\n[1] Mohri, Mehryar, 2018. Foundations of machine learning, Chapter 2, The PAC Learning Framework, Section 2.1, The PAC learning model.\n\n5. *Generalization of theoretical results*\n\nWe provide empirical support (Figure 9) by training GPT2 with APE and RPE on different functions. Probing reveals the learned models roughly align with minimum-degree interpolators w.r.t. their respective bases.\n\n6. *... the computational and memory overhead ...*\n\nSuppose the sequence length is N and the hidden dimension is d.\nRPE-Square: $O(N^4 + N^3 d)$ compute and $O(N^2 + N d)$ memory.\n\nStandard PE: $O(N^2 d)$ compute and $O(N^2 + N d)$ memory.\n\nWhile RPE-Square is less efficient than the standard position embedding, it mainly serves as a proof-of-concept for separating LDHD and format handling. It is focused on the length generalization challenge instead of the challenge of long sequence. Designing efficient variants is a promising direction for future work.", "claims_and_evidence": "The paper presents several key claims:\n1. LDHD Generalization Necessitates Inductive Biases: The authors assert that without appropriate inductive biases, models cannot generalize from low-dimensional training data to high-dimensional testing scenarios. This claim is supported by the No-Free-Lunch Theorem for LDHD generalization, which mathematically establishes the necessity of inductive biases. \n\n2. CoT Enhances Length Generalization: The paper suggests that Chain-of-Thought prompting improves length generalization by restructuring the latent space, facilitating better LDHD generalization. While the theoretical analysis supports this claim, direct empirical comparisons of models trained with and without CoT on reasoning tasks are lacking.\n\n3. RPE-Square Improves Length Generalization: The authors propose that RPE-Square, their novel position embedding method, addresses both LDHD generalization and data format inconsistencies, leading to improved length generalization. Experimental results on addition and unaligned copy tasks demonstrate its effectiveness. However, comparisons with alternative position embeddings designed for length generalization, such as ALiBi, RoPE, NoPE, Abacus, as well as position coupling are not included.\n\nOverall, the claims are theoretically substantiated, but additional empirical validation on diverse tasks would strengthen the conclusions.", "methods_and_evaluation_criteria": "Theoretical analysis is sound and clearly presented. The evaluation tasks (Unaligned Copy, URF Addition) are appropriate for illustrating their claims. However, broader validation on more varied or real-world data could strengthen applicability claims.", "theoretical_claims": "Yes, I checked Theorem 1 (No Free Lunch Theorem) and the proof in the supplementary material seems correct.", "experimental_designs_or_analyses": "The experiments are well-structured but exhibit certain limitations:\n\n- Lack of Generalization Tests on Longer Sequences: It remains unclear whether models trained with RPE-Square can extrapolate to significantly longer sequences than those encountered during training.\n- Absence of Robustness Analysis: The performance of RPE-Square across various tasks beyond addition and unaligned copy is not explored.\n- Experiments comparing with and without CoT seems to be missing.", "relation_to_broader_scientific_literature": "The paper appropriately references key works on:\n- Length Generalization in Transformers: Citing studies by Anil et al. (2022) and Jelassi et al. (2023).\n- Inductive Bias in Neural Networks: Expands on works by Gunasekar et al. (2017), Abbe et al. (2023), providing a clear link to length generalization.", "essential_references_not_discussed": "I am not aware of essential references not discussed.", "other_strengths_and_weaknesses": "### Strengths:\n- Novel Formalization: The introduction of LDHD generalization offers a valuable perspective on length generalization challenges.\n- Practical Contribution: The development of RPE-Square presents a tangible improvement for length generalization tasks.\n\n### Weakness:\n- I find it hard to make connections between sections and get a clear intuition behind the approach. \n- Limited diversity of tasks considered: The paper only considers two simple tasks of unaligned copy and URF addition of small scale. \n- Lacks comparison with diverse Positional Embeddings, developed to tackle length generalization, leaving the relative effectiveness of RPE-Square unclear\n- Computational overhead of RPE-Square not discussed.", "other_comments_or_suggestions": "- typo: line 168 overparatermization -> overparameterization\n- typo: line 202 equation X_m instead of X^m in the subscript of E\n- discuss briefly computational complexity of RPE-Square in practice.\n- the definition of \"concept c\" in Abstraction 1 seems missing.", "questions_for_authors": "- Generalization of theoretical results: Do you have empirical evidence confirming that actual Transformers trained in realistic scenarios indeed favor minimum-degree interpolants?\n- Computational Complexity: What is the computational and memory overhead of RPE-Square compared to standard positional encoding, especially for longer sequences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNrD2SPJeQ", "reviewer": "Reviewer_osYw", "summary": "Summary: The paper examines Low-Dimension-to-High-Dimension (LDHD) generalization and theoretically demonstrated that LDHD\ngeneralization is unattainable without appropriate inductive bias, focusing on Boolean functions and how different architectures and inductive biases influence this generalization. The study introduces RPE-Square to handle data format nuisances and improve LDHD generalization. Key formulas include the computation of latent variables h and labels y through mappings phi, and the modeling of low-dimension-to-high-dimension generalization with the Boolean function framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your careful evaluation and positive assessment of our work. All experiments results are in the anonymous link: https://www.dropbox.com/scl/fi/52t23nfzev1lo1sq5dmyj/ICML_2025_5857_Rebuttal.pdf?rlkey=5nde6aampze744klvn0rsb3mp&st=37lg3nzo&dl=0.\n\n1. Has RPE-Square been evaluated in diverse or challenging environments? What were the challenges? What were the outcomes?\n\nWe evaluate RPE-Square in three additional tasks: Parity (with CoT), Multiplication (1 * N), and Division (N / 1). RPE-Square achieves length generalization in all the three tasks (Figures 3 - 5 in the link). These results are consistent with our insight of the inductive bias of RPE-Square and further justify the proposed position embedding design principle.\n\n2. Could you provide an insight on the computational overhead of RPE-Square, especially in large-scale or real-time applications?\n\nSuppose the sequence length is N and the hidden dimension is d. The computational overhead of an attention with RPE-Square is $O(N^4 + N^3 d)=O(N^2 (N^2 + N^d))$ (with the implementation that saves the token attention scores for all query j and key i and reuse them when computing $\\text{RPE-Square}_{i,j}$). This overhead can lead to inefficiency for large-scale or real-time applications. \n\nWe note that RPE-Square is primarily intended to illustrate the design principle for position embeddings. Our focus is on the challenge of length generalization instead of long-sequence efficiency. We consider it a promising direction to develop more efficient variants of RPE-Square in future work.\n\n3. Are there specific conditions or assumptions in Theorem 3 that might limit its general applicability? If so, what are they?\n\nTheorem 3 is derived with a simplified model (PLAA-APE). The conclusion in Theorem 3 may not hold precisely for more complex models. Also, the target functions considered in Theorem 3 are restricted to Boolean functions and the notation \"min-degree profiler w.r.t. linearly independent set\" may not be applied to non-Boolean functions directly. Additionally, the result include a limit ($\\hat{P}$) but we cannot use the exact limit in practice.\n\nDespite the above theoretical limitations, we think Theorem 3 can provide sufficient insight on the inductive bias of APE. The simplified model captures the essential effect of APE on the inductive bias. The analysis in the Boolean functions can be naturally extended to functions defined on finite domains. The limit can be well approximately by choosing a sufficient small initialization.\n\n\nWe also appreciate your suggestions on writing to improve clarity and readability. We will revise our manuscript accordingly.", "claims_and_evidence": "While the paper presents strong theoretical underpinnings and some experimental results, there are areas where the evidence might not be as convincing or where further clarification and experimentation could strengthen the claims:\n\n1. Generalization of Theoretical Concepts to Practical Applications: The paper does an excellent job in theoretical exposition, especially with the introduction of the No-Free-Lunch Theorem for LDHD generalization. However, the transition from theoretical models to practical, real-world applications is not deeply explored. The claims regarding the practical effectiveness of RPE-Square would be more convincing if supplemented by more diverse and extensive empirical evidence across various domains and tasks.\n\n2. Robustness of RPE-Square: The paper claims that RPE-Square enhances the model's ability to handle data format nuisances effectively. While initial experimental results are provided, the robustness of RPE-Square across a broader range of scenarios and its comparison to other state-of-the-art position embeddings are not thoroughly examined. Claims about its superiority could be better supported by more comprehensive comparative studies that include a variety of datasets and model configurations.\n\n3. Scalability and Performance Across Different Scales: The paper discusses the potential of RPE-Square in managing the challenges associated with LDHD generalization. However, the scalability of this approach—how it performs as the dimensionality of data dramatically increases—is not fully addressed.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are appropriate for addressing the challenges LDHD.\n\nProposed Methods:\n- RPE-Square: Enhances traditional Relative Position Embedding by better handling data format nuisances and the shift from low to high-dimensional spaces. This method theoretically supports improved out-of-distribution generalization in deep learning models.\n- Chain-of-Thought (CoT): Applies CoT to enhance the understanding of sequences in transformer models, suitable for complex reasoning tasks where sequence understanding is crucial.\n\nEvaluation Criteria:\n- Task-Specific Benchmarks: Uses arithmetic reasoning and symbolic logic tasks to evaluate performance, which are appropriate for testing model capabilities in handling length generalization and increased problem complexity.\n- Comparison with Baselines: Measures improvements over traditional embeddings and other state-of-the-art methods, providing a clear benchmark for assessing the efficacy of RPE-Square.\n\nAreas for Improvement:\n1. Wider Range of Datasets: Including datasets with inherently high-dimensional structures could better test the robustness of the proposed methods.\n2. More Diverse Tasks: Expanding evaluations to include varied tasks like image processing could demonstrate the versatility of RPE-Square.\n3. Quantitative Metrics: More rigorous metrics that measure performance in high-dimensional settings and assess computational efficiency would provide a comprehensive evaluation.", "theoretical_claims": "The mathematically oriented proofs, such as those for the No-Free-Lunch Theorem, are rigorous and seem to be correct according to the details provided. However, there's a chance that I might have missed an error. Additionally, the proofs do not thoroughly address the computational complexity of implementing RPE-Square, which is vital for practical applications, particularly in very high-dimensional spaces.", "experimental_designs_or_analyses": "In reviewing the paper, I examined the soundness and validity of the experimental designs and analyses, particularly those related to the implementation and evaluation of RPE-Square and its impact on LDHD generalization.\n\n- Data Variability: The experiments would benefit from a more detailed examination of the test data's variability and distribution, particularly in how they mirror real-world scenarios essential for LDHD generalization.\n- Computational Load: The paper does not adequately address the computational demands of implementing RPE-Square, crucial for practical applications, especially in resource-constrained environments.\n\nOverall, while the experimental designs and analytical methods employed in the paper are sound for the most part, enhancing the range of benchmarks, incorporating a broader set of evaluation metrics, and providing more details on computational demands and data scales could significantly strengthen the validity of the findings.", "relation_to_broader_scientific_literature": "The key contributions of the paper, particularly the development of RPE-Square and its application in LDHD generalization, build on existing scientific literature by enhancing position embedding techniques previously established for transformer models. This development provides a nuanced approach to handling high-dimensional data challenges, extending the utility of relative position embeddings for complex reasoning tasks. The paper's theoretical additions, like the No-Free-Lunch Theorem for LDHD generalization, contribute a rigorous framework to the discussion of dimensionality in machine learning, bridging theoretical concepts with practical machine learning applications. Overall, these contributions effectively integrate and advance the current understanding of embedding designs and generalization in machine learning.", "essential_references_not_discussed": "Not essential but might beneficial:\n1. Bellman, R. (1961). Adaptive Control Processes: A Guided Tour. Princeton University Press.\nThis reference is foundational in the discussion of the curse of dimensionality, providing the initial formal description of the problem which is crucial for understanding the theoretical underpinnings of dimensionality challenges.\n\n2. Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2319-2323.\nThis paper introduces Isomap, a manifold learning technique that provides insights into how high-dimensional data can be effectively reduced, maintaining the intrinsic geometry of the data, which is highly relevant for LDHD generalization.\n\n3. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning (pp. 1096-1103).\nAutoencoders, particularly denoising autoencoders, are useful for dimensionality reduction and feature extraction in noisy datasets, relevant for LDHD generalization where maintaining data integrity across dimensions is critical.", "other_strengths_and_weaknesses": "Strengths:\nIt becomes increasingly interesting if you can get through the first two pages!\n1. Novel Conceptual Framework: The paper effectively frames LDHD generalization within the broader context of Out-of-Distribution (OOD) generalization, offering a novel perspective on handling high-dimensional data spaces derived from low-dimensional training data.\n2. Nice Theoretical Support: The theoretical underpinnings, including the No-Free-Lunch Theorem for LDHD generalization, are robust, providing a strong foundation for the arguments regarding the necessity of inductive biases.\n3. Practical Implications: The introduction of RPE-Square as a novel embedding to address both LDHD generalization and data format nuisances is both innovative and practically relevant, offering direct applicability to transformer models.\n4. Empirical Validation: The use of Chain-of-Thought (CoT) to demonstrate the practical effectiveness of the proposed methods in restructuring latent space for improved generalization is well-executed, with clear experimental setups and results.\n\nWeaknesses: \nAs noted earlier, the main areas for improvement include:\n1. Enhanced Mathematical Clarity: While foundational proofs like those for the No-Free-Lunch Theorem are well-established, additional mathematical detail could strengthen the theoretical support for RPE-Square.\n2. More Empirical Evidence: Broader empirical testing would help validate the practical effectiveness of RPE-Square across various applications.\n3. Computational Complexity: Further discussion on the computational demands of implementing RPE-Square is needed to assess its practicality in high-dimensional settings.", "other_comments_or_suggestions": "Commnets: \n- I recommend including \"Relative Position Embedding (RPE)\" in the abstract for clearer context.\n- Figure 1c was unclear until I read lines 117-123 on page 3. Adding a brief explanation beneath the figure could assist readers who seek immediate clarity.\n- The first two pages would benefit from a more detailed discussion of goals and applications.", "questions_for_authors": "Questions: \n1. Has RPE-Square been evaluated in diverse or challenging environments? What were the challenges? What were the outcomes?\n2. Could you provide an insight on the computational overhead of RPE-Square, especially in large-scale or real-time applications?\n3. Are there specific conditions or assumptions in Theorem 3 that might limit its general applicability? If so, what are they?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wlTmlmClU5", "reviewer": "Reviewer_9ohk", "summary": "This paper conducts a theoretical analysis of the problem of low-dimension-to-high-dimension generalization, with an application to the problem of length generalization. The main theoretical results are based on Boolean function analysis and extend the analysis in Abbe et al. (2023) by considering different functional bases in the Boolean function space and analyzing more neural network models, including random feature models with projection and Position-Only Linear Attention with Advice (PLAA) models with Absolute Position Embedding (APE) and Generalized Absolute Position Embedding (GAPE). The paper shows that gradient descent on these models results in min-degree solutions with regard to different functional bases. On the algorithm side, the paper proposes RPE-Square, an ad-hoc improvement of Relative Position Embedding (RPE), and demonstrates its effectiveness on length generalization problems including unaligned copy and URF addition.\n\n**Post-rebuttal update:** The authors have addressed some of my concerns. At this point, my main criticisms of the paper are (1) the limited implications of the formulation and theoretical analysis (compared with the prior work by Abbe et al. (2023)), and (2) the overall presentation (see my rebuttal comment for details). The authors have responded to both points in the rebuttal, but not to a very satisfactory level. I decided to keep my original score but would let AC and other reviewers decide whether this paper should be considered to reach the acceptance bar of ICML.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your effort and insightful comments. We respond to your main concerns below.\nAll experiments results are in the anonymous link: https://www.dropbox.com/scl/fi/52t23nfzev1lo1sq5dmyj/ICML_2025_5857_Rebuttal.pdf?rlkey=5nde6aampze744klvn0rsb3mp&st=37lg3nzo&dl=0.\n\n1. *The theoretical contributions of the paper are rather incremental.*\n\nOur main contribution is the LDHD generalization formulation (Abstraction 1), which offers a more precise characterization of length generalization by separating it into two challenges: (1) LDHD generalization in the latent space (which is inherent), and (2) data format nuisance. Under the formulation, we can derive the fundamental limitation of length generalization, explain the effectiveness of certain methods, and design novel position embeddings for challenging length generalization problem.\n\nThe theoretical results in Section 4 demonstrates how different model designs can introduce different inductive bias to handle LDHD generalization, only related to the LDHD generalization aspect in Abstraction 1. \n\nWhile our Boolean analysis in Section 4 is technically inspired by the remarkable work of Abbe et al. (2023), our core novelty lies in Abstraction 1 and its implications, not in extending prior Boolean analysis.\n\n2. *The proposed method RPE-Square is interesting but is only evaluated in limited settings.*\n\nWe evaluate RPE-Square in three additional tasks: Parity (with CoT), Multiplication (1 * N), and Division (N / 1). RPE-Square achieves length generalization in all the three tasks (Figures 3 - 5 in the link).\n\n3. *The theory and algorithm seem disconnected.*\n\nAll components of our paper are built around Abstraction 1. Here’s how they connect:\n**Section 3** focuses on the LDHD challenge and shows that length generalization requires prior knowledge. **Section 4** illustrates how inductive bias can be introduced through model design to address the LDHD challenge. **Section 5** consider practical length generalization techniques from the perspective of Abstraction 1. **Section 5.1** explains the effectiveness of CoT as extending the latent space, which may promote LDHD generalization. **Section 5.2** derive the position embedding design principle according to Abstraction 1, highlighting the importance of handling LDHD generalization in the latent space and the data format separately.\n\nRPE-Square is designed closely following the principle implied by Abstraction 1. The inner \"relative distance to some special token\" is to handle the unaligned format. The outer \"relative distance of relative distance to some special token\" is to handle the LDHD generalization in the latent space, which is partially explained by the analysis of PLAA-GRPE. Further details are in Appendix D.1.\n\n4. *I did not understand Sec. 5.1: how did you determine the latent space of each instance? Why should the latent space take the form in the paper?*\n\nThe latent space is determined according to Abstraction 1. We consider the scale $n$ of each instance and a proper $\\Sigma$ such that each instance of scale $n$ corresponds to an element in $\\Sigma^n$.\n\nWith CoT, we treat each step (which requires a model prediction) as an instance of the same scale.\nFor example, in the addtion with CoT, the two predition steps, i.e., given $x_0 \\dots x_{n-1} + y_0 \\dots y_{n-1} =$ to predict $z_0$, and given $x_0 \\dots x_{n-1} + y_0 \\dots y_{n-1} = z_0$ to predict $z_1$, are treated as separate CoT-step instances of the same scale $n$.\n\nWe then decide a proper domain $\\bar{\\Sigma}$ such that each CoT-step instance of scale corresponds to an element in $\\bar{\\Sigma}^n$. Furthermore, since CoT typically inserts intermidate results to the original instances, $\\bar{\\Sigma}$ will take the form $\\Sigma\\times\\Sigma'$. For the addition, $\\Sigma'=\\lbrace *,0,\\dots,9\\rbrace$ satisfies the requirement.\n\n\n5. *...the formulation of LDHD generalization (Def. 1) is quite similar to length generalization... Are there any other applications of this formulation except length generalization?*\n\nChanging \"length\" to \"dimension\" is not merely a notion replacement. By the notion \"dimension\", we highlight the **exponential growth** in sample space as scale increases, and the **complete absence of information** in low-dimension training data about high-dimension behavior. These aspects are not captured by \"length\", which can be confounded by formatting. Thus, “dimension” more precisely reflects the scaling challenge.\n\nWhile LDHD generalization is originally to characterize the inherent scaling challenge in length generalization, it could be extended to other applications to character the exponential growth in the sample space size and imperfect information provided by the training data. For example, in graph learning, we may consider the number of nodes as the dimension and use this formulation to describe the generalization from small to large graphs.", "claims_and_evidence": "Most of the claims made in the paper are supported by theoretical or empirical evidence. However, I did not understand the claims on Chain-of-Thought (CoT) prompting in Sec. 5.1. See \"Questions for authors\" for more details.", "methods_and_evaluation_criteria": "The proposed RPE-Square is evaluated on two representative length generalization tasks including unaligned copy and URF addition. I am not an expert in benchmarking length generalization, but given the scope and scale of the considered tasks, they should mainly serve as proof-of-concept demonstrations at this stage. Since the main contents of the paper are of theoretical nature, I think the current evaluation is acceptable.", "theoretical_claims": "I reviewed some proofs in the appendix and did not find major issues.", "experimental_designs_or_analyses": "Experimental designs are OK, though it would be better if the authors could also compare RPE-square with more position embedding methods (e.g., RoPE) in the experiments.", "relation_to_broader_scientific_literature": "Prior work by Abbe et al. (2023) has shown that several types of NNs converge to min-degree interpolators when trained by gradient descent; this work extends this result by showing that other types of NNs, including simplified attention models with position embeddings, may converge to min-degree interpolations under different bases in the function space.\n\nThe proposed RPE-Square is an ad-hoc improvement to RPE in mathematical reasoning problems.", "essential_references_not_discussed": "To my knowledge, most of the related works are properly cited and discussed.", "other_strengths_and_weaknesses": "*Strengths:* Theoretical results are clearly presented.\n\n*Weaknesses:*\n- The paper is not very easy to read, especially for those who are not familiar with the prior work by Abbe et al. (2023).\n- Many definitions and results heavily build on those in Abbe et al. (2023), e.g., degree profile, min-degree interpolators, Theorem 2, and Corollary 1. Overall I think the theoretical contributions of the paper are somewhat incremental.\n- Many parts of the paper, in my opinion, are kind of disconnected: \n  - I did not get how Abstraction 1 is used in the theoretical results/proofs.\n  - The theory and the algorithm seem not connected; in fact, I think that Theorems 3 and 4 are not relevant to the proposed RPE-Square (which seems to have nothing to do with functional bases induced by the position embedding), nor are they relevant to Example 1 (regarding the mismatch between the problem scale and the input length) which serves as the main motivation of the paper.\n- I think that the main theoretical contributions of the paper, i.e., introducing min-degree interpolators under different bases, can only address very limited length generalization problems: for example, the target function in Example 3 only has degree 1. From my angle, introducing additional projections/linear operations cannot solve length generalization problems with higher-degree target functions, e.g., the one considered in Theorem 5.1 of Abbe et al. (2023).", "other_comments_or_suggestions": "In general, at this point my overall evaluations of this paper are as follows:\n\n- The theoretical contributions of the paper are rather incremental;\n- The proposed method RPE-Square is interesting but is only evaluated in limited settings;\n- The theory and algorithm seem disconnected.\n\nI am now between weak accept and weak reject and decided to choose weak accept. Yet, I may also lower my rating if my main concerns are not well addressed in the rebuttal.", "questions_for_authors": "- I did not understand Sec. 5.1: how did you determine the latent space of each instance? Why should the latent space take the form in the paper?\n\n- It appears to me that the formulation of LDHD generalization (Def. 1) is quite similar to length generalization (the main difference is that LDHD considers different dimensions in train/test data subspaces, while length generalization replaces the notion of \"dimension\" to \"length\"). Are there any other applications of this formulation except length generalization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper studies Low-Dimension-to-High-Dimension (LDHD) generalization for Boolean functions using different architectures, under inductive bias. Based on the insights, the authors propose a position encoding method, called RPE-Square to facilitate LDHD generalization. They perform experiments on LLMs with transformer architectures and show promising results on simple Math tasks.

The reviewers have raised two main concerns -- lack of readability and limited experimental evaluation. The authors have addressed both these issues by providing explanations in the rebuttal (some reviewers have acknowledged that they now have a better understanding of the results) and providing additional experimental results. If these improvements are included in the final version of the paper, the paper makes solid contributions.
Decision: Accept (poster)