A Length-Extrapolatable Transformer
Year: 2023

Abstract:
Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better efficiency. The proposed architecture is named LengthExtrapolatable (LEX) Transformer. We evaluate different Transformer variants on language modeling. Experimental results show that our model achieves better performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/ LeX-Transformer.

heading: 1 Introduction
text: Transformer (Vaswani et al., 2017) has shown strong performance in NLP and become a de-facto backbone (Dosovitskiy et al., 2020; Radford et al., 2021; Wang et al., 2022). However, most of them have a crucial shortcoming: they can only deal with the in-distribution size of inputs. Figure 1 shows that the perplexity of previous Transformers increases rapidly when the input sequence is getting longer. It is usually infeasible to train a model with all possible input lengths. Therefore, a length-extrapolatable Transformer is essential for wider usage.
In sequence modeling, position information plays a crucial role in building the correct representation and understanding of the latent meaning. For Recurrent Neural Networks such as LSTM (Hochreiter and Schmidhuber, 1997), the calculation is done along the sequence order in O(N) time. However, the parallel attention module
∗Work done during internship at Microsoft Research.
makes it hard to encode position effectively. First, Vaswani et al. (2017) proposes absolute sinusoidal position embedding, and Devlin et al. (2019) adjusts it to a learnable one. The absolute design is computation-efficient, but not comparable with subsequent relative ones (Shaw et al., 2018; Su et al., 2021; Press et al., 2021). Among many relative position embeddings, ROPE (Su et al., 2021) shows better performance and is used to many PLMs such as PaLM (Chowdhery et al., 2022). However, it can’t deal with sequences with exceeding length. Alibi (Press et al., 2021) mitigates the extrapolation problem but sacrifices the general performance.
Since different strategies concentrate on some part of the position feature, it is essential to build a comprehensive view and guide the Transformer’s design systematically. First, a Transformer should be sensitive to order. Otherwise, it will degenerate into a bag-of-word model which confuses the whole meaning. Then, position translation can’t hurt the representation, especially, when a prefix is added to the target sentence, the representation should stay the same with an attention mask on the prefix. After that, a good sequence model needs to
14590
deal with any input length. As illustrated before, the length problem is not universal but special for Transformer. Especially, when a Transformer is pre-trained under a maximal length, it is not affordable to re-train for applying to tasks with longer sequences. Finally, when a Transformer satisfies the principles above, we evaluate its performance, which requires thorough experiments and empirical analysis.
Considering all the properties above, we propose Extrapolatable Position Embedding (XPOS), which is a universal-good design for Transformers. Based on ROPE’s design, we propose attention resolution as a metric to measure position monotonicity. Then, we generalize its mathematical form, where an exponential decay is added to the rotation matrix. XPOS preserves the advantage of ROPE, and behaves stably at long-term dependency. Besides, inspired by sparse attention methods (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020; Xiong et al., 2021), we choose blockwise causal attention to increase attention resolution, which improves the performance of length extrapolation for language modeling.
We train different Transformers from scratch. We evaluate models on PG22 and QMSum (Zhong et al., 2021) with various input lengths. On the interpolation experiments, LEX Transformer reaches minimal perplexity. In the extrapolation experiments, our methods can continue decreasing the perplexity while other methods either can’t extrapolate (i.e., perplexity increases) when the input length is very long. Figure 1 shows clearly that LEX Transformer has an opposite tendency compared with others.
We summarize our contributions as follows:
• We summarize the design principles of Transformers for position modeling.
• We define attention resolution to indicate a Transformer’s capability on encoding position.
• We propose an extrapolatable position embedding and use blockwise causal attention to improve length extrapolation.
• We conduct experiments on language modeling and show that the proposed LEX Transformer achieves strong performance on both short and long texts.

heading: 2 Design Principles of Transformers for Position Modeling

heading: 2.1 Order Variance
text: A transformer without position information is actually a bag-of-word model. Although bag-of-words models can achieve comparable performance for some tasks (Wang et al., 2020a), position information is essential for sequence modeling. Most of the existing position modeling satisfies this goal (Vaswani et al., 2017; Devlin et al., 2019; Shaw et al., 2018; Wang et al., 2020a; Raffel et al., 2020; Su et al., 2021). With effective position information, Transformer models should be variant with permuting the order (Dufter et al., 2022). Give a permutation function Pπ(X) : [x1, x2, ..., xn] → [xπ1 , xπ2 , ..., xπn ], where [π1, π2, ..., πn] is a random order, a Transformer model f(input) should satisfy:
f(Pπ(X)) ̸= Pπ(f(X)) (1)

heading: 2.2 Translation Invariance
text: The representation of a sequence should be robust with the translation of its positions. For instance, a sentence’s meaning is invariant with padding before or after the whole sentence. Similar
to (Wang et al., 2020a), we give a general form for translation invariance: given a Transformer model f(input,mask), any input sequence X = [x0, x1, ..., xn] with mask M = [m0,m1, ...,mn], the output should be same with the padding ones:
Xpad = [0]i ⊕X ⊕ [0]j Mpad = [0]i ⊕M ⊕ [0]j
f(X,M) = f(Xpad,Mpad)[i : i+ n]
(2)
Relative positions (Shaw et al., 2018; Raffel et al., 2020; Wang et al., 2020a; Su et al., 2021) satisfy this condition, while most of the absolute positions do not (Vaswani et al., 2017; Devlin et al., 2019). Although sinusoidal embedding has a similar property (Vaswani et al., 2017): PEpos+k can be represented as a linear function of PEpos, the addition operation in the initial word embedding messes the attention weight, where the spread form of QKT has 4 components whose geometric connection with position is unclear.

heading: 2.3 Length Extrapolation
text: As the cost of pre-training is getting bigger due to the larger model size and corpus, it is infeasible to retrain a model for a longer context. A Transformer model with a suitable design should be capable of dealing with any input length.
First, learnable absolute position embedding (Devlin et al., 2019) is not able to extrapolate because it does not have any pre-defined position knowledge. With the evaluation of perplexity on different lengths (Press et al., 2021), almost every position embedding’s performance drops significantly (Vaswani et al., 2017; Raffel et al., 2020; Su et al., 2021). Alibi (Press et al., 2021) solves this problem by adding an exponential decay on the attention matrix, which lower the influence of outof-distribution position like a soft sliding window. However, the absence of long-term dependency contributes to a performance drop compared with other relative strategies. Table 2 shows that Alibi’s perplexity is larger than ROPE by about 0.3.
However, the extrapolation ability needs a systematic design where position embedding is a crucial but not only component. With the proper attention mask, the relative position can deal with long text. The ideal situation is to use the long context in the right way, in that case, the perplexity should decrease as the input length increases.

heading: 3 A Length-Extrapolatable Transformer
text: We define attention resolution as the indicator of the Transformer’s capability on encoding position in Section 3.1. Then we propose two ways to maximize the resolution metric, i.e., improve the length interpolation and extrapolation of Transformers. First, we introduce a relative position encoding method (Section 3.2) to explicitly maximize attention resolution. Second, we propose to use blockwise causal masking (Section 3.3) during extrapolation inference for improved resolution.
In the following section, we denote d as the hidden dimension and l as the input length. For each attention layer, query, key, and value are calculated by input x: q = WQx, k = WKx, v = WV x.

heading: 3.1 Attention Resolution
text: The monotonicity of attention scores is essential to represent distance in language models. In an attention layer of the vanilla Transformer, we measure the attention score expectation as s[n] when the distance of two tokens is n:
s[n] = E 0≤i≤N
( qi+nk T i√
d ) (3)
We define attention resolution R(s) as a metric to evaluate attention’s ability to recognize position:
R(s) = N∑
i=0
es[i](es[i] − es[i+1]) ( ∑N
i=0 e s[i])2
(4)
First, s[i] > s[i+1] is preferred to ensure monotonicity. Besides, we implement softmax operation on s[n] to simulate the attention probability. To mitigate the influence of long-tail distribution, the factor es[i] is multiplied. We can estimate s[n] and R(s) quantitatively when we design Transformers.

heading: 3.2 Improve Resolution by Position Encoding
text: Su et al. (2021) propose that by adding absolute position embedding on query and key, the attention matrix is actually encoded with relative position information. ROPE shows a strong performance in interpolation tasks, but its s[n] oscillates dramatically in Figure 2, which harms the resolution.
We use a similar but generalized strategy to improve resolution. First, a pseudo inner product is defined as ⟨x, y⟩ = ∑Re(xi ·y∗i ), which is consistent with the exact inner product’s definition when we map Cd/2 → Rd. Before calculating attention,
the query and key are encoded with position information. Generally, the attention function is as follows:
aij = ⟨fq(qi, i), fk(kj , j)⟩√
d
oi = i∑
j=0 eaij∑i j=0 e aij vj
(5)
Formally, the encoding must satisfy:
⟨fq(q, n+r), fk(k, n)⟩ = ⟨fq(q, r), fk(k, 0)⟩ (6)
A simple solution is as follows:
fq(q, n) = Aqqe λn fk(k, n) = Akke −λ∗n (7)
The scaling factor Aq, Ak is unnecessary because q, k is obtained by a linear transformation. Since λ ∈ Cd/2, it can be represented as λ = ξ+iθ where ξ, θ ∈ Rd/2:
fq(q, n) = qe ξn+iθn fk(k, n) = ke −ξn+iθn (8)
If ξ = 0, the form is the same as ROPE (Su et al., 2021). Geometrically, the transformation provides a rotation on vectors. If the relative angle between q and k is larger, the inner product is smaller. However, the cosine function is not monotonic if the rotating angle is large than π, which causes an unstable phenomenon in that the expectation of the inner product oscillates dramatically with the growth of relative distance. Following the parameters (Vaswani et al., 2017; Su et al., 2021) θ = {θi = 10000−2i/d, i ∈ [0, 1, ..., d/2]}, we will calculate the expectation as follows. For generative models, we assume E(∠q) ≤ E(∠k) to ensure the monotonicity:
E[⟨qemξ+imθ, ke−nξ+inθ⟩]
=
d/2∑
x=0
E[Re(qxkxe(m−n)ξx+i(m−n)θx)]
≤ d/2∑
x=0
Re(E[|qxkx|]e(m−n)ξx+i(m−n)θx)
∝ d/2∑
x=0
cos(m− n)θxe(m−n)ξx
(9)
The inference here is different from (Su et al., 2021) because of two reasons: 1) there is an additional assumption brought by generative language
models where E(∠q) ≤ E(∠k); 2) the inequality scaling of (Su et al., 2021) is too strong to lose generality. We calculate expectation instead of the upper bound.
Now we define a function to represent the property of relative position:
gζ [n] =
d/2∑
i=0
cosnθiζ n i (10)
g[n] simplifies Equation 9 by defining ζi = eξi . Stabilizing the g[n] curve is intuitive. Although attention bias can achieve this goal, we try to avoid additional position calculations. Instead, we can accomplish this goal using a good ζ to maximize R(gζ).
Obviously, the oscillation mainly comes from large θi. Manually setting ζ can achieve this goal:
ζ̃i = i/(d/2) + γ
1 + γ ∈ [0, 1] (11)
where ζ̃i becomes smaller when θi is larger. In this way, we punish the oscillation of unstable dimensions and keep the distribution of stable ones.
Numerical optimization methods are tried to find optimal values for ζ. However, the results rely on the initial value and lack control when the hidden dimension changes. Besides, the numerical precision should be considered because of fp16’s range. Finally, we find a sub-optimal solution by manually setting γ to both satisfy the resolution is recognizable (R(gζ) is partially optimized) and ζni can be represented by fp16 when n is big (8192 in our setting). Besides, in implementation, the position is re-scaled with base B in the exponential calculation to avoid overflow and underflow (eξn → eξn/B in Equation (8)). We use γ = 0.4 and B = 512 as the final implementation in LEX Transformer.
The curves of ζ = 1, ζ̂ are shown in Figure 2. The default rotary embedding contributes to a dramatic oscillation, especially in the large relative distance, which causes bad extrapolation performance and restricts the model’s convergence speed. After adding a decay, the curve is almost stable, especially on long-term dependency. What’s more, it does not hurt pure rotation’s fitting ability because ζni ≈ 1 when i is large or n is small. In that way, short-term and long-term dependencies are divided continuously.
Finally, we have Extrapolatable Position Embed-
Algorithm 1: Attention with XPOS def rot(x):
return [−x1, x0,−x3, x2, ...] Initialization: θi = 1/10000
2⌊i/2⌋/d, θ ∈ Rd ζ̂i = (2⌊i/2⌋/d+ γ)/(1 + γ), ζ̂ ∈ Rd Input: Q,K, V ∈ Rh×l×d,M ∈ Rd×d Cmn = cosmθn, C ∈ Rl×d Smn = sinmθn, S ∈ Rl×d Tmn = ζ̂ m n , T ∈ Rl×d Q = Q⊙ (C ⊙ T ) + rot(Q)⊙ (S ⊙ T ) K = K⊙(C⊙T−1)+rot(K)⊙(S⊙T−1) output = softmax(QK
T √ d ·M)V return output
ding (XPOS):
fq(q, n) =   q1 cosnθ1ζ̂ n/B 1 − q2 sinnθ1ζ̂n/B1 q2 cosnθ1ζ̂ n/B 1 + q1 sinnθ1ζ̂ n/B 1 ... qn−1 cosnθd/2ζ̂ n/B d/2 − qn sinnθd/2ζ̂ n/B d/2
qn cosnθd/2ζ̂ n/B d/2 + qn−1 sinnθd/2ζ̂ n/B d/2
 
fk(k, n) =   k1cosnθ1ζ̂ −n/B 1 − k2 sinnθ1ζ̂−n/B1 k2 cosnθ1ζ̂ −n/B 1 + k1 sinnθ1ζ̂ −n/B 1 ... kn−1 cosnθd/2ζ̂ −n/B d/2 − kn sinnθd/2ζ̂ −n/B d/2
kn cosnθd/2ζ̂ −n/B d/2 + kn−1 sinnθd/2ζ̂ −n/B d/2
 
(12) In the implementation, the transformation for key and value can be easily calculated by parallel addition and multiplication as shown in Algorithm 1. Since position embedding’s size C, S, T ∈ Rl×d is much smaller than batched multi-head at-
tention matrix and doesn’t require gradients, the cost is almost the same with ROPE and bigger than Absolute Position with 6% additional time.

heading: 3.3 Blockwise Causal Attention
text: To deal with length extrapolation, a simple way to improve attention resolution (Section 3.1) is using windowed attention. During inference, we use blockwise masking (Dai et al., 2019; Zaheer et al., 2020; Xiong et al., 2021) for self-attention. Notice that other window strategies, such as sliding window (Child et al., 2019), also work. We use blockwise causal attention because it is cache-friendly and easy to implement.
As shown in Figure 3, if the pre-training length is l, we divide the query as blocks with l/2 length, and each query interacts with its own block and the last block. In this way, the context information can be delivered by the reuse of key and value. The window constraint helps models encode longer input with improved resolution.
Different from training a long-sequence model with a stop gradient, we use vanilla attention in the training phase, because the pre-training corpus is not very long on average. However, during the inference phase, when dealing with long sequences, we directly implement BCA to help the model to be more position-recognizable.

heading: 4 Experiments

heading: 4.1 Pre-training
text: To fairly evaluate different Transformer variants, we pre-train the Transformer from scratch. We use 1024 hidden dimensions, 16 heads, and 24 layers,
i.e., comparable to medium-size GPT-3 (Brown et al., 2020). The training corpus includes a subset of the Pile (Gao et al., 2020): Books3, OpenWebText2, Stack Exchange, PubMed Abstracts, Wikipedia, Gutenberg (PG-19), BookCorpus2, NIH ExPorter, and Pile-CC datasets. The training procedure is performed on 16×V100 GPUs. We use the tokenizer from GPT2 (Radford et al., 2019). The maximal length is 1024 for saving memory and extrapolation evaluation. The learning rate is 3×10−4 and polynomial decay is used to adjust the learning rate. The global batch size is 512 to follow GPT-3 (Brown et al., 2020), i.e., 0.5M token size. We use Adam (Kingma and Ba, 2015) optimizer with β1 = 0.9, β2 = 0.98, ϵ = 10−6. The code is based on TorchScale (Ma et al., 2022a).

heading: 4.2 Language Modeling
text: We measure perplexity on long document datasets, which can show the model’s ability for longdependency modeling. We use books from Project Gutenberg whose years are later than 2019 to ensure no overlap with PG19, and we name it as PG22. Besides, we pick QMSum (Zhong et al., 2021) from SCROLLS (Shaham et al., 2022) with above 9k length on average. We care about the performance on different input lengths to evaluate the model’s interpolation and extrapolation capability. For experiment results in Table 2, we divide the same input into the target length to fairly compare the perplexity of different lengths.
For interpolation capability, we analyze the results where the length is no more than 1024. Since the validation distribution is very similar to training data, all Transformers’ generalization capabilities are also close. XPOS have a stable advantage on others with a 0.09 perplexity drop on PG22, and 0.27 on QMSum, which proves that XPOS increases the interpolation ability.
For extrapolation lengths, we do not use BCA in other Transformers, and the following ablation study will discuss the performance with that. Press et al. (2021)’s experiment shows that most of the position strategies can’t deal with input length longer than pre-training directly. XPOS shows a stable decrease when the sequence length increases, which satisfies the assumption that a longer context makes the prediction better. While others’ perplexity increases when the input length is 4096.
To illustrate the tendency of perplexities, Figure 1 visualizes the relation between input length
and perplexity. When the length is larger than 4096, Alibi’s perplexity increases gradually. However, LEX’s perplexity decreases continuously when the length extends to 8192.
The experiment shows that XPOS gets better performance on language modeling. With the stable advantage of any length, users can input any sentence freely without the concern of position. Besides, results also indicate that is not essential to build an explicit decay on the attention matrix, Instead, a proper design for an attention mask is actually better to deal with long-context tasks.

heading: 4.3 Measuring Resolution
text: We empirically evaluate the resolution of different Transformer variants. In the previous section, we define attention resolution as a quality indicator of position modeling in Transformers. The expectation of s[n] is computed as:
E[s[n]] = 1 N − nE[ N−1∑
i=n
ai(i−n)] (13)
where aij has the same meaning in Equation 5. Then the attention resolution can be calculated by combining Equation (4) and Equation (13). The final expectation is averaged over input texts and different layers.
Table 3 reports the average resolution of various Transformer variants. The results show that XPOS makes the position more recognizable in both 1024 (i.e., training length) and 2048 (i.e., length extrapolation). For Alibi (Press et al., 2021), the stable resolution comes from explicit decay, but it prevents the model from learning position dependency itself. In addition, we ablate BCA in 1024 and 2048. The results support that BCA helps the model distinguish positions better, achieving higher attention resolution.

heading: 4.4 Ablation Studies

heading: 4.4.1 Rotation Computation
text: As shown in Table 4, we discuss the necessity of the combination of vector rotation and exponential decay. XPOS without rotation means Equation (12) degenerates to θi = 0:
ḟq(q, n) =   q1ζ̂ n 1 q2ζ̂ n 1
... qn−1ζ̂nd/2 qnζ̂ n d/2
  ḟk(k, n) =   k1ζ̂ −n 1 k2ζ̂ −n 1 ... kn−1ζ̂ −n d/2
knζ̂ −n d/2
 
Moreover, the setting of ζ = 0 is RoPE (Su et al., 2021), which can be viewed as a special case of our method. Besides, we discuss the situation when ζ is a scalar instead of a vector, where we choose ζ = γ/(1 + γ) as the value.
After pre-training on 1024, we evaluate the perplexity of PG22 with 1024 and 8192 lengths. Table 4 shows that simple scaling operation cannot match the performance of LEX. The vector ζ also performs better than ζ = 0 and γ/(1 + γ). Therefore, the combination of rotation and decay means the combination of in-distribution and outof-distribution capability in terms of length.

heading: 4.4.2 Blockwise Causal Attention
text: As shown in Table 5, we run the evaluation using different position embeddings (i.e., Absolute, Alibi, ROPE, and XPOS) with or without blockwise
causal attention. First, Blockwise Causal Attention works for ROPE whose perplexity will explode without that. Alibi performs well without windowed attention because its “soft window” is broader than a hard block window. However, when the sequence length increases to 8192, windowed attention outperforms vanilla attention again (also shown in Figure 1). XPOS’s perplexity without BCA increases by about 1.5 in 2048, and 40 in 8192. However, with its high resolution, XPOS can recognize position with BCA’s constraint.
Besides, we compare BCA with Sliding Attention (Child et al., 2019). In this experiment, we set the window size as 1024 to align with pre-training. Sliding Attention performs better as shown in the last row of Table 5 because its interaction range is broader than Block Causal Attention. The reason to use block windows instead of sliding windows is efficiency. According to (Xiong et al., 2021), the training speed of Blockwise Attention is 1.5x
faster than using sliding windows. Therefore, LEX makes a trade-off and uses BCA in our implementation. Without losing generality, our method is also compatible with Sliding Attention and other local attention variants.

heading: 5 Related Work

heading: 5.1 Long-Sequence Transformers
text: Long-sequence Transformers aim to solve two key problems. First, the computation or memory consumption is not efficient enough for long sequences. Second, there is a trade-off between performance and efficiency.
One popular solution (Wang et al., 2020b; Katharopoulos et al., 2020; Choromanski et al., 2020) is linear attention, i.e., using a kernel-based or low-rank approximation to replace vanilla attention. The methods typically target efficiency while underperforming vanilla Transformers for regular length. Another strand is sparse attention (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020; Xiong et al., 2021), which usually leverages structured sparsity to reduce computation. For causal sequence modeling, the recurrent-style designs (Dai et al., 2019; Hutchins et al., 2022; Ma et al., 2022b) are also competitive.
In comparison, we focus on length extrapolation (Press et al., 2021) for language modeling, i.e., training on short texts while evaluating long texts. The training process is kept the same as vanilla Transformers. The capability of long-sequence modeling is given for free during inference. So training efficiency (which is typically expensive for large-scale language models) is not affected
compared with previous work. Moreover, the performance on regular length is perfectly retained, without trade-offs for long-sequence modeling.

heading: 5.2 Position Modeling

heading: 5.2.1 Absolute Position Embedding
text: Absolute sinusoidal position embedding is proposed by Vaswani et al. (2017), which is the initial design of the Transformer. For each dimension, different frequencies are encoded from 2π to 10000× 2π:
PE(pos,2i) = cos(pos/10000 2i/dmodel)
PE(pos,2i+1) = sin(pos/10000 2i/dmodel)
(14)
where PEpos+k is represented as a linear function of PEpos to restore a relative-position property.

heading: 5.2.2 Relative Position Embedding
text: Shaw et al. (2018) propose relative position embedding as an alternative approach. Denote aij as attention weight, αij = softmax(aij), oi as output, we have:
aij = qi · kj√
d =⇒ qi · (kj + pKij )√ d
oi = ∑
j
αijvj =⇒ ∑
j
αij(vj + p V ij)
(15)
where pKij = ω K min(i−j,k),p V ij = ω V min(i−j,k), and ωK and ωV are learnable parameters. The clipping strategy helps length generalization but cannot distinguish the positions that are larger than k. Yang et al. (2019) and He et al. (2020) further reparameterize the relative position vectors for better performance. T5 (Raffel et al., 2020) uses a simpler strategy to encode relative position:
aij = qi · kj√
d + pbucket(i−j) (16)
where log-bucket scalars are added to attention scores. Recently, pre-defined position embedding is brought back by ROPE (Su et al., 2021). Alibi (Press et al., 2021) proposes to explicitly build an exponential decay on the attention matrix, which contributes to length extrapolation:
aij = qi · kj√
d −m(i− j), m(·) > 0 (17)
where the values of m(·) are manually defined. However, Alibi (Press et al., 2021)’s performance tends to be inferior to ROPE for the context whose
length is shorter than the pre-training length. In this work, we propose a theoretically derived relative position embedding XPOS that optimizes the attention resolution between tokens. The XPOS method not only has the nice property of length extrapolation but also achieves strong performance.

heading: 6 Conclusion
text: We propose LEX Transformer to accurately capture position information for Transformers. We define attention resolution as the metric of length extrapolation and design a solution to improve the modeling. Extensive experiments on language modeling show that our method achieves lower perplexity on longer sequences while training on short texts. The simplicity also makes the method a go-to augmentation for Transformer-based language models. In addition, attention resolution provides a more principled view for position modeling, which sheds light on future architecture design.
Limitations
In this work, we focus on causal language modeling. It needs additional efforts to integrate the proposed methods into bidirectional attention, such as masked language modeling (Devlin et al., 2019). Moreover, XPOS introduces about 6% inference cost compared with absolute position embeddings, although it accelerates training convergence.

heading: A Additional Experiments
text: Besides the experiments in Section 4, we run language modeling evaluation on Arxiv and NarrativeQA (Kočiskỳ et al., 2018). The results are shown in Table 6. These datasets have their shortcomings. The article length of Arxiv is usually less than 8192, and part of NarrativeQA’s corpus is sampled from PG19, which is in the training dataset. Therefore, we show them in the appendix instead of the main content.

heading: B Hyperparameters for Pre-Training
text: As shown in Table 7, we present the hyperparameters for pre-training. The setting keeps the same among all Transformer variants. We follow medium-size GPT3 (Brown et al., 2020), 24 layers, 1024 hidden size, 4096 FFN inner hidden size, and 16 attention heads. The number of batch tokens is 0.5M, for pre-training 1024, and the number of batch sentences is 512. We use Adam (Kingma and Ba, 2015) optimizer with β1 = 0.9, β2 = 0.98, ϵ = 10
−6. The warmup steps are 20k, and we use 50k checkpoints for evaluation.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the final part after conclusion
A2. Did you discuss any potential risks of your work? Not applicable. It is fundamental research and not tied to particular applications.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. These tools and models are publicly available and free of use for research purposes.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The use is consistent with their intended use.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 4

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section4.1 and Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section4.1 and Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In addition, attention resolution provides a more principled view for position modeling, which sheds light on future architecture design. Limitations
In this work, we focus on causal language modeling. It needs additional efforts to integrate the proposed methods into bidirectional attention, such as masked language modeling (Devlin et al. , 2019). Moreover, XPOS introduces about 6% inference cost compared with absolute position embeddings, although it accelerates training convergence