Interpreting Positional Information in Perspective of Word Order
Year: 2023

Abstract:
The attention mechanism is a powerful and effective method utilized in natural language processing. However, it has been observed that this method is insensitive to positional information. Although several studies have attempted to improve positional encoding and investigate the influence of word order perturbation, it remains unclear how positional encoding impacts NLP models from the perspective of word order. In this paper, we aim to shed light on this problem by analyzing the working mechanism of the attention module and investigating the root cause of its inability to encode positional information. Our hypothesis is that the insensitivity can be attributed to the weight sum operation utilized in the attention module. To verify this hypothesis, we propose a novel weight concatenation operation and evaluate its efficacy in neural machine translation tasks. Our enhanced experimental results not only reveal that the proposed operation can effectively encode positional information but also confirm our hypothesis.

heading: 1 Introduction
text: In recent years, attention mechanism (Bahdanau et al., 2015; Luong et al., 2015; Lin et al., 2017) has made remarkable progress on a wide range of natural language processing (NLP) tasks, such as machine translation (Vaswani et al., 2017; Radford et al., 2018), question answering and language inference (Devlin et al., 2019). It updates the contextual representation of a word by aggregating information from other words in the context, enabling it capture the content-based relevance of any two words, regardless of the distance between them.
In contrast to widely used recurrent neural networks (RNNs) or convolutional neural networks (CNNs), attention mechanism suffers from a notable disadvantage, i.e. its permutation invariance (Yun et al., 2020). This limitation results in its
∗Corresponding author.
failure to distinguish sequences with different word orders. As a result, numerous studies have focused on encoding positional information of a sequence for the attention mechanism, such as learnable fixed-length positional encoding (Gehring et al., 2017), sinusoidal positional encoding (Vaswani et al., 2017) and relative positional encoding (Shaw et al., 2018). Typically, these methods assign positional embeddings to words at different positions through vector addition in various indexing manners, such as absolutely indexing and relatively indexing. However, it remains unclear how the attention mechanism incorporates positional information into a sequence through the addition of positional embeddings to word embeddings.
Besides, previous studies have probed the influence of word order perturbation on natural lan-
9600
guage understanding (NLU) tasks (Abdou et al., 2022; Pham et al., 2021; Clouâtre et al., 2021) to determine whether these tasks are sensitive to word order. The findings indicate that some NLU tasks do require word order information although others do not. It should also be noted that word order is particularly important for natural language generating (NLG) tasks, such as machine translation. This is because metrics used to evaluate generated results, such as BLEU (Papineni et al., 2002), are sensitive to word order. Therefore, word order is a critical aspect of natural language processing.
In linguistics, grammars can be viewed as explanations for rules or principles of word order (Hawkins, 1990) to some extent. Given a sentence of arbitrary length, a set of positional embeddings can be seen as a group of distributed vectors that represent temporal information (Elman, 1990), namely word order in the context of language processing. This insight motivates us to correlate positional information with word order, interpret positional information in perspective of word order, and comprehend how word order impacts attention models.
In this paper, we first examine how the attention module of Transformer works and discover that its weight sum operation (Figure 1(a)) is the reason it struggles to encode positional information. Furthermore, this sheds light on how the word order impacts on NLP models, which means that the impact is achieved by its connection with positional information.
Based on this finding, we make further modifications to the attention module’s working mechanism. Our goal is to implicitly encode positional information in attention module, which we posit is superior to explicit representation of positional information, as suggested by Elman (1990). To be more concrete, we devise a novel weight concatenation operation (Figure 1(b)) to calculate the final contextual representation, as an alternative to the weight sum operation (Figure 1(a)) in the attention module. To test the effectiveness of this approach, we evaluate the novel operation on the widely used, big neural machine translation datasets, including WMT 2014 English⇒German and WMT 2014 English⇒French. Our experimental results demonstrate that the proposed operation is capable of effectively encoding positional information for a sequence, leading to consistently improved performance and verifying our hypothesis.

heading: 2 Background

heading: 2.1 Attention Mechanism
text: In this section, we provide a brief introduction about the attention mechanism in Transformer (Vaswani et al., 2017). To simplify, we consider attention layers only with a single head rather than multiple heads. Also, let s (·) denote the softmax operator that performs softmax operation to each row of a matrix. By dropping the residual connection (He et al., 2016) and the layer normalization (Ba et al., 2016), the attention layer can be generally formed as:
Attn (Q,K,V) = s ( QK⊤/ √ dk ) V , (1)
Q,K,V = XWQ,XWK ,XWV , (2)
where Q ∈ RL×dq ,K ∈ RL×dk ,V ∈ RL×dv are packed queries, keys and values respectively, which are results of affine transformation on the input. X ∈ RL×d is the embedding result or the hidden representation of previous layer with L being the sequence length and d being the model dimension. dq, dk, dv are the dimension of queries, keys and values respectively, typically with dq = dk = dv. WQ ∈ Rd×dq ,WK ∈ Rd×dk ,WV ∈ Rd×dv are trainable linear projections.
To clearly show the integration of positional encodings in Section 2.2, only consider the contextual result of a single word. Given the embedding result or hidden representation X ∈ RL×d as input, the i-th output of the attention mechanism, denoted as oi, can be calculated as:
oi = s ( qiK ⊤/ √ dk ) V = L∑
j=1
aijvj , (3)
where aij is the attention score of qi over kj and calculated as:
aij = exp
( qik ⊤ j / √ dk )
∑L j=1 exp ( qik⊤j / √ dk ) . (4)
For convenience, define the function a (qi,K) : Rdq × RL×dk → RL as:
a(qi,K) := s ( qiK ⊤/ √ dk ) . (5)

heading: 2.2 Positional Encoding Mechanism
text: To remove the permutation invariance constraint (Yun et al., 2020), various positional encodings have been proposed and they are usually integrated into embedding layers (embedding-level) or attention layers (attention-level).
Embedding-Level Positional encodings at embedding level are added to the embedding results of a sequence, immediately following the embedding layers. They are usually a set of predefined or trainable vectors, indexed with absolute position numbers. After that, the contextual representation oi can be represented as:
oi = a (xi + bi,X+B) (X+B)WV , (6)
where bi ∈ Rd is the positional embedding for the i-th word, B ∈ RL×d is the packed positional embeddings for the whole sequence.
Attention-Level Positional encodings at attention level are based on an insight into the attention mechanism. Since they are employed in attention layers, they can access to both the query and the key. This enables them to encode positional information in a more complex manner, such as indexing with the relative distance between the query and the key. Taking relative positional encoding (Shaw et al., 2018) as an example, the contextual representation oi can be calculated as:
oi = L∑
j=1
aij ( vj + r v ij ) , (7)
aij = a(qi,kj + r k ij) , (8)
where rkij , r v ij ∈ Rdk are trainable positional encodings for the j-th key and j-th value respectively. These encodings are indexed based on the distance between xi and xj .

heading: 3 Analysis and Method

heading: 3.1 Correlation Between Positional Encoding and Word Order
text: In this section, we first demonstrate that the attention mechanism of Transformer is permutation invariant and analyse the reason. After that, we show how to correlate positional information with the word order of a sequence and perform positional encoding through preserving word order.
It is straightforward to obtain the permutation invariance of attention mechanism in Transformer from Equation (3). Given the input X ∈ RL×d and the permutation matrix P ∈ RL×L, the permutation invariance constraint can be demonstrated as follows:
oi = a (qi,PXWK) (PXWV )
= a (qi,XWK)P ⊤PXWV
= a (qi,XWK)XWV , (9)
where s ( XP⊤ ) = s (X)P⊤ and P⊤P = I are used. From Equation (9), it is obvious that the contextual representation oi is invariant to whatever permutation transformations applied to the input.
However, the permutation invariance constraint does not generalize to RNNs and CNNs, which compute the contextual representation in a different manner. Therefore, we may focus on the way of computing the contextual representation in the attention mechanism. We speculate that the constraint is a result of the weight sum operation in Equation (3). In other words, the weight sum in the attention mechanism eliminates the positional distinctions among words of a sequence.
Hence, to encode positional information for the attention mechanism, we place emphasis on how to keep the word order of a sequence when calculating the contextual representations of these words. Following this way, we propose the weight concatenation operation as an alternative to the weight sum operation. We denote the weight concatenation of attention score ai ∈ RL and value matrix V ∈ RL×dv as ai ⊕ V : RL × RL×dv → RLdv , which can be formally represented as:
ai ⊕V = [ai1v1 : ai2v2 : · · · : aiLvL] , (10)
where : represents the vector concatenation. Then, the contextual representation oi can be represented as:
oi = (ai ⊕V)Φ , (11)
where Φ ∈ RLdv×dv is a linear projection matrix. In Equation (11), the value vectors are first weighted by the corresponding attention scores and then concatenated, followed by the linear projection Φ to reduce the dimension of the concatenated result to the original dimension. Since Equation (11) concatenates the weighted value vectors exactly according to the word order of the sequence, it is obvious that the attention mechanism is permutation variant now, which can be shown as:
oi = (a (qi,PXWK)⊕ (PXWV ))Φ = ( a (qi,XWK)P ⊤ ⊕P (XWV ) ) Φ
= ( a (qi,XWK)P ⊤ ⊕ (XWV )P⊤ ) Φ
(12)
̸= (a (qi,XWK)⊕ (XWV ))Φ , (13)
where PX = XP⊤ is used. Equation (12) means that the permutation transformation applied to the
input gives rise to a change in the order of both the attention scores and value vectors. Then the attention mechanism can not be permutation invariant any longer (Equation (13)).
Therefore, the weight concatenation implicitly encodes positional information for attention mechanism. To sum up, the retention of the word order information prevents the positional information from being lost during computing the contextual representation, which correlates positional encoding with the word order from a linguistic perspective.

heading: 3.2 Positional Kernels
text: As shown in Section 3.1, the weight concatenation is proposed to eliminate the permutation invariance constraint. However, practical space complexity concerns arise with the weight concatenation since it requires extending the dimension of each word vector from dv to Ldv, given a sequence of length L. Besides, we want to circumvent increasing time complexity and utilize highly efficient parallel matrix calculation on GPU. Fortunately, both of these issues can be resolved through factoring the matrix Φ as:
oi = (ai ⊕V)Φ = L∑
j=1
aijvjϕj (14)
= L∑
j=1
aij ( vjϕj ) = ai (V ⊗Φ) , (15)
where ϕj ∈ Rdv×dv , j ∈ {1, . . . , L} is a subblock of Φ. ( vjϕj ) means vj is multiplied by ϕj first. ⊗ is the batched matrix multiplication that broadcasts along the sequence length dimension of V and Φ. V ⊗Φ can be defined as:
[V ⊗Φ]i := viϕi , (16)
where vi ∈ Rdv and ϕi ∈ Rdv×dv . In Equation (14), we reshape the matrix Φ ∈ RLdv×dv as a tensor Φ ∈ RL×dv×dv , namely splitting Φ into a set of subblock ϕj ∈ Rdv×dv . Multiplying vj by ϕj first in Equation (15) makes it possible to apply efficient parallel matrix multiplication through tensor reshaping and broadcasting. The highly efficient matrix form can be represented as:
Attn (Q,K,V) = s ( QK⊤/ √ dk ) (V ⊗Φ) . (17)
It is amazing that the attention mechanisms with matrix blocks ϕj in Equation (17) are highly similar to the existing positional encodings in Equations (6) and (7). Both are position-specific and capable of encoding positional information, however, the way that they compute the contextual representation is totally different. Therefore, ϕj in Equation (15) can be regarded as a novel positional encoding strategy and we dub it as “positional kernel”.

heading: 3.3 Positional Encoding Network
text: In this section, we integrate the proposed positional kernels into Transformer and demonstrate the existing positional encodings as special cases of positional kernels, then develop two types of positional encoding networks.
Recall that positional encodings (Vaswani et al., 2017; Shaw et al., 2018) are typically integrated into the model through adding positional embeddings to hidden representations, either in the embedding layers or attention layers. Omitting trivial distinctions, such as indexing manners, the result of incorporating positional encodings can be uniformly represented as:
Hpe = X+M , (18)
where X ∈ RL×d is the input, M ∈ RL×d is the positional encoding matrix and Hpe is the hidden representation after applying the positional embedding M.
Likewise, we apply the proposed positional kernel tensor Φ ∈ RL×d×d to the embedding layers or attention layers in the following manner:
Hpk = X⊗Φ . (19)
It can be proved that existing positional encodings in Equation (18) can be regarded as special cases of the method with positional kernels in Equation (19), which is established as the following theorem:
Theorem 3.1. Let ϵ > 0, then for any given X ∈ RL×d and M ∈ RL×d, there exists a Φ∗ ∈ RL×d×d such that, for f (X) = X ⊗ Φ∗ and g (X) = X+M, ∥f (X)− g (X)∥2 ≤ ϵ. Remark 3.2. We provide the detailed proof of Theorem 3.1 in Appendix A.1. To sum up, given the input X ∈ RL×d and positional encodings M ∈ RL×d, there exists a set of corresponding positional kernels ϕj ∈ Rd×d, j ∈ {1, . . . , L},
such that f (X) = X⊗Φ can closely approximate g (X) = X+M. Therefore, the existing positional encodings (Equation (18)) can be regarded as special cases of the proposed positional kernels (Equation (19)).
To integrate the positional kernel into Transformer and make it work well, we further develop the corresponding Positional encoding Network (PosNet) at attention level and embedding level.
Attention-Level In a manner similar to Shaw et al. (2018), we apply the positional kernel to attention layers to preserve positional information. Given the input X ∈ RL×d and positional kernel tensor Φ ∈ RL×dk×dk , the attention mechanism with PosNet can be represented as follows:
Attn (Q,K,V) = s ( QK⊤/ √ dk ) PosNet (V;Φ) , (20)
where PosNet (·) is the function of the proposed PosNet, which can be formed as:
PosNet (X;Φ) = σ (X⊗Φ) , (21)
where σ (·) is the non-linear activation function.
Embedding-Level The equivalent implementation of the weight concatenation in Equation (15) makes it possible to match vj with ϕj , which enlightens us to apply the proposed positional kernel to the embedding layer. Besides, inspired by the feed forward network (FFN) in Transformer, we also use two linear projections to transform the input to a desired low-dimensional representation space (Yun et al., 2020). Given the embedding result X ∈ RL×d and the positional kernel tensor Φ, the output of PosNet can be formally represented as:
PosNet (X;Φ) = σ (XW1 ⊗Φ)W2 , (22)
where W1 ∈ Rd×d1 and W2 ∈ Rd2×d are trainable linear projections. Φ ∈ RL×d1×d2 and we use d1 = d2 < d to alleviate the increase of parameter counts.
Residual Connection and Dropout The identity term in the proof of Theorem 3.1 (Appendix A.1) indicates that it is also helpful to retain the original information, i.e. X in Equations (21) and (22). This form exactly corresponds to the residual connection (He et al., 2016), which can facilitate the
training of the network. Hence, the residual connection is employed to wrap PosNet modules, both at embedding level and attention level. Then, Equations (21) and (22) can be reformed as:
PosNet (X;Φ) = σ (X⊗Φ) +X , (23) PosNet (X;Φ) = σ (XW1 ⊗Φ)W2 +X .
(24)
Apart from the residual connection, the dropout (Srivastava et al., 2014) is also applied to the output of PosNet modules as a regularizer and immediately followed by the residual connection, which is also inspired by the delicate design of Transformer. The dropout rate of the PosNet is independent from that of Transformer. The illustration of applying the PosNet to Transformer is available in Appendix A.3.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets The proposed methods are evaluated on the widely used machine translation benchmarks, including WMT 2014 English⇒German (En-De) and WMT 2014 English⇒French (En-Fr), with about 4.43 million and 35.76 million parallel sentence pairs for training respectively. For both language pairs, the newstest2013 dataset is used as the validation set and the newstest2014 dataset as the test set.
Model In all experiments, English, German and French datasets are tokenized via the scripts in Moses (Koehn et al., 2007). For all language pairs, the byte pair encoding (BPE) compression algorithm (Sennrich et al., 2016) is employed to reduce the size of the vocabulary and enable the model to translate rare and unknown tokens. The vocabulary size is set to 40,000 for both En-De and En-Fr translation tasks. The Adam algorithm (Kingma and Ba, 2015) is used as the optimizer with β1 = 0.9, β2 = 0.98, ϵ = 10−9. The label smoothing (Szegedy et al., 2016) is adopted as well, with ϵls = 0.1.
We implement all methods on top of FAIRSEQ (Ott et al., 2019) and follow its preprocessing and training instructions on neural machine translation. We take advantage of efficient half-precision floating point (FP16) arithmetic for both the training and evaluation stage. To maintain approximately 25,000 source tokens and 25,000 target tokens in each training batch as Vaswani et al. (2017), we limit the number of
tokens in each batch to 4096 and accumulate the gradients of 8 batches for each update (Ott et al., 2018). We re-implement all compared methods and keep experimental sets same for these methods to ensure a consistent and fair comparison. We run all experiments with NVIDIA RTX3090 24 GB GPU cards.
Evaluation We employ beam search with a beam size of 4 and a length penalty a = 0.6. We evaluate the performance of models with both the common used BLEU (Papineni et al., 2002) and recently proposed ChrF++ (Popović, 2017; Marie et al., 2021), which shows better correlation with human judgments. Besides, we perform statistical significance test with bootstrap resampling (Koehn et al., 2003).
Compared Methods Compared positional encoding methods mainly include
• w/o PE: without any positional encoding, to provide another reference of positional encod-
ing ability besides the baseline, • SAPE: sinusoidal absolute positional encod-
ing (Vaswani et al., 2017), • LAPE: learnable absolute positional encoding,
to evaluate the effect of trainable positional encoding, • RPE: relative positional encoding (Shaw et al., 2018), • Adaptive-T5: adaptive version of T5’s relative positional encoding (Wu et al., 2021), • PosNet-Attn: attention-level PosNet (Section 3.3), • PosNet-Embed: embedding-level PosNet (Section 3.3).
We provide the relevant codes and scripts1 of our experiments, which is helpful to reproducibility. Besides, the datasets2 used in our experiments are all publicly available.

heading: 4.2 Results
text: Experimental results are shown in Tables 1 and 2 and Figure 2. We make the following observations:
Strong Baseline To isolate the impact of different positional encoding strategies from any other unrelated factors, such as the underlying implementation detail and experimental configuration, we re-implement the baseline model (SAPE) and other compared methods. In comparison to the reported result (Vaswani et al., 2017), the reproduced baseline result achieves better performance, especially
1https://github.com/vesterchang/ interpret_positional_encoding
2https://www.statmt.org/wmt14/index. html
on the WMT 2014 En-Fr translation task. This indicates that the system in this work is a strong baseline.
Positional Information is Critical To assess the significance of positional information, we conduct experiments on Transformer without any positional encodings (w/o PE). As a result, the performance decreases drastically. This suggests that positional information is critical for Transformer, which relies solely on the position insensitive attention mechanism.
Performance of PosNet The proposed approaches achieve competitive performance or statistically significant improvement over the baseline. Specifically, PosNet-Attn outperforms the w/o PE, indicating that it effectively encodes positional information, although it is worse than the baseline. However, the PosNet-Embed achieves better performance in terms of both BLEU and ChrF++ than the baseline on both of WMT 2014 En-De and En-Fr translation tasks. This demonstrates the superiority of the PosNet-Embed in encoding positional information.
Comparison with Other Methods In order to make a direct and fair comparison with other positional encoding methods (LAPE and RPE), we re-implement them and evaluate them with new metric, i.e. ChrF++. For RPE, we follow the configuration in Shaw et al. (2018). As shown in Table 1, both LAPE and RPE are worse than PosNet-Embed on both BLEU and ChrF++, which demonstrates that the PosNet-Embed encodes the positional information better again.
Computation Complexity of Different Methods To evaluate the computation complexity of different positional encoding methods, we manually con-
struct a batch of samples and only perform the forward stage with Transformer base model since the model with different positional encoding methods would perform different numbers of decoding steps in practical translation scenario. We perform the forward stage with the same batch 10 times and record the time consumed by the model. As shown in Figure 2, all methods exhibit similar time complexity, with the exception of the RPE and AT5, which is obviously more time-consuming.
Furthermore, we conduct an analysis of the GPU memory consumption of the Transformer base model with different methods on the WMT 2014 En-De and En-Fr tasks. This is done to determine their relative space complexity. The results of this analysis can be found in Appendix A.2, which demonstrate that all methods exhibit comparable space complexity, except for RPE, which consumes more GPU memory on the WMT 2014 En-Fr task.
Scaling to Transformer Big We also implement these methods in the Transformer big model to observe the performance when scaling to big models. As shown in Table 2, the reproduced SAPE and RPE achieve competitive or better results in terms of BLEU compared with their reported results (Vaswani et al., 2017; Shaw et al., 2018). Besides, the PosNet-Embed achieves better performance than other methods, especially its statistically significant improvement on most results with significance level 0.05/0.10.

heading: 5 Discussion
text: Effect of Positional kernel’s dimension As stated previously in Equation (22), the positional kernel tensor Φ has the shape of L× d1 × d2. In practice, we use d1 = d2 = dp. To further investigate the effect of the dimension dp, we conduct experiments on PosNet-Embed with a series of varying dp and evaluate the performance of dif-
ferent dp with the BLEU score. Figure 3 shows the result of ablation experiment on the validation dataset, i.e. the newstest2013 dataset. Notably, dp = 96 has a detrimental effect on the performance of PosNet-Embed and dp = 128 with the best performance. Thereby, we conduct the experiments on the PosNet-Embed in Table 1 with d1 = d2 = 128. Besides, we use d1 = d2 = 256 for the Transformer big model, which is just twice the size of the base model.
Evaluation with different n-grams The default evaluation setup is 6-gram character F2 and 2- gram word F2 for ChrF++, 4-gram word precision for BLEU. However, since the PosNet preserves positional information through concatenating the weighted value vectors, it is intriguing what the performance is when evaluating with coarser-grain unit. Therefore, to observe the performance superiority of the proposed method on coarser grain, we evaluate the performance gap of PosNet-Embed over SAPE on WMT14 En-De with different ngrams. Specifically, we test the ChrF with n-gram character F2 and 2-gram word F2 (1 ≤ n ≤ 30), the ChrF with 6-gram character F2 and n-gram word F2 (1 ≤ n ≤ 8), and the BLEU with ngram word precision (1 ≤ n ≤ 8) respectively. As shown in Figure 4, it is amazing that the performance superiority of the PosNet-Embed over the SAPE consistently enlarges as the evaluating grain becomes coarser, i.e. bigger n-gram character or word. We speculate that this is attributable to the weight concatenation of PosNet since the concatenation operation promotes the model to capture the whole context precisely according to the word order of a sequence, which preserves the coarser-grained feature.

heading: 6 Related Work
text: Since Transformer relies solely on the attention mechanism, it is position-insensitive to sequences with different word orders in the absence of positional encodings. Many studies attempt to utilize various positional encodings to inject the sequence’s positional information into the model. Typically, embedding-level positional encodings (Vaswani et al., 2017; Devlin et al., 2019; Gehring et al., 2017) build positional dependencies via adding positional encodings to embedding results. In addition, attention-level positional encodings (Shaw et al., 2018; Raffel et al., 2020; Huang et al., 2020; Wu et al., 2021) usually capture positional information based on an analysis of attention mechanisms and leverage delicate indexing manners, such as the relative distance to the querying word. However, it is unknown why the attention mechanism of the Transformer is position agnostic and what the working principle behind these positional encodings is.
There has been increasing interest in understanding and explaining how these positional encodings construct positional dependencies for a sequence. The proof in Yun et al. (2020) suggests that the key of eliminating Transformer’s permutation invariance is to quantify the embedding results to distinct intervals, which ensures a one-to-one mapping. Wu et al. (2021) analysed and improved the scalar relative positional encoding of T5 (Raffel et al., 2020) from a probabilistic perspective and took it as a prior distribution. The difference is that, in this work, we associate the positional information with the word order of a sequence, providing a novel linguistic perspective on positional encoding. Furthermore, we propose a new and effective positional encoding mechanism. We perform the
comparison of different positional encoding methods and the results are in Tables 1 and 2 except that the theoretical positional encoding scheme in the proof of Yun et al. (2020) doesn’t work well in practice and thus are not reported.

heading: 7 Conclusion
text: Positional encoding is a critical issue for Transformer-like models. However, it has not been explored how positional encoding builds positional dependencies for a sequence. In this paper, we analyse the working manner of the attention module and find that its weight sum operation leads to the failure to encode positional information. Then, we modify the attention module with a proposed novel weight concatenation operation, following the guideline of retaining positional information according to the word order of a sentence. The modification correlates a sequence’s word order with positional information, thus shedding light on how the word order impacts on NLP models. Competitive experimental results substantiate our hypothesis.
Limitations
In this paper, we present a novel approach to remove the permutation invariance of the attention module. Specifically, we propose a weight concatenation operation that exactly follows the word order of a sentence, leading to an increase in dimensionality and the introduction of affine transformations aimed at reducing it. Hence, the effect of increased parameter counts cannot be well isolated. While our preliminary experiments show that an increase in the number of parameter counts does not necessarily enhance the experimental results, we acknowledge the increased complexity resulting from direct concatenation and, thus, have utilized the equivalent form of the proposed operation in practice. In the future, we aim to explore alternative operations that implicitly encode positional information based on word order, without resorting to affine transformations, to replace the weight sum operation of the attention module.

heading: Acknowledgments
text: This work was supported by the Provincial Natural Science Foundation of Shaanxi of China (No. 2019JZ-26). We acknowledge all the anonymous reviewers for their valuable comments and suggestions.

heading: A Appendix
text: A.1 Proof of Theorem 3.1
Proof. To make the proof clear and concise, take into account only the individual word xi. The corresponding positional embedding and positional kernel are mi and ϕi. Let’s ignore the subscript i which indexes position for simplification if no confusion is possible and then f (·) and g (·) can be represented as:
f (x) = xϕ , (25)
g (x) = x+m . (26)
Then, to obtain the conclusion, we reconstruct the ϕ as ϕ ′ + I, which exactly corresponds to the widespreadly used and effective residual connection (He et al., 2016). Then,
f (x) = x ( ϕ ′ + I ) = xϕ ′ + x . (27)
The l2 norm is
∥f (x)− g (x)∥2 = ∥∥∥xϕ′ + x− x−m ∥∥∥ 2 (28)
= ∥∥∥xϕ′ −m ∥∥∥ 2
(29)
Then, we only need to find a ϕ ′∗ such that, for
ϵ > 0, ∥∥∥xϕ′∗ −m ∥∥∥ 2 ≤ ϵ.
Let’s take into account the following optimization problem:
min ϕ ′ ∥∥∥xϕ′ −m ∥∥∥ 2
(30)
Since we adopt stochastic gradient method to train the network, it is straightforward that if h (x) =∥∥∥xϕ′ −m
∥∥∥ 2 is convex, then we can obtain a ϕ ′∗
such that ∥∥∥xϕ′∗ −m ∥∥∥ 2 ≤ ϵ (Zinkevich, 2003). Therefore, we have only to prove h (x) is convex.
For any x1,x2 ∈ Rd,
h (x2)− h (x1)− ⟨∆h (x1) ,x2 − x1⟩ (31) = ∥∥∥x2ϕ ′ −m
∥∥∥ 2 − ∥∥∥x1ϕ ′ −m ∥∥∥ 2
−
( x1ϕ ′ −m ) ϕ
′⊤ ∥∥∥x1ϕ′ −m
∥∥∥ 2 (x2 − x1)⊤ (32)
= ∥∥∥x2ϕ ′ −m ∥∥∥ 2
−
( x1ϕ ′ −m )( x2ϕ ′ −m )⊤ ∥∥∥x1ϕ′ −m
∥∥∥ 2
(33)
Since [∥∥∥x2ϕ
′ −m ∥∥∥ 2 · ∥∥∥x1ϕ ′ −m ∥∥∥ 2 ]2
= [( x1ϕ ′ −m )( x2ϕ ′ −m )⊤]2 , (34)
then ∥∥∥x2ϕ
′ −m ∥∥∥ 2 · ∥∥∥x1ϕ ′ −m ∥∥∥ 2
− ( x1ϕ ′ −m )( x2ϕ ′ −m )⊤
≥ 0 . (35)
Therefore
h (x2)− h (x1)− ⟨∆h (x1) ,x2 − x1⟩ ≥ 0 , (36)
namely
h (x2) ≥ h (x1) + ⟨∆h (x1) ,x2 − x1⟩ . (37)
Now, we have proved that h (x) is convex. Thus, after training for certain steps with proper learning rate setting, we can obtain a ϕ
′∗ such that, for ϵ > 0,
∥∥∥xϕ′∗ −m ∥∥∥ 2 ≤ ϵ. It is easy to generalize
the proof above when taking into account the whole sequence, i.e. X ∈ RL×d.
A.2 GPU Memory Consumption The results of GPU memory consumption is presented in Table 3. A lower GPU memory consumption translates to reduced space complexity, which is a desirable trait.
A.3 Illustration of the PosNet The illustration of applying the PosNet to Transformer is presented in Figure 5.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
7 A2. Did you discuss any potential risks of your work? no risks
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract,1
3 A4. Have you used AI writing assistants when working on this paper? quillbot, a system providing writing assistance, check grammar mistakes and typo, 1-8
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? datasets and tools used in this paper are widely used and publicly available
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? datasets and tools used in this paper are widely used and publicly available
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? datasets and tools used in this paper don’t include private information or offensive content
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? datasets and tools used in this paper are widely used and publicly available, detailed documentations are available on the corresponding website.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we aim to explore alternative operations that implicitly encode positional information based on word order, without resorting to affine transformations, to replace the weight sum operation of the attention module