ENCODING WORD ORDER IN COMPLEX EMBEDDINGS
Year: 2020

Abstract:
Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions is extended to complex-valued domain due to richer representations. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available). Experiments 1 on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complexvalued representations to concrete meanings (i.e., word order).

heading: 1 INTRODUCTION
text: When processing text, the sequential structure of language is important, but can be computationally costly to model with neural networks (NNs) (Socher et al., 2011) due to the difficulty in parallelization. This has been alleviated by modeling word sequence not on the NN architecture level, but by adding position embeddings on the feature level. This has been done by the convolutional sequence model (ConvSeq) (Gehring et al., 2017) and the Transformer model (Vaswani et al., 2017) that replaces recurrent and convolution operations with purely attention mechanisms. More generally, vanilla position embeddings (Gehring et al., 2017) assume that individual word positions are independent and do not consider relations between neighbouring word positions. We posit that both the global absolute positions of words and their inner sequential and adjacent relationships are crucial in language. This is supported by recent empirical findings by Shaw et al. (2018) and Dai et al. (2019) who show the importance of modeling distance between sequential elements, and explicitly use extra relative position encodings to capture the relative-distance relationship of words.
We present a novel and principled approach to model both the global absolute positions of words and their inner sequential and adjacent relationships as follows: we extend each word embedding, previously defined as an independent vector, as a continuous function over an independent variable i.e., position. The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions
∗First two authors contributed equally. †Corresponding author: pzhang@tju.edu.cn 1The code is on https://github.com/iclr-complex-order/complex-order
𝑟1
𝑟2 𝑟3
𝑝2 𝑝1
𝑝3
Figure 1: 3-dimensional complex embedding for a single word in different positions. The three wave functions (setting the initial phases as zero) show the real part of the embedding; the imaginary part has a π 2 phase difference and shows the same curves with its real-valued counterpart. The x-axis denotes the absolute position of a word and the y-axis denotes the value of each element in its word vector. Colours mark different dimensions of the embedding. The three cross points between the functions and each vertical line (corresponding to a specific position pos) represent the embedding for this word in the pos-th position.
can correlate with each other in a continuous function. Fig. 1 illustrates this type of word representation with a three-dimensional complex-valued embedding, where the amplitudes {r1, r2, r3} denote semantic aspects corresponding to classical word vectors, and periods {p1, p2, p3} denote how sensitive the word is to positional information. We further discuss the necessary properties of these functions to model sequential information and obtain a general solution in the form of a complexvalued embedding. Interestingly, there is a direct connection between a specific case of our general embedding and the well-known positional encoding in Vaswani et al. (2017) (see App. A).
We contribute (i) a novel paradigm that extends word vectors as continuous functions over changing variables like word position, and (ii) a general word embedding that models word order in a mathematically-sound manner. We integrate our complex word embeddings in state-of-the-art (SOTA) NN architectures (CNN, RNN, Transformer and experimentally find that it yields gains over both classical word embeddings and position-enriched word embeddings in text classification, machine translation and language modeling. Note that this is the first work in NLP to link imaginary numbers in complex-valued representation to concrete meanings (i.e., word order).

heading: 2 MODELLING WORD ORDER IN EMBEDDING SPACE
text: A Word Embedding (WE) generally defines a map fwe : N → RD from a discrete word index to a D-dimensional real-valued vector and N = {0, 1, 2, . . .}. Similarly, a Position Embedding (PE) (Gehring et al., 2017; Vaswani et al., 2017) defines another map fpe : N → RD from a discrete position index to a vector. The final embedding for word wj (wj ∈ W with index j in a given vocabulary W) in the pos-th position in a sentence is usually constructed by the sum
f(j, pos) = fwe(j) + fpe(pos), (1)
and f(j, pos) ∈ RD. Since both the word embedding map fw and the position embedding map fp only take integer values as word indexes or position indexes, embedding vectors for individual words or positions are trained independently. The independent training for each word vector is reasonable, since a word index is based on the order of a given arbitrary vocabulary and does not capture any specific sequential relationship with its neighboring words. However, the position index captures an ordered relationship, for instance adjacency or precedence, leading to the problem that position embeddings in individual positions (Gehring et al., 2017) are independent of each other; the ordered relationship between positions is not modelled. We refer to this as the position independence problem. This problem becomes more crucial when position embeddings are used in position-insensitive NNs, e.g., FastText (Mikolov et al., 2013b), ConvSeq (Gehring et al., 2017) and Transformer (Vaswani et al., 2017), because it is hard for such position-insensitive NNs with vanilla position embeddings (Gehring et al., 2017) to infer that wj1 in the pos-th position is close to wj2 in the pos + 1-th position, or that wj1 precedes wj2 ; instead, it is only inferred that wj1 and wj2 are in different positions, while the relative distance between them is almost unknown. Thus vanilla position embeddings (Gehring et al., 2017) cannot fully capture the sequential aspect of language.
Next, we first introduce the necessary properties to model word order in embeddings, and then give a unique solution to meet such properties.

heading: 2.1 EXTENDING VECTORS TO FUNCTIONS
text: In the general definition in Eq. 1, each dimension of the position embedding is obtained based on the discrete position indexes {0, 1, 2, ..., pos, ...}. This makes it difficult to model the ordered relationship between the positions. One solution to this problem is to build continuous functions over a variable (i.e., position index) to represent a specific word in an individual dimension. Formally, we define a general embedding as
f(j, pos) = gj(pos) ∈ RD, (2) where gj is short for gwe(j) ∈ (F)D, indicating D functions over position index pos, and gwe(·) : N → (F)D is a mapping from a word index to D functions. By expanding the D dimension of gj , a word wj in the pos-th position can be represented as a D-dimensional vector as shown in
[gj,1(pos), gj,2(pos), ..., gj,D(pos)] ∈ RD, (3) in which ∀gj,d(·) ∈ F : N→ R, d ∈ {1, 2, ..., D} is a function over the position index pos. To move the word wj from the current position pos to another one pos′, it needs only replace the variable pos to pos′ without changing gj .
Functions for words, especially continuous functions, are expected to capture smooth transformation from a position to its adjacent position therefore modeling word order. The position-independent position embedding (Gehring et al., 2017) can be considered as a special case of our definition when it only takes independent values for individual positions in the embedding function.

heading: 2.2 PROPERTIES FOR THE FUNCTIONS TO CAPTURE WORD ORDER
text: Relative distance is hard to compute because position indices are not visible in NNs after vector embedding (discrete position indices are necessarily embedded as vectors like words to be backpropagated with the gradient). Hence, we claim that the modeling of relative distance in NNs should be position-free: absolute position indices cannot be directly accessed in intermediate layers. Instead of processing position-free operations in NNs to capture relative distance between words, prior work (Shaw et al., 2018; Dai et al., 2019) first calculates the relative distance between words, and then feeds the relative distance as an additional feature or as embeddings/weights to NNs, instead of directly feeding with the raw position indices.
Assume that words are embedded into RD, and let, for 1 ≤ d ≤ D, the function gj,d : N→ R be the embedding function giving the d-th coordinate of the representation of word wj (i.e., gj,d(pos) is the d-th coordinate of the embedding ofwj if it occurs at position pos. In the following, we simply write g instead of gj,d when there is no risk of confusion. Ideally, one would like there to exist a function Transformn : R → R that transforms the embedding of any word at some position pos to the embedding of a word at position pos + n such that Transformn is only dependent on the embedded value itself, but independent of the position pos, that is ∀pos : g(pos + n) = Transformn(g(pos)). Prior work in NLP (Li et al., 2019), Information Retrieval (Van Rijsbergen, 2004) and Machine Learning (Trabelsi et al., 2017) has shown the usefulness of complex numbers as richer representations. Complex word embeddings (Wang et al., 2019; Li et al., 2019; Li et al., 2018) have been used to model language. To investigate the potential of complex-valued representation, we extend the target domains of g(·) from RD to CD without losing generality, since real-valued numbers are specific complex numbers with their imaginary part being zero. This property regarding “position-free offset transformation” in complex-valued domains is formally defined in Property 1 below.
Property 1. Position-free offset transformation: An embedding function g : N → C is said to be a position-free offset transformation if there exists a function Transform : N × C → C (called the witness) such that for all n ≥ 1, the function Transformn(·) = Transform(n, ·) satisfies ∀pos ∈ N : g(pos+n) = Transformn(g(pos)). A position-free offset transformation g is said to be linearly witnessed if there is a function w : N→ C such that g has a witness Transform satisfying, for all n, Transform(n, pos) = Transformn(pos) = w(n) (i.e., each Transformn is a linear function).
Additionally, a boundedness property is necessary to ensure that the position embedding can deal with text of any length (pos could be large in a long document).
Property 2. Boundedness: The function over the variable position should be bounded, i.e. ∃δ ∈ R+,∀pos ∈ N, |g(pos)| ≤ δ.
Formally, we prove the following claim that there is a unique solution that meets Properties 1 and 2 under the condition that the embedding function is linearly witnessed. We use linear functions because they are well-understood and simple with a single floating-point operation in NNs.
Claim 1. A function g : N → C is a bounded and linearly witnessed position-free offset transformation iff it is on the form g(pos) = z2z pos 1 for z1, z2 ∈ C with |z1| ≤ 1.
Proof. Assume that g is a bounded and linearly witnessed position-free offset transformation. Then, by linear witnessing, we have for all pos, n1, n2 ∈ N:
w(n1)w(n2)g(pos) = w(n2)g(pos + n1) = g(pos + n1 + n2) = Transformn1+n2(g(pos)) = w(n1 + n2)g(pos)
whencew(n1+n2) = w(n1)w(n2). Writew(1) = z1 and g(0) = z2. As n1, n2 ∈ N were arbitrary, we have w(n) = (w(1))n = zn1 for all n ∈ N. But then g(pos + n) = w(n)g(pos) = zn1 g(pos). Furthermore, observe that for pos ≥ 1, we have g(pos) = g(1+pos−1) = w(pos)g(0) = zpos1 z2 = z2z pos 1 . For pos = 0, g(0) = z2 = z2z 0 1 , whence g(pos) = z2z pos 1 , as desired. Observe that if |z1| > 1, then g(pos) is unbounded, whence we have |z1| ≤ 1. Conversely, assume that g is on the form g(pos) = z2z pos 1 with |z1| ≤ 1. Then, |g(pos)| ≤ |z2z pos 1 | ≤ |z2||z pos 1 | ≤ |z2|, whence g is bounded. Define, for each n ∈ N, w(n) = zn1 and Transformn(pos) = w(n)pos. Then, for all pos, n ∈ N,
g(pos + n) = z2z pos+n 1 = z2z pos 1 z n 1 = g(pos)z n 1 = Transformn(g(pos))
showing that g is a linearly witnessed position-free offset transformation.
For any z ∈ C, we may write z = reiθ = r(cos θ + i sin θ). Thus, for the general form of the embedding g from Theorem 1, we have:
g(pos) = z2z pos 1 = r2e iθ2(r1e iθ1)pos = r2r pos 1 e i(θ2+θ1pos) subject to |r1| ≤ 1 (4)
In implementations, the above definition of g will lead to an optimization problem due to the constraint |r1| ≤ 1. A natural and simple way to avoid this is to fix r1 = 1; note that |eix| ≡ 1, thus automatically satisfying the constraint, in contrast to a real-valued embedding where one would need to explicitly devise functions satisfying the constraint. Finally, Eq. 4 can be written in the simplified form: g(pos) = rei(ωpos+θ). Thus, one can think of g as embedding positions counterclockwise on a complex circle of radius r with a fixed period (r is the amplitude term, θ is the initial phase term, ω 2π is the frequency, and 2π ω is the period term).

heading: 2.3 COMPLEX-VALUED WORD EMBEDDING
text: We now define our complex-valued word embedding g as a map taking a word index j and position word index pos to CD. For a word wj in position pos, our general complex-valued embedding is defined as f(j, pos) = gj(pos) = rjei(ωjpos+θj). Therefore, f(j, pos) is defined as:
[rj,1e i(ωj,1pos+θj,1), ..., rj,2e i(ωj,2pos+θj,2), · · · , rj,Dei(ωj,Dpos+θj,D)] (5)
Note that each coordinate d (1 ≤ d ≤ D) has a separate amplitude rj,d, period pj,d = 2πωj,d , and initial phase θj,d. In Fig. 1 each dimension is represented as a wave which is parameterized by an amplitude, a period/frequency, and an initial phase. The trainable parameters of the embedding are the amplitudes vector rj = [rj,1, ..., rj,D], the period/frequency related weights ωj = [ωj,1, ..., ωj,D], and the initial phase vector θj = [θj,1, ..., θj,D]. Note that the mean values of f(j, ·) over all positions are linearly dependent on the amplitude. Observe that the period/frequency determines to what degree the word is sensitive to the position. With an extremely long period (i.e., ωj very small), the complex-valued embedding is approximately constant for all possible values of pos, and hence approximates a standard word embedding. Conversely, if the period is short, the embedding will be highly sensitive to the position argument.
In our embedding, the mean vectors of f(j, ·) taken over all positions are linearly correlated to the amplitude embedding rj = [rj,1, ..., rj,K ] with a coefficient 2π . The amplitude rj,d of our embedding depends only on the word wj (and coordinate d), not on the position of the word, whence one can
think of the vector gpe(j, pos) = [ei(ωj,1pos+θj,1), · · · , ei(ωj,Dpos+θj,D)] as a “purely” positional embedding. Consequently, our complex embedding can be considered an element-wise multiplication between the word embedding gwe(j) = [rj,1, ..., rj,K ] and position embedding gpe.
f(j, pos) = gwe(j) gpe(j, pos) (6)
Prior work (Gehring et al., 2017; Vaswani et al., 2017) uses mean-weight addition between word embeddings fwe and position embeddings fpe (all words share the weights). In our work, word embeddings and position embeddings are decoupled to some extent by element-wise multiplication and therefore the frequency/period terms (related to ωj,d) can adaptively adjust the importance between semantic and position information for each word and each dimension. In particular, with higher frequency (i.e., large ωj,d), the final embedding will change dramatically with the changing positions, while it can be fixed for any positions with an extremely-small frequency (i.e., small ωj,d). Interestingly, the well-known position embedding in Transformer (Vaswani et al., 2017) can be seen as a degraded version of one of our specific complex word embeddings (see the Appendix A).

heading: 3 EXPERIMENTAL EVALUATION
text: We evaluate our embeddings in text classification, machine translation and language modeling.

heading: 3.1 TEXT CLASSIFICATION
text: Experimental Setup. We use six popular text classification datasets: CR, MPQA, SUBJ, MR, SST, and TREC (see Tab. 1). We use accuracy as evaluation measure based on fixed train/dev/test splits or cross validation, as per prior work. We use Fasttext (Joulin et al., 2016), CNN (Kim, 2014), LSTM and Transformer (Vaswani et al., 2017) as NN baselines2. We use each of them: (1) without positional information; (2) with Vanilla Position Embeddings (PE) (randomly initialized and updated during training using the sum between word and position vectors (Gehring et al., 2017); (3) with Trigonometric Position Embeddings (TPE) (defining position embeddings as trigonometric functions as per Eq. 7); (4) with Complex-vanilla word embeddings (where the amplitude embedding is initialized by the pre-trained word vectors, and the phrase embedding is randomly initialized in a range from −π to π without considering word order (Wang et al., 2019)); and (5) with our order-aware complex-valued word embeddings, Complex-order (which encode position in the phase parts, train the periods, and where the amplitude embedding is also initialized by pretrained word vectors). For more details on the complex-valued extensions of NNs, see App. B and App. C.
Our embedding generally has 3 × D × |W| parameters with D-dimensional word vectors and |W| words, while previous work (Mikolov et al., 2013b; Pennington et al., 2014) usually employs only D × |W| parameters for embedding lookup tables. To increase efficiency and facilitate fair comparison with previous work we set initial phases θj = [θj,1, ..., θj,D] to a shared constant value (such as zero). Furthermore, the period vectors ωj,d depend on word index j with length |W| and the coordinate index d with length D. To decrease the number of parameters, one can either use a word-sharing scheme (i.e., ωj,d = ω·,d), or a dimension-sharing scheme (ωj,d = ωj,·), leading to |W| ∗D + |W| and |W| ∗D +D parameters in total for the embedding layer.
2Graph convolutional networks (GCNs) (Beck et al., 2018; Sahu et al., 2019) also encode positional information. We do not compare against them because they encode positional information inherently as part of the model, which makes redundant any additional encoding of positional information at the embedding level.
We search the hyper parameters from a parameter pool, with batch size in {32, 64, 128}, learning rate in {0.001, 0.0001, 0.00001}, L2-regularization rate in {0, 0.001, 0.0001}, and number of hidden layer units in {120, 128}. We use pre-trained 300-dimensional vectors from word2vec (Mikolov et al., 2013a) in all models except for Transformers. The models with trainable trigonometric position embedding produce nearly identical results compared to the non-trainable version, therefore we report the result of fixed position embeddings as per Vaswani et al. (2017). We adopt narrow convolution and max pooling in CNN, with number of filters in {64, 128}, and size of filters in {3, 4, 5}. In all Transformer models, we only use the encoder layer to extract feature information, where the layer is 1, dimension of word and inner hidden are 256 and 512 respectively, and head number is 8.
Results. The results are shown in Tab. 2. Our complex-order embeddings outperform all other variations at all times. This gain in effectiveness comes at a negligible (or non-existent) cost in efficiency (it varies per NN architecture – see Fig. 2). CNNs are the best performing NN as expected following Bai et al. (2018). Tranformer NNs benefit the most from our complex-order embeddings, most likely because they are our weakest baseline. To contextualise these results, Tab. 3 shows classification accuracy of five typical approaches on the same datasets (as reported in the original papers). Our complex-order embeddings outperform all methods, except for the CR dataset, where InferSent is marginally better. Overall, our approach is on a par with the SOTA in embeddings.
We perform an ablation test (Tab. 4) on Transformer because it is the most common NN to be used with position embeddings. The two period-sharing schemas (dimension-sharing and word-sharing)
slightly drop performance, because fewer parameters limit the representative power. Adding initial phases also hurts performance, although we observed that the loss could decrease faster in early epochs compared to the setting without offset. The negative effect of initial phases may be due to periodicity, andω cannot be directly regularized with L2-norm penalties. The sharing schemes slightly decrease the performance with less parameters. More details of the learned periods/frequencies (e.g. the distributions of periods/frequencies and case studies) are shown in App. D.
Note that the word-sharing schema outperform the Vanilla Transformer, (both have a comparable number of parameters). If we choose<(WQ/K/V ) = =(WQ/K/V ), the additional parameters in the embedding layers will affect much less the whole parameter scale in the multiple-layer Transformer, since a embedding layer is only used in the first layer instead of the following Transformer layers.

heading: 3.2 MACHINE TRANSLATION
text: Experimental Setup. We use the standard WMT 2016 English-German dataset (Sennrich et al., 2016), whose training set consists of 29,000 sentence pairs. We use four baselines: basic Attentional encoder-decoder (AED) (Bahdanau et al., 2014); AED with Byte-pair encoding (BPE) subword segmentation for open-vocabulary translation (Sennrich et al., 2016); AED with extra linguistic features (morphological, part-of-speech, and syntactic dependency labels) (Sennrich & Haddow, 2016); and a 6-layer Transformer. Our approach (Transformer Complex-order) uses a batch size of 64, a head of 8, 6 layers, the rate of dropout is 0.1, and the dimension of the word embedding is 512. The embedding layer does not use initial phases, i.e., following f(j, pos) = rjei(ωjpos). We evaluate MT performance with the Bilingual Evaluation Understudy (BLEU) measure.
Results Tab. 5 shows the MT results. Our approach outperforms all baselines. Two things are worth noting: (1) Both the vanilla Transformer and our Transformer Complex-order outperform the three Attentional encoder-decoder baselines which are based on an LSTM encoder and decoder, even when AED uses additional features. (2) Our Transformer Complex-Order outperforms the Vanilla Transformer and complex-vanilla Transformer by 1.3 and 1.1 in absolute BLEU score respectively.

heading: 3.3 LANGUAGE MODELING
text: Experimental Setup. We use the text8 (Mahoney, 2011) dataset, consisting of English Wikipedia articles. The text is lowercased from a to z, and space. The dataset contains 100M characters (90M for training, 5M for dev, and 5M for testing, as per Mikolov et al. (2012)). We use as baselines BN-LSTM, LN HM-LSTM RHN and Large mLSTM, which are typical recurrent NNs for language modeling in this dataset. We evaluate performance with the Bits Per Character (BPC) measure, (the lower, the better). We run the coder in Dai et al. (2019) with 6 layers for Transformer XL 6L; our model, named Transformer XL complex-order, directly replaces the word embedding with our proposed embedding under the same setting. We choose 6 layers due to limitations in computing resources. For Transformer XL Complex-order, all other parameter settings are as for Transformer XL (Dai et al., 2019). Our complex-order model does not use initial phases.
Results. We see in Tab. 6 that our method outperforms all baselines. The first four baselines (BNLSTM, LN HM-LSTM, RHN and Large mLSTM) are based on recurrent NNs and rely on different regulation methods to become applicable in multiple-layer recurrent architectures. Transformerbased architectures can easily be stacked with multiple layers due to their advantages in parallelization, however the vanilla Transformer does not outperform the multiple-layer recurrent baselines, most likely due to its limitation of 6 layers. Our Transformer XL Complex-order outperforms its vanilla counterpart under the 6-layer setting (and also strong recurrent network baselines), demonstrating that our embedding also generalizes well in tasks with long-term dependency. With limited resources, slightly increasing the parameters in the feature layer like our proposed embedding could be more beneficial than stacking more layers with linearly increasing parameters.

heading: 4 RELATED WORK
text: Complex-valued NNs are not new (Georgiou & Koutsougeras, 1992; Kim & Adalı, 2003; Hirose, 2003). Complex-valued weights have been used in NNs, motivated by biology (Reichert & Serre, 2013), and also as signal processing in speech recognition (Shi et al., 2006). More recently, Arjovsky et al. (2016) shifted RNNs into the complex domain and Wolter & Yao (2018) proposed a novel complex gated recurrent cell. Trabelsi et al. (2017) also developped a complex-valued NN for computer vision and audio processing.
Complex numbers have also been applied to text processing like (Van Rijsbergen, 2004; Melucci, 2015; Blacoe et al., 2013). Trouillon et al. (2016) adopt complex embedding for entities in Knowledge Graph Completion to represent antisymmetric relations with Hermitian dot product. Li et al. (2019); Wang et al. (2019) extend word embeddings to complex-valued fashion in quantum probability driven NNs, seeing the overview in Wang et al. (2019). However, the physical meaning of both the complex-valued entity and word embeddings is unknown, since a complex number was considered as two real numbers in a black-box learning paradigm. Our work first links the phase in complex numbers to word position to define concrete physical meaning in document representations.

heading: 5 CONCLUSIONS
text: We extended word vectors to word functions with a variable i.e. position, to model the smooth shift among sequential word positions and therefore implicitly capture relative distances between words. These functions are well-defined to model the relative distances, therefore we derive a general solution in complex-valued fashion. Interestingly, the position embedding in Vaswani et al. (2017) can be considered a simplified version of our approach. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding. Experiments on text classification, machine translation and language modeling show that our embeddings are more effective than vanilla position embeddings (Vaswani et al., 2017).

heading: ACKNOWLEDGMENTS
text: We thank Massimo Melucci and Emanuele Di Buccio for their helpful comments, Xindian Ma for his detailed experimental suggestions.
This work is supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union‘s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321. Peng Zhang and Donghao Zhang are supported in part by Natural Science Foundation of China (grant No. 61772363, U1636203)

heading: A LINKING TO THE POSITION EMBEDDINGS IN (VASWANI ET AL., 2017)
text: Vaswani et al. (2017) proposed a new initialization for position embedding, resulting in comparable performance with previous one (Gehring et al., 2017) even without fine-tuning. The position embedding is empirically selected as
PE2k(·, pos) = sin(pos/100002k/dmodel) PE2k+1(·, pos) = cos(pos/100002k/dmodel)
(7)
Where pos is the position index, 2k and 2k + 1 is the dimension index and dmodel is the dimension size of embedding. The reason for choosing this position embedding was not well-explained and its general extension is unknown, leading to some difficulties to improve it.
We claim that the proposed position embedding in (Vaswani et al., 2017) is a degraded version of one of our specific complex word embedding in word-sharing schema (i.e., ωj,d = ω·,d), in which pj,k = 2π × 100002k/dmodel and the initial phases are set as zero. In our complex-valued position embedding, let fpe,k(·, pos) = ei×10000
2k/dmodel = cos(100002k/dmodelpos) + i sin(100002k/dmodelpos). Note that there exists a bi-jection between PE(·, pos) and fpe,k(·, pos):
PE2k(·, pos) = =(fpe,k(·, pos)), PE2k+1(·, pos) = <(fpe,k(·, pos))
(8)
where < and = are the operations to take the real and imaginary part of a complex-valued number. Its inverse transformation is
fpe,k(·, pos) = PE2k+1(·, pos) + iPE2k(·, pos) (9)
In our overall embedding, each dimension fk(j, pos) = fwe,k(j) fpe,k(·, pos) in our approaches, while it is Ek(j, pos) = WEk(j) + PEk(·, pos) in (Gehring et al., 2017; Vaswani et al., 2017). Hence the position embedding in (Vaswani et al., 2017) is equivalent, albeit not identical, to our complex-valued position embedding with pj,k = 2π × 100002k/dmodel . It is therefore a particular case of our complex-valued position embedding, the word-sharing schema in which all words share the same period at a certain dimension, i.e, pj,k = p·,k is irrelevant to the choice of j.
B INTEGRATING COMPLEX-VALUED EMBEDDING TO GENERAL NEURAL NETWORKS
Neural networks are typically given real numbers as inputs and return real numbers as outputs. To accommodate complex numbers as in- and output, we devise a complex-valued version of various neural network layers i.e. complex-valued FastText with dense layer, CNN, and RNN. Unlike existing complex-valued neural networks (Trabelsi et al., 2017; Wolter & Yao, 2018), our feature layers are also converted into complex-valued layers.
Complex-valued FastText FastTest (Joulin et al., 2016) is a simple and efficient neural network architecture using a dense layer over the sum of all word embeddings for general text classification. For a linear dense layer, i.e., z = dense(x + iy), where x + iy and z denote the complex-valued in- and output, respectively. Let W = A + iB and b = c + id be complex-valued linear weights and bias, respectively. Then, the complex-valued dense layer is given by:
z = σ (Ax−By + c) + iσ(Bx+Ay + d) (10)
where σ is a real-valued activation function such as the sigmoid function. By rewriting (10) in matrix form (Trabelsi et al., 2017), we obtain:[
<(z) =(z)
] = [ σ(Ax−By + c) σ(Bx+Ay + d) ] (11)
where, for z = x + iy, <(z) = x and =(z) = y. To save parameters and fairly compare with our real-valued baselines, the weights for real-part and imaginary-part input can be shared, i.e., A = B, c = d.
Complex-valued CNN For the complex-valued version of the convolution operation Trabelsi et al. (2017), we similarly define a complex-valued convolution with separate real and imaginary kernels A and B, to compute convolutions on the real and imaginary parts of the input in Eq. 10. A complex-valued CNN network is constructed by stacking the operations based on a complex-valued convolution kernel, adding a complex dense layer in (10), and taking their norm as the final prediction in the last layer.
Complex-valued RNN The basic complex RNN formulation is: hCt = f ( W hht−1 +W zzt + b )
(12)
where zt and ht represent the complex-valued input and complex-value hidden state vectors at time t, b is a complex-valued bias, W h and W z are complex-valued weight transitions for hidden state and input state, and f (z) = σ (< (z)) + iσ (= (z)) is the activation function. The multiplication W hht−1 and W zzt is computed as defined in (10) above. Similarly, the complex-valued gates are used in LSTM via operations as in (12). In the final layer, a l2-norm operation is adopted to obtain a real-valued loss for backpropagation.
Complex-valued Transformer The main components in the Transformer are self-attention sublayers and position-wise feed-forward (FFN) sublayers. A self-attention sublayer employs h attention heads and the concatenation of all heads is used as the output followed by a parameterized linear transformation. For a sequence embedded as complex-valued vector input = {w1,w2, ...,wn}, the output of each head is computed as a weighted sum of a linear transformation of the input sequence itself, namely
outputi = ∑ j ai,jwjW V , (13)
where wj is a complex-valued vector and W V is a complex linear transformation; therefore outputi is also complex. Hence, output is a sequence of complex-valued vectors with the same shape as input. The weight coefficient, ai,j , which is defined as in real-valued domain, is calculated as the softmax of the product between complex-valued query vectors and key vectors: ai,j = softmax
ei,j∑n i=1 ei,j , and
ei,j =
√ <(z)2 + =(z)2
n , z =
( wiW Q ) ( wjW K )† , (14)
z is a complex number since wi and wj are complex-valued vectors and WQ,WK are complexvalued transformation. To extend another variant of Transformer called Transformer XL, we keep its original relative position embedding and additionally replace its word embedding with our proposed embedding.
Correspondingly, the FFN sublayer can easily be extended to a complex-valued version by replacing the real-valued layers with complex-valued ones. We use batch-normalization separately for the real and imaginary parts.
C IMPLEMENTATION OF THE PROPOSED EMBEDDING
Words functions are implemented in neural networks by storing the function parameters {r,ω,θ} and then construct the values based on the arguments. Based on the definition, the implementation of the proposed embedding can easily be implemented with only modifying the embedding layer. We list the basic code to construct our general embedding as below:
import t o r c h import math c l a s s ComplexNN ( t o r c h . nn . Module ) :
def i n i t ( s e l f , o p t ) : super ( ComplexNN , s e l f ) . i n i t ( ) s e l f . word emb = t o r c h . nn . Embedding ( o p t . n token , o p t . d model ) s e l f . f r equency emb = t o r c h . nn . Embedding ( o p t . n token , o p t . d model ) s e l f . i n i t i a l p h a s e e m b = t o r c h . nn . Embedding ( o p t . n token , o p t . d model )
def g e t e m b e d d i n g ( s e l f , x ) :
a m p l i t u d e = s e l f . word emb ( x ) f r e q u e n c y = s e l f . f r equency emb ( x ) s e l f . i n i t i a l p h a s e e m b . we ig h t = t o r c h . nn . P a r a m e t e r ( s e l f . i n i t i a l p h a s e e m b . we ig h t
% (2 ∗ math . p i ) )
s e n t l e n =x . s i z e (−1) p o s s e q = t o r c h . a r a n g e ( 1 , s e n t l e n + 1 , 1 . 0 , d e v i c e = a m p l i t u d e . d e v i c e )
p o s s e q = p o s s e q . unsqueeze ( 0 ) . unsqueeze (−1) p o s s e q = p o s s e q . r e p e a t ( [ x . s i z e ( 0 ) , 1 , a m p l i t u d e . s i z e ( −1) ] )
d i m e n s i o n b a i s = s e l f . i n i t i a l p h a s e e m b ( x )
e n c o u t p u t p h a s e = t o r c h . mul ( pos seq , f r e q u e n c y )+ d i m e n s i o n b a i s e n c o u t p u t r e a l = a m p l i t u d e ∗ t o r c h . cos ( e n c o u t p u t p h a s e ) e n c o u t p u t i m a g e = a m p l i t u d e ∗ t o r c h . s i n ( e n c o u t p u t p h a s e ) # r e t u r n t o r c h . c a t ( [ e n c o u t p u t r e a l , e n c o u t p u t i m a g e ] ,−1) re turn e n c o u t p u t r e a l , e n c o u t p u t i m a g e
def f o r w a r d ( s e l f , x ) : re turn s e l f . g e t e m b e d d i n g ( x ) . . .
Note that both the frequency vectors ω and initial-phase vectors θ can be shared between words or dimensions, to save parameters. The proposed embedding can be also used in real-valued neural networks if one directly concatenates the real-part numbers and imaginary-part numbers as a doublesize real-valued vector; therefore it could easily be extended in any existing networks without any complex-valued components. For instance, it could be a good extension for Transformer based pretrained models like (Devlin et al., 2018) by enriching the feature layer.
D VISUALIZATION OF FREQUENCIES/PERIODS
After training, we obtain the frequency vectorω for each word. For each word, the mean value of the absolute frequency values, i.e., δj = 1|D| ∑D d=1 |ωj,d| is considered as a metric to test the positional sensitivities of the word, since a period value could be negative during training. The density of the δj is shown in Fig. 3.
Words with the 50 greatest, and the 50 smallest, frequencies in the SST dataset are shown in Tab. 7. For the words with greatest frequencies, most of them are strong sentiment words like “worst”
,“stupid” and “powerful”; a reason for this may be that such words appear in many positions in many documents during training, and thus they are more sensitive to the positions. Conversely, there are fewer words expressing strong sentiment among words with smaller frequencies, as shown in the second row.

Review summaries:
This paper proposes to learn position varying embeddings of words using complex numbers. Specifically, this work learns a position embedding by learning a continuous function that respects relative position based constraints and is bounded. The authors show that complex representations are an ideal fit for this purpose wherein the amplitude of the complex wave represents the base word embedding that is positionally invariants and the "wave" part encodes encodes the evolution of each dimension with position  as a periodic function with a learnable phase and period.