Encoding word order in complex embeddings
Authors: Benyou Wang
Donghao Zhao
Christina Lioma
Qiuchi Li
Peng Zhang
Jakob Grue Simonsen
Venue:  (2020)
Keywords: word embedding
complex-valued neural network
position embedding

Abstract: Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions can be extended to complex-valued variants. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available). Experiments on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complex-valued representations to concrete meanings (i.e., word order).

Published as a conference paper at ICLR 2020
ENCODING WORD ORDER IN COMPLEX EMBEDDINGS
Benyou Wang
Donghao Zhao
Christina Lioma
University of Padua it
Tianjin University zhaodhêt t juu.uu..ccu edu.cn
University of Copenhagen chrh@di. ku.dk
Qiuchi Li University of Padua
Peng Zhang
Jakob Grue Simonsen University of Copenhagen simonsenedi di ku. dk
Tianjin University pzhang@t ju. edu. on
qiuchili@dei. unipd. it
ABSTRACT
Sequential word order is important when processing text. Currently, neural net- works (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individ- ual word positions. We present novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solu- tion generalizes word embeddings, previously defined as independent vectors, continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can cor- relate with each other a continuous function. The general solution of these functions is extended to complex-value We extend CNN, RNN and Transformer NNs to complex-valued versions to in- corporate complex embedding (we make all code available). Experiments [T] on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complex- valued representations to concrete meanings (i.e., word order).
domain due to richer representations.
INTRODUCTION
When processing text, the sequential structure of language important, but can be computationally costly to model with neural networks (NNs) (Socher et al. (01T) due to the difficulty in paralleliza- tion. This has been alleviated by modeling word sequence not on the NN architecture level, but by adding position embeddings on the feature level. This has been done by the convolutional se- quence model (ConvSeeq) (Gehring et al. 2017) and the Transformer model (Vaswani et al. 2017) that replaces recurrent and convolution operations with purely attention mechanisms. More gener- ally, vanilla position embeddings (Gehring et al., 2017) assume that individual word positions are independent and do not consider relations between neighbouring word positions. We posit that both the global absolute positions of words and their inner sequential and adjacent relationships are cru- cial in language. This is supported by recent empirical findings by (Shaw et al. (2018) and Dai al. (2019) who show the importance of modeling distance between sequential elements, and explicitly use extra relative position encodings to capture the ellatve-distance relationship of words.
We present a novel and principled approach to model both the global absolute positions of words and their inner sequential and adjacent relationships as follows: we extend each word embedding, previously defined an independent vector, as a continuous function over an independent variable iee. position. The benefit of continuous functions over variable positions that word represen- tations shift smoothly with increasing positions. Hence, word representations in different positions
"Frstt two authors contributed equally.
1 'Corresponding author: pzhang@rju.cdie
'The code is httpp:////iitit.....ooaooooooopp
order

---

Published as a conference paper at ICLR 2020
Figure 1: 3-dimensional complex embedding for a single word in different positions. The three wave functions (setting the initial phases as zero) show the real part of the embedding; the imagi- nary part has 2 phase difference and shows the same curves with its real-valued counterpart. The x-axis denotes the absolute position of a word and the y-axis denotes the value of each element in its word vector. Colours mark different dimensions of the embedding. The three cross points between the functions and each vertical line (correspond- ing to specific position pos) represent the em- bedding for this word in the pos-th position.
P1
Ps
1/ 1
11
can correlate with each other in a continuous function. Fig. Jillustraae this type of word representa- tion with three-dimensional complex- valued embedding, where the amplitudes {r1, r2, denote semantic aspects corresponding to classical word vectors, and periods {P1. P2: Ps denote how sen- sitive the word is to positional information. We further discuss the necessary properties of these functions to model sequential information and obtain general solution in the form of complex- valued embedding. Interestingly, there is a direct connection between a specific case of our general embedding and the well-known positional encoding in Vaswani et al. (2017) (see A)
We contribute (i) a novel paradigm that extends word vectors as continuous functions over chang- ing variables like word position, and (ii) a general word embedding that models word order in a mathemattiaally-sonnd manner. We integrate our complex word embeddings in state-of-the-arr (SOTA) NN architectures (CNN, RNN, Transformer and experimentally find that it yields gains over both classical word embeddings and position-enrichee word embeddings in text classification, machine translation and language modeling. Note that this is the first work in NLP to link imaginary numbers in complex-valued representation to concrete meanings (i.e., word order).
2 MODELLING WORD ORDER IN EMBEDDING SPACE
Word Embedding (WE) generally defines a map fwe : N - TR from a discrete word index to D-dimensional real-valued vector and N = {0, 1. Similarly, Position Embedding (PE) Gehring et al., 2017 Vaswani et al. 2017) defines another map fpe N + DR from a discrete position index to vector. The final embedding for word Wy (wj W with index j in given vocabulary W) in the pos-th position in a sentence is usually constructed by the sum
(os) = fwe(() + {pos)
and fos) E R., Since both the word embedding map fw and the position embedding map fp only take integer values as word indexes position indexes, embedding vectors for individ- ual words or positions are trained independently. The independent training for each word vector reasonable, since a word index is based on the order of given arbitrary vocabulary and does not capture any specific sequential relationship with its neighboring words. However, the position index captures an ordered relationship, for instance adjacency or precedence, leading to the prob- lem that position embeddings in individual positions (Gehring et al. 2017) are independent of each other; the ordered relationship between positions not modelled. We refer this as the position independence problem. This problem becomes more crucial when position embeddings are used in position-insensittee insensitivee NNs, e.g., FastText (Mikolov et al. 20136), ConvSeq (Gehring et al. (017) and Transformer (Vaswani et al. (2017), because is hard for such resition-insensitiive NNs with vanilla position embeddings (Gehring et al.) 2017) to infer that wj, in the pos-th position is close to Wj, in the pos + 1-th position, or that w;; precedes W,z; instead, it is only inferred that Wj, and Wy, are in different positions, while the relative distance between them is almost unknown. Thus vanilla position embeddings (Gehring al. 2017) cannot fully capture the sequential aspect of language.
Next, we first introduce the necessary propertics to model word order in embeddings, and then give unique solution to meet such properties.
2

---

Published as a conference paper at ICLR 2020
2.1 EXTENDING VECTORS TO FUNCTIONS
In the general definition in Eq. I each dimension of the position embedding is obtained based on the discrete position indexes {0, :. This makes it difficult to model the ordered relationship between the positions. One solution to this problem is to build continuous functions over variable (i.e.. position index) to represent specific word in an individual dimension. Formally, we define general embedding as
pos) gp E R..
where gj is short for (i) (F)P, indicating D functions over position index pos, and Gwe (+) N (F) is mapping from word index to D functions. By expanding the D dimension of gj. a word wy in the pos-th position can be represented as a D-dimensional vector as shown in
19g,1 (pos), (pos) -.. (pos) E RR?,
which (gga(+) F + R E 21,2,... .... D) is function over the position index pos. To move the word wy from the current position pos to another one pos', it needs only replace the variable pos to pos' without changing 9,-
Functions for words, especially continuous functions, expected to capture smooth transformation from a position to its adjacent position therefore modeling word order. The position embedding (Gehring et al., 2017) can be considered as special case of our definition when it only takes independent values for individual positions in the embedding function.
2.2 PROPERTIES FOR THE FUNCTIONS TO CAPTURE WORD ORDER
Relative distance is hard to compute because position indices are not visible in NNs after vector embedding (discrete position indices are necessarily embedded as vectors like words to be back- propagated with the gradient). Hence, we claim that the modeling of relative distance in NNs should be position-frre: absolute position indices cannot be directly accessed in intermediate layers. Instead of processing position-frre operations in NNs capture relative distance between words, prior work (Shaw et al. 2018 Dai et al. 2019) first calculates the relative distance between words, and then feeds the relative distance as an additional feature or as embeddingss/weihht to NNs, instead of directly feeding with the raw position indices.
Assume that words are embedded into RD,, and let, for S d D, the function 9j.d : -> R be the embedding function giving the d-th coordinate of the representation of word Wy (i.e., ( is the d-th coordinate of the embedding of w/ if it occurs at position pos. In the following, we simply write g instead of Jj.a when there is no risk of confusion. Ideally, one would like there to exist a function Transform, : R + F' that transforms the embedding of any word at some position pos to the embedding of word at position pos such that Transform, is only dependent on the embedded value itself, but independent of the position pos, that is Vppos g(pos + n) = Transform, (g (poo))
Prior work in NLP (Li et al. 2019) Information Retrieval (Van Rijsbergen 2004) and Machine Learning Trabelsi et al. 2017 7) has shown the usefulness of complex numbers as richer representa- tions. Complex word embeddings (Wang et al., 2019 Li Li al. 2019 Li et (018)) have been used to model language. To investigate the potential of complex-valued representation, we extend the target domains of (-() from R to CP without losing generality, since real-valued numbers are spe- cific complex numbers with their imaginary part being zero. This property regarding "position- free offset transformation" in complex-vail valued domains is formally defined in Property [ below.
Property 1. Position-free offsst transformation: An embedding function g N + is said to (called
be a position--ree offset transformation if there exists a function Transform N C the witness) such that for all n 2 1, the function Transform, (-) Transform (n. satisfies Vpos E N g(pos + n1) = Transform,, (g(pos)). A position-free offset transformation g is said to be linearly witnessed if there is function w N - C such that y has witness Transform satisfying, for all n, Transform(1e (n. pos) = Transform, (pos) = w(n) (i.e., each Transform, is a linear function).
Additionally, a boundedness property is necessary to ensure that the position embedding can deal with text of any length (pos could be large in a long document).
Property 2. Boundedness: The function over the variable position should be bounded, i.e. 38 E R+, Vpos N, [g(po))] & 6.
3

---

Published as a conference paper at ICLR 2020
Formally, we prove the following claim that there is a unique solution that meets Properties I and 2 under the condition that the embedding function is linearly witnessed. We use linear functions because they well-understood and simple with single floating-pon -oint operation in NNs.
Claim 1. A function y N + C is a bounded and linearly witnessed position-free offset transfor- mation iff it is on the form g(pos) = =2211 for 21, 22 E with [21| S 1.
Proof. Assume that is bounded and linearly witnessed position-free offset transformation. Then, by linear witnessing, we have for all pos, R2 E N:
(((p)))(n)))(pps) w: ((22)g(po) + m1) = g(pos + n1 + m2) -y (g((s))) w((n + (pos)
whence w(n1 +m2) wwn))(nnuu (n2). Write w(1) = 21 and g(0) 22. As n1, n2 E N were arbitrary, we have w(n) = (w(1))* = 21 for all n E N. But then g(pos + n) = w(n)g(pos) = 2 (pos). Furthermore, observe that for pos 1, we have g(pos) g p-spppossp - 1) ((po))g(0)
2 22 =221 pos For pos = 0, g(0) 22 = =22.. whence (pos)) = =22 pos as desired. Observe that if |ai| > 1, then (pos) is unbounded, whence we have |21| < 1. Conversely, assume that y is on the form gppos) = with |21| S L. Then, [p(os]] S [2221 S |221lz] S [2gl. whence
is bounded. Define, for each n E N, w(n) = o and Transform, (pos) = w(n)pos. Then, for all pos, N,
g(pos + n) = 2 pos-mn pos =1 = =pos)= Transform, (Tansform,(f(ppos))
2
showing that g is a linearly witnessed position- free offset transformation.
For any C, we may write re:0 r(cos 0 isi 0). Thus, for the general form of the embedding from Theorem I we have:
g(pos) 2 pus r2e (r1e jou- = 122 (=00= O pos) subject to |na| S
(4)
In implementations, the above definition of will lead to optimization problem due to the con- straint |r1] S 1. A natural and simple way to avoid this is to fix "1 = 1; note that |e**| = 1, thus automatically satisfying the constraint, in contrast to real-valued embedding where one would need to explicitly devise functions satisfying the constraint. Finally, Eq. c4an be written in the simplified form: g(po) = to) Thus, one can think of g as embedding positions counterclockwis. on complex circle of radius with fixed period (r is the amplitude term, 0 is the initial phase term, w. is the frequency, and 2m is the period term).
2n
2.3
COMPLEX- VALUED WORD EMBEDDING
We now define our complex-valued word embedding g as a map taking word index j and position word index pos to CP, For word Wj in position pos, our general valuee embedding is defined as f(j. pos) = (pos) rye" (l0sppppt))); Therefore, f (j, pos) is defined
1oo , ..... 2j (=s,apm)0 De (us..,ppttttt.))
(5)
2x and ini- Note that each coordinate d (1 <d D) has separate amplitude rj.t, period Pia ay tial phase Ojdd In Fig. each dimension is represented as a wave which parameterized by an am- plitude, period/frequency. and an initial phase. The trainable parameters of the embedding are the amplitudes vector r, = [r .....
T..... p. the period/frequency related weights W/ (w,.,...,,,,,... W.D,, and the initial phase vector 02 (0,.1..., .......).. Note that the mean values of (jj, .) over all posi- tions are linearly dependent on the amplitude. Observe that the period/frequency determines to what degree the word is sensitive to the position. With an extremely long period (i.c., W, very small), the complex-valuee embedding is approximately constant for all possible values of pos, and hence approximates standard word embedding. Conversely, if the period is short, the embedding will be highly sensitive to the position argument.
In our embedding, the mean vectors of f(j--) taken over all positions linearly correlated to the amplitude embedding r, = [r..1.... [x.]] with a coefficient 2. The amplitude Tj.a of our embedding depends only on the word wy (and coordinate d), not on the position of the word, whence can
4

---

Published as a conference paper at ICLR 2020
Dataset
Irain
test
vocah.
task
Classes
CR (Hu & Liu 2014)
4K
CV
6K
product reviews
MPQA
(Wiebe el al.
2005
11k
CV
6K
opinion polarity
2
SUBJ
Pang Lee
(205
10k
CV
21k
subjectivity
2
MR
Pang Lec
20155
11.9k
CV
20k
movie reviews
SST
Socher ct al.
2013
67k
2.2k
18k
movie reviews
TREC
(Li & Roth
2002)
5.4k
0.5k
10k
Question
Table 1: Dataset Statistics. train/dev/test splits.
CV means 10-fold cross validation.
The last 2 datasets come with
think of the vector Ope (J. pos) po.)
[e (post ... (wsssspo as "purely" positional em- bedding. Consequently, our complex embedding can be considered an element- wise multiplication between the word embedding Swe(j) = [rj.1, [s and position embedding Spc-
f(j.pos, = gwe (j) ④ (p pos)
(6)
Prior work (Gehring al., 2017 Vaswani et al. 2017) uses mean-weight addition between word embeddings wwe and position embeddings free (all words share the weights). In our work, word em- beddings and position embeddings are decoupled to some extent by element-wise multiplication and therefore the frequency/period terms (related to wi,a) can adaptively adjust the importance between semantic and position information for each word and each dimension. In particular, with higher frequency (i.e., large Wj.a), the final embedding will change dramatically with the changing posi- tions, while it can be fixed for any positions with an extremely-smal frequency (i.e., small wj,a)) Interestingly, the well-known position embedding in Transformer (Vaswani et al., (007)) can be seen as degraded version of one of our specific complex word embeddings (see the Appendix A)
3 EXPERIMENTAL EVALUATION
We evaluate our embeddings in text classification, machine translation and language modeling.
3.1 TEXT CLASSIFICATION
Experimental Setup. We use six popular text classification datasets: CR, MPQA, SUBJ, MR, SST, and TREC (see Tab. D. We use accuracy as evaluation measure based on fixed train/dev/test splits or cross validation, as per prior work. We use Fasttext (Joulin al.] 2016), CNN (Kim 2014),
LSTM and Transformer (Vaswani et al., 2017) as NN baselines]] We use each of them: (1) with- out positional information; (2) with Vanilla Position Embeddings (PE) (randomly initialized and updated during training using the sum between word and position vectors (Gehring et al.) 2017);
(3) with Trigonometric Position Embeddings (TPE) (defining position embeddings as trigono- metric functions as per Eq. 7 (4) with Complex-vanile word embeddings (where the amplitude embedding is initialized by the pre- pre-trained word vectors, and the phrase embedding is randomly ini- tialized in range from -I to without considering word order (Wang al.) (019)); and (5) with our order-aware complex-valued word embeddings, Complex-order (which encode position in the phase parts, train the periods, and where the amplitude embedding is also initialized by pretrained
word vectors). For more details on the complex-value extensions of NNs, see App. B and App. C
Our embedding generally has D W parameters with D-dimensional word vectors and [W] words, while previous work (Mikolov et al. 2013b Pennington et al. 2014) usually employs only D |W] parameters for embedding lookup tables. To increase efficiency and facilitate fair com- parison with previous work we set initial phases 0, = [0,.,.... pp. ] to a shared constant value (such as zero). Furthermore, the period vectors wjdd depend on word index j with length [W] and the coordinate index d with length D. To decrease the number of parameters, one can either use word-sharing scheme (i.e., Wjdd = w..a), or a dimension-sharins scheme (wij.d = Wj.-), leading to [ * D + W and |W| * D + D parameters in total for the embedding layer.
2 Graph convolutional networks (GCNs) (Beck et al. 2018 Sahu et al. 2019) also encode positional infor- mation. We do not compare against them because they encode positional information inherently part of the model, which makes redundant any additional encoding of positional information the embedding level.
5

---

Published as a conference paper at ICLR 2020
Text classification accuracy without position embeddings, with random position embed- dings (PE), with trigonometric position embeddings (TPE), with complex-valued NNs without po- sition embeddings (complex-vanille) and with our complex-order embeddings. Superscripts 5. f. and mean significant improvement over a baseline without position embeddings 8 PE TPE! and Complex-vanilla using Wilcoxon's 's signed-rank test p<0..05.
Table 2:
Method
MR
SUBJ
CR
MPQA
SST
TREC
Fasttext
0.765
0.916
0.789
0.874
0.788
0.874
Fasttext-PE:
0.774
0.922
0.789
0.882
0.791
0.874
Fasttext-TPP
0.776
0.921
0.796
0.884
0.792
0.88
Beattex--Comlee--vvnii
0.773
0.918
0.79
0.867
0.803
0.872
Fasttext- Destext-Coomplex-odeer
0.787
0.929811
0.800
0.889511
0.809
0.892811-
LSTM
0.775
0.896
0.813
0.887
0.807
0,858
LSTM-PE
0.778
0.915
0.822
0.889
0.811
0.858
LSTM-TPE
0.776
0.912
0.814
0.888
0.813
0.865
LSTM-CComplex-vailla
0.765
0.907
0.810
0.823
0.784
0.784
LSTM-Commlex-orrer
0.790
0.926 12+
0.828
0,8975144
0,819
0,869811-
CNN
0.809
0.928
0.830
0.894
0.856
0.898
CNN-PE
0.816
0,938
0.831
0.897
0.856
0.890
CNN-TPE
0.815
0,938
0.836
0.896
0.838
0.918
CNN-Complex-vanilla
0.811
0.937
0.825
0.878
0.823
0,900
CNN-Complex-order
0.8258t1~
0.951
0.852
0.906
0.864 t
0.939
Transformer w/o position embedding
0.669
0,847
0.735
0.716
0.736
0,802
Transformer-Pi PE
0.737
0.859
0.751
0.722
0.753
0.820
Transformer- TPE /Vaswani al. 2017
0.731
0.863
0.762
0.723
0.761
0.834
Transformer Cowpeeee--ooooex-vaiaeea
0.715
0.848
0.753
0.786
0.742
0.856
Translorne--ompllex-rrree
0.746
0.895
0.806
0.863814+
0.813
0.896811-
Table 3: Text classification accuracy.
means that scores are reported from other papers.
Method
MR
SUBJ
CR
MPQA
SST
TREC
Word2vec Bow (Conneau et al.
2017
0.777
0.909
0.798
0.883
0.797
0.836
Sent2Noc (Pagliardini et al. 2011
2017)*
0.763
0.912
0.791
0.872
0.802
0.858
QuickThoughts (Logeswaran & Lec 2018
0.824
0.948
0.860
0.902
0,928
InferSent (Conncau et al. 2017)
0.811
0.924
0.863
0.902
0.846
0.882
QPDN (Wang 2019
0.801
0.927
0.810
0.870
0.839
0.882
We search the hyper parameters from a parameter pool, with batch size {32, 64, 128], learning rate in {0.001, 0.0001, 0.00001), L2-regularization rate in {0, 0.001, 0.0001), and number of hid- den layer units in { {120, 128). We use pre-trained 300-dimensional vectors from word2vee (Mikolov et al. (013a) in all models except for Transformers. The models with trainable trigonometric posi- tion embedding produce nearly identical results compared to the non-trainable version, therefore we report the result of fixed position embeddings as per Vaswani al. (017). We adopt narrow convo- lution and max pooling in CNN, with number of filters in {64, 128), and size of filters in {3, 4, 5}. In all Transformer models, we only the encoder layer to extract feature information, where the layer is 1, dimension of word and inner hidden are 256 and 512 respectively, and head number is 8.
Results. The results are shown in Tab. 2 Our complex-order embeddings outperform all other variations at all times. This gain in effectiveness comes at negligible (or non-existent) cost in efficiency (it varies per NN architecture see Fig. 2 CNNs are the best performing NN as expected following Bai et al. (2018). Tranformer NNs benefit the most from complex-order embeddings, most likely because they are our weakest baseline. To contextualise these results, Tab. 3 shows classification accuracy of five typical approaches the same datasets (as reported in the original papers). Our complex-order embeddings outperform all methods, except for the CR dataset, where InferSent is marginally better. Overall, our approach is par with the SOTA in embeddings.
We perform an ablation test (Tab. 4 on Transformer because it is the most common NN to be used with position embeddings. The two period-sharing schemas cdimension-sharigg and word-sharing)

---

Published as a conference paper at ICLR 2020
Fastteat BILSTM CNN Transformer
23
21
10
Figure 2: Computation time (seconds) per epoch in Tensorflow on TITAN X GPU.
Table 4: Ablation test for Transformer, showing the effect of (i) the definition of embedding layer( falj, pos)), and (ii) whether the real-part and imaginary transition share the weights, i.c., R(W2/K/V) 3(W9/K/V)
Method
Setting
Params
Accuracy
A
Sali. pos)
share in Q/K/vv
Transformerr-omplee--rrddd
Vi Tg (lsg,apo))
8.33M
0.813
adding initial phases
11.89M
0.785
-0.028
period schema
5.82M
0.797
-0.016
word-sharing period schema
T ae paposs
5.81M
0.805
-0.008
dimension-sharing amplitude schema
5.82M
0.798
-0.015
word-sharing amplitude schema
T. T.
5.81M
0.804
-0.009
w/t encoding positions (complex-vanilla)
9.38M
0.764
-0.049
dimension maaeeiinn-sharing period schema
4.77M
0.794
-0.019
" arros
word-sh word-sharing period schema
4.76M
0.797
-0.016
dimension- cimension-ssaring amplitude schema
ry "0....o
4.77M
0.792
-0.021
word-sharing amplitude schema
4,76M
0.801
-0.012
encoding positions (complex-vanilla)
Tg 3.
8.33M
0.743
-0.07
vanilaa Transformer /Vaswani et al. 2017)
W WEga + PEa
4.1M
0.761
-0.052
slightly drop performance, because fewer parameters limit the representative power. Adding ini- tial phases also hurts performance, although we observed that the loss could decrease faster in early epochs compared to the setting without offset. The negative effect of initial phases may be due to pe- riodicity, and w cannot be directly regularized with L2-norm penalties. The sharing schemes slightly decrease the performance with less parameters. More details of the learned periods/frequeecces (e.g. the distributions of periods/frequencies and case studies) are shown in App. D
Note that the word-sharing schema outperform the Vanilla Transformer, (both have comparable 3 3/K// ), the additional parameters in the
number of parameters). If we choose R( R((QQ/K/V) embedding layers will affect much less the whole parameter scale in the multiple-layer Transformer, since embedding layer is only used in the first layer instead of the following Transformer layers.
3.2 MACHINE TRANSLATION
Experimental Setup. We use the standard WMT 2016 English-German dataset (Sennrich et al, 2016), whose training set consists of 29,000 sentence pairs. We use four baselines: basic Attentional encoder-decoder (AED) (Bahdanau et al. 2014); AED with Byte-pair encoding (BPE) subword segmentation for open-vocabulary translation (Sennrich al. 2016); AED with extra linguistic features (morphological, part-of-speech, and syntactic dependency labels) (Sennrich & Haddow) 2016); and a 6-layer Transformer. Our approach (Transformer Complex-order) uses batch size of 64, a head of 8, 6 layers, the rate of dropout is 0.1, and the dimension of the word embedding is 512. The embedding layer does not use initial phases, i.c. following f(j. (j.pos) = (wwpos)) We evaluate MT performance with the Bilingual Evaluation Understudy (BLEU) measure.

---

Published as a conference paper at ICLR 2020
Table 6:
Language modeling results.
Table 5: Machine translation results. marks scores
marks scores reported from other papers.
reported from other papers.
Method
BPC
Method
BLEU
BN-LSTM (Cooijmans et
2016 * 1.36
AED (Bahdanau et al.
2014)x
26.8
LN HM-LSTM (Chung el al.
2016 * 1.29
AED+Lnguste (Sennrich & Haddow
2016)
28.4
RIIN Zilly et al 2007)
1.27
AED+BPE (Sennnich el al. 2016)
34.2
Large mLS I' Krauss ct al.
2016
1.27
Transformer (Ma el al. 2019)
34.5
Transformer XL 6L (Dai al.
2019
1.29
Transformer complex vamila
34.7
Transformer complex vanilla
1.30
Transformer Complex-order
35.8
Transformer XL Complex-order 6L
1.26
Results Tab. shows the MT results. Our approach outperforms all baselines. Two things worth noting: (I) Both the vanilla Transformer and our Transformer Complex- outperform the three Attentional encoder-decoder baselines which based on LSTM encoder and decoder, even when AED uses additional features. (2) Our Transformer Complex-Oree outperforms the Vanilla Transformer and complex-vanin Transformer by 1.3 and 1.1 in absolute BLEU score respectively,
3.3 LANGUAGE MODELING
Experimental Setup. We use the text8 (Mahoney (011] dataset, consisting of English Wikipedia articles. The text is lowercased from a to Z, and space. The dataset contains 100M characters (90M for training. 5M for dev, and 5M for testing, as per Mikolov et al. (2012). We use as baselines BN-LSTM, LN HM-LSTM RHN and Large MLM,, which are typical recurrent NNs for language modeling in this dataset. We evaluate performance with the Bits Per Character (BPC) measure, (the lower, the better). We run the coder in Dai et al. (2019) with 6 layers for Transformer XL 6L; model, named Transformer XL complex-order, directly replaces the word embedding with proposed embedding under the same setting. We choose layers due to limitations in computing for Transformer
resources. For Transformer XL Complex-order all other parameter settings XL (Dai et al. 2019). Our complex-order model does not use initial phases.
Results. We Tab, that our method outperforms all baselines. The first four baselines (BN- LSTM, LN HM-LSTM, RHN and Large mLSTM) are based on recurrent NNs and rely on different regulation methods to become applicable in multiple-layer recurrent architectures. Transformer- based architectures can easily be stacked with multiple layers due to their advantages in paralleliza- tion, however the vanilla Transformer does not outperform the multiple-laye recurrent baselines, most likely due to its limitation of 6 layers. Our Transformer XL. Complex-order outperforms its vanilla counterpart under the 6-layer setting (and also strong recurrent network baselines). demon- strating that our embedding also generalizes well in tasks with long-ter dependency. With limited resources, slightly increasing the parameters in the feature layer like our proposed embedding could be more beneficial than stacking more layers with linearly increasing parameters.
RELATED WORK
Complex-valued NNs are not new (Georgiou Koutsougeras 1992 Kim Adali 2003 Hirose
2003 Complex-valued weights have been used in NNs. motivated by biology (Reichert & Serre
2013) and also signal processing in speech recognition (Shi et al., 2006 More recently, Ar-
jovsky et al. (016)) shifted RNNs into the complex domain and Wolter & Yao (2018) proposed novel complex gated recurrent cell. Trabelsi et al. (2017) also developped complex-valuee NN for computer vision and audio processing.
Complex numbers have also been applied to text processing like (Van Rijshergen. 2004 Melueei 2015 Blacoe et al. 2013). Trouillon et al. (2016) adopt complex embedding for entities in Knowl- edge Graph Completion to represent antisymmetric relations with Hermitian dot product. Li et al. 2011 Wang al. (2019) extend word embeddings to complex-valued fashion in quantum prob- ability driven NNs, seeing the overview in Wang et al. (2019). However, the physical meaning of both the complex- camplex-valued entity and word embeddings is unknown, since complex number was considered as two real numbers in a black-bol box learning paradigm. Our work first links the phase in complex numbers to word position define concrete physical meaning in document representations.
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSIONS
We extended word vectors to word functions with a variable i.c. position, to model the smooth shift among sequential word positions and therefore implicitly capture relative distances between words. These functions are well-defined to model the relative distances, therefore we derive a general solu- tion complex-valuee valued fashion. Interestingly. the position embedding in Vaswani al. (2017) can be considered a simplified version of our approach. We extend CNN, RNN and Transformer NNs to complex-vai -valued versions to incorporate our complex embedding. Experiments on text classifica- tion, machine translation and language modeling show that our embeddings are more effective than vanilla position embeddings Vaswani et al. 2017).
ACKNOWLEDGMENTS
We thank Massimo Melucci and Emanuele Di Buccio for their helpful comments, Xindian Ma for his detailed experimental suggestions.
This work is supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union's 's Horizon 2020 research and innovation programme under the Marie Skfodowska-Cus cursic grant agreement No. 721321. Peng Zhang and Donghao Zhang supported in part by Natural Science Foundation of China (grant No. 61772363, U1636203)
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on International Conference on Machine Learning, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 2014.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional
and recurrent networks for sequence modeling. CoRR, abs/1803.01271, 2018. URL http:
aaaaiiv. azxiv..rg//bs//1800000
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequence learning using gated graph 09835, 2018.
neural networks. arXiv preprint
William Blacoe, Elham Kashefi, and Mirella Lapata. A quantum-theoretic approach to distributional semantics. NAACL, pp. 847-857, 2013.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural net- works. arXiv preprint arXiv: 0e.11169007700 2016.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Super- vised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv: 700.002305. 2017.
Tim Cooijmans, Nicolas Ballas, César Laurent, Çaglar Gülçehre, and Aaron Courville. Recurrent batch normalization. arXiv preprint arXiv: 1603.0000000 2016.
Zihang Dai, Zhilin Yang. Yiming Yang. William W Cohen, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint Arvi .0286000000 2019.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.888000 2018.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1243-1252. JMLR. org, 2017.
George M Georgiou and Cris Koutsougeras. Complex domain backpropagation. IEEE transactions Circuits and systems II: analog and digital signal processing, 39(5):330-334, 1992.

---

Published as a conference paper at ICLR 2020
Akira Hirose. Complex-valued neural networks: theories and applications, volume 5. World Sci- entific, 2003.
Minqing Hu and Bing Liu. Mining and Summarizing Customer Reviews, pp. 10, 2014.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint
2016.
Taehwan Kim and Tülay Adali Approximation by fully complex multilayer perceptrons. Neural computation, 15(7):1441-1666, 2003.
Yoon Kim. Convolutional neural networks for sentence classification.
arXiv preprint
14X8: 2014.
Ben Krause, Liang Lu, lain Murray, and Steve Renals. Multiplicative Istm for sequence modelling. arXiv preprint 1609./660007000 2016.
Qiuchi Li, Sagar Uprety, Benyou Wang, and Dawei Song. Quantum-inspired complex word embed- ding. Proceedings of The Third Workshop on Representation Learning for NLP, pp. 50-57, 2018.
Qiuchi Li, Benyou Wang, and Massimo Melucci. Com: interpretable complex-valus network for matching, In NAACL-HLT (I), pp. 4139-4148, 2019.
Xin Li and Dan Roth. Learning Question Classifiers. COLING. Association for Computational Linguistics, 2002.
Lajanugen Logeswaran and Honglak Lee, An efficient framework for learning sentence representa- tions. arXiv preprint arXiv:1803 17X.::1803.0299,,, 2018.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou. A tensorized transformer for language modeling. arXiv preprint 19XXi:1906609777,, 2019.
Matt Mahoney. Large text compression benchmark. URL: http://ww.. mattmahoney. net/text/text. html, 2011.
Massimo Melucci. Introduction to information retrieval and quantum mechanics, volume 35.
Springer, 2015.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed representations ofwords and phrases and their compositionality. Advances Neural Information Processing Systems, 26: 3111-3119, 2013a.
Tomáx Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cer- nocky. Subword language modeling with neural networks. preprint (http://www. fit. vuthr. c=/mikoloovmmmimcchar pdf), 8, 2012.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3781, 2013b.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence embed- dings using compositional n-gram features. arXiv preprint arXiv:1703.02500 2017.
Bo Pang and Lillian Lee. Seeing stars: exploiting class relationships for sentiment categoriza- tion with respect to rating scales. pp. 115-124, Association for Computational Linguistics, 2005. doi: 01111/5/121944022199555
URL
c2mPdo18-1219840..222 1219855
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pp. 1532-1543, 2014.
David P Reichert and Thomas Serre. Neuronal synchrony in complex-vale valued deep networks. arXiv preprint arXiv: 2013.
10

---

Published as a conference paper at ICLR 2020
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. Inter-sentence relation extraction with document-level graph convolutional neural network. arXiv preprint arXiv: 04684, 2019.
Rico Sennrich and Barry Haddow. Linguistic input features improve neural machine translation. arXiv preprint
2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for wmt 16. arviv preprint arXiv:1606.02855 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self Self-attention with relative position representa- tions. arXiv preprint arXiv: 2018.
Guangji Shi, Maryam Modir Shanechi, and Parham Aarabi. On the importance of phase in human speech recognition. IEEE transactions audio, speech, and language processing, 14(5):1867- 1874, 2006.
Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-II), pp. 129-136, 2011.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP, pp. 1631-1642, 2013.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Fe- lipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher Pal. Deep complex networks. arXiv preprint arXiv:1705.09773 2017.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071-2080, 2016.
Comelis Joost Van Rijsbergen. The geometry of information retrieval. Cambridge University Press, 2004.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Benyou Wang, Emanuele Di Buccio, and Massimo Melucci. Representing words in vector space and beyond. In Quantum-Like Models for Information Retrieval and Decision-Making, pp. 83-113. Springer, 2019.
Benyou Wang, Qiuchi Li, Massimo Melucci, and Dawei Song. Semantic hilbert space for text representation learning. The World Wide Web Conference on, pp. 3293-3299, 2019.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating Expressions of Opinions and Emo- tions in Language. Language Resources and Evaluation, 39(2-3):165-210, May 2005. ISSN 1574-020X, 1572-8412. doi:
URL
com/ 10.1000016.00000 0011///011119000-788888
Moritz Wolter and Angela Yao. Gated complex recurrent neural networks. arXiv preprint orXiv:1806.08200 2018.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189-4198. JMLR. org, 2017.

---

Published as a conference paper at ICLR 2020
A LINKING TO THE POSITION EMBEDDINGS IN (VASWANI ET AL., 2017)
Vaswani et al. (017)) proposed a new initialization for position embedding, resulting in compara- ble performance with previous one (Gehring al. 2017) even without fine-tuning. The position embedding is empirically selected as
PE2a( = sin 1on(pos/100000 /dmoniie)
P (1 pos) cos(pos/10000 2k dmmddel)
Where pos is the position index, 2k and 2k + 1 is the dimension index and dmade is the dimension size of embedding. The reason for choosing this position embedding was not well-explained and its general extension is unknown, leading to some difficulties to improve it.
We claim that the proposed position embedding in (Vaswani al. 2017 is degraded version of one of our specific complex word embedding in word-sharing schema (i.e., Wj.d w..)., in which Pj, = 2T x 1000024/dmodde and the initial phases are set zero. In our complex-valued position embed- ding, let ( pos) e*x10000 /fmmedee cos( 100002k/dmmmtt pos) + (isin100002k///mmm pos). Note that there exists a bi-jection between PE(,, pos) and (-po):
PE pos) () ( = M(spee (s)o)s)
where R and 3 are the operations take the real and imaginary part of a complex-valuse valuue number. Its inverse transformation is
fpe,k (:.ps)) = P 1 (s, pos) + i iPPP2.(..pss
In our overall embedding, each dimension fa (j. pos) = fw) O &. in our approaches, while it pos) = W (i) Pfx pos) in (Gehring et al. 2017; Vaswani et al. 2017). Hence the position embedding in Vaswani et al. 2017) is equivalent, albeit not identical, to our complex-valued position embedding with PYA = 2x x 100002k/00 200002k/dmant /dmaeet It is therefore a particular case of our complex- -aluued position embedding. the word-sharing schema in which all words share the same period at certain dimension, i.e, Pj.k P., is irrelevant to the choice of j.
B INTEGRATING COMPLEX-VALUTE EMBEDDING TO GENERAL NEURAL NETWORKS
Neural networks are typically given real numbers as inputs and return real numbers as outputs. To accommodate complex numbers as in- and output, we devise complex-valued version of various neural network layers i.e. complex-values FastText with dense layer. CNN, and RNN. Unlike exist- ing complex- complex-valued valued neural networks (Trabelsi et al., 2017 Wolter & Yao 2018), our feature layers are also converted into complex-valued layers.
Complex-valued FastText FastTest (Juulin et al. 2016) is simple and efficient neural network architecture using a dense layer over the sum of all word embeddings for general text classification. For linear dense layer, i.e., dense(æ: + iy), where + iy and denote the complex- valuud in- and output, respectively. Let W = A + iB and b = c + id be complex-vales valuee linear weights and bias, respectively. Then, the complex-valued dense layer is given by:
== ( (Ax By + c) + io io Bx + Ay + d)
(10)
where a a real-valued activation function such as the sigmoid function. By rewriting (10) in matrix form (Trbellsi et al. 2017). we obtain:
AA By + cc] 32)) Ba Ay d)
R =))
(11)
where, for 2 + iy, R(2) and 3(2) y. To parameters and fairly compare with our valled basclines, the weights for real-part and imaginary-parr input can be shared, i.c., A B,c c d.
12

---

Published as a conference paper at ICLR 2020
Complex-valuee CNN For the complex- valued version of the convolution operation Trabelsi al. (2117), we similarly define a complex-valuee convolution with separate real and imaginary kernels A and B, to compute convolutions complex-valued CNN network is constructed by stacking the operations based on a complex-valued convolution kernel, adding complex dense layer in (TO), and taking their norm the final predic- tion in the last layer.
the real and imaginary parts of the input in Eq. 10 A
Complex-valued RNN
The basic complex RNN formulation is:
h (W" + W +b
(12)
where and h2 represent the complex-value input and complex-value hidden state vectors at time t, b is complex-valued bias, W and W are complex-valued weight transitions for hidden state and input state, and f (z) o (R (=)) + io (3 (z) is the activation function. The multiplication -hh and W 2, computed as defined in (10) above. Similarly, the complex-valus gates are used in LSTM via operations as in (12). In the final layer, a 12-norm operation is adopted to obtain a real-valued loss for backpropagation.
Complex-valued Transformer The main components in the Transformer are self self-attention sub- layers and position-wise forrffo (FFN) sublayers. A self-attention sublayer employs h atten- tion heads and the concatenation of all heads is used the output followed by parameterized linear transformation. For a sequence embedded as complex-valuee vector input = {w,, W2, .... Wn J. the output of each head computed as a weighted sum of linear transformation of the input sequence itself, namely
outpul; =
Σ oi.j wij,w;W' WV,
(13)
where wy is complex-valued vector and W is complex linear transformation; therefore output, is also complex. Hence, output a sequence complex-values vectors with the same shape imput. The weight coefficient, azjj. which is defined in real-valued domain, is cal- culated as the softmax of the product between complex-values query vectors and key vectors: and
di,j softmax E 61.
+3()) = (w: (w, W
(14)
Gijj
Z is a complex number since w, and wy are complex-valuee valued vectors and W., W are complex- valued transformation. To extend another variant of Transformer called Transformer XL, we keep its original relative position embedding and additionally replace its word embedding with our proposed embedding.
Correspondingly, the FFN sublayer can easily be extended to a complex-valued version by replacing the real-valued layers with complex-valuedd ones. We use batch-normalizatioo separately for the real and imaginary parts.
C IMPLEMENTATION OF THE PROPOSED EMBEDDING
Words functions are implemented neural networks by storing the function parameters (r, w. 0) and then construct the values based on the arguments. Based on the definition, the implementation of the proposed embedding can easily be implemented with only modifying the embedding layer. We list the basic code construct our general embedding as below:
import torch import math class (topee nn. Module)
def
-.init_- - (self. opt):
super ( (CompleXNN,
self, word_emb torch nn Embedding ( (opt. n.token. d_mdde)
self. frequency.emb torch nn. Embedding ( (opt. n_token opt dmmoeel)
self. initial = torch nn Embedding (opt. n_token, opt d_model)
13

---

Published as a conference paper at ICLR 2020
3753
1503
word number
2003
250
0.060 0.065 0.070 0.075 0.080 0.085 0.090 0.095
mean of absolute frequency values
Figure 3: The distribution of the ój. Higher values mean that the word representations are more sensitive the word positions.
def get.embedding ( (self, x):
amplitude self word_emb(x) (x)
frequency self frequency emb(x)
self initial phase emb. weight torch nn. Parameter initial phase emb. weight
% (2 math. pi))
sent (-1) pos_seq torch ar, sent.len 1, 1.0, device=ampllin device)
pos = pos_seq unsqueeze (0). unsqueeze(-1)
pos_seq = pos_seq repeat ([x size (0), 1 amplitude. size
dimension bais self initiniiilll..... phattellpphaseeeee emb (x)
enc.output.phase = torch mul(possee frequency)+ damension.bais enc-output_real amplitude * torch. (eccooupuuupppaa) enc_output_image amplitude torch sin cnc_oouput.phase)
return torch. car enc output real, enc output output_imagee saruttimagee,,. return enc mnc.output.c. real, ene output.pppii...... image
def forward x) return self. get.embeddding())
Note that both the frequency vectors w and initial-phase vectors can be shared between words or dimensions, save parameters. The proposed embedding can be also used in real-valued neural networks if one directly concatenates the real numbers and imaginary- pprt numbers as a double- size real-valued vector; therefore it could easily be extended any existing networks without any complex-valued components. For instance, it could be good extension for Transformer based pretrained models like (Devlin et al. 2018) by enriching the feature layer.
D VISUALIZATION OF FRECQUENCIESSPPRIIODS
After training, obtain the frequency vector for each word. For each word, the mean value the absolute frequency values, i.e., 6, = bbj Eb (wy.el is considered as metric test the positional sensitivities of the word, since a period value could negative during training. The density of the 5, is shown in Fig.
Words with the 50 greatest, and the 50 smallest, frequencies in the SST dataset are shown in Tab. For the words with greatest frequencies, most of them strong sentiment words like "worst"
14

---

Published as a conference paper at ICLR 2020
"stupid" and "powerful"; reason for this may be that such words appear in many positions in many documents during training, and thus they are more sensitive to the positions. Conversely, there are fewer words expressing strong sentiment among words with smaller frequencies, as shown in the second row.
words
worst solid stupid powerful mess wonderful remarkable suffers intoxicating thoughtful rare captures portrait gem frontal terrific unique wannabe witty lousy pointless contrived
greatest frequencies in descending order
worse refreshingly charming inventive amazing junk incoherent
refreshing mediocre unfunny thinks enjoyed heartbreaking delightfully crisp brilliant heart spirit perfectly nowhere mistake engrossing fashioned excellent unexpected wonderfully means slowly proposal schemes roiling juliette titles fabric superstar ah wow choreographed tastelessness beg fabulous muccino jacobi legendary jae example code sensation counter deaths hall cun drug metiernan storylinee cellophane wild motion ups trick comedy entertained mission frightening witnesses snoots liners african groan satisfaction calm saturday estranged holm refuses inquisitive
smallest frequencies in ascending order
Table Words with greatest frequencies and frequencies periods (based on 8,) in SST (a sentiment classification task), all words are converted to lower-case. The strong sentiment words bold based on manual labeling.
15


Reviews:
[{"review_id": "r1lxJDIcKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe authors present a \"natural\" way of encoding position information into word embeddings and present extensive empirical evidence to support their method. I believe that paper meets the bar for acceptance.\n\n### Details\n\nThe paper \"Encoding word order in complex embeddings\" presents a method for making word embeddings position dependent. The idea in a nutshell is to map each discrete position , n, to a value  `   A exp( freq_{word, dim} × n )` . So a word embedding is a collection of complex valued signals sampled at discrete points.\n\nThe frequency is dependent on each word and each dimension in general. The authors motivate/justify this particular formulation via their Claim 1, which argues that their particular formulation uniquely satisfies two intuitive constraints. Although one of those constraints (i.e. linearly witnessed Position-free offset) almost completely specifies the solution.\n\nThe experiments in the paper are fairly thorough and cover text classification, machine translation and language modeling. Through the comparative experiments the complex embeddings we can see that the formulation in this paper outperforms existing SOTA methods, sometimes with significantly difference such as a difference of 1.3 BLEU point on the MT task. \n\nI would have liked to say that the ablation are similarly conclusive but there seem to be a problem in the table, which eroded my confidence:\n\n1. The number of parameters in rows 5 and 8  (w/t encoding positions, share / not-share respectively) are reported to be 9.38M and 8.33M which has to be wrong. Similar problem happens with other pairs. And now I am not sure whether the results were also swapped or not. Still the results in general trend in the right direction.\n\n### Possible improvements to the paper\n\n1. The main weakness of the paper is that the authors repeatedly mention that encoding the position as a multiplicative factor which is multiplied to the frequency gives leads to a more decoupled/interpretable embedding but their experiments are solely focused on accuracy measurement. I would have liked to see the authors carry out more experiments to see whether the frequency parameters really are interpretable? For example, \n      -  What is the histogram of the frequencies ? Are some of them negative? \n      - Which word has the highest frequencies (pooled over all dimensions) in absolute term? Does it make sense that that word's meaning is so position dependent? For example, positions can capture subjects versus objects in english, but they will more reliably reflect the subject versus verb distinction in hindi. \n      - Are the word frequencies by themselves predictive of anything? For example, what happens if the word embedding amplitudes are tied across words or dimensions? We expect the performance to be bad but how bad? \n\nThese kinds of ablations  / qualitative analysis will really make the paper more informative and interesting. Right now it just seems like yet another paper where the capacity of the model is increased and the accuracy increases. Specially because the delta improvement over the fixed positional embeddings of (Transformer-TPE Vaswani et al. 2017) is so limited.\n\n\n\n### Edit after the author response\n\nThe authors have made the required corrections and added the necessary analysis. One interesting outcome was that the \"word-sharing amplitude schema\" seems to drop so little in performance, it's almost like all the information can be coded in just the phase vectors alone. It will be nice if the authors release their trained phase embeddings as well, for the words. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SygmcOJ0YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to learn position varying embeddings of words using complex numbers. Specifically, this work learns a position embedding by learning a continuous function that respects relative position based constraints and is bounded. The authors show that complex representations are an ideal fit for this purpose wherein the amplitude of the complex wave represents the base word embedding that is positionally invariants and the \"wave\" part encodes encodes the evolution of each dimension with position  as a periodic function with a learnable phase and period. \n\nResults are shown on text classification baselines and improvements are small. In the experiments related to machine translation and language modeling, some relevant baselines are missing (which were covered in the text classification case) , most importantly, Vaswani etal 2017 variant of complex position embeddings and \"complex-vanilla\", and all the numbers for other baselines are reported from the corresponding papers, hence it is unclear whether the improvement shown is strictly comparable or not.\n\nMoreover, a question that is unanswered is how does the periodicity affect the quality of embeddings. Basically, because of the periodic nature, the dimensions will take the same value for multiple positions spaced out according to the period. Is this a good assumption? Now, with large periods, for a finite practical length value, the periodic effects might end up not being observed but is that the case in the models that this approach learns? It would be great if authors could characterize the contexts/ words for which the periods are small and the contexts for which they are large. Basically, my concern is about the effect of getting the same embedding as output for different positions which is very likely if most of the periods learnt are small. \n\nAlso, empirical results with some other functions (maybe unbounded, or non-linear functions that do not respect relative positional constraints) would be insightful in order to assess the need for the desiderata laid out for the position sensitive functions. Finally, the \"iff\" proof needs to be cleaned up because I am not still not convinced if the proof holds oin both the directions and I believe there could be other functions with desired properties.\n\nThere are minor typos in equations like last line of \"Property 1\" related to g_pos, x.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1e32fPEcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Problem and Previous Research\nThis paper tackles the problem of incorporating the sequential structure of words for text processing. \nPrevious research [Gehring et al., ICML'17; Vaswani et al., NeurIPS'17] tackles the problem by adding position embeddings at the feature level. \nSupported by recent empirical results [Shaw et al., NAACL'18; Dai et al., ACL'19], the paper argues that these position embeddings are independent i.e. do not consider relations between neighbouring word positions.\n\n### Contributions\nTo address this key limitation of previous research, the paper proposes to define the embedding of each word through a continuous function over its position (so that embeddings shift smoothly with increasing positions thereby modelling word order). \nThe paper then lists two properties that such a function needs to satisfy and proposes to use the complex space as the target domain of the function. \nExperimental results on text classification, machine translation, and language modelling show gains over classical and position-enriched word embeddings.\n\n### Pros and Cons\nOverall, the paper tackles an important problem in word embeddings and proposes a principled approach to the problem. \nHowever, the paper could be further strengthened by positioning itself with respect to other existing neural network-based approaches that incorporate sequential structure e.g. graph neural networks (GNNs).\nGated-graph neural networks (GGNN) [Beck et al., ACL'18] and graph convolutional networks [Sahu et al., ACL'19] use GNNs on graphs with words as nodes and labelled edges (adjacence, precedence, etc.) between nodes to model the sequential structure between words. \n\n### Possible Improvements\nOn the empirical side, GGNN of Beck et al. seems esp. relevant since they show improvements on machine translation.\nA GNN-based baseline to compare against is to use Transformer - Complex - vanilla embeddings as features to a GGNN on the graph with words as nodes and adjacence, precedence labelled edges between neighbouring words.\n[Beck et al., ACL'18] Graph-to-Sequence Learning using Gated Graph Neural Networks\n[Sahu et al., ACL'19] Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network\n\nI am open to revising my rating based on the responses of the authors.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1x_HUU8cH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes present an original way to encode the position of the token when encoding them in a sequence. The classical additive encoding of positions creates several issues, such as the lack of flexibility when dealing with pooling layers, and the authors refer it as the position-independence problem. \n\nInstead, the proposed approach is based on the encoding of a term-specific frequency (through the complex argument) and modulus in the complex-space, applied once per embedding dimension. This enables the embedding of a word to be dependent on the position in a non-linear manner. The intuition is similar to the use of complex numbers in signal analysis.\n\nSorry this is not scientifically, but I have to mention that I find the axiomatic derivation of the approach simply beautiful. It is amazing to find such a simple formula from two obvious properties that someone would want from a positional encoding: Position-free offset transformation and boundedness to handle arbitrary length. \n\nThe fact that the offset does not have positive effect is interesting, and the discussion about it is limited. I would assume it is due to some redundancy in the other two parameters, but more experiences would be needed.\n\nThe rest of the paper shows impressive results, both for text-classification and for machine translation, with a clear comparison with the state-of-the-art. The gains are really significant, providing a clear validation of the approach.\n\nIn short, it is quite rare to find such a clear and simple idea with so much empirical gains. I would love to meet the authors once the review period is over.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new language model that captures both the position of words, and their order relationships.  This redefines word embeddings (previously thought of as fixed and independent vectors) to be functions of position.  This idea is implemented in several models (CNN, RNN and Transformer NNs) to show improvements on multiple tasks and datasets.

One reviewer asked for additional experiments, which the authors provided, and which still supported their methodology.   In the end, the reviewers agreed this paper should be accepted.
Decision: Accept (Spotlight)