Encoding word order in complex embeddings

1 INTRODUCTION . When processing text , the sequential structure of language is important , but can be computationally costly to model with neural networks ( NNs ) ( Socher et al. , 2011 ) due to the difficulty in parallelization . This has been alleviated by modeling word sequence not on the NN architecture level , but by adding position embeddings on the feature level . This has been done by the convolutional sequence model ( ConvSeq ) ( Gehring et al. , 2017 ) and the Transformer model ( Vaswani et al. , 2017 ) that replaces recurrent and convolution operations with purely attention mechanisms . More generally , vanilla position embeddings ( Gehring et al. , 2017 ) assume that individual word positions are independent and do not consider relations between neighbouring word positions . We posit that both the global absolute positions of words and their inner sequential and adjacent relationships are crucial in language . This is supported by recent empirical findings by Shaw et al . ( 2018 ) and Dai et al . ( 2019 ) who show the importance of modeling distance between sequential elements , and explicitly use extra relative position encodings to capture the relative-distance relationship of words . We present a novel and principled approach to model both the global absolute positions of words and their inner sequential and adjacent relationships as follows : we extend each word embedding , previously defined as an independent vector , as a continuous function over an independent variable i.e. , position . The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions . Hence , word representations in different positions ∗First two authors contributed equally . †Corresponding author : pzhang @ tju.edu.cn 1The code is on https : //github.com/iclr-complex-order/complex-order 𝑟1 𝑟2 𝑟3 𝑝2 𝑝1 𝑝3 Figure 1 : 3-dimensional complex embedding for a single word in different positions . The three wave functions ( setting the initial phases as zero ) show the real part of the embedding ; the imaginary part has a π 2 phase difference and shows the same curves with its real-valued counterpart . The x-axis denotes the absolute position of a word and the y-axis denotes the value of each element in its word vector . Colours mark different dimensions of the embedding . The three cross points between the functions and each vertical line ( corresponding to a specific position pos ) represent the embedding for this word in the pos-th position . can correlate with each other in a continuous function . Fig . 1 illustrates this type of word representation with a three-dimensional complex-valued embedding , where the amplitudes { r1 , r2 , r3 } denote semantic aspects corresponding to classical word vectors , and periods { p1 , p2 , p3 } denote how sensitive the word is to positional information . We further discuss the necessary properties of these functions to model sequential information and obtain a general solution in the form of a complexvalued embedding . Interestingly , there is a direct connection between a specific case of our general embedding and the well-known positional encoding in Vaswani et al . ( 2017 ) ( see App . A ) . We contribute ( i ) a novel paradigm that extends word vectors as continuous functions over changing variables like word position , and ( ii ) a general word embedding that models word order in a mathematically-sound manner . We integrate our complex word embeddings in state-of-the-art ( SOTA ) NN architectures ( CNN , RNN , Transformer and experimentally find that it yields gains over both classical word embeddings and position-enriched word embeddings in text classification , machine translation and language modeling . Note that this is the first work in NLP to link imaginary numbers in complex-valued representation to concrete meanings ( i.e. , word order ) . 2 MODELLING WORD ORDER IN EMBEDDING SPACE . A Word Embedding ( WE ) generally defines a map fwe : N → RD from a discrete word index to a D-dimensional real-valued vector and N = { 0 , 1 , 2 , . . . } . Similarly , a Position Embedding ( PE ) ( Gehring et al. , 2017 ; Vaswani et al. , 2017 ) defines another map fpe : N → RD from a discrete position index to a vector . The final embedding for word wj ( wj ∈ W with index j in a given vocabulary W ) in the pos-th position in a sentence is usually constructed by the sum f ( j , pos ) = fwe ( j ) + fpe ( pos ) , ( 1 ) and f ( j , pos ) ∈ RD . Since both the word embedding map fw and the position embedding map fp only take integer values as word indexes or position indexes , embedding vectors for individual words or positions are trained independently . The independent training for each word vector is reasonable , since a word index is based on the order of a given arbitrary vocabulary and does not capture any specific sequential relationship with its neighboring words . However , the position index captures an ordered relationship , for instance adjacency or precedence , leading to the problem that position embeddings in individual positions ( Gehring et al. , 2017 ) are independent of each other ; the ordered relationship between positions is not modelled . We refer to this as the position independence problem . This problem becomes more crucial when position embeddings are used in position-insensitive NNs , e.g. , FastText ( Mikolov et al. , 2013b ) , ConvSeq ( Gehring et al. , 2017 ) and Transformer ( Vaswani et al. , 2017 ) , because it is hard for such position-insensitive NNs with vanilla position embeddings ( Gehring et al. , 2017 ) to infer that wj1 in the pos-th position is close to wj2 in the pos + 1-th position , or that wj1 precedes wj2 ; instead , it is only inferred that wj1 and wj2 are in different positions , while the relative distance between them is almost unknown . Thus vanilla position embeddings ( Gehring et al. , 2017 ) can not fully capture the sequential aspect of language . Next , we first introduce the necessary properties to model word order in embeddings , and then give a unique solution to meet such properties . 2.1 EXTENDING VECTORS TO FUNCTIONS . In the general definition in Eq . 1 , each dimension of the position embedding is obtained based on the discrete position indexes { 0 , 1 , 2 , ... , pos , ... } . This makes it difficult to model the ordered relationship between the positions . One solution to this problem is to build continuous functions over a variable ( i.e. , position index ) to represent a specific word in an individual dimension . Formally , we define a general embedding as f ( j , pos ) = gj ( pos ) ∈ RD , ( 2 ) where gj is short for gwe ( j ) ∈ ( F ) D , indicating D functions over position index pos , and gwe ( · ) : N → ( F ) D is a mapping from a word index to D functions . By expanding the D dimension of gj , a word wj in the pos-th position can be represented as a D-dimensional vector as shown in [ gj,1 ( pos ) , gj,2 ( pos ) , ... , gj , D ( pos ) ] ∈ RD , ( 3 ) in which ∀gj , d ( · ) ∈ F : N→ R , d ∈ { 1 , 2 , ... , D } is a function over the position index pos . To move the word wj from the current position pos to another one pos′ , it needs only replace the variable pos to pos′ without changing gj . Functions for words , especially continuous functions , are expected to capture smooth transformation from a position to its adjacent position therefore modeling word order . The position-independent position embedding ( Gehring et al. , 2017 ) can be considered as a special case of our definition when it only takes independent values for individual positions in the embedding function . 2.2 PROPERTIES FOR THE FUNCTIONS TO CAPTURE WORD ORDER . Relative distance is hard to compute because position indices are not visible in NNs after vector embedding ( discrete position indices are necessarily embedded as vectors like words to be backpropagated with the gradient ) . Hence , we claim that the modeling of relative distance in NNs should be position-free : absolute position indices can not be directly accessed in intermediate layers . Instead of processing position-free operations in NNs to capture relative distance between words , prior work ( Shaw et al. , 2018 ; Dai et al. , 2019 ) first calculates the relative distance between words , and then feeds the relative distance as an additional feature or as embeddings/weights to NNs , instead of directly feeding with the raw position indices . Assume that words are embedded into RD , and let , for 1 ≤ d ≤ D , the function gj , d : N→ R be the embedding function giving the d-th coordinate of the representation of word wj ( i.e. , gj , d ( pos ) is the d-th coordinate of the embedding ofwj if it occurs at position pos . In the following , we simply write g instead of gj , d when there is no risk of confusion . Ideally , one would like there to exist a function Transformn : R → R that transforms the embedding of any word at some position pos to the embedding of a word at position pos + n such that Transformn is only dependent on the embedded value itself , but independent of the position pos , that is ∀pos : g ( pos + n ) = Transformn ( g ( pos ) ) . Prior work in NLP ( Li et al. , 2019 ) , Information Retrieval ( Van Rijsbergen , 2004 ) and Machine Learning ( Trabelsi et al. , 2017 ) has shown the usefulness of complex numbers as richer representations . Complex word embeddings ( Wang et al. , 2019 ; Li et al. , 2019 ; Li et al. , 2018 ) have been used to model language . To investigate the potential of complex-valued representation , we extend the target domains of g ( · ) from RD to CD without losing generality , since real-valued numbers are specific complex numbers with their imaginary part being zero . This property regarding “ position-free offset transformation ” in complex-valued domains is formally defined in Property 1 below . Property 1 . Position-free offset transformation : An embedding function g : N → C is said to be a position-free offset transformation if there exists a function Transform : N × C → C ( called the witness ) such that for all n ≥ 1 , the function Transformn ( · ) = Transform ( n , · ) satisfies ∀pos ∈ N : g ( pos+n ) = Transformn ( g ( pos ) ) . A position-free offset transformation g is said to be linearly witnessed if there is a function w : N→ C such that g has a witness Transform satisfying , for all n , Transform ( n , pos ) = Transformn ( pos ) = w ( n ) ( i.e. , each Transformn is a linear function ) . Additionally , a boundedness property is necessary to ensure that the position embedding can deal with text of any length ( pos could be large in a long document ) . Property 2 . Boundedness : The function over the variable position should be bounded , i.e . ∃δ ∈ R+ , ∀pos ∈ N , |g ( pos ) | ≤ δ . Formally , we prove the following claim that there is a unique solution that meets Properties 1 and 2 under the condition that the embedding function is linearly witnessed . We use linear functions because they are well-understood and simple with a single floating-point operation in NNs . Claim 1 . A function g : N → C is a bounded and linearly witnessed position-free offset transformation iff it is on the form g ( pos ) = z2z pos 1 for z1 , z2 ∈ C with |z1| ≤ 1 . Proof . Assume that g is a bounded and linearly witnessed position-free offset transformation . Then , by linear witnessing , we have for all pos , n1 , n2 ∈ N : w ( n1 ) w ( n2 ) g ( pos ) = w ( n2 ) g ( pos + n1 ) = g ( pos + n1 + n2 ) = Transformn1+n2 ( g ( pos ) ) = w ( n1 + n2 ) g ( pos ) whencew ( n1+n2 ) = w ( n1 ) w ( n2 ) . Writew ( 1 ) = z1 and g ( 0 ) = z2 . As n1 , n2 ∈ N were arbitrary , we have w ( n ) = ( w ( 1 ) ) n = zn1 for all n ∈ N. But then g ( pos + n ) = w ( n ) g ( pos ) = zn1 g ( pos ) . Furthermore , observe that for pos ≥ 1 , we have g ( pos ) = g ( 1+pos−1 ) = w ( pos ) g ( 0 ) = zpos1 z2 = z2z pos 1 . For pos = 0 , g ( 0 ) = z2 = z2z 0 1 , whence g ( pos ) = z2z pos 1 , as desired . Observe that if |z1| > 1 , then g ( pos ) is unbounded , whence we have |z1| ≤ 1 . Conversely , assume that g is on the form g ( pos ) = z2z pos 1 with |z1| ≤ 1 . Then , |g ( pos ) | ≤ |z2z pos 1 | ≤ |z2||z pos 1 | ≤ |z2| , whence g is bounded . Define , for each n ∈ N , w ( n ) = zn1 and Transformn ( pos ) = w ( n ) pos . Then , for all pos , n ∈ N , g ( pos + n ) = z2z pos+n 1 = z2z pos 1 z n 1 = g ( pos ) z n 1 = Transformn ( g ( pos ) ) showing that g is a linearly witnessed position-free offset transformation . For any z ∈ C , we may write z = reiθ = r ( cos θ + i sin θ ) . Thus , for the general form of the embedding g from Theorem 1 , we have : g ( pos ) = z2z pos 1 = r2e iθ2 ( r1e iθ1 ) pos = r2r pos 1 e i ( θ2+θ1pos ) subject to |r1| ≤ 1 ( 4 ) In implementations , the above definition of g will lead to an optimization problem due to the constraint |r1| ≤ 1 . A natural and simple way to avoid this is to fix r1 = 1 ; note that |eix| ≡ 1 , thus automatically satisfying the constraint , in contrast to a real-valued embedding where one would need to explicitly devise functions satisfying the constraint . Finally , Eq . 4 can be written in the simplified form : g ( pos ) = rei ( ωpos+θ ) . Thus , one can think of g as embedding positions counterclockwise on a complex circle of radius r with a fixed period ( r is the amplitude term , θ is the initial phase term , ω 2π is the frequency , and 2π ω is the period term ) .

Summary: This paper proposes to learn position varying embeddings of words using complex numbers. Specifically, this work learns a position embedding by learning a continuous function that respects relative position based constraints and is bounded. The authors show that complex representations are an ideal fit for this purpose wherein the amplitude of the complex wave represents the base word embedding that is positionally invariants and the "wave" part encodes encodes the evolution of each dimension with position  as a periodic function with a learnable phase and period.