Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper makes present an original way to encode the position of the token when encoding them in a sequence. The classical additive encoding of positions creates several issues, such as the lack of flexibility when dealing with pooling layers, and the authors refer it as the position-independence problem. 

Instead, the proposed approach is based on the encoding of a term-specific frequency (through the complex argument) and modulus in the complex-space, applied once per embedding dimension. This enables the embedding of a word to be dependent on the position in a non-linear manner. The intuition is similar to the use of complex numbers in signal analysis.

Sorry this is not scientifically, but I have to mention that I find the axiomatic derivation of the approach simply beautiful. It is amazing to find such a simple formula from two obvious properties that someone would want from a positional encoding: Position-free offset transformation and boundedness to handle arbitrary length. 

The fact that the offset does not have positive effect is interesting, and the discussion about it is limited. I would assume it is due to some redundancy in the other two parameters, but more experiences would be needed.

The rest of the paper shows impressive results, both for text-classification and for machine translation, with a clear comparison with the state-of-the-art. The gains are really significant, providing a clear validation of the approach.

In short, it is quite rare to find such a clear and simple idea with so much empirical gains. I would love to meet the authors once the review period is over.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
### Problem and Previous Research
This paper tackles the problem of incorporating the sequential structure of words for text processing. 
Previous research [Gehring et al., ICML'17; Vaswani et al., NeurIPS'17] tackles the problem by adding position embeddings at the feature level. 
Supported by recent empirical results [Shaw et al., NAACL'18; Dai et al., ACL'19], the paper argues that these position embeddings are independent i.e. do not consider relations between neighbouring word positions.

### Contributions
To address this key limitation of previous research, the paper proposes to define the embedding of each word through a continuous function over its position (so that embeddings shift smoothly with increasing positions thereby modelling word order). 
The paper then lists two properties that such a function needs to satisfy and proposes to use the complex space as the target domain of the function. 
Experimental results on text classification, machine translation, and language modelling show gains over classical and position-enriched word embeddings.

### Pros and Cons
Overall, the paper tackles an important problem in word embeddings and proposes a principled approach to the problem. 
However, the paper could be further strengthened by positioning itself with respect to other existing neural network-based approaches that incorporate sequential structure e.g. graph neural networks (GNNs).
Gated-graph neural networks (GGNN) [Beck et al., ACL'18] and graph convolutional networks [Sahu et al., ACL'19] use GNNs on graphs with words as nodes and labelled edges (adjacence, precedence, etc.) between nodes to model the sequential structure between words. 

### Possible Improvements
On the empirical side, GGNN of Beck et al. seems esp. relevant since they show improvements on machine translation.
A GNN-based baseline to compare against is to use Transformer - Complex - vanilla embeddings as features to a GGNN on the graph with words as nodes and adjacence, precedence labelled edges between neighbouring words.
[Beck et al., ACL'18] Graph-to-Sequence Learning using Gated Graph Neural Networks
[Sahu et al., ACL'19] Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network

I am open to revising my rating based on the responses of the authors.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper proposes to learn position varying embeddings of words using complex numbers. Specifically, this work learns a position embedding by learning a continuous function that respects relative position based constraints and is bounded. The authors show that complex representations are an ideal fit for this purpose wherein the amplitude of the complex wave represents the base word embedding that is positionally invariants and the "wave" part encodes encodes the evolution of each dimension with position  as a periodic function with a learnable phase and period. 

Results are shown on text classification baselines and improvements are small. In the experiments related to machine translation and language modeling, some relevant baselines are missing (which were covered in the text classification case) , most importantly, Vaswani etal 2017 variant of complex position embeddings and "complex-vanilla", and all the numbers for other baselines are reported from the corresponding papers, hence it is unclear whether the improvement shown is strictly comparable or not.

Moreover, a question that is unanswered is how does the periodicity affect the quality of embeddings. Basically, because of the periodic nature, the dimensions will take the same value for multiple positions spaced out according to the period. Is this a good assumption? Now, with large periods, for a finite practical length value, the periodic effects might end up not being observed but is that the case in the models that this approach learns? It would be great if authors could characterize the contexts/ words for which the periods are small and the contexts for which they are large. Basically, my concern is about the effect of getting the same embedding as output for different positions which is very likely if most of the periods learnt are small. 

Also, empirical results with some other functions (maybe unbounded, or non-linear functions that do not respect relative positional constraints) would be insightful in order to assess the need for the desiderata laid out for the position sensitive functions. Finally, the "iff" proof needs to be cleaned up because I am not still not convinced if the proof holds oin both the directions and I believe there could be other functions with desired properties.

There are minor typos in equations like last line of "Property 1" related to g_pos, x.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: ### Summary

The authors present a "natural" way of encoding position information into word embeddings and present extensive empirical evidence to support their method. I believe that paper meets the bar for acceptance.

### Details

The paper "Encoding word order in complex embeddings" presents a method for making word embeddings position dependent. The idea in a nutshell is to map each discrete position , n, to a value  `   A exp( freq_{word, dim} × n )` . So a word embedding is a collection of complex valued signals sampled at discrete points.

The frequency is dependent on each word and each dimension in general. The authors motivate/justify this particular formulation via their Claim 1, which argues that their particular formulation uniquely satisfies two intuitive constraints. Although one of those constraints (i.e. linearly witnessed Position-free offset) almost completely specifies the solution.

The experiments in the paper are fairly thorough and cover text classification, machine translation and language modeling. Through the comparative experiments the complex embeddings we can see that the formulation in this paper outperforms existing SOTA methods, sometimes with significantly difference such as a difference of 1.3 BLEU point on the MT task. 

I would have liked to say that the ablation are similarly conclusive but there seem to be a problem in the table, which eroded my confidence:

1. The number of parameters in rows 5 and 8  (w/t encoding positions, share / not-share respectively) are reported to be 9.38M and 8.33M which has to be wrong. Similar problem happens with other pairs. And now I am not sure whether the results were also swapped or not. Still the results in general trend in the right direction.

### Possible improvements to the paper

1. The main weakness of the paper is that the authors repeatedly mention that encoding the position as a multiplicative factor which is multiplied to the frequency gives leads to a more decoupled/interpretable embedding but their experiments are solely focused on accuracy measurement. I would have liked to see the authors carry out more experiments to see whether the frequency parameters really are interpretable? For example, 
      -  What is the histogram of the frequencies ? Are some of them negative? 
      - Which word has the highest frequencies (pooled over all dimensions) in absolute term? Does it make sense that that word's meaning is so position dependent? For example, positions can capture subjects versus objects in english, but they will more reliably reflect the subject versus verb distinction in hindi. 
      - Are the word frequencies by themselves predictive of anything? For example, what happens if the word embedding amplitudes are tied across words or dimensions? We expect the performance to be bad but how bad? 

These kinds of ablations  / qualitative analysis will really make the paper more informative and interesting. Right now it just seems like yet another paper where the capacity of the model is increased and the accuracy increases. Specially because the delta improvement over the fixed positional embeddings of (Transformer-TPE Vaswani et al. 2017) is so limited.



### Edit after the author response

The authors have made the required corrections and added the necessary analysis. One interesting outcome was that the "word-sharing amplitude schema" seems to drop so little in performance, it's almost like all the information can be coded in just the phase vectors alone. It will be nice if the authors release their trained phase embeddings as well, for the words. 
review_title: Official Blind Review #2

Meta-review: This paper describes a new language model that captures both the position of words, and their order relationships.  This redefines word embeddings (previously thought of as fixed and independent vectors) to be functions of position.  This idea is implemented in several models (CNN, RNN and Transformer NNs) to show improvements on multiple tasks and datasets.

One reviewer asked for additional experiments, which the authors provided, and which still supported their methodology.   In the end, the reviewers agreed this paper should be accepted.
Decision: Accept (Spotlight)