Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: I think the paper needs a deep review in the English part. For example, in the abstract, they repeat "In this paper" a couple of time and it is complicated to understand the introduction and methodology. Also, I think the paper needs a better structure. The related work should be first in order to understand the relevance of this paper. 

From the experiment point of view, it is necessary a better explanation about the hyperparameters or the experiments which were carried out. In addition, the single database was used to evaluate the technology which is not enough to show the big different respect to the transformed paper. Also, the comparison is not really fair. In each "layer" of the proposed phrase transformer, it has actually two self-attention layers, but the baseline has only one self-attention layer. In addition more methodologies should be necessary to compare the results of the experiment. 

The architecture part is complicated to follow and I don't understand the big contribution of this paper. 

For that reason, I recommend a reject the paper and work more for the final version 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This submission proposes to consider to put attention on "phrases" in NLP. The phrases are generated by taking consecutive words in sentences. Each phrase is treated as a "node" in the same way as words. Then representations of phrases are learned in the network. The algorithm is applied to two applications, translation and pos tagging. The proposed method achieved better performance than transformer. 

Critics: 

1. In the abstract and the introduction, the submission argues that usefulness of phrases, which are sementic units represented by word groups. However, in the model development, "phrases" are really bigrams and trigrams. I don't know how much the previous argument is still valid. Particularly, there are so many bigrams and trigrams. The effect from these word combinations should have strong effect on the model, but the effect may not be explained as the argument. 

2. I think transformer can somewhat capture word combinations in bigrams and trigrams. In higher layers, transformer actually combine words in representations. What is the advantage of the proposed method over the type of combination done in transformer? 

3. The experiment only compares to transformer in the translation task. It only compares to transformer and semantic phrase transformer. Other SOTA methods (e.g. different versions of transformers) should be compared. 

4. The comparison is not really fair. In each "layer" of the proposed phrase transformer, it has actually two self-attention layers, but the baseline has only one self-attention layer. 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper addresses an issue of compositionality in self-attention models such Transformer. A simple idea of composing multiple words into a phrase as a hypernode and representing it using a non-linear function to capture the semantic mutation is proposed. In the machine translation and PoS tagging tasks, the proposed PhraseTransformer achieves impressive gain, especially +13.7 BLEU score compared to the Transformer.

The motivation of the paper is very clear, and I love this kind of paper; with a simple idea, making a huge impact on the field. I appreciate the real example to compare how word-level self-attention is different from the phrase-level self-attention in Abstract and Figure 1. The problem itself; tackling the semantic compositionality of self-attention, is a very important problem, and I like the part that authors described it as an inductive bias as a model perspective. 

However, this work seems to be problematic in terms of presentation, clarity, and meaningful comparisons. Please see my detailed comments below.

First, what exactly is “semantic mutation”? The term has been used here and there to describe the inductive bias in semantic compositionality and to show how the nonlinearity can effectively capture it. But, I couldn’t find any definition from the paper, couldn’t find any formal definition from any ACL papers, and couldn’t guess myself based on the context. I am guessing it is probably some sort of combination of meaning in phrase-level words. If so, more importantly, how does the simple non-linear function (i.e., sigmoid) can capture such semantic combinations of words? How could it make such a huge gain (PhraseTransformer vs Linear PhraseTransformer) in Table 1 on MT task? This seems to be the most important contribution of this paper, of which I don’t understand yet. 


What is the “concept of hypernodes in hypergraph”? I think it is not a common word that I can understand it clearly without any references or any background. It would be better to add some references for the concept. Again, I am guessing it is a sort of graph theory that decomposes a large node into small pieces but keeps their connectivity. But, then how is that exactly linked to phrases of words? If you make phrases only on consecutive words, it is basically just chunking. I don’t find any relevance of phrases in a sentence with the (hyper)graph something. 

In Figure 2, how is the bidirectional path made between the word representations and phrase representations? In my understanding of your algorithm based on Equation 2-6, I only see the attention of phrases is computed by word attention, but not the other way.  Please clarify this. If so, how does the gradient back-propagate to each other?

The biggest concern of this work is the scores reported in Table 1. I have checked the recent papers which used the Mutli30K(de-en) and other results from WMT[16-18] reports, but the BLEU score reported in Table 1 (20.90) seems way lower than the scores reported by any systems trained by either non-Transformer or Transformer systems. For that reason, it would be fair to include some results from the state-of-the-art systems on the same dataset. 

A minor point but in the complexity analysis, your m is basically n because you take consecutive words from n length of sentence. You better distinguish which variables are dependent on each other first. 


There are MANY typos, missing captions, grammatical errors in the paper. Here are only some of them:
There is no caption for Figure 1. 
When you cite a reference with its model name, you better make the citation under parenthesis. 
“equation equation 5” -? “equation 5”
“Wee” -> “We”

review_title: Official Blind Review #1

Meta-review: This paper incorporates phrases within the transformer architecture.

The underlying idea is interesting, but the reviewers have raised serious concerns with both clarity and the trustworthiness of the experimental evaluation, and thus I cannot recommend acceptance at this time.
Decision: Reject