Attention over Phrases
Authors: Wanyun Cui
Venue:  (2020)
Keywords: representation learning
natural language processing
attention

Abstract: How to represent the sentence ``That's the last straw for her''? The answer of the self-attention is a weighted sum of each individual words, i.e. $$semantics=\alpha_1Emb(\text{That})+\alpha_2Emb(\text{'s})+\cdots+\alpha_nEmb(\text{her})$$. But the weighted sum of ``That's'', ``the'', ``last'', ``straw'' can hardly represent the semantics of the phrase. We argue that the phrases play an important role in attention.
If we combine some words into phrases, a more reasonable representation with compositions is 
$$semantics=\alpha_1Emb(\text{That's})+Emb_2(\text{the last straw})+\alpha_3Emb(\text{for})+\alpha_4Emb(\text{her})$$.
While recent studies prefer to use the attention mechanism to represent the natural language, few noticed the word compositions. In this paper, we study the problem of representing such compositional attentions in phrases. In this paper, we proposed a new attention architecture called HyperTransformer. Besides representing the words of the sentence, we introduce hypernodes to represent the candidate phrases in attention. 
HyperTransformer has two phases. The first phase is used to attend over all word/phrase pairs, which is similar to the standard Transformer. The second phase is used to represent the inductive bias within each phrase. Specially, we incorporate the non-linear attention in the second phase. The non-linearity represents the the semantic mutations in phrases. The experimental performance has been greatly improved. In WMT16 English-German translation task, the BLEU increases from 20.90 (by Transformer) to 34.61 (by HyperTransformer).

Under review as a conference paper at ICLR 2020
ATTENTION OVER PHRASES
Anonymous authors Paper under double-blind review
ABSTRACT
How to represent the sentence "That's the last straw for her"? The answer of the self-attention is weighted sum of each individual words, i.e.
semontics ai Emb((Tha) + og Emb((s) + ... Em,b(((rrr)
But the weighted sum of "Thatss, "the", "last", "straw" can hardly represent the semantics of the phrase. We argue that the phrases play an important role atten- tion. If we combine some words into phrases, more reasonable representation with compositions
semantics = an Emb(Tttt'sss) Emb (he last straw) Eaa
While recent studies prefer to use the attention mechanism to represent the nat- ural language, few noticed the word compositions. In this paper, we study the problem of representing such compositional attentions in phrases. In this paper, we proposed a attention architecture called Phrase Transforrr.. Besides rep- resenting the words of the sentence, we introduce hypernodes to represent the candidate phrases in attention. PhraseeTransformer has two phases. The first phase is used to attend over all word/phrase pairs, which is similar to the standard Trans- former. The second phase is used to represent the inductive bias within each phrase. Specially, we incorporate the non-linear attention in the second phase. The non-linearity represents the the semantic mutations in phrases. The exper- imental performance has been greatly improved. In WMT16 English-German translation task, the BLEU increases from 20.90 (by Transformer) to 34.61 (by PhraseTrame Transformer).
INTRODUCTION
The word-to-word attention (Bahdanau et al. 2015) Vaswani et al. 2017) updates the state of word by the weighted sum of the current states of the words. The weights depend on the attention scores. Such mechanism incurs two major problems. Firstly. it is unfeasible to incorporate the semantics of phrases in the attention. For example, in figure [a] and figure [b] using the attention over phrase "the last straw" is more intuitive than using the attention over the three individual words. However, such word composition (i.c. "the last straw"") cannot be represented atomically by word level attentions. Secondly, the literal meaning of each individual words can be quite different from the implied meaning of the phrase. The weighted sum in attention cannot capture the semantic mutation. For example, it's hard to represent the semantics of the unpleasantness, the struggling, and being unable to continue by the weighted sum of individual words "the",, "last", and "straw". Deepening the attention layers as in Devlin et al. (2019); Liu et al. (2019) cannot solve these issues.
Instead of representing the natural language via word-to-word attentions, incorporating phrase as atoms in the attention provides more intuitive representation. In figure [b] if we regard "the last straw" atom node in the attention in figure lb it is much more intuitive to represent the semantics.
For multiple words that form semantic compositions (e.g. idioms, chunkings, phrases), their seman- ties should be computed uniformly as an atom in the attention mechanism. The traditional word level attention mechanism should be expanded represent the attentions over such compositions. based on Transformer.
To this end, we make a very simple framework called We leverage the concept of hypernodes in hypergraph. The traditional Transformer only consists of

---

Under review as a conference paper at ICLR 2020
That's
the
last
straw
for
her.
semantics 01 Emb(That's) a...mb((hh)) + a,, Emb(her)
(a) Word level self-attention
That's
the last straw for her.
semantics ax Emb (That's) ) Emb last straw) + Emb(for)) an Emb(her)
(b) Attention over adjacent phrases.
Representation learning
not only popular
but also important.
semanlies =o Reppeeeesssnetaaion learning) + 02 Emb(is) + 0g Emm(not only but also)
Emb(ot (mEmm(importan))
(c) Attention nonadjacent phrases.
nodes representing single words. From the perspective of graph, Transformer a complete bipar- tite graph, that the information transfers through all node pairs according to their attention scores. In hypergraph, a hypernode consists of one or more nodes. Likewise, we use such hypernodes in Phrase Transssrranfrmmer to represent the semanties of the composition. Each hypernode represents mul- tiple words in composition.
The information propagation in Transforre has two stages: the propagation across arbitrary node pairs and the propagation within a hypernode.
Attention of arbitrary node pairs works similar to the classical Transformer. We compute the attention scores for all nodes and all hypernodes. The information propagates accordingly. We use hypernodes represent the atomic information propagation of semantic compositions in attentions.
Non-linear attention within each phrase reflects the inductive bias of the phrases. The nodes with- in phrase should be updated by each other via attention. For example, the states of the hypernode the last straw should be updated by the node of the, last, straw etc. And the states of node straw should also be updated by the node of the last straw. To do this, limit the attention in Transformer to nodes within each phrase.
Compared the attention arbitrary node pairs, significant difference of the attention within each phrase is that we incorporate nonlincarity. As suggested in figure [a the literal meaning of each individual words can be quite different from the implied meaning of the phrase. We use the non-linearity to represent the semantic mutations of phrases.
Our proposed Transoerrnnsoommrr is able capture any multi-word compositions. In this paper, mainly study one of the simplest form of phrase: sequence of adjacent words. For detecting phrases with adjacent words, we do not utilize existing identification algorithms (e.g. chunking). We roughly added all sequences of words in the sentence whose length is less than given threshold. The attention mechanism itself decides the weights of these candidate phrases by assigning them different attention scores during the inference.
summarize the contributions of this paper below:
We propose to represent semantic compositions the attention mechanism. We show that such representation is more intuitive than the traditional word-to-word attention.
We incorporate the hypernodes in Transformer as PhraseTransforee The semantics of composition can be computed atomically via cross-node attentions. The inductive bias is reflected by the information propagation within each hypernode.
2

---

Under review as a conference paper at ICLR 2020
We evaluate approach two widely used types of semantic compositions, i.e., chunking and semantic dependency. We conduct extensive experiments to verify the fectiveness of our proposed approaches.
MODEL
2.1 ARCHITECTURE
In this subsection, we show how the PraaseTrrnsformmrr works on an abstract level. In the atten- tion, use "simple node" and "hypernode" to denote nodes that correspond a phrase, respectively. We denote both the simple nodes and hypernodes as nodes. The Phrase- Transformer contains simple nodes, where is the length of the sentence, and m hypernodes representing m phrases. For each node s, we denote the words it corresponds to as word(s). For simple nodes, [word(s)]
single word and
For hypernodes, [ > For example, in figure 2 represented by single node $1. word(s)) (straw). The phrase "the last straw" is represented by a hypernode 82, (oord((2) (the, last, straw).
"straw" is
The attention in PhraseTrransforme has two phases. The first phase uses the node-to-node attention over all node pairs. This is represented by the blue edges in figure 2 For example, the state of the placeholder [CLS] will be updated according to its attention scores to nodes of "That's", "for", "the last straw", etc. Similarly, the states a hypernode (e.g. "the last straw") will updated according to its attention scores to all nodes (e.g. "That's", "for", "last straw")..
The second phase uses limited attention mechanism to update the states within each phrase. In figure the orange edges show the attention within the phrase "the last straw". The state of 82 will be updated according to its attention scores to "the", "last", "straw", but will not be updated by "for" "her". And the state of will also be updated by $2-
[CLS]
S2
[CLS]
That's
the
last
straw
for
her
the straw
Nodes for individual words
Hypemodes for phrases
Attention across arbitrary node pairs
Attention within each phrase
Figure 2: Architecture of Phrase Transformer.
2.2 PHRASE GENERATION
As mentioned in the introduction section, a hypernode represents a phrase formed by multiple words. In this paper, we consider the phrases formed by adjacent words in the original sentences. For given length threshold & and word sequence W1. ... Wn, we generate hypernodes that consists of adjacent word sequence whose length less than or equal k. More specifically, we generate all hypermode such that
word( {) (wi, Wi+1, wy). s.. - iis
Our proposed neural network is feasible for any phrase generation strategies. We also compare with other the phrase generation w.r.t. the dependency parsing trees. The details of such strategy are shown in section 4.3

---

Under review as a conference paper at ICLR 2020
2.3 MUTLI-HEAD ATTENTION OVER ALL NODE PAIRS
The attention over all node pairs in the first phase works similar to the standard Trans- former (Vaswani et al. 2017). Given sequence of vectors H Rat", where m is the number of hypernodes, we compute the attention of all nodes by:
QKT )V
K) softmart Vd
where Q = HWQ, K = HW* V = HWV, and Wº, W WV are trainable parameters.
By following Transformer (Vaswani et al. 2017). we use multi- head attention to jointly attend to different representation subspaces:
ealli = Concatt ... heeadd heads)WWo
where head, = Alnn(HW HWK, HWW). ), the number of heads.
2.4 NON-LINEAR ATTENTION WITHIN EACH PHRASE
The attention within each phrase captures the inductive bias of the phrase. For example, we use the semantics of "representation", "learning" to update the semantics phrase "representation learn- ing", , and vice versa. We consider two nodes have an edge between them if the words of one node the subset of the words of another node. More specific, for two nodes $1, $2, we consider their attention within phrase iff
word(s1) S word($2) or word((2) S Word(s1)
(4)
Given sequence of vectors H E IR*+,, the attention within phrases are computed by: QKT
Altn' (Q, K, V) assoftmazt
A)V)
(5)
Vd
where Q HWV K HWK', V HW denotes the Hadamard product, A [0, (](+m))((+m)) denotes the adjacency of the nodes:
{1 if word(s1) S word(s2) or word(s2) C word(s1)
A..
(6)
otherwise.
Note that we use an active function a in Attn' K. V), which is not used in the standard Trans- former. We found that the literal meaning of each individual words can be quite different from the implied meaning of the phrase. For example, in figure Ta the semantics of "the last straw" is not simple combination of the semantics of "the", "last", "straw". So using weighted sum of the three words to represent the phrase cannot precisely represent its semantics. We use the non-linearity to represent the semantic mutation. this paper, we use the sigmoid function as a.
We also use multi-head attention in the attention within phrases to jointly attend to different repre- sentation subspaces::
MultiHHead' (H, A) Concat(headd, head", W
where head' = Altn' KW. VW.").
2.5 ALGORITHM
The overall update algorithm of k-layer Phrase Transforrre is shown in Algorithm [ We are given the initial states of the words
[gnxd first generate m phrases and the adjacent matrix A. We extend the states of nodes H E m(++m)xd by padding m zeros. Then we iteratively perform the attention over all node pairs and the attention within phrases k times. The output is the states of the first n nodes, which correspond to the original words of the sentence.
Complexity analysis The standard Transformer considers the attention over nodes. Thus its complexity is (n)d)). Similarly, the first phase of PhraseTransforee pairs) is O(1 + m))d). since it consider the attention over all + nodes. The second phase of PhraseTransfar Transformer (i.e. attention within phrases) considers attentions within each phrase. Since the attention within phrases is sparse than attention over all nodes. overall complexity is O((n + dd = O(k2n2d)) Such complexity is acceptable when is small. In the experiments, we set k 2 or 3.
(i.e. attention over node
4

---

Under review as a conference paper at ICLR 2020
Algorithm The update algorillun Pransserran
Data: H. .r;
//Generate the hypernodes and init the states to zeros;
Generate m hypernodes and A;
Ho <-Pad m x d zeros to H;;
for 1...k do
//Attention over all node pairs:
H; <- MultiHead H::--
AAtteetion within phrases:
Multi ead (Hs, A);
return Ha]: m,,
RELATED WORK
Attention-based Natural Language Representation There is trend to use attention and self- attention BBahdanau (2015); Vaswani et al. (2017) for modeling the natural language. Attention is able to jointly model the semantics of the words and their alignments Bahdanau et al. (2015) or compositions Vaswani et al. 2017). It captures compositions of all word pairs, which superior to traditional sequential representation (e.g. LSTM) in many tasks Wang et al. (2018). However, previ- ous attention mechanisms work over the word level. They are infeasible to capture the semantics of phrases, especially if the semantics of the phrase is different from the literal semantics of its words. Instead, our proposed Phrase Transformer solves such issue by introducing hypernodes to represent phrases in the attention.
Attention-baaee pre-trainnn vre-traniin language models achieve state-of-the-ar results in many NLP tasks. BERT Devlin al. (2019), GPT-2 Radford et al. (2019), Roberta Liu et al. (2019) use multi-layer Transformers as the major network structure. We believe that our proposed Phrase' PhraseTransfone Transformer is good alternative of Transformer for pre-training models, because using phrase-based knowledge is more intuitive than the word-based knowledge. On the hand, as TrrrassTranffommrr has more trainable parameters than Transformer, the large amount of training data in pre-training guarantees these parameters can be trained sufficiently.
Incorporating Inductive Bias in Natural Language Representation is another trend in NLP. These inductive bias are usually from semantic dependencies SSrubell et al. (2018); Zhang et al. [Fu8); et al. (2019) or other external knowledge. LISA (Stubelll et al. (201 8) introduce the de- pendency parsing inductive bias to Transformer. It only allows the information transfer through edges of the parsing tree. C-GCN Zhang et al. (2018) and GraphRel Fu et all (2019) uses graph convolution networks, in which the edges are from the dependency tree, to represent such induc- tive bias. These approaches all relies on good algorithm (e.g. dependency parsing) for the given domain. This is usually unsatisfied for different languages and different domains. In contrast, approach do not rely such knowledge to represent the compositions.
4 EXPERIMENTS
In this section, we evaluate our proposed approaches over two classical NLP tasks: machine trans- lation and POS tagging, All the experiments run over computer with Intel Core i7 4.0GHz CPU, 96GB RAM, and a Geforce GTX 2080 GPU.
4.1 MACHINE TRANSLATION
Hyper-parameters We set the dimension of hidden states as 512, the number of heads as 8, and the dimension of each head as 64. We set the number of layers to 6.
Dataset We use the WMT' 16 multimodal translation: Multi30k (de-en), which consists of 29k samples for training, 10k samples for development, and 1K samples for testing.
5

---

Under review as a conference paper at ICLR 2020
Neural network details: Each layer of the Transformer has two self-attention modules: one for the encoder, and one for the decoder. We apply the PhraseTranstne Transformer to the self attention module in the encoder layer.
Model
BLEU
Transformer
20.90
Linear PhraseTTanstonee Transtormer (k=2)
21.60
Linear PhraseTranssforee Transformer (k=3)
20.95
TrrrsssTrans (k=2)
34.61
Phrase Transformer (k=3)
34.03
Table 1: Results over WMT" 16 multimodal translation.
The results of proposed approaches shown in table L The PhraseTransformec outperforms Transformer by a large margin. This shows that the intuition of representing natural language via phrases works.
Effectiveness of the non-linearity We want to figure out what make the Phrase Transformer works so well. Note that Transforree uses non-linear function in the attention within phrases. We further verify the effectiveness of such non-linearity by using a linear competitor, which is denoted as i nnaa PhraseTres former. It works the same PhraseTranssfommer, except that it does not use the activate function equation equation We report the results of the linear Phrase Transformer in [able
The comparison clearly suggests that most improvement of the PhraseTransformer is obtained from the non-linear activation. This intuitive because we want to use the non-linearity represent the semantic mutation in phrases.
4.2 VISUALIZZTIION OF RANAAFRRRAN
We visualize the attention matrix of PhraseTTaan Transformer the self-attention module for WMT' multimodal translation in figure 3 We analyze the attention over these phrases. As can be seen, the attention is over the single nodes and the hypernodes. This verifies that the successfully use phrases as atoms in attention. For example, in figure 3a we can see that the phrase "a building" has a higher attention score than other phrases. This makes sense because the two words form valid phrase. In the attention matrix for attention within phrases figure we found that the word "on" attends to the phrase "working on". This is intuitive because the work "on" itself does not have clear semanties. needs the collocation with "working" represent semantics.
4.3 POS TAGGING
Hyper-parameters We follow the parameters in Guo al. (2019). We the dimension hidden states as 300, the number of heads as 6, and the dimension of each head as 50. We set the number of layers to 2.
Dataset We use the POS tagging dataset from Penn Treehank (PTB) Marcus et al. 1993), which consists of 38k samples for training. samples for development, and 5k samples for testing.
Competitor with semantic inductive bias Noticing that the semantic dependency is also possible inductive bias (Strubell al., 2018) Zhang et al., (018), we also compare with Phrase Transformer with dependency parsing trees. We denote it semantic
Each hypernode in it represents an edge in the dependency parsing tree. In this way, the non-adjacent words can form one phrase. For example, is possible to represent the phrase "not only ... but also" in figure Ic More formally, in semantic ChraseTransformee we generate hypernode s such that
word(s) (s) (w;, w,) s.t. edge (w, wy) is in the dependency parsing tree
(8)
this paper, we use spaCy (Honnibal Montani 2017) for dependency parsing. The two phases in the semantic Phrase Transformer works the same in Phrase Transforrans.rmmer..

---

Under review as a conference paper at ICLR 2020
1.0
guy
works
building
</5>
guy
0.4
works
works
on a
building
0.2
building....




0.0


guy works
on
building

 <5> a guy guy works works
on on building
building
</5>>
</S>
(a) Attention matrix for attention over all nodes.
1.0
guy
works
on
a
building
0.6

>
guy
0.4
works
works
on
building
0.2
building.




0.0
<5>
guy works on
building
</>> <5> guy guy works works on on building
building

>
</57
(b) Attention matrix for attention within phrases.
Figure 3: Attention matrix visualization.
The results are shown in table 2 The PhraseTransformer outperforms Transformer and semantic PhraseTrannffrmea Note that semantic PhraseTTransforee uses external data to incorporate induc- tive bias. This verifies incorporate simple but suitable inductive bias, i.e., the phrases.
CONCLUSION
We noticed that representing phrase as the atoms in attention is more intuitive than the word-to- word attention. Wee proposed the Transforrre, which incorporates the hypernodes in Transformer. Each hypernode represents possible phrase in the sentence. Without using extra phrase identifi-

---

Under review as a conference paper at ICLR 2020
Model
use external data
accuracy
Transformer
96.36
Semantic Phrase'Trame Transformer (k=2)
96.35
Semantic Phrase Transformer (k=3)
96.23
Linear PhraseTranstonne Transformer (k=2)
96.54
Linear Phrase PhraseTransii Transformer (k=3)
96.52
Phrase Transformer (k=2)
96.87
Phrase Transformer (k=3)
96.80
Table 2: Results over PTB POS tagging.
cation algorithms, jointly detect the phrases and attend them. Our proposed PhraseTrans- former has two phases. The first phase achieves phrase level attentions like the word level attention. The second phrase represents the inductive bias between words and phrases. The experiments show a significant improvement in the machine translation task.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Jsi Fu, Peng-Hsuaa Li, and Wei-Yun Ma. Graphrel: Modeling text relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1409-1418, 2019.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star- transformer. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1315-1325, 2019.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. 1993.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. Linguistically- informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 5027-5038, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp. 353, 2018.
8

---

Under review as a conference paper at ICLR 2020
Yuhao Zhang, Peng Qi, and Christopher D Manning. Graph convolution pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2205- 2018.


Reviews:
[{"review_id": "ryg1sde8qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the paper needs a deep review in the English part. For example, in the abstract, they repeat \"In this paper\" a couple of time and it is complicated to understand the introduction and methodology. Also, I think the paper needs a better structure. The related work should be first in order to understand the relevance of this paper. \n\nFrom the experiment point of view, it is necessary a better explanation about the hyperparameters or the experiments which were carried out. In addition, the single database was used to evaluate the technology which is not enough to show the big different respect to the transformed paper. Also, the comparison is not really fair. In each \"layer\" of the proposed phrase transformer, it has actually two self-attention layers, but the baseline has only one self-attention layer. In addition more methodologies should be necessary to compare the results of the experiment. \n\nThe architecture part is complicated to follow and I don't understand the big contribution of this paper. \n\nFor that reason, I recommend a reject the paper and work more for the final version ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxkd5v0Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses an issue of compositionality in self-attention models such Transformer. A simple idea of composing multiple words into a phrase as a hypernode and representing it using a non-linear function to capture the semantic mutation is proposed. In the machine translation and PoS tagging tasks, the proposed PhraseTransformer achieves impressive gain, especially +13.7 BLEU score compared to the Transformer.\n\nThe motivation of the paper is very clear, and I love this kind of paper; with a simple idea, making a huge impact on the field. I appreciate the real example to compare how word-level self-attention is different from the phrase-level self-attention in Abstract and Figure 1. The problem itself; tackling the semantic compositionality of self-attention, is a very important problem, and I like the part that authors described it as an inductive bias as a model perspective. \n\nHowever, this work seems to be problematic in terms of presentation, clarity, and meaningful comparisons. Please see my detailed comments below.\n\nFirst, what exactly is “semantic mutation”? The term has been used here and there to describe the inductive bias in semantic compositionality and to show how the nonlinearity can effectively capture it. But, I couldn’t find any definition from the paper, couldn’t find any formal definition from any ACL papers, and couldn’t guess myself based on the context. I am guessing it is probably some sort of combination of meaning in phrase-level words. If so, more importantly, how does the simple non-linear function (i.e., sigmoid) can capture such semantic combinations of words? How could it make such a huge gain (PhraseTransformer vs Linear PhraseTransformer) in Table 1 on MT task? This seems to be the most important contribution of this paper, of which I don’t understand yet. \n\n\nWhat is the “concept of hypernodes in hypergraph”? I think it is not a common word that I can understand it clearly without any references or any background. It would be better to add some references for the concept. Again, I am guessing it is a sort of graph theory that decomposes a large node into small pieces but keeps their connectivity. But, then how is that exactly linked to phrases of words? If you make phrases only on consecutive words, it is basically just chunking. I don’t find any relevance of phrases in a sentence with the (hyper)graph something. \n\nIn Figure 2, how is the bidirectional path made between the word representations and phrase representations? In my understanding of your algorithm based on Equation 2-6, I only see the attention of phrases is computed by word attention, but not the other way.  Please clarify this. If so, how does the gradient back-propagate to each other?\n\nThe biggest concern of this work is the scores reported in Table 1. I have checked the recent papers which used the Mutli30K(de-en) and other results from WMT[16-18] reports, but the BLEU score reported in Table 1 (20.90) seems way lower than the scores reported by any systems trained by either non-Transformer or Transformer systems. For that reason, it would be fair to include some results from the state-of-the-art systems on the same dataset. \n\nA minor point but in the complexity analysis, your m is basically n because you take consecutive words from n length of sentence. You better distinguish which variables are dependent on each other first. \n\n\nThere are MANY typos, missing captions, grammatical errors in the paper. Here are only some of them:\nThere is no caption for Figure 1. \nWhen you cite a reference with its model name, you better make the citation under parenthesis. \n“equation equation 5” -? “equation 5”\n“Wee” -> “We”\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxidjcRtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes to consider to put attention on \"phrases\" in NLP. The phrases are generated by taking consecutive words in sentences. Each phrase is treated as a \"node\" in the same way as words. Then representations of phrases are learned in the network. The algorithm is applied to two applications, translation and pos tagging. The proposed method achieved better performance than transformer. \n\nCritics: \n\n1. In the abstract and the introduction, the submission argues that usefulness of phrases, which are sementic units represented by word groups. However, in the model development, \"phrases\" are really bigrams and trigrams. I don't know how much the previous argument is still valid. Particularly, there are so many bigrams and trigrams. The effect from these word combinations should have strong effect on the model, but the effect may not be explained as the argument. \n\n2. I think transformer can somewhat capture word combinations in bigrams and trigrams. In higher layers, transformer actually combine words in representations. What is the advantage of the proposed method over the type of combination done in transformer? \n\n3. The experiment only compares to transformer in the translation task. It only compares to transformer and semantic phrase transformer. Other SOTA methods (e.g. different versions of transformers) should be compared. \n\n4. The comparison is not really fair. In each \"layer\" of the proposed phrase transformer, it has actually two self-attention layers, but the baseline has only one self-attention layer. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper incorporates phrases within the transformer architecture.

The underlying idea is interesting, but the reviewers have raised serious concerns with both clarity and the trustworthiness of the experimental evaluation, and thus I cannot recommend acceptance at this time.
Decision: Reject