Attention over Phrases

semantics = α1Emb ( That ) + α2Emb ( ’ s ) + · · ·+ αnEmb ( her ) . But the weighted sum of “ That ’ s ” , “ the ” , “ last ” , “ straw ” can hardly represent the semantics of the phrase . We argue that the phrases play an important role in attention . If we combine some words into phrases , a more reasonable representation with compositions is semantics = α1Emb ( That ’ s ) +Emb2 ( the last straw ) +α3Emb ( for ) +α4Emb ( her ) . While recent studies prefer to use the attention mechanism to represent the natural language , few noticed the word compositions . In this paper , we study the problem of representing such compositional attentions in phrases . In this paper , we proposed a new attention architecture called PhraseTransformer . Besides representing the words of the sentence , we introduce hypernodes to represent the candidate phrases in attention . PhraseTransformer has two phases . The first phase is used to attend over all word/phrase pairs , which is similar to the standard Transformer . The second phase is used to represent the inductive bias within each phrase . Specially , we incorporate the non-linear attention in the second phase . The non-linearity represents the the semantic mutations in phrases . The experimental performance has been greatly improved . In WMT16 English-German translation task , the BLEU increases from 20.90 ( by Transformer ) to 34.61 ( by PhraseTransformer ) . 1 INTRODUCTION . The word-to-word attention ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) updates the state of a word by the weighted sum of the current states of the words . The weights depend on the attention scores . Such mechanism incurs two major problems . Firstly , it is unfeasible to incorporate the semantics of phrases in the attention . For example , in figure 1a and figure 1b , using the attention over phrase “ the last straw ” is more intuitive than using the attention over the three individual words . However , such word composition ( i.e . “ the last straw ” ) can not be represented atomically by word level attentions . Secondly , the literal meaning of each individual words can be quite different from the implied meaning of the phrase . The weighted sum in attention can not capture the semantic mutation . For example , it ’ s hard to represent the semantics of the unpleasantness , the struggling , and being unable to continue by the weighted sum of individual words “ the ” , “ last ” , and “ straw ” . Deepening the attention layers as in Devlin et al . ( 2019 ) ; Liu et al . ( 2019 ) can not solve these issues . Instead of representing the natural language via word-to-word attentions , incorporating phrase as atoms in the attention provides a more intuitive representation . In figure 1b , if we regard “ the last straw ” as an atom node in the attention in figure 1b , it is much more intuitive to represent the semantics . For multiple words that form semantic compositions ( e.g . idioms , chunkings , phrases ) , their semantics should be computed uniformly as an atom in the attention mechanism . The traditional word level attention mechanism should be expanded to represent the attentions over such compositions . To this end , we make a very simple framework called PhraseTransformer , based on Transformer . We leverage the concept of hypernodes in hypergraph . The traditional Transformer only consists of That ’ s the last straw for her . semantics = α1Emb ( That ’ s ) + α2Emb ( the ) + · · ·+ αnEmb ( her ) ( a ) Word level self-attention That ’ s the last straw for her . semantics = α1Emb ( That ’ s ) + α2Emb ( the last straw ) + α2Emb ( for ) + αnEmb ( her ) ( b ) Attention over adjacent phrases . Representation learning is not only popular but also important . semantics =α1Emb ( Representation learning ) + α2Emb ( is ) + α3Emb ( not only ... but also ) + α3Emb ( hot ) + α4Emb ( important ) ( c ) Attention over nonadjacent phrases . nodes representing single words . From the perspective of graph , Transformer is a complete bipartite graph , that the information transfers through all node pairs according to their attention scores . In hypergraph , a hypernode consists of one or more nodes . Likewise , we use such hypernodes in PhraseTransformer to represent the semantics of the composition . Each hypernode represents multiple words in a composition . The information propagation in PhraseTransformer has two stages : the propagation across arbitrary node pairs and the propagation within a hypernode . Attention of arbitrary node pairs works similar to the classical Transformer . We compute the attention scores for all nodes and all hypernodes . The information propagates accordingly . We use hypernodes to represent the atomic information propagation of semantic compositions in attentions . Non-linear attention within each phrase reflects the inductive bias of the phrases . The nodes within a phrase should be updated by each other via attention . For example , the states of the hypernode the last straw should be updated by the node of the , last , straw etc . And the states of node straw should also be updated by the node of the last straw . To do this , we limit the attention in Transformer to nodes within each phrase . Compared to the attention of arbitrary node pairs , a significant difference of the attention within each phrase is that we incorporate nonlinearity . As suggested in figure 1a , the literal meaning of each individual words can be quite different from the implied meaning of the phrase . We use the non-linearity to represent the semantic mutations of phrases . Our proposed PhraseTransformer is able to capture any multi-word compositions . In this paper , we mainly study one of the simplest form of phrase : a sequence of adjacent words . For detecting phrases with adjacent words , we do not utilize existing identification algorithms ( e.g . chunking ) . We roughly added all sequences of words in the sentence whose length is less than a given threshold . The attention mechanism itself decides the weights of these candidate phrases by assigning them different attention scores during the inference . We summarize the contributions of this paper below : • We propose to represent semantic compositions in the attention mechanism . We show that such representation is more intuitive than the traditional word-to-word attention . • We incorporate the hypernodes in Transformer as PhraseTransformer . The semantics of a composition can be computed atomically via cross-node attentions . The inductive bias is reflected by the information propagation within each hypernode . • We evaluate our approach over two widely used types of semantic compositions , i.e. , chunking and semantic dependency . We conduct extensive experiments to verify the effectiveness of our proposed approaches . 2 MODEL . 2.1 ARCHITECTURE . In this subsection , we show how the PhraseTransformer works on an abstract level . In the attention , we use “ simple node ” and “ hypernode ” to denote nodes that correspond to a single word and a phrase , respectively . We denote both the simple nodes and hypernodes as nodes . The PhraseTransformer contains n simple nodes , where n is the length of the sentence , and m hypernodes representing m phrases . For each node s , we denote the words it corresponds to as word ( s ) . For simple nodes , |word ( s ) | = 1 . For hypernodes , |word ( s ) | > 1 . For example , in figure 2 , “ straw ” is represented by a single node s1 . word ( s1 ) = ( straw ) . The phrase “ the last straw ” is represented by a hypernode s2 , word ( s2 ) = ( the , last , straw ) . The attention in PhraseTransformer has two phases . The first phase uses the node-to-node attention over all node pairs . This is represented by the blue edges in figure 2 . For example , the state of the placeholder [ CLS ] will be updated according to its attention scores to nodes of “ That ’ s ” , “ for ” , “ the last straw ” , etc . Similarly , the states of a hypernode ( e.g . “ the last straw ” ) will be updated according to its attention scores to all nodes ( e.g . “ That ’ s ” , “ for ” , “ last straw ” ) . The second phase uses limited attention mechanism to update the states within each phrase . In figure 2 , the orange edges show the attention within the phrase “ the last straw ” . The state of s2 will be updated according to its attention scores to “ the ” , “ last ” , “ straw ” , but will not be updated by “ for ” or “ her ” . And the state of s1 will also be updated by s2 . 2.2 PHRASE GENERATION . As mentioned in the introduction section , a hypernode represents a phrase formed by multiple words . In this paper , we consider the phrases formed by adjacent words in the original sentences . For a given length threshold k and a word sequence w1 , · · · , wn , we generate hypernodes that consists of adjacent word sequence whose length is less than or equal to k. More specifically , we generate all hypernode s such that word ( s ) = ( wi , wi+1 , · · ·wj ) , s.t . j − i < k ( 1 ) Our proposed neural network is feasible for any phrase generation strategies . We also compare with other the phrase generation w.r.t . the dependency parsing trees . The details of such strategy are shown in section 4.3 . 2.3 MUTLI-HEAD ATTENTION OVER ALL NODE PAIRS . The attention over all node pairs in the first phase works similar to the standard Transformer ( Vaswani et al. , 2017 ) . Given a sequence of vectors H ∈ Rn+m , where m is the number of hypernodes , we compute the attention of all nodes by : Attn ( Q , K , V ) = softmax ( QKᵀ√ d ) V ( 2 ) , where Q = HWQ , K = HWK , V = HWV , and WQ , WK , WV are trainable parameters . By following Transformer ( Vaswani et al. , 2017 ) , we use multi-head attention to jointly attend to different representation subspaces : MultiHead ( H ) = Concat ( head1 , · · · , headh ) WO ( 3 ) , where headi = Attn ( HW Q i , HW K i , HW V i ) , h is the number of heads . 2.4 NON-LINEAR ATTENTION WITHIN EACH PHRASE . The attention within each phrase captures the inductive bias of the phrase . For example , we use the semantics of “ representation ” , “ learning ” to update the semantics of phrase “ representation learning ” , and vice versa . We consider two nodes have an edge between them if the words of one node is the subset of the words of another node . More specific , for two nodes s1 , s2 , we consider their attention within a phrase iff word ( s1 ) ⊆ word ( s2 ) or word ( s2 ) ⊆Word ( s1 ) ( 4 ) . Given a sequence of vectors H ∈ Rn+m , the attention within phrases are computed by : Attn′ ( Q , K , V ) = σ ( softmax ( QKᵀ√ d ◦A ) V ) ( 5 ) where Q = HWQ ′ , K = HWK ′ , V = HWV ′ , ◦ denotes the Hadamard product , A ∈ [ 0 , 1 ] ( n+m ) × ( n+m ) denotes the adjacency of the nodes : Ai , j = { 1 if word ( s1 ) ⊆ word ( s2 ) or word ( s2 ) ⊆ word ( s1 ) 0 otherwise . ( 6 ) Note that we use an active function σ in Attn′ ( Q , K , V ) , which is not used in the standard Transformer . We found that the literal meaning of each individual words can be quite different from the implied meaning of the phrase . For example , in figure 1a , the semantics of “ the last straw ” is not a simple combination of the semantics of “ the ” , “ last ” , “ straw ” . So using a weighted sum of the three words to represent the phrase can not precisely represent its semantics . We use the non-linearity to represent the semantic mutation . In this paper , we use the sigmoid function as σ . We also use multi-head attention in the attention within phrases to jointly attend to different representation subspaces : : MultiHead′ ( H , A ) = Concat ( head′1 , · · · , head′h ) WO ( 7 ) where head′i = Attn ′ ( QWQ ′ i , KW K′ i , V W V ′ i ) .

Summary: This submission proposes to consider to put attention on "phrases" in NLP. The phrases are generated by taking consecutive words in sentences. Each phrase is treated as a "node" in the same way as words. Then representations of phrases are learned in the network. The algorithm is applied to two applications, translation and pos tagging. The proposed method achieved better performance than transformer.