Attention over Phrases

semantics = α1Emb ( That ) + α2Emb ( ’ s ) + · · ·+ αnEmb ( her ) . But the weighted sum of “ That ’ s ” , “ the ” , “ last ” , “ straw ” can hardly represent the semantics of the phrase . We argue that the phrases play an important role in attention . If we combine some words into phrases , a more reasonable representation with compositions is semantics = α1Emb ( That ’ s ) +Emb2 ( the last straw ) +α3Emb ( for ) +α4Emb ( her ) . While recent studies prefer to use the attention mechanism to represent the natural language , few noticed the word compositions . In this paper , we study the problem of representing such compositional attentions in phrases . In this paper , we proposed a new attention architecture called PhraseTransformer . Besides representing the words of the sentence , we introduce hypernodes to represent the candidate phrases in attention . PhraseTransformer has two phases . The first phase is used to attend over all word/phrase pairs , which is similar to the standard Transformer . The second phase is used to represent the inductive bias within each phrase . Specially , we incorporate the non-linear attention in the second phase . The non-linearity represents the the semantic mutations in phrases . The experimental performance has been greatly improved . In WMT16 English-German translation task , the BLEU increases from 20.90 ( by Transformer ) to 34.61 ( by PhraseTransformer ) . 1 INTRODUCTION . The word-to-word attention ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) updates the state of a word by the weighted sum of the current states of the words . The weights depend on the attention scores . Such mechanism incurs two major problems . Firstly , it is unfeasible to incorporate the semantics of phrases in the attention . For example , in figure 1a and figure 1b , using the attention over phrase “ the last straw ” is more intuitive than using the attention over the three individual words . However , such word composition ( i.e . “ the last straw ” ) can not be represented atomically by word level attentions . Secondly , the literal meaning of each individual words can be quite different from the implied meaning of the phrase . The weighted sum in attention can not capture the semantic mutation . For example , it ’ s hard to represent the semantics of the unpleasantness , the struggling , and being unable to continue by the weighted sum of individual words “ the ” , “ last ” , and “ straw ” . Deepening the attention layers as in Devlin et al . ( 2019 ) ; Liu et al . ( 2019 ) can not solve these issues . Instead of representing the natural language via word-to-word attentions , incorporating phrase as atoms in the attention provides a more intuitive representation . In figure 1b , if we regard “ the last straw ” as an atom node in the attention in figure 1b , it is much more intuitive to represent the semantics . For multiple words that form semantic compositions ( e.g . idioms , chunkings , phrases ) , their semantics should be computed uniformly as an atom in the attention mechanism . The traditional word level attention mechanism should be expanded to represent the attentions over such compositions . To this end , we make a very simple framework called PhraseTransformer , based on Transformer . We leverage the concept of hypernodes in hypergraph . The traditional Transformer only consists of That ’ s the last straw for her . semantics = α1Emb ( That ’ s ) + α2Emb ( the ) + · · ·+ αnEmb ( her ) ( a ) Word level self-attention That ’ s the last straw for her . semantics = α1Emb ( That ’ s ) + α2Emb ( the last straw ) + α2Emb ( for ) + αnEmb ( her ) ( b ) Attention over adjacent phrases . Representation learning is not only popular but also important . semantics =α1Emb ( Representation learning ) + α2Emb ( is ) + α3Emb ( not only ... but also ) + α3Emb ( hot ) + α4Emb ( important ) ( c ) Attention over nonadjacent phrases . nodes representing single words . From the perspective of graph , Transformer is a complete bipartite graph , that the information transfers through all node pairs according to their attention scores . In hypergraph , a hypernode consists of one or more nodes . Likewise , we use such hypernodes in PhraseTransformer to represent the semantics of the composition . Each hypernode represents multiple words in a composition . The information propagation in PhraseTransformer has two stages : the propagation across arbitrary node pairs and the propagation within a hypernode . Attention of arbitrary node pairs works similar to the classical Transformer . We compute the attention scores for all nodes and all hypernodes . The information propagates accordingly . We use hypernodes to represent the atomic information propagation of semantic compositions in attentions . Non-linear attention within each phrase reflects the inductive bias of the phrases . The nodes within a phrase should be updated by each other via attention . For example , the states of the hypernode the last straw should be updated by the node of the , last , straw etc . And the states of node straw should also be updated by the node of the last straw . To do this , we limit the attention in Transformer to nodes within each phrase . Compared to the attention of arbitrary node pairs , a significant difference of the attention within each phrase is that we incorporate nonlinearity . As suggested in figure 1a , the literal meaning of each individual words can be quite different from the implied meaning of the phrase . We use the non-linearity to represent the semantic mutations of phrases . Our proposed PhraseTransformer is able to capture any multi-word compositions . In this paper , we mainly study one of the simplest form of phrase : a sequence of adjacent words . For detecting phrases with adjacent words , we do not utilize existing identification algorithms ( e.g . chunking ) . We roughly added all sequences of words in the sentence whose length is less than a given threshold . The attention mechanism itself decides the weights of these candidate phrases by assigning them different attention scores during the inference . We summarize the contributions of this paper below : • We propose to represent semantic compositions in the attention mechanism . We show that such representation is more intuitive than the traditional word-to-word attention . • We incorporate the hypernodes in Transformer as PhraseTransformer . The semantics of a composition can be computed atomically via cross-node attentions . The inductive bias is reflected by the information propagation within each hypernode . • We evaluate our approach over two widely used types of semantic compositions , i.e. , chunking and semantic dependency . We conduct extensive experiments to verify the effectiveness of our proposed approaches . 2 MODEL . 2.1 ARCHITECTURE . In this subsection , we show how the PhraseTransformer works on an abstract level . In the attention , we use “ simple node ” and “ hypernode ” to denote nodes that correspond to a single word and a phrase , respectively . We denote both the simple nodes and hypernodes as nodes . The PhraseTransformer contains n simple nodes , where n is the length of the sentence , and m hypernodes representing m phrases . For each node s , we denote the words it corresponds to as word ( s ) . For simple nodes , |word ( s ) | = 1 . For hypernodes , |word ( s ) | > 1 . For example , in figure 2 , “ straw ” is represented by a single node s1 . word ( s1 ) = ( straw ) . The phrase “ the last straw ” is represented by a hypernode s2 , word ( s2 ) = ( the , last , straw ) . The attention in PhraseTransformer has two phases . The first phase uses the node-to-node attention over all node pairs . This is represented by the blue edges in figure 2 . For example , the state of the placeholder [ CLS ] will be updated according to its attention scores to nodes of “ That ’ s ” , “ for ” , “ the last straw ” , etc . Similarly , the states of a hypernode ( e.g . “ the last straw ” ) will be updated according to its attention scores to all nodes ( e.g . “ That ’ s ” , “ for ” , “ last straw ” ) . The second phase uses limited attention mechanism to update the states within each phrase . In figure 2 , the orange edges show the attention within the phrase “ the last straw ” . The state of s2 will be updated according to its attention scores to “ the ” , “ last ” , “ straw ” , but will not be updated by “ for ” or “ her ” . And the state of s1 will also be updated by s2 . 2.2 PHRASE GENERATION . As mentioned in the introduction section , a hypernode represents a phrase formed by multiple words . In this paper , we consider the phrases formed by adjacent words in the original sentences . For a given length threshold k and a word sequence w1 , · · · , wn , we generate hypernodes that consists of adjacent word sequence whose length is less than or equal to k. More specifically , we generate all hypernode s such that word ( s ) = ( wi , wi+1 , · · ·wj ) , s.t . j − i < k ( 1 ) Our proposed neural network is feasible for any phrase generation strategies . We also compare with other the phrase generation w.r.t . the dependency parsing trees . The details of such strategy are shown in section 4.3 . 2.3 MUTLI-HEAD ATTENTION OVER ALL NODE PAIRS . The attention over all node pairs in the first phase works similar to the standard Transformer ( Vaswani et al. , 2017 ) . Given a sequence of vectors H ∈ Rn+m , where m is the number of hypernodes , we compute the attention of all nodes by : Attn ( Q , K , V ) = softmax ( QKᵀ√ d ) V ( 2 ) , where Q = HWQ , K = HWK , V = HWV , and WQ , WK , WV are trainable parameters . By following Transformer ( Vaswani et al. , 2017 ) , we use multi-head attention to jointly attend to different representation subspaces : MultiHead ( H ) = Concat ( head1 , · · · , headh ) WO ( 3 ) , where headi = Attn ( HW Q i , HW K i , HW V i ) , h is the number of heads . 2.4 NON-LINEAR ATTENTION WITHIN EACH PHRASE . The attention within each phrase captures the inductive bias of the phrase . For example , we use the semantics of “ representation ” , “ learning ” to update the semantics of phrase “ representation learning ” , and vice versa . We consider two nodes have an edge between them if the words of one node is the subset of the words of another node . More specific , for two nodes s1 , s2 , we consider their attention within a phrase iff word ( s1 ) ⊆ word ( s2 ) or word ( s2 ) ⊆Word ( s1 ) ( 4 ) . Given a sequence of vectors H ∈ Rn+m , the attention within phrases are computed by : Attn′ ( Q , K , V ) = σ ( softmax ( QKᵀ√ d ◦A ) V ) ( 5 ) where Q = HWQ ′ , K = HWK ′ , V = HWV ′ , ◦ denotes the Hadamard product , A ∈ [ 0 , 1 ] ( n+m ) × ( n+m ) denotes the adjacency of the nodes : Ai , j = { 1 if word ( s1 ) ⊆ word ( s2 ) or word ( s2 ) ⊆ word ( s1 ) 0 otherwise . ( 6 ) Note that we use an active function σ in Attn′ ( Q , K , V ) , which is not used in the standard Transformer . We found that the literal meaning of each individual words can be quite different from the implied meaning of the phrase . For example , in figure 1a , the semantics of “ the last straw ” is not a simple combination of the semantics of “ the ” , “ last ” , “ straw ” . So using a weighted sum of the three words to represent the phrase can not precisely represent its semantics . We use the non-linearity to represent the semantic mutation . In this paper , we use the sigmoid function as σ . We also use multi-head attention in the attention within phrases to jointly attend to different representation subspaces : : MultiHead′ ( H , A ) = Concat ( head′1 , · · · , head′h ) WO ( 7 ) where head′i = Attn ′ ( QWQ ′ i , KW K′ i , V W V ′ i ) .

Summary: This paper addresses an issue of compositionality in self-attention models such Transformer. A simple idea of composing multiple words into a phrase as a hypernode and representing it using a non-linear function to capture the semantic mutation is proposed. In the machine translation and PoS tagging tasks, the proposed PhraseTransformer achieves impressive gain, especially +13.7 BLEU score compared to the Transformer.