# Selectively Connected Self-Attentions for Semantic Role Labeling
## 3. Selectively Connected Attentive Representation
### Attentive Representation  
Basically, our model mainly relies on the self-attention mechanism [14] to capture the semantic roles of the words in a sentence. The self-attention has been successfully applied to draw the dependency structure within a single sequence. In recent studies [15; 19], the self-attention is widely used because it is not only avoiding recurrent updates during training, but also capturing the intra-relationship in sentences. The most relevant work [15] to ours has achieved state-of-the-art performance (without pre-trained embedding models) in 2017 by parallelizing the encoding and decoding processes by using multi head self-attention. Inspired by the prior work, we build the dependency structure featured for SRL by densely connecting the every word parts to each other. Figure 1 illustrates the multi head self-attention module.  
As a basis, the scaled dot-product attention is adopted to compute the attention score. The dotted block denoted as \(h\)-heads of self-attention depicts a single computation flow of the scaled dot-product attention. Also, two variants can be used for efficient computation: dot-product attention and additive attention. The dot-product attention utilizes matrix production, which allows faster computation [15]. Given a matrix of \(n\) query vectors \(\mathbf{Q}\in\mathbb{R}^{n\times d}\), keys \(\mathbf{K}\in\mathbb{R}^{n\times d}\) and values \(\mathbf{V}\in\mathbb{R}^{n\times d}\) for \(\mathbf{X}\), the scaled dot product attention computes the attention scores based on the following equation:  
\[a(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{ \sqrt{d}})\mathbf{V}, \tag{7}\]  
where \(d\) is the number of hidden units of the network. Without \(\sqrt{d}\), the score computes the dot-product attention. Additive function is just a simple concatenation.  
For multiple heads of the self-attention, we adopt linear projections by \(\mathbf{W}_{i}^{Q}\in\mathbb{R}^{n\times d/h}\), \(\mathbf{W}_{i}^{K}\in\mathbb{R}^{n\times d/h}\), \(\mathbf{W}_{i}^{V}\in\mathbb{R}^{n\times d/h}\), which correspond to queries, keys and values, respectively. The attention score \(a\) calculates the association between queries and keys for values to generate the attentive representation of the vectors \(X\). As a result, our model is incorporating multiple parallel channels, that is, \(h\)-multi heads \(m_{i}\), to represent the different learned semantics from the inputs:  
\[m_{i}=a(\mathbf{Q}\mathbf{W}_{i}^{Q},\mathbf{K}\mathbf{W}_{i}^{K},\mathbf{V} \mathbf{W}_{i}^{V}), \tag{8}\]  
To combine the multiple channels as a single representation, we concatenate them and apply linear projection via a feed forward layer as \(X^{\prime}=Concat(\mathbf{m}_{1},...,\mathbf{m}_{h})\times\mathbf{W}_{\mathbf{ A}}\). The Figure 2 illustrate the module called _attentive representation_ that makes the multi head self-attention module to be trainable. Each module contains the multi head self-attention sub-module and fully connected sub-layer, followed by a sub-layer for normalization. Also, we adopt residual connections around each of the sub-layers described as blocks. The number \(h\) of the parallel layers is given as a hyper-parameter. To deepen the model, we stack the attentive representation modules by \(N\) sequential layers. It is worth noting that the depth of our model affects the performance on labeling accuracy, which means the learned semantic representation would be more robust according to the depth. By adopting the multiple heads and the stacking, our model can capture the correlative relationships within a sentence effectively.  
Figure 1: Multi head self-attention.