# Selectively Connected Self-Attentions for Semantic Role Labeling
## 1 Introduction  
Semantic representation is important for the machine to understand the meaning of data. Semantic role labeling (SRL) is a task of constructing a dependency structure to represent the semantics in natural language. In specific, it assigns pre-defined labels, called _semantic roles_, about 'when', 'who', 'what to whom' or 'where', to the non-predicate words dependent to predicates. The definition of semantic roles have been well posed in the literature [1; 2]. The aim of SRL is accurately predicting the labels among verbs and other words given a sentence. SRL studies have been applied to benefit many natural language processing (NLP) applications, such as question answering [3] machine translation [4], and many others [5]. Many traditional SRL approaches have used the shallow models that rely on the syntactic features. With the recent development of deep learning [6], more and more SRL studies using deep neural networks improve state-of-the-art approaches only with large training data sets. For example, recurrent neural networks (RNN) have shown satisfactory results [7] in SRL owing to the _sequential nature_ of natural language data. The variants of RNN using long short-term memory (LSTM) are popularly used in many NLP applications [8] due to their great performance in training sequential data. The LSTMs, which enable the RNNs to memorize selectively, have been regarded as viable modules that can learn the latent features over an alignment between the pair of long sentences. In SRL, several recent studies [9; 10; 11] have been adopted the LSTM-based networks to generate a sequence of labels, that is, semantic roles, corresponding to the input words. A seminal work [9] has introduced a stacked, bidirectional LSTM-based model for SRL, and has outperformed the existing studies. However, the existing approaches [9; 10; 11; 7] have several limitations. First, LSTM-based approaches overwrite a large number of parameters of the model during training. The updates of the hidden states may perform poorly when the training sequences are long. For example, the overwritten memory of anLSTM module may not maintain the features learned at the early stage until it processes the last word. Second, due to the inherent recurrence, the current hidden state is repetitively updated by the previous hidden state. Although this process makes the models learn the dependency through time, it prevents parallel computation. Third, training with temporal dependency over the sequence may lose some spatial constraints, for example, hierarchical structure over the words in a sentence. For example, a sequence of words, construct a phrase, and a sentence is made of phrases. It is worth noting that the hierarchical representation is an effective form of natural language. Although the existing SRL studies perform greatly, there still remain challenges in training long sequences and spatial constrains, such as hierarchical information. To address the problems, we present a novel deep architecture for SRL based on stacked attentive representations. The _attention mechanism_[12] has become a prevalent method in neural models for NLP tasks because of its impressive performance [6]. The attention mechanism works to focus the salient information over an input while generating an output. Our model is based on the self-attention mechanism [13] for sentence representation because we note that the self-attention is a proper method to capture the associations for SRL. Since the self-attention has been considered to derive the intra-connections among every word within a sentence, it has been popularly adopted in recent NLP studies [6]. The basis of our architecture is the encoder of the _transformer_[14] as in the prior work [15], which is similar to ours. However, we differ from the work [15] by stacking the self-attentions with the original module called, the _selective connection_. The module makes our model can capture the hierarchical information over the attentive representations, which are built on the self-attention modules, while training.