# Selectively Connected Self-Attentions for Semantic Role Labeling
## 4 Experiments
### Experimental Results
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Development**} & \multicolumn{3}{c}{**OntoNotes Test**} \\ \cline{2-7}  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline Ours+ELMo & 82.9 & 81.6 & 81.9 & 83.2 & 83.8 & 83.6 \\ Ours & 81.8 & 81.2 & 81.3 & 81.4 & 82.4 & 82.8 \\ \hline [PERSON] et al. (2018) & 81.8 & 81.4 & 81.5 & 81.7 & 83.3 & 83.4 \\ [PERSON] et al. & 82.2 & 83.6 & 82.9 & 81.9 & 83.6 & 82.7 \\ [PERSON] et al. (2017) & 81.7 & 81.4 & 81.5 & 81.8 & 81.6 & 81.7 \\ [PERSON] and [PERSON] & - & - & 81.1 & - & - & 81.3 \\ [PERSON] et al. & 81.0 & 78.5 & 78.9 & 81.2 & 79.0 & 80.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental results on the CoNLL-2012. Table 3 summarizes \(F1\) values of our model ablations on the CoNLL 2005 development set. We ablate our full model by removing, (1) the BiLSTM layers (denoted as Seq), (2) the selective connection module (denoted as SC) and (3) the ELMo model (denoted as Emb). Without the selective connection, the stack of attentive representation is constructed only with the fully connected layer followed by layer normalization. As the increasing number of layers, the model achieves higher \(F1\) scores until it has 8 layers. We report the ablation result with the increasing numbers 4, 8 and 12 layers of the attentive representations to minimize the effect of network depth. Without the BiLSTM layers, the model achieves, at most, 79.7 \(F1\), which means that the sequential nature in language should be encoded as an input as well as the positional encoding. Without the selective connection module, the performance drops by 7.1 in worst case. With Word2 Vec model, \(F1\) values cannot be reached to 80, which is the score of the early study [7]. All the performance results show the decreases with each ablation model. For parallelization, we have observed the training time decreased with the increasing number of GPUs used. Due to the limited GPU environment conducted, we cannot present the result in detail. We note that our model reduce the training time by 62 percentage from the baseline. As a future work, we plan to investigate the holistic effects of parallelization in training the models. Figure 5: Performance changes with respect to model depth. Figure 6: Training time difference for parallelization. To investigate the relationship between the selective connection and the SRL accuracy improvements, we qualitatively analyze the generated labels with phrases at each layer. Specifically, for each predicted label in the development set, we collect 7 predicted labels with their corresponding spans from the training set. Table 4 shows the inference results with given the predicate word \"chased\". We have observed the higher levels of attentive representation learn the higher syntactic structure captured by varying the depth of the network. At the ninth layer, the label for _accepted-from_[A1] is assigned to the phrase \"the bird that saw the cat\". This means the our model predicts the role labels for the span of words, which contains the new acceptor \"the bird\" and the new predicate \"saw\". The span could be understood by introducing a new sentence with label [A0] and [V] as specified in the lower layer 8. Our results show that our model understand the hierarchical structure over the compounding sentences consisting of acceptors to be accepted.