# Selectively Connected Self-Attentions for Semantic Role Labeling
## 4 Experiments
### Experimental Results  
We compare our models to five existing studies: two bidirectional LSTM-based models denoted as [PERSON] et al. [9; 11], include the state-of-the-art SRL model, and a self-attention-based model denoted as [PERSON] et al. [15] and others [7; 25]. Tables 1 and 2 shows our model performance with the reported results of the previous studies. We summarize the results of the best single models with the metrics in terms of precision (P), recall (R) and F1. Our proposed model achieves \(F1\) scores of 86.6 and 77.2 on WSJ test and Brown test of CoNLL 2005 shared tasks, respectively. In CoNLL 2012 shared task, our model achieves best \(F1\) scores of 83.6. In some senses, these may be small improvements on both datasets compared to the existing studies. However, we have observed several useful characteristics in the SRL task. For example, with ELMo pretrained model, \(F1\) scores are increased from 83.3 to 86.6. Based on the study, it is worth noting that the pretrained embedding models with our architecture are important to general downstream tasks as well as sentence encoding. The key issue in SRL would be incorporating the fine-tuning with the language models. The results explain our model is competitive to predict the semantic role labels compared to previous work. To better understand the effect of the proposed modules, we compare the performance of using the selective connection in self-attention based architectures. The Figure 5 shows the model performance, according to the different numbers of the depth of the model. With increasing number of model depths, \(F1\) values of both models are gradually increased until it has 9 layers of attentive representation modules. The training times in the Figure 5 are normalized by setting the medians to be zero to present the improvement of efficiency. Training time difference due to the selective connection is relatively small from 6 to 10 layers, which is within the region of higher performance. Although the hyperparameters we used are slightly different from the work [15], the tendencies of training times increase are similar. For more than 4 layers of multi head self-attentions, our model with the selective connections performs much better than others. This means that the adoption of selective connection enhances the training efficiency. For parallelism, we compare the time to complete varying the number of epochs given GPU settings. By assigning a model to 2 GPUs, we have observed that the training time is reduced by 0.62 compared to the single GPU model. For data parallelization, the training time is reduced by 0.67. Figure 6 shows how the different parallelization method competes with single models. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Development**} & \multicolumn{3}{c}{**WSJ Test**} & \multicolumn{3}{c}{**Brown Test**} \\ \cline{2-9}  & **P** & **R** & **F1** & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline Ours+ELMo & 83.1 & 85.6 & 85.5 & 86.4 & 81.0 & 86.6 & 75.8 & 77.6 & 77.2 \\ Ours & 82.8 & 82.6 & 82.2 & 83.6 & 80.1 & 83.3 & 73.5 & 74.0 & 74.9 \\ \hline [PERSON] et al. (2018) & - & - & 85.3 & 84.8 & 87.2 & 86.0 & 73.9 & 78.4 & 76.1 \\ [PERSON] et al. & 82.6 & 83.6 & 83.1 & 84.5 & 85.2 & 84.8 & 73.5 & 74.6 & 74.1 \\ [PERSON] et al. (2017) & 81.6 & 81.6 & 81.6 & 83.1 & 83.0 & 83.1 & 72.9 & 71.4 & 72.1 \\ [PERSON] and [PERSON] & 79.7 & 79.4 & 79.6 & 82.9 & 82.8 & 82.8 & 70.7 & 68.2 & 69.4 \\ [PERSON] et al. & 81.2 & 76.7 & 78.9 & 82.5 & 78.2 & 80.3 & 74.5 & 70.0 & 72.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Experimental results on the CoNLL-2005.