# Sentence Compression Using BERT and Graph Convolutional Networks
## 4 Experiments
### Results and Evaluation  
As this study is the first attempt to compress Korean sentences using a deep learning model, the experimental results cannot be compared with those of previous studies. Therefore, we establish a pruning model for an initial comparison model that compresses a sentence by pruning its dependency tree at a specified depth. The experimental results for Korean sentence compression are described in Table 2. The CR is the average number of characters of a compressed sentence divided by that of an original sentence, and \(\triangle\)C is the CR of sentences compressed by a model minus the CR of a training set. We set the depth of the pruning model to 5 to generate compressed sentences as close as possible to the CR of the training set. The baseline model with only BERT and the score layer achieved a high F1 score of 90.33. This result implies that contextualized information of a sentence calculated by self-attention operations in BERT is helpful for deciding which words should remain in compressed Korean sentences. When the model was equipped with the GCN, its F1 score was similar to the baseline model, but the CR increased. The model that adopted the \(d\)-GCN achieved a higher F1 score, but the CR became higher than the comparison models. We reason that the \(d\)-GCN layers were well-trained in choosing appropriate nodes for compressing a sentence while preserving the grammaticality of the compressed sentences. Although the model with \(d\)-GCN produced the longest compressed sentences, the number of sub-trees in the compressed sentences was the smallest among the comparison models besides the pruning model. The model with the \(d\)-GCN tends to generate a longer compressed sentence in which all words are connected in a valid syntactic tree. \"LONG\" is the evaluation result only for the sentences in which the average lengths exceeded those in the testing set. The percentage of the \"LONG\" sentences in the testing set is also presented in the table. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameters** & **Values** \\ \hline Number of LSTM layers & 2 \\ \hline Hidden dimension of LSTM & 768 \\ \hline Dimension of dependency label embedding & 768 \\ \hline Number of layers in GCN and \(d\)-GCN & 2 \\ \hline Hidden dimension of GCN and \(d\)-GCN & 1536 \\ \hline Optimizer & Adam \\ \hline Learning rate of BERT & \(5\times 10^{-5}\) \\ \hline Learning rate of other deep learning models & \(1\times 10^{-3}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameters used in experiments. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{4}{c}{**ALL**} & \multicolumn{4}{c}{**LONG (43.7\%)**} \\  & \multicolumn{4}{c}{**(Length \(\geq\) 16.34)**} \\ \cline{2-9}  & **F1** & **CR** & \(\triangle\)**C** & **\# of Sub-Trees** & **F1** & **CR** & \(\triangle\)**C** & **\# of Sub-Trees** \\ \hline PRUNING (depth = 5) & 80.16 & 82.37 & 6.9 & 1 & 76.14 & 71.75 & \(-\)1.2 & 1.0 \\ \hline BASELINE & 90.33 & 75.51 & 0.6 & 1.18 & 89.32 & 73.31 & 0.3 & 1.29 \\ \hline BASELINE + GCN & 90.34 & 76.74 & 1.2 & 1.16 & 89.27 & 72.97 & 0.8 & 1.25 \\ \hline BASELINE + \(d\)-GCN & **90.98\({}^{\dagger}\)** & 78.00 & 2.5 & 1.04 & **90.12\({}^{\dagger}\)** & 74.79 & 1.8 & 1.07 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation results for Korean sentence compression. \({}^{\dagger}\) indicates that the difference of the F1 score from the BASELINE is statistically significant with 0.95 confidence. We used paired-bootstrap-resampling [13] with 14,605,000 random samples. The experimental results for English sentence compression are described in Table 3. The compression model with GCN had a similar F1 score as the baseline model and the previous studies. However, the model with \(d\)-GCN achieved state-of-the-art performance. The gap in the F1 scores of the models with \(d\)-GCN and without \(d\)-GCN widened more for \"LONG\" sentences than for \"ALL\".