# Sentence Compression Using BERT and Graph Convolutional Networks
## 4 Experiments
### Results and Evaluation
The performance improvements are attributed to the propagation rules in the \(d\)-GCN layers that differentiate parent and child nodes and aggregate child nodes with a max-pooling operation. In order to conduct human evaluation of the three compression models, we collected 100 sentences each from the Korean and English testing sets. The compression ratio strongly correlates with human judgments of informativeness and grammaticality [14]. Therefore, only sentences with a compression ratio difference of less than 10.0% were collected and evaluated. All sentence-compression pairs were assessed by three human raters who were asked to rate them on a five-point Likert scale from one to five. These pairs were evaluated for both readability and informativeness. Table 4 summarizes the human evaluation results. In Korean and English, the system with \(d\)-GCN outperformed the other models under both metrics. The performance of the compression model using GCN is not as good as we would expect, and the readability score of English is even lower than that of the baseline. However, the model adopting \(d\)-GCN has shown significant improvements in both metrics. Examples of the compressed sentences are listed in Table 5. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Models** & **Readability** & **Informativeness** \\ \hline \multirow{3}{*}{Korean} & BASELINE & 3.55 & 3.64 \\ \cline{2-3}  & BASELINE + GCN & 3.68 & 3.79 \({}^{\ddagger}\) \\ \cline{2-3}  & BASELINE + \(d\)-GCN & **3.84**\({}^{\ddagger}\) & **3.99**\({}^{\ddagger}\) \\ \hline \multirow{3}{*}{English} & BASELINE & 3.72 & 3.57 \\ \cline{2-3}  & BASELINE + GCN & 3.59 & 3.57 \\ \cline{1-1} \cline{2-3}  & BASELINE + \(d\)-GCN & **3.89**\({}^{\ddagger}\) & **3.91**\({}^{\ddagger}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results of human evaluation of Korean and English sentence compression. \({}^{\ddagger}\) indicates that the difference of the score from the BASELINE is statistically significant with 0.95 confidence. We used the same method as in Tables 2 and 3 with 100,000 random samples. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**ALL**} & \multicolumn{4}{c}{**LONG (41.7\%)**} \\ \cline{2-9}  & **F1** & **CR** & \(\vartriangle\)**C** & **\# of Sub-Trees** & **F1** & **CR** & \(\vartriangle\)**C** & **\# of Sub-Trees** \\ \hline Evaluator-LM [8] & 85.5 & 39.0 & \(-\)2.7 & - & - & - & - & - \\ \hline SLAHAN [9] & 85.5 & 40.7 & \(-\)1.5 & - & 83.3 & 28.4 & \(-\)1.9 & - \\ \hline BASELINE & 85.2 & 40.1 & \(-\)2.3 & 1.70 & 82.9 & 29.6 & \(-\)1.3 & 1.91 \\ \hline BASELINE + GCN & 85.5 & 40.3 & \(-\)2.1 & 1.65 & 82.4 & 30.0 & \(-\)0.9 & 1.88 \\ \hline BASELINE + \(d\)-GCN & **86.2**\({}^{\ddagger}\) & 40.9 & \(-\)1.4 & 1.55 & **84.2**\({}^{\ddagger}\) & 31.3 & 0.4 & 1.72 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Evaluation results for English sentence compression. \({}^{\ddagger}\) indicates that the difference of the F1 score from the baseline is statistically significant with 0.95 confidence. We used paired-bootstrap resampling [13] with 1,000,000 random samples.