# Incorporating Multi-Hypotheses as Soft-Templates in Neural Headline Generation
## 4 Results
### Main Results  
Table 3 provides the experimental results on the Gigaword test dataset. We compare our model with several models introduced in Section 3.2. The first row presents the evaluation results of the models trained only on the paired data without any pre-training information; the second row presents the experimental results of the models trained in the paired data and the template information retrieved from the training data; and the final row corresponds to the model performance when pre-training methods were adopted. \(R1\), \(R2\) and \(RL\) stand for the F1 scores of ROUGE-1, ROUGE-2 and ROUGE-L, respectively. We illustrate the model architecture of each system in Table 4 to better analyze the possible impact factors on the model performance. **Comparison with models without pre-training:** As expected, our model, the baseline, and the proposed MH-NHG model exhibit a superior performance compared to the models without pre-training. This superiority can be attributed not only to the absence of a pre-training information module in these models but also to the fundamental architecture of the baseline model. Our model incorporates the Transformer [30], a proven architecture known for its successful application in sequence learning tasks, featuring multi-head attention and feed-forward layers that contribute significantly and effectively to the learning process. **Comparison with template-based models:** Re\({}^{3}\)Sum and BiSET are two template-based models that brought significant improvements in NHG. Our model outperforms Re\({}^{3}\)Sum on all three ROUGE evaluation metrics and BiSET on the ROUGE-1 and -2 scores. However, our model exhibited an inferior performance to BiSET on the ROUGE-L score. One possible explanation for this is that the soft template in our model is made of system-generated hypotheses, while theirs is an actual headline from the training dataset. Re\({}^{3}\)Sum and BiSET demonstrate the exact template retrieval mechanism. Given a source article, they first utilize the information retrieval library, Apache Lucene ([[https://lucene.apache.org](https://lucene.apache.org)]([https://lucene.apache.org](https://lucene.apache.org)) (accessed on 1 January 2022)), to retrieve related source articles from the training dataset and \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{3}{c}{**GigaWord**} \\ \cline{2-4}  & **RG-1** & **RG-2** & **RG-L** \\ \hline Re\({}^{3}\)Sum & 37.04 & 19.03 & 34.46 \\ BiSET & 39.11 & 19.78 & 36.87 \\ \hline MASS & 38.73 & 19.71 & 35.96 \\ UniLM & 38.90 & 20.05 & 36.00 \\ PEGASUS & 39.12 & 19.86 & 36.24 \\ PEGASUS + DotProd & **40.60** & **21.00** & **37.00** \\ ProphetNet & 39.51 & 20.42 & 36.69 \\ OFA & 39.81 & 20.66 & 37.11 \\ \hline Basic model & 38.92 & 20.01 & 36.07 \\ MH-NHG & 39.44 & 20.02 & 36.69 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results on Gigaword test dataset. The highest scores are displayed in bold. take the corresponding headlines as soft templates. While the retrieval process mitigates the need for manual rule creation in traditional template-based methods, it necessitates meticulous design to ensure its effectiveness. Enhancing the precision and efficacy of the retrieved information involves eliminating all non-alphabetic characters, thereby minimizing their impact on article matching. The retrieval system identifies a limited set of candidate articles based on the processed text. In contrast, our method solely relies on the model itself, simplifying its applicability to various NHG models. **Comparison with pre-trained models:** We also compare our model with the previous pre-training methods for the headline generation tasks. Utilizing unsupervised pre-trained language models in supervised tasks has become a common practice in NLP. When training a pre-trained language model, the training data and model sizes are the two key factors significantly affecting the model performance. The authors of [PERSON] et al.