Prompt-based Pseudo-labeling Strategy for Sample-Efficient Semi-Supervised Extractive Summarization
Authors: Gaurav Sahu, Olga Vechtomova, Issam H. Laradji


Reviewer paper summaries: This paper targets semi-supervised extractive summarization where we have only few documents with binary labels for extraction. The authors propose to use PreSumm, a supervised classifier trained on the small data, and GPT. Specifically, they first train the supervised classifier and then use GPTs to score documents in two steps i.e., score each sentence and then score generated whole summary based on the scored sentences.

The evaluations are done common datasets for summarization; WikiHow, ArXive/PubMed, and TweetSum.
Summary of strengths: - The effectiveness of the proposed approach (after stage 2) are shown in the experiments.
Summary of weaknesses: - More simple baselines: The proposed method uses summaries created by a zero-shot approach with GPT as training data. Therefore, if high-quality training data can be obtained with this method, it might be possible that good summaries can also be obtained with the simple zero-shot approach by GPT4/Llama-3. However, the experiments lack a comparison with a simple zero-shot extractive summarization method.

- Novelty: the proposed approach seems a straightforward application of the existing idea used in similar summarization settings. A basic motivation behind the proposed method is iterating the refinement of pseudo-generated summary and then use them for training new extractor. This idea of iterating a summary has been used several studies e.g., abstractive summarization (https://aclanthology.org/2023.findings-emnlp.714.pdf).

- Human evaluation: the experiments lack human evaluation.

- Hyperparameters: Some hyperparameters are included in the model but explanations on how to set them are missing e.g., how to set the number 50 for the stage 1. Is it tuned on the test data? How did you set N_{cycle}?
Comments, suggestions and typos: - In Figure 2, “Figure 4” -> “Figure 3 (left)”?

- Line225: hard to read two-line equation in a single-line.
Overall assessment scores: 2