Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals? (2024)

Abstract: In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-finetuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly finetuning remains an open problem for domainspecific applications.1

heading: 1 Introduction
text: Recent work on the evaluation of large language models (LLMs) has shown unprecedented performance on diverse language generation tasks, even in zero-shot settings (Clark et al., 2021). Specifically in text summarization, Goyal et al. (2022) found that human evaluators prefer summaries produced by GPT-3 over strong supervised baselines. In a similar experiment testing ten large language models, Zhang et al. (2023) concluded that instruction-tuned models performed on par with human freelance writers. Additionally, their results suggest that the evaluation of news summarization is hindered by the low quality of reference summaries. Hence, the conventional summarization
1Code is available at https://github.com/ thefonseca/scisum
paradigm based on fine-tuning and evaluating on reference datasets is questionable in the face of increasingly competent language models.
However, most of those studies highlighting the success of LLM summarizers are based on news articles and the validity of those claims for long technical documents deserves further investigation. In this work, we aim to fill this gap by posing the following research questions: 1) How do LLM sum-
8599
maries compare to human-generated paper summaries? 2) To what extent LLM summarizers can be controlled to fulfill different goals of scientific communication? In both cases, we focus on the zero-shot setting, since the cost to fine-tune such large models is often prohibitive.
To address the first question, we design an experiment to compare human and machine summarizers, in which human summaries are judged in terms of their similarity with respect to another set of human-written reference summaries. By evaluating the lexical overlap with reference summaries from the multi-perspective scientific summarization dataset - MuP (Cohan et al., 2022), we find that human reviewers achieve lower ROUGE scores (Lin, 2004) compared to LLMs. The high quality of LLM summaries is confirmed by a human judgment experiment, where machine-generated summaries are preferred in 83% of the instances.
The strong preference for LLM summaries hints that the variability in communicative intentions (Giulianelli et al., 2023; Andreas, 2022) outweighs the usual quality criteria such as coherence, fluency, and relevance (Fabbri et al., 2021). In fact, an inspection of human summaries for the same article reveals arbitrary decisions related to conciseness and coverage of scientific aspects. We argue that those subjective decisions affect the perceived quality of summaries and that evaluation protocols should consider the adaptability of summarizers to diverse contexts (Jones, 1998).
Motivated by the experiment on the MuP dataset and previous literature on controllable summarization (He et al., 2022), we elect three intentional aspects: conciseness, narrative perspective (first or third person), and keyword coverage. For each of those aspects, we define two elements: intention prompts, and intention control metrics. Intentional prompts are designed to specify the task and to elicit a given intentional behavior from LLMs (e.g., summarize this article in 5 sentences). Finally, the control metrics assess the intentional alignment of prompts and the generated summaries.
To investigate the second research question, we evaluate our intentional prompts on the tasks of abstract generation (Cohan et al., 2018) and lay summarization (Goldsack et al., 2022), using GPT3.5 and LLAMA-2 (Touvron et al., 2023). We find that both models can follow conciseness and narrative perspective intentions accurately to generate short abstracts for arXiv and PubMed. Furthermore, by using keyword-based intentional prompts
and classifier-free guidance (Sanchez et al., 2023), we can direct to the lexical content of summaries leading to ROUGE scores comparable to strong supervised baselines.
However, the eLife lay summarization benchmark presents a harder challenge for LLMs. Besides being highly abstractive, these lay summaries are much longer (18 sentences on average), and have a particular concept distribution emphasizing research background. Our controllability results show that intentional prompts achieve limited success in replicating these characteristics.
Overall, our experiments spanning four benchmarks indicate that LLMs are effective summarizers for long scientific documents, both in terms of lexical alignment and human preferences. However, when summarization tasks deviate from the training distribution (e.g., lay summarization), LLMs cannot consistently match the features of human summaries. Thus, LLMs do not usher the “death” of summarization (Pu et al., 2023) but the transition from reference-based evaluation to more nuanced, domain-specific evaluation protocols, as envisioned by Jones (1998) 25 years ago.

heading: 2 Guiding Summarizer Intentions
text: As uncertainty in intentionality is intrinsic in language generation (Giulianelli et al., 2023), we argue that summarization performance relates to the capacity of a system to adapt its behavior given not only the source documents but also the target communicative intentions. We investigate language model intent adaptability by changing the summarization context via prompting, and measuring how it affects the perceived intention in the summaries (see Figure 1 for an example). Let Pθ be a language model parameterized by θ. Then, we define summarization as a conditional sequence generation:
S ∼ Pθ(· | D, pI1 , . . . , pIN ),
where D is an input article and pI1 , . . . , pIN are prompts inducing the intentions I1, . . . , IN . In this work, we consider three types of intentions that are important to adapt scientific summaries to different goals, namely conciseness, narrative perspective, and keyword coverage:
• Conciseness: as one of the most important intentional factors in summarization, conciseness defines the compression rate between the source document and the summary. We are
interested in the ability of LLMs to follow specific conciseness instructions ranging from short abstracts (6-8 sentences) to longer lay summaries (14 or more sentences).
• Narrative Perspective: depending on the perspective of the summarizer, the summary is written using first or third-person narrative. In our experiments, we consider that paper abstracts from arXiv and PubMed use firstperson narrative, and other summaries such as paper reviews from MuP (Cohan et al., 2022) and lay summaries from eLife (Goldsack et al., 2022) use third person narrative.
• Keyword Coverage: by guiding the coverage of keywords, we can indirectly manipulate the level of abstractiveness of a summary and favor simplified language, which is particularly relevant for lay summarization. Furthermore, when reference summaries are available, keyword guidance provides a mechanism to guide LLM generation using relatively small keyword models (Section 2.1.2) that demand fewer resources to fine-tune.
Since we specify intention guidance via prompts, we have no control over how the language models trade off their unconditional summarization behavior and the intention instructions. To address this issue, we modify the decoding process in Eq. 2 to include classifier-free guidance (CFG) weighting (Ho, 2022; Sanchez et al., 2023):
P̂θ(S|D, pI , pϵ) ∝ Pθ(S|D, pI)γ Pθ(S|D, pϵ)γ−1 ,
where γ ≥ 1 is the guidance strength and pϵ is a “non-intentional” summarization prompt:
Write a summary of the article above.
When γ > 1, the next-token probabilities are changed so that the generated summary is closer to the target intentions than its default summarization behavior elicited by the prompt pϵ.
To measure the adherence of summaries to intention prompts, we define reference-free intention control metrics kI(yI , S) that gauge the intentional alignment of S with respect to a target value yI . In the next sections, we detail the intention prompts pI and control metrics kI used in our experiments.

heading: 2.1 Intention Prompt Templates
text: The next component of our methods refers to the way we prompt language models to induce the in-
tentions described above. The general prompt template is defined as follows:
Article: {text} {pI1 . . . pIN }
where the placeholder {text} denotes the input article and {pI1 . . . pIN } refers to the concatenation of intention prompts, namely conciseness, narrative perspective, and keyword coverage.

heading: 2.1.1 Style Intention Prompts
text: Conciseness prompt We define an intention prompt for a target number of sentences yconciseness:
Write a summary of the article above in {yconciseness} sentences.
We choose to specify the target in sentences as it was found to be effective in previous work (Goyal et al., 2022) and in our own experiments.
Narrative perspective prompt For paper abstract generation tasks, we use the following prompt to instruct the model to write in the usual first-person plural perspective:
Write in first person “we” when applicable.
For other tasks, we do not have to prompt the model to use third person voice, as we observe this is the default behavior of the language models we evaluate (see Tables 4 and 6).

heading: 2.1.2 Keyword Coverage Prompt
text: The last prompt type instructs the model to focus on a collection of keywords to generate the summary:
Focus on the following keywords: {ykeywords}.
where the {ykeywords} placeholder indicates a comma-separated list of terms, which are provided by a keyword model.
Keyword model In our experiments, we use pretrained encoder-decoder summarizers as keyword generators. Specifically, we use FACTORSUM (Fonseca et al., 2022) and BART (Lewis et al., 2020) for abstract generation and lay summarization respectively. Using these models, we generate summaries for each document in the evaluation dataset. Then, we extract part-of-speech tags for tokens in summaries using the NLTK library (Bird et al., 2009) and keep as keywords only nouns, verbs, adjectives, foreign words, and symbols. The full list of keyword POS tags is presented in Appendix B.

heading: 2.2 Intention Control Metrics
text: In this section, we identify the controllability metrics kI for stylistic features and keyword coverage. Although we present summary-level definitions, in our experiments we report dataset-level metrics, that is, their average over all evaluation samples.
Conciseness In this work, we measure the conciseness of a summary S by counting the number of sentences |S|. Then, the conciseness controllability kI for a summary S and a target number of sentences yI is defined by:
kconciseness(yI , S) = abs(yI − |S|),
where abs is the absolute difference function.
Narrative perspective We define the narrative perspective metric based on a text classifier fnarrative(s) that maps each summary sentence s ∈ S to the label ŷI ∈ {first, third}. Then, we defined the summary-level perspective controllability as the percentage of sentences written in the target perspective yI :
knarrative(yI , S) = 100 |S| ∑
si∈S ✶[fI(si) = yI ],
where |S| is the number of sentences in S. For the classifier fnarrative(s), we use a simple heuristic where the sentence is classified as first-person perspective if 1) it starts with a first-person pronoun FPP ∈ {“we”, “our”} or 2) if it contains the pattern ", FPP", i.e., a comma followed by a FPP.
Keyword Coverage We also measure the coverage of a target set of keywords yI in summaries using the ROUGE-1 recall metric (Lin, 2004):
kkeywords(yI , S) = ROUGE-1recall(yI , S).

heading: 3 Experiments
text: In this section, we present the experimental settings spanning three styles of scientific summaries: abstracts, reviews, and lay summaries.

heading: 3.1 Datasets
text: Multi-perspective scientific summarization The MuP dataset (Cohan et al., 2022) is a corpus of summaries extracted from publicly available scientific peer reviews2 capturing multiple summarization perspectives for a single document. In our
2https://openreview.net/
experiments, we use 1,060 review summaries (and their corresponding papers) from the validation set, covering topics primarily from the AI, Machine Learning, and Natural Language Processing fields.
Abstract generation We use arXiv and PubMed (Cohan et al., 2018), which are two large-scale benchmarks for abstract generation. In our experiments, we report results for 1,000 random samples from each dataset test set.
Lay summarization We also evaluate LLMs on the task of lay summarization using the eLife dataset (Goldsack et al., 2022). In contrast to reviews and abstracts, these summaries are much longer (around 18 sentences on average), more abstractive, and exhibit a strong bias towards background content, which makes them easier to parse by non-specialists. We report results on the 241 samples from the eLife test set.

heading: 3.2 Models
text: Supervised baselines For the abstract generation task, we use BIGBIRD (Zaheer et al., 2020), a transformer-based model that uses a sparse attention mechanism to handle long input sequences. Also, we include FACTORSUM (Fonseca et al., 2022), a model that breaks the summarization task into shorter subtasks, achieving strong performance on the arXiv and PubMed datasets. For lay summarization, we train our own version of the BART model (Lewis et al., 2020) on the eLife dataset (refer to Appendix A for training details).
LLAMA-2 A collection of pre-trained and finetuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters (Touvron et al., 2023). The intention prompts described in Section 2.1 are wrapped in a model-specific instruction-tuning prompt (see Appendix A for details). Unless otherwise stated, we report results for the chat variant with 7B parameters (16-bit floating point), with 4,096 maximum context tokens using nucleus sampling (Holtzman et al., 2020) with temperature 0.8 and p = 0.95.
GPT-3.5 A proprietary model based on INSTRUCTGPT (Ouyang et al., 2022). We use the model version gpt-3.5-turbo-0301 via the chat completion API endpoint3. Although we do not have access to the implementation details behind
3https://platform.openai.com/docs/ api-reference/chat
commercial APIs, they were extensively studied in recent work (Zhang et al., 2023; Goyal et al., 2022). This model version is trained on data up to September 2021, and we generate summaries in December 2023.

heading: 3.3 Evaluation Metrics
text: In addition to the intention control metrics defined in Section 2.2, we report other relevant metrics for summarization, namely ROUGE (Lin, 2004) and summary abstractiveness (percentage of n-grams in summary that are absent in the source document). For the lay summarization task, we also report the Flesch-Kincaid Grade Level (FKGL) readability score (Kincaid et al., 1975).

heading: 4 Results and Discussion
text: In this section, we discuss our experimental results comparing human and LLM-generated summaries (Section 4.1), and the controllability experiments on the abstract generation and lay summarization tasks (Section 4.2).

heading: 4.1 Comparing Human and LLM Summaries: Reviewer Perspectives
text: We leverage the MuP dataset (Cohan et al., 2022) for a controlled comparison between human and machine summaries. First, we select documents from the validation set with more than one humanwritten summary and randomly choose one of the summaries to serve as a reference and a second human-written summary as a fictitious system summary. This setting puts humans in a similar condition as usual reference-based summarization benchmarks, that is, humans are evaluated on their ability to guess reference summaries.
As machine summarizers, we evaluate LLAMA2 and GPT-3.5. To minimize the confounding factors related to summary length, we employ conciseness prompts (Section 2.1.1) to instruct the LLMs to generate summaries with a number of sentences such that the number of generated tokens approximates the human summaries4. In Table 1, we report metrics comparing LLM and human summaries for 1,060 samples from the MuP validation set.
In addition to automatic evaluation, we perform human evaluation based on 30 random samples of human and LLM-generated summaries. Similarly
4We take the number of tokens in each human summary and divide by the average number of tokens per sentence generated by LLAMA-2 and GPT-3.5, resulting in a different conciseness target for each document.
to Goyal et al. (2022), we ask evaluators to perform blind A/B judgments based on the paper abstract and a triplet consisting of one human and two LLM summaries. Evaluators are tasked to elect from the triplet which summary (or summaries) they judge is the best/worst alternative to the paper abstract. For each answer, the evaluators need to provide a short justification. Each summary triplet is annotated by two graduate students specializing in NLP. We provide detailed task instructions in Appendix C.
ROUGE penalizes abstractiveness in human summaries The results in Table 1 show that LLMs achieve comparable albeit higher ROUGE scores compared to human reviewers, which by standard reference-based protocols, would be an indicator that LLM summaries have “superior quality.” However, we observe quantitative differences between human and machine reviews across other metrics. Notably, human summaries are significantly more abstractive as measured by the percentage of novel bigrams in the summaries. Given the more creative nature of human reviews, it is expected that they offer more possibilities for lexical disagreement than a conservative summary that prioritizes content from the source document.
Humans strongly prefer LLM summaries The results in Table 2 reveal the marked preference of evaluators towards LLM summaries, which account for 83% of votes for best summary. The preference for the worst summary is more balanced, with humans and GPT-3.5 taking the majority of votes. Also, we note that Cohen’s κ scores (Artstein and Poesio, 2008) are low, which indicates the high subjectivity of the task.
To understand the factors underlying human pref-
erences, we categorize the evaluators’ comments into quality criteria, as shown in Table 3. For both best and worst summary choices, the dominant reason is coverage of research aspects such as background or experimental results. Interestingly, factuality only appears as a third factor for choosing a bad summary, which is an indicator of the high quality of LLM summaries. However, we note that factuality issues might be underrepresented in our results, as the annotators do not use the original article in their judgments.
Takeaways Despite the advantage of LLMs on reference-based metrics and human evaluation, our aim with this experiment is not to claim that LLM summaries are superior. Instead, we emphasize the limitations of evaluation based on reference summaries, as bias in communicative intentions impacts ROUGE scores and even preferences of human annotators. Since LLM summaries exhibit human-level quality in terms of coherence, fluency, and informativeness, we argue that the real challenge for LLMs lies in the adaptability to different tasks, which we explore in the next experiment.

heading: 4.2 Controlling Style and Keyword Coverage
text: In this section, we gauge the performance of LLMs on benchmarks for abstract generation and lay summarization. Our goal is to compare the performance of zero-shot inference in LLMs to supervised baselines, and most importantly, to assess the flexibility of LLMs, via intention prompts, to replicate preferences such as narrative perspective
and conciseness. In addition to LLAMA-2 and GPT-3.5, we evaluate strong fine-tuned encoderdecoder baselines: BIGBIRD and FACTORSUM (Zaheer et al., 2020; Fonseca et al., 2022) for abstract generation and BART (Lewis et al., 2020) for lay summarization. Also, we report results for LLAMA-CFG, denoting LLAMA-2 with classifierfree guidance (refer to Section 2 for details).
LLMs capture style intents We analyze the effects of style prompts compared to two baseline prompts: the non-intentional prompt pϵ and a more direct prompt that specifies the type of summary
(e.g., write a lay summary of the article above). For abstract generation, LLMs are able to follow conciseness instructions (pconciseness) consistently, as indicated by the significant reduction in the average deviation from the target number of sentences (kconciseness in Table 4). In this aspect, GPT-3.5 exhibits superior performance compared to LLAMA2, achieving a deviation of 0.28 and 0.53 sentences for arXiv and PubMed, respectively.
By adding narrative perspective prompts (pconciseness + pnarrative), we observe a similar level of controllability, with the percentage of sentences in first person voice increasing from zero to 84% for LLAMA-2 (knarrative in Table 4). This change of perspective also results in higher similarity to reference summaries (as measured by ROUGE), which validates the effectiveness of narrative guidance. Interestingly, GPT-3.5 generates fewer sentences in first person perspective, balancing first and third voices to achieve better fluency compared to LLAMA-2 summaries.
Long summaries remain challenging While conciseness prompts are effective for abstract generation, LLMs cannot achieve low conciseness deviation for long lay summaries (Table 6). To evaluate the limits of conciseness guidance, we compare LLAMA-2 and GPT-3.5 summaries (100 samples from the eLife validation split) with target conciseness varying from 1 to 16 sentences (Figure 2). We find that both LLAMA-2 and GPT-3.5 reproduce conciseness intentions almost perfectly up to 6 sentences when it starts to degrade its instruction adherence. We hypothesize that longer generation targets conflict with the notion of “summary” that LLMs learn from pre-training data.
Keyword prompts improve lexical alignment We introduce the keyword coverage prompt pkeywords with a keyword model derived from FACTORSUM summaries for arXiv and PubMed. The results in Table 4 demonstrate that the keyword recall for LLAMA-2 and GPT-3.5 increases significantly with keyword guidance, leading to ROUGE scores higher than BIGBIRD summaries. Similarly, keyword prompt improves lexical alignment in lay summarization (Table 6). In this case, in addition to keyword recall, we also observe that keyword guidance results in better readability scores and higher abstractiveness (Table 7), which are desirable properties for lay summaries.
Classifier-free guidance improves intention control We report results for LLAMA-2 with classifier-free guidance (CFG), using γ = 1.5 for all prompt variants5 (Tables 4 and 6). In all cases, we observe an improvement in intention control metrics versus LLAMA-2 with regular decoding. In Figure 2, we can clearly observe how LLAMACFG can follow conciseness instructions for up to 10 sentences but fail to produce longer summaries.
LLMs perform well out-of-distribution One important concern is that LLAMA-based models might simply be memorizing their training data, which likely include scientific papers that overlap with the arXiv and PubMed summarization datasets. To check this possibility, we collected 500 arti-
5We set γ = 1.5 based on the results reported by Sanchez et al. (2023) and our experiments on the validation set.
Model R-1 R-2 R-L Control Metrics kI
Con↓ Nar↑ Key↑
Reference 100 100 100 10.8 99.5 44.1 BART 48.1 13.9 30.5 2.0 99.9 100
Write a lay summary of the article above.
cles (category cs.CL) submitted to arXiv between December 2023 and January 2024 and removed all the information before the introduction section (including title, abstract, and other information). Then, we evaluate the same models without further fine-tuning, using the narrative perspective and conciseness prompts. The results in Table 5 show that instruction-tuned models achieve even higher ROUGE scores (compared to Table 4), suggesting that memorization is not a cause for their good summarization performance. In contrast, we observe a pronounced loss of performance for BIGBIRD, which tends to generate repeated sentences.

heading: 5 Related Work
text: Our work fits in the context of recent research of large language models applied to summarization, mostly for news articles (Goyal et al., 2022; Zhang et al., 2023). In this work, we explore summarization in the scientific domain and its specific
Model FKGL↓ Novel Bigram↑ Reference 10.8 66.6 BART 10.2 57.8
Write a lay summary of the article above.
challenges related to document length (Beltagy et al., 2020; Fonseca et al., 2022) and technical writing style. Furthermore, we consider our work as a contribution towards understanding production variability by humans and language models, which was explored on tasks such as translation, story generation, and open-domain dialogue by Giulianelli et al. (2023).
Previous work has explored summarization controllability, including length control and entity coverage (He et al., 2022; Fan et al., 2018). In this work, we explore similar summary features and expand our analysis to the challenging setting of lay summarization. This task requires the production of long, highly abstractive summaries that cover more background information. Several other challenging scientific summarization tasks (e.g., focusing on paper contributions and related literature) would lead to promising research directions.
Finally, our work is related to attempts to improve the prompt-adherence in generation tasks. Pu and Demberg (2023) apply prompt-based methods to control summarization according to target audience (expert vs. layman) and style (formal vs. informal). Kumar et al. (2022) propose a nonautoregressive generation method to introduce soft and hard constraints, including adherence to keywords. In a different direction, (Sanchez et al., 2023) showed that Context-Free Guidance (CFG)
(Ho and Salimans, 2022), a technique originally used in text-to-image models, improves the performance of language models across several tasks. While Sanchez et al. (2023) applied CFG to tasks such as question answering, code generation and translation, our work is the first to explore this technique for summarization controllability.

heading: 6 Conclusion
text: Our experimental results on four scientific summarization benchmarks confirm that large language models are effective summarizers, as measured by conventional lexical overlap metrics and human preferences. Moreover, we find that LLMs can follow intentional prompts for style and keyword coverage, especially for short summaries. This summarization controllability can be further improved using simple decoding changes such as classifierfree guidance. However, longer summaries that require a higher level of abstractiveness are not easily achievable with our prompting techniques. Our findings suggest that analyzing summaries as an expression of communicative intentions leads to informative and actionable insights for future model improvements, where traditional reference-based evaluation shows its limits.
Limitations
Model scale and proprietary APIs Our experiments operate at model scales up to 7 billion parameters (LLAMA-2), and we expect to observe more performance gains for larger models (Wei et al., 2022). By including results from the OpenAI API, we can get an estimation of the level of performance of larger models, although we do not have access to information about the number of parameters of the underlying model and whether it includes extra machinery on top of language modeling. Lastly, we limit our experiments to zero-shot summarization as it represents a relevant application of LLMs, given the high costs involved in obtaining high-quality reference summaries.
Excluded models and reproducibility issues Despite our best prompting efforts, we could not make encoder-decoder instruction-tuned models T0 (Sanh et al., 2022) and FLAN-T5 (Chung et al., 2022) generate long summaries (at least six sentences), which makes them unsuitable for comparison. We also tried other models that claim high ROUGE performance but do not provide public code (Pang et al., 2023), fine-tuned checkpoints for
the scientific papers (Guo et al., 2022; Xiong et al., 2022), or reproducible results with the provided resources (Phang et al., 2022).
Human evaluation scope Our human evaluation experiment uses a limited number of samples from a dataset of computer science papers, and our conclusions might not generalize to other domains. Furthermore, we limit our analysis of variability in communicative intents from the point of view of summarizers, but we believe human evaluators are also subject to these factors. Future investigation could explore how different annotation conditions and guidelines potentially bias human judgments and to what extent human evaluation is still valid as the gold standard.
Other summarization domains Related work in news summarization explores the generation of summaries based on guidelines for topic and entity coverage (Ahuja et al., 2021; Maddela et al., 2022). We believe that the methods we presented in this paper could be applied to these types of coverage requirements. However, given that news summarization has been extensively investigated in previous work, we choose to perform a more comprehensive evaluation in the scientific domain.
Ethics Statement
The strong zero-shot performance of large language models dramatically reduces the costs of deploying high-quality summarization services. As a consequence, we expect that much of the information available on social media and other sources will be abridged versions generated by commoditized summarization systems. In this context, special attention is needed regarding existing issues of generative language models such as hallucination (McKenna et al., 2023). Crucially, we have shown that language models can adapt their outputs according to specific communication intentions, which might bias the summary contents towards the (potentially harmful) beliefs and desires of the actors behind those systems.

heading: Acknowledgements
text: This work was supported by Actelligent Capital and used the Baskerville UK National Tier-2 HPC (https://www.baskerville.ac.uk) at the University of Birmingham. We also thank Zheng Zhao and Ronald Cardenas for their collaboration in the human evaluation experiments.

heading: A Fine-tuning and Generation Details
text: In this section, we provide details on the summary generation implementation. The main generation parameters are provided in Table 8. LLAMA-2 and BIGBIRD checkpoints are downloaded from the HuggingFace model hub6.
For abstract generation, we use the FACTORSUM checkpoints provided by Fonseca et al. (2022)7. We perform inference with a fixed budget of 6 and 8 sentences for arXiv and PubMed respectively, and content guidance from BIGBIRD summaries.
For lay summarization, we fine-tune our own BART-base model8 on the eLife dataset9. We use the Hugging Face summarization training script (commit 5c6768210) with the hyperparameters listed in Table 9. We choose the checkpoint with the best ROUGE-2 (F1) score on the 241 of the validation set. The fine-tuning process takes about 8 GPU hours (2 Nvidia GTX 1080 12GB GPUs).
A.1 Prompt Details In addition to the prompt specification provided in Section 2.1, we provide model-specific prompt details in the section. For LLAMA-2, we use the following prompt:

 {instruction} 

where {instruction} is an intention prompt from Section 2.1.

heading: B Keyword Model Details
text: As described in Section 2.1.2, we perform partof-speech tagging on tokenized summaries by FACTORSUM and BART. Specifically, we use the averaged_perceptron_tagger provided by NLTK (Bird et al., 2009)11. Then, we select only tokens tagged as adjectives, nouns, verbs, symbols, and foreign words. The complete set of keyword POS tags is listed in Table 10.

heading: C Human Evaluation Task Description
text: Human evaluators are presented with the following short description of the tasks:
6https://huggingface.co/models 7https://github.com/thefonseca/factorsum 8https://huggingface.co/facebook/bart-base 9https://huggingface.co/datasets/tomasg25/
scientific_lay_summarisation 10https://github.com/huggingface/transformers/ commits/5c67682b169576c4859700d551090ff79d450a9a/ examples/pytorch/summarization/run_summarization. py
11www.nltk.org/api/nltk.tag.perceptron.html
BIGBIRD
Checkpoint (arXiv) bigbird-pegasuslarge-arxiv Checkpoint (PubMed) bigbird-pegasuslarge-pubmed Number of parameters 576.9M Max source length 3072 Generation beams 5 Length penalty 0.8
FACTORSUM
Checkpoint bart-base Number of parameters 139.4M Max source length 1024 Max source length 128 Generation beams 4 Length penalty 1.0
LLAMA-2
Checkpoint Llama-2-7b-chat-hf # parameters 7B Max context length 2048 Parameter type float16 Nucleus temperature 0.8 Nucleus top-p 0.95
GPT-3.5
The goal of this study is to evaluate machinegenerated summaries of scientific articles. Each reference article abstract will be presented with 3 alternative summaries. Your task is to identify which of the alternatives is the best according to your personal preferences and experience reading papers. The criteria may include good coverage of the abstract content, factuality issues, and linguistic fluency, among others. It is expected that in some cases the alternative summaries cover details that cannot be verified based on the reference abstract. You can disregard the factuality of those details if you consider it appropriate.
A sample task with a reference abstract, the 3 candidate summaries, and the questions is shown in Table 11.

heading: D PubMed Summarization Results
text: We report summarization and intention control metrics for the PubMed dataset in Table 12.

heading: E Sample Summaries
text: In Tables 13 to 17, we provide sample summaries for the MuP, arXiv, PubMed, and eLife datasets.
Reference Abstract (Zhou et al., 2020)
Learning disentangled representations is regarded as a fundamental task for improving the generalization, robustness, and interpretability of generative models. However, measuring disentanglement has been challenging and inconsistent, often dependent on an ad-hoc external model or specific to a certain dataset. To address this, we present a method for quantifying disentanglement that only uses the generative model, by measuring the topological similarity of conditional submanifolds in the learned representation. This method showcases both unsupervised and supervised variants. To illustrate the effectiveness and applicability of our method, we empirically evaluate several state-of-the-art models across multiple datasets. We find that our method ranks models similarly to existing methods. We make our code publicly available at https://github.com/stanfordmlgroup/disentanglement.
Summary A
The article proposes a new evaluation metric for disentanglement in generative models, which is based on the concept of persistent homology. The metric measures the topological similarity between submanifolds conditioned on different latent dimensions in the generative model, and can be used to evaluate the disentanglement of different models and datasets. The proposed metric provides a more reliable and robust way of evaluating disentanglement compared to existing methods, and can be applied across different model architectures and datasets without requiring additional training or hyperparameter tuning.
Summary B
The paper proposes a novel metric for evaluating disentanglement by taking a manifoldtopological perspective on the representations learnt. The key insight is that for a disentangled representation, when we fix a certain factor of variation at different values the topology of the conditional sub-manifolds should be similar. Using this insight the paper proposes a metric for disentangling which does not require annotations of the factors of variation and is more general than previous such tests.
Summary C
Researchers have proposed an unsupervised method for evaluating the level of disentanglement in deep generative models. This evaluation metric uses topology and density analysis of samples to measure the degree of topological similarity across submanifolds conditioned on disentangled latent dimensions versus entangled ones. The approach shows consistent results across 10 models and three datasets, without relying on training additional models or introducing a dataset-specific preprocessing step.
Human Summary 1 (Reference)
This paper presents a fast method for solving certain classes of optimal control problems, by implementing a value iteration (VI) algorithm that takes advantage of convex duality. The basic idea is to take advantage of the fact that for two functions f1, f2, we have (f1□f2)∗ = f∗1 + f∗2 , where f∗ denotes the Legendre-Fenchel transform of f , and (f1□f2)(x) = inf{f1(x1) + f2(x2) : x1 + x2 = x} denotes the infimal convolution of f1, f2. Thus, the slow inf operation can be replaced with a fast addition operation. The paper provides analyses of the convergence, time complexity, and error of the algorithm. When compared to a “naive” VI algorithm in which each iteration takes O(XU) (where X , U are the grid sizes of the state and input spaces, respectively), the algorithm presented in this paper can achieve O(X + U). Finally, the paper provides a numerical example implemented in MATLAB.
Human Summary 2
The paper develops an approximate value iteration method for an infinite-horizon, discountedcost Markov Decision Processes (MDPs) that satisfy a given set of regulatory assumptions. The idea is to work on a dual space that replaces the DP value function by a reformulation written in terms of biconjugate operations. The authors present an algorithm that applies the conjugate operators iteratively, and present convergence and other structural results associated with the resulting errors. Numerical results compare the proposed approach against a traditional value iteration and a variant that generated dynamic discretization grids.
Prompt: Write a summary of the article above in 4 sentences.
LLAMA-2 Summary
The article discusses the problem of optimal control of continuous-state systems with discretetime dynamics and stochastic cost. The authors propose a new algorithm, called Conjugate Value Iteration (ConjVI), which combines the power of the DP operator with the efficiency of discrete-time methods to solve this problem. ConjVI is based on a modified version of the Discrete Dual Domain (d-CDP) operator, which allows for a more efficient computation of the value function. The authors provide theoretical guarantees on the convergence of ConjVI and analyze its error, showing that it can achieve a linear time complexity of O(X+U) in each iteration, where X and U are the cardinalities of the discrete state and input spaces, respectively.
GPT-3.5 Summary
Value iteration (VI) is an algorithm used for problems in reinforcement learning (RL) and optimal control formulated as Markov decision processes (MDPs). However, VI suffers from a high computational cost for large-scale finite state spaces. A solution to this problem is to incorporate function approximation techniques and compute the output of the DP operator for a finite sample of the underlying continuous state space. The article proposes the conjugate VI (ConjVI) algorithm based on a modified version of the d-CDP operator that extends existing results in three directions for solving optimal control problems of discrete-time systems with a continuous state-input space.
Table 13: Sample summaries for Kolarijani et al., 2021 (MuP validation set).
Reference Abstract
This study examines the composition and activity of the planktonic community during the polar night in the high Arctic Kongsfjord, Svalbard. Our results are the first published evidence of bioluminescence among zooplankton during the Arctic polar night. The observations were collected by a bathyphotometer detecting bioluminescence, integrated into an autonomous underwater vehicle, to determine the concentration and intensity of bioluminescent flashes as a function of time of day and depth. To further understand community dynamics and composition, plankton nets were used to collect organisms passing through the bathyphotometer along with traditional vertical net tows. Additionally, using a moored bathyphotometer closed to the sampling site, the bioluminescence potential itself was shown not to have a diurnal or circadian rhythm. Rather, our results provide evidence for a diel vertical migration of bioluminescent zooplankton that does not correspond to any externally detectable changes in illumination.
Prompt: Write a abstract for the article above in 8 sentences. Write in first person “we” when applicable.
LLAMA-2 Summary
We conducted a study on plankton distribution and bioluminescence in the polar night at Kongsfjorden, Svalbard, during January 2010. We used a remotely operated vehicle (ROV) equipped with bioluminescence, depth, and temperature sensors to collect data on plankton abundance and distribution. Our results showed that bioluminescence potential was detected throughout the water column, with higher intensity at depth during the day and increased surface bioluminescence at night. We found significant differences in acoustic backscatter coefficients between day and night, indicating the presence of zooplankton migration. Our plankton enumerations showed an increase in the majority of the most abundant zooplankton taxa at night, including Pseudocalanus spp., which are known to be present throughout the year in this region. We also found that Metridia lucens, Metridia longa, Oncaea borealis, Thysanoessa inermis, and Thysanoessa longicaudata likely contributed to the increase in highintensity bioluminescent flashes at 15 and 45 m during the night. Our results suggest that there is a coordinated movement of biomass indicative of vertical migration in this region.
GPT-3.5 Summary
The article discusses the importance of bioluminescence and other adaptations in marine environments. Bioluminescence has evolved independently over 40 times and has various functions including defensive functions such as counter-illumination and burglar alarm and offensive mechanisms such as prey attraction and intra-specific communication. Some adaptations involve entering a dormant state and overwintering at depth, such as seen in the copepods calanus glacialis and c. hyperboreus. The study aimed to characterize plankton abundance and distribution patterns during a time of year rarely studied, using vertical net tows and autonomous underwater vehicle (AUV) surveys. The AUV was fitted with upward and downward facing RD instruments, a CTD, and a bioluminescence bathyphotometer. The data collected off the coast of Svalbard showed bioluminescence was detected throughout the water column both day and night, and higher bioluminescence was present at depth during the day and increased surface bioluminescence at night. Vertical net hauls showed that there was an increase above 60 meters in the majority of the most abundant zooplankton taxa at night, indicating dvm. Lastly, plankton enumerated from the > 20m net collection of the bp exhaust suggests that during the day, the greatest biomass occurred at 45m and was dominated by copepod nauplii, copepod eggs, and the tintinnid acantostomella norvegica.
Table 15: Sample summaries for Berge et al., 2012 (PubMed test set).
Reference Lay Summary
Imagine a gymnastics competition in which participants take turns to cartwheel and somersault across the floor. The routines on display comprise sequences of precisely timed movements learned through practice. This is also true for many of the actions we perform every day, such as reaching for a cup of coffee. A region of the brain called the cerebellum helps us learn sequences of movements. But how does it do this? To find out, Khilkevich et al. came up with a new version of an old experiment. Rabbits were first trained to blink their eye in response to a specific external cue. This type of learning, called associative learning, has been shown before in the cerebellum. But Khilkevich et al. wondered whether the cerebellum could also use internal feedback signals from the eyeblink as a cue to learn the next movement? If so, this might explain how the cerebellum can chain movements together in a sequence. As predicted, Khilkevich et al. found that rabbits could learn to blink their eye in response to an initial signal, and then blink again in response to the first blink. Control experiments confirmed that the second eyeblink was coupled to the first, and not to the original cue. Moreover, on many trials the rabbits showed a third and even fourth eyeblink. This is because feedback signals from the first, second or third blink were the same. Thus, the feedback signals from the first blink triggered the second blink, feedback from the second triggered the third, and so forth. Rabbits could also learn to use a blink of the left eye as a cue for a blink of the right eye. Similar patterns of neuronal activity accompanied each blink, suggesting that the same mechanism generated them all. The cerebellum can thus use feedback from one movement as a cue to learn the proper timing of the next movement in a sequence. A key question is whether this mechanism of sequence learning extends beyond movement. The cerebellum has extensive connections to the brain’s outer layer, the cortex, including many areas involved in cognition. Future experiments should test whether the cerebellum might help guide sequences of cortical activity during cognitive tasks.
GPT-3.5 Summary Prompt: Write a summary of the article above in 14 sentences.
The article focuses on how the cerebellum, which is responsible for accurate movement execution and learning, can learn and execute movement sequences. Previous studies have shown that cerebellar pathology leads to severe deficits in the execution and learning of sequences. To test the sufficiency of feedback signals from a movement as a cue for the next component in a sequence, the researchers trained rabbits using electrical stimulation of mossy fibers as a cue. Once subjects acquired a sequence of conditioned eyelid responses, the researchers tested whether the feedback information about the first component is sufficient for the cerebellum to use it as a new “CS” to learn the subsequent CR. The results show that the cerebellum can learn to chain together a sequence of inter-connected movement components by using feedback signals from one component to serve as a cue for the next component in the sequence. In vivo recordings from cerebellar cortex showed that Purkinje cell activity relates as strongly to sequences as it relates to single-component eyelid responses. The experiments suggest that the feedback signals are strong enough to serve as cues for the cerebellum to learn subsequent movements. The authors suggest that cerebellar mechanisms of learning single movement components can be extended to learning of movement sequences, providing a general framework for how the cerebellum can use feedback signals to learn to chain together correctly timed responses to produce a movement sequence. The researchers emphasize that most human movements are comprised of sequences, and movement sequences as well as multi-joint movements are sensitive to cerebellar dysfunction. The article provides insights into how researchers can improve the treatment of movement disorders involving the cerebellum.
Table 16: GPT-3.5 summary for Khilkevich et al., 2018 (eLife test set).
Prompt: Write a summary of the article above in 14 sentences.
LLAMA-2 Summary
Most movements are composed of sequences, and the cerebellum is involved in learning and executing accurate movements. The cerebellum has been linked to learning and execution of movement sequences, and studies have shown that cerebellar dysfunction can lead to deficits in sequence learning or the inability to learn sequences at all. However, most of what we know about cerebellar mechanisms of learning comes from studies using single-component movements. Three possible ways of implementing cerebellar learning of movement sequences have been proposed, and one of these ways involves the use of feedback signals from one movement component to learn the next component. To test the hypothesis that the cerebellum can use feedback signals from a movement to learn the next movement in a sequence, researchers trained rabbits using electrical stimulation of mossy fibers as a cue. The cerebellum can learn to chain together a sequence of interconnected movement components by using feedback signals from one component to serve as a cue for the next component in the sequence. The design of most experiments does not permit distinguishing between different possibilities of sequence learning, but the use of eyelid conditioning to test the sufficiency of feedback signals (FS) for sequence learning provided direct evidence that the cerebellum can learn to use FS to chain together movement components. In the experiment, the first movement was a conditioned eyelid response (CR) elicited by a sensory stimulus, and the second movement was a sequence of movements produced by the same muscle or different muscle groups used for the different components of the sequence. The results showed robust learning of later components, including sequences of movements produced by the same muscle and sequences with different muscle groups used for the different components. The in vivo recordings from cerebellar cortex showed that Purkinje cell activity related as strongly to the latter components of sequences as it has been shown to relate to single-component eyelid responses. These findings provide a general framework for how the cerebellum can use feedback signals to learn to chain together appropriately timed responses to produce a movement sequence. The results demonstrate the importance of cerebellar learning mechanisms in mediating the acquisition of movement sequences and suggest that the cerebellum can use feedback signals from a movement to learn the next movement in a sequence.
Table 17: LLAMA-2 summary for Khilkevich et al., 2018 (eLife test set).

heading: Future Work
text: Future experiments should test whether the cerebellum might help guide sequences of cortical activity during cognitive tasks. GPT-3. 5 Summary Prompt: Write a summary of the article above in 14 sentences. The article focuses on how the cerebellum, which is responsible for accurate movement execution and learning, can learn and execute movement sequences. Previous studies have shown that cerebellar pathology leads to severe deficits in the execution and learning of sequences. To test the sufficiency of feedback signals from a movement as a cue for the next component in a sequence, the researchers trained rabbits using electrical stimulation of mossy fibers as a cue. Once subjects acquired a sequence of conditioned eyelid responses, the researchers tested whether the feedback information about the first component is sufficient for the cerebellum to use it as a new “CS” to learn the subsequent CR. The results show that the cerebellum can learn to chain together a sequence of inter-connected movement components by using feedback signals from one component to serve as a cue for the next component in the sequence. In vivo recordings from cerebellar cortex showed that Purkinje cell activity relates as strongly to sequences as it relates to single-component eyelid responses. The experiments suggest that the feedback signals are strong enough to serve as cues for the cerebellum to learn subsequent movements. The authors suggest that cerebellar mechanisms of learning single movement components can be extended to learning of movement sequences, providing a general framework for how the cerebellum can use feedback signals to learn to chain together correctly timed responses to produce a movement sequence. The researchers emphasize that most human movements are comprised of sequences, and movement sequences as well as multi-joint movements are sensitive to cerebellar dysfunction. The article provides insights into how researchers can improve the treatment of movement disorders involving the cerebellum. Table 16: GPT-3. 5 summary for Khilkevich et al. , 2018 (eLife test set). Prompt: Write a summary of the article above in 14 sentences. LLAMA-2 Summary
Most movements are composed of sequences, and the cerebellum is involved in learning and executing accurate movements. The cerebellum has been linked to learning and execution of movement sequences, and studies have shown that cerebellar dysfunction can lead to deficits in sequence learning or the inability to learn sequences at all. However, most of what we know about cerebellar mechanisms of learning comes from studies using single-component movements. Three possible ways of implementing cerebellar learning of movement sequences have been proposed, and one of these ways involves the use of feedback signals from one movement component to learn the next component. To test the hypothesis that the cerebellum can use feedback signals from a movement to learn the next movement in a sequence, researchers trained rabbits using electrical stimulation of mossy fibers as a cue. The cerebellum can learn to chain together a sequence of interconnected movement components by using feedback signals from one component to serve as a cue for the next component in the sequence. The design of most experiments does not permit distinguishing between different possibilities of sequence learning, but the use of eyelid conditioning to test the sufficiency of feedback signals (FS) for sequence learning provided direct evidence that the cerebellum can learn to use FS to chain together movement components. In the experiment, the first movement was a conditioned eyelid response (CR) elicited by a sensory stimulus, and the second movement was a sequence of movements produced by the same muscle or different muscle groups used for the different components of the sequence. The results showed robust learning of later components, including sequences of movements produced by the same muscle and sequences with different muscle groups used for the different components. The in vivo recordings from cerebellar cortex showed that Purkinje cell activity related as strongly to the latter components of sequences as it has been shown to relate to single-component eyelid responses. These findings provide a general framework for how the cerebellum can use feedback signals to learn to chain together appropriately timed responses to produce a movement sequence. The results demonstrate the importance of cerebellar learning mechanisms in mediating the acquisition of movement sequences and suggest that the cerebellum can use feedback signals from a movement to learn the next movement in a sequence. Table 17: LLAMA-2 summary for Khilkevich et al. , 2018 (eLife test set)