ATLAS: Improving Lay Summarisation with Attribute-based Control (2024)

Abstract: Automatic scientific lay summarisation aims to produce summaries of scientific articles that are comprehensible to non-expert audiences. However, previous work assumes a one-sizefits-all approach, where the content and style of the produced summary are entirely dependent on the data used to train the model. In practice, audiences with different goals and levels of expertise will have specific needs, impacting what content should appear in a lay summary and how it should be presented. Aiming to address this disparity, we propose ATLAS, a novel abstractive summarisation approach that can control various properties that contribute to the overall “layness" of the generated summary using targeted control attributes. We evaluate ATLAS on a combination of biomedical lay summarisation datasets, where it outperforms state-of-the-art baselines using both automatic and human evaluations. Additional analyses provided on the discriminatory power and emergent influence of our selected controllable attributes further attest to the effectiveness of our approach.

heading: 1 Introduction
text: Lay summarisation is defined as producing a summary of a scientific article that is comprehensible to non-experts (King et al., 2017). Recent work has shown that, when compared to technical abstracts, lay summaries typically are more readable (lexically and syntactically), more abstractive, and contain more background information, enabling a non-technical reader to better understand their contents (Luo et al., 2022; Cohen et al., 2021; Goldsack et al., 2023b). However, the extent to which these attributes are required within a lay summary depends largely on the specific needs of the reader. For example, a scientist from a related field will require less background information to understand an article’s contents than an entirely non-technical
∗ Corresponding author
reader, but they might still require domain-specific jargon to be simplified or explained. Despite its obvious benefits, to our knowledge, no work has yet explored how we can enable such fine-grained control over comprehensibility-related aspects for lay summary generation.
In this paper, we propose ATLAS (ATtributecontrolled LAy Summarization), a novel scientific summarisation approach that aims to control four attributes targeting distinct properties contributing to the overall “layness" of the generated summary, thus allowing it to cater to the specific needs of different audiences. Although recent attempts at text simplification and story generation have had success influencing the style (Martin et al., 2020; Kong et al., 2021; Sheang and Saggion, 2021) and content (Kong et al., 2021; Tang et al., 2024) of generated text using fine-grained controllable attributes, no work to our knowledge has explored this for scientific summarisation. Luo et al. (2022) recently addressed the task of readability-controlled scientific summarisation, however, this is only done at a binary level, training a model to produce either a technical or non-technical summary based on a single control token.
Our approach innovates by enabling a greater degree of controllability through the flexible handling of multiple attributes, allowing it to produce more diverse summaries and better address the specific needs of different audiences. Our results show that ATLAS outperforms state-of-the-art baselines in both automatic and human evaluations across three summary types with varying levels of technicality. Additional analyses confirm that attribute control positively influences performance, and suggest the selected control attributes are able to effectively capture the difference between technical and nontechnical summaries.
337

heading: 2 Methodology
text: As discussed in §1, ATLAS aims to control four targeted attributes. We use BART-base as the base model for ATLAS as it represents the state-of-theart benchmark in previous lay summarisation works (Guo et al., 2021; Goldsack et al., 2022).
Formally, each document x = (x1, x2, ..., xn) of length n, where xi is the i-th token, is prepended with a control token sequence l such that x = (l,x1,x2s, ...,xn). l consists of our four selected control tokens, each of which targets distinct characteristics of the output summary that contributes to its overall comprehensibility. We describe each aspect below:
Length (L) The length of the output summary in characters. A more lay audience may require a longer summary to aid comprehension.
Readability (R) How easy it is to read the text. This is measured using the Flesh-Kincaid Grade Level (FKGL) metric, which estimates the reading grade level (US) required to understand the generated text based on the total number of sentences, words, and syllables present within it.
Background information (BG) The percentage of sentences classified as containing primarily background information. Intuitively, a more lay audience will require greater levels of background information to contextualise an article.
Content word entropy (CWE) The average entropy of content words. We hypothesise that jargon terms are likely to possess higher entropy values, thus lower average CWE is likely to be a property of more lay text. Since jargon terms are predominately nouns, we extract noun phrases as content words using CoreNLP library (Manning et al., 2014).We then follow Xiao et al. (2020) to calculate I(xi) entropy of a given token xi as the negative logarithm of its generation probability P (xi), which is directly extracted from a pre-trained language model.
I(xi) = −logP (xi) (1)
During model training, true attribute values (as calculated on reference summaries) are used, allowing the model to learn to associate attribute values with summary properties. For all attributes, values are discretized into 10 fixed-width bins depending on their respective range in the train split (from
minimum to maximum observed value), resulting in 10 unique control tokens for each attribute which are added to the vocabulary. For each attribute at test time, we use the most common bin value observed for reference summaries of the training set as attribute values.

heading: 3 Experimental Setup
text: Data. We experiment on the biomedical lay summarisation datasets introduced in Goldsack et al. (2022), eLife (4.8k articles) and PLOS (27.5k articles), for which target lay summaries have been shown to contain different levels of “layness". Specifically, eLife’s lay summaries have been characterized as longer, more readable, and more abstractive than those of PLOS, as well as being empirically observed to be suitable for a more lay audience. We, therefore, combine both of these datasets, allowing us to expose ATLAS to a greater variety of attribute values during training.1 For each article in the combined dataset, we train our ATLAS to produce both the technical abstract and lay summary, using our control attributes to differentiate between them.
Evaluation. We employ several automatic metrics to evaluate the performance of ATLAS. In line with common summarisation practice, we calculate ROUGE-1,2, and L variants (Lin, 2004) and BERTScore (Zhang et al., 2019). We also measure Dale-Chall Readability Score, a metric that estimates US grade level based on the frequency of common words.
Baselines. To enable fair comparison, we rerun many of the baseline approaches used by Goldsack et al. (2022) (which have the abstract included in the input) on the combined datasets. Specifically, we rerun the Lead-3, Lead-K, and oracle heuristic baselines; TextRank (Mihalcea and Tarau, 2004), LexRank (Erkan and Radev, 2004), and HipoRank (Dong et al., 2021) unsupervised models; and BART and BARTScaffold supervised models. Here, we use the transformer-based BART base model (Lewis et al., 2020), which we fine-tune on our own datasets. BARTScaffold is the recreation of a model from Goldsack et al. (2022) which is trained using a binary control token (<abs> or <lay>) to produce either an abstract or lay summary for an article. This model is equivalent to that pro-
1To combine the datasets, we merge the training and validation sets. We evaluate on the test sets separately.
posed by Luo et al. (2022), the only previous work on controllable lay summarisation.2
Finally, we include two baselines based on ChatGPT (3.5-turbo), so as to compare against an accessible and widely used method of controlling text generation (i.e., prompt engineering). Our first GPT baseline (GPT3.5-zs) uses the following zeroshot prompts: (i) “Summarize the following article for an expert audience that is familiar with the technical aspects of the content” to generate technical abstracts; (ii) “Summarize the following article for a non-expert audience that has some familiarity with the technical aspects of the content” to generate PLOS lay summaries, and (iii) “Summarize the following article for a non-expert audience that has no familiarity with the technical aspects of the content” to generate eLife lay summaries. Our second GPT baseline (GPT3.5-mdc) replicates the method of Turbitt et al. (2023), the best-performing team of the recent BioLaySumm shared task (Goldsack et al., 2023a). Based on in-context learning, this method dynamically selects the maximum number of input-output examples that fit in the context window (separated by the simple prompt “Explanation:”) to generate lay summaries based on only the article abstract.
Implementation Details. As mentioned in §2, we employ BART-base as our base model. We train our ATLAS for a maximum of 5 epochs on a GeForce GTX-1080Ti GPU, retaining the checkpoint with the best average ROUGE-1/2/L score on the validation set. We set the batch size to 1 and keep the α scale factor (§2) at the default value of 0.2 from Kong et al. (2021).
For calculating control attributes, we use SciBERT (Beltagy et al., 2019) for entropy calculation, and we employ a BERT-based sequential classi-
2The original code for Luo et al. (2022) is not yet available at the time of writing and their results are reported on a different dataset and thus are not comparable.
fier (Cohan et al., 2019) trained on the PubMedRTC dataset (Dernoncourt and Lee, 2017) for background sentence classification (as described in Goldsack et al. (2022)). We compute the FKGL readability score using the textstat package.

heading: 4 Experimental Results
text: Discriminatory ability of control attributes. To validate the ability of our controllable attributes to distinguish between different summary types, we plot the distribution of attribute values for each type in Figure 1. The figure suggests that, in combination, the attributes are able to capture characteristic differences between summary types, as instances in which two summary types share a similar distribution for one attribute can typically be separated by other attributes.3
To further evidence this, we use the training set to train a simple logistic regression classifier, using only the attribute values of the reference summaries as features, to discriminate between reference summary types. The test set results in Table 1 show that all summary types are classified with an F1-score above 0.7, attesting to the discriminatory power of our control attributes.
Summarisation performance. Table 2 presents the performance of ATLAS and baseline models using automatic metrics on the test sets of PLOS
3E.g., PLOS lay summaries and abstracts have similar readability distributions but differ in their comprehensibility, length, and entropy distributions. Similarly, PLOS and eLife lay summaries have similar comprehensibility distributions but differ in their readability and length.
and eLife. We include the results for ATLAS under two conditions: 1) one utilizing the average value for each attribute observed in the training data for each summary type (ATLAS); and 2) one using true attribute values obtained from gold standard summaries (ATLASOracle), where ATLASOracle is intended to provide an upper bound of the obtainable performance using our control attributes.
For all metrics, it is evident from Table 2 that ATLAS exceeds the performance of all baseline approaches for both eLife lay summaries and abstracts, demonstrating a strong ability to control the technicality of generated text whilst producing high-quality summaries. Interestingly, although the GPT3.5-mdc baseline achieves a slightly stronger all-round performance for PLOS lay summaries, it fails to maintain this for the more “lay" summaries of eLife where ATLAS achieves significantly better performance, indicating that our control attributes can effectively capture these differences.
In all cases, ATLAS also achieves scores that are comparable to (and sometimes exceeding) that of ATLASOracle, suggesting that the use of the most frequently observed bin value for control attributes is effective for producing the appropriate characteristics for each summary type.
Ablation study. To assess the contribution of each attribute to model performance, we conduct an ablation study, evaluating ATLASOracle under different configurations.4 Table 3 reports the results of this study for abstracts and lay summaries on the combined test sets of PLOS and eLife.
The table shows that the removal of control attributes has a significant detrimental effect on performance. Additionally, when only a single attribute is included, the length-based control has
4We use ATLASOracle as the subject of this experiment rather than ATLAS to get a true reflection of each attribute’s influence, rather than an approximation.
the highest ROUGE scores, particularly for lay summaries. This is to be expected, as lay summaries are known to differ significantly in length between PLOS (avg. 175.6 words) and eLife (avg. 347.6 words). When employing attributes in combination, we can see that the addition of content word entropy control and the subsequent addition of background information control have the greatest benefit to performance for ATLAS with 2 and 3 attributes, respectively. Interestingly, no attribute emerges clearly as the least effective as, although readability score control is the only one not included in the 3 attribute model, its inclusion in the single attribute model has clear benefits for lay summary performance. This provides further evidence that, in combination, our control attributes are able to capture the differences between summary types and effectuate them during generation.
Human evaluation. To provide a comprehensive assessment of the summaries generated, we conducted a human evaluation involving our proposed model ATLAS and the strongest baseline model
(BART) using two experts.5 Specifically, adopting a similar setting to the original that of Goldsack et al. (2022), we take a random sample of 10 articles from the test split of each dataset. Alongside each model-generated lay summary, judges are presented with both the abstract and reference lay summary of the given article. We choose not to provide judges with the full article text in an effort to minimise the complexity of the evaluation and the cognitive burden placed upon them. Using 1-5 Likert scale, the judges are asked to rate the model output based on three criteria: (1) Comprehensiveness: to what extent does the model output contain the information that might be necessary for a non-expert to understand the high-level topic of the article and the significance of the research; (2) Layness: to what extent is the content of the model output comprehensible (or readable) to a non-expert, in terms of both structure and language; (3) Factuality: to what extent is the model generated lay summary factually consistent with the two other provided summaries (i.e. abstract and reference lay summary).6
Table 4 presents the average ratings from our manual evaluation. We calculate the Cohan Kappa scores to measure inter-rater reliability, where we obtain values of 0.50 and 0.57 for eLife and PLOS, attesting to the reliability of our evaluation. The overall results suggest that our proposed method performs better than the BART baseline in terms of all three criteria on both datasets, attesting to their quality. In terms of layness, the higher layness scores observed in the eLife dataset compared to the PLOS dataset align with the previous analysis for the two datasets from (Goldsack et al., 2022). Moreover, compared to baseline, it is worth noting that our model outputs are judged to produce much more factually correct outputs on both datasets, suggesting our method generates fewer hallucinations.
5Both judges have experience in scientific research and hold at least a bachelor’s degree.
6For example, for the “Layness“ criteria, a score of 5 is equal to “highly lay" and a score of 1, “highly technical".
Controllability analysis. To assess the extent to which our control attributes enable controllability over the overall layness of the text, we conduct a further analysis using two additional versions of ATLAS with highly lay or technical values. Specifically, we create ATLASlay and ATLAStechnical by selecting the lowest and highest attribute bins, respectively, for which there are at least 100 observations in the training data (for all attributes other than length which is kept constant).
We examine how these extreme attributes manifest themselves in generated summaries by calculating the average readability values obtained by the generated summaries for both datasets. We present the results of the analysis in Table 5, which show a significant divergence in the readability values obtained by each model on both datasets. Interestingly, this divergence is substantially wider for summaries generated on eLife, the dataset which is identified by Goldsack et al. (2022) as containing lay summaries that are more “lay” than those of PLOS, suggesting that exposure to more extreme values whilst training on this dataset may enable even greater controllability at inference time.7

heading: 5 Conclusion
text: In this paper, we introduce ATLAS, a model for controllable lay summarisation that employs controllable attribute tokens to influence various properties of the generated summary, enabling it to cater to users of different levels of expertise. Using combined datasets for biomedical lay summarisation we perform multiple experiments whereby we confirm the ability of our selected control attributes to discriminate between summary types, demonstrate their effectiveness for controllable lay summarisation, and further investigate their ability to effectuate desired differences during generation.
7Examples of summaries generated by these models are included in the Appendices.
Limitations
Although our results demonstrate that our selected control attributes are able to effectively capture the characteristics between summary types, it is highly likely that there are additional attributes that we have not explored that could benefit performance for controllable lay summarisation. We plan to explore this in future work, in addition to experimenting with more complex methods for enabling controllability.

heading: A Appendix
text: ChatGPT Baseline Prompts The prompts provided to ChatGPT for each summary type are given in Table 6. To ensure a fair comparison, we control the length of the GPT baselines using the generation arguments, (e.g., max_new_tokens). Note that we differentiate between the lay summary types (namely, PLOS and eLife) based on distinctions made by Goldsack et al. (2022), who recognise
PLOS’ summaries as the less “lay" of the two, making them better suited to an audience with some technical knowledge.

heading: Future Work
text: We plan to explore this in future work, in addition to experimenting with more complex methods for enabling controllability