`Don't Get Too Technical with Me': A Discourse Structure-Based Framework for Automatic Science Journalism
Authors: Ronald Cardenas
Bingsheng Yao
Dakuo Wang
Yufang Hou
Venue: EMNLP 2023 Main (2023)
Keywords: automatic scientific journalism
summarization
style transfer
TL;DR: We introduce a new dataset for `scientific journalism', where a summary in journalistic style is generated conditioned on a scientific article. The proposed systems leverage a paper's discourse structure and its metadata to guide generation.

Abstract: Science journalism refers to the task of reporting technical findings of a scientific paper as a less technical news article to the general public audience. We aim to design an automated system to support this real-world task (i.e., automatic science journalism ) by 1) introducing a newly-constructed and real-world dataset (SciTechNews), with tuples of a publicly-available scientific paper, its corresponding news article, and an expert-written short summary snippet; 2) proposing a novel technical framework that integrates a paper's discourse structure with its metadata to guide generation;  and, 3) demonstrating with extensive automatic and human experiments that our model outperforms other baseline methods (e.g. Alpaca and ChatGPT) in elaborating a content plan meaningful for the target audience, simplify the information selected, and produce a coherent final report in a layman's style.

'Don't Get Too Technical with Me': A Discourse Structure-Based Framework for Science Journalism
Ronald Cardenas', Bingsheng Yao², Dakuo Wang', and Yufang Hou"
"University of Edinburgh, "Rensselaer Polytechnic Institute "Nottheateem University, 'IBM Research Europe, Ireland ronald. caa.lldddddddaaaeeedeeee ac. uk, yaobêrpi. edu d. wangenortheasterr. edu, yhoußic ibm. com
Abstract
The enormous quantity of scientific literature and the huge amount of manual effort required produce high-quality science journalistic content inspired recent interest in tasks such as generating blog titles or slides for scientific papers (Vadapalli et al., 201 8; Sun et al., 2021), extracting structured knowledge from scientific literature (Hou et al.. 2019; Mondal al., 2021; Zhang et al., 2022), simplifying technical health manuals for the gen- eral public (Cao et al., 2020), and creating plain language summaries for scientific literature (Dan- govski et al., 2021; Goldsack et al., 2022).
Science journalism refers to the task report- ing technical findings of a scientific paper less technical news article to the general pub- lic audience. We aim to design an automated system to support this real-world task (i.e., tomatic science journalism) by 1) introduc- ing newly-conttructed and real-world dataset (SCITECHNEWS) with tuples of a publicly- available scientific paper, its corresponding article, and expert-written short sum- mary snippet; 2) proposing a novel technical framework that integrates paper's discourse structure with its metadata to guide generation; and, 3) demonstrating with extensive auto- matic and human experiments that our frame- work outperforms other baseline methods (e.g. A Alpaca and ChatGPT) in claborating a content plan meaningful for the target audience, sim- plifying the information selected, and produc- ing coherent final report a layman's style.
Our work focuses on generating simplified jour- nalistic summaries of scientific papers for the non-technical general audience. To achieve this goal, we introduce new dataset, SCiTECH which pairs full scientific papers with their corre- sponding press release articles and newswire snip- pets as published in ACM TechNews. We further carry out in-depth analysis understand the jour- nalists' summarization strategies from different di- mensions (Section 3.2). Then, we explore novel model designs to generate short journalistic sum- maries for scientific papers. Unlike previous stud- ies that model this problem to-sequence task and ignore erucial metadata infor- mation of scientific papers (Dangovski et al., 2021; Goldsack et al., 2022), we propose a technical framework that integrates author and affiliation data as they are important information in scientific news articles. Furthermore, we encode each sen- tence with its corresponding discourse rhetoric role (e.g., background or methods) and apply a hi- erarchical decoding strategy to generate summaries. trained decoding
1 Introduction
Science journalism refers producing journalistic content that covers topics related different areas of scientific research (Angler, 2017). It plays an important role in fostering public understanding science and its impact. However, the sheer vol- ume of scientific literature makes it challenging for journalists to report on every significant dis- covery, potentially leaving many overlooked. For instance, in the year 2022 alone, 185, 692 papers were submitted to the preprint repository arXiv.org spanning highly diverse scientific domains such as biomedical research, social and political sciences, engineering research and multitude of others!. To this date, PubMed contains around 345, 332 sci- entific publications about the novel coronavirus Covid-192, nearly 1.6 times many as those pro- duced in 200 years of work influenza.
"flat" sequence-
As illustrated in Figure model first generates content plan, which is then employed guide the model in producing sum- maries that adhere to the plan's 's structure.
'nttps://infe" awxiv. about/ about/reports/ (r22__arxi__annaa__ee pdf
In summary, our main contributions are two- fold. First, we construct a new open-access high- quality dataset for automatic science journalism
Phttps://wwe http://///www../....oooo wwb.i.nm.....................m.m hi...ggv research/coronavee

---

Input article and Metadata
[AUTHOR] shmelkin I tel aviv university [AUTHOR]... ... [BACKGROUND] master face is face image that passes facebased identity authentication for a large portion of the population ... [CONCLUSIONS] this demonstrated for multiple face representations and explored with multiple. state of the art optimization methods.
Content Plan and Target Summary
[PLAN] [AUTHOR] [BACKGROUND] [BACKGROUND] [RESULTS] [BACKGROUND] [METHODS] [RE- SULTSJ I [AUTHOR] [METHODS] [RESULTS] [SUMMARY] computer scientists at isracl's tel aviv university ( tau say they have developed a "master face" method for circumventing a large number of facial recognition systems, by applying artificial intelligence to generate facial template. the rescarchers say the technique exploits such systems" usage of broad sets of markers to identify specific people producing facial templates that match many such markers essentially creates an omni face that can bypass numerous safeguards the researchers created the master face by plugging an algorithm into generative adversarial network that builds digital images of artificial human faces, the tau team said testing showed the template was able unlock over 20 % of the identities in open source database of 13,000 facial images operated by the university of massachusetls.
An example of scientific article enriched with metadata and scientific rhetoric roles, along with its content plan and target press summary. Colors relate to the plan at the sentence level.
Figure I:
that covers wide range of scientific disciplines. Second, we propose a novel approach that learns the discourse planning and the writing style of journalists, which provides users with fine-grained control over the generated summaries. Through extensive automatic and human evaluations (Sec- tion 6). we demonstrate that our proposed ap- proach generate more coherent and informa- tive summaries in comparison to baseline meth- ods, including zero-shot LLMs (c.g., ChatGPT and Alpaca). In principle, our framework can assist journalists to control the narrative plans and craft various forms of scientific news sum- maries efficiently. We make the code and datasets publicly available at //githu./ com/ htwelldaaeed/siitteedceeee
Automatic Science Journalism. Vadapalli et al. (2018) developed a pointer-generator network generate blog titles from the scientific titles and their corresponding abstracts. Cao et al. (2020) built a manually annotated dataset for expertise style transfer in the medical domain and applied multiple style transfer and sentence simplification models to transform expert-level sentences into lay- man's 's language. The works most closely related to ours are Dangovski al. (2021) and Goldsack et al. (2022). Both studies employed standard seq- to-seq models to generate news summaries for sci- entific articles. In our work, we propose a novel framework that integrates metadata information of scientific papers and scientific discourse structure to learn journalists' writing strategies.
The SCITECHNEWS Dataset
2
Related work
In this section, we introduce SCITECHNEWS, new dataset for science journalism that consists of scientific papers paired with their corresponding press release snippets mined from ACM TechNews. We elaborate on how the dataset was collected and curated and analyze how it differs from other lay summarization benchmarks, both qualitatively and quantitatively.
There are few datasets for generating journalistic summaries for scientific pa- pers. Dangovski et al. (2021) created the Science Daily dataset, which contains around 100K pairs of full-text scientific papers and their corresponding Science Daily press releases. However, this dataset is not publicly available due to the legal and licens- ing restrictions. Recently, Goldsack et al. (2022) constructed two open-access lay summarisation datasets from two academic journals (PLOS and eLife) in the biomedical domain. The datasets con- tain around 31k biomedical journal articles along- side expert-written lay summaries. Our dataset SCITECHNEWS is valuable addition to the ex- isting datasets, with coverage of diverse domains, including Computer Science, Machine Learning, Physics, and Engineering.
Existing Datasets.
3.1 Data Collection
ACM TechNews" is a news aggregator that pro- vides regular news digests about scientific achieve- ments and technology in the areas of Computer Science, Engineering, Astrophysics, Biology, and others. Digests are published three times a week as a collection of press release snippets, where each
9https://technewe acm.org!

---

snippet is written by a journalist and consists of title, summary of the press release, metadata about the writer (e.g., name, organization, date), and a link to the press release article.
Dhiaset
Dous
Doc
words
Summary
SeTTccNewss
non-stigned
29,069
612.56
205.93
6.74
aligne)
176.07
aligted
216:77
We collect archived TechNews snippets between 1999 and 2021 and link them with their respective press release articles. Then, parse each news article for links the scientific article it reports about. We discard samples where we find more than one link to scientific articles in the press re- lease. Finally, the scientific articles are retrieved in PDF format and processed using Grobid*. Fol- lowing collection strategies of previous scientific summarization datasets (Cohan et al., 2018), sec- tion heading names are retrieved, and the article text is divided into sections. We also extract the title and all author names and affiliations.
PLOS (Gooldaack et 2012))
27.525
5.366.70
175.00
(Golback 2022)
4,828
547.60
15.70
LaySumm (Chendnsccizan et 20200
4,426.10
$2.15
Farcka-Alen (Zamin et 2030)
5,304
3,022
635.60
Duily Nowemom (Groky et al. 2018)
2015)
1,200,077
770.09
30.36
(Cotan
177.22
(Colan stal.,
215.913
5,282.27
237.79
Table Comparison of aligned and non-aligned sub- sets of SCITECHNEWS with benchmark datasets for tasks of lay, newswire, and scientific article sum- marization. For SCITECHNEWS statistics for both the Press Release (PR) and the Scientific (Sci.) side are shown. The number of tokens (words) and sentences (sent.) are indicated average.
Table presents statistics of our dataset in com- parison with datasets for lay, newswire, and sci- entific article summarization. Tokenization and sentence splitting was done using spaCy (Honnibal et al., 2020). In total, gathered 29 069 press release summaries, from which 18 933 were linked their corresponding press release articles. From these, 2431 instances -aligned rows in Table 1- were linked to their corresponding scientific arti- cles. In this final subset, all instances have press release metadata (e.g.. date of publication, author). press release summary and article, scientific article metadata (e.g., author names and affiliations), and scientific article body and abstract. We refer to divide
#Instances
Source
Valid
Test
nature
188
320
arxiv
263
231
journals.aps
21
73
dlaacm
67
64
iccexplore.icee
126
14
usenix
11
journals.plos
60
7
author
222
68
other
480
212
Total
1431
1000
Table 2: Most frequent sources of scientific articles in the validation and test set SCITEcHNEWS. The 'au- thor' category refers to papers obtained from authors' personal websites.
this subset as SCITECHNEWS-ALIGNER it into validation (1431) and test set (1000), and leave the rest of the unaligned data as non-parae par-aalllle training data. Figure in the appendix showcases a complete example of the aligned dataset. The train- test division was made according to the source and availability of each instance's s corresponding scien- tific article, i.e., whether it is open access not, The test set consists of only open-access seientific articles, whereas the validation set contains open- access as well as articles accessible only through institutional credentials. For this reason, we re- lease the curated test set to the research community but instead provide download instructions for the validation set.
variation in content type and writing style between scientific abstracts and press summaries.
Knowledge Domain. SCITECHNEWS gathers scientific articles from a diverse pool of knowledge domains, including Computer Science, Physics, En- gincering, and Biomedical, as shown in Table 2. Sources include journals in Nature, ACM, APS, well as conference-style articles from arXiv, IEEE, BioArxiv, among others. Note that sizable chunk of articles was obtained from the authors' personal websites, as shown by the category 'author'.
The readability of scientific arti- cle abstracts and press summaries in our dataset is assessed using the following standard metrics: Grade Level (I'KGL; (Kincaid et al., 1975)), Coleman-Liau Index (CLI; (Coleman and Liau, 1975)), Dale-Chall Readability Score (DCRS; (Dale and Chall, 1948)) and Gunning
Readability.
3.2 Dataset Analysis
We conduct an in-depth analysis of our dataset and report the knowledge domains covered and the
'https://g3 http://///tthabb..c//eeeeett grobidd

---

Metric
NelneeisNosss Sti-PPas
PLOS
2.0
Readability
FKGL]
14.81
14.79
1.5
CLL
15.17
14.23
DCRS,
11.08
11.13
Deatte
Donedte
Gunning4
16.33
16.75
Average.
14.35
14.23
0.5
Abstractivity (%)
Novel unigrams"
14.07
32.12
Novel bigramst
47.32
72.50
1.00
0.26
11.00
0.70
100
0.00
0.25
0.500
0.75
1.00
Choenage
Novel trigramst
70.38
90.21
(a) Scientific Other
Table 3: Differences between scientific abstracts (Sci) and press release (PR) summaries in EEWS in terms of text readability (L. the lower, the more read- proxy for
SerToch News Sei-aSei
arXiv
15
able) and percentage of novel ngrams abstractiveness (1, the higher, the more abstractive).
(Gunning, 1968). These metrics aim to measure the simplicity readability of text by applying experimental formulas that consider the number of characters, words, and sentences in a text. For all these metries, the lower the score, the more read- able or simpler a text is. As shown Table 3, the readability of abstracts and press summaries are on comparable levels (small gaps in scores), in line with observations in previous work in text simplifi- cation (Devaraj et al., 2021) and lay summarization (Goldsack al., 2022). Nevertheless, all differ- ences are statistically significant by means of the Wincoxin-Mamie Banooxin-Man--hhinne Whinney test.
0.55
0.75
1.00
0.50 000
0.55
Covssage
(b) Scientific
Scientific
Figure 2: Extractivity levels of SCITECHNEWS and other summarization datasets in terms of coverage and density distributions, when the reference summary is of different style (Other, a) and when it the same style (Scientific, b) the source document. Warmer colors indicate more data entries.
distribution in terms coverage. Such features in- dicate that SCITeCHNEWs is much less extractive than XRV and constitutes a more challenging dataset for scientific article summarization, as we corroborated with preliminary experiments. Second, we examined the amount information in the reference summary not mentioned verbatim in the source document, proxy for abstractive- ness. Table 3 (second row) presents the percent- age of novel n-grams in scientific abstracts (Sci) and press release summaries (PR). PR summaries show significantly higher level of abstractivity than abstracts, indicating the heavy presence of in- formation fusion and rephrasing strategies during summarization.
Summarization Strategies. We examined and quantified the differences in summarization strate- gies required in our dataset.
First, we assessed the degree of text overlap be- tween the source document (i.e., the scientific arti- cle body) and either the abstract the press sum- mary as the reference summary, as shown in Fig- ure 2. Specifically, we examine the extractiveness level dataset samples in terms of extractive frag- ment coverage and density (Grusky et al., 2018).
When the reference summary is of non-scientific style (Fig. 2a), our dataset shows lower density than PLOS (Goldsack et al., 2022), recent bench- mark for lay summarization. This indicates that the task of science journalism, as exemplified by our dataset, requires following a less extractive strategy, i.e., shorter fragments are required to be copied verbatim from the source document. Simi- larly, when the reference summary is of scientific style (Fig. 2b), our dataset shows far lower density levels compared to ARXIV and more concentrated
Distribution of Named Entities. What type of named entities are reported in a summary can be indicative of the writing style, more precisely of the intended audience and communicative goal of said summary. We quantify this difference by compar- ing the distribution of named e in scientific abstracts against that press summaries.
As shown in Figure 3, press summaries show a
*Calculted using the textstat package.
'Extracted using the spaCy library.

---

4 Problem Formulation and Modelling
PR Summary
3.0
Average Ehtity Henneney
Sei Summary
We cast the problem of science journalism as an encoder-decoder generative task and propose a model that performs content planning and style transferring while summarizing the content. Given scientific article text D, enriched with metadata information M, the task proceeds in two steps. First, a plan is generated conditioned on the input document, p(s) D, M), followed by the summary y, conditioned on both the document and the plan, p(y|s, D, M). Following Narayan et al. (2021), we train encoder-decoder model that encodes input document and learns generate the concate- nated plan and summary as single sequence.
2.5
2.0
1.5
1.0
0.5
0.0
ORG
NUMBER
MISC
PERSON
DATE
LOC
Named Entity
Figure 3: Average frequency of named entities in press
Let = (Io. xN) be a scientific article text, modeled as a sequence sentences, let M be the set of author-affiiation pairs associated with the said article, and let Y be the target summary. We define D' (m, mo, ... M[M I to, To. ... IN, NN) the input to the encoder, where is special token indicating the beginming of metadata information, m, M is an author name concatenated the cor- responding affiliation, and lj a label indicating the scientific rhetorical role of sentence 24z.
release (PR) summaries and scientific abstracts (Sci).
high presence of organization and person entities, whereas scientific abstracts report mostly number entities. It is worth noting the low, however ticeable, presence of organization and person enti- ties in the scientific abstracts. Upon closer inspec- tion, these entities referred scientific instruments and constants named after real-life scientists, c.g., the Hubble telescope. In contrast, person entities in press summaries most often referred to author names, whereas the organization entity referred to their affiliations.
Given the encoder states, the decoder proceeds to generate plan s conditioned on D' p(s) ((ss D': θ), where θ are the model parameters. The plan is defined as ([PLAN] 80; 9/y)) where 8% is label indicating the rhetorical role sentence yk in summary y should cover. Figure shows an example the annotated document and content plan. We use Bart encoder-decoder architecture (Lewis et al., 2020) and train it with D' as the source and [s; v] (plan and summary concatenated) as the target. We call this model Bart plan.
Discourse Structure Next, we examine the dif- ference scientific discourse structure between abstracts and press release summaries. We employ the model proposed by Li et al. (2021) trained on the PubMed-RCT dataset (Dernoncourt and Lee, 2017), and label each sentence in summary with its rhetorical role, e.g., background, conclusion, method, among others? Figure presents the pres- ence of rhetorical roles along with their relative positions in the summaries. Scientific abstracts tend to start with background information, then present methods, followed by results, and finish with conclusions. In contrast, press release sum- maries tend to emphasize conclusions way sooner than abstracts, taking the spotlight away from re- sults and, a lesser degree, from methods. Surpris- ingly, the relative presence of background informa- tion seems to be similar in both abstracts and press release summarics, in contrast with its emphasized presence in lay summaries, as reported in previous work (Goldsack et al., 2022).
Experimental Sctup
In this section, we elaborate the baselines used and evaluation methods, both using automatic met- rics and eliciting human judgments. Following previous work (Goldsack et al., 2022), we use the abstract followed by the introduction as the arti- cle body and prepend the metadata information as described in the previous section.
5.1 Comparison Systems
We compare against the following standard base- lines: Extractive Oracle, obtained by greedily se- lecting N sentences from the source document
'Li ct al. (2021) report F-scores of 0.95 and 0.84 for sei- entific discourse tagging on two datasets from the biomedical
and computational linguistic domains, respectively.

---

BACKGROUND
CONCLUSIONS
RESULTS
OBJECTIVE
METHODS
Sei Summary
PR Summary
10-
2.0
Label Freqnney
Label Eequeeey
0.6
1.6-
0.4-
1.4
0.0
0.0
0.6 Relative Sentence Position
0.0
0.2
0.8
Relative Sentence Position
Figure 4: Distribution of scientific discourse tags in scientific abstracts (Sci) and press release (PR) summaries in SCITECHNEWS.
maximizing the ROUGE score (rouge 1 + rouge 2) against the reference summary; LEAD, which picks the first N sentences of the source document; and RANDOM, which randomly selects N sentences following a uniform distribution. For all periments, we use N 5, the average number of sentences in PR summaries. Additionally, we report the performance of the scientific abstract, ABSTRACT, which provides an upper bound to ex- tractive systems and systems that do not perform style transfer nor include metadata information.
based on GPT3 (Brown et al., 2020); FlanT5- Large (Chung et al., 2022), fine-tuned on T5- 3B (Raffel et al., 2020); and Alpaca 7B (Taori et al., 2023), an LLaMA (Touvron al., 2023). We employ the same instruction prompt followed by the abstract and introduction for all systems, "Write a report of this paper in journalistic style."
version of
5.2 Evaluation Measures
Given the nature of our task, we evaluate the intrin- sic performance of our models across the axes of summarization and style transfer.
For unsupervised baselines, we compare against LexRank (Erkan and Radev, 2004) and TextRank (Mihalcea and Tarau, 2004), two extractive systems that model the document as a graph of sentences and score them using node centrality measures. For supervised systems, we choose BART (Lewis et al., 2020) as encoder-decoder architecture and use the pretrained checkpoints for BART-LARGE avail- able the Hugging library (Wolf al., 2020). The following BART-based systems are compared: Bart finetuned on the ARXIV dataset (Cohan et al., 2018); Bart finetuned on SCITECH- NEWS with only the abstract and introduction text input, without metadata information or scientific rhetoric labels, and tasked to generate only the tar- get summary without plan; and finally, Bartmeta- trained with metadata and article input and sum- mary without plan the target.
Summarization. The informativeness, rele- vance, and fluency of the generated summaries are evaluated using ROUGE 1, 2, and L, respectively (Lin, 2004).² Semantic relevance is evaluated with BertScore (Zhang et al.) using RoBERTa-large as base model (Liu et al., 2019) and in-domain importance weighting." 10 All evaluations made over the summary text after stripping away the generated content plan.
Style Transfer To distinguish between press re- lease style and scientific style, we train binary sentence classifier using press release summary sentences from the unaligned split of SCITECH- NEWS positive samples, and equal amount of sentences from scientific abstracts from arXiv (Cohan et al., 2018) as negative examples. We
Finally, we benchmark on recently proposed large language models (LLM) fine-tuned on the instruction-following paradigm: GPT-3.5-Turbo!.
"As calculated by the roage-score library. "BertScore has been proven a reliable metric when equipped with importance weighting in highly technical do- mains such as medical texts (Miura et al., 2021; Hossain et al., 2020).
"We used model gpt-3. 9pt-3.5-turee 5-tt-bb--0000 in https: //pl allaforrm openai. com/ docs comddocs/mmmelll.....c

---

the RoBERTa-base model as implemented in the huggingface library in sequence classification setup. Then, the style score (Sy(S) of summary S is defined as the probability of the positive class, averaged over all sentences in S.
Systems
R2
BSc
ABSTRACT
32.94
6.26
28.84
81.20
EXT-ORACLE
39.73
10.43
34.10
84.49
Lead
32.46
5.79
28.17
83.81
Random
29.58
3.99
25.50
82,60
LexRank
31.40
5.21
27.16
82.98
TextRank
31.86
5.38
27.38
82.92
Faithfulness. Factual consistency of generated summaries with respect their source document quantified using QuestEval (Scialom et al., 2021).
Bartera
32.28
6.01
28.12
82.81
Bartsar
36.42
7.51
31.71
84.12
Bartmete
38.07
9.03
33.14
84.76
Barppian
38.84
8.89
33.50*
84.78
Human Evaluation. We take random sample of 30 items from the test and conduct study using Best-Worst Scaling (Louviere et al., 2015), method that measures the maximum difference be- tween options and has been shown to produce more robust results than rating scales (Kiritchenko and Mohammad, 2017). Human subjects were shown the source document (abstract, introduction, and metadata) along with the output of three systems. They were asked to choose the best and the worst according the following dimensions: mativeness how well the summary captures im- portant information from the document; (2) Factu- ality whether named entities were supported by the source document:"! (3) Non-Redundancy- - if the summary presents less repeated information; (4) Readability if the summary is written in sim- ple terms; (5) Style whether the summary text follows journalistic writing style; and finally, (6) Usefulness how useful would the summary be as a first draft when writing a press release summary of a scientific article. Systems are ranked across a dimension by assigning them a score between -1.0 and 1.0, calculated as the difference between the proportion of times it was selected as best and selected worst. See Appendix D for more details,
Alpacu
21.24
3.24
18.16
81.20
F-annS-
26.26
4.98
20.13
80.98
GPr-3.5-Turbo
35.67
6.75
28.68
82.86
Table 4: Results terms of ROUGE-FI scores (R1, R2, and RL) and BertScore FI (BSc). Best systems in bold. statistically significant w.r.t. to the closest baseline with 95% bootstrap confidence interval.
Systems CLI
QEval f
Styt
Bartare
15.33
47.90
0.18
(1) Infor-
Bartsas
13.70
36.54
0.96
Bartmeta
13.43
36.91
0.98
Bartzian
13.55
38. 18.16
0.98
Alpaca
13.82
38.00
0.25
FlanTT--large
16.36
44.36
0.10
GPT-3.5-Tume
16.36
46.51
0.81
Summary 16.52
33.95
0.99
Table 5: Performance systems terms of readability (CLI), faithfulness (QuestEvval score; QEval), and our style score (Sty). (t.)): higher, lower is better.
to previously reported extractive upper-bounds in lay summarization (Goldsack et al., 2022). This further confirms that the kind of content covered in press release summaries and scientific abstracts are fundamentally different, as explored in Sec- tion 3.2. For the abstractive systems, we notice that Bartmeta significantly improves over Bartsar, highlighting the critical importance of adding meta- data information the source document. Generat- ing scientific rhetorical plan as part of the output further improves informativeness (Rouge-1) and fluency (Rouge-L., as well as semantic relevance (BertScore) of the produced content. is worth noting that the assessed LLMs, Alpaca, FlanTS, and GPT-3.5, underperform the proposed models, indicating that the task poses significant challenge to them under zero-shot conditions.
6 Results and Discussion
In this section, present and discuss the results of our automatic and human evaluations, provide comprehensive analysis of the factuality errors our systems incur, and finish with a demonstration of controlled generation with custom user plans.
6.1 Automatic Metries
Informativeness and Fluency. Table 4 presents the performance of the compared systems in terms of ROUGE and BertScore. We notice that the extractive upper-bounds, ABSTRACT and EXt- ORACLE, obtain relatively lower scores compared
Readability, Faithfulness, and Style. First, we find that adding metadata and plan information reduces syntactic and lexical complexity and im- proves faithfulness, as shown in Table Inter-
We consider names of people, locations, organization, as well as numbers.

---

System
Inf.
N-Rd.
Fact.
Read.
Sty.
Use.
Redundancy and Style can be due to its memoriza- tion of highly frequent patterss in the dataset, e.g.. 'researchers at the university of.... In contrast, 0.02
Baflmett
-0.31
0.01
0.16
-0.22
part
0.08
0.08
-0.10
0.22
0.30
GPT-35
-0.07
-0.01
0.02
-0.23
-0.24
-0.21
Bartplan generates more diverse and stylish text. Third, whereas GPT-3.5's high Factuality score can be attributed to the difference in the number of ar- chitecture parameters, its low Readability and Style scores indicate that the simplification and styliza- tion of complex knowledge still pose significant challenge. Finally, in terms of Usefulness, users preferred Bartplan as a starting draft for writing a press release summary, demonstrating the model's effectiveness for this task.
Som.
0.68
0.43
0.79
0.57
Table 6: System ranking according human judg- ments, along (Inf)ormativeness. WNon-RedJundanyy (Fact)uality, (Read)ability, Press Release (Sty)le, and (Use)sulness, Best system is shown in bold.
estingly, FlanT5 and GPT-3.5 generate seemingly more complex terms, followed by Bartarz- Upon inspection, these systems showed highly extractive behavior, i.e., the produced summaries are mainly composed of chunks copied from the input verba- tim. We hypothesize that this property also inflated their respective faithfulness scores. Note that gold PR summaries show a low QEval score, indicat- ing that faithfulness metries based on pre-trained models are less reliable when the source and target texts belong to a highly technical domain or dif- fer in writing style. In terms of style scoring, we observe that models finetuned on our dataset are capable of producing summaries in press release style, specific kind of newswire writing style. See Appendix E for few generation examples by mart and Bartplan-
6.3 Factuality Error Analysis
We further analyzed the types of factuality errors our systems incurred on. We uniformly sampled 30 instances from the test set and manually annotated their respective reference summary and summaries generated by Bartptan and GPT3.5-Turbo.
We adapt the error taxonomy employed in Goyal and Durrett (2021) and consider three categories the span level. (i) Entity-related, when the span is a named entity (same entity categories considered in Section 3.2.); (ii) Noun Phrase-related, when the span is an NP modifier: and (iii) Other Errors, such as repetitions and grammatical errors. Each error category except "Other' is further divided into sub-categories: Intrinsic, Extrinsic, and World Knowledge, depending on where the supporting information is found (Cao and Wang, 2021). In- trinsic errors are caused when phrases or entities found in the input are generated in the wrong place. In contrast, extrinsic errors happen when the gener- ated span cannot be supported by the input or any external source (e.g., Wikipedia). Finally, word knowledge errors are caused when the span cannot be verified with the input but it can be verified us- ing external knowledge, e.g. author X being the director of the C.S. department at university Y.
6.2 Human evaluation
Table 6 shows the results of our human evalua- tion study, comparing models effective at encoding metadata (Bartmeta). generating a plan (Bartpion) and strong LLM baseline (GPT-3.5). Inter- annotator agreement Krippendorff's alpha (Krip- pendorff, 2007) was found to be 0.57. Pairwise statistical significance was tested using a one- one-way ANOVA with posthoc Tukey-HSD tests and 95% confidence interval. The difference between prefer- ences across dimensions was found to be significant (p < 0.01) for the following pairs: expert-wing -epite-written gold PR summaries vs. all systems, in all dimen- sions; for Non-Redundancy, Bartyian and GPT3.5 against : for Factuality, Bartmeto GPT- 3.5; for Readability, Bartptan vs all systems and Bartmeto vs GPT-3.5; and for Style and Usefulness, all pairs combinations were significant.
Table presents the proportion of error cate- gories found in the inspected summaries, along with the total number of error spans found for each system. It is worth noting that the total number of errors follows the ranking trend in Table 6, with PR summaries having the least number of errors, fol- lowed by GPT3.5, and then Bartpion First, we ob- serve that reference summaries exhibit only Entity and NP-related errors of type World Knowledge.
The results indicate the following. First, scores for PR summaries are higher than machine- generated text, further confirming the difficulty of the task and showing ample room for improve- ment. Second, Bart rather low scores in Non-
12ehe event-related category is not considered here since the source documents in SCITECHNEWS do not contain events.

---

Oracl Plan Generuted Summary
[PLAN] HHUT (CONCLUSIONS]] [CONCLUSIONS] LAUTHOR JCUNCLUSIONS) MMMMY)
Researchers at the University of Basel in Switzerland say they have developed a method for generating a secure quantum key via photonic conversion. to the initial messrremnn data, which cumnot be known costrolled by advensury, to guaruniee that the key will be obiained by both Alice and Bob, who are connected by a quanium channel. By adling the noise the researchers were able to increase the threshold at which they guarantce the key obtained. Althhugh the researchers their methed not perfect, they believe is still effective and could lead new generation of highly secure and provable quantum key distribution. They nore that new method provides security even when assumptions ahout the securily of QKD are not fulfilled.
The method adds artificial
User Plan & Generated Summary
[PLANI [AUTHORI [BACKGROUND] [METIODS] [RESULTS] (CONCLUSIONS) ISUMMARY]
Researchers at University of Basel in Switzerland say they have developed method for generating a secure quantum key via photonic conversion. The methed involves adding artificial noise to the initial measurement dara, which cannot he known or controlled by an adversary, to increase the probability fully device independent, they say il provides security even when assumptions
that key obtuine.. Allhough reseurchers say method are made about the security of the quantum key distribution. "There is fundamental obstacle in the development of QKD, i.e.. the requirement that an adversary cannot fully control the quantum channel," says Bosel Professor Sangoward Bancal, However, be says method provides sufliciont bounds on the minimum required global detection efficiency to ensure that the informarion sent over the qubit channel is good and nocurate.
Figure 5: Example of generation by Bartplon conditioned to a user plan. Text and plan labels are color-coded.
Entity Ext.
Noun Phrase
System
Other
Total
Int.
W.K.
Int.
Ext.
W.K.
PR Sum.
0.0
0
0.79
0.0
0.0
0.21
0
43
part
0.1
0.34
0.20
0.07
0.16
0.02
0.11
61
GPT3.5
0.0
0.08
0.0
0.02
0.18
0.0
0.72
50
Table 7: Proportion of factuality errors in different systems with error proportions normalized by system.
The majority of them include completed names of affiliated institutions (e.g., the metadata mentioning only the abbreviation 'MIT' but the reference sum- mary showing the complete name), country names where these institutions are located, or the posi- tion an author holds within an institution. We also found cases where an author held more than one affiliation, with only one of these being listed in the metadata and another mentioned in the reference summary. Second, we observe that Bartpian extrin- sically hallucinates mostly entities (e.g., country names, 34% of all its errors), followed by extrinsic NPs. Among intrinsic errors, entity-related ones in- cluded mixed-ups of author first names, last names, and affiliations, whilst NP-related errors included mistaking resources mentioned in the source (e.g. a github repository) for institutions. In contrast, GPT3.5 produced sizable amount of extrinsic hal- lucinations of noun phrases, most of them stating publication venues (e.g., "In a paper published in ..."). Since the metadata added to the source does not include publication venue, the model must have surely been exposed to this kind of information during training. However, somewhat surprisingly, GPT3.5-Turbo did not exhibit world knowledge errors of any kind, potentially highlighting the con- servative approach generation followed during its training. Errors of type 'Other' consisted mainly of orphaned phrases at the beginning of generation, i.e., the model starts the generation by attempting
to continue the last sentence of the input in a 'con- tinue the story' fashion. We hypothesize that the GPT3.5 model employed (checkpointed at March, 2023) struggled with the length of the input, reach- ing point where the prompt instruction (stated at the beginning of the input) is completely ignored and the model just continues the "story' given.
6.4 Controlled Generation with User Plans
The proposed framework opens the possibility of controlling the content and the rhetorical structure of the generated summary by means of specifying custom plans of rhetorical labels. Figure 5 presents an example of this, showing that Bartplan generates content from all the requested roles in the plan, following most of the precedence order stated.
7 Conclusions
This paper presents a novel dataset, SCITECH- NEWS, for automatic science journalism. We also propose a novel approach that learns journalistic writing strategy and style by leveraging the pa- per's discourse structures. Through extensive auto- matic evaluation and human evaluation with base- line methods (e.g.. ChatGPT and Alpaca), we find that our models can generate high-quality informa- tive news summaries that closely resemble those crafted by professional journalists.

---

8 Limitations
References
The introduced dataset in English, as a result, our models and results are limited to English only, Future work can focus on the creation of datasets and the adaptation of science journalism to other languages. Also of relevance is the limited size of our dataset, and the potential lack of balanced cov- erage on the reported knowledge domains. Finally, LLMs results suggest that a more extensive prompt engincering might be critical to induce generation with adequate press release journalistic style.
M.W. Angler. 2017. Science Journalism: An Introduc- lion. Routledge.
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on NeuRIPS, pages 1877-1901.
Shuyang Cao and Lu Wang. 2021. Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on EMNLP, pages 6633-6649.
Another limitation of our approach is the usage of only author and affiliation metadata as additional input information. We decide to only consider this metadata for the following reason. Considering the distribution of named entities found in Press Release reference summaries (analyzed in Section 3.2 and depicted in Figure 3), it worth noting that entities of type Organization and Person are the most frequent entities- after numbers and miscel- laneous. Hence, we decided to restrict the usage of metadata in our framework to author's 's names and affiliations. However, other metadata information collected, both from the scientific article (e.g. publication venue and year, title) and press release articles (e.g. title, PR publication date, journalistic organization), as detailed in Scction 3.1. We in- clude the complete metadata in the released dataset so that future investigations can leverage them.
Yixin Cao, Ruihao Shui, Liangming Pan, Min Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communi- cation between experts and laymen. In Proceedings of the 58th Annual Meeting of the ACL, pages 1061- 1071. Association for Computational Linguistics.
Muthu Kumar Chandrasekaran, Guy Feigenblat, Ed- uard Hovy, Abhilasha Ravichander, Michal Shmueli- Scheuer, and Anita de Waard. 2020. Overview and insights from the shared tasks scholarly document processing 2020: Cl-scisumm, laysumm and long- summ. In Proceedings of the First Workshop on Scholarly Document Processing, pages 214 224.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling
language mod- els. arXiv preprint 120X::22100111116
9 Ethics Statements
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Scokhwan Kim. Walter Chang, and Na- zli Goharian. 2018. A discourse- aware attention model for abstractive summarization of long docu- ments. In Proceedings of the 2021 Conference on NAACL-HLT, pages 615-621.
The task of automatic seience journalism is in- tended to support journalists or the researchers themselves writing high-quality journalistic con- tent more efficiently and coping with information overload. For instance, a journalist could use the summaries generated by our systems as an initial draft and edit it for factual inconsistencies and add context if needed. Although we do not foresee the negative societal impact of the task the accompa- nying data itself, we point at the general challenges related to factuality and bias in machine-generated texts, and call the potential users and developers of science journalism applications to exert caution and follow up-to-date ethical policies.
Meri Coleman and Ta Lin Liau, 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology: 60(2):283.
Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational re- search bulletin, pages 37-54.
Rumen Dangovski, Michelle Shen, Dawson Byrd, Li Jing, Desislava Tsvetkova, Preslav Nakov, and Marin Soljačić. 2021. We can explain your search in layman's terms: Towards automating sci- ence journalism at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14): 12728 12737.
Franck Dernoncourt and Ji- Yo-Y Lee. 2017. Pubmed 200k RCT: dataset for sequential sentence classi- fication in medical abstracts. In Proceedings of the 8th IJCNLP pages 308-313.

---

Ashwin Devaraj. Byron C. Wallace, lain Marshall, and Junyi Jessy Li. 2021. Paragraph-level simplification of medical texts. In Proceedings of the 2021 Con- ference on NAACL-HLI, volume 2021, page 4972. NIH Public Access.
Svetlana Kiritchenko and Saif Mohammad. 2017. Best- worst scaling more reliable than rating scales: A case study sentiment intensity annotation. In Pro- ceedings of the 55th Annual Meeting of the ACL, pages 465-470.
Klaus Krippendorff. 2007. Computing krippendorf''s alpha-reliabillty. Departmental Pupers (ASC), UPenn.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph- based lexical centrality as salience in text summarization, Journal of artificial intelligence re- search, 22:457-479.
Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli- based models for inconsistency detection in summa- rization. Transactions of the ACL, 10:163-177.
Tomas Goldsack, Zhihao Zhang. Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Corpora for the lay summarisation of scientific lit- crature. In Proceedings of the 2022 Conference on EMNLP, pages 10589-10604. Association for Com- putational Linguisties.
Mike Lewis Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising Sruuence----sqqeence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th An- nual Meeting of the ACL, pages 7871-7880.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality summarization. In Proceedings of the 2021 Conference on NAACL- HLT. pages 1449 1462.
Xiangei Li, Gully Burns, and Nanyun Peng. 2021. Sci- entific discourse tagging for evidence extraction. In Proceedings of the 16th Conference of the EACL, pages 2550-22 2562.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference on NAACI-IILT, pages 708-719.
Chin- Yew Lin. 2004. Rouge: evaluation of summaries. In Text summarization branckes out, pages 74-81.
A package for automatic
Robert Gunning. 1968. The technique of clear writing, rev. ed edition. McGraw-Hill
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining ap- proach. CoRR, abs/1907.11692
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th Inter- national Conference on NeuRIPS, pages 1693-1701.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Con- ference on Learning Representations.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy:
Industrial-strength Natural Language Processing in Python.
Jordan Louviere, Terry N Flymn, and Anthony Al- fred John Marley, 2015, Best-worst scaling: The- ory, methods and applications. Cambridge Univer- sity Press.
Tamanna Hossain, Robert L Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. Covidlies: Detecting covid-19 misin- formation on social media. In Proceedings of the 1st Workshop NLP for COVID-19 19 (Part 2) at EMNLP 2020.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring- ing order into text. In Proceedings of the 2004 con- ference on EMNLP, pages 404-411.
Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Lan- glotz, and Dan Jurafsky. 2021. Improving factual completeness and consistency image-to-text radi- ology report generation. In Proceedings of the 2021 Conference on NAACL-HLT, pages 5288-5304.
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metries, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the ACL, pages 5203-5213. Association for Computational Linguistics.
Ishani Mondal, Yufang Hou, and Charles Jochim. 2021. End-to-end construction of NLP knowledge graph. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages Association for Computational Linguistics.
Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability in- dex, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Re- search Branch.
Shashi Narayan, Shay Cohen, and Mirella Lapata. 2019. What is this article about? extreme summa- rization with topic-aware convolutional neural net- works. Journal of Artificial Intelligence Research. 66:243-278.

---

Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simöes, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with learned entity prompts for abstractive summarization. Transactions of the ACL, 9:1475- 1492.
turn humans a question: Question-ans answer pair generation for children's 's story books. In Proceed- ings of the 60th Annual Meeting of the ACI., pages 731-744.
Farooq Zaman, Matthew Shardlow, Saced-Ul Hassan, Naif Radi Aljohani, and Raheel Nawaz. 2020. Htss: A novel hybrid text summarisation and simplifica- tion architecture. Information Processing Man- agement, 57(6):102351.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Ad- versarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the ACL, pages
Shao Zhang, Hui Xu, Yuting Jia, Ying Wen, Dakuo Wang, Luoyi Fu, Xinbing Wang. and Chenghu Zhou, 2022. Geodeepshovel: A platform for building sci- entific database from geoscience literature with ai as- sistance. Geoscience Data Journal.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former, The Journal of Machine Learning Research, 21(1):5485 5551.
Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Confer- ence on Learning Representations.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Questeval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on EMNLP, pages 6594 6604.
A Example Snippet
Figure showcases a complete example of an ACM TechNews snippet paired with the scientific paper talks about.
Edward Sun, Yufang Hou, Dakuo Wang. Yunfeng Zhang. and Nancy X. R. Wang. 2021. D2S:
Document-to-sie to-iide generation via query-based text summarization. In Proceedings of the 202/ Con- ference on NAACL-HLT, pages 1405-1418. Associ- ation for Computational Linguistics.
B Training and Resource Details
BART models were trained two NVIDA A A100 GPUs, each with 80GB of memory, using Adam optimizer (Loshchilov and Hutter, 2018) with a learning rate of le 6, batch size of 128, for a maximum of 5.000 steps. LLM experiments were run on one NVIDIA A100 40G graphic card. For Laarge, we use maximum length of 256, beam size of 5, temperature of 0.9, top_k of 100, and use early stopping. For Alpaca, the default generation parameters are used.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- ford alpaca: An instructionn-ollloiing llama model. https:/ /ttp://gttttttttt com/tatsu-labl stanford_alpace
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13977
C Supplementary Readability and Faithfulness Evaluation
Raghuram Vadapalli, Bakhtiyar Syed. Nishant Prabhu, Balaji Vasan Srinivasan, and Vasudeva Varma. 2018. When science journalism meets artificial intelli- gence: An interactive demonstration. In Pro- ceedings of the 2018 Conference on EMNLP: Sys- tem Demonstrations, pages 163-168. Association for Computational Linguistics.
Table 8 presents supplementary performance re- sults of our systems w.r.t. readability and faith- fulness. In addition to QuestEval, we report entailment-based scores SummaC (Laban al., 2022) and Adversarial NLI (Nie et al., 2020).
D Human Evaluation
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. 2020. Transformers: State-of-the the- art natural language processing. In Proceedings of the 2020 conference on EMNLP: System demonstra- tions, pages 38-45.
Following a typical human evaluation setup as in the previous literature (Yao et al., 2022), cruited 5 volunteers for human evaluation, all PhD students in Computer Science, and hosted the study on an internal server. Participants were selected so that their area of expertise do not overlap signif- icantly with the topic of the articles in the study.
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Li, Mo Yu, and Ying Xu. 2022. It is ai's

---

ACM TechNews Snippet
Title
Researchers Say They' 've Found Wildly Successful Bypass for Face Recognition Tech
Press Summary
Computer scientists Israel's Tel Aviv University (TAU) say they have developed a "master face" method for circum- venting large number of facial recognition systems, by applying artificial intelligence to generate facial template. The rescarchers say the technique exploits such systems' usage hroad of markers identify specific people; producing facial templates that match many such markers essentially creates an omni face can bypass numerous safeguards. ...
Press Release
addition helping police arrest the wrong person or monitor how often you increasingly used by companies as routine security procedure: it's a way unlock your phone or log into social media, for example. This practice comes an exchange of privacy for the promise comfort security but, according a recent study. ...
Gap, facial recognition
Scientific Article
Title
Generating Master Faces for Dictionary Attacks with Network-Assisted Latent Space Evolution Abstract
A master face is face image that passes face- sase-based auttiiitaathhetiaatiionn for large portion of the population. These faces be used impersonate, with high probability of success, any without having access any user-inforre inrrinfforaaion. We optimize these faces, using evolutionary algorithm in the latent embedding space of the StyleGAN face generator. Multiple evolutionary strategies compared.... Main Body
I. INTRODUCTION
dictionary attacks, one attempt pass authentication system sequentially trying multiple inputs. real-world biometric systems, one can typically attempt only handful of inputs before heing blocked. However, the matching biometrics not exact, and the space of biometric data uniformly distributed. This may suggest that with handful samples, can
larger portion of the population....
Figure 6: Example from our SCITECHNEWS dataset showing a complete scientific article (title, abstract, and main body; bottom) and its associated ACM TechNews snippet (title, press summary, and press release article; top).
The study comprised a sample of 30 scientific arti- cles, and each participant annotated all articles but were allowed to do so in their own pace and time. Moreover, we discouraged participants from doing more than 5 articles in single sitting.
ing summaries generated by systems Bartmeta and Bart
F Controlling Generation with User Plans
shown brief description of the task, followed by the sci- entific article (abstract and introduction), metadata information, along with the output of three systems (Narayan et al., 2019). Then, they were asked to select the best and worst systems according to the dimensions mentioned in Section 4. In case there significant difference between all systems, participants were instructed to select all systems as best and worst. Similarly, if there cant difference between the best and second best, or worst and second worst, participants were allowed to select both systems. The score of a system is cal- culated as the proportion of times it was selected as best minus the proportion of times it was selected worst. Hence, the score can be value between -1 and 1.
As shown in Figure 7, participants
Figure 10 presents a complete example of summary generation with Oracle, system- seneem-eeeeatdd, and user plans. Notice that Bartpian generates content from all the requested roles in the plan, following most of the precedence order stated.
no signifi-
E Example of System Outputs
Figure and 9 showcase press relcase sum- maries from SCITECHNEWS and the correspond-

---

Readability
Faithfulness
System
FKGL4
CLL
DCRS}
Gunning,
QEvalt
Sumc
ANLI
Bartaræ
15.21
15.33
11.71
17.27
47.90
80.12
69.95
BartsaT
15.40
13.70
10.74
17.36
36.54
28.62
22.77
Bartmeta
15.22
13.43
10.66
17.21
36.91
28.33
25.30
Bartplan
15.35
13.55
11.03
17.59
38.16
28.54
28.96
Alpaca
12.21
13.82
11.00
14.04
38.00
67.97
48.76
FlanT5-large
15.12
16.36
11.92
16.97
44.36
73.76
63.26
GPT-3.5-Turbo
14.68
16.52
11.29
16.03
46.51
55.02
49.82
PR Summary 15.16
14.61
11.51
17.25
33,95
27.09
31.10
Table 8: Supplementary performance results of systems in terms of readability (the lower the better) and faithful- ness (the higher the better). QEval: QuestEval; Sumc: SummaC; ANLI: Adversarial NLL
Bleman Stady
Figure 7: Web interface used for the human study introduction in Section

---

Press Release Summary
university of illinois at urbuna- champaign uiuc) scientists have created a computer model of simple brain network slug, , which was trained to find food. the asimov simulated shug placed in confined setting
hased on that of randomly encounter food pellets that were alternately delicious noxious, emitting a characteristic odor either attract repulse the slug. asimov was programmed to experience satiation well reward, and the addition of rewarding but nutritionally empty pellet enticed the slug pursue it the exclusion of all else. although consuming this pellet caused satiation, that sense of fullness dissipated over time, with homeostatic plasticity mechanism causing asimov experience withdrawal. uiue 's thanor gillette said,, by watching how this brain makes sense of its environment, expect to learn more about how real- world brains work."
Bartmeta
researchers at the university of illinois at urbana- champaign uiue) have developed artificial intelligence ( ai system that simulates the addictive process, incorporating sense of taste and disgust well as learning mechanisms that drive addiction. the team used virtual reality to create cyberslug, agent that forages based value and probability, with the artificial intelligence system learning to approach or avoid prey hased on information such as nutritional value, comfort, and the potential of infection. asimov eventually feel the effects of withdrawal. uiuc's thanor gillette said, through that simulation, can see that has very similar aesthetic sense addiction and that the feelings that people with addiction have may be similar to those that other people have."
exposed to stimuli that it did not like, such as choking, and would
Bartaian
a university of illinois at urbana- champaign ( uiuc) scientist has developed an artificial intelligence ( ai system that simulates drug- like addiction, with the goal of making easy to visualize the effects of such addiction computer. the program gives the ai visual cues to expect to receive high reward, but also warns the agent of the potential unpleasant feelings that may follow after consuming that reward. ekaterina gribkova's 's lab developed the artificial intelligence replicate the neural circuitry involved in addiction, and showed that ai can be programmed to watch movies, listen music, and act visual cues. said gribkov. if you want understand what addiction like, you start with the analogs and find out how the addictive processes connected to each other."
GPT-3.5
Aesthetic decision-making in primitive generalist foragers, shedding light on the coevolutionary origins of aesthetics and addiction. According to recent study published in the journal PLOS Biology. rudimentary aesthetic sense can be found in the stimulus valuations and cost-henefit decisions made by primitive generalist foragers. This aesthetic sense is based on factors governing personal economic decisions such as incentive, appetite, and learning. The study found that the addictive process is extreme expression of aesthetic dynamics. The rescarchers used an interactive, agent-based model called ssioov to reproduce simple aesthetic sense from known neural relations of cost-benefit foraging. In the presence of very high reward, addiction-likk process emerges. A drug-like prey provides extreme reward with nutritive value, initiating high selectivity and prolonged cravings for the drug through reward learning.
First example of reference summary along with corresponding output summaries of systems Bartmeta- Bartpion and GPT-3.5.
Figure 8:

---

Press Release Summary
a leam researchers from massachusetts institute of technology (mit) ), arizona state university. and the university massachusetts at amherst developed a new control system that enhances the speed and agility of legged robots as they leap across gaps. the control system algorithmically processes and translates real time forefront video input into instructions for bodily movement. the researchers combined the best elements of controllers that do not incorporate vision into a separate module that handles vision in real time, and trained the controller through reinforcement learning. tests of the system when installed into mit mini cheetah robot found it outperformed other systems that use a single controller, enabling it to successfully cross 90 % of physical terrains.
Bartmeta
researchers the massachusetts institute of technology ( mmi) ) have developed a quadruped robot that can successfully jump through wide gaps, and across uneven terrain. the team notes this an important milestone toward the development " blind walking. which relies on having the robot navigate without vision. without vision, the robot would either step the gap or fall or otherwise treat the gap as an obstacle and stop. the researchers say. their approach relies on making the robot anticipate how the ground will hehave next. this enables the rohot to execute novel maneuvers, such as jumping over large gaps that require planning beforehand. in addition, the team uses control scheme called depth based impulse control dic ), which provides the robot with a flexible way to control the system when it is near an unseen obstacle. dic involves the robot acting on impulse based on small changes in the surrounding environment, which can be very useful when controlling a robot that has no visual information. said mit's 's's pulkit agrawal, we' 're trying to create a system that can be adaptable to any type of environment, even though it has visual impairment.
Bartpian
themasses institute of technology's tao chen has designed quadruped robot that can successfully jump across small gaps, but would not know if safe to continue moving forward.
the rohot uses depth- based impulse control system, which avoids the traditional control scheme of arranging the robot a straight line along the edge the obstacle, but instead jumps depending on the momentum of body. as the rohot approaches the edge, senses an obstacle getting closer, and then uses light to predict how is from the robot and whether it will be able to jump across the gap.
to control the robot, chen says," we need to provide large range of possible control schemes that the robot can use to produce dynamic behavior, such as jumping. using reinforcement learning."
GPT-3.5
leam of researchers has developed a new method for creating agile locomotion robotic quadrupeds that can navigate discontinuous terrains, such as those with gaps or obstucles. The method, called depth-bused impulse control (DIC), uses visual inputs to plan ahcad and execute agile hchaviors heyond robust walking, such as jumps. However, dynamic motion can result in significant motion onboard sensors, which poses new challenges for real-time visual processing. DIC combines model-fred learning with explicit model-based optimization ground reaction forces to regularize hehavior. The team evaluated the method in both simulation and the real world, using a gap- gap-world environment containing flat regions and randomly placed variable- width gaps. The ability to traverse discontinuous terrains with agility and terrain awareness reinforces the need for robust control, especially as the robotics community seeks to construct legged systems that can navigate novel and complex landscapes.
Figure 9: Second example of reference summary along with corresponding output summaries of systems Bart Bartpion. and GPT-3.5.

---

Oracle Plan & Press Release Summary
[PLAN] [AUTHOR] [CONCLUSIONS] [CONCLUSIONS] [AUTHOR] I [[CONCLSSONN]] [SUMMARY] Researchers at Switzerland's University of Basel and ETH Zurich have established the theoretical foundations for 100% secure communication, ensuring protection not just against quantum computers but also in instances where the communication devices' operational mechanisms and trustworthiness unknown. The protocol adds artificial noise to information about the cryptographic key. guarantecing eavesdroppers receive so little actual data as to render the protocol unhreakable. Nicolas Sangouard from the University of Basel said, "Since the first small-scal quantum computers
available, urgenty need new solutions for protecting privacy. Our work represents significant step toward the next milestone in secure communications.
Oracle Plan & Generated Summary [PLAN] [AUTHOR]] [CONCLUSIONS] [CONCLUSIONS] [AUTHOR]] I [CONCLUSIONS [SUMMARY] Researchers at the University of Basel in Switzerland say they have developed method for generating secure quantum key via photonic conversion. The method adds artificial noise to the initial measurement data, which cannot be known controlled by adversary, guarantee that the key will be obtained by both Alice and Bob, who are connected by quantum channel. By adding the noise, the rescarchers were able to increase the threshold at which they can guarantce the key is obtained. Although the rescarchers say their method is not perfect, they believe it is still effective and could lead to new generation of highly secure and provable quantum key distribution. They note that the new method provides security even when assumptions about the security of QKD are not fulfilled.
Generated Plan & Generated Summary [PLAN] [AUTHOR] [BACKGROUND [BACCGRRUUD (CONCLUSIONS] ICCOCCUSIONS) IMETH- ODS) I [RESULTS] [METHOODS] (RESULTS) [SUMMARY]
Researchers at the University of Basel in Germany have developed a method for generating key for quantum computer in which both Alice and Bob are connected by quantum channel. The method adds artificial noise to the initial measurement data to increase the probahility that the signals sent over the quantum channel are successfully received. However, in practice, only small amount of information about the key can be obtained. In a fully device-independent QKD. the rescarchers assume that any information contained in the measurement device must be known and controlled by an external party. Unlike traditional quantum key distribution, Eve does not directly control the measurement outcome, and in the absence of such information, the key is sent to an untrusted source who also distributes the information to the other user. By adding artificial noise, Eve can not ensure that the information in the raw key is real. "Our method provides way to significantly relax the threshold provide provable security." says Basel Professor Sangouard.
in such a way
User Plan & Generated Summary [PLAN] [AUTHOR] [BACGGROUN] [METHODS] [RESULTS] [CONCLUSIONS [SUMMARY] Researchers the University of Basel in Switzerland say they have developed method for generating secure quantum key via photonic conversion. The method involves adding artificial noise the initial measurement data, which cannot be known controlled by adversary, to increase the probability that the key is obtained. Although the researchers say the method is not fully inviee-indeeendent. they say provides security even when assumptions are made about the security of the quantum key distribution. "There fundamental obstacle in the development of QKD, i.e... the requirement that adversary cannot fully control the quantum channel." " says Basel Professor Sangouard Bancal. However, he says the method provides sufficient bounds the minimum required global detection efficiency ensure that the information sent over the qubit channel is good and accurate.
An example of generating summaries with custom user content plans and Bartpton. Text and corre- sponding rhetorical roles and color-coded.
Figure 10:


Reviews:
[{"review_id": "9HkbXXWIuG", "reviewer": "Reviewer_6Mgm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive review. We really appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- There is very good motivation for such a research area to exist: given the volume of published scientific work, journalists cannot manually report on all scientific findings, which means an important part of meaningful science is never reported to the general public.\n- The framing of the task has some usefulness/originality in that it uses metadata+discourse structure and includes both in the generation process instead of having plain seq2seq models that train only on text content. \n- Evaluation against baselines and current LLMs (eg GPT3.5) shows promising results. \n- The paper is well written/structured and has good readibility. ", "reasons_to_reject": "- No particular reason to reject the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about automatic science journalism, the task is about generating a news-article-like summary of a scientific paper, the goal is for the summary to look like what a professional journalist would write both in terms of linguistic style and content. The main contributions are a dataset (SciTechNews) with tuples of (scientific paper, corresponding news article, expert-written summary snippet), and a methodology to generate text guided by metadata + discourse structure where the model first generates a plan based on the scientific paper (eg introduction, background, methods) then generates a summary text that follows the plan. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mnHG2iheia", "reviewer": "Reviewer_qvrA", "summary": "", "questions": "The only thing that I find a little alarming is that in Table 6 the proposed system scores lower on factuality than GPT-3.5, which itself is not known for its faithfulness. I know that the Bart system is not the main contribution of the paper, but I think there should be at least a discussion of the apparent tradeoff between quality and factuality, especially since this is arguably a domain in which factuality is very important. Error analysis of the shortcomings in factuality would make a great addition to this paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments, we are glad you found our contributions and analysis useful.\nAs per your suggestion, we conducted an analysis of factuality errors (see below for details) and will add this analysis to the final version of the paper.\n\n## Trade-off between factuality and other summary qualities\nThere is indeed a clear trade-off between factuality and other summary properties, such as Style or Readability, as can be seen in Table 6.\nWe found that this limitation is inherent to our approach, and we leave the improvement in factuality to future work.\n\n## Analysis of Factuality Errors\nWe analyzed the types of factuality errors present in candidate summaries.\nWe uniformly sampled 30 instances from the test set and manually annotated their respective reference summary and summaries generated by $Bart_{plan}$ and GPT3.5-Turbo.\n\nWe adapt the error taxonomy employed in [0] and consider three categories at the span level: (ii) *Entity-related*, when the span is a named entity (same entity categories considered in Section 3.2.); (ii) *Noun Phrase-related*, when the span is an NP modifier; and (ii) *Other Errors*, such as repetitions and grammatical errors.\nContrary to [0], the Event-related category is not considered here since the source documents in SciTechNews do not contain events.\nEach error category is further divided into sub-categories: *Intrinsic*, *Extrinsic*, and *World Knowledge*, depending on where the supporting information is found [1].\nIntrinsic errors are caused by phrases or entities found in the source being generated in the wrong place.\nIn contrast, extrinsic errors happen when the generated span cannot be supported by the input source or any external source (e.g. Wikipedia). \nFinally, word knowledge errors are caused when the span cannot be verified with the source but it can be verified using external knowledge, e.g. author X being the director of the Computer Science department at university Y.\n\nTable R.2. presents the proportion of error categories found in the inspected summaries, along with the total number of error spans found for each system. It is worth noting that the total number of errors follows the ranking trend in Table 6, with PR summaries having the least number of errors, followed by GPT3.5, and then Bart_{plan}.\nFirst, we observe that reference summaries exhibit only Entity and NP-related errors of type World Knowledge. The majority of them include completed names of affiliated institutions (e.g. the metadata mentioning only the abbreviation “MIT” but the reference summary showing the complete name), country names where these institutions are located, or the position an author holds within an institution.\nWe also found cases where an author held more than one affiliation, with only one of these being listed in the metadata and another mentioned in the reference summary.\nSecond, we observe that Bart_{plan} extrinsically hallucinates mostly entities (e.g. country names, 34% of all its errors), followed by extrinsic NPs.\nAmong intrinsic errors, entity-related ones included mixed-ups of author first names, last names, and affiliations, whilst NP-related errors included mistaking resources mentioned in the source (e.g. a github repository) for institutions.\nIn contrast, GPT3.5-Turbo produced a high proportion of extrinsic hallucinations of noun phrases, most of them stating publication venues (e.g. *In a paper published in …*). Since the metadata added to the source does not include publication venue, the model must have surely been exposed to this kind of information during training. However, somewhat surprisingly, GPT3.5-Turbo did not exhibit world knowledge errors of any kind, potentially highlighting the conservative approach to generation followed during its training.\nErrors of type “Other” consisted mainly of orphaned phrases at the beginning of generation, i.e. the model starts the generation by attempting to continue the last sentence of the input in a `continue the story’ fashion. Consider the following example:\n\n**Input**\n```\nWrite a report of this paper in journalistic style\nPaper:\n[AUTHOR] sachit menon | duke university durham [AUTHOR] … [OBJECTIVE] the primary aim of single - image super - resolution is to construct a … and also downscale correctly . [CONCLUSIONS] the ( weighted ) pixel\nReport:\n```\n**GPT3.5-Turbo**\n```\n-wise average distance between the super-resolved and high-resolution images. However, this often results in blurring, especially in detailed regions…\n\n```\n\nHere, we observe that the input document was truncated in the middle of a sentence, and GPT3.5 attempted to complete it, hallucinating the orphaned span (*-wise average...images.*).\nWe hypothesize that the GPT3.5 model employed(model checkpointed at March, 2023) still struggled with the length of the input, reaching a point where the prompt instruction (stated at the beginning of the input) is completely ignored and the model just continues the `story’ given. \n\n\n\n---\n\n| System | Entity-Int.  | Entity-Ext. | Entity-W.K. | NP-Int. | NP-Ext. | NP-W.K. | Other | Total |\n| ------ | ------ | ---- | ---- | ---- | ---- | ----- | ----- | ----- |\n| PR Summary | 0 | 0 | 0.79 | 0 | 0 | 0.21 | 0 | 43 |\n| $Bart_{plan}$ | 0.1 | 0.34 | 0.20 | 0.07 | 0.16 | 0.02 | 0.11 | 61 |\n| GPT3.5-Turbo | 0 | 0.08 | 0 | 0.02 | 0.18 | 0 | 0.72 | 50 |\n\nTable R.2. Proportion of factuality errors in reference summaries (PR Summary), summaries generated by Bart_{plan}, and summaries generated by GPT3.5-Turbo (bottom). Error categories are organized by Entity-related, Noun Phrase-related (NP), and Other, and further subcategorized into Extrinsic (Ext.), Intrinsic (Int.), and World Knowledge (W.K.). The total is showed by system, with error proportions also normalized by system.\n\n\n## References\n[0] Goyal, Tanya, and Greg Durrett. \"Annotating and Modeling Fine-grained Factuality in Summarization.\" NAACL-HLT 2021.\n\n[1] Cao, Shuyang, and Lu Wang. \"CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization.\" EMNLP 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an excellent and thorough dataset. The analysis in section 3.2 is meticulous and very informative, and gives a very clear picture of the process which the authors seek to automate. The figures are very illuminating, especially Fig. 4. The set of comparison systems is extensive and includes recently released LLMs including open source options, making the claims in the paper very convincing.", "reasons_to_reject": "I don't really have any strong reasons to reject. My main concern is about factuality (described in questions section), but I don't think it is a sufficient reason to reject this paper.", "questions_to_authors": "The only thing that I find a little alarming is that in Table 6 the proposed system scores lower on factuality than GPT-3.5, which itself is not known for its faithfulness. I know that the Bart system is not the main contribution of the paper, but I think there should be at least a discussion of the apparent tradeoff between quality and factuality, especially since this is arguably a domain in which factuality is very important. Error analysis of the shortcomings in factuality would make a great addition to this paper.", "paper_topic_and_main_contributions": "This paper presents a substantial dataset for exploring the process of science journalism, in which scientific findings are presented in accessible terms for a non-specialist audience. The authors carefully extract papers, their abstracts and press releases and associated press release snippets. The dataset is accompanied with analyses that demonstrate the changes in style between these genres. Finally, the authors show that a controlled generation method leveraging Bart is most successful at producing high-quality journalistic summaries using a number of different metrics as well as human evaluation on multiple axes. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tUqcnfNTV7", "reviewer": "Reviewer_MWo1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and comments. We appreciate the attention to the points raised and provide a thorough discussion which will be integrated into the final version.\n\n## Contribution of extra signal in the source and target\n\nThe comparison systems introduced in Section 5.1. intend to present an incremental story of how metadata and rhetorical labels in the source improve summary qualities as well as the benefits of generating a content plan before the actual summary.\nMoreover, the usefulness of the presented two-step generation framework lies in the option to control crucial summary properties (discourse structure, style, and content) by providing custom plans. We showcased this situation in Section 6.3, and provided further examples in Appendix F.\n\nFinally, for the sake of completeness, we conducted an ablation study to quantify the impact of each extra signal in the source and the impact of generating a summary conditioned in a content plan.\nTable R.3 presents the performance of $Bart_{plan}$ when the source is ablated of metadata, rhetorical labels, and when no content plan is generated.\nIt can be seen that removing each signal has a negative impact in terms of informativeness (ROUGE scores), readability (CLI), and style (Sty).\nThis outcome highlights the benefits of using metadata and scientific rhetorical roles in the input, as well as the benefit of our two-step generation framework.\n\n\n| System | R1 | R2 | RL | CLI | Sty |\n| --- | --- | --- | --- | --- | --- |\n| $Bart_{plan}$ | 38.84 | 8.89 | 33.5 | 13.55 | 0.98 |\n| w/o Metadata | 37.16 | 8.07 | 32.48 | 14.37 | 0.96 |\n| w/o Rhet. Labels | 37.06 | 7.96 | 32.25 | 14.18 | 0.94 |\n| w/o Content Plan | 37.77 | 8.52 | 33.06 | 13.84 | 0.95 |\n\nTable R.3. Ablation study of the impact of each extra signal in the input (Metadata, Rhetorical labels) and the impact of generating the plan before the summary (Content Plan).\n\n## Using only Author and Affiliation Information as Metadata\n\nWhen enriching the source document with article metadata, we decide to only consider author’s names and affiliations for the following reason.\nConsidering the distribution of named entities found in Press Release reference summaries (analyzed in Section 3.2 and depicted in Figure 3), it is worth noting that entities of type Organization and Person are the most frequent entities –after numbers and miscellaneous–.\nHence, we decided to restrict the usage of metadata in our framework to author’s names and affiliations.\nHowever, other metadata information was collected, both from the scientific article (e.g. publication venue and year, title) and press release articles (e.g. title, PR publication date, journalistic organization), as detailed in Section 3.1.\nWe include the complete metadata in the released dataset so that future investigations can leverage them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n* Constructed a new dataset for this task which contains more domains than existing datasets. The new dataset could be beneficial for the research community. \n\n* Presented some comprehensive analysis in Section 3 to show the style difference between scientific abstracts and press summaries. The analysis could be helpful and inspire future work to leverage these characteristics.\n\n* Proposed to utilize discourse structure as extra information, which was shown to improve the model.\n\n* The paper is well-written and easy to follow. \n", "reasons_to_reject": "\n* The contribution of using meta information is quite weak. For the meta information, I originally thought it was to use a set of meta data to help summarize texts.  However in the paper it is basically using the author name and affiliation to fill in necessary information. \n\n* The motivation of the two-step framework is unclear, so more analyses about this are needed. Specifically, why is it useful to generate the plan first before generating the summary? It is not clear if the improvement of BART_plan mostly comes from the discourse role labels added in the inputs. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the task of summarizing scientific abstracts into scientific news for public audience. It presents a public dataset which contains more domains than existing datasets. The authors propose methods to utilize the discourse structure in the input abstract and show improvement. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a dataset and an automatic summarization method to generate journalistic-style summaries from scientific papers. The method takes advantage of the discursive structure and metadata of the papers to generate the summaries. The method is compared with current LLM-based tools (Alpaca and ChatGPT) showing good results. The paper makes valuable contributions and is well presented.
Decision: Accept-Main