Linguistic Compression in Single-Sentence Human-Written Summaries
Authors: Fangcong Yin
Marten Van Schijndel
Venue: EMNLP 2023 Findings (2023)
Keywords: summary writing
linguistic compression
text summarization

Abstract: Summarizing texts involves significant cognitive efforts to compress information. While advances in automatic summarization systems have drawn attention from the NLP and linguistics communities to this topic, there is a lack of computational studies of linguistic patterns in human-written summaries. This work presents a large-scale corpus study of human-written single-sentence summaries. We analyzed the linguistic compression patterns from source documents to summaries at different granularities, and we found that summaries are generally written with morphological expansion, increased lexical diversity, and similar positional arrangements of specific words compared to the source across different genres. We also studied how linguistic compressions of different factors affect reader judgments of quality through a human study, with the results showing that the use of morphological and syntactic changes by summary writers matches reader preferences while lexical diversity and word specificity preferences are not aligned between summary writers and readers.

Linguistic Compression in Single-Sentence Sluann-Writee Summaries
Fangeong Yin Department of Information Science Cornell University fy95@cornell. edu
Marten van Schijndel Department of Linguistics Cornell University
mv443@cornell.cm edu
Abstract
advances in transformer-basee neural models, au- tomatic text summarization systems have achieved a breakthrough in performance. But abstractive summaries generated by neural summarizers still suffer from hallucinated generation (Maynez et al.. 2020; Wang et al., 2022) and verbose copying from the source text (Wilber et al., 2021: Zhang et al., 2023). Studies have shown that mimicking human behaviors and preferences toward more naturalistic generations for machine- learning tasks (Jaidka al., 2013; Xiao et al., 2022) and consistent patterns human summary writing can provide helpful references to this end.
Summarizing texts involves significant cogni- tive efforts compress information. While advances in automatic summarization systems have drawn attention from the NL.P and lin- guistics communities to this topic, there is lack of computational studies of linguistic pat- terns in presents large-scale corpus study of human- written single-sentence summaries. We ana- lyzed the linguistic compression patterns from source documents to summaries different granularities, and we found that summaries are generally written with morphological expan- sion, increased lcxical diversity, and similar positional arrangements of specific words com- pared to the source across different genres. We also studied how linguistic compressions of dif- ferent factors affect reader judgments of quality through human study, with the results show- ing that the use of morphological and syntactic changes by summary writers matches reader preferences while lexical diversity and word specificity preferences are not aligned between summary writers and readers.
summaries. This work
promising approach
Previous literature has investigated cognitive op- erations behind human summary writing, especially the operation of compression. Compressing text into concise form requires the deletion of irrele- vant information, generalization of concepts, and construction of fluent expressions (Hidi and An- derson, 1986). This entails complicated operations on the language features of the produced texts at the levels of syntax, semantics, and discourse. The strategies of compression during summarization can be studied using corpus linguistics (Sherrard, 1989). However, there is lack of fine-grained analysis of the compression strategies present in summarization, and dearth of large-scale quan- titative studies that examine the general patterns of linguistic compression strategies in human sum- mary writing.
1 Introduction
Summarization requires condensing important in- formation from longer text into brief form (John- son, 1983). It involves complex cognitive capabili- ties related to logical thinking and language profi- ciency and has been commonly used in classroom settings to teach and assess these capabilities of students. Extensive education and psychology re- search on the development of students has shown that summary writing is rich resource to study the human cognitive process of comprehension. reten- tion, and organization (Brown et al., 1983).
To provide more in-depth insight into human behaviors during summary writing, this study presents a large-scale computational analysis of linguistic compression patterns in human- summaries of expository texts. To amplify any consistent linguistic compression patterns, fo- cus extreme summaries (Narayan al., 2018), i.e. single-sentence summaries that are highly com- pressed and abstractive without verbatim copying from the source text. The patterns are extracted from large summarization corpora of different sub-
Recently, human summary writing has drawn increased attention from the NL.P and psycholin- guistics communities for the purposes of compu- tational modeling of language production. Given

---

# Source-Summary Pair
Avg Source Length (sd)
Avg Summary Length (sd)
(train/val/test)
WikiHow
10607332/77952/41182
82.32 (60.10)
8.28 (5.27)
XSum
004045/11332/11533
433.05 (355.51)
23.19 (5.81)
SciTLDR-Auth 1992/618/618
1142.09 (469.71)
20.34 (8.38)
SciTLDR-Peer
1992/834/1349
1142.09 (469.71)
23.17 (7.65)
Table I: Descriptive statistics of the datasets used in this paper (in tokens).
genres at multiple granularities to obtain more fine-grained picture of human summary writing. We then conduct a human study to evaluate which linguistic compressions are most preferred by read- ers.
the usage of sentiment, named entities, phrases, syntactic structure, and punctuation in news head- lines with the popularity of the associated articles. The only cross-domain study of the linguistic in- fluences on the general task of summary writing is Arroyo-Ferrnndez et al. (2019), which com- pared the differences in the use of several language features, including part-of-speech tags, sentence sentiment, named entity tags, and relation tags from rhetoric structure theory, between human- written and machine seabhine-geeertted summaries. Arroyo- Fernández et al. found that machine-generatee summaries show significantly different behaviors in the usage of named entities from humans. How- ever, these existing works did not discuss the fine- grained distributional differences between source document and summary pairs evaluate how the identified linguistic patterns are related to the per- ceived summary quality by summary readers. contrast, we propose a more ane-grained analy- sis framework of the linguistic patterns present summarization, analyzing the pairwise relations between source documents and the associated sum- maries at different granularities in addition to lin- guistic features of the summaries in isolation.
2 Related Work
The cognitive operations of compression during summary writing were formulated as "macrorules" by Brown and Day (1983). Brown and Day pro- posed four common operations in summary writ- ing macrorules, which experienced high school and college students self-reported to intentionally and frequently when writing summaries: dele- tion, selection, substitution, invention, and combi- nation. Empirical research has shown that linguis- tic features can reflect the usage of some of the macrorules (Sherrard, 1989). Denhiere (2005) pro- posed a computational model using sentence rep- resentations and latent semantic analysis to model the use of macrorules by participants in an experi- mental study.
Recently, studics in journalism and education have investigated linguistic features of summaries for specific domains by viewing news headlines (Pi- otrkowicz et al., 2017; Xu and Guo, 201 8), research article abstracts (Amnuai et al., 2020), and paper reviews (Leijen and Leontjeva, 2012) as summaries. These works have identified patterns for each do- main and shown that the patterns can help writ- ers improve the quality and effectiveness of their summary writing. For example, Leijen and Leon- tjeva (2012) examined the correlations between the linguistic features of peer reviews of academic papers and the acceptability of the revision sug- gestions provided in the reviews using random forest model, and found that the paper authors more likely accept and implement the reviewer comments if the comments have more directives. Piotrkowicz et al. (2017) studied news headlines as a form of extreme summaries and correlated
3 Linguistic Compression Patterns
3.1 Data
We focused on extreme summarization of exposi- tory texts, the task of creating a single-sentence ab- stractive summary of a single document (Narayan et al., 2018). Extreme summarization a chal- lenging yet commonly studied task by the auto- matic text summarization community. Given its tight length constraint, extreme summarization re- quires more abstractive and compressed summaries compared with other forms of summarization, help- ing ensure that non-trivial efforts of compression will occur in summarization. In addition, extreme summary datasets tend to have fewer supplemental materials that are non-essential the source docu-

---

(Bommasani and Cardie, 2020), providing purer window on the souree compression.
Syntactic features2
ceastitueney-parseee tree height is the con-
We selected three popular datasets for extreme au- tomatic summarization: WikiHow (Koupaee and Wang, 2018), XSum (Narayan et al., 2018), and Sc- iTLDR (Cachola et al., 2020). WikiHow uses the first introductory sentence of each paragraph of the community-gennerated instructional how-to articles from WikiHow.com summary of the paragraph. XSum collects the introductory sentence of news articles from BBC written by professional journal- ists as a single-sentence summary. SciTLDR uses one-line summaries for academic arXiv papers writ- ten by the authors (SeiTLDR-Auth) and written by peer reviewers (SeiTLDR-Pere Peer). Writing sum- maries for the same source text but for different purposes likely involves different production strate- gies (c.g.. Hidi and Anderson, 1986, found that cer- tain planning strategies were used for reader-based summaries, but not for writer-based summaries), so we analyzed the two subsets of SciTLDR as two separate datasets in our study. Statistics of the datasets are summarized in Table 1.
stituency tree height of each sentence.
Unique part-of-speech tag is the sentence- level average of the number of unique part-of- speech (POS) tags.
Number of modifiers is the average number of modifiers modifying each token in a sen- tence. It has been shown to correlate with the cognitive effort involved in creating written text (Ravid and Berman, 2010).
Dependency length is the average depen- dency length in each sentence.
Semantic features
Word specificity represents the semantic dis- tinguishability of noun and verb phrases that represent unique entities, nouns, referents, or events a given context (Frawley, 1992). The specificity of each noun and verb is mea- sured by the taxonomy depth of the contex- tualized word sense (obtained by in-context word sense disambiguation) in the WordNet hierarchy (Miller, 1994).3
3.2 Sentence-level Analysis
In order to compress all the important information from the source document into a single-sentence summary, the extreme summary might show dif- ferent morphological, syntactic, and/or semantic pattern from the individual source sentences. We measured such compression in extreme summa- rization by the change the sentence-level values each linguistic feature from source to summary. The significance of change indicates which features are consistently manipulated by writers during sum- marization. We modeled the change of each lin- guistic feature as single classification feature in binary logistic regression model trained classify sentences either coming from a source document from summary. We also investigated the di- rection and magnitude change by comparing the average paired difference of each feature between the source sentences and their associated summary.
sentence
Type-token ratio represents the lexical di- versity of a sentence. It is measured by the number of unique word types divided by the total number of tokens in each sentence.
Morphological features
Based on previous works in education and chil- dren's psychology research (Green al., 2003; Deacon al., 2017), the capability to control mor- phological structures writing correlates well with the reading comprehension capability for children. Therefore, we also investigated morphological fea- tures to quantify the change in sizes of words after summarization in terms of the numbers of seman- tic word-internal units. Two morphological fea- tures are chosen to potentially capture subtle word choices in writing beyond changes in the distribu- tion of different types of words (e.g. the distribu- tion of stop words):
3.2.1 Sentence-level features
For each source-summary pair in the datasets, we obtained automatic annotations' for both the source document and the summary with features as de- scribed below.
2 All syntactic features are computed with the Python pack- age spaCy.
'The part-of-speech tag and the word frequency affects the depth of the synset tree of a word in the WordNet hierarchy. Therefore, we z-scored word specificity of cach word by the mean specificity and standard deviation of its part-of-speed- tag (rare words were excluded in the calucation of mean).
Feature data and annotation scripts can he found here: wtwp://wwww..ooooooooooiiinnn//i

---

Feature
Regression Coefficient (Prediction Accuracy)
WikiHow
XSum
SciTLDR-Auth
SciTLDR-Peer
# Morphemes
-3.842 (0.637)
-2.877 (0.595)
-5,290 (0.705)
-7.404 (0.690)
Word Length
-0.192 (0.531)
-0.274 (0.512)
-2.452 (0.768)
-2,218 (0.724)
# Unique POS
0.722 (0.858)
-0.170 (0.573)
0.333 (0.695)
0.401
Tree Height
0.678 (0.810)
-0.152 (0.593)
0,194 (0.632)
-0.128 (0.573)
Avg # Modifiers
0.034 (0.498)
-0.299 (0.590)
0.004 (0.520)
0.046 (0.493)
Avg Dependency Length
2.369 (0.746)
-0.174 (0.518)
0.822 (0.604)
0.756 (0.610)
Type-Tobee Token Ratio
22.308 (0.772)
-3.938 (0.547)
-4.945 (0.661)
-0.787 (0.525)
Word Specificity
-0.155 (0.554)
-0.500 (0.587)
-0.107 (0.501)
-1.448 (0.597)
Table Summary of separate logistic regression models using each linguistic feature isolation to predict whether text is source document (label=I) or summary (label=0). Each model was tested on held-out validation data to obtain prediction accuracy. Prediction accuracies lower than the random baseline (50%) are greyed out.
Number of morphemes' is the average num- ber of morphemes in words in each sentence.
positive feature difference indicates larger value in the source sentences relative the summary sentence.
Word length is the average number of letters in words in each sentence.
In Figure 1, the average number of morphemes and word lengths are larger for summaries across datasets of different genres, indicating that sum- maries tend to be morphologically expanded. Simi- larly, summarics tend to use more specific words and be lexically more diverse than source sentences. Between the two feature groups, the morphological features more clearly distinguish between summary and source than semantic features as the prediction accuracies are higher in Table 2.
Given that source documents have multiple sen- tences, we computed grand averages of the sentence-level feature averages over the source doc- uments.
3.2.2 Sentence-level Results
Table 2 summarizes logistic regressions using each linguistic feature separately to predict whether sentence is summary from the source. 5 A positive coefficient indicates greater compression of linguistic feature at the sentence level (larger values the source sentences compared with the summary) while a negative regression coefficient indicates expansion. Higher prediction accuracy indicates higher distinguishability of the feature between source sentences and summary and thus greater consistent manipulation by writers during summarization.
Syntactic features, especially constituency- parsed tree height and the number unique POS tags, are strong indicators of whether a sentence is summary or not as suggested by their high pre- diction accuracies. However, the regression coeffi- cients display genre-dependent variability. XSum summaries tend to be syntactically expanded than the source sentences while WikiHow and SciTLDR summaries are syntactically compressed.
Between SCiTLDR-Auth and SciTLDR- the change patterns linguistic features are similar except for syntactic tree height. The similarity implics that when writers create summaries for the same genre of scientific academic paper but for different audiences, they apply similar linguistic compression and expansion strategies.
In Figure 1, visualize the change of each feature as the average paired difference berween each feature in the source sentences and in the associated summary. We first normalized feature values by min-max normalization to account for the scales of different features (unnormalized raw numerical values can be found in Appendix B). A
It is noteworthy that the average number of mod- ifiers is neither an above-random-basslin indicator nor significantly changed from the source sum- mary, implying that the number of modifiers is not an important factor for extreme summarization.
"This was computed by the pre-trained morphology anno- tator morfessor (Smit et al., 2014).
WWe also confirmed the results of our individual regressions
using a regression involving joint factors for syntax, semantics, and morphology (sce Appendix A).

---

0.00
0.125
Wikihow
XSum
Normazlied Feature Values
0.02
0.100
autth
0.1
Mean Paired Difference
SciTLDR-peer
0.2
-0.04
-0.06
0.050
[
0.4
0.025
-0.08
0.5
0.000
II
00.255
0.7
-0.12
0.050
Norphemes
Word Length
Unique free Height Modiliers
Avg Dependency Length
Type-Token Ratio
Word Specificity
Morphological Feature
Semantic Fealure
Syntactic Feature
Figure I: Paired differences of normalized linguistic features between source and summary sentences (source minus summary), A positive value means the feature was compressed during summarization (greater in the source), while a negative value indicates expansion (greater in the summary). Whiskers show 95% level confidence intervals.
This is contrary to findings of previous works that the number of modifiers used in written text is good indicator of the writer's capability of process- ing complex information and handling language (Ravid and Berman, 2010; Durrant and Brenchley, 2022), but might grounded the differences between extreme summary writing and other types of writing (e.g., Ravid and Berman, 2010, exam- ine paragraph-long compositions that describe an event).
in the aligned set. 6 We averaged the DBA outcome over 3 separate runs. As bascline to represent the positional patterns present in random English sentences, incorporated the average contours of the test split of the English portions of the English- to-German parallel multi-genre corpus OPUS-100 (Zhang et al., 2020).
3.3.1 Word-level features
We analyzed the word-level behavior two of the features that have shown strong influences in our previous sentence-level analysis.
3.3 Word-level Analysis
Word specificity uses the same definition in the sentence-level analysis, without averag- ing over each sentence.
We also examined the feature contours within each sentence investigate whether there are consis- tent positional patterns in how word-level linguis- tic features are distributed within each sentence. We generated separate average feature contours for source and summary sentences and qualitatively analyzed the contour patterns. We applied dynamic time warping (DTW) with barycenter averaging technique (DBA; Petitjean al., 2011) to align the sentence features and obtain the average contours. DBA approximates the average sequence of group of sequences by recursively updating the average aligned sequences to minimize the sum of DTW distances from the average. has been shown to be more accurate and informative representation of the average pattern of a group of time series than geometric averaging (Forestier et al., 2017). We first z-scored the features in each sentence and then interpolated each sentence standardization. We then applied DTW to align the sequences each feature and used barycenter aver- aging to find the average sequence of each feature
Dependency length reflects the psycholin- guistic cost of resolving dependency are (Gibson, 2000) and has been shown to cor- relate well with human syntactic processing (Demberg and Keller, 2008; Grodner and Gib- son, 2005; Warren and Gibson, 2002; Lewis et al., 2006). The word-level incremental de- pendency length is computed by summing the length of terminating dependency along with the number of ongoing unresolved dependen- cies in left-to-right parse.?
"We initialized the averaging process using randomly selected sentence from the
the suggested by Petitjean (201 make the initialization not skewed distribution of sentence length, we randomly selected one sentence of each possible sentence length (before interpola- tion) as an initial average, updated this average with sentences same length for iterations using DBA. and finally applied DBA to all the resulting averages of different initial- izations to obtain final grand average.
length of 100 for
"In the original definition of Gibson (2000), dependency

---

Dataven
25um SOITLDS-AUth SUTLLR OPUS
ScITLDR
Standardd aed Dependenc Langth
0%
Netatiwp Wostiin Sonttene
(a) Word specificity of summary sentences
-2
Viki--oow
XSum
Relative Position Sertence
Suurce Summary
Sounue Summary
(a) Dependency length of summary sentences
Wikihow
Xtum
Source
Source
Summarr
Summmar ry
Standardized Specifilty
Standardized Dependency Length
07
25%
50%
75%
100%
096
2556
30%
75%e
.00%
0%
50%
100%
0%
50%
100%
Summary
ScITLDR-Auth
ScITLDR-Peed Peer
Source
Sourde Summarr
Summary
25%
50%
75%
100%
0%
20:0
30%
75%
200%
Relative Position
(96)
0%
50%
100%
0%%
500%
105%
(b) Comparisons of word specificity contours between summary and source sentences
Relative Position in Sentence (963
(b) Comparisons of dependency length contours between summary and source sentences
Figure Average contours of word specificity.
Figure 3: Average contours of dependency length.
3.3.2 Word-level Results
Figure 2(a) shows that summary sentences gener- ally start with a relatively high word specificity followed by drastic dip before in the first 10% of the sentence and reach the peak of word specificity near the 40% location of the sentence. Summaries contours have several different (at some locations, even opposite) high and low locations from the OPUS contour. By comparing the contours be- tween the summary and source sentences in Fig- ure 2(b), we can see that summary contours are mostly the same as those of the source sentences. This indicates that summary sentences might be written to replicate the positional arrangements of specific words in sentences from the source. This is likely consequence of the fact that specific words exhibit consistent semantic roles through-
out the source document, which is repeated in the summaries.
Figure 3(a) shows that summaries generally have a small spike of dependency length near the begin- ning of the sentence and a high peak near the end of the sentence (the WikiHow contour is an exception as it does not have small spike in the heginning; this is because WikiHow summaries short imperative sentences). Similar patterns can be found in the dependency length contours of OPUS in 3(a) and of the source sentences in 3(b). The similarity indicates that the construction of depen- dencies in summary and source sentences is similar to that of random sentences in that long dependen- cies are resolved near the end of sentences.
mostly
Reader Preference Evaluation
length measured using intervening nouns and verbs. In our case, we still measured the dependency lengths of nouns and verbs, hut we used intervening tokens rather than intervening nouns and verbs only.
Common summarization strategies among writers do not necessarily entail quality: good strategies

---

of summary writing should help readers better un- derstand the gist of the source, and readers' per- ceptions of summary texts might different from the writers' intentions. Therefore, we evaluated the extent to which readers preferred each of the com- pression patterns described in the previous section.
60s per pair). We collected additional human data to replace the filtered items.
We obtained automatic annotations for linguistic features of the summary only and the paired differ- ences of linguistic features between the source and summary, Features in the summary alone can only be correlated with overall quality, fluency, and ab- straction level because these quality aspects could be judged by reading the summary sentence alone while others have to be determined with reference to the source text.
4.1 Study Setup
Because XSum source documents are news arti- cles that require little background knowledge to comprehend and XSum summaries contain richer information from the source given their length, we randomly sampled 20 source articles that have less than 400 words from XSum along with their sum- maries for our human evaluation.
4.2 Results
Intercorrelation of quality aspects To explore which quality aspects readers emphasize, we fit a linear regression model to predict the overall qual- ity rating using the other quality aspects. We re- moved any insignificant (p>0.05) or strongly mul- ticollincar (variance inflation factor > 10) quality aspect from the regression models. Results show that readers prefer summaries with high novelty (t=2.39.=00.019) relevance (=6.600..<00001) and factual coverage (t=3.178.ppe 178.p<0.01). They reveal a reader preference for summaries with a clear focus, comprehensive coverage of facts in the source, and new expressions of existing information. These cor- relations align with the findings of Sanchan (2018 8) for reader preferences of summaries in the online debate domain: participants prefer summaries that are well-arranged with concise list of logical points.
Using the crowdsourcing platform ProlificAca- demic, we recruited native English speakers for each source-summary pair to read the source and summary and rate 7 aspects of summary quality (defined below) 1-5 scale. Exact protocols can be found in Appendix C.
Overall Quality is the overall quality of the summary.
Fluency is the grammaticality and readability of the summary.
Relevance indicates a clear focus and no redun- dant information in the summary.
Consistency indicates factuality and faithful- ness to the information that appears in the source document.
Factual Coverage is the coverage of important facts from the source.
Linguistic features and summary quality. Val- of linguistic features and the rating scores have different scales, and ratings by different raters dis- play lot of variance. Therefore, we utilized repre- sentational similarity analysis (RSA; Kriegestkorte and Bandettini, 2008), which studies second-order rather than first-order relations, to obtain correla- tions between linguistic features and quality as- pects. As common method used in cognitive science, RSA reveals correlations between mea- sures that operate over different scales. By first correlating each measure with itself across the in- put stimuli, the scales of the first pass are internally comparable while the outputs are all similarities and therefore are comparable with each other in a subsequent second pass.
Novelty is the use of new words and expressions that have not appeared in the source to paraphrase existing information.
Abstraction level is the use of high-level reca- pitulations of important concepts and ideas.
In each run, raters were presented with source- summary pairs of two different sources in ran- domized orders. Each rater was paid the equiv- alent of $12 per hour. Low-quality ratings were filtered based on the correct responses to manually- constructed comprehension questions (2 per source document) and task completion speed (removed if
"The definitions of fluency, relevance, and consistency are adapted from Fabbri et al. (2021).
"Novelty and abstraction level are used to evaluate the ab- stractiveness of summaries, the degree to which the summary is not copied from the source text yet retains the semantics of the source text (Katwe et al., 2022). Abstractiveness is an important characteristic of extreme summaries and abstractive summaries in general.
We constructed representational similarity ma- trices (RSMs) for each quality aspect by comput- ing the pairwise cosine similarity score of cach stimulus' quality aspect rating with the aspect rat-

---

Avg iependency Length Word Length
Wordlevel Word Specificity Avg Word Specificity
these linguistic features important when dealing with extreme summaries. Figure 4(b) indicates that changes in type-token ratio from the source sen- tences to the summary are correlated with most quality aspects for readers, but this feature did not receive much attention from summary writers as the small average paired differences in Figure I indicated. Conversely,
Aug é Modiners
Type- Taken Ratio
Morpheme
Unique POS
Wee Height
0.3
0
Overall Quality 0.32 0.10
3.19
0.15
0.2
Fluensy
0.15
0.1
Abstraction Level
see that changes in word specificity hardly associated with sum- mary quality by readers while writers emphasize (as the large average paired differences Fig- ure indicated). Finally, writers should take note that syntactic features, especially average depen- dency length and the number of unique POS tags, are correlated with the readers' perceptions of the factual (consistency and factual coverage) and in- novative qualities (novelty and abstraction level) of summaries.
(a) Correlations of features and reader preferences (Summary Only)
Avg Dependence Length Word Length
Avg Word Specificlty
# Morphemes
Aug Modifiers
# Unique POS
Type-Tkknn Töken Ratio
free Height
Overall Quality
0.18
0.16
Fluency
0.11
0.14
aopooo a Seaamma
Relevance
0.15
Consistency
0.17
0.1
Factual Coverage 0.16
0.23
0.16
Discussion
0.0
Novelty-
This work provides a large-scale computational corpus study towards understanding how humans generally write single-sentence summaries. We performed an analysis of changes in morphologi- cal, syntactic, and semantic features and used dy- namic time warping with barycenter averaging analyze the fine-grained positional patterns of lin- guistic features. Our analyses found that syntactic and morphological changes are helpful when distin- guishing whether a sentence is summary or from source document. Single-sentence summaries are generally written with morphological expansion, in- creased specificity of words, and increased lexical diversity compared to the source sentences. At the word level, summaries are written to replicate the positional arrangements of their source sentences while the temporal arrangements of long dependen- cies in summary sentences are similar to those of random sentences.
Abstraction Level
0.16
0.15
0.14
(b) Correlations of changes and reader preferences Chource-Summat Paired Differences)
Figure 4: Significant (p 0.05) RSA correlations be- tween quality ratings and linguistic features. (a) shows that more morphemes and unique POS tags a sum- mary are correlated with overall quality. (b) shows that changes in type-token ratio and changes in syntactic features are correlated with most quality aspects. Non- significant RSA correlations are represented by" "-".
ings of all other stimuli. Similarly, we constructed RSMs for each sentence-level linguistic feature by computing the pairwise Euclidean distance (con- verted similarity score) of that feature across all the stimuli. For werd-level linguistic features, DTW distance average is used instead of the Eu- clidean distance and is converted to the similarity metric. We computed the final second order RSA correlations as the Spearman rank-order correlation between RSMs.
We also evaluated the impact of linguistic feature compression on human readers. Using representa- tional similarity analysis over correlations between linguistic features and quality ratings, we showed that morphological features and lexical diversity (type-token ratio) are correlated well with qual- ity ratings. Readers' perceptions of consistency, factual coverage, novelty, and abstraction level of summaries are correlated with changes in syntac- tic features such as the average dependency length. The results suggest that summary writers and read- ers both emphasize certain syntactic features, but
Figure 4(a) summarizes RSA correlations for the linguistic features of the summary alone while Fig- ure 4(b) examines the paired differences of features between the source sentences and summary. Fig- ure 4(a) shows that morphological features and the number of unique POS tags of the summary sen- tence are strongly correlated with the overall qual- ity. Therefore, both writers and readers consider

---

type-token ratio and word specificity exhibit mis- match between writer usage and reader preference. This implies that semantic compression and ex- pansion need to be carefully applied by summary writers to effectively express the main information to readers.
and filtering out low-quality data points, but we found it difficult obtain data with high agreement among raters. The variability might be caused by the nature of evaluating the summary quality as the interrater agreements in human evaluations of text summarization are generally low (Iskender et al., 2021).
It should be noted that we only indirectly tested the influence of linguistic feature compression on readers through our human study since we con- strained our survey to summaries obtained from corpora. In the future, we plan to incorporate direct testing by asking readers to rate summary stimuli that are specifically manipulated with particular compression patterns. Future works might also in- clude investigations of the incremental processing of text production under experimental setups with the help of techniques such as eye tracking and cloze tasks. As motivated by the emerging focus text summarization in the NLP community, our work can serve as a syycholingistticall-iffooom resource for improving automatic summarization systems (e.g.. explicitly imitating human behaviors during summary writing improves coherence and organization of text summarization; Jaidka al., 2013; Xiao et al., 2022) and probing model under- standing of text generation.
Ethics Statement
The human evaluation was conducted under IRB approval. Experimental stimuli were carefully se- lected from the random samples to not include top- ics and descriptions of violence and sensitivity, and raters were presented with the option to leave any time during the study.
Factuality and faithfulness are one of the most important qualities of summaries, and applying strategies to match readers' preferences should not be in trade-off at any time with inaccurate, hallu- cinated, or distorted summaries of facts. While in- depth studies of the factuality of summaries might need further efforts of fact-checking and de-biasing, we have taken into account the study by dis- cussing the quality aspects of consistency, factual coverage, and relevance in detail.
A Acknowledgements
Limitations
This work is supported by Cornell Engincering Learning Initiatives Undergraduate Research Grant and Wood Excellence Engineering Edu Award. We thank John R. Starr, Maria Antoniak, Jacob Matthews, Debasmita Bhattacharya, and members of the Corell NLP group and Cornell Computa- tional Psycholinguisties Discussion (C.Psyd) group for their valuable advice. We also appreciate the precious feedback from participants of the Sec- ond Workshop on Processing and Evaluating Event Representations (PEER2023) at the University of Rochester and from the reviewers.
Our corpus analyses only focus on expository texts. Studies have shown that linguistic compressions applied by grade students are substantially different for expository, urgumentative, and descriptive texts (Jagaiah and Kearns, 2020). While single-sentence summaries are less common for these genres and thus very limited data exists for large-scale analy- sis, the observations drawn from our study might be limited to the expository genre. They might also be limited to single-sentence summarics because discourse factors, which are used in general un- constrained summaries, are not considered in our study. However, given that extreme summarization is increasingly common in online settings with lots of real-world applications, we view this work as an important contribution to the understanding of how humans summarize texts.
References
Wirada Amnuai, Pansupa Kotuta, and Maythinee Duangprasertchai. 2020. Textual and linguistic char- acteristics of research article abstracts. LLT Jour- nal: Journal on Language and Language Teaching, (11)):668181.1
Regarding human evaluation, the rating data show large variability that might result in insignifi- cant first-order correlations between linguistic fea- tures and quality ratings. During multiple pilot runs, we attempted to reduce the variability by in- creasing the sample size and the number of raters
Ignacio Arraya-Fernnndez, Arturo Curiel, and Carlos- Francisco Méndez-Cruz. 2019. Language features in extractive summarization: Humans vs. machines. Knowledge-based Systems, 180(

---

Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075-8096, Online. Association for Computational Linguistics.
Daniel Grodner and Edward Gibson. 2005, Conse- quences of the serial nature of linguistic input for sentenial complexity. Cognitive Science, 29(2):261- 290.
Suzanne Hidi and Valerie Anderson. 1986. Producing written summaries: Task demands, cognitive oper- ations, and implications for instruction. Review of Educational Research, 56(4))473-93.
Ann I.. Brown and Jeanne D. Day. 1983. Macrorules for summarizing texts: the development of exper- tise. Journal of Verbal Learning and Verbal Behav- ior, 12.1):1114
Neslihan Iskender, Tim Polzehl, and Sehastian Möller. 2021. Reliability of human evaluation for text sum- marization: Lessons learned and challenges ahead. In Proceedings of the Workshop on Human Evalua- tion of NLP Systems (HumEval), pages 86-96 Online. Association for Computational Linguistics.
Ann L. Brown, Jeanne D. Day, and Roberta S. Jones. 1983. The development of plans for summarizing texts. Child Development, 54(4):968-979
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld, 2020. TLDR: Extreme summarization of sci- entific documents. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766-4777, Online. Association for Computational Linguistics.
Naitaii G. Jagaiah, Thilagha and Devin M. Kearns. 2020. Syntactic complexity measures: varia- tion by genre, grade-level. students' writing abilities, and writing quality. Reading and Writing, 33:2577- 2638.
Hélène Deacon, Shelley Xiuli Tong, and Kathryn Fran- cis. 2017. The relationship of morphological analysis and morphological decoding reading comprehen- sion. Journal of Research in Reading, 40(1):
Kokil Jaidka, Christopher Khoo, and Jin-Cheon Na. 2013. Deconstructing human literature reviews - a framework for multi-document summarization. In Proceedings of the 14th European Workshop on Nat- urul Language Generation, pages 125-135, Sofia, Bulgaria. Association for Computational Linguistics.
Vera Demberg and Frank Keller. 2008. Data from eye- tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193-2100
Nelson Johnson. 1983. What do you do if you can't I tell the whole story? the development of summarization skills. Children's 's language, 4:315-383.
Pessus-Philippe Lamaire Benoit Mandin Sonia Den- hiere, Guy. 2005. Computational cognitive models of summarization assessment skills. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 27. Cognitive Science Socity.
Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala, Rohit Vardhan, and Shravan Patil. 2022. A comparative study of abstractive- ness in summarization evaluation. In 2022 OITS International Conference on Information Technology (OCIT), pages 61-66.
Philip Durrant and Mark Brenchley. 2022. Develop- ment of Noun Phrase Complexity Across Genres in Children's Writing. Applied Linguistics, 44(2):239- 264.
Mahnaz Koupace and William Yang Wang. 2018 Wiki- how: A large scale text summarization dataset. arXiv, Abs/1810.09305.
Alexander R. Fabbri, Wojciech Kryšciński, Bryan Mc- Cann, Caiming Xiong. Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-400 409..
Marieke Kriegestkorte, Nikolaus and Peter Ban- dettini. 2008 Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2(4).
Germain Forestier, François Petitjean, Hoang Anh Dau, Geoffrey Webb, and Eamonn Keogh. 2017. Gener- ating synthetic time series to augment sparse datasets. In Data Mining (ICDM), 2017 IEEE International Conference on, pages 865-870. IEEE
Djuddah J. Leijen and Anna Leontjeva. 2012. Lin-
guistic and review features of peer feedback and their effect on implementation of changes in academic writing: A corpus based investigation. Journal of Writing Research, 4(2):177-202.
William Frawley. 1992. Linguistic Semantics. L. Erl- baum Associates.
Richard Lewis. Shravan Vasishth, and Julie Van Dyke. 2006. Computational principles of working mem- ory in sentence comprehension. Trends in cognitive sciences, 10:447-54.
Edward Gibson. 2000. The dependency locality theory: A distance-based theory of linguistic complexity. Im- age, language, brain: Papers from the first mind articulation project symposium, pages 94-126.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Laura B. Green, Deborah Mecutchen, Catherine Schwiebert, and Tom Quinlan. 2003. Morpholog- ical development in children's writing. Journal of Educational Psychology, 95(4)

---

George A. Miller. 1994. WordNet: A lexical database for English. In Human Language Technology: Pro- ceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Liqiang Xiao, Hao He, and Yaohui Jin. 2022. Fu- sionsum: Abstractive summarization with sen- tence fusion and cooperative reinforcement learning. Knowledge- Systems, 243(C).
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguisties.
Zhan Xu and Hao Guo. 2018. Using text mining com- pare online pro- and anti-vaccine headlines: Word usage, sentiments, and online popularity. Communi- cation Studies, 69(1):103 122.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639, Online. Association for Computational Linguis- tics.
François Petitjean, Alain Ketterlin. and Pierre Gançarski. 2011. A global averaging method for dynamic time warping, with applications to clustering. Pattern Recognition, 44(3):678-69) 693.
Alieja Piotrkowicz, Vania Dimitrova, Jahna Otterbacher, and Katja Markert. 2017. The impact of news values and linguistic style on the popularity of headlines on twitter and facebook. In Proceedings of the Interna- tional AAAI Conference on Web and Social Media, volume I I, pages 767-774. Association for Compu- tational Linguistics.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Perey Liang. Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv, abs/2301.1384
A Multi-factor Logistic Regression
To confirm the results from our individual regres- sions, we selected one significant feature from each of the feature categories (morphological, syntactic, and semantic) in combination to fit logistic regres- sion models for the same classification task as our sentence-level analysis. The results in Table 3 confirm our prior observations on the relative ranking of distinguishability among features and the genre- dependee variability for the syntactic feature.
Dorit Ravid and Ruth A. Berman. 2010. Develop- ing noun phrase complexity at school age: A text- embedded cross- loosu-iinguistic analysis. First Language, 30(1):3-26.
Nattapong Sanchan. 2018. Comain-Focused Summa- rization of Polarized Debates. Ph.D. thesis, De- partment of Computer Science, the University of Sheffield, the United Kingdom.
Carol Sherrard. 1989. Teaching students to summarize: Applying textlinguistics. System, 17(1):1-11.
B Difference of Unnormalized Linguistic Features
Peter Smit, Sami Virpioja, Stig-Aree Grönroos, and Mikko Kurimo. 2014, Morfessor 2.0: Toolkit for sta- tistical morphological segmentation. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages 21-24, Gothenburg, Sweden. Association for Computational Linguistics.
Table 4 includes the difference in unnormalized linguistic features between the source sentences and summary. All the means of paired differences statistically significant from 0 (p<0.05) except for the ones greyed out. Statistical significance is tested with paired t-test the distribution approx- imately normal, and with the Wilcoxon signed-ran test otherwise.
Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, and Haizhou Li. 2022. Analyzing and evaluating faithful- ness in dialogue summarization. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4897-4908 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Tessa Warren and Edward Gibson. 2002. The influ- ence of referential processing on sentence complex- ity. Cognition, 85(1):79-112
Matt Wilber, William Timkey, and Marten van Schijndel. 2021. To point or not to point: Understanding how abstractive summarizers paraphrase text. In Find- ings of the Association for Computational Linguis- fies: ACL-IJCNLP 2021, pages 3362-3376, Online. Association for Computational Linguistics.

---

Dataset
Regression Coefficient Morphemes
Accuracy
Tree Height
Type-Token Ratio
WikiHow -5,463
34.862
-20.857
0.916
XSum
-6.247
-7.190
-2.550
0.657
Sctt
-5.579
1.472
-3.714
0.779
SeiTLDR-Peer -8.344
-3.213
-5.766
0.795
Table Summary of logistic regression models using three significant linguistic features in combination predict whether a text is source document or a summary.
Feature # Morphemes
Dataset
Mean
Median
Q1
Q3
WikiHow
-0.082
-0.055
-0.178
0.059
XSum
-0.040
-0.025
-0.126
0.063
SciTLDR-Auth
-0.111
-0.069
-0.188
0.016
SciTLDR-Pess
-0.814
-0.729
-1.322
-0.202
Word Length
WikiHow
-0.101
-0.054
+05933
0.434
XSum
-0.055
-0.018
-0,499
0.338
SeiTLDR-Auth
-0.087
-0.070
-0.18
0.013
SciTLDR-Pes
-0.587
-0.557
-1.008
-0.097
# Unique POS
WikiHow
4.867
5.167
3.000
7.000
XSum
-0.880
-0.800
-3.000
1.000
SciTLDR-Aut
2.019
2.000
-0.099
4.467
SciTLDR-
1.747
1.856
0.000
3.744
Tree Height
WikiHow
3.563
3.750
2.000
5,500
XSum
-1.007
-0.800
-3,200
1,250
SeiTLDR-Auth
1.123
1.232
-0,822
3,462
SciTLDR-
-0.772
-0.636
-2.625
1.529
Avg # Modifiers
WikiHow
0.034
0.000
0.000
0.000
XSum
-0.268
-0.050
-0.875
0.125
SciTLDR-AAth
0.003
0.000
-0.444
0.500
SciTLDR-Ped
0.040
0.000
-05000
0.564
Avg Dependency Length
WikiHow
0.497
0.532
0.126
0.902
XSum
-0.026
0.000
-0.61
0.329
SciTLDR-Auth
0.178
0.236
-0.171
0.622
SciTLDR-Peer
0.161
0.257
-0.198
0.626
Type-Token Ratio
WikiHow
-0.052
-0.055
-0.087
-0.024
XSum
-0.008
-0.012
-0.048
0.028
SciTLDR-Auid
-0.030
-0.044
-0.085
0.011
SciTLDR-Peet
-0.003
-0.009
-0.062
0.043
Word Specificity
WikiHow
-0.058
-0.039
-0.456
0.366
XSum
-0.077
-0.065
-0.334
0.209
ScitLDR-Auth
-0.080
-0.085
-0.309
0.168
SciTLDR-Peer -0.126
-0.103
-0.324
0.106
Table 4: Statistics of the paired difference of the unnormalized values of linguistic features between source sentences and summary. Means that are not significantly different from (p>0.05) are greyed out.

---

C Survey Questions of the Human Evaluation
Very Good
Consistency How is the consistency of the sum- mary? A consistent summary should extract facts from the text and should NOT have information that contradicts the text does not appear in the text.
Overall Quality What is the overall quality of the summary?
Very Poor
Very Poor
Poor
Poor
Barely Acceptable
Barely Acceptable
Good
Good
Very Good
Factual Coverage How is the factual coverage of the summary? A summary with high factual coverage contains all most of the important in- formation and facts in the source text. summary with low factual coverage misses some important facts in the source text.
Very Good
Fluency How is the fluency of the summary? A fluent summary should be easy read. should NOT have capitalization errors or obviously un- grammatical sentences (c.g.. fragments, missing components) that make the text difficult to read.
Very Low
Low
Very Poor
Medium
Poor
High
Very High
Barely Acceptable
Novelty How is the novelty of the summary? A novel summary should use new words and phrases that do not appear the source text to paraphrase the same information. It should NOT be a verba- tim copy of the exact words and phrases from the source text.
Good
Very Good
Relevance How is the relevance of the summary? A relevant summary should have clear focus and sentences should only contain important informa- tion from the source text. should NOT have irrelevant content or redundant repetition of infor- mation from the text.
Very Poor
Poor
Barely Acceptable
Good
Very Poor
Very Good
bstraccioo Level What is the abstraction level of the summary? A highly abstract summary should be a high-level recapitulation of important concepts and ideas in the text. It should NOT be verbatim copy of technical details from the source text.
Poor
Barely Acceptable
Good
Very Low

---

Low
Medium
High
Very High


Reviews:
[{"review_id": "oE6UiJiUTn", "reviewer": "Reviewer_XKkb", "summary": "", "questions": "- You state that extreme summarization would amplify the linguistic compression patters. Could you please elaborate on this?\n\n- Is there a reason you chose this set of morphological features and not things like numbers of stop words, content words, and function words?\n\n- To improve readability, consider placing figures and tables on the same page as their corresponding references. This would eliminate the need for readers to constantly flip back and forth to interpret figures and tables.\n\n- L345: where the claim of correlation between human syntactic processing and dependency length is stated, please provide an appropriate citation.\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper and provide helpful feedback and suggestions. We would like to answer your questions about our work as the following. \n1. “Explain how extreme summarization could amplify linguistic compression patterns”\nExtreme summarization amplifies linguistic compression because it is inherently abstractive and more compressed compared with other forms of summarization. Being abstractive and having such a tight length constraint helps ensure that maximal compression will occur, compared with arbitrary-length summaries, which usually have some form of verbatim copying from the source document. In addition, extreme summary datasets tend to have fewer supplemental materials that are non-essential in the source document (Bommasani and Cardie 2020), providing a purer window on the source compression.\n2. “Is there a reason you chose this set of morphological features and not things like numbers of stop words, content words, and function words?” \nWe specifically chose the two morphological features (the number of morphemes and word lengths) to potentially capture subtle word choices in writing beyond changes in the distribution of different types of words (e.g. stop, function, content words). This assumption comes from education and children’s psychology research that control of morphological structures in writing correlates with capabilities of reading comprehension (e.g. McCutchen et al. 2003 and Deacan et al. 2015). Morphological features are also natural to use to study compressions because they quantify the change of sizes of words in terms of the numbers of semantic word-internal units. We will revise the paper to provide more explanation of this.\n3. “To improve readability, consider placing figures and tables on the same page as their corresponding references.” \nThank you for this suggestion. We will take this into consideration in the future revision of our work. \n4. “Where the claim of correlation between human syntactic processing and dependency length is stated, please provide an appropriate citation.” \nWe will provide more citations on the correlation between human sentence processing and dependency length besides the citation of Gibson 2000 in footnote 7. The following is a representative but not exhaustive list of literature that examines such correlation: Demberg and Keller 2008; Grodner and Gibson 2005; Warren and Gibson 2002; Lewis et al. 2006. (We will give the full references in the end)\n5. “The focus on extreme summarization limits the scope and applicability of this work.”\nWe agree that our results on the linguistic compression of extreme summaries may or may not fully generalize to longer summaries, which often involve direct copying of the document. However, our work focuses on summarization as a human activity and linguistic process, which has not been studied thoroughly in the community before. Investigating extreme summaries is a natural first step because it is a natural task with many real-world applications (e.g., headline writing, topic sentence writing, gist distillation) from both the human and machine perspectives and has long been a very influential setup in the automatic summarization community. In addition, most computational linguistics tools work significantly better in terms of reliability and interpretability at the sentence level than at the paragraph/document level. We are not focusing on generalizing to summaries at normal or arbitrary length in this work, but this is indeed a very interesting question that we are interested in pursuing in the future.\nThanks again for your helpful comments. We hope the above answers can address your questions, and we will include the above clarifications in revisions of the paper!\n\nReference\nRishi Bommasani and Claire Cardie. 2020. Intrinsic Evaluation of Summarization Datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075–8096, Online. Association for Computational Linguistics.\nGreen, L., McCutchen, D., Schwiebert, C., Quinlan, T., Eva-Wood, A., & Juelis, J. (2003). Morphological Development in Children's Writing. Journal of Educational Psychology, 95(4), 752–761. https://doi.org/10.1037/0022-0663.95.4.752\nDeacon, S. H., Tong, X., and Francis, K. (2017) The relationship of morphological analysis and morphological decoding to reading comprehension. Journal of Research in Reading, 40: 1–16. doi: 10.1111/1467-9817.12056.\nDemberg V, Keller F. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition. 2008 Nov;109(2):193-210. doi: 10.1016/j.cognition.2008.07.008. Epub 2008 Oct 18. PMID: 18930455.\nGrodner, D.J. and Gibson, E.A. (2005) Consequences of the serial nature of linguistic input for sentential complexity. Cogn. Sci. 29, 261–290\nWarren, T. and Gibson, E. (2002) The influence of referential processing on sentence complexity. Cognition 85, 79–112\nLewis RL, Vasishth S, Van Dyke JA. Computational principles of working memory in sentence comprehension. Trends Cogn Sci. 2006 Oct;10(10):447-54. doi: 10.1016/j.tics.2006.08.007. Epub 2006 Sep 1. PMID: 16949330; PMCID: PMC2239011.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The theoretical insights presented in this study can hold significant value in the field of summarization, offering the potential to provide insight to enhance the generation of summaries that align more closely with human preferences.\n\n- Furthermore, this study has the potential to contribute to the establishment of more general reference-independent metrics for evaluating the desirability of AI-generated summaries, as well as for addressing other text generation tasks.\n", "reasons_to_reject": "The focus on extreme summarization (single-sentence summaries) limits the scope and applicability of this work. The outcomes of this study might not necessarily extend to more commonly used summarization tasks that involve longer summaries than a single sentence, and it’s not clear how these results may extend to larger span of text as summary.", "questions_to_authors": "- You state that extreme summarization would amplify the linguistic compression patters. Could you please elaborate on this?\n\n- Is there a reason you chose this set of morphological features and not things like numbers of stop words, content words, and function words?\n\n- To improve readability, consider placing figures and tables on the same page as their corresponding references. This would eliminate the need for readers to constantly flip back and forth to interpret figures and tables.\n\n- L345: where the claim of correlation between human syntactic processing and dependency length is stated, please provide an appropriate citation.\n\n\n", "paper_topic_and_main_contributions": "This paper presents a corpus study investigating the strategy and patterns employed in creating single-sentence human summaries. The findings indicate that single-sentence summaries typically exhibit higher lexical diversity, contain a larger number of morphemes, and feature longer words when compared to the source text.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tIZstWgbmD", "reviewer": "Reviewer_xvMe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your time in reading our manuscript and your helpful suggestions. We would like to address a few of your comments below:\n1. ”What are some surprising new insights in the analysis?”\nWe acknowledge that our results might not be counter-intuitively surprising or state-of-the-art, but our work used a novel combination of computational methods to study linguistic compression patterns and correlations at scale and to shed light on intuitive hypotheses of the summarization process with quantitative evidence. We agree that the summarization process is typically treated as a black box and we hope to provide tools and a starting point for investigating summarization as a human process in more detail though our findings in this paper were largely intuitive.\n2. “Some of the correlations between reader preferences and linguistic features are hard to use in building summarization systems.”\nAlthough our main focus in this paper is to better understand the basic science underlying summarization as a cognitive and linguistic activity, we do believe that the features identified as helpful in our work can be potentially helpful for engineering automatic summarization systems for future works. Such features, like the number of unique parts of speech and the type-token ratio, could be easily manipulated in controllable generation settings, though this is beyond the scope of this paper.\n3. “The strength of the reader preference study is weak given the number of participants.”\nRegarding our human study, we agree with the feedback on its limited scale. However, since our main focus is on understanding linguistic patterns in summarization, we set up the human study mainly to supplement our quantitative corpus findings, especially considering the cost of scaling up human evaluations for our specific setting where each participant needs to do a significant amount of reading and answer a comprehensive set of questions, and to showcase a first step of this novel direction towards understanding summarization as a human activity. We believe this supplementary study is worth incorporating into the main body of our computational work, and we think our results in the human study are interesting to expand upon in future human-focused studies.\nThanks again for your helpful comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Very clear, well motivated problem statement; nicely replicable experiments on reasonably large datasets (which are easily accessible datasets); interesting problem, especially since so many of the use cases for all the deep learning approaches involve summarization (but the actual process is non-transparent). The analysis is useful in understanding how summaries differ from the source documents in terms of their linguistic features and will likely (at least it did for me) raise interesting questions of how we may want to structure systems to explicitly model some of the desired transformations for an improved reader experience.", "reasons_to_reject": "The analysis while well done yields no surprising new insights. Some of the correlations between reader preferences and linguistic features are difficult to use in building summarization systems since linguistic choices in the summary are constrained primarily by source material. The reader preferences study was weak with \"6 native English speakers\" rating the summary quality on 7 aspects. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an analysis of human generated single-sentence 'headline' summaries. The analysis compares various aspects of the original source stories and companion summaries. The analyses were conducted on over a million article summary pairs from three data sets from three different genres (instructional texts, news stories and academic/scientific papers). The importance of various features in summarization was computed using a regression model for classifying sentences as belonging to either source/summary buckets. Finally, the summary-features were correlated with reader preferences using feedback from a reader-panel.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m7ZI5qPEjf", "reviewer": "Reviewer_fk6U", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time and consideration in reading our manuscript. We are really excited about this direction of studying summarization from a human-centered perspective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper could improve automatic summarization quality. It is refreshing to see the effort put into consolidating traditional and automated approaches.", "reasons_to_reject": "I do not see a reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses extreme summarization (one sentence) and compares automatic summarization systems to features of quality human summarization. It further provides human evaluation of automatic summarization and reports the correlations among observed features and human ratings. The novel and interesting thing about this paper is the heavy use of literature on summarization as a human activity and the way in which insights in that literature are turned into features that can be measured in both human and automated summaries. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The results are sound and convincing but their contribution to linguistics and/or cognition are not immediately perceivable.
Decision: Accept-Findings