Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects - A Survey (2024)

Abstract: Generic text summarization approaches often fail to address the specific intent and needs of individual users. Recently, scholarly attention has turned to the development of summarization methods that are more closely tailored and controlled to align with specific objectives and user needs. Despite a growing corpus of controllable summarization research, there is no comprehensive survey available that thoroughly explores the diverse controllable attributes employed in this context, delves into the associated challenges, and investigates the existing solutions. In this survey, we formalize the Controllable Text Summarization (CTS) task, categorize controllable attributes according to their shared characteristics and objectives, and present a thorough examination of existing datasets and methods within each category. Moreover, based on our findings, we uncover limitations and research gaps, while also exploring potential solutions and future directions for CTS. We release our detailed analysis of CTS papers at https://github. com/ashokurlana/controllable_text_ summarization_survey.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 1603–1623 August 11-16, 2024 ©2024 Association for Computational Linguistics
Generic text summarization approaches often fail to address the specific intent and needs of individual users. Recently, scholarly attention has turned to the development of summarization methods that are more closely tailored and controlled to align with specific objectives and user needs. Despite a growing corpus of controllable summarization research, there is no comprehensive survey available that thoroughly explores the diverse controllable attributes employed in this context, delves into the associated challenges, and investigates the existing solutions. In this survey, we formalize the Controllable Text Summarization (CTS) task, categorize controllable attributes according to their shared characteristics and objectives, and present a thorough examination of existing datasets and methods within each category. Moreover, based on our findings, we uncover limitations and research gaps, while also exploring potential solutions and future directions for CTS. We release our detailed analysis of CTS papers at https://github. com/ashokurlana/controllable_text_ summarization_survey.

heading: 1 Introduction
text: Despite the significant advancements in automatic text summarization, its one-size-fits-all approach falls short in meeting the varied needs of different segments of users and application scenarios. For example, generic automatic summarization may struggle to produce easily understandable summaries of scientific documents for non-expert users or create extremely brief summaries of news stories for online feeds. Lately, a myriad of works have emerged aimed at generating more controlled (Fan et al., 2018a; Maddela et al., 2022; He et al., 2022; Zhang et al., 2023b; Pagnoni et al., 2023) and tailored text summaries that meet a wide range of user needs.
CTS task is centered around creating summaries of source documents that adhere to specific criteria.
These criteria are managed through various controllable attributes (CA) or aspects like summary length (Kwon et al., 2023), writing style (Goyal et al., 2022), coverage of key information (Li et al., 2018; Jin et al., 2020b), content diversity (Narayan et al., 2022), and more. These criteria vary depending on the task, user needs, and specific application context. For example, length-controlled summaries (Hitomi et al., 2019) are particularly useful in situations where brevity is crucial, like in social media posts, headlines, and abstracts. In areas such as marketing, academic writing, or professional communication, a style-controlled summary (Chawla et al., 2019) is essential to ensure that the information aligns with the intended tone and messaging strategy. Similarly, topic-controlled summaries (Bahrainian et al., 2021) are commonly used in research papers, reports, and content curation, providing an emphasis on a specific topic to enhance clarity and coherence in the presented information.
There is an uneven distribution of attention within the research community towards various CAs as depicted in Fig 1. The majority of CTS works concentrate on managing length, topic, and style. This could be attributed to two main factors. First, it is comparatively simpler to develop datasets for evaluating length, topic, and style compared to aspects like structure and diversity. Second, there is a plethora of application scenarios for
ar X
iv :2
31 1.
09 21
2v 3
[ cs
.C L
] 2
8 M
ay 2
02 4
1603
length or topic-oriented summaries, such as generating concise news feeds or focused legal reports.
In this survey, we collect and analyze 61 research papers pertaining to various possible CAs. The filtration criteria for the selection of papers are described in Appenedix A. Subsequently, we classify these CAs into 10 categories, grouping similar ones based on shared characteristics and objectives. Moreover, we delve into the existing datasets, evaluating their creation methods and appropriateness for the respective task in each CA category. Furthermore, we scrutinize the current CTS methodologies for each CA category, drawing comparisons between their overarching frameworks and discussing relevant limitations. Subsequently, we discuss in detail the generic and specific evaluation strategies for CAs utilized by various works. Finally, we attempt to critique the current approaches and unravel potential future research trajectories. To the best of our knowledge, it is the first comprehensive survey on CTS.

heading: 2 Task Formulation
text: This section introduces the Controllable Text Summarization (CTS) task by outlining its definition and offering a categorized breakdown of different controllable attributes along with concise descriptions for each. Given a set of source documents D = {d1, d2, . . . , dk}. Each document, di, consists of a sequence of n tokens: {xi,1, xi,2, . . . , xi,n}. Si is the target summary of document di, which comprises of a sequence of m
Attribute Definition Length Controlling the length of the summary Style Controlling the readability levels, politeness, humor, and emotion Coverage Controlling the salient information in summary Entity Summary specific to pre-defined entities Structure Create summaries with predefined structure or order Abstractivity Controlling the novelty in sentence formation Salience Adjusting the presence of prominent information Role Providing role-specific summaries Diversity Generating semantically diverse summaries Topic Controlling topic-focused summary generation
Table 2: Controllable attributes definitions.
tokens: {si,1, si,2, . . . , si,m}, where m ≪ n. The user wants to control a set of controllable attributes C. The task can be framed as a conditional generative problem: P (S|D,C) = ∏ki P (Si|di, C)

heading: 2.1 Controllable Attributes
text: A Controllable Attribute or Aspect (CA) refers to a user or application-driven trait of a summary designed to meet specific criteria or conditions, such as Length, Style, Role, etc. In the literature, it is evident that various authors use different terms to describe the same CAs, which exhibit similar characteristics and objectives (such as “Salience: Key information", and “Coverage: Granularity"). Additionally, numerous attributes can be encapsulated by a representative class; for instance, “Style” may serve as a class encompassing Tone, Readability, Humor, Romance, and similar aspects, facilitating their classification within the same category as shown in Table 3. Based on these observations, we group the CAs into 10 categories as listed in Table 2.

heading: 3 Related Surveys
text: In the literature, a multitude of surveys center around conventional text summarization methods (El-Kassas et al., 2021; Nazari and Mahdavi, 2019; Allahyari et al., 2017; Gambhir and Gupta, 2017), including task-specific surveys such as multi-document summarization (Sekine and Nobata, 2003), cross-lingual summarization (Wang et al., 2022), and dialogue-based summarization (Tuggener et al., 2021). There are a few surveys that concentrate on text generation techniques (Zhang et al., 2023a; Prabhumoye et al., 2020) and the causal perspective (Wang et al., 2024; Hu and Li, 2021) on the same. On the contrary, this is the first survey, that focuses on controllable summarization by offering a thorough analysis of CTS methods, challenges, and prospects.

heading: 4 Datasets
text: This section provides a broad overview of the CTS datasets and corresponding creation/acquisition strategies. The CTS methods are evaluated in several ways: 1) by utilizing publicly available summarization datasets, 2) by datasets derived from generic datasets, and 3) by creating humanannotated datasets.

heading: 4.1 Generic Datasets
text: CTS research predominantly leverages widely used news summarization datasets. Notably, about 57% of CTS studies utilize either CNN-DailyMail (Nallapati et al., 2016) or DUC (Over and Yen, 2004; Dang, 2005). Other popular datasets, including Gigaword (Napoles et al., 2012), XSum (Narayan et al., 2018), NYTimes (Sandhaus, 2008), NEWSROOM (Narayan et al., 2018), and dialogue-based SAMSUM (Gliwa et al., 2019), along with opinionbased datasets (Angelidis and Lapata, 2018; Angelidis et al., 2021), are employed for controllable summarization. However, these generic datasets lack explicit annotations and nuances to evaluate the CA-specific summarization. CTS requires specialized datasets (as detailed in Table 4) to provide evaluation opportunities for specific aspects like length, topic, style, etc.

heading: 4.2 Derived Datasets
text: The derived datasets are obtained by applying the aspect-specific heuristics to the widely used generic datasets. In this section, we list out a few derived datasets and their creation strategies.
JAMUL. Hitomi et al. (2019) collect lengthsensitive headlines for the Japanese language. Each article consists of three headlines with varying lengths of 10, 13, and 26 characters respectively. TS and PLS. In order to enhance the readability of biomedical documents, Luo et al. (2022) introduce two types of summaries. The Technical Summary (TS) is an abstract of a peer-reviewed bio-medical research paper and the Plain Language Summary (PLS) is the authors submitted summary as part of the journal submission process. Wikiasp. In order to construct the multi-domain aspect-based summarization corpus, Hayashi et al. (2021) utilize the Wikipedia articles from 20 domains. Further, the section titles and paragraph boundaries of each article are obtained as a proxy of aspect annotation. In another study, Ahuja et al. (2022) create the ASPECTNEWS dataset for aspect-oriented summarization. They achieve it by utilizing articles from the CNN/DailyMail dataset and identifying documents related to ‘earthquakes’ and ‘fraud investigations’ by using the universal sentence encoder (Cer et al., 2018). Further, Mukherjee et al. (2020) collect a CA-based opinion summarization dataset consisting of tourism reviews. These are obtained from the TripAdvisor website and identified the relevant aspects using the unsupervised attentionbased aspect extraction technique (He et al., 2017).

heading: 4.3 Human annotated
text: This section provides the details of the humanannotated CTS datasets. GranDUC. By re-annotating the DUC-2004 (Dang, 2005), Zhong et al. (2022) release a novel benchmark dataset for the granularity control. Annotators are instructed to create summaries of multiple documents with coarse, medium, and fine granularity levels. Multi-LexSum. Shen et al. (2022c) create a human-annotated corpus of 9,280 civil rights lawsuits and corresponding summaries with different degrees of granularity. The target summary length ranges from one-sentence to multiparagraph level. EntSUM. (Maddela et al., 2022) is a human-annotated entity-specific controllable summarization dataset. It utilizes the articles from The New York Times Annotated Corpus (NYT) (Sandhaus, 2008) and includes annotated summaries for PERSON and ORGANISATION tags. The recent release of EntSUMV2 (Mehra et al., 2023) is the more abstractive version of EntSUM. NEWTS. Bahrainian et al. (2022) introduce the top-
ically focused summarization corpus by leveraging documents from CNN-DailyMail and employing crowd-sourcing to generate two distinct summaries with different thematic aspects for each document. CSDS. Lin et al. (2021) introduce the role-oriented Chinese Customer Service Dialogue Summarization (CSDS) dataset. It is meticulously annotated, segmenting the dialogues based on their topics and summarizing each segment as a QA pair. MReD. To tackle the task of structure-controllable summarization, Shen et al. (2022b) introduce the MetaReview Dataset (MReD). It is created by gathering meta-reviews from the open review system and categorizing each sentence into one of nine predefined intent categories (abstract, strength, weakness, etc.,). MACSUM. Zhang et al. (2023b) develop a human-annotated corpus to control the mix of CAs (Topic, Speaker, Length, Extractiveness, and Specificity) together. MACSUM covers source articles from CNN/DailyMail and QMSUM (Zhong et al., 2021) datasets.

heading: 5 Approaches to Controlled Summarization
text: Various CAs have been investigated in controllable summary generation tasks, including style (politeness, humor, formality), content (length, entities, keywords), and structure. In this section, we describe various approaches to achieve CTS for the attributes mentioned in Table 2. Additionally, we list out the novel contributions and limitations for each paper in the Appendix C Table 9. Length. Earlier methods lacked length control and only employed heuristics such as stopping the generation after a fixed number of tokens. To overcome this, four different approaches to integrate length as a learnable parameter are proposed.
Adding length in input: Fan et al. (2018b) propose a convolutional encoder-decoder-based summarization system, where it quantizes summary lengths into discrete bins of different size ranges. During training, the input data is prepended with the gold summary length represented by bin lengths. Due to a fixed number of length bins, the system fails to generate summaries of arbitrary lengths. CTRLSUM (He et al., 2022) presents a generic framework to generate controlled summaries using keywords specific to length. Instead of controlling a single attribute, Zhang et al. (2023b) allow different length attribute values (normal, short, long) to be used as inputs along with the source text for hard prompt tuning (Brown et al., 2020).
Adding length in encoder: Yu et al. (2021) propose a length context vector that is generated at each decoding step derived from the positional encodings. This vector is then concatenated with the decoder hidden state and encoder attention vectors. The limitation of the system is the generation of incomplete summaries for short desired lengths. Liu et al. (2022b) propose a length-aware attention model that adapts the source encodings based on the desired length by pretraining the model. Zhang et al. (2023b) add a hyperparameter for learning the prefix embeddings for different attributes at each layer of the encoder and decoder for soft prefix tuning (Li and Liang, 2021).
Adding length in decoder: Kikuchi et al. (2016) propose the first method to control length using a BiLSTM encoder-decoder architecture with attention (Luong et al., 2015) for sentence compression. In each step of the decoding process, an additional input for the remaining length is provided as an embedding. Instead of pre-defined length ranges, Liu et al. (2018) add a desired length parameter
at the decoding step to each convolutional block of the initial layer of the convolutional encoderdecoder model. Févry and Phang (2018) design an unsupervised denoising auto-encoder for sentence compression, where the decoder has an additional input of the remaining summary length at each time step. While it produces grammatically correct summaries, but they are nonsensical or semantically different from the input. This leads to the generation of unfaithful summaries. To handle the length constraint, Takase and Okazaki (2019) propose two modifications to the sinusoidal positional embeddings on the decoder side: length-difference positional encoding and length-ratio positional encoding. Sarkhel et al. (2020) present a multi-level summarizer that models a multi-headed attention mechanism using a series of interpretable semantic kernels to control lengths, reducing the trainable parameters significantly. The model does not encode the length attribute directly. Song et al. (2021) design a confidence-driven generator that is trained on a denoising objective with a decoder-only architecture, where the source and summary tokens are masked with position-aware beam search. Goyal et al. (2022) use a mixture-of-experts model with multiple transformer-based decoders for identifying different styles or features of summaries. Kwon et al. (2023) introduce the summary length prediction task on the encoder side and this predicted summary length is inserted with a length-fusion positional encoding layer. Adding length in loss/reward function: Makino et al. (2019) propose a global minimum risk training optimization method under length constraint for the neural summarization tasks which is faster and generates five times fewer over-length summaries on an average than others. Chan et al. (2021) use an RL-based Constrained Markov Decision process with a mix of attributes. Hyun et al. (2022) devise an RL-based framework that incorporates both length and quality constraints in the reward function to generate multiple summaries of different lengths and according to the experimental results present in Hyun et al. (2022), the model is computationally expensive.
Style. The generation of user-specific summaries has gained significant interest, but achieving distinct styles has posed an enduring challenge. These stylistic variations may encompass tone, readability control, or the modulation of user emotions. Style
control aims to generate source-specific summaries (Fan et al., 2018a) by utilizing the convolutional encoder-decoder network. Chawla et al. (2019) obtain formality-tailored summaries by utilizing the input-dependent reward function. The pointer-generator (See et al., 2017) network is used as the under-laying architecture and the loss function is modified with the addition of a formality-based-reward function. In another study, Jin et al. (2020a) attempt to control humor, romance, and clickbait in headlines using a multitask learning framework. By employing an inference style classifier, Cao and Wang (2021) adjust the decoder final states to obtain stylistic summaries. Moreover, they obtain lexical control by utilizing the word unit prediction that can directly constrain the output vocabulary. Similarly, Goyal et al. (2022) extend the decoder architecture to a mixture-of-experts version by using multiple decoders. The gating mechanism helps to obtain multiple summaries for a single source. However, the major limitation in this model is its manual gating mechanism. To control various fine-grained reading grade levels, Ribeiro et al. (2023) present three methods: instruction-prompting, reinforcement learning-based reward model, and look-ahead readability decoding approach. Coverage. Managing the information granularity is essential to measure the semantic coverage between the source text and the summary. To regulate the granularity, Wu et al. (2021) introduce a two-stage approach, where the model incorporates a summary sketch, that encompasses user intentions and key phrases, serving as a form of weak supervision. They leverage a text-span-based conditional generation to govern the level of detail in the generated dialogue summaries. Zhong et al. (2022) propose a multi-granular event-aware summarization method composed of four stages: event identification, unsupervised event-based summarizer pretraining, event ranking, and summary generation by adding events as hints. Extraction of events from source text may lower the abstractiveness. Zhang et al. (2023b) use the hard and soft-prompting strategies to control the amount of extracted text from the source in the summary. Additionally, Huang et al. (2023) utilize the natural language inference models to improve the coverage.
Entity. Entity-centric summarization concentrates on producing a summary of a document that is specific to a given target entity (Hofmann-Coyle et al.,
2022). Zheng et al. (2020) extract the named entities using a pre-trained BERT (Devlin et al., 2019) based model and feed both the article and the selected entities to a bidirectional LSTM (Hochreiter and Schmidhuber, 1997) encoder-decoder model. In another study, Liu and Chen (2021) extract the entities (speakers and non-speaker entities) from a dialogue to form a planning sequence. The entities extracted are concatenated to the source dialogue for training the conditional BART-based model. This model introduces factual inconsistency due to paraphrasing from a personal perspective. Maddela et al. (2022) extend the GSum (Dou et al., 2021) by feeding it either sentences or strings, which mention extracted entities as guidance. The model is an adapted version of BERTSum (Liu and Lapata, 2019), where only the sentences containing the entity string mention and its coreferent mentions are fed. Hofmann-Coyle et al. (2022) model entity-centric extractive summarization as a sentence selection task. Building upon BERTSum (Liu and Lapata, 2019), they use a BERT (Devlin et al., 2019) based encoders to represent the sentence and target entity pair and train with a contrastive loss objective to extract sentences most relevant to the target entities. Structure. Generic datasets lack key elements for emphasizing specific aspects in the corresponding ground truth summaries. To address this limitation and emphasize summary structure, Shen et al. (2022b) achieve structure-controllable text generation by adding a control sequence at the beginning of the input text and treating summary generation as a standalone process. However, this approach has two main limitations, 1) generated tokens are solely based on logits predictions without ensuring that the sequence satisfies the control signal, 2) Auto-regressive models face error propagation in generation due to self-attention, causing subsequent generations to deviate from the desired output. To overcome these challenges, the sentence beam-search (SentBS) (Shen et al., 2022a) approach produces multiple sentence options for each sentence and selects the best sentence based on both the control structure and the model’s likelihood probabilities. In a related study, Zhong and Litman (2023) utilize predicted argument role information to control the structure in legal opinion documents. Additionally, in the work of Zhang et al. (2023b), the prompt of entity chains, representing an ordered sequence of entities, is used for
pre-training and fine-tuning with a planning objective to control the summary structure.
Abstractivity. It measures the degree of textual novelty between the source text and summary. See et al. (2017) introduce a pointer-generator network to control the source copying via pointing and generate novel sentence formations by using generator mechanism. However, this scheme fails to generate higher abstraction levels. Kryściński et al. (2018) tackle this problem in two ways: 1) decompose the decoder into a contextual network to retrieve the relevant parts of the text and generate the summary by utilizing a pretrained model, 2) a mixed RL-based objective jointly optimizes the n-gram overlap with the ground truth summary. Similarly, Song et al. (2020) control the copying behavior by using a mix-and-match strategy to generate summaries with varying n-gram copy rates. Based on the seen, unseen words from the source text, the system controls the copying percentage by acting as a language modeling task. Moreover, methods such as ControlSum (Fan et al., 2018a) allow the users to explicitly specify the control attribute to facilitate better control. However, it does not provide any supervision on violating the controllability. To alleviate this issue, Chan et al. (2021) propose an RL-based framework on the constrained Markov decision process and introduced a reward to penalize the violation of attribute requirement.
Salience. This attribute captures the most important information in a document. In SummaRuNNer (Nallapati et al., 2017), salience is modeled as a feature in a classification objective. It uses GRU-based encoders and decoders to frame summarization as a text-to-binary sequence learning task at the sentence level (Bahdanau et al., 2014; Cho et al., 2014). A binary score is assigned to each sentence, indicating its membership in the summary. The system performs poorly on out-ofdomain datasets. To retain key content from the source, Li et al. (2018) introduce a Key Information Guide Network, where keywords are identified by the TextRank algorithm with a modified attention mechanism that accommodates this key information as an additional input. However, it focuses mostly on informativeness ignoring coherence and readability features. Deutsch and Roth (2023) model salience in terms of noun phrases using QA signals where the generation of the summary is conditioned on these identified phrases. This approach is not applicable
to languages for which question generation and question answering models are not available. In long document CLS tasks, summarization systems often fail to respond to user queries. To resolve this issue, Pagnoni et al. (2023) propose a pre-training approach that involves two tasks of salient information identification from sentences having the highest self ROUGE score and a question generation system to generate questions whose answers are the salient sentences. Role. Role-oriented dialogue summarization generates summaries for different roles/agents present in a dialogue (e.g. doctor and patient) (Liang et al., 2022). Lin et al. (2021) propose the CSDS dataset (see Section 4.3) and benchmark a variety of existing state-of-the-art summarization models for the task of generating agent and user surveys. They find that agent summaries generated by the existing methods lack key information, that needs to be extracted from dialogues of the other role. To bridge this gap, Lin et al. (2022) build a role-aware summarization model for two users (agent and user) present in the dataset. They use two separate decoders for generating the user and agent summaries by utilizing user and agent masks. A role attention mechanism is introduced to each decoder so that it can leverage the overall context by attending to the hidden states of the other role. Liang et al. (2022) use a role-aware centrality scoring model that computes role-aware centrality scores for each utterance, which measures the relevance between the utterance and the role prompts (signaling whether the summary is for the user or agent). This is then used to reweight the attention scores for each utterance, which is subsequently used by the decoder to generate the summary. Diversity. Traditional decoding strategies, like beam search, excel at generating single summaries but often struggle to produce diverse ones. Techniques such as top-k and nucleus sampling are effective in generating diverse outputs but may sacrifice faithfulness. In response to these challenges, Narayan et al. (2022) introduce compositional sampling, a decoding method to obtain diverse summaries. This method initiates by planning a semantic composition (Narayan et al., 2021) of the target in the form of entity chains, and then leverages beam search to generate diverse summaries. Topic. Long documents often cover multiple topics, and a generic summary might not fully encompass the diverse scope. Krishna and Srinivasan (2018) train a topic-conditioned pointer-generator network (See et al., 2017) by concatenating one hot encoding representation of the topic with the embedding of each token in the input document. However, news categories are used as the predefined topics, that limits the generalization to other tasks. To handle diverse topics, Tan et al. (2020) utilize external knowledge sources like Wikipedia and ConceptNet to create a weakly supervised summarization framework compatible with any encoder-decoder architecture. Suhara et al. (2020) propose an unsupervised method, where aspect-specific opinions are extracted from a set of reviews by a pre-trained opinion extractor, and the summary of the opinion is generated by a generator model trained to reconstruct the reviews from the opinions. Similarly, given a set of reviews for a product (e.g. Hotels), Amplayo et al. (2021) train a Multiple Instance Learning (MIL) model, to extract the predictions for aspect (like cleanliness) codes at the document, sentence, token level (Mukherjee et al., 2020). These predicted aspects transform the input such that relevant sentences and keywords along with aspect tokens are fed into the pre-trained T5 (Raffel et al., 2020) model. Hsu and Tan (2021) introduce the task of generating decision-supportive summaries. The focus is on predicting future Yelp ratings from the set of reviews using a Longformer-based (Beltagy et al., 2020) regression model. They propose an iterative algorithm that selects the sentences of the summary from a set of representative sentences. Mukherjee et al. (2022) extend topic-focused summarization for multimodal documents by creating a joint image-text context vector.

heading: 6 Evaluation Strategies
text: This section catalogs and briefly describes the variety of automatic and human evaluation metrics that are being used to evaluate the summaries generated by the different methods studied in this paper.

heading: 6.1 Automatic Evaluation
text: The automatic evaluation metrics can be categorized based on how they are defined. We categorize the metrics into n-gram-based, language-modelbased, and aspect-specific. N-gram based evaluation metrics like ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) are based on matching n-grams from candidate summaries to a set of reference summaries. ROUGE
is the most widely used metric in CTS literature. Language-model based metrics are computed using Pre-trained Language Models (PLM) like BERT (Devlin et al., 2019) or BART (Lewis et al., 2019). One class of approach computes the distance between the PLM embeddings of the reference and the generated summary. Another way is based on computing the log probability of the generated text conditioned on input text as demonstrated in BARTScore (Yuan et al., 2021). Summarization specific metrics including ROUGE-WE (Ng and Abrecht, 2015), S3 (Peyrard et al., 2017), Sentence Mover’s Similarity (SMS) (Clark et al., 2019), SummQA (Scialom et al., 2019), BLANC (Vasilyev et al., 2020), and SUPERT (Gao et al., 2020), (Lite)3Pyramid (Zhang and Bansal, 2021) are prominent for controllable summary evaluation. Aspect specific metrics do not fall cleanly into either of the above-mentioned categories. These metrics focus on evaluating specific controllable aspects such as Flesh Reading Ease (Flesch, 1948), Gunning Fog Index, and Coleman Liau Index for readability, control correlation, and error rate (Zhang et al., 2023b) for topic, abstractivity and role attributes. Appendix B Table 7 describes more details about the automatic evaluation metrics.

heading: 6.2 Human Evaluation
text: Human evaluation is an indicator of the robustness and effectiveness of different summarization systems on specific aspects that cannot be directly captured by automatic evaluation metrics. These aspects include generic properties of a summary such as truthfulness (Song et al., 2020; Hyun et al., 2022), relevance (Goyal et al., 2022; He et al., 2022; Shen et al., 2022b), fluency (Narayan et al., 2022; Suhara et al., 2020), and readability (Cao and Wang, 2021; Kryściński et al., 2018) or specific properties such as completeness (Yu et al., 2021; Liu et al., 2022a) for length-controlled summaries, coverage (Mukherjee et al., 2020, 2022) for the entity, and topic-controlled summary generation. Broadly two kinds of scoring mechanisms are used for human evaluation: binary and rank-based. The rank-based scores usually range from 1 to 5. Despite these widely adapted mechanisms, human evaluation of summarization is challenging due to ambiguity and subjectivity. Aspects like coherence and fluency help mitigate ambiguity, but remain subjective to individual annotators. Accurately defining annotation descriptions is crucial, yet achieving a stan-
dardized approach across annotators remains difficult (Iskender et al., 2021; Ito et al., 2023). The details about different human evaluation metrics are detailed in Appendix B Table 8.

heading: 7 Challenges and Future Prospects
text: Generic vs specialized benchmarks. We observe that more than 75% of CTS works either utilize or alter the generic news summarization datasets to evaluate the controllable summarization. As shown in Table 2, out of the 10 categories, we could find CA-specific datasets for only seven categories. We envisage that conducting evaluations with specialized datasets that align closely with real-world application scenarios or user requirements will help better in assessing the practical utility, robustness, and performance of CTS. It is evident from our survey of CTS systems that evaluations are often confined to specific domains, like news, possibly due to the abundance of available datasets in that domain. However, this narrow focus limits the evaluation of the CTS model’s robustness. Standardization of metrics. The goal of the CTS task is to produce CA-specific summaries, warranting a metric tailored to capture the nuances of this particular attribute. We observe that comparing models for a specific CA-based CTS task is challenging due to the use of varying metrics, leading each study to redo evaluations for a fair comparison with prior work. Standardizing CA-specific evaluation metrics could offer a valuable solution. Explainability. For effectively controlling user or application-specific attributes, it is imperative to leverage the understanding of the decision-making process within CTS systems. Also, this comprehension is essential for users or stakeholders, enabling them to discern how the system generates summaries from source text. This holds particular significance in applications where human decisionmaking or interpretation plays a pivotal role, such as in legal, medical, or financial domains. The existing CTS efforts lack proper emphasis on the explainability aspects, which can be readily addressed through the incorporation of suitable explanation methodologies (Abnar and Zuidema, 2020; Sundararajan et al., 2017; Lundberg and Lee, 2017). Multi-lingual, multi-modal, and code-mixed CTS. The existing literature on CTS predominantly focuses on works in English, with only one study addressing the topic in a Japanese context. We could not find any studies and datasets related
to multilingual and code-mixed CTS approaches. Moreover, the task of controllable summarization in multi-modal and multi-document settings remains largely unexplored, presenting unique challenges for models to address and offering avenues for intriguing research problems.
Multi-CA control. Even though, few of the works perform multi-attribute controllable summarization (Goyal et al., 2022; He et al., 2022; Zhang et al., 2023b), we observe that existing works predominantly investigate combinations of length and entity attributes (see Table 5). As a future research direction, it’s essential to design models that consider other important combinations of control attributes, such as length, style, and saliency. Furthermore, creating standardized multi-CA benchmarks is crucial to facilitate the evaluations.
Reproducibility. In the detailed analysis outlined in Table 10, we note that 35% of research studies do not share their code publicly. Furthermore, 25% of the papers did not carry out any human evaluation, and among the remaining studies, 79% did not conduct Inter Annotator Agreement (IAA) assessments. The lack of reproducibility (Ito et al., 2023; Gao et al., 2023; Iskender et al., 2021) measures hinders the scientific community’s ability to validate and build upon existing work. On the other hand, the human study component should be a must for a text summarization evaluation scheme, otherwise, we are potentially overlooking essential aspects of real-world applicability. Standing on the shoulders of LLMs. The rise and success of large language models (LLMs) have opened up unparalleled possibilities for leveraging their capabilities across diverse stages of the Natural Language Processing (NLP) pipeline. In the context of CTS, LLMs can be fine-tuned to grasp context-specific nuances about CAs without the need for a dedicated training set. Additionally, when it comes to evaluating CTS models, LLMs
can serve as effective substitutes for human experts or judges (similar to Liu et al. (2023)), offering an efficient method for assessing performance.

heading: 8 Conclusions
text: We present a comprehensive survey on controllable text summarization (CTS) by offering a detailed analysis, from formalizing various controllable attributes, classifying them based on shared characteristics, and delving into existing datasets, proposed models, associated limitations, and evaluation strategies. Moreover, we discuss the challenges and prospects, making it a helpful guide for researchers interested in CTS. We plan to keep the GitHub repository regularly updated with the latest CTS works.

heading: 9 Limitations
text: Although we attempt to conduct a rigorous analysis of existing literature on controllable summarization, some works might have been possibly left out due to variations in search keywords. Furthermore, due to limited space, our survey primarily concentrates solely on the high-level aspects of the approaches, omitting a very fine-grained experimental comparison. Finally, our exploration of multilingual works was limited as we encountered challenges in finding them, likely influenced by the relatively low attention from the research community. We aim to further investigate the potential reasons behind the challenges associated with multilingual CTS tasks.

heading: 10 Ethics statement
text: To uphold transparency and accountability, the papers utilized in this survey are detailed in Appendix D Table 10. We have provided a comprehensive set of papers, accompanied by our qualitative classification and annotations, enabling public scrutiny and examination. Moreover, to alleviate qualitative bias, each paper underwent review by
at least three different individuals independently, aiming to minimize misclassification. We adhere to the same methodology to validate the presence of diverse observations in each paper. By incorporating these ethical considerations, we affirm our dedication to conducting research in an ethical and accountable manner.

heading: A Survey papers selection criteria
text: We used keywords such as “controllable summarization”, “text summarization” and “text generation” for selecting the initial pool of 105 papers. We selected the majority of papers from the reputed databases including the ACL Anthology1, ACM Digital library2, Google Scholar3, which are known for hosting peer-reviewed articles that meet high academic standards. Among these 105 papers, six papers are pertinent to CTS, albeit they have not undergone peer review. Additionally, 23 papers touch upon the summarization aspect to some
1https://aclanthology.org/ 2https://dl.acm.org/ 3https://scholar.google.com/
extent, although they may not be directly aligned with controllable summarization. Furthermore, we have excluded 15 papers as they primarily discuss controllable text generation or focus on enhancing the summarization task without specifically controlling any CTS attributes. Post to applying the above three filters we are left with 61 peer-reviewed and relevant papers to CTS. We have listed the filtration details in Table 6).

heading: B Evaluation Approaches
text: We have listed the automatic and human evaluation methodologies along with their respective metric details in Table 7 and Table 8. The automatic evaluation metrics are categorized into three groups: embedding-based, n-gram-based, and miscellaneous. Additionally, we present a compilation of papers organized by aspects, each associated with the relevant metrics, along with concise descriptions. As for human evaluation, we specify the corresponding metrics and provide definitions based on the attributes under consideration.

heading: C Model Descriptions
text: As outlined in Table 9, we augment novel contributions, utilized dataset, and the corresponding limitation for each paper, all aligned with the respective controllable attribute.

heading: D Survey papers checklist explanation
text: To underscore the comprehensiveness of our survey, as mentioned in Table 10, we include 23 features for each paper. For easier understanding, we briefly describe each feature in the master table below.
• Paper: Citation of the paper.
• Year: Year of the publication.
• Venue: Paper publishing conference or journal.
• Controllable attribute: Controllable attribute(s) concentrated in the paper.
• Controlling more than one aspect: Whether the paper handles more than one controllable aspect or not?
• Model type: Type of the model used in the paper such as encoder-decoder, encoder, or decoder architecture.
• Training strategy: Training approaches employed to perform CTS task.
• Approach: Type of the training approach employed to perform CTS task.
• Code access: Whether the code is publicly accessible or not?
• Code link: Address of the public repository.
• Dataset: Dataset utilized in the paper.
• Source: Source of the dataset used in the paper.
• Nature of the data: Dataset creation/acquisition strategy.
• Data release: Public availability of the dataset.
• Domain: The corresponding domain of the dataset.
• Data link: Public repository link to the dataset.
• Metric name: Name of the metric used in the paper.
• Proposed new metric: Names of the proposed new automatic evaluation metrics.
• Human evaluation: Human evaluation performed or not?
• Metric names: Name of the metrics used to perform human evaluation.
• IAA: Whether Inter Annotator Agreement assessment performed or not?
• Limitation: Any limitations of the paper mentioned or not?
• Reproducibility: Rate the reproducibility of the paper.
From the master table, we have represented our observations in Figures 2, 3, 4, 5, 6.
Type of method
Figure 3: Various training approaches utilized to perform CTS tasks.
Pa pe
r Pu
bl ic
at io
n in
fo C
on tr
ol la
bl e
as pe
ct M
od el
de ta
ils D
at as
et de
ta ils
A ut
om at
ic ev
al ua
tio n
H um
an E
va lu
at io
n L
im ita
tio ns
R ep
ro du
ci bi
lit y
Ye ar
Ve nu
e C
on tr
ol la
bl e
at tr
ib ut
e
C on
tr ol
lin g
m or
e th an on e as pe ct
M od
el ty
pe *
Tr ai
ni ng
st ra
te gy
A pp
ro ac
h C
od e
ac ce
ss C
od e
lin k
D at
as et
So ur
ce N
at ur
e of
th e
da ta
D at
a re
le as
e D
om ai
n D
at a
lin k
M et
ri c
na m
e Pr
op os
ed N
ew M
et ri
c H
um an
E va
lu at
io n
M et
ri cs
na m
es IA
A Ye
so r
N o
M en
tio ne
d? H
ig h/
M ed
iu m
/N ot
su re
*
1 C
ha w
la et
al .(
20 19
) 20
19 C
oN L
L St
yl e
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
no -
C N
N D
M ,
W eb
is -T
L D
R -1
7 co
rp us
do cu
m en
t ge
ne ri
c no
ne w
s, so
ci al
m ed
ia no
R O
U G
E no
no no
no no
no ts
ur e
2 G
oy al
et al
.( 20
22 )
20 22
E M
N L
P St
yl e
ye s
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
C N
N D
M ,
X Su
m ,N
E W
SR O
O M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E ab
st ra
ct iv
en es s, de gr ee of sp ec ifi
ci ty , le ng th ,r ea da bi lit y
ye s
re le
va nc
e, co
he re
nc e,
gr am
m at
ic al
ity ,
fa ct
ua lit
y no
ye s
hi gh
3 So
ng et
al .(
20 21
) 20
21 N
A A
C L
L en
gt h
no de
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k G
ig aw
or d,
N E
W SR
O O
M do
cu m
en t
ge ne
ri c
ye s
ne w
s L
in k
R O
U G
E no
ye s
ye s-
no ty
pe qu
es tio
ns no
no hi
gh
4 So
ng et
al .(
20 20
) 20
20 A
A A
I A
bs tr
ac tiv
ity no
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
G ig
aw or
d, N
E W
SR O
O M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E ,B
E R
T Sc
or e
no ye
s in
fo rm
at iv
en es s, gr am m at ic al ity , tr ut hf ul ne
ss no
no hi
gh
5 Fa
n et
al .(
20 18
b) 20
18 A
C L
-N M
T (W
) L
en gt
h, St
yl e
E nt
ity ye
s en
co de
rde
co de
r pr
eho
c su
pe rv
is ed
no -
C N
N D
M do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no ye
s ra
te ra
sk ed
to pr
ef er
re d
su m
m ar y, bi na ry ev al ua tio
n no
no m
ed iu
m
6 L
iu et
al .(
20 22
b) 20
22 A
C L
L en
gt h
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k C
N N
D M
,X Su
m do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no ye
s gr
am m
at ic
al ly
co rr
ec t,
in fo
rm at
iv en
es s,
ov er
al l
no no
hi gh
7 Z
ha ng
et al
.( 20
23 b)
20 23
TA C
L L
en gt
h, To
pi c
C ov
er ag
e ye
s en
co de
rde
co de
r pr
eho
c su
pe rv
is ed
ye s
L in
k C
N N
D M
,Q M
Su m
do cu
m en
t, di
al og
ue s
hu m
an an
no ta
te d
ye s
ne w
s, so
ci al
m ed
ia L
in k
C on
tr ol
E rr
or R
at e,
C on
tr ol
C or
re la
tio n
co nt
ro le
rr or
ra te , co nt ro lc or re la tio n
ye s
ye s
or no
qu es
tio ns
ye s
no hi
gh
8 Ta
ka se
an d
O ka
za ki
(2 01
9) 20
19 N
A A
C L
L en
gt h
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k JA
M U
S do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no no
no no
no m
ed iu
m
9 Sh
en et
al .(
20 22
a) 20
22 E
M N
L P
St ru
ct ur
e no
en co
de r-
de co
de r
po st
-h oc
su pe
rv is
ed ye
s L
in k
M et
a R
ev ie
w D
at as
et (M
R eD
) re
vi ew
s ge
ne ri
c no
sc ie
nt ifi
c no
R O
U G
E ,B
E R
T SC
O R
E no
ye s
flu en
cy ,c
on te
nt re
le va
nc e,
st ru
ct ur
e si
m ila
ri ty , de ci si on co rr ec tn es
s no
ye s
hi gh
10 G
ol ds
ac k
et al
.( 20
23 )
20 23
A C
L -B
Io N
L P(
W )
St yl
e no
- -
- -
- PL
O S
an d
eL if
e -
- -
- -
R el
ev an
ce ,R
ea da
bi lit y, Fa ct ua lit y
- -
- no
- -
11 C
ao an
d W
an g
(2 02
1) 20
21 N
A A
C L
St yl
e no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
C N
N D
M do
cu m
en t
ge ne
ri c
no ne
w s
no R
ea da
bi lit
y, Pe
rp le
xi ty , B E R T Sc or e
re ad
ab ili
ty ,
pe rp
le xi
ty ye
s in
fo rm
at iv
en es s, flu en cy ,r el ev an
ce no
no hi
gh
12 N
ar ay
an et
al .(
20 21
) 20
21 TA
C L
St ru
ct ur
e no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- C
N N
D M
,X Su
m ,
SA M
Su m
,a nd
B ill
Su m
do cu
m en
t, di
al og
ue s
ge ne
ri c
no ne
w s,
le ga
l no
R O
U G
E en
tit y
pl an
ni ng , sp ec ifi ci ty
ye s
re le
va nt
,a cc
ur at e, co nc is e, flu en t
no no
m ed
iu m
13 A
m pl
ay o
et al
.( 20
21 )
20 21
E M
N L
P To
pi c
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k SP
A C
E ,O
PO SU
M +
re vi
ew s
ge ne
ri c
no so
ci al
m ed
ia no
R O
U G
E no
ye s
in fo
rm at
iv en
es s,
co he
re nc e, co nc is en es s, flu en cy
no no
hi gh
14 D
eu ts
ch an
d R
ot h
(2 02
3) 20
23 E
A C
L Sa
lie nc
y no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- C
N N
D M
,X Su
m ,N
Y Ti
m es
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E ,B
E R
T Sc
or e,
Q A
E va
l no
ye s
se le
ct io
n of
be st
su m
m ar y (b in ar y ev al ua tio n)
no no
no ts
ur e
15 W
u et
al .(
20 21
) 20
21 A
L C
-I JC
N L
P C
ov er
ag e
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k SA
M Su
m di
al og
ue s
ge ne
ri c
no so
ci al
m ed
ia no
R O
U G
E no
ye s
fa ct
ua lc
on si
st en
cy ,
in fo
rm at
iv en
es s
no no
hi gh
16 H
su an
d Ta
n (2
02 1)
20 21
E M
N L
P To
pi c
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k Y
el p’
s bu
si ne
ss es
,r ev
ie w s, an d us er da
ta re
vi ew
s ge
ne ri
c no
re st
au ra
nt re
vi ew
s L
in k
M SE
no ye
s -
no no
hi gh
17 H
yu n
et al
.( 20
22 )
20 22
E M
N L
P L
en gt
h ye
s R
L an
te -h
oc un
su pe
rv is
ed ye
s L
in k
D U
C 20
04 do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no ye
s flu
en cy
,fi de
lit y
ye s
ye s
hi gh
18 C
ha n
et al
.( 20
21 )
20 21
TA C
L L
en gt
h, E
nt ity , A bs tr ac tiv
ity ye
s R
L an
te -h
oc su
pe rv
is ed
ye s
L in
k C
N N
D M
,N E
W SR
O O
M ,D
U C
-2 00
2 do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
,B E
R T
SC O
R E , M ov eS co re ,Q A
-F ye
s ye
s flu
en cy
,e nt
ity re
le va
nc e,
fa ith
fu ln
es s
ye s
no hi
gh
19 K
ri sh
na an
d Sr
in iv
as an
(2 01
8) 20
18 N
A A
C L
-H LT
To pi
c no
en co
de r-
de co
de r
pr e-
ho c
su pe
rv is
ed no
- C
N N
D M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
ye s
to pi
c re
le va
nc e
no no
no ts
ur e
20 K
ri sh
na et
al .(
20 18
) 20
18 A
C L
To pi
c no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- C
N N
D M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
ye s
qu al
ity ye
s no
no ts
ur e
21 R
us h
et al
.( 20
15 )
20 15
E M
N L
P L
en gt
h no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- N
Y T,
D U
C 20
04 do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no no
no no
no hi
gh
22 Fé
vr y
an d
Ph an
g (2
01 8)
20 18
C oN
L L
L en
gt h
no en
co de
rde
co de
r an
te -h
oc un
su pe
rv is
ed ye
s L
in k
G ig
aw or
d do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no ye
s gr
am m
at ic
al ity , pr es er vi ng or ig
in al
in fo
rm at
io n
no no
hi gh
23 L
ie ta
l. (2
01 8)
20 18
N A
A C
L -H
LT Sa
lie nc
y no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- C
N N
D M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
no no
no no
no ts
ur e
24 Ji
n et
al .(
20 20
b) 20
20 A
A A
I C
ov er
ag e
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
no -
G ig
aw or
d, D
U C
20 04 , M SR da ta se
t do
cu m
en t
ge ne
ri c
no ne
w s
L in
k R
O U
G E
W or
d M
ov er
’s D
is ta
nc e,
B er
tS co
re ye
s fa
ith fu
ln es
s, in
fo rm
at iv
en es s, flu en cy
no no
no ts
ur e
25 L
iu et
al .(
20 22
a) 20
22 N
eu rI
PS L
en gt
h no
N on
A ut
o R
eg re
ss iv
e M
od el
an te
-h oc
su pe
rv is
ed ye
s L
in k
G ig
aw or
d, D
U C
20 04
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
ye s
ov er
al lq
ua lit y, co m pl et en es s/
flu en
cy no
ye s
m ed
iu m
26 K
w on
et al
.( 20
23 )
20 23
E A
C L
L en
gt h
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
no -
C N
N D
M ,N
Y T,
W ik
iH ow
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E ,L
en gt
h V
ar ia
nc e
no ye
s in
fo rm
at iv
en es s, co nc is en es
s no
ye s
no ts
ur e
27 Pa
gn on
ie ta
l. (2
02 3)
20 23
A C
L E
nt ity
,S al
ie nc
y ye
s en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k Q
M Su
m an
d SQ
uA L
IT Y
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E ,B
E R
T SC
O R
E ,
le ve
ns ht
ei n
di st
an ce
no ye
s un
am bi
gu ity
,c on
ci se
ne ss
no ye
s hi
gh
28 H
of m
an n-
C oy
le et
al .(
20 22
) 20
22 A
A C
L E
nt ity
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
no -
E nt
Su m
do cu
m en
t ge
ne ri
c no
ne w
s no
F1 no
no no
no no
no ts
ur e
29 L
in et
al .(
20 22
) 20
22 A
C L
R ol
e no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
C SD
S, M
C di
al og
ue s
ge ne
ri c
no cu
st om
er se
rv ic
e da
ta ,
m ed
ic al
in qu
ir ie
s L
in k
R O
U G
E ,B
er tS
co re , B L E U ,M ov er Sc or
e no
ye s
in fo
rm at
iv en
es s,
no n-
re du
nd an
cy ,fl
ue nc
y ye
s no
hi gh
30 Z
ho ng
et al
.( 20
22 )
20 22
E M
N L
P C
ov er
ag e
ye s
en co
de r-
de co
de r
an te
-h oc
un su
pe rv
is ed
no -
G ra
nu D
U C
,M ul
tiN ew s, D U C 20 04 ,A rx
iv do
cu m
en t
ge ne
ri c
ye s
ne w
s, sc
ie nt
ifi c
pa pe
rs L
in k
R O
U G
E B
E R
T Sc
or e
ye s
flu en
cy ,
re le
va nc
e, fa
ith fu
ln es
s no
ye s
m ed
iu m
31 L
ia ng
et al
.( 20
22 )
20 22
A A
C L
R ol
e ye
s en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
no -
C SD
S, M
C di
al og
ue s
ge ne
ri c
no cu
st om
er se
rv ic
e da
ta ,
m ed
ic al
in qu
ir ie
s L
in k
R O
U G
E ,B
L E
U ,
B E
R T
Sc or
e, M
ov er
Sc or
e no
no no
no no
no ts
ur e
32 L
iu an
d C
he n
(2 02
1) 20
21 E
M N
L P
E nt
ity no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
L in
k SA
M Su
m di
al og
ue s
ge ne
ri c
no so
ci al
m ed
ia no
R O
U G
E no
ye s
fa ct
ua lc
on si
st en
cy ,
in fo
rm at
iv en
es s
no no
m ed
iu m
33 Z
he ng
et al
.( 20
20 )
20 20
C O
L IN
G E
nt ity
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k G
ig aw
or d,
D U
C 20
04 do
cu m
en t
ge ne
ri c
no ne
w s
no B
E R
TR
E O
,R O
U G
E B
E R
TR
E O
ye s
in fo
rm at
iv en
es s,
gr am
m at
ic al
ity ,
co he
re nc
e no
no hi
gh
34 H
ua ng
et al
.( 20
23 )
20 23
E A
C L
C ov
er ag
e no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
D IA
L O
G -S
U M
,S A
M SU
M di
al og
ue s
ge ne
ri c
no so
ci al
m ed
ia no
B A
R T
Sc or
e, R
O U
G E , Fa ct C
C no
ye s
re ca
ll, pr
ec is
io n,
fa ith
fu ln
es s
ye s
ye s
hi gh
35 M
uk he
rj ee
et al
.( 20
22 )
20 22
A A
C L
To pi
c no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
M SM
O do
cu m
en t
se m
iau
to m
at ic
ye s
ne w
s L
in k
R O
U G
E no
ye s
C ov
er ag
e, G
ra m
m ar , To pi cA w ar eTe xt
no ye
s hi
gh
36 Ji
n et
al .(
20 20
a) 20
20 A
C L
St yl
e ye
s en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k N
Y T,
C N
N do
cu m
en t
ge ne
ri c
no ne
w s
no B
L E
U ,R
O U
G E , M E T E O R ,C ID E
r no
ye s
re le
va nc
e, at
tr ac
tiv en
es s,
la ng
ua ge
flu en
cy ,
st yl
e st
re ng
th no
no hi
gh
37 Se
e et
al .(
20 17
) 20
17 A
C L
A bs
tr ac
tiv ity
,C ov
er ag
e ye
s en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k C
N N
D M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
no no
no no
hi gh
38 K
ry śc
iń sk
ie ta
l. (2
01 8)
20 18
E M
N L
P A
bs tr
ac tiv
ity no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- C
N N
D M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
ye s
re ad
ib ili
ty ,r
el ev
an ce
no no
no ts
ur e
39 B
ah ra
in ia
n et
al .(
20 21
) 20
21 A
C M
To pi
c no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
C N
N D
M ,A
M I,
IC SI
,A D
SE do
cu m
en t
ge ne
ri c
no ne
w s.
,m ee
tin gs
no R
O U
G E
no ye
s in
fo rm
at iv
en es
s, re
ad ab
ili ty
ye s
no hi gh 40 K ik uc hi et al .( 20 16 ) 20 16 E M N L P L en gt h no en co de rde co de r an te -h oc ,p os tho c su pe rv is ed ye s L in k D U C 20 04 ,G ig aw or d do cu m en t ge ne ri c no ne w s no R O U G E no no no no no hi gh 41 L iu et al .( 20 18 ) 20 18 E M N L P L en gt h no en co de rde co de r an te -h oc su pe rv is ed ye s L in k C N N D M ,D M Q A do cu m en t ge ne ri c no ne w s no R O U G E , VA R ,S im ila ri ty no no no no no hi gh 42 M ak in o et al .( 20 19 ) 20 19 A C L L en gt h no en co de rde co de r an te -h oc su pe rv is ed no - C N N D M ,M ai ni ch i do cu m en t ge ne ri c no ne w s no R O U G E no ye s po st -e di tin gtim e no no no ts
ur e
43 Y
u et
al .(
20 21
) 20
21 A
C L
L en
gt h
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k C
N N
D M
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
ye s
C or
re ct
ne ss
,c om
pl et
en es s flu en cy
no no
hi gh
44 Sa
rk he
le ta
l. (2
02 0)
20 20
C O
L IN
G L
en gt
h no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed no
- M
SR N
ar ra
tiv es , T hi nk in gM ac
hi ne
s do
cu m
en t
ge ne
ri c
no so
ci al
m ed
ia ,
sc ie
nc e/
ed uc
at io
n no
R O
U G
E ,M
E T
E O
R no
ye s
pr es
en ce
of ke
y fa
ct s
no no
no ts
ur e
45 H
e et
al .(
20 22
) 20
22 E
M N
L P
L en
gt h,
E nt
ity ye
s en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
ye s
L in
k C
N N
D M
,a rX
iv ,B
IG PA
T E
N T
do cu
m en
t ge
ne ri
c no
ne w
s, sc
ie nt
ifi c
da ta
no R
O U
G E
no ye
s C
on tr
ol A
cc ur
ac y,
C on
tr ol
R el
ev an
ce no
ye s
hi gh
46 Ta
n et
al .(
20 20
) 20
20 E
M N
L P
To pi
c no
en co
de r-
de co
de r
an te
-h oc
su pe
rv is
ed ye
s L
in k
C N
N D
M ,M
A N
ew s
do cu
m en
t ge
ne ri
c no
ne w
s no
R O
U G
E no
ye s
A cc
ur ac
y ,I
nf or
m at
iv en
es s,
Fl ue
nc y
ye s
no hi
gh
47 N
ar ay
an et
al .(
20 22
) 20
22 A
C L
D iv
er si
ty no
en co
de r-
de co
de r
po st
-h oc
su pe
rv is
ed ye
s L
in k
C N
N D
M ,X
Su m , SQ uA D
) do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
,B er
tS co
re ,
SE L
FB
L E
U no
ye s
fa ith
fu ln
es s
an d
di ve
rs ity
no no
m ed
iu m
48 Z
ho ng
an d
L itm
an (2
02 3)
20 23
IJ C
N L
PA
A C
L St
ru ct
ur e
no en
co de
rde
co de
r an
te -h
oc su
pe rv
is ed
no -
C an
L II
do cu
m en
t ge
ne ri
c no
le ga
l no
R O
U G
E ,B
E R
T Sc
or e,
st ru
ct ur
e si
m ila
ri ty
. no
ye s
co he
re nc
e, co
ve ra
ge no
ye s
no ts
ur e
49 Su
ha ra
et al
.( 20
20 )
20 20
A C
L To
pi c
ye s
ex tr
ac to
r+ ge
ne ra
to r(
de co
de r)
an te
-h oc
un su
pe rv
is ed
ye s
L in
k H
ot el
,Y el
p re
vi ew
s ge
ne ri
c no
re vi
ew s
no R
O U
G E
no ye
s in
fo rm
at iv
en es
s, co
he re
nc e,
no n-
re du
nd an
cy ,
co nt
en t-
su pp
or t
no no
hi gh
50 R
ib ei
ro et
al .(
20 23
) 20
23 E
M N
L P
St yl
e no
en co
de r-
de co
de r
pr e-
ho c,
an te
-h oc
su pe
rv is
ed ye
s L
in k
C N
N D
M do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
,B E
R T
Sc or e, FR E ,G FI ,C L I
ye s
ye s
m os
tr ea
da bl
e an
d le
as tr
ea da
bl e
no ye
s hi
gh
51 N
al la
pa ti
et al
.( 20
17 )
20 17
A A
A I
A bs
tr ac
tiv ity
,S al
ie nc
y ye
s G
R U
-R N
N an
te -h
oc su
pe rv
is ed
ye s
L in
k C
N N
/D M
,D U
C 20
02 do
cu m
en t
ge ne
ri c
no ne
w s
no R
O U
G E
no no
no no
no hi gh 52 H ito m ie ta l. (2 01 9) 20 19 IN L G L en gt h no en co de rde co de r pr eho c, an te -h oc su pe rv is ed no - JA M U S do cu m en t ge ne ri c ye s ne w s L in k R O U G E no no no no no m
ed iu m 53 Sh en et al .( 20 22 c) 20 22 N IP S C ov er ag e no en co de rde co de r pr eho c su pe rv is ed ye s L in k M ul tiL ex Su m do cu m en t H um an -a nn ot at ed ye s L eg al L in k R O U G E ,B E R T SC O R E no ye s R at e th e ge ne ra tio n (0 -3 sc al e) no ye s hi gh 54 M ad de la et al .( 20 22 ) 20 22 A C L E nt ity no en co de r, en co de rde co de r pr eho c, an te -h oc su pe rv is ed ye s L in k C N N D M ,N Y T do cu m en t ge ne ri c ye s ne w s L in k R O U G E ,B E R T Sc or e no no no no no hi gh 55 B ah ra in ia n et al .( 20 22 ) 20 22 A C L To pi c no en co de rde co de r, de co de r - su pe rv is ed no - N E W T S do cu m en t H um an -a nn ot at ed ye s ne w s L in k R O U G E no ye s B in ay ev al ua tio n ye s no hi gh 56 H ay as hi et al .( 20 21 ) 20 21 TA C L To pi c no en co de r, te xt R an k pr eho c su pe rv is ed ye s L in k W ik iA sp W ik ip ed ia A ut om at ic ye s W ik ip ed ia L in k R O U G E no ye s no no no hi gh 57 L uo et al .( 20 22 ) 20 22 E M N L P St yl e no en co de rde co de r pr eho c su pe rv is ed no - T S an d PL S do cu m en t A ut om at ic ye s Sc ie nt ifi c L in k R O U G E ,F K G , C L I,A R I, R N T PC ye s ye s R el ev an ce ,G ra m m ar , C oh er en ce no ye s hi gh 58 A hu ja et al .( 20 22 ) 20 22 A C L To pi c no en co de r, en co de rde co de r pr eho c su pe rv is ed ye s L in k A SP E C T N E W S do cu m en t A ut om at ic ye s ne w s L in k R O U G E no no no no no M ed iu m 59 M uk he rj ee et al .( 20 20 ) 20 20 SI G IR To pi c no G ra ph -b as ed an te -h oc U ns up er vi se d ye s L in k To ur is m R ev ie w s do cu m en t A ut om at ic ye s To ur is m L in k R O U G E no ye s as pe ct -c ov er ag e, re ad ab ili ty , di ve rs ity no no hi gh 60 L in et al .( 20 21 ) 20 21 E M N L P R ol e no al l pr eho c su pe rv is ed ye s L in k C SD S re vi ew s H um an -a nn ot at ed ye s re vi ew s L in k R O U G E ,B E R T Sc or e no ye s In fo rm at iv en es s, no nre du nd an cy , flu en cy ,m at ch in g ra te ye s no hi gh 61 Sh en et al .( 20 22 b) 20 22 A C L St ru ct ur e no en co de rde co de r pr eho c su pe rv is ed ye s L in k M R eD re vi ew s H um an -a nn ot at ed ye s Sc ie nt ifi c L in k R O U G E no ye s Fl ue nc y, co nt en tr el ev an ce , st ru ct ur e, de ci si on co rr ec tn es s no no hi gh
Ta bl
e 10
:M as
te rs
ur ve
y ta
bl e
(* m
ar ke
d fie
ld s
ar e
fil le
d to
be st
of ou
ru nd
er st
an di
ng ba
se d
on th
e av
ai la
bl e
in fo
rm at
io n)
.

heading: Future Work
text: As a future research direction, it’s essential to design models that consider other important combinations of control attributes, such as length, style, and saliency. Furthermore, creating standardized multi-CA benchmarks is crucial to facilitate the evaluations. Reproducibility. In the detailed analysis outlined in Table 10, we note that 35% of research studies do not share their code publicly. Furthermore, 25% of the papers did not carry out any human evaluation, and among the remaining studies, 79% did not conduct Inter Annotator Agreement (IAA) assessments. The lack of reproducibility (Ito et al. , 2023; Gao et al. , 2023; Iskender et al. , 2021) measures hinders the scientific community’s ability to validate and build upon existing work. On the other hand, the human study component should be a must for a text summarization evaluation scheme, otherwise, we are potentially overlooking essential aspects of real-world applicability. Standing on the shoulders of LLMs. The rise and success of large language models (LLMs) have opened up unparalleled possibilities for leveraging their capabilities across diverse stages of the Natural Language Processing (NLP) pipeline. In the context of CTS, LLMs can be fine-tuned to grasp context-specific nuances about CAs without the need for a dedicated training set. Additionally, when it comes to evaluating CTS models, LLMs
can serve as effective substitutes for human experts or judges (similar to Liu et al. (2023)), offering an efficient method for assessing performance