Active Learning for Natural Language Generation
Authors: Yotam Perlitz
Ariel Gera
Michal Shmueli-Scheuer
Dafna Sheinwald
Noam Slonim
Liat Ein-Dor
Venue: EMNLP 2023 Main (2023)
Keywords: active learning
NLG
generation
TL;DR: A systematic empirical study of applying active learning to NLG tasks

Abstract: The field of Natural Language Generation (NLG) suffers from a severe shortage of labeled data due to the extremely expensive and time-consuming process involved in manual annotation. A natural approach for coping with this problem is active learning (AL), a well-known machine learning technique for improving annotation efficiency by selectively choosing the most informative examples to label. However, while AL has been well-researched in the context of text classification, its application to NLG remains largely unexplored. In this paper, we present a first systematic study of active learning for NLG, considering a diverse set of tasks and multiple leading selection strategies, and harnessing a strong instruction-tuned model. 
Our results indicate that the performance of existing AL strategies is inconsistent, surpassing the baseline of random example selection in some cases but not in others. We highlight some notable differences between the classification and generation scenarios, and analyze the selection behaviors of existing AL strategies. Our findings motivate exploring novel approaches for applying AL to generation tasks.

Active Learning for Natural Language Generation
Yotam Perlitz', Ariel Gera', Michal Shmueli-Scheuer,
Dafna Sheinwald, Noam Slonim, Liat Ein-Dor
IBM Research AI poolam.perritz,ariie geral @ @ibm.com,
@il.ibm.ccm
Abstract
Nev+Q4
The field of Natural Language Generation (NLG) suffers from severe shortage of labeled data due to extremely expensive and time- consuming process involved in manual annota- tion. A natural approach for coping with this problem is active learning (AL), well-known machine learning technique for improving notation efficiency by selectively choosing the most informative examples label. However, while AL has been well-researched in the con- text text classification, application NL. remains largely unexplored. In this paper, present first systematic study of active learn- ing NLG, considering diverse set tasks and multiple leading selection strategies, and harnessing strong instruction-tumel model. Our results indicate that the performance of existing AI. strategies is inconsistent, surpass- ing the baseline of random example selection in cases but not in others. We highlight some notable differences between the classifi- cation and generation scenarios, and analyze the selection behaviors of existing AL strate- gies. Our findings motivate exploring novel ap- proaches applying generation tasks.
MSCOCO
Datperfoome
ParabankV:
ratiooss
ParabankV2
Non
signficant
TL.0R
CHAFC E&M
GYAPC F&R
MC Strategy
Figure Statistical significance AI. benefits. plot depicts the results of paired Wilcoxon signed-rank tests of AL. performance, for each dataset-strategy com- bination. Cells marked in green indicate that the AL strategy significantly outperformed random selection for the dataset, with p < .05 after Bonferroni correction.
Major advances in the architecture and scale of machine learning models in general, and pre- trained language models in particular, have given rise to the emerging field of Natural Language Gen- eration (NLG) (Li et al., 2021; Dong et al., 2022). However, a major practical barrier in tackling NLG tasks is the shortage of annotated data, exacerbated by the increased burden of human annotation for such tasks. As paradigm for minimizing label- ing effort, AL is a natural avenue for coping with these challenges. Nevertheless, it has hardly been studied in the context of NLG problems.
1 Introduction
Active learning (AL) (Cohn et al., 1996) is well- known machine learning approach for reducing annotation effort, aiming to train models with less data by selecting the most informative examples label. This paradigm was introduced and devel- oped the context classification (Settles, 2009; Lewis and Gale, 1994), and has been successfully applied to machine learning problems from wide range of domains, including computer vision (Gal and Ghahramani, 2016; Sener and Savarese, 2018; Gissin and Shalev-Shwartz, 2019) and text classi- fication (Zhang et al., 2017; Siddhant and Lipton, 2018; Prabhu et al., 2019; Ein-Dor et al., 2020). "These authors contributed equally to this work.
In this work, we explore the AL paradigm as applied to NLG. Our aim twofold: first, to exam- ine how well AL strategies perform in NLG; and second, to gain insight into how these strategies operate in the unique context of text generation. To this end, we conduct an extensive set of experi- ments with leading AL strategies on various NLG tasks, accompanied by a comprehensive analysis of strategy behaviour. Our results reveal that these AL strategies do not perform consistently across different datasets and tasks (Figure 1), suggesting that new AL methods are required to bring value

---

this domain.
annotation effort on the most informative instances. The AL setting assumes access to a large amount of unlabeled data, and a limited annotation budget. The core idea of AL is that the current model can be leveraged to maximize the utility of the label- ing budget; thus, the goal of AL strategy is identify unlabeled examples whose annotation is expected to yield the largest performance improve- ment when used to train the model.
To the best of our knowledge, this is the first systematic study of AL for NLG tasks. Moreover, in this work we introduce strong instruction-tun -untd models into the AL paradigm. In outlining the behavior of existing strategies, we aim to lay the groundwork for future research, leading to effective use of AL in practical NLG scenarios.
2 Related Work
Differing approaches have been put forth for predicting given set of unlabeled examples which of those examples would be most beneficial as training examples for the model.
In the field of natural language processing, AL was mainly studied for text classification (Ein-Dor et al.. 2020; Schröder et al., 2021; Margatina et al., 2021). AL has also been successfully applied in the con- text of neural machine translation (NMT), where the focus is on low-resource pairs of languages (Zhao et al., 2020; Liu and Yu, 2023). Some works investigated strategies that are tailored specifically for NMT, such as using a backward translator to check round-trip translations (Haffari et al., 2009; Zeng et al., 2019) or quality estimation (Chimoto and Bassett, 2022). Zeng et al. (2019) conducted a systematic comparison of different AL methods in the context of NMT. Thus, do not include NMT in the present work and focus on NLG tasks that had not been systematically explored.
Broadly, much of the AL literature has focused on two general directions: Representativeness and Informativeness. Representativenes, methods fo- cus on the data distribution of the examples. As- suming that an informative set of training examples is one that accurately represents the overall pop- ulation of instances, they aim to seleet diverse and representative set of examples for labeling. Un- der the Informativeness umbrella, key focus has been uncertainty. In the uncertainty-based ap- proach, the core assumption is that examples for which the model is least certain are the most infor- mative for model training. Thus, uncertainty- strategies aim to estimate the uncertainty for each unlabeled example u, and to choose those with the highest model uncertainty.
There exists a wide variety of NLG tasks, and these have attracted much attention in recent years (Dong et al., 2022). Nevertheless, outside the context of NMT there are very few works that apply AL to NLG tasks (Zhang et al., 2022). Specifically, for summarization, Gidiotis and Tsoumakas (2021, 2022) propose Bayesian strategy in which they select samples label by optimizing the uncer- tainty using the Monte Carlo BLEU variance met- ric. More recently, Tsvigun et al. (2022) propose an embedding- based method and show improvements in certain summarization datasets. Paraphrase gen- eration with LSTMs was reported in Karaoguz (2018), where the authors use n-gram coverage measures their sampling strategy, aiming cap- turing the informativeness of source paraphrases. Thus, while there have been some focused stud- ies of AL in the context of a specific NLG task, in this work we aim for a more systematic and com- prehensive view, across multiple tasks and datasets.
3.2 AL in Generation vs. Classification
Text classification and text generation differ in many important aspects. Next, we consider these differences through the lens of AL strategies. One major difference is that, for most NLG tasks, there are multiple legitimate outputs for a given input text. For example, in paraphrase generation, there are many ways to rephrase a given sentence; the ability to generate a diverse set of outputs is in fact a desired attribute of the model.
Generally, in AL, a model's uncertainty about an example is considered a strong signal for informa- tiveness; the underlying assumption is that exam- ples where the model is uncertain are those where it is prone to error, and would thus derive the most benefit from supervision. However, uncertainty in an NLG scenario namely, situation where the model considers several outputs to be equally prob- able is not necessarily associated with errors, and may even reflect a desirable propensity to generate diverse generation outputs. Therefore, the family of
3 Background
3.1 The Active Learning Scenario
Active learning is an iterative process that aims to reduce labeling costs, by focusing the human

---

uncertainty-based active learning strategies, consid- ered a top strategy for classification (Schröder et al., 2021), may not be directly applicable to NLG. Another fundamental difference between clas- sification and generation is in the dimensionality of the prediction space. In classification tasks, the number of classes is typically small, ranging from two classes, e.g., in the case of spam detection, up few hundreds for intent detection tasks. In contrast, NLG, the number of possible "classes" in predicting the next token is the vocabulary size which is typically O(10*) and correspondingly the number of options to select from when gener- ating sequence of tokens is exponentially large. The dimension of the prediction space is crucial in the case of expected model change strategies, which aim directly estimate the effect instance would have on the model. While classification a strategy like Expected Gradient Length (Huang et al., 2016) can compute the expected gradient norms over the posterior distribution of labels, the very large dimension of the prediction space in gen- eration tasks makes this computation intractable.
domly sampled from the training set of D. Altogether, we report results of 18 AL steps be- tween 0 and 1000 labeled examples: 10 batches of 20, followed by 8 batches of 100. As our focus is on practical scenario of small annotation budgets, opted to sample the first iterations (i.e., 0 - 200 training examples) more densely, to gain a detailed view of the behavior of AL in this area.
4.2 Base Model
To represent a practical scenario of applying AL over strong pretrained models, we use the instruction-tune
FLAN-T5 Large' as the basc model for our experiments. This model was trained on wide range of language tasks, including many NLG tasks, and has been shown to exhibit better performance and faster convergence in downstream task fine-tuning (Longpre al., 2023). As this base model was trained using instruction-tuning (Wei et al., 2022), we formulate each NLG task as a sim- ple natural language instruction that given to the model. The instruction prompts for each task are listed in Appendix A.2.
To ensure appropriate simulation of the AL process, we only experiment datasets that were not included in the FLAN-T5 training data?.
4 Experimental Setup
This work aims to systematically study the appli- cation of active learning to NLG. To this end, we conduct a comprehensive set of experiments, ex- ploring the behavior of the different families of AL strategies across wide range of NLG tasks.
Note that the use of a strong base model with zero-shot capabilities allows for starting the AL process from an empty labeled data pool LD. This is unlike the traditional AL setup, where a mundomly-selecter seed Lp is required to jump- start the process.
4.1 Active Learning Setup
We use the pool-based active learning (Settles, 2009) variant, in batch mode. At the beginning of the active learning process for a dataset D, we start with a pre-trained base model Mo. a pool of unlabeled data Up and an empty pool of labeled data LD.
4.3
Tasks and Datasets
We consider four prominent tasks in NLG: para- phrase generation, style transfer (formality), sum- marization, and question generation. We chose 2 or 3 representative datasets for each task. As men- tioned above, we avoid datasets that were used to fine-tune FLAN.
At each active learning step i, the AL strategy selects batch of n, samples from UD for labeling; these samples removed from UD and added to the labeled data pool Lp, along with their ground- truth labels. Then, the base model Mo fine-tuned on the labeled samples in LD. i.e., on all the data labeled thus far, creating a new model M,.
The datasets for each task are listed in Table 1, and a full description of the datasets can be found in Appendix A.I.
4.4 Active Learning Strategies
This process is repeated N times, where at each step the AL strategy can utilize the predictions over the unlabeled pool of the previous model M,-1, in order to select the next batch of examples.
We test a representative group of AL strategies, covering different data acquisition approaches. Fol- lowing Zhang et al. (2022), we divide AL query
'nttps://hugginggacc http:/faaacccccooolgeeeee /oggoogllelleentt--lrgg chttps://gitthi thub. com/ htt.///oglleeereeer//////aaaaaaa inw.eetttooonoo.css.
For runtime considerations, we restrict the size of the unlabeled pool Up to 10K examples ran-

---

Task
Datasets
Metric
Mean Token Entropy (MTE) selects instances the model is least certain about, according the max-entropy decision rule. For NLG, the notion of max-entropy is expanded by taking the mean over the entropies of each generated token, as in Zhao et al. (2020).
Paraphrase
MSCOCO,
iBLEU
Generation
Purabank v1.0/2.00
Summarizutio
DebateSUM, Reddit TEDDR
ROUGE-
Question Gen.
NewsQA, MLQA
BLEU
Formality
G&M, GYAFC-F&R
G-Score
Table I: Datasets and evaluation metries.
Monte Carlo Dropout (MC Dropout) selects instances the model is least certain about, by har- nessing model stochasticity (Gal and Ghahramani. 2016). For NLG, instance uncertainty estimated using Monte Carlo BLEU variance (Gidiotis and Tsoumakas, 2021; Xiao et al., 2020). In this ap- proach, after stochasticity is induced by activating dropout, the uncertainty of specific sample is es- timated by how different its generated outputs are in terms of their BLEU score.
strategies into two broad categories representa- tiveness and informativeness strategies.
Where necessary, we adapt the strategy imple- mentation to accommodate the NLG scenario. Note that certain types of AL strategies are inherently un- suitable for NLG. Specifically, gradient-bate -baeedd--aadddeeee AL methods like EGL (Huang et al., 2016) or BADGE (Ash et al., 2020) cannot be straightforwardly ap- plied to NLG, due to the sequential nature of NLGG
4.5 Evaluation Metries
4.4.1 Representativeness Strategies
We use standard automatic metries to evaluate the tasks, summarized in Table 1. For paraphrase generation we use iBLEU (Sun and Zhou, 2012); for summarization and question generation, we use ROUGE-L (Lin and Och, 2004), and BLEU (Pap- ineni et al., 2002), respectively. To evaluate the for- mality transfer task, we follow Xu et al. (2018) and use G-Score, the geometric mean of the formality- score and BERTScore (Zhang et al., 2020) with the reference text; further details can be found in Appendix A.6.
For batch-mode AL, the AL variant we use in this study, the relevant representativenes: strategies are those that aim to optimize the diversity and repre- sentativeness of the selected batch. We take greedy Core-Set and IDDS as examples of this family.
Core-Set (Sener and Savarese, 2018) aims to increase representativeness by selecting instances with maximal distance from the labeled pool. We follow the greedy method described in Sener and Savarese (2018) 8). As in our scenario start from empty labeled data pool, for the first AL step we modify this strategy to enable it to be applied without labeled data. For details see Appendix A.5. In-Domain Diversity Sampling (IDDS) aims to select diverse instances while avoiding outliers (Tsvigun et al., 2022). IDDS scores for an example strike balance between large mean distance of the example from the instances in the labeled pool, and a small mean distance from those in the unlabeled pool.
4.6 Implementation Details
We base training and inference implementation on Hugging Face Transformers (Wolf al., 2019) v4.26 and pytorch (Paszke al., 2019) v2.0. Each experiment was repeated times, with each rep- etition using a different random seed on a single NVIDIA A A100 GPU. Thus, we performed a total of 4050 training and inference runs (9 datasets x 5 strategies 18 iterations x 5 repetitions) for the main experimental results.
The above strategies rely on vector representa- tions for each instance; following Ni et al. (2022), calculate these representations the hidden state of the last layer of the encoder, averaged across input tokens.
To keep the computation budget manageable, we opted for single base model and standard set of hyperparameter values. In each AL step, the base model was fine-tuned for 3 epochs over L.p, using the adafactor optimizer (Shazeer and Stern, 2018) with constant learning rate of 5 x 10-5, and train batch size of 8.
4.4.2 Informativeness Strategies
Informativeness strategies rank unlabeled exam- ples according measures that estimate example informativeness, where model uncertainty is often a proxy of informativeness. We take MTE as an ex- ample of an uncertainty sampling strategy, and MC Dropout which is disagreement-bar based strategy.
Results
For each dataset, we report performance metrics across 18 AL iterations i.e., the performance of

---

Paraphrase generation ParabankV1
Formality transfer GYAFC-F&R
21
0.8
22
0.7
0-Sc00e
strategy
strategy
10
Cuearsed
Cone-dd
MCDropout
MCDropout
16
0.4
Random
Random
14
0.3
300
100
400
Train Examples
Exumpless
Figure 2: AL performance for selected datasets. The lines depict the evaluation performance the different selection strategies along the AL process. Each line represents average (#: 95% Bootstrapped CI) over 5 experimental repetitions. In this plot we focus on the range of up to 500 labeled examples. Plots for all datasets, for the full range 1000 examples, are shown in Appendix A.3.
models fine-tuned on different amounts of labeled instances - comparing results between different active learning strategies.
all the strategies tested is strikingly similar to that of random selection. Granted, there are specific instances where a strategy outperforms random se- lection for a specific dataset (for example, see the MC Dropout strategy over the Reddit TL;DR data); however, we find no consistent pattern of benefits to using AL, even within specific NLG task.
In addition, we report the performance when using the baseline of Random selection randomly sampling the batch of examples to be added to the labeled data pool at each iteration.
Figure/ depicts AL results for two of the datasets tested, Parabank1.0 and GYAFC-F&R. As can be seen in the figure, for these datasets we do not see clear advantage to any of the AL strate- gies tested; moreover, the various AL strategies do not seem outperform the baseline of randomly selecting instances for labeling. These plots are representative of the overall pattern of results see across tasks and datasets, with none of the AL strategies convincingly overtaking the others across datascts. Individual plots for all the datasets tested are shown in Appendix A.3.
To better quantify these results, we perform a Wilcoxon signed-rank test, comparing each strat- egy to the random selection baseline (refer to Ap- pendix A.4 for details). Figure shows the results of the significance testing. As can be seen, none of the strategies exhibit a clear benefit for the tasks of paraphrase generation and formality transfer, and results for question generation and summarization are somewhat mixed. Notably, both Core-Set and MC Dropout fail to provide benefits across more than one dataset.
A recurring pattern in the results is that most of the performance gains from supervision occur in the first few AL iterations, and times even in the first AL iteration, where the model is trained just 20 labeled instances. Thus, appears the FLAN base model needs only a small number of examples to learn the basic premise of the target generation task; while larger amounts of labeled data are of course beneficial, the few-shot performance of this model across datasets is quite strong.
6 Analysis
Given the failure to systematically outperform the random baseline, this section we examine the dif- ferent AL strategies by the properties of the batches they select, aiming to gain insights into how they operate. In line with the two families of strate- gies survey, analyses focus notions of representativennss ($6.1) and uncertainty ($6.2).
We perform a comparative analysis of the strate- gies, using the random strategy as a reference point. This analysis also serves as sanity check, to vali- date that the properties exhibited by the different strategies aligned with their expected behavior. To ensure all strategies are compared with the
In Figure 3 present the full distribution of the performance of the various AL strategies relative the random selection baseline, for each of the NLG tasks. Overall, across tasks, datasets, AL iter- ations and experiment repetitions, the behavior of

---

Question generator:
Detasat
Paraphrese generation
Dataset
Retative Galin Kanndom 1%6 HE..1.
600
MEDAA
NewsOA
Relative Ceia Vs. Randon (%) 18L,B
MBcOCO
Puu buubk2
500
440
380
200
100
-2h
-50
-100
-7G
Drepont
Strategy
Summarization
Formality transfer
Relattiv Eaain Vs. Kandom (16) ROLGG:
Reddit
Relative Gein Vs. Rantom (%) C-Scoree
GAFGE&M
CAAPC PAR
-40
80
50
Tropoott
Mr Dropent
MTR
AL Strategy
Strategy
Figure 3: Dataset/Strategy Summary. The plots depict the relative gains of each strategy with respect to the random selection baseline. Gains are computed as the performance difference with respect to the zero-shot performance. Relative gains are computed the percentage change between the gains of the AL strategy and the gains of random selection. Each point represents the relutive gain between a given strategy and random selection for particular setting - i.c., at specific iteration and for a specific experimental repetition. Thus, for each datasct-stran stratcy combination 90 points shown (18 AL iterations repetitions). The distribution patterns reveal that although in some cases, a strategy might beat random selection, no strategy consistently outperforms the random baseline.
same initial conditions, this analysis performed solely for the first AL iteration, where all strate- gies rely on the same base model and the same unlabeled set Up. Each strategy makes its own selection of 100 examples² for labeling from Up.
for the representativeness strategies in $4.4.1.
Outliers: A known issue with AL strategies, particularly those focusing on uncertainty, is their tendency to select outliers that do not faithfully represent the overall data distribution. To mea- sure the severity of the outlier problem, we use the density in representation space of points in the selected batches. Specifically, following Ein-Dor et al. (2020), rely the KNN-density measure proposed by Zhu al. (2008), where the density of an instance is quantified by the average (Euclidean) distance between the instance in question and nearest neighbors within Up. We define the outlier- score of batch by the average KNN-density its instances (K 10), where high density corre- sponds to low outlier-score.
6.1 Diversity and Outliers
Two properties that are known in the literature to impact the effectiveness of AL strategies, at least in the context of classification, are the diversity and the propensity for outlicrs of the batches selected for annotation (Kee et al., 2018). Thus, examine these two properties in the selected batches.
For the purpose of analyzing these properties, we define the input example representation as the average of the input tokens' embeddings over the last encoder hidden state (Ni et al., 2022), done
Diversity: Choosing batch of diverse examples is often better than choosing one containing very similar and perhaps redundant examples. We define
"To obtain more accurate estimate of strategy hehavior, we use a larger initial batch size than for the results in $5.

---

NewsQA
+
random
Oracle
MSCOCO
x +
+
MIE
Drupout
IDDS Core-set
Reddit T2,DR
MLDA
Dataset
ParabankV2
ParabankV1
3++
DehateSum
GYAFC-F&R
CYAPC-E&M
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.30
0.35
0.40
0.45
0.50
0.55
0.60
Outlter Score
Diversity
Figure 4: Strategy selection characteristies. The plots depict the outlier score (left, lower is better) and diversity (right, higher is better) of the batches selected by the different AL strategies, as well as the Oracle strategy of $6.2.2.
the Diversity of a batch as the average Euclidean distance of its instances from the center.
beled examples with larger error rate; and second, that labeling examples with larger model errors is in fact more beneficial for training the model. The following analyses examine these two conditions.
The diversity and outlier-score of the different strategies are depicted in Figure 4. As expected, Core-Set, a batch-aware strategy, which was de- signed to increase diversity, characterized by also charac-
6.2.1 Selecting Error-prone Examples
In classification, there exists a clear association be- tween uncertainty and prediction errors. Here we examine whether similar association also holds in generation; and specifically, whether examples that are scored higher by the uncertainty strategies are associated with poor generated outputs, as mea- sured by the automatic evaluation metrics.
batches with the highest diversity. terized by the highest outlier score, in line with the tendency of the greedy version of Core-Set to select outliers (Sener and Savarese, 2018). In contrast, IDDS, which explicitly attempts to avoid outliers (Tsvigun et al., 2022), correspondingly has a low outlier-score, and also relatively low diversity. Meanwhile, the uncertainty strategies exhibit diver- sity and outlier scores that are closer to those of the random selection baseline, indicating that they do not suffer from severe diversity or outlier issues.
To this end, obtain the generation predictions of the base model for all the instances in the un- labeled pool UD. Then, we compute the average evaluation score of the model generations that cor- respond to instances in the selected batch B, and compare them to the average score over the en- tire unlabeled pool. The results of this analysis for the various AL strategies are presented in Fig- ure 5. As expected, batches selected by the MC Dropout uncertainty strategy are characterized by lower evaluation scores. The other uncertainty ap- proach, MTE, exhibits similar tendency but is less consistent across datasets.
6.2 Uncertainty and Model Performance
The major premise underlying uncertainty-basee AL that the information gained by labeling an example is higher if the current model's prediction on this example is erroneous. Thus, relying on the potential association between uncertainty and error rate, uncertainty-basse based strategies aim to find examples from the unlabeled data that are more prone to errors (Lewis and Gale, 1994).
Thus, there is some association between gener- ation performance and uncertainty. Nevertheless, there are no consistent performance gains from the uncertainty strategies.
The failure of the uncertainty-baat based strategies ex- amined here to consistently outperform the random baseline, raises the question if - and to what extent they applicable to the generation scenario. In order for the uncertainty approach succeed, two conditions must be fulfilled: First, that the strategies are able to identify subsets of the unla-
6.2.2 Are Error-prone Examples Informative? So far, we have established that the uncertainty strategies do tend to capture poor generation per- formance. However, in order for this to reflected

---

tasks. Through rigorous experimentation and anal- ysis, we have shown that no AL stralegy system- atically demonstrates a clear superiority over the random baseline terms of NLG quality.
Strategy Selections
Relative performance SD)
+1
Our findings indicate that despite the potential promises and advancements in active learning tech- niques, when it comes to NLG, the inherent com- plexity and diversity of human language poses sig- nificant challenges. AL strategies, which aim improve efficiency by actively selecting informa- tive data points for labeling, may not effectively capture the intricacies of language structure, seman- tics, and context required for generating coherent and meaningful text.
-1
Core set
IDDS MC Dropout
MTE
Oracle
Figure 5: Strategy selections by relative generation performance. The plot compares AL strategies in terms the relative model generation performance over batches selected by the strategy. Relative performance is defined as the difference between the average perfor- mance over the batch and the average performance over the full unlabeled pool Up, measured in standard devi- ations. Results are averaged across datasets. A large negative value indicates that the examples selected by the strategy are associated with poor model generation outputs.
Our results provide a wider and somewhat con- trasting perspective to previous works. While previ- ous papers had typically reported the effectiveness of AL on specific dataset and task, our compre- hensive analysis spanning multiple datasets and tasks suggests that the potential gains reported before are not very robust, hence do not show up a consistent manner. Thus, while our experiments were done on a single base model and hyperparam eter setting, they indicate that existing AL methods cannot be assumed to work out-of-the-box.
AL gains, the basic assumption that low perfor- mance examples are more useful to the model must be satisfied.
The failures of cxisting AL methods for NLG cannot easily be associated with a single underlying factor. More likely, there multiple issues play that violate some of the premises and assumptions of AL strategies that were designed with classifi- cation in mind. Some of these potential causes for instance, the complex relation between model uncertainty and erroneous outputs reflect a fun- damental difference between the tasks of classifi- cation and generation. Others, however, may be more question of degree. For instance, the out- put space for generation is overwhelmingly larger than that of a typical classification task, and is also characterized by a large degree of label imbalance, properties that may lead to difficulties capturing informative examples. Notably, similar issues exist in some multi-label classification tasks, which also exhibit difficulties with leveraging AL (Wang and Liu, 2023; Wertz et al., 2022).
To test the validity of this assumption more directly, we implement "illegal" AL strategy, named Oracle, that has direct access the evalua- tion metric scores with respect the ground-tran -ruth references, and selects the examples with the lowest evaluation scores (as seen in Fig. 5). If the afore- mentioned assumption is valid, Oracle is expected to yield optimal learning progress, as measured by the corresponding evaluation metric.
However, the results of this experiment, as shown in Appendix A.3, indicate that the Oracle strategy generally performs very poorly.
Thus, we see that the basic assumption uncer- tainty sampling - that labeling examples with poor model output will be most beneficial for improving the model does not necessarily hold in NLG.
To conclude, we have shown that uncertainty- based strategies are able to select prror-prone in- stances; However, even optimal selection is not guarantee of gains in model training, as demon- strated by the Oracle strategy.
In this work we combine AL with a strong instruction-tuned model, highlighting the impor- tance of the base model used. The behavior ob- served here, where much of the performance gain is achieved with a small number of training exam- ples, is encouraging with respect to the practical scenario of a limited labeling budget; at the same
7 Discussion
this work, we examined the effectiveness of var- ious active learning strategies across multiple NLG

---

time, this may entail new AL approaches that fo- cus on small batches or on bringing value to the selection of a single batch of few-shot instances, In our view, the takeaway from our results is not that the paradigm of active learning should be aban- doned. The goal of reducing manual annotation efforts remains as relevant as ever, all the more so for the particularly expensive annotation process associated with NLG. Rather, our hope is that these results will stimulate new efforts in devising novel AL strategies and approaches, ones that are specif- ically designed for the NLG scenario, and suited for strong instruction-tuned base models.
the AL experiments are performed on fully-labeled datasets, here we only look the selection of ex- amples for labeling, and not at the labeling process itself. Specifically, it is plausible that the exam- ples selected by an AL strategy would prove to be more difficult for a human annotator, and/or that different labelers will write very different outputs for the selected instances. Such questions, which are highly relevant for NLG,, are beyond the scope of this work.
In this study we report AL behavior for more practical labeling budgets, ranging between 20 and 1000 training examples. The effects of AL strate- gies when working with larger scales of labeled NLG data may be quite different than the pattern of results shown here.
Ethics and Broader Impact
This paper is submitted in the wake of a tragic terrorist attack perpetrated by Hamas, which has left our nation profoundly devastated. On October 7, 2023, thousands of Hamas terrorists infiltrated the Israeli border, launching a brutal assault 22 Israeli villages. They methodically moved from home to home brutally torturing and murdering more than a thousand innocent lives, spanning from infants to the elderly. In addition to this horrifying loss of life, hundreds of civilians were abducted and taken to Gaza. The families of these abductees have been left in agonizing uncertainty, no infor- mation, not even the status of their loved ones, has been disclosed by Hamas.
Finally, is common in NLG, we rely on au- tomatic metries to evaluate model performance. While these metrics are likely correlated to human judgements of task performance, the metrics may suffer from various artifacts and biases, and thus provide only a partial window into the true model performance.
References
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy. John Langford, and Alekh Agarwal. 2020. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learn- ing Representations.
We fervently call for the immediate release of all those who have been taken hostage and urge the academic community to unite in condemnation of these unspeakable atrocities committed by Hamas. We call all to join us in advocating for the prompt and safe return of the abductees, we stand to- gether in the pursuit of justice and peace.
Everlyn Asiko Chimoto and Bruce A Bassett. 2022. COMET-QE and active learning for low-resource machine translation. arXiv:2210.15600
David A Cohn, Zoubin Ghahramani, and Michael Jordan. 1996. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145 145.
Limitations
Chenhe Dong. Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2022. A survey of natural language generation. ACM Computing Surveys, 55(8):1-38.
Generally, there are some inherent gaps between AL experiments such as those conducted here and the ultimate goal of achieving label efficiency with a human in the loop. As outlined in Margatina and Aletras (2023), prominent gaps between aca- demic AL studies and practical AL applications include the potential effects of differing dataset qualitics, as well as of temporal drifts in data distri- butions, that are characteristic of real-world data; additionally, while practitioners may pursue hyper- parameter tuning for their trained model, this is not feasible in the context of a large AL study like the present work. Perhaps most crucially, given that
Qingxiu Dong, Xiaojun Wan, and Yue Cao. 2021. ParaSCI: large scientific paraphrase dataset for longer paraphrase generation. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume. pages 424-434, Online. Association for Computa- tional Linguistics.
Liat Ein-Dor. Alon Halfon, Ariel Gera, Eyal Shnarch. Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim, 2020. Active Learning for BERT: An Empirical Study. Proceedingy of the 2020 Conference on Empirical

---

Methods in Natural Language Processing (EMNLP), pages 7949-7962 Online. Association for Computa- tional Linguisties.
Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evalu- ating cross-lingull extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7315- 7330, Online. Association for Computational Lin- guistics.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation Representing model un- certainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learn- ing. volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA. PMLR.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji- ti-Rong Wen. 2021. Pretrained language model for text gener- ation: A survey. In Proceedings of the Thirtieth Inter- national Joint Conference on Artificial Intelligence, IJCAI-21, pages 4492-4499, International Joint Con- ferences on Artificial Intelligence Organization. Sur- vey Track.
Alexios Gidiotis and Grigorios Tsoumakas. 2021. Bayesian active summarization. arXiv:2110004400
Alexios Gidiotis and Grigorios Tsoumakas. 2022. Should we trust this summary? Bayesian abstrac- tive summarization to the rescue. In Findings of the Association for Computational Linguistics: ACL 2022, pages 41 19-4131, Dublin, Ireland. Association for Computational Linguistics.
Chin-Yew Yew Lin and Franz. Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and skip-bigram statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- 04), pages 605-612, Barcelona, Spain.
Daniel Gissin and Shai Shalev-Shwart., 2019. Discrim- inative active learning. arXiv:1907.00634.
Tsung- Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L.. Zit- nick. 2014. Microsoft COCO: Common objects in context. In ECCV.
Gholamreza Haffari, Maxim Roy, and Anoop Sarkar. 2009. Active learning for statistical phrase-based machine translation. In Proceedings of Human Lan- guage Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 415-423.
Chuanming Liu and Jingqi Yu. 2023. Uncertainty-awee non-autoregressive neural machine translation. Com- puter Speech & Language, 78:101444.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. 2023. The FLAN collection: Designing data and methods for effective instruction tuning. arXiv:2301.1366
Edward Hu, Rachel Rudinger, Matt Post, and Ben- jamin Van Durme. 2019a. Parabank: Monolingual bitext generation and sentential paraphrasing via constriied neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6521-6528.
Katerina Margatina and Nikolaos Aletras. 2023. On the limitations of simulating active learning.
J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. 2019b. Large-scale, diverse, paraphrastic bitexts via sampling and clus- tering. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). pages 44-54, Hong Kong, China. Association for Computational Linguistics.
arXiv:2305.13342
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650-663, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Jiaji Huang, Rewon Child, Vinay Rao, Hairong Liu, Sanjeev Satheesh, and Adam Coates. 2016. Active learning for speech recognition: the power of gradi- ents. arXiv: 1612.0222..
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfci Yang. 2022. Sentence-TS: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022. pages 1864-1874, Dublin, Ireland. Association for Computational Linguistics.
Ethem Can Karaoguz. 2018. Adaptive learning strate- gies for neural paraphrase generation. Master's the- sis, Universität Hamburg.
Seho Kee, Enrique Del Castillo, and George Runger. 2018. Doory-by-committee improvement with diver- sity and density in batch active learning. Information Sciences, 454:401-418.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR'94 94, pages 3-12. Springer.

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems,
Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Proceed- ings of the 50th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 38-42, Jeju Island, Korea. Association for Computational Linguistics.
Ameya Prabhu, Charles Dognin, and Maneesh Singh. 2019. Sampling bias in deep active classification: An empirical study. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pages 4058-40666 Hong Kong, China, Association for Computational I Linguistics.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehen- sion dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200 Vancouver, Canada. Association for Computational Linguistics.
Akim Tsvigun, Ivan Lysenko, Danila Sedashov, Ivan Lazichny, Eldar Damirov, Vladimir Karlov, Artemy Belousov, Leonid Sanochkin, Maxim Panov, Alexan- der Panchenko, Mikhail Burtsev, and Artem Shel- manov. 2022. Active learning for abstractive text summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5128-5152, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguisties.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, bench- marks and metries for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen, Denmark. Association for Com- putational Linguistics.
Allen Roush and Arvind Balaji. 2020. DebateSum:
A large-scale argument mining and summarization dataset. In Proceedings of the 7th Workshop on Ar- gument Mining. pages 1-7, Online. Association for Computational Linguistics.
Christopher Schröder, Andreas Niekler, and Martin Potthast. 2021. Revisiting uncertainty-based query strategies for active learning with transformers. arXiv:2107.05687.
Mengqi Wang and Ming Liu. 2023. An empirical study active learning for multi-label text classification. In The Fourth Workshop on Insights from Negative Results in NLP, pages 94-102, Dubrovnik, Croatia. Association for Computational Linguisties.
Ozan Sener and Silvio Savarese. 2018. Active learn- ing for convolutional neural networks: A core-set approach. In International Conference on Learning Representations.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Burr Settles. 2009. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences.
Lukas Wertz, Katsiaryna Mirylenka, Jonas Kuhn, and Jasmina Bogojeska. 2022. Investigating active learn ing sampling strategies for extreme multi label text classification. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 4597-4605, Marseille, France. European Language Resources Association.
Noam Shazeer and Mitchell Stern. 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, PMLR.
pages
Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language pro- cessing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 2904-2909, Brussels, Belgium. Association for Com- putational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2019. Huggingface's 's transformers: State-of-the the- art natural language processing. arXiv:1910.0377
Nisan Stiennon. Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
Tim Z Xiao, Aidan N Gomez, and Yarin Gal. 2020. Wat zei je? detecting -iitt--iiittitttion translations with variational transformers. arXiv:2006.08344
Jingjing Xu, Xu Sun, Qi Zeng. Xiaodong Zhang, Xu- ancheng Ren, Houfeng Wang, and Wenjie Li. 2018.

---

Unpaired sentiment- sereeeetttt--eetiine sentiment translation: A cy- cled reinforcement learning approach. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 979-988, Melbourne, Australia. Association for Computational Linguistics.
Parabank1.0 and Parabank2.0: contain clusters of sentential paraphrases, produced from a bilin- gual corpus using lexical constraints to the NMT decoding procedure (Hu et al., 2019a) or negative constraints, inference sampling, and clustering (Hu et al., 2019b) respectively. These datasets are com- posed of an average of 5 paraphrases in every clus- ter and close to 80 and 100 million pairs in total. After filtering we were left with around 50K sam- ples pairs in each of the datasets.
Xiangkai Zeng, Sarthak Garg, Rajen Chatterjee, Ud- hyakumar Nallasamy, and Matthias Paulik. 2019. Empirical evaluation of active learning techniques for neural MT. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 84-93, Hong Kong, China. Association for Computational Linguistics.
Formality transfer
Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Interna- tional Conference Learning Representations.
The Formality task is defined as the transition from informal to formal style.
GYAFC: The dataset was obtained from Rao and Tetreault (2018), and it contains 106K formal- informal pairs of sentences. Informal sentences were extracted from Yahoo Answers from two cat- egories "Entertainment & Music (E&M)" and "Family & Relationships (F&R)". The parallel for- mal sentences were produced with crowd workers. Due to its way of creation. it is considered high- quality dataset, and hence further filters were applied. Using the categories, we split GYAFC into two datasets, GYAFC-E&M and GYAFC-F&R, each with around 52K samples.
Ye Zhang, Matthew Lease, and Byron Wallace. 2017. Active discriminative text representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Zhisong Zhang, Emma Strubell, and Eduard H. Hovy. 2022. survey of active learning for natural lan- guage processing. arXiv:2210.1 10109.
Yuckai Zhao. Haoran Zhang. Shuchang Zhou, and Zhi- hua Zhang. 2020. Active learning approaches to enhancing neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1796-1806, Online. Association for Computational Linguistics.
Summarization
Jingbo Zhu, Huizhen Wang, Tianshun Yao, and Ben- jamin K Tsou. 2008. Active learning with sampling by uncertainty and density for word sense disam- biguation and text classification. In Proceedings of the 22nd International Conference Computational Linguistics (Coling 2008), pages 144.
DebateSUM: This dataset (Roush and Balaji. 2020) consists of around 187K arguments, with corre- sponding evidence texts and extractive summaries that were compiled by experts (competitors within the National Speech and Debate Association), We consider the evidence extracts as the input texts and the arguments as the target abstractive summaries.
A Appendix
Reddit TL,DR (openAl): The Reddit TL;DR dataset (Völske et al., 2017) contains more than 3 million reddit posts along with human-written summaries composed by the original posters. We use subset of this dataset introduced by Stiennon et al. (2020), which consists of more than 123K posts and summaries with higher quality (removed duplications, removed summaries with certain lev- els of profanity, etc.). Summaries contain between 24 and 48 tokens.
A.1 Datasets and Tasks
Paraphrase Generation
Paraphrase generation datasets include pairs of an input text and its paraphrase, typically created by automatic alignment. To ensure high-quality can- didate samples, we followed Dong et al. (2021), and kept only pairs where the BERTScore (Zhang et al., 2020) between the input and the paraphrase is higher than 0.8.
MSCOCO: This dataset consists of 123K images, each being associated with at most five human- labeled captions (Lin al., 2014). Following pre- vious studies, we consider different captions of the same image as paraphrases. After applying the filtering we were left with more than 13K samples.
Question Generation
Question answering datasets are also used for the Question generation task, where given a context and an answer the model is asked to generate a question that leads to this answer.

---

NewsQA: a collection of more than 100K human- generated questions and answers. Questions are posed by crowd workers on a set of news articles from CNN, and the relevant span is annotated as the answer (Trischler et al., 2017).
randomly selecting single unlabeled example serve as the initial seed. We then use this single example as Lp and apply the standard Core-Set algorithm for selecting n1 - instances, where ni is the batch size of the first iteration. The subse- quent AL iterations of Core-Set are selected using the standard algorithm.
MLQA: multilingual question answering dataset, with questions generated by the crowd over English paragraphs from Wikipedia that were found to have corresponding parallel paragraphs in other languages. Professional translators then translate these questions into all target languages, and answer spans are annotated within the aligned contexts. In this work, we use the English subset only, which consists of 12K pairs (Lewis et al.. 2020).
A.6 Evaluation metries
Formality: To obtain formality scores for model outputs, train classifiers by fine-tuning DeBERTa-v3 once over the GYAFC-E&M dataset, and once over GYAFC-F&R (to accuracies of 92%); these classifiers are used to evaluate the level of formality of the unseen dataset, respec- tively. Then, we follow Xu et al. (2018) and use G-Score of the formality score and BERTScore (Zhang et al., 2020) with the reference text.
A.2 Instructional Prompts
Table 2 reports all prompt templates used for the different tasks.
A.3 Full Active Learning Plots
Figure presents the active learning performance ($5) of the datasets tested.
Figure 7 depicts the full results for the Oracle strategy from the analysis section ($6.2.2).
A.4 Statistical Significance Analysis
We perform statistical significance analysis to evaluate the benefits of the different AL strategies in comparison to random selection.
Following Ein-Dor et al. (2020), we opt for the Wilcoxon signed-rank test due to its non-parametric nature. To calculate the p-value for a strategy over dataset D, we compare the performance of the relevant evaluation metric (Table I) for all pairs (Sij, Raj), such that R is the Random selection strategy, i (1...18) is the iteration index, and j (1...5) is the experiment repetition number. We apply a Bonferroni correction adjust for the multiple strategies examined.
A.5 Core-Set Adaptation
As stated in $4.4, we adapt the greedy Core-Set algorithm from Sener and Savarese (2018) for the scenario of starting with zero-shot base model and an empty initial labeled pool T.p-
The original algorithm starts from seed of la- beled data, and then relies it in order to greedily choose unlabeled examples one at time to add to the labeled pool. In this work we begin with an empty pool LD. Thus, for the first AL iteration of the Core-set strategy, we jump-stan start this process by

---

Paraphiuse yeneration MSCCCC
Parapluese geueralion ParabankVi
5.3
3.0
4.5
4.0
20
strategy
Core
TDDS
IDS
MCDoropout
MTE
2000
Train Exampies
TTain Esausios
Paraphrase generation ParabankV2
Question generation -MQA
15
14
10.30
Al. strategy Corn-see se.
Al. stratagy Core-ant
1/7
12
Mc3Drpppot
McDDropout
MIY
16
Mil:
Thanlor
Random
10
15
200
400
600
B00
1000
0
200
400
600
800
1000
Examplis
Train Esumakes
Question generation NewsQA
Summarization DebateSum
0.175
17
0.165
15
strategy
strategy
14
Core ser
Core-set
IDDS
IDS
MCDripout
MCDropout
Random:
Rendom
12
0.135
1000
Irein honmgres
Summarization Reddit TL,DR
Formality transfer- GYAPC-E&M
0.8
0.25
30L084
0.74
0.6
AL siretegy
AL sicategy
0.3
MCDropout
McDropout
MTE
0.4
MTL
Randomm
Handems
0.20
200
800
600
800
1000
200
400
600
800
1000
Train Exangles
Train Exampiee
Formality transfer GYAFC-F&R
0.8
5,0000
0.6
stratogy
Care set
TDDS
04
Randon:
Examples
Figure 6: AL performance. The lines depict the evaluation performance of the different selection strategies along the AL process. Each line represents average 95% Bootstrapped CI) over 5 experimental repetitions.

---

Paraphrase generation MSCOCO
Paraphrase generation ParabankV1
5.5
25
5.0
4.0
3.5
3.0
AL straterry
AL strategy
2.5
Oracie 3BLBU
16
Oracle IBLEU Random
Handom
14
1.5
100
200
300
400
500
160
200
300
406
500
# Triin Examples
0 Tixin Exumplos
Poraphrasc gencration
ParabonkV2
Question generation NewsQA
17
16
14
13
AL strategy
AL stratesg
Oracie .BLEU
Oracle BLEU
Handom
Rondom
10
10
100
200
300
100
500
100
200
300
100
500
Train Examplex
Tixin Exomplle
Summarizzaton DebateSum
Summarization Roddit TL,DR
0.27
0.165
0.26
HOUGH-L
0.155
BODOGE
0.24
0.150
strategy
AL strategyy
0.110
Oracle ROUGE
Oracie ROUGE-L
Rendom
0.21
Randon:
0.153
0.20
100
200
300
401
500
Trala Examples
Drampled
Formality transfer GYAPC-E&M
Formality transfer F&R
33,00
3.25
2.75
2.50
2.75
2.25
Score
2.25
2.00
1.75
strategF Oracie G Score
1.75
Al. strateggy Oreale G Soure Rendoo
Handom:
1.50
1.25
1.25
0
100
200
300
400
500
A Train Exurr piles
Examppee
Figure 7: AI performance of an "Oracle" strategy. The lines depict evaluation performance along AL. process. The Oracle strategy is exposed to ground-truth labels, and selects examples based poor performance the automatic task evaluation metric. Each line represents an average (+t 95% Bootstrapped CI) over experimental repetitions. For the sake of presentation, in these plots we focus the range of up to 500 labeled examples.

---

Task
Prompt
Paraphrase generation
Here is text: (input text)
Write a paraphrase for this text:
Summarization *
Here is a text: (document text]
Write a short summary for this text:
Question generation
Here is some context: [context]
And an answer: fanswer)
Given the context, write question that leads to this answer:
Formality
Here is an informal text: (informal text)
Write this text in formal manner:
Table 2: Prompts. *The summarization prompt was used only for the DebateSum dataset, as Reddit TL:DR dataset includes its own prompts.


Reviews:
[{"review_id": "VaeGOShdpe", "reviewer": "Reviewer_hX5h", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful input.\nOur goal was to understand the state of affairs in active learning for generation. As this is an empirical study, we did not know what results we would get, but indeed we were somewhat surprised that no strategy consistently outperformed the random strategy. As the focus of the paper was to evaluate and analyze the current state-of-the-art, devising new approaches is beyond the scope of this work. Given the results of our systematic study, the hope is that future work would offer novel algorithms for this setting. This is also something we are actively pursuing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper describes multiple AL(Active Learning) strategies, divided into two categories: i.e. representativeness and informativeness. Also, uses these on 4 primary tasks: para-phrase generation, style transfer(formality), summarization, and question generation.\nThe paper uses an underlying pre-trained model i.e. Flan-T5 model hence avoids training data used in the model.\nThe paper presents AL performance using different strategies on Parabank1.0 and GYAFC-F&R dataset.", "reasons_to_reject": "The paper poses multiple AL strategies to evaluate the performance of the model on unlabeled datasets.\n1. After trying multiple techniques to label data, the author shows that the random strategy is the best strategy to use random data.\n2. Author should devise novel algorithms for using unlabeled data to get better model performance after fine-tuning.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. The paper presents a systematic study of how well AL(Active Learning) strategies perform in NLG.\n2. Gain insights into how well these strategies perform in the context of text generation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "53X5pwp2CZ", "reviewer": "Reviewer_i9ma", "summary": "", "questions": "NA", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful input.\nIndeed, as we note under Limitations, hyperparameter tuning in the context of a large-scale AL study is simply not feasible. But importantly, the capacity to tune hyperparameters is usually very limited in real-world settings as well, since where the goal is to minimize the amount of labeled data a validation set is not available (or is very small).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a first systematic study of active learning for NLG, which is a novel contribution to the field. The authors explore a diverse set of tasks and selection strategies, and analyze the selection behaviors of existing AL strategies. They also highlight some notable differences between the classification and generation scenarios, which affect the performance of AL strategies. The results of their study indicate that the performance of existing AL strategies is inconsistent, surpassing the baseline of random example selection in some cases but not in others. Overall, this paper provides valuable insights into the application of active learning to NLG and lays the foundation for future research in this area.", "reasons_to_reject": "While practitioners may pursue hyper-parameter tuning for their trained model, this is not feasible in the context of a large AL study like the present work. These limitations may affect the generalizability of the findings to real-world scenarios. However, these limitations are common to many academic studies and do not necessarily invalidate the contributions of the paper.", "questions_to_authors": "NA", "paper_topic_and_main_contributions": "This paper is about Active Learning for Natural Language Generation. The main contributions of this paper are a systematic study of active learning for NLG, exploring a diverse set of tasks and selection strategies, an analysis of the selection behaviors of existing AL strategies, and highlighting some notable differences between the classification and generation scenarios.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "XhMag3S5hA", "reviewer": "Reviewer_74zp", "summary": "", "questions": "A: I am curious if you have an intuition as to why the Oracle strategy was worse than all the others? Is it because the outliers it selects are in fact mislabeled/corrupted examples?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful input.\nIndeed, as we note under Limitations, hyperparameter tuning and testing multiple models in the context of such a large-scale AL study is simply not feasible. The gaps between AL experiments and real world use, which we certainly acknowledge, are indeed a general issue with the field. We tried our best to account for what we can, but indeed these issues are beyond the scope of this study as well as of all classic AL empirical studies.\n\nA: Indeed, the key point here as we see it is the propensity to select outliers, which may be more extreme than those chosen by other strategies. We do not think this is necessarily one specific type of outliers - the Oracle can both tend to select mislabeled examples, examples with typos/tokenization issues etc., but also these can simply be examples that are uncharacteristic of the data distribution. In all of these different types of outliers we expect that training on these examples would not be very informative for the model, as they are not good representatives of the typical distribution of the data and task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Systematic and Novel. The strengths of this paper lie in its systematic approach to studying AL for NLG, a highly relevant area of study. The authors conduct experiments across multiple tasks and datasets, testing and comparing known AL strategies. The authors are also the first to look into effective AL for NLG  (as far as I could find). \n\n- Motivation for further research. The paper's findings provide valuable insights into the performance of existing AL strategies, highlighting their inconsistencies and the need for new methods. This could stimulate further research and innovation. The paper also offers a comparative analysis of the strategies, serving as a reference point for future studies. The authors' discussion on the inherent complexity and diversity of human language in the context of AL strategies could also benefit researchers working to gather data for LLMs. ", "reasons_to_reject": "- Limited model variation. The paper's findings are based on a single base model and a specific set of hyperparameters. This could limit the generalizability of the results, as the performance of AL strategies might vary with different models or hyperparameters.\n  \n- Gaps with real-world datasets. The authors acknowledge inherent gaps between AL experiments, such as those conducted in the paper and the goal of achieving label efficiency with a human in the loop. For instance, factors such as differing dataset qualities and temporal drifts in data distributions, characteristic of real-world data, are not accounted for in the experiments.\n", "questions_to_authors": "A: I am curious if you have an intuition as to why the Oracle strategy was worse than all the others? Is it because the outliers it selects are in fact mislabeled/corrupted examples?", "paper_topic_and_main_contributions": "This paper analyses the challenge of applying Active Learning (AL) to Natural Language Generation (NLG). The AL technique selectively chooses the most informative examples to label, improving annotation efficiency. The paper presents a systematic study of AL for NLG tasks, considering a diverse set of tasks, multiple leading selection strategies, and a strong instruction-tuned model baseline. The study reveals that the performance of existing AL strategies is inconsistent, sometimes surpassing the baseline of random example selection. The authors' findings indicate that despite the potential of active learning techniques, human language's inherent complexity and diversity still pose a significant challenge to naively applying AL in NLG.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:
The reviewers agree that the paper's positives including a systematic and novel study of active learning (AL) for natural language generation (NLG). Overall, the paper is said to lay the foundation for future NLG AL research and is a pioneering contribution to the field. It categorizes AL strategies into representativeness and informativeness and applies them to four primary tasks: paraphrase generation, style transfer (formality), summarization, and question generation. The study utilizes the Flan-T5 pre-trained model, avoiding the need for additional training data. The research provides valuable insights into the performance of existing AL strategies, highlighting their inconsistencies and the need for new methods. It motivates further research in NLG AL by offering a comparative analysis of strategies and addressing the complexities of human language. 

The reviewers share the following limitations of the paper: (1) hyper-parameter tuning is not feasible in the context of a large active learning (AL) study like the one presented. This limitation may impact the applicability of the findings to real-world scenarios. (2) The study's findings are based on a single base model and specific hyperparameters and could restrict the generalizability of the results, as AL strategy performance may vary with different models or hyperparameter settings. (3) The authors recognize inherent gaps between AL experiments conducted in the paper and the ultimate goal of achieving label efficiency with human involvement in a real-world context. Factors such as varying dataset qualities and temporal shifts in data distributions, typical of real-world data, are not addressed in the experiments. 

Reasons to Accept:
(1) The paper provides a systematic study of active learning (AL) for natural language generation (NLG), addressing a novel and relevant research area.
(2) It explores a diverse set of NLG tasks and selection strategies, shedding light on their performance and behaviors.
(3) The paper highlights the inconsistencies in the performance of existing AL strategies, providing valuable insights into the challenges and opportunities in NLG-related AL.
(4) The findings of this study can motivate further research and innovation in AL for NLG, serving as a reference point for future studies.

Reasons to Reject:
(1) The study primarily relies on a single base model and specific hyperparameters, potentially limiting the generalizability of the results to other models or settings.
(2) The experiments do not fully account for real-world dataset challenges, such as variations in dataset quality and temporal drifts in data distributions.
Decision: Accept-Main