Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation
Authors: Zijian Ding
Alison Smith-Renner
Wenjuan Zhang
Joel R. Tetreault
Alejandro Jaimes
Venue: EMNLP 2023 Findings (2023)
Keywords: human-centered NLP
large language model
human-AI collaboration
text summarization
TL;DR: We explored common human-AI interaction types in the context of LLM-assisted news headline generation.

Abstract: To explore how humans can best leverage LLMs for writing and how interacting with these models affects feelings of ownership and trust in the writing process, we compared common human-AI interaction types (e.g., guiding system, selecting from system outputs, post-editing outputs) in the context of LLM-assisted news headline generation. While LLMs alone can generate satisfactory news headlines, on average, human control is needed to fix undesirable model outputs. Of the interaction methods, guiding and selecting model output added the most benefit with the lowest cost (in time and effort). Further, AI assistance did not harm participants’ perception of control compared to freeform editing.

Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generation
Zijian Ding', Alison Smith-Renner', Wenjuan Zhang", Joel R. Tetreault Alejandro Jaimes² University of Maryland, College Park, Dataminn Inc.
Abstract
There are several methods of human-AI inter- action for writing, which vary terms of effort and control afforded by cach (Cheng et al., 2022). For example, humans might simply select from al- ternative model outputs or have the flexibility to revise them (e.g.. post-editing). Recent research by Dang et al. (2023) explored the dynamics of inter- action between crowdworkers and Large Language Models (LLMs) within the scenario of composing short texts. Extending from their work, we evaluate human-Al interaction in the context of news head- line creation, focusing on three of the interaction types identified by Cheng et al. (2022)-giddonce, selection, and post-editing. Our study advances Cheng et al.'s Wizard-of- framework by using implemented AI system, providing more real- istic insights into human-Al co-creation for news headline generation.
To explore how humans can best leverage LLMs for writing and how interacting with these models affects feelings of ownership and trust the writing process, we compared com- mon human-AI interaction types (e.g., guiding system, selecting from system outputs, post- editing outputs) in context of -Las-isstted news headline generation. While LLMs alone can generate satisfactory news headlines, on average, human control is needed fix unde- sirable model outputs. Of the interaction meth- ods, guiding and selecting model output added the most benefit with the lowest cost (in time and effort), Further, Al assistance did not harm participants' perception control compared to freeform editing.
1 Introduction
Recent advancements in Large Language Mod- cls (LLMs), including the Generative Pre-trained Transformer (GPT) series (Brown et al., 2020), have shattered the previous ceiling of human-like text generation (Bommasani al., 2021). This has led to a paradigm shift in NLP tasks, with task-agnoste LLMs surpassing the performance of other state-of-the-art task-specific models (Lee et al., 2022). However, LLM-bbcked systems are not without their flaws, often suffering from hal- lucinations, bias, occasional generation of inap- propriate content, such toxic, discriminatory, or misleading information (Wang et al., 2022).
We explore three main aspects of co-creation with LLMs, through the lens of headline generation: which type of assistance is most effective for help- ing humans write high-quality headlines (RQ1)? Which type of assistance can reduce perceived fort for such task (RQ2)? And, how does interact- ing with these models affect trust and feelings of ownership of the produced headlines (RQ3)?
In a controlled experiment, 40 participants wrote news headlines -eeed manually or assisted by one of three variants of an LLM-powered headline writ- ing system: (1) selection, (2) guidance + selection, and (3) guidance + selection post-editing- To explore the balance between control and effort in tasks involving human-Al interaction (Figure 1), we examined the different interaction types in com- bination rather than separately. Next, 20 evaluators rated the headlines generated by five methods- manual, Al-only, and the three assisted conditions. While LLMs alone generated the highest qual- ity headlines on average, they were not perfect, and human control is needed to overcome errors. The simpler interactive condition guidance + selec-
Human-Al text co-creation-ov or writing with Al assistance-allose allows some control over the genera- tion process and the opportunity to overcome some LLM deficiencies. Co-creation approaches have shown tremendous promise in areas such as sum- marization (Goyal al., 2022; Bhaskar al., 2022) and creative writing (Moore et al., 2023; Cao, 2023; Yuan et al., 2022; Ding et al., 2023). Yet, the lack of accountability of AT (Shneiderman, 2022) shifts the burden of responsibility to humans when mis- takes arise in the text co-creation process.

---

8
Al-Only
Selection
Guidance
Guidance Selection Post-editing
Manual-Only
Less human control effort
More human control effort
Figure I: Human-Al interactions for text generation can fall within a range of no human control effort (AI-only) to full human control effort (manual methods) (Ding and Chan, 2023). Selecting from model outputs (Selection) alone provides less control (but also is easier) than when adding additional interactions in the form of guiding the model (Guidance) and post-editing the outputs.
tion resulted in the rapid creation of high-quality headlines compared to conditions involving further human intervention (i.e., post-editing or manual writing). All conditions yielded similar perceived trust and control in our task.
creation. For human-AI co-creation in the context of journalism, recent study explored the potential of harnessing the common sense reasoning capabil- ities inherent in LLMs to provide journalists with diverse range of perspectives when reporting on press releases (Petridis et al., 2023).
This work contributes: (1) a systematic eval- uation of three variants of Au interaction for headline generation, (2) design guidelines for builders of these systems, and (3) dataset' of 840 human-rated news written by humans, AI, or the combination, which could be used as an evaluation set for further leveraged with reinforce- ment lcarning, c.g., RLHF (Stiennon et al., [n. d.),, to adapt LLMs for news headline generation tasks.
2.2 Human-Al Interactions for Text Summarization
The taxonomy from Cheng et al. (2022) of human- AI interactions for text generation serves as the framework for our investigation. We focus on the
2 Related Work
first three types - guiding model output, selecting or rating model output, and post-editing, which offer varying degrees of control over the output and require different levels of human effort. Other identified types, such as interactive writing, outside the scope of this study.
Our study draws upon convergence of prior re- search on news headline generation and evaluation, and human-AI interactions for text summarization.
2.1 News Headline Generation and Evaluation
Guiding model output entails human interven- tion in the automatic text generation process, such as specifying crucial parts of input text to be in- cluded in the output (Gehrmann et al., 2019), or pro- viding semantic prompts or style parameters (Os- one et al., 2021: Chang et al., 2021; Strobelt et al.. 2022). Our study involves human guidance in head- line generation by specifying keyword prompts. Se- lecting rating model output involves humans choosing from multiple model outputs (Kreutzer et al., 2018; 8; Rosa et al., 2020) or rating them (Lam et al., 2018; Bohn and Ling, 2021). In our ex- periment, participants chose from three generated headlines. Post-editing refers to humans editing generated text (Chu and Komlodi, 2017; Huang al., 2020), which could be accompanied by addi- tional supports. such suggestions for edits (Liu et al., 2011: Weisz al., 2021). Our study in- vestigates inline post-editing of selected headlines without any additional support.
News headline generation, conventionally consid- ered as a paradigm of text summarization tasks, has been extensively researched (Tan et al., 2017; Goyal et al., 2022). Advances in automation range from heuristic approaches like parse-and- trim (Dorr al., 2003) to sophisticated machine learning algorithms like recurrent neural networks (Lopyrev, 2015), Universal Transformer (Gavrilov et al., 2019), reinforcement learning (Song al., 2020; Xu et al., 2019), large-scale generation mod- els trained with a distant supervision approach (Gu et al., 2020), and large language models (Zhang et al., 2023). Zhang et al. (2023) demonstrated that news summaries generated by freclance writers Instruct GPT-3 Davinci received an equivalent level of preference from human annotators. In these studies, human involvement is for evaluation, not hhanss//ggthha.com//////////PPP Mha.EEMMn

---

Al co-creation of news headlines with four variants of human-Al interactions:
Lif da upturc Cheae
Selection: The LLM generates three headlines for each news article (generate headlines), and the user selects the most appropriate one;
paroal cooks oobii (1paer
Lioke prtas ard Necio dorics
inckuiin for your
Guidance + Selection: The LLM extracts sev- eral potential perspectives (keywords) from each news article (extract perspectives), the user chooses one or more perspectives to em- phasize in the headline, the LLM then gen- erates three headlines for each news article based on the selected perspectives (generate headlines w/ perspectives), and finally, the user selects the best one;
L.C dyland,,
molder
Figure Interface for human-Al news headline co- creation for guidance + selection + post-ediring con- dition: (A) news reading panel, (B) perspectives (key- words) selection panel (multiple keywords can be se- lected), (C) headline selection panel with post-editing capability, and (D) difficulty rating slider. Note: (B), (C) and (D) are hidden from the user until the requisite step is finished (c.g.. the user does not see the difficulty rating slider until after finalizing the headline).
Guidance + Seleccionn + Post-editing: This is similar to Guidance + Selection, but the user can further edit the selected headline (post- editing);
3 Assessing Human-AI Co-creation of News Headlines
Manual only: The participant writes the news headline without any AI assistance.
We compared three variants of human-AI co- creation for news headlines against fully manual and fully automatic approaches. Specifically, explored three research questions (RQs): RQ1: Which type of assistance most effective for help- ing humans write high-quality headlines? RQ2: Which type of assistance can reduce perceived ef- fort for such task? RQ3: How does interacting with these models affect trust and feelings of ownership of the produced headlines?
Our goal was to create a continuum of human-Al interaction methods ranging from less to more con- trol. The layering approach of selection, guidance, and post-editing was informed by prior literature (Cheng et al., 2022) and allowed us to assess the impact of each new layer by comparing adjacent conditions. Specifically, for these variants, we add additional functionality opposed to comparing each interaction individually to allow for more explicit comparison of conditions that afford less control (and require less effort)-e.g. selection only opposed to more control (more effort)- c.g., guidance + selection + post-editing (Figure 1).
We chose news headline creation it is familiar task many participants would already have some intuition about what constitutes com- pelling (or not) news headline-of manageable complexity-hoth task and evaluation, compared to free-form creative writing or long-form summa- rization (e.g., authoring paper abstracts).
We used OpenAT's GPT-3.5 text-davinci-002 model, the state-of-the-art LLM as of the study period (July 2022). Our study system directly in- terfaced with OpenAl's GPT-3.5 API, employing prompt programming to extract perspectives (key- words) and generate headlines with and without supplied perspectives and presented the results to the participants (see Figure 2).
3.1 Co-creation Methods, Implementation, and Prompts
Participants wrote headlines in one of four ways: manually or aided by AI using one of three com- mon human-AI interaction types identified by pre- vious research (Cheng et al., 2022): guidance of the LLM's headline generation using perspectives (keywords facets), selection from three LLM- generated headlines, and post-editing the LLM- generated headlines.
To determine the optimal prompts for extract per- spectives, generate headlines, and generate head- lines w/ perspectives with the GPT-3.5 API, we conducted multiple configuration tests, ultimately selecting the zero-shot learning paradigm (with no examples of news headlines included) for extensive coverage of news topics. The GPT-3.55 API takes
Specifically, we developed a system for human-

---

prompts (provided in Appendix A) and generates the requested outputs.
Participants first learned how utilize the sys- tem with various types of Al assistance (instruction round) and wrote headlines for two practice articles (practice round).
To rigorously examine the LLM's reliability, we used temperature setting of 1, ensuring maximum output variety. Additionally, we set a token count of 400, providing sufficient space for both input data which included prompts and news articles and output data, encompassing several keywords news headlines.
During the main study, participants wrote head- lines for the designated 20 task articles with assis- tance based their assigned condition. The order of news articles was randomized for each partic- ipant. For each article, participants first read the article and clicked "Continue to Write Headline". Participants then either wrote the headline manu- ally (manual) or with some assistance as described in Section 3. After completing each headline, participants rated the difficulty. Figure 2 demon- strates the system for the Guidance + Selection Post-editing condition.
3.2 Study Design
The study was structured into two phases: (1)
human-Al co-creation of news headlines and (2)
human evaluation of the generated headlines. The first phase utilized a between-subjects experimen- tal design with four conditions, ranging from mini- mal to maximal human control: (1) Selection, (2) Guidance + Selection, (3) Guidance + Selection Post-editing, and (4) Manual only. The second phase involved human evaluators who ranked the headlines generated in the first phase alongside the original article headlines and headlines generated by GPT-only without any human input.
Finally, after writing headlines for all 20 articles, participants completed post-task survey. See the Appendix for all study figures, including instruc- tions, all conditions, and post-task survey.
3.2.2 Phase II: Human Evaluation of Headline Quality
Participants Another 20 evaluators were recruited through Upwork. We required that they were expe- rienced in reviewing or editing news headlines ensure high-quality evaluations.
3.2.1 Phase I: Human-Al Co-creation of Headlines
Participants We enlisted 40 participants from the freelancing platform, Upwork, 2 ensuring diversity in gender (28 females, 11 males, non-binary). All the participants, who read news articles at least few times week if not daily, had experience in journalism, editing, or writing. Their educational backgrounds were diverse: 18 with Bachelor's De- grees, 11 with Master's Degrees, 10 with High School education, and 1 chose not to disclose.
Procedure In total there were 840 headlines requir- ing evaluation, including 800 headlines from Phase I (20 articles 4 conditions 10 participants per condition), 20 Original headlines, and 20 headlines generated by GPT only.
To ensure every headline was reviewed by at least two evaluators, we asked each evaluator to review 120 headlines-the six headlines for 20 articles-includding 80 from Phase I, 20 Original, and 20 GPT only. We provided the evaluators with Excel forms containing instructions, news articles, and news headline ranking tables. For cach article, the evaluators ranked the quality of six different headlines (the Original, GPT Only, and a headline generated by each of the four study conditions) us- ing the Vaste-AtrractivennssCCie Clarity--TTuuii Truth (TACT) test 6 from I (best) to 6 (worst) and shared their reasons:
Procedure Each participant was asked to create headlines for 20 articles, published in June 2022 Yahoo! News, 4 the most popular news website in the US during Q1 2022.5 The articles were carefully chosen across four distinct themes: World Politics, COVID-19, Climate Change, and Science, with five articles each. The overall study procedure included an instruction round, a practice round, main study, and a post-task survey. Participants were randomly assigned to one of the four study conditions and compensated $20 for their 1-hour participation.
Taste: Is it in good taste? Is there anything possibly offensive in the headline? Can any- thing in the hcadline be taken the wrong way?
https///www.ppwo....cm
'The articles varied in length from 300 to 1000 words.
httpss//newwsyyhhooooo comm
wwwp://wwww..o..oooooooooemmmmmo news-websites/all
http://wwwwiiiiiiii.//iiiiiiiinm Headlincs.html

---

Attractiveness: Is attractive to the reader? Can it be improved so is more engaging and interesting, without sacrificing accuracy?
trust of the system based on prior research: the question to gauge participants perceived control was "I could influence the final headlines for the news articles" (Rader et al., 2018). The question to gauge perceived trust was trusted that the system would help me create high-quality head- lines" (Cramer et al., 2008), via 5-point Likert rating scales (from strongly disagree to strongly agree).
Clarity: Does it communicate the key points of the article? it clear and simple? Does use the active voice and active verbs? Are there any odd words or double meanings that could confuse the reader?
Truth: Is it accurate? Are the proper words or terms from the article used in the headline? Is the headline factually correct?
3.4 Data Analysis
We employed the Kruskal-Wallis test (Kruskal and Wallis, [n. d.J), the non-parametric version a one-way ANOVA test for data with three or more conditions, to assess the quantitative results of head- line quality rankings, task efficiency, and average and overall task difficulties, and perceived control and trust ratings. We then conducted pairwise comparison of headline quality using the Mann- Whitney test (Mann and Whitney, 1947), the non-parametric version of the Student t-test for two conditions. For headline quality, we compared the four conditions (Manual, Selection, Guidance Se- lection, and Guidance + Selection + Post-editing) against the original headlines (Original) and the automatic method (GPT only). For task efficiency and user experience, compared only within the four human study conditions. Finally, analyzed participants general comments and feedback on task difficulty.
When two or more headlines for the same article were of similar quality, evaluators were able to assign the same ranking them. Each evaluator was compensated $20 each for this task, estimated to take around an hour. 8
3.3 Measures
We measured headline quality (Section 3.2.2), per- ceived task difficult, headline creation time, and perceived control and trust of the system. For com- paring efficiency between conditions, we care most about the headline creation time (from clicking "Continue to Write Headline" to "Submit Head- line"). We further compute the article reading time (from starting to read the news article to clicking "Continue to Write Headline") and overall time (ar- ticle reading time + headline creation time). For user experience, we measured the task difficulty-afie after creating each headline (instance- level) and upon completing all headlines (overall). well perceived control and trust of the system. For instance-level task difficulty, participants an- swered "How difficult was to create this headline for the news article with the system?" on slider from 0 (very easy) to 100 (very difficult) after cach headline writing task.
Results
We compared headline quality (RQ1), task effort (RQ2) and perceived control and trust (RQ3) be- tween the different headline writing conditions.
4.1 RQ1: Headline Quality
Rankings from Phase II: Human Evaluation of Headline Quality (Section 3.2.2) were used to as- sess headline quality across six conditions. Table shows number of ranking data points and average rankings for each condition.?
The remaining subjective measures were col- lected in the post-task survey: For overall task dif- ficulty, participants agreed or disagreed with "It difficult to create headlines for the news ar- ticles with the system." on Likert scale from 1 (strongly disagree) to (strongly agree) and an- swered a follow-up question, "why do you feel this way?". We operationalized perceived control and
As shown in Table and 2, an approximate rank- ing of headline quality as follows: Original GPT only (best) > Guidance + Selection Guid- ance Selection + Post-editing > Selection Man- ual only. The variance in quality rankings across the six conditions was significant, as confirmed by
'A small amount of headlines (1.3%, or 11/840) were found be identical to another headline. These were to participants selecting an identical GPT-generated headline without making further edits.
"Rankings (from 1st to 6th place) for total of 120 head- lines (20 article 6 conditions) were obtuined (as shown in Table 1). However, ten were missed due to annotator error, resulting in 2390 total data points for analysis.
"Based on our pilot, evaluating one news article with six headlines took approximately three minutes.

---

Condition Ranking
3
Mean
Std
Manual only
56
56
98
1.8
Selection
56
67
75
1.7
Guidance + Selection
71
75
58
3.4
1.7
Guidance + Selection Post-editing
70
71
74
60
58
1.7
Original
87
75
68
65
38
1.7
GPT only
70
73
89
79
51
37
3.2
1.6
Table 1: Counts of rankings from (best) 6 (worst) across six conditions, with mean rank and standard deviation. Lower values indicate superior rankings. Bold counts represent the highest number of 1st place rankings and the fewest 6th place rankings. The original headlines (Original) received the most 1st place rankings and the fewest 6th place rankings, while the Manual condition received the fewest 1st place and the most 6th place rankings.
Condition Condition 2
U-value
p-value
Original Manual only
61766.0
<0.01
Original Selection
63550.0
<0.01
Original Guidance Selection
72364.5
0.028
Original Guidance Selection Post-editing
72688.5
0.048
GPT only Manual only
63168.5
<0.01
GPT only Selection
64751.5
<0.01
Guidance + Selection Manual only
89877.5
<0.01
Guidance + Selection > Selection
88536.5
<0.01
Guidance + Selection + Post-editing > Manual only
89972.5
<0.01
Guidance + Selection + Post-editing > Selection
88734.0
<0.01
Table 2: Pairwise comparison of headline quality rankings with significant difference (p<0.05, Mann-Whitney U).
Kruskal-Wall Wallis tests, with an H value 50.8 and a p value < 0.01. Additional pairwise comparisons (detailed in Table 2) offered more granular insights into the discrepancies: headlines penned by participants in the Manual condition were deemed the lowest quality, whereas the origi- nal headlines and headlines generated by GPT only were deemed highest quality.
into the reasons behind their rankings. For exam- ple, one annotator favored the concise headline generated by GPT only and critiqued the errors that undermined the Manual headline. For more details, see Appendix B.
4.2
RQ2: Task Effort
4.2.1 Headline Creation Time
Within the Al-assisted conditions, human guid- ance, in the form of selecting keyword perspectives (under Guidance + Selection or Guidance + Se- lection + Post-editing), improved headline quality compared to direct Selection of AI Al-generated op- tions. Interestingly, post-editing had no significant effect on the quality of the headlines: no noticeable difference in ranking was identified between the headlines generated by Guidance Selection and Guidance + Selection + Post-editing conditions.
We used task time as an indicator of task effort because it often reflects the workload someone ex- periences. Specifically, we compare headline cre- ation time-the time spent to generate the news headline, exclusive of the time it took to read the news article. As expected, the Selection and Guid- ance + Selection conditions were markedly faster -HIval = 13.0, p pavalue = 0.005) than condi- tions that involved manual editing, such as Manual and Guidance + Selection + Post-editing (Figure 3). A detailed comparison of creation time, reading time, and total time is provided in Appendix C.
To delve deeper into group comparisons, we ex- amined the count of "top" (ranked Ist) and "bot- tom" (ranked 6th) rankings (as shown in Table 1). As anticipated, the Original headlines of the news articles received the most "top" rankings (87) and the fewest "bottom" rankings (38). The GPT only condition had fewer "top" rankings than Original (70), but a comparable number of "bottom" rank- ings (37). The Manual and Selection conditions had the fewest "top" rankings (56), while Manual also had the most "bottom" rankings (98). Evaluators comments gave additional insight
4.2.2 Perceived Task Difficulty
On the whole, participants felt the headline writing task was easy (e.g., average overall difficulty rating 1.9 out of 5). And, although Guidance + Selection had the lowest percieved instance-level difficulty (M=19.7 out of 100. SD=24.3) and Manual had the highest (M=30.9, SD=17.1), 1), these differences were not statistically significant in part due to high variance between participants and our small sam-

---

4000
Therefore, system designers should consider lever- aging these types of assistance better support humans performing similar writing tasks.
2002
2500
While our findings could imply that completely automated methods are adequate for generating headlines, we urge system designers to critically evaluate when and where is appropriate to fully rely on Al assistance. Headlines by GPT only were not flawless, receiving the lowest ranking in some instances. Human intervention remains necessary for editing and final decisions, particularly in high- stakes tasks. Upon reflection, this study focused on overall headline quality in the evaluation task. Additional evaluation on the type, quantity, and severity of errors from headlines could reveal valu- able insights. While multiple reasons could lead to headline ranked low quality, problematic hal- lucinations that are commonly seen with large lan- guage models (Bender al., 2021; Ji al., 2023) could be more harmful than omission of details.
2200
300
Nemual only
Sedection
Guccance &
Guidacee Paalediing
Haman condiiins
Figure 3: News headline creation time across four hu- man conditions (standard deviation as error bars): Se- lection and Guidance + Selection are faster than the other two conditions which required more human edit- ing (Mamual and Guidance + Selection Post-editing eostiedii
ple size. Additional results for instance-level and overall task difficulties are detailed in Appendix D.
4.3 RQ3: Perceived Trust and Control
We did not observe significant difference in per- ceived trust or control across the conditions (see Appendix E for more details). We had expected participants with more "control" over the final out- put (i.e., those writing manually or with option to post-edit the final output) have more perceived control than the other conditions. Yet, even while most participants (9/10) in the Guidance + Selec- tion + Post-editing condition did edit at least one headline during the task, with an average of 8.5 headlines edited (median 7.5, sld 6.0), post- editing did not seem to enhance participants' sense of ownership the generated headlines. We discuss possible reasons for this Section 5.3.
Among the Al assistance, providing Guidance (using keywords) resulted in higher headline qual- ity compared to Selection alone, yet did not outper- form the GPT only condition.
Participants comments revealed that the qual- ity of keywords given in the Guidance interaction played crucial role in the final headline quality. When the keyword quality was poor, e-g., vague or not well aligned with key points of the article, the generated captions were lower quality as well. The design of the Guidance interaction may have limited its utility. One participant noted "it would have been nice if could go back and change the keywords and see how the headlines change", sug- gesting a more interactive experience could help them better understand how their guidance influ- ences the model's output.
5 Discussion
Our study explored how diverse forms of Al assis- tance for news headline writing influence headline quality (RQ1), task effort (RQ2), and perceived trust and control (RQ3). We interpret our results for each the RQs and discuss further implications in the following.
Headline evaluation is subjective task, which can influence these outcomes. To mitigate this, we used the TACT evaluation framework as a standard- ized rubric, which was shared with both headline generators and evaluators. Importantly, human trol over the final headline is promising direction given the subjective nature (i.e., simply relying on the system to pick the best headline problematic if what is best differs by the audience and their desired headline qualities.
5.1 RQI: Which type of assistance helped humans write high-quality headlines?
Headlines produced by the GPT only condition were, on average. of the highest quality, whereas headlines written manually were of the lowest, And, all types of AT assistance explored in the study helped participants achieve higher quality as compared to when no assistance was available.

---

Type
Refore Revision
After Revision
1) Hedging
Is your lawn ruining euvironment?
Is your lawn contributing climate change? Soldiers given second chance to get vaccinated and avoid penalties
2) Hedging
Soldiers given second change to get vaccinated and avoid expulsion
3) Catering
Mapping the Seafloor: New technologies erucial for completion
Mapping the Scafloor: New technologies erucial for climate change
4) Clarifying
Forest Service employees made several mistakes when planning a controlled burn in New Mexico, resulting in the largest fire the state's history
Forest Service mistakes when planning a controlled burn in New Mexico, resulting in the largest fire in New Mexico history
Table 3: Examples of humans' post-editing Al-generated headlines.
5.2 RQ2: Which type of assistance reduced perceived effort?
and had no way to compare with other conditions. A within-suhject study would be more effective for comparing perceived difficulty. control, and trust across multiple experiences. However, we chose between-subjer design avoid the fatigue of having complete similar tasks with multiple in- terfaces and the confusion from switching from one experience to another.
In terms of headline creating time, interactions such guiding the model output and selecting generated headlines were quicker than post-editing, as ex- pected. This means when efficiency is key to sys- tem's performance, these type of interactions would be most effective, especially when the model out- puts are generally satisfactory. Post-editing would likely be more critical if the quality of the model's output was suboptimal, or when humans need the ability to fully control the outcome.
Finally, participants may have felt they did not need to exert much control for the system to per- form well and, therefore, did not think about the level of control critically. Counter to what the results showed, we had expected post-editing to improve the sense of control. Analysis of their post-editing process (Table 3) revealed that changes made by participants during post-editing were typi- cally minor, falling into three categories:
Interestingly, in perceived task difficulty did differ across all conditions. This likely because most participants were experienced in headline writing. While their experience level varied greatly, from working for school newspaper to being in journalism for more than 40 years, the hands-on ex- perience made the task in our study easy for them. Future research direction might evaluate if such AT assistance lower the expertise requirements for headline writing task, enabling people with less experience write good quality headlines.
Participants generally avoided sensationalism or "click-bait" language, fa- voring more measured phrasing. For instance, "ruining the environment" was altered to "con- tributing to climate change", and "expulsion" revised to "penalties".
"Hedging":
2. "Catering": Participants frequently tweaked the headlines to make them more relevant to the target audience, for instance, changing "completion" to "climate change".
5.3 RQ3: How does interacting with these models affect feelings of trust and ownership of the produced headlines?
We did not observe any significant difference in perceived trust or control. While surprising, we expect this might be from a flaw in our operational- ization of perceived control; the statement "I could influence the final headlines for the news articles" does not specify the level or type of control one had over the system, and the participants could "influ- ence the final headlines" to some extent no matter which condition they experienced. We recommend future research investigate more robust methods for measuring perceived control or sense of ownership. Another reason for the lack of significant differ- ence in this (and other subjective measures) that each participant only experienced one condition
"Clarifying": Some edits involved minor text adjustments, such as replacing "the state's" with "New Mexico".
Participants may have associated smaller degree of changes with smaller level of control to the sys- tem output. However, highlight that these vision types could be supported natively by future systems.
5.4 Control, Transparency, and Guardrails in Co-Creation
The interaction options we introduced extend beyond mere control, instead providing both

---

guardrails and way to expose what is possible with LLMs. First, both the selection and guidance interactions provide a set of guardrails toward bet- ter output opposed to freeform prompting of the LLM. Next, as participants can guide the model via theme or keyword inputs, they are also afforded a level of transparency to comprehend the influence they may exert on LLMs to yield headlines with varying focuses. These approaches which combine control and transparency seem critical to successful co-creation settings (Liao and Vaughan, 2023).
Limitations
This study, while comprehensive, acknowledges several limitations that outline areas for potential future research. One of the constraints pertains to the evaluation of different models. Specifically, the research primarily focused on types of human-Al interaction, utilizing GPT-3.5 due to its state-of- the-art performance during the research phase. The choice was made to prevent the complication of the study design by introducing performance as ad- ditional variable, which could have overshadowed the main objective of analyzing three interaction variants. However, understanding the impact of varying models and their performances remains an essential prospect for subsequent studies and system design.
We did not evaluate a setting where users prompted the LLM themselves (this is suggestion for future work): instead, our settings- corporated optimized prompts (Appendix A) -are meant to provide a more directed scenario, without requiring the user to come up with possible key- words from scratch. This scenario progressively ex- ploded users to how the model works and how they can control it. Future iterations of the guidance interaction could benefit from additionally offer- ing users choice between system-suggested and user-defined keywords to flexibly steer the model.
in-
Morcover, in terms of evaluation measures, the study employed the Yaste-Attractivennss- -lariit--- Truth (TACT) framework assess headline quality. Despite its standardization, this framework may not fully encapsulate all the nuances of a model's ss performance, such as verbosity, indicating a need for future work to refine these evaluation standards. Additionally, the research's 's scope was limited in terms of the number and types of articles, as it pri- oritized the examination of interaction differences over article diversity. This limitation underscores the importance of future studies exploring a broader array of articles and involving more participants for an extended period. Similarly, the study did not evaluate the efficiency of the model in generating diverse perspectives, maintaining consistency in ar- ticle selection based length and domain instead. The assessment of content generation difficulty is identified as a crucial element for more in-depth research.
6 Conclusion
We explore human-Al text co-creation through the context of news headline generation: how can hu- mans best harness the power of LLMs in writing tasks, in terms of both quality and efficiency? And, how does interacting with these models affect trust and feelings of ownership of the produced text? 40 participants with editing backgrounds wrote news headlines- either manually or assisted by a variant our LLM- -oeeeedweeed headline generation system utilizing some combination of three common in- teraction types (guiding model outputs, selecting from model outputs. and post-editing). Following this, 20 expert evaluators rated the headlines gen- erated by five methods (human-only. Al-only, and the three assisted conditions). While the LLM on its own can generate satisfactory news headlines on average, human intervention is necessary to correct undesirable model outputs. Among the interac- tion methods, guiding model output provided the most benefit with the least cost (in terms of time and effort). Furthermore, AI assistance did not diminish participants' perception of control com- pared freeform editing. Finally, while focus on a simpler, low-stakes text generation scenario, this work lays the foundation for future research in more complex areas of human-AT co-creation.
Concerning participant expertise, the study con- ducted was concise, one-hour session centered on human-AI collaboration in news headline creation, taking into account participant expertise without enforcing strict control. This approach points to the necessity for more extensive research, particularly in more complex and specialized domains.
Finally, regarding study design and comparison, the research adopted between-suhject approach, preventing participants from making direct compar- isons of the different human-Al interaction meth- ods. As result, certain participants might not have fully grasped how AI could potentially en- hance the task, as their experience was confined to their specific condition. This highlights an oppor-

---

tunity for future studies to possibly implement a within-subjects design for a more holistic partici- pant experience.
Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma. Ali Malik, Christopher D. Manning. Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the Opportunities and Risks of Foundation Models. 07X02:2108.07258 [cs] (Aug. 2021). http: www.arii..or/aab...ee.eeee....ee 07258 00021 arXiv: 2108.07258.
Ethies Statement
Although the performance of headlines generated by GPT only were comparable to the original head- lines and those headlines did not demonstrate evi- dent biases or ethical issues, there is an open ques- tion of to what extent AI should be relied upon. Shneiderman proposes that "we should reject the idea that autonomous machines can exceed or re- place any meaningful notion of human intelligence, creativity, and responsibility" (Shneiderman, 2022). We echo that serious consideration is needed be- fore Al is relied upon for creating text, even for low stakes tasks. It is important for humans to use their expertise and judgment when working on con- tent generation with Al and ensure that the content produced is fair, ethical, and aligned with societal values.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. ar arXiv:2005. 14]65 [cs] (June 2020). http: / //axxiv. or org/abs 14165 00030 arXiv: 2005. 14165.
References
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM con- ference on fairness, accountability, and transparency. 610-623.
Chen Cao. 2023. Scaffolding CSI Courses with a Large Language Model-Powerts Intelligent Tutoring Sys- tem. In Companion Proceedings of the 28th Inter- national Conference on Intelligent User Interfaces. 229-232.
Adithya Bhaskar, Alexander R. Fabbri, and Greg Dur- rett. 2022. Zero-Shot Opinion Summarization with www.xx..org/ab//2//002 15914 arXiv:2211.15914 [cs].
GPT-3.
Tanner Bohn and Charles X. Ling. 2021. Hone as You Read: A Practical Type of Interactive Sum- marization. http://arxiv.coe wwtp//r////////////e// 02923 arXiv:2105.02925 [es].
Haw- Haw-Shiuan Chang. Jiaming Yuan, Mohit lyyer, and An- drew McCallum. 2021. Changing the Mind of Trans- formers for Topically-Comtrollabbe Language Gen- eration. http:// http::/axxvvoorooeee /b333 arXiv:2103. [cs].
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis Dora Demszky. Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, I.i Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman. Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle
Ruijia Cheng, Alison Smith-Renner, Ke Zhang, Joel Tetreault, and Alejandro Jaimes-Larrarte 2022. Map- ping the Design Space of Human-Al Al Interaction in Text Summarization. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. Association for Computational Linguistics, Seattle, United States, 431-455. https: adoi..og//i.oooooooooooooo
/ //doi. org/ /0/. ww.1388111222222 naacl-main 33
Peng Chu and Anita Komlodi. 2017. TranSearch: A Multilingual Search User Interface Accommodating User Interaction and Preference. In Proceedings of the 2017 CHI Conference Extended Abstracts on Hu- man Factors in Computing Systems. ACM, Denver Colorado USA, 2466-24772 https://doi. org//00. 1145/3027063.305222

---

Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten van Someren, Lloyd Rutledge. Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of content- based art recommender. User Modeling and User- Adapted Interaction 18, 5 (Nov. 2008), 455-496. https://doi. org/10. 1007/ 088-9951-3
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler, 2018. Reliability and Learnaillity of Human Bandit Feedback for Sequeence-o-Seqqeeee Reinforcement Learning. http://arxiv. wwtps/iioorooooooooo..o org orggabs///eeao aws/aaas.,... 10627 arXiv:180S.10627 [cs, stat].
William H Kruskal and W Allen Wallis. [n.d.] d.). Use of Ranks in One-Criterion Variance Analysis. (In. d.J..
Hai Dang. Sven Goller, Florian Lehmann, and Daniel Buschek. 2023. Choice over control: How users write with large language models using diegetic and non-diegetic prompting. In Proceedings of the 2023 CHI Conference on Human Factors Computing Systems. 1-17.
Tsz Kin Lam, Julia Kreutzer, and Stefan Riezler. 2018. A Reinforcement Learning Approach to Interactive- Predictive Neural Machine Translation. http: //arxiv.org org.
www.:/oggg////////////o /a//a/////..... 01553 arXiv:1805.01553 [cs, stat]. Zijian Ding and Joel Chan. 2023. Mapping the Design Space Interactions Human-AI Text Co-creation Tasks, arXiv e-prints (2023), ar Xiviv-2333.
Mina Lee, Perey Liang, and Qian Yang. 2022. CoAu- thor: Designing a Human-Al Collaborative Writing Dataset for Exploring Language Model Capabilities. (2022), 19. https://doi. watgg///..11 110...115555 1145//99...////91102.. 3502030
Zijian Ding, Arvind Srinivasan, Stephen MacNeil, and Joel Chan, 2023. Fluid Transformers and Creative Analogies: Exploring Large Language Models' Ca- pacity for Augmenting Cross-Domain Analogical Creativity. In Proceedings of the 15th Conference on Creativity and Cognition. 505-
Vera Liao and Jennifer Wortman Vaughan. 2023. Al Transparency in the Age of LLMs: A Human- Centered Research Roadmap. arXiv preprint arXiv:2306.0199 01944 (2023).
Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge Trimmer: parse- and- trim approach to head- line generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop Vol. 5. Associ- ation for Computational Linguistics, Not Known, 1-8. https://doi. org/10. 3115/1119466 1104467.1119445 1119468
Chien-Liang Liu, Chia-Hoang Lee, Ssu-Han Yu, and Chih-Wei Chen. 2011. Computer assisted writing system. Expert Systems with Applications 38, (Jan, 2011), 804-811. https://doi.org 10. /
eswa. 2010.
Daniil Gavrilov, Pavel Kalaidin, and Valentin Ma- lykh. 2019. Self-Attentive Model for Headline Gen- eration.
Konstantia Lopyrev, 2015. Generating News Headlines with Recurrent Neural Networks. http://arxiv, org/abs/1512, 01712 arXiv:1512.01712 [cs].
wwwrxivv.or/aaae wtgg/ba///////// 19911007811 McXiv:1901007780 [cs].
Sebastian Gehrmann, Hendrik Strobelt, Robert Krüger, Hanspeter Pfister, and Alexander M. Rush. 2019. Vi- sual Interaction with Deep Learning Models through Collaborative Semantic Inference. / org/abs/19e) 10739 190i::197.11000 [es].
H. B. Mann and D. R. Whitney. 1947. On Test of Whether one of Two Random Variables Stochasti- cally Larger than the Other. The Annals of Math- ematical Statistics 18, (1947), 50-60. http: wwww.jstoon orggstaalee/22611
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News Summarization and Evaluation in the Era of GPT-3. http://arxiv htt.vorggabb////// 12356 arXiv:2209.1223 12356 [cs].
Steven Moore, Richard Tong, Anjali Singh, Zitao Liu, Xiangen Hu, Yu Lu, Joleen Liang, Chen Cao, Hassan Khosravi, Paul Denny, et al. 2023. Empowering edu- cation with lims-the next- gen interface and content generation. In International Conference on Artificial Intelligence in Education. Springer, 32-37.
Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating Represen- tative Headlines for News Stories. In Proceedings of The Web Conference 2020. ACM, Taipei Taiwan, 1784. https://doi.. org/10. . 1145/3366423. 3388247
Hiroyuki Osone, Jun-Li Lu, and Yoichi Ochiai. 2021. BunCho: AT Supported Story Co-Creation via Un- supervised Multitask Learning to Increase Writers' Creativity in Japanese. In Extended Abstracts of the 2021 CHI Conference on Human Factors
Cheng-Zh Zhi Anna Huang, Hendrik Vincent Koops, and Ed Newton-Rex. 2020. AI SONG CONTEST: HUMAN-AI CO-CREATION SONGWRITING. (2020).
Computing Systems. ACM, Yokohama Japan, 1-10. https://doi. org/10. 11111.....14// 1145/3411763 345019 3450391
Savvas Petridis, Nicholas Diakopoulos, Kevin Crow- ston, Mark Hansen, Keren Henderson. Stan Jastrzeb- ski, Jeffrey V Nickerson, and Lydia Chilton. 2023, Anglekindling: Supporting journalistic angle ideation with large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Comput- ing Systems. 1-16.
Ziwei Ji. Nayeon Lee, Rita Frieske. Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung, 2023. Survey of hal- lucination in natural language generation. Comput. Surveys 55, 12 (2023), 1-38.

---

Emilee Rader, Kelley Cotter, and Janghee Cho. 2018. Explanations as Mechanisms for Supporting Algo- rithmic Transparency. In Proceedings of the 2018 CHI Conference Human Factors Computing Systems. ACM, Montreal QC Canada, 1-13. https: //doi.org// /10. 1145/3173574 31775777777 3173677
Peng Xu, Chien-Sheng Wu, Andrea Madotto, and Pas- cale Fung. 2019. Clickbait? Sensational Head- line Generation with Auto-tuned Reinforcement Learning.
http//aav..g..aab...e//o/e/////e .35882 1rXiv:1999033522 [es].
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ip- polito. 2022. Wordcraft: Story Writing With Large Language Models. 27th International Confer- ence on Intelligent User Interfaces. ACM, Helsinki Finland, 841-852. https://doi.ors 1145/ 3490099. 3511105
Rudolf Rosa, Ondiej Dušck, Tom Koemi, David Mareček, Tomás Musil, Patrícia Schmidtová, Do- minik Jurko, Ondřej Bojar, Daniel Hrbek, David Košt' ák, Martina Kinská, Josef Doležal, and Klára Vosecká. 2020. THEaiTRE: Artificial Intelligence Write Theatre Play. http://arxx //wxxva..vvv.///a///// wrgg org/abs/ 2006. 14668 arXiv:2006.1 14668 [es].
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization, arXiv preprint arXiv:2301. (2023).
Ben Shneiderman. 2022. Humman-centered AI. Oxford University Press.
Yun-Zhu Song, Hong-Han Shuai, Sung-Lin Yeh, Yi- Lun Wu, Lun-Wei Ku, and Wen-Chih Peng. 2020. Attractive or Faithful? Eapularity-Reinforee Learn ing for Inspired Headline Generation. Proceed- ings of the AAAI Conference on Artificial Intelli gence 34, 05 (April 2020), 8910-8917. //doi. //0g//0.00 609/aaa1, v34105.644
https:
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. [n. d.]. Learning to summarize from human feedback. d.]).
Hendrik Strobelt, Jambay Kinley, Robert Krueger, Jo- hanna Beyer, Hanspeter Pfister, and Alexander M. Rush. 2022. GenNI: Human-AI Collaboration for Data-Backed Text Generation. IEEE Transactions on Visualization and Computer Graphics 28, (Jan. 2022), 1106-11 16. https://dol: wor.oooo.oo////ooooooooooooooooo orgoorgg/// 10. 1109/ TVCG. 2021, 3114845 1rX1v:2110.10185 [es].
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao, 2017. From Neural Sentence Summarization Headline Gener- ation: Coarse-to-Fine Approach. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. International Joint Confer- ences on Artificial Intelligence Organization, Mel- bourne, Australia, 4109 4115. https://dol org/ 10. 29.24993/ijccl 2017/574
Boxin Wang. Wei Ping. Chaowei Xiao, Peng Xu. Mostofa Patwary, Mohammad Shoeybi. Bo Li, An- ima Anandkumar, and Bryan Catanzaro. 2022. Ex- ploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models. http: 84173 1rXiv:2202.04173
//arxiv. [cs].
Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human- Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces. ACM, College Station TX USA, 402-412. https://doo /oi. org/10. 145/ 3397481 3450656

---

APPENDIX
E Perceived Control and Trust
A Prompts for tasks
Table 7 presents the self-reported trust in the sys- tem (trust) and feelings of ownership over the final headline (control).
To determine the optimal prompts for extract per- spectives. generate headlines, and generate head- lines w/ perspectives with the GPT-3.5 API, we conducted multiple configuration tests, ultimately selecting the zero-shot learning paradigm (with no examples of news headlines included) for extensive coverage of news topics.
F Qualitative feedback on Al-assisted headline generation
Here are the finalized prompts for the corre- sponding tasks:
After the headline generation task, participants were asked for open-ended responses as a supple- ment to their perceived task difficulty and were invited to share "any other comments you want to tell us". These responses illuminated perceived advantages of Al assistants, for example in facili- tating brainstorming: "Overall, I'd say this system could definitely be helpful as long as the keyword list lines up with the main keywords in the arti- cle. Additionally, it would always be helpful brainstorming tool for headlines." (P40, Guidance + Selection + Post-editing)
extract perspectives: "Please identify five key- words and key phrases (nouns) for the fol- lowing news article. \nNews article:n + [news article] 'nFFive keywords and key phrases (nouns), separated with commać'n"
generate headlines: "Please come up with three high-quality (attractive, clear, accurate and inoffensive) headlines for the following news article. AnNews article:AnIn + [news arti- cle] TnhnThree high-quality headlines:An"
Some participants proposed specific considera- tions for the design of future Al-assisted systems: 1) inclusion of main points Al-generated head- lines "Some of the main points were left out of certain headlines, but otherwise, this proves to be useful tool for journalists. The main points were combined into easy-to-use database for selee- tion." " (P23, Guidance Selection); 2) high effi- ciency but limited creativity and reasoning "I could see how this would be beneficial in situa- tions where time limited, but does not replace the creativity and reasoning a human has. Some seemed spot on and others seemed chaotic. But with few tweaks, it was easy to create a headline. Some of the keywords suggested seemed totally unnecessary. The keywords were the weakest part of the system my opinion." (P39, Guidance Se- lection Post-editing); 3) guidance on AI model and selection of Al outputs as a non-linear pro- cess "I do wish could have gone back to choose different perspectives and see alternate variations than just the 3 original headlines. If those first 3 options don't 't really cover the central theme of the article, it would be nice to go back and choose al- ternate perspectives see if populates precise headline. Overall though, this is great tool to save time and many headaches. I really enjoyed this experience and would love to participate again in future exercises." " (P28, Guidance Selection)
generate headlines w/ perspectives: "Please come up with three high-quality (attractive, clear, accurate and inoffensive) headlines for the following news article including key- word(s): [keyword(s) selected by partici- pantsJAnNNews article:\n'n [news article] + in'nThree high-quality headlines:\n"
B Annotator Rationale Case Study
Table 4 illustrates the rankings assigned by an an- notator to six different headlines, each generated under unique condition, for a single article. Ac- companying comments provide valuable insights into the annotator's 's ranking rationale.
Task Time
Table 5 presents the time spent on different parts of the task-reading the article, creating the headline, and the total time for both reading and creating the conditions.
D Perceived Task Difficulty
Table presents the self-reportee instance-leved and overall task difficulty ratings between conditions.

---

Type Manual only
Title
Rank
Comment
National Army Guard Demands Sol- 6 diers be Vaccinated against Covid-
The title of the Army is wrong and it's also not the subject of the article.
19
Selection
Army National Guard facing recruit- ing crisis amid vaccination deadline
This title does not include COVID which is prominent subject and good for SEO considering the article is about it
Guidance
Army National Guard misses Covid 2 vaccine deadline, faces recruiting cri-
This title is good well. The topic of recruiting crisis should be out before the COVID vaccine deadline This the main subject of the ticle. Just a highlight in the article
Selection
sis
Guidance
Army National Guard Members Refuse Vaccine, Face Potential Expul-
Selection
Post- eost-nd
sion
Original
Deadline passes and in 10 Army 4 National Guard soldiers still unvac- cinated for Covid. Will they be ex- pelled?
This title sums up the article but it needs to be more concise. The sen- tence is too long.
GPT only
Army National Guard Faces Recruit- ing Crisis Amid Covid Vaceine Dead- line
Clear title Contains subject and issue that has arised.
Table Representative evaluation task, showing one (of 20) annotators evaluation of one (of 20) articles. Headlines are ranked from best (1) to worst (6).
Condition / Time Duration (s)
Reading
Creating
Total
mean
std
mean
std
mean
std
Manual only
1711
953
2248
1733
3959
1887
Selection
2723
1503
644
313
3367
1417
Guidance Selection
1998
1581
838
379
2835
1634
Guidance + Selection + Post- editing
1362
885
1961
1588
3324
1700
H-value
p-value
H-value
p-value
H-value
p-value
4.2
0.24
13.0
0.005
1.9
0.59
Table Duration (in seconds) of reading time (the time spent on reading the news article), creating time (the time
spent on generating the news headline), and total time. The differences were tested with Kruskal-Wallis H test.
Condition Difficulty
Instance-level
Overall
mean
std
mean
std
Manual only
30.9
17.1
1.9
1.0
Selection
25.0
19.4
1.9
0.7
Guidance + Selection
19.7
24.3
1.7
0.5
Guidance + Selection + Post- bost-ed
20.3
25.1
1.9
0.7
H-value
p-value
H-value
p-value
4.0
0.26
0.4
0.94
Table 6: Participants self-reported instance-level task difficulty rating (average across all instances, from 0 100) and overall task difficulty rating (Likert-scale, from 1 to 5) for four conditions. In both cases, higher scores means more difficult.

---

Statements
Manual only
Selection
Guidance Selection
Guidance Selection Post-editing
H-value p-value
[Control] could influence the final headlines for the news articles.
3.8 (0.7) 3.4 (1.3)
3.9 (0.7)
(1.
1.6
0.66
[Trust] trusted that the system would help me create high-quality headlines.
3.9 (1.1) 3.6 (0.7)
3.6 (0.5)
3.7 (0.8)
0.8
0.85
Table 7: Mcans, standard deviation (in parenthesis) and Kruskal-Waliis H test results of users' level of agreement to statements of participants' control and trust on a Likert scale of (Strongly Disagree) to 5 (Strongly Agree) for Condition I, and 3. No significant difference observed for control trust among the four study conditions.
G Interfaces for human study
The screenshots of the interfaces for the human study were attached below. All the practice rounds and human studies shared the same interfaces and interactions. We took the first round of practice as an example to demonstrate the interfaces for four Al-assisted conditions: Manual only, Selec- tion, Guidance + Selection, and Guidance + Selec- tion + Post-editing.

---

LLIGN
Inasruct and Writing News Headlines
cuiddl
headlines artcies tefore tawk, revieer
High quality artide headlinns and news artcies.
headlines for two
During the task, you will write hoaditnes 20 news artides, and atterwads, you wil fiI
short survey your experience. The whole process take about on hour
Figure GI: General task instruction for participants.
heedines
What makes Nigh-quaiity headline? high quality headiine should Toste- Atradivanees CarttyTTuuh (TACT) test.
Taoe: is in good laste? there anything possiily offersive the headlive? Can the wrong way?
anything
Attaacivenesss sttractive
engaging
Carlty: communicate the there words double meanings that could contuse the roadert
artide? is cimple?
volce
Truth: In accunate? Are the proper words terma from the artide used the heacline?
Is the headlins factually comoc??
How high-quality haadline?
headlins..
Undersland peglangle. the Koyword syskam: select from artidee Select words than reflect the central theme the artice
Summmrize the nons ortice and be bridl.
Highight trinuus, contratt contt
therne, using keywords.
Figure G2: Instructions how generate high quality news headlines for participants.

---

This is practice: article 1/2
Please read the news article below carefully.
The United Nations Security Council vated unanimously on Friday to extend the mandate of the U.N. Integrated Office Halti, which would otherwise have expired Friday, for year, but not provide additional security violence
assistance to the
by faces its
adopted resolution, between the United States and China. Beijing had been pushing for stronger language that would impose ban on arms and munitions sales to Haltian gangs.
Mexico, extensive consultation
"The mandate reflects the challenges facing Haltt, including the need address illegal arms trafficking and Illicit financial flows, and that the Security Council is ready to consider taking measures, es appropriate, to address challenges, Jeffrey Delaurentis,
diplomat,
council
"We must also note It is long past time Halt''s stakeholders to set aside their differences, added, "and reach agreement political framework that will allow Halti hold presidential and legislative elections when conditions permit."
A representative Mexico, Juan Ramón de Fuente Ramirez, told the Security Council that the resolution "calls for cessation of violence."
The U.N. political mission in Halti was established in 2019 to promote stability in Port-au-Prince.
ongoing security since the assassination of its president last year, led some call for more robust security package to accompany the renewel.
devastated by gang
political paralysis
Continue Write Headline
Figure G3; Interface for news article reading (same for all conditions).
This is a practice: article 1/2
unarimously Councl voted
Plooee nrite headline.
mandari
Integrated Office fartt, would have expired Friday, one year but not provide additional security assistance the country as foces Is gravest spate vielence years.
The council adopted the resolution, drafted by the United States and Mextoo, after extendive consultation betwoon United States and China. Beijing had been pushting stronger languoge would impose and munitions
Subrit Headline
Italtan
mandate
challenges
including
taking
challenges," Jeffrey DeLaurents,
diplomat,
must alto note long past Haatt's stakeholddrs to set aside their differences" added,
"and reach ag agrcement on political flamework that will allow Hais to hold presidential and legsslative etectons when conditions permit"
represeetativv Mesico, Juan Ramáó de la Fuente
Figure G4: Interface for Manual only condition of news headline generation.

---

This is practice: article 1/2
genersted
Security Council of violence in Haiti
Please select the best headline and click "Submit Heedliiee.
The United Security Counci
unantmously friday estend the
UK Security Coundil unanimsucly exends mandate for Integrated Office Harti
U.N. Integrated Office Halls which would otherwise have explred Friday, for one your but did not provide additionll security assistance to the country as it faces Its gravest spate of violence in years.
traffikingg
UK Security Counci calls for crecation of videece in Hatt
The council adopted the resolution, dratted the United States and Medoc, after extensive consuitation between the United States and Chino, Beijing had been pusling stronger language that would impose arma munitions sakes Hattian
"The and
Halti traffickigg diplomat, challenges" appropriats,
dificit (100)
Nost
long past staischelders set aaide differencss" added, reach agneement political frameworn will slow Helt hord preditental and legiciative etections wihee conditions nerma""
Figure G5: Interface for Selection condition of news headline generation.
practice:
Security
UNSC seeks to end violence in Halti
United Nations
United Nations Security Coundi
mandate
unanimoudy Friday extend mandate
Intogroted Office
which otherwise
Generate Headi nes
UN Security Council extends Halli mandate amid vilence
Beijing
been pushing for stranger language that ban and
Hallian gangs.
Hall violence continues as UN Security Council exten mandote
"The mandate reffects key challerges facing Halts including the need to addness illeal arms tratfiiingg and illiht financal flows, and that the Security Council is ready to consider taking measures, appropriate, to address these challenges, Jeffrey Delzurents, U.S. diplomat, bold the councl after the voto.
Very
Wey dffiuut
"We must also nobe it long past time for taatt's stoceholdders adide their differences"" adied,
Next
"and to reach agreemen ment on political framework that allow Hart hald presidential and logisiative etectionn sebeo con.itioos ooumnt"
Figure G6: Interface for Guidance Selection condition of news headline generation.

---

This practice: article 1/2
UN Security Counci
UN Security Council votes extend UN mission Haiti
Herti
mandate
Fridaw,
arts cales
security assittance country gravest spate
years.
Gerente headines
adopted reselution, drafted by Stabes and Hesdon, after extensive consultation
UN Security Council
between the United States and China, Beijing had
UN Security Council wama hotton gango potental sandtions
Blen pushing for stronger language that would impose ben on arms and munitions sales Hatian gangs.
"The mandete reflects the bay challengee facing nelt, including the need to addness illegl arme traffccing and iliit finandaa flowes and that the Security Council roady consider taking measures, approprioe lono, to address these challanges," Jeffrey DeLiurents, U.S. diplomat, told council after
Subrn freddine
How dificullt was to write this headline?
B1
Heilt's differences'
stakihhllders
Figure G7: Interface for Guidance Selection Post-ediiing condition of news headline generation.
The "Finish" button is clickable after you finish all the questions.
was difficult to write headlines for the news articles. [Strongly disagree (1) to Strongly (5))
Why feel this
Please write your answer here.
2. I could influence the final news headlines submit. [Strongly disagree (1) to Strongly agree (5))
3. trusted that the headline writing assistance would help me generate high-quality headlines. [Strongly disagroe (1)
Strongly agree (5))
4. Any other comments you want to tell us?
Please your
Figure G8: Interface of the post-study survey.


Reviews:
[{"review_id": "Kv4CCZVGE8", "reviewer": "Reviewer_YiDy", "summary": "", "questions": "A. Why do you choose that specific combination of selection, guidance + selection, etc.? Why not selection + post-editing or just each individual method by themselves?\nB. Newsline generation/editing is subjective to the audience and the selections, edits, and prompting depend heavily on the person and may not be preferred by others. Can you discuss how this may impact your experiment/results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and constructive suggestions offered by Reviewer YiDy, especially for recognizing that this study represents an initial exploration in a highly impactful domain, as well as for the points regarding the study design, discussion of results and documentation of participants’ behaviors. Although it was not feasible to provide a comprehensive treatment of all relevant aspects given the limitations of paper length and the nascent nature of this research area, deeply engaging with the comments has resulted in a number of impactful revisions we will make to the paper. Below, we address each raised point.\n\nGuidance Variant is Highly Constrained\n\nWe appreciate this insightful observation. The purpose of the guidance variant is to offer a controlled scenario that both allows for guiding the headline generation and assists the user to do so by suggesting possible directions (keywords). This is designed to help users unfamiliar with how to interact with an LLM, especially considering the study's timeframe in July 2022, and give some suggestions of possible perspectives (based on LLM analysis of the article content) without the user needing to come up with these alone. However, we recognize the benefit of further supporting user-specified keywords, allowing a more flexible interaction. We will update the Discussion to include this suggestion of providing users the choice between system-generated and user-generated keywords as a future variant.\n\nSubjectivity Not Addressed, and Subjectivity Based on Audience\n\nWe concur that subjectivity is a significant factor, particularly in the evaluation of headlines. To counteract this, we employed the Taste-Attractiveness-Clarity-Truth (TACT) framework as a standardized evaluation rubric. This framework offers concrete benchmarks across four dimensions to minimize individual evaluator subjectivity (Section 3.2.2). Despite having a robust evaluation framework and recruiting evaluators with editorial experience, this subjectivity may account for discrepancies between human-selected and solely AI-generated headlines. We will elaborate on this aspect in the Discussion.\n\nScale of the Study\n\nWe acknowledge the limitation in scope due to the specialized skills required for both writing and evaluating headlines. However, our sample size (40) is consistent with similar human-AI interaction studies, such as the 22 participants in the study conducted by Li et al. (2023). The aim was to provide an initial understanding of different interaction variants, and future studies should expand on this with larger participant samples and more diverse skill sets. We include this as a suggestion for future work and in the Limitations. \n\nEvaluating Different Models\n\nWe acknowledge the importance of evaluating different language models, while our study's primary focus was on human-AI interaction. Building upon Cheng et al. work, which used a 'Wizard of Oz' setup, we employed a real-world system with GPT-3.5 with state-of-the-art performance at the time of our research. Adding performance as an additional factor (and comparing additional models) would have complicated the study design making it hard to understand our key factor: the three interaction variants. We agree that examining the impact of different models (performance) is a valuable avenue for future work and we will add this to the Limitations and Future Work sections.\n\nChoice of Combination Methods\n\nOur goal was to create a continuum of human-AI interaction methods ranging from less to more control. The layering approach of selection, guidance, and post-editing was informed by prior literature (Cheng et al.) and allows us to assess the impact of each new layer by comparing adjacent conditions. This will be further clarified in the paper.\n\nWe hope these responses address the concerns of Reviewer YiDy and help clarify the contributions and limitations of our work. Thank you for considering our paper for EMNLP 2023!\n\nReference\n\nLi, Daniel, et al. \"Improving Automatic Summarization for Browsing Longform Spoken Dialog.\" Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presented a well-designed study aside from a few points in an impactful domain, given the increasing popularity of LLMs in downstream tasks such as news headline generation. There is signficant discussion of results and many behaviors of participants (e.g., post-editing) is well-documented/explained. ", "reasons_to_reject": "Generally, the weaknesses of this paper can be summarized as:\n1. The guidance variant is highly constrained, only allowing certain keywords to be emphasized. Ideally, this should be user-specified.\n2. Subjectivity is not addressed in this work - indviduals and audiences have preferences as to what constitutes a good headline\n3. The scale of the study made it difficult to observe significant trends, especially for RQ2 and RQ3. Furthermore, since experience varied significantly, many trends seemed obscured. \n\nOne additional note: I would have liked to see how different models (e.g., GPT-3, fine-tuned BART for headline generation) impacted perceived difficulty of writing as well as quality. This would provide insights into whether more expensive models are actually beneficial for this task.", "questions_to_authors": "A. Why do you choose that specific combination of selection, guidance + selection, etc.? Why not selection + post-editing or just each individual method by themselves?\nB. Newsline generation/editing is subjective to the audience and the selections, edits, and prompting depend heavily on the person and may not be preferred by others. Can you discuss how this may impact your experiment/results?", "paper_topic_and_main_contributions": "This paper studies human-AI text co-creation (i.e., humans writing with different levels of AI assistance) in the context of news headline generation through a well-designed human subject evaluation study. They consider what types of assistance are most effective for helping humans write headlines, how the types of assistance affect the perceived effort of writing, and how crowd workers’ interactions with LLMs affect their trust/feelings of ownership of the headlines.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "In the related works, might be relevant to add “Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning” (Xu et al., 2019)"}, "extra_text": {}}, {"review_id": "rvmx2QAp9Y", "reviewer": "Reviewer_Nsef", "summary": "", "questions": "QA. Novelty: Can you specify exact points of distinction from Cheng et. al, since the interactions paradigms and evaluation measures are both borrowed from that work?\n\nQB. How often did you notice that GPT3 generated headlines were absurdly factual? Could it be that factuality or verbosity were factors that confounded the *quality* of headlines? How exactly would you define quality in this case?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the detailed feedback and constructive criticisms offered by Reviewer Nsef, especially the remarks on the timeliness of our work, our meticulous study design and detailed presentation. Despite the constraints of paper length and the emerging nature of this research area, your insights have been instrumental in shaping impactful revisions that will enhance both the quality and depth of our paper. Herein, we respond to each of the points.\n\nEvaluation Measures Underspecified, especially for Verbosity and Factuality\n\nThank you for highlighting the need for rigorous evaluation measures in our study. To address this, we've employed the Taste-Attractiveness-Clarity-Truth (TACT) framework, a well-established and standardized rubric commonly used in journalism education (http://www.columbia.edu/itc/journalism/isaacs/client_edit/Headlines.html), as our measure for evaluating headline quality. Referenced in both the main text and Appendix G2, this framework specifically addresses critical quality metrics such as clarity and truthfulness. It prescribes guidelines across four dimensions to limit evaluator subjectivity (Section 3.2.2) and preemptively tackles potential pitfalls like verbosity and factual inaccuracies (with clarity and truthfulness) that may occur with GPT-3.5.\n\nWe also leveraged the expertise and experience of our evaluators to enhance the validity of our assessments. While we believe the TACT framework provides a robust basis for evaluation, we acknowledge its limitations in capturing all nuances of a model's performance. Therefore, we will discuss these limitations and propose directions for refining the evaluation criteria in Future Work.\n\nLimited Interaction Strategies\n\nThank you for your observation, which dovetails with the 'Guidance Variant is Highly Constrained' concern raised by Reviewer YiDy. We intentionally concentrated on the first three interaction strategies outlined in Cheng et al. to steer clear of the complexities inherent in multiple interaction rounds, which are generally better suited for longer text editing interactions. Nevertheless, we acknowledge the need for more adaptable interaction mechanisms and will actively consider incorporating such elements in our future research.\n\nNovelty Relative to Cheng et al.\n\nOur study builds upon Cheng et al.'s Wizard-of-Oz framework by employing an actual AI-powered system for text news headline generation and editing. This methodological advancement enables us to capture more realistic and reliable insights into human-AI co-creation, while still respecting the original framework. Building a functional AI system enables measurements that are more reflective of genuine human-AI co-creation with realistic AI performance. We will clarify this novelty in comparison to Cheng et al. in the paper.\n\nWe hope that the above responses adequately address the queries and concerns put forth by Reviewer Nsef. Your feedback has been invaluable in clarifying both the strengths and areas for improvement in our paper. Thank you for your time and consideration in reviewing our submission for EMNLP 2023!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "RA1: Timeliness: The paper provides a timely analysis of different human-AI interaction paradigms for a co-writing task. Given the prevalence of GPT3 in writing tasks in the present, the paper is a good attempt to quantify the benefits on interacting with LLMs along three measure axes.\n\nRA2: Study Design: The authors did a great job at designing a controlled experiment study, as well as presenting details of their study.", "reasons_to_reject": "RR1: Evaluation Measures: Evaluators were ranking generations by GPT3.5, Human-only and three other assisted settings. However, there is no control for the reasons evaluators' rankings. GPT3.5 is known to be very verbose even in short generations, and for settings like news which GPT3.5 would have probably even been trained on, there is no guarantee that their generations were actually better or just verbose for that matter. While the paper reports these reasons in Table 4 (appendix), they are not accounted during the actual quality evaluation, which makes the actual evaluation underspecified.\n\nRR2: Interaction strategies: The paper only uses the first three interaction strategies from Cheng. et al. The writers are not allowed to prompt or interact with the model, and is thus only constrained to guide the generation (Setting 2) based on the options they are provided (which are also generated via prompting). Firstly, not only is this constrained to the quality of keyword generations by GPT itself, there is no option for writers to abstain from GPT's selections (in case of low-quality generations). Furthermore, given that annotators are not actually allowed to interact with the model (Setting 4/5 in Cheng et. al), model guidance in terms of keywords is not a realistic guidance setting. \n\nUpdate post rebuttal: The authors have addressed RR1 with satisfactory arguments.", "questions_to_authors": "QA. Novelty: Can you specify exact points of distinction from Cheng et. al, since the interactions paradigms and evaluation measures are both borrowed from that work?\n\nQB. How often did you notice that GPT3 generated headlines were absurdly factual? Could it be that factuality or verbosity were factors that confounded the *quality* of headlines? How exactly would you define quality in this case?", "paper_topic_and_main_contributions": "This paper proposes a study that compares how different human-AI interaction strategies (guidance, selection and post-editing) compare on the task of news headline generation based on three measures -- headline quality, task effort and perceived control and trust, based on prior work (Cheng et. al, 2022). The work operationalizes Cheng. et al (conducted on a simulated task) on a real-world text-generation setting. These interactions are conducted in a controlled setting with curated participants, where the participants involved in the interactions are separate from the evaluators.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. \"four\" --> \"three\" L367"}, "extra_text": {}}, {"review_id": "lyAP30lnFd", "reviewer": "Reviewer_DQTz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, Reviewer DQTz, for your comprehensive review and constructive suggestions, particularly your recognition of the value in exploring human-AI text co-creation and the contribution of our human-rated news headlines dataset. Given the limitations of space and the emerging nature of this field of study, your feedback has been instrumental in identifying areas for improvement, and we have made a concerted effort to address each of your points below.\n\nInsufficient Experimental Discussion\n\nWe appreciate the insightful comments concerning the depth of our experimental results section.\n\nOur results show that AI-generated headlines already possess a high level of quality, which enables them to outperform manually crafted headlines within a constrained time frame. Furthermore, the high initial quality of the AI-generated headlines likely explains why post-editing does not lead to significant improvements. \n\nTo address the notion that \"the quality of the condition combined with post-editing should be reduced with a high probability,\" we will delve into the differential experiences that manual crafting and post-editing offer. Specifically, post-editing permits humans to focus primarily on fine-tuning and minor adjustments, as the AI has already done much of the initial work. This stands in contrast to the more labor-intensive process of generating a headline manually from scratch.\n\nWe will incorporate these insights for a more comprehensive interpretation of the findings in the Discussion.\n\nLimited Number and Types of Articles\n\nWe appreciate the critique on the limited number of articles used for evaluating different conditions. The scope of our study was a first step beyond Cheng et al.'s Wizard-of-Oz approach, incorporating real system interactions. Given the constraints of our study's scale, our primary focus was understanding the differences between the three interactive variants rather than exhaustively exploring variations in articles and headline tasks. Future research should build on this  study through the inclusion of a more diverse set of articles, a longer-term assessment, and a varied participant pool. We will add these important points to the Limitations and Future Work. These combined efforts will offer a more nuanced understanding of the model's performance across a range of scenarios.\n\nExperimental Limitations and Task Difficulty\n\nThank you for pointing out the issue of \"accuracy of perspective selection\" and the potential impact of task difficulty. We acknowledge the limitation of not having evaluated the model's accuracy for generating perspectives, despite employing a state-of-the-art system available at the time and undertaking prompt engineering coupled with qualitative analysis to enhance the quality of generated perspectives. We are open to sharing the user-selected perspectives in our dataset for further scrutiny. As a future improvement, we are considering the incorporation of user-specified keyword options, which would enable us to assess the performance of system-generated keywords.\n\nRegarding task difficulty, we endeavored to minimize variability by choosing articles of similar lengths and from common domains, and all participants worked with the same set of articles. Nevertheless, we acknowledge that task difficulty could be a relevant factor in future, more expansive studies.\n\nWe hope that our responses successfully address Reviewer DQTz's concerns to clarify the scope and limitations of our research. Your feedback is greatly appreciated and will help strengthen our revised paper. Thank you for considering our work for inclusion in EMNLP 2023!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The exploration of human-AI text co-creation is valuable and could be of interest to researchers in the community. \n- The work provides a dataset of human-rated news headlines.", "reasons_to_reject": "- The paper is lacking some deeper discussion on the experimental results of using their method.  E.g., why post-editing has no significant effect and why the manual condition has the lowest quality, which are counterintuitive. Since the manual condition's effect is the worst, then the quality of the condition combined with post-editing should be reduced with a high probability.\n- I don't think 20 articles are enough to analyze the actual effects of different types, and instead of increasing the number of participants, consider increasing the number of different articles. As a straightforward example, choosing the best of the three outputs generated by the LLM, the results of 10 participants would not vary excessively.\n- The experiment is not sufficient. I am curious whether the accuracy of perspectives selection will have a significant impact on the quality of generated titles. If there is a task difficulty label, whether the effects of different types under different difficulty levels are considered, and whether the number of different task difficulty levels is balanced.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper adopts three interactive types of guidance, selection, and post-editing to evaluate the human-AI text co-creation in the news headline generation. The work experiments and analyzes the combination of three common interaction types in human-AI interaction for headline generation, and provides design guidelines and human-evaluated datasets. I think the idea is valuable, but it lacks the sufficient experiments and in-depth analysis.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the studies were well designed and the discussion was well written. Reviewer YiDy appreciated both. Reviewer YiDy also emphasized the timeliness of this analysis of different Human-AI Interaction paradigms. Reviewer DQTz appreciated the human-rated news dataset.

Reviewer YiDy had a valid concern that the guidance variant is highly constrained and it should be user-specified. The authors acknowledged it and suggested making edits to the discussion to reflect that. They also had a concern that subjectivity remains unaddressed in the work.The authors pointed to their use of TACT framework. Finally they had a concern about the scale of the study, and the authors pointed out that the same size is consistent with the prior work.

Reviewer Nsef also had similar concerns about lack of control in evaluation metrics as well as not allowing users to prompt/interact with the model. The authors also made the novelty relative to Cheng et al.’s wizard-of-oz framework, and pointed out that they used an actual AI-powered system for news text generation.

Reviewer DQTz had most notable concerns about the methodology including limited number of articles and task difficulty balance. The authors acknowledge it in the rebuttal.
Decision: Accept-Findings