The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions
Authors: Siru Ouyang
Shuohang Wang
Yang Liu
Ming Zhong
Yizhu Jiao
Dan Iter
Reid Pryzant
Chenguang Zhu
Heng Ji
Jiawei Han
Venue: EMNLP 2023 Main (2023)
Keywords: Large language models
Real-world NLP applications
User-GPT interaction analysis

Abstract: Recent progress in Large Language Models (LLMs) has produced models that exhibit remarkable performance across a variety of NLP tasks. However, it remains unclear whether the existing focus of NLP research accurately captures the genuine requirements of human users. This paper provides a comprehensive analysis of the divergence between academic research in NLP and the needs of real-world NLP applications via a large-scale collection of user-GPT conversations. We analyze a large-scale collection of real user queries to GPT. We compare these queries against existing NLP benchmark tasks and identify a significant gap between the tasks that users frequently request from LLMs and the tasks that are commonly studied in academic research. For example, we find that tasks such as "design" and "planning" are prevalent in user interactions but largely neglected or different from traditional NLP benchmarks. We investigate these overlooked tasks, dissect the practical challenges, and provide insights toward a roadmap to make LLMs better aligned with user needs.

The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions
Siru Ouyang", Shuohang Wang', Yang Liu', Ming Zhong', Yizhu Jiao', Dan Iter?
Reid Pryzant Chenguang Zhu?, Heng Ji' Jiawei Han!
University of Illinois Urbana- Chamappga Microsoft Azure AI
Atruo2@lllioois.
Abstract
as machine translation (Bahdanau et al., 2015) and document summarization (Tan et al., 2017).
Recent progress in Large Language Models (L.LMs) has produced models that exhibit re- markable performance across variety of NLP tasks. However, it remains unclear whether the existing focus of NLP research accurately captures the genuine requirements of human users. This paper provides a comprehensive analysis of the divergence between current NL.P research and the needs of real- world NLP appli- cations via large-scale collection of uSe conversations. We analyze scage-scale collec- tion of real user queries to GPT. We compare these queries against existing NLP benchmark tasks and identify significant gap between the tasks that users frequently request from LLMs and the tasks that commonly studied in aca- demic research. For example, we find that tasks such as "design" and "planning" are prevalent in user interactions but are largely neglected or different from traditional NLP benchmarks. We investigate these overlooked tasks, dissect the practical challenges they pose, and provide in- sights toward a roadmap to make L.LMs better aligned with user needs.
Most recently, large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022) such as CharGPT, emerged as powerful tools capa- ble of achieving unprecedented success across a broad spectrum of NLP tasks (Jiao et al., 2023; Hendrycks et al., 202 Clark et al., 2018). These models have become accessible and popular among non-NLP experts, opening the door for many new user applications.
The flood of new applications and the sharing of user interactions with LLMs (Tay et al., 2023) provide a great opportunity to closely examine the distribution of real applications users need on a daily basis. After detailed analysis, we identify conspicuous gap between real-world user queries and established NLP benchmarks, suggesting an- other shift in NLP focus is needed. To systemat- ically analyze the phenomenon and to bridge the gap, we conduct battery of experiments aiming to examine the following aspects:
What is the distribution of real-world user queries in terms of domain and task types, and how do they shift from traditional NLP benchmarks ($ 3)?
1 Introduction
Over the past years, the NLP community has wit- nessed several paradigm shifts in technology fol- lowed by renewed research focus on applications that test the limits of this technology (Sun et al., 2022). For example, distributed word representa- tions (Landauer et al., 1998; Mikolov et al., 2013) enabled better characterization of the semantic similarity between words, entailing NLP research gravitated towards tasks like sentiment analysis and dependency parsing (Klein and Manning, 2003). Subsequent technologies like the transformer ar- chitecture (Vaswani et al., 2017) and contextual word representations (Devlin et al., 2019; Peters et al., 2018) further expanded the space of possible applications and the edge of NLP research, such
What are the emerging tasks and requirements from real-world user queries that may be over- looked in previous studies ($ 4)?
We start by investigating ShareGPTI, large- scale collection of user-GPT conversations in the real world, containing 94,145 145 split data samples. ShareGPT has been used for training powerful LLMs (Chiang et al., 2023; Xu et al., 2023) and incorporated into new datasets (Zheng et al., 2023; Gudibande et al., 2023), both showing substan- tial advantages. Specifically, we design an annota- tion framework where we employ GPT-4 (OpenAl,
"ook partially done during internship at Microsoft.

---

2023) to generate the related information for every user query that appears ShareGPT. We subse- quently delve into the obtained data and conduct comprehensive analyses to answer the aforemen- tioned questions". We summarize our key findings as follows:
requirements expressed in user queries. We chose to employ GPT-4 conduct a self-demonstrated an- notation due to its superior precision and coverage. The annotation process consists of three stages: 1) thain- prompting, 2) demonstration sam- pling, and 3) demonstration pool expansion.
1. Generally, real-world user queries demon- strate tendency towards more aligned with daily life with enlarging diverse user bases.
Chain-of-thought prompting. Chain-of-thought (CoT) (Wei et al., 2022) uses intermediate steps for text generation and improves the performance of LLMs (Chen et al., 2023). To augment GPT- 4's proficiency in comprehending and interpreting queries, craft our instructions in manual- CoT style, compelling the LLM to deliver the req- uisite information in a methodical, step-by-step manner. Specifically, we first ask LLM to iden- tify the domain or topic related to the user query. Then, the LLM is prompted to generate a concise one-sentence summary for the given user query as a reference. Finally, drawing on the insights obtained thus far. the LLM is capable of devising creative and accurate task types corresponding to the user query. The generated task types are fine-grained and diverse, spanning from email editing dream analysis. Overall, we obtain 13, 783 task types and 8, 392 domain labels for all samples ShareGPT.
2. We discovered several lasks, including provid- ing advice, designing, planning, etc., that are seldom touched and pose new requirements in the era of LLM.
3. We summarized the shifting trends and chal- lenges, providing insights to fill the gap for both stakeholders and users.
2 Methodology
this section, we employ GPT to annotate the topic/domain and task type of every sample in the ShareGPT collection. The whole annotation and post-processing pipeline is illustrated in Figure 1. We use human evaluation to verify the quality of our automatic annotation.
2.1 ShareGPT
Demonstration sampling. While the CoT prompting can generate reasonable annotation, is known that in-context demonstrations can further improve LLM output quality (Wang et al., 2022a). Thus, we select examples from CoT outputs to serve as demonstrations in the second stage. We initiate the demonstration pool with 20 samples of different domains topics and task types. For every sample, we randomly select k demonstrations from the pool and append them to the instruction in the first stage. The input and output format could be found Figure
ShareGPT is publically available large-scale col- lection of user-GPT conversation histories?. It is based on a Chrome Extension* where users can choose to upload their interactions with GPT. The version we used contains 94, 145 split user-GPT conversations and is previously used to train LLMs such Vicuna (Chiang et al., 2023). Every sam- ple in ShareGPT is a multi-turn conversation with utterances from both users and the GPT engine.
2.2
Seff-demonstrated annotation
The goal of anmotating each user query is twofold: to identify the underlying task user is intending to perform (task types), and to understand the subject matter or field (domains) the query pertains to. The annotation process is challenging because i) the an- notation requires comprehensive and suitable pre- defined set of task types and domains/topics, and ii) the annotation should accurately reflect the genuine
Demonstration pool expansion. To encourage diversity and avoid potential bias in demonstration selection (Wang et al., 2022b), we gradually expand the demonstration pool. Since we are asking GPT- 4 to generate free-form task types, one challenge here is to avoid generating too divergent task types. Therefore, we maintain a dictionary to document the time of appearance for every task type. If task type appears more than fixed ratio À among all the current samples, we then add the current sample containing the task type into the demonstration pool. By enforcing such constraints, the generated free-form task types could be better "clustered" for
*oode is available https: //gi gi thub. con/ozyyshr/
'The collection ShareGPT already contains 305,000 samples to the date of this submission, and is dynamically expanding. Since all the data are not publically available. we use the portion that could be downloaded from Huggingface. 'https: //tbthu..b com/ coooecclestonn weweeeeestteeeeeeeeeeeeeeeeeee shareegp

---

Manual-COT Instruction
Input: Advice
External dictionary
You will given query in user-GPT conversation. Please classify
queries with respect to task types, as fine-grained as possible following steps:
Suggestion, counseling,
(1) Identify specific domain/lopie. (2) Give brief summary.
(3) Generate lask types. Note there could multiple types.
Find summary sentences
Demonstration sampling
Demonstration pool
union
[user query]
been feeling really overwhelmed anxious lately because of my workload. How handle this stress? emotion, psychology. [summary] Seek advice manage stress
(oomain/oopic) [task type]
emotional support
Find original sumples
New user query.
[user query] Write 800-word blog about counting
[summary] Provide details on applying internship... [task type] job application advice.
[domain/topig hopie]
education, social media, Write blog .... including...
[summary] [task type] blug writing ... frequency threshold
GPT-4
Estimate Green Card timeline for EB2 [task type] jmmieration counseling. ...
[summaryy]
Self-demoostmteed unnotation
(b) Postpmeessing for amalysis
Figure I: General framework of how we conduct step-to-step and self-demonstrated prompting for annotation using GPT-4 (a), and the post-processigg for certain words/phrases (b).
further analysis.
Table 1: Human evaluation results in terms of complete- ness and correctness.
Experiment settings. We download ShareGPT from Huggingface", where the 51k conversations are split into 94k ones due to the length limit for input. For every step, At is set to 3 and À is 0.05. We concatenate 3 samples together and let GPT-4 generate the annotation once for the balance of speed and quality. To encourage diversity, we set the temperature to 0.4 and it takes around 10 days due to speed limitations in GPT-4 to annotate all the 94k samples in ShareGPT. We plan to release all the annotated results for future related research.
Completeness
Correctness
Score
0.95
1.76
K
0.96
0.83
for correctness, the scoring scale is 0 (incorrect), (partly correct) and (correct). The detailed rubric and the interface are shown in Appendix
Table demonstrates the results of human evalu- ation. We can see that with GPT-4 we got reliable annotations for ShareGPT. Notably, none of the 100 samples got "incorrect" annotations. Apart from the scores, (Fleiss, 1971) for each metric, both indicating "almost perfect agreement".
2.3 Human Evaluation
also calculate Fleiss kappa
To assess the quality of annotation produced by GPT-4, a human evaluation is conducted with a spe- cific focus on the generated free-form task types. We designed and distributed our human assess- ment task with Doccano". We recruited 3 graduate students our human assessors (all of which are paid as research assistants). The assessors all have rich experiences with related NLP and ML research but were not involved in the development of our framework. We randomly selected 100 samples for evaluation. For every sample, we ask the asses- sors to judge the generated task types in terms of completeness and correctness. This is to evaluate whether the generated task types are complete and faithful to the original user query. For complete- ness, the scoring scale is 0 (bad) and (good), and
2.4 Post-processing for analysis
As the domain/topic and task type annotations gen- erated by GPT-4 are free-form words and phrases, clustering the samples poses a significant chal- lenge. For example, "reccppe suggestions", "cook- ing tips" and "cooking advice" all belong to the same task type. To tackle this challenge, we pro- pose a post-processing framework that incorporates three stages shown in Figure 1: (1) statistical cal- culation based heuristic rules, (2) an ensemble with the embedding similarity of summary sen- tences, and (3) manual checking process to en- sure the best possible quality. Detailed narrations could be found in Appendix A.
'ht tps: http:://huuggngg http://gggggg..c/geeaaaaaaeeeeaa anon8231 28022231119233//33 10899123/SharGGPP_L SeiereeeP__iccaaaa wwwuuaauunfllteeeeeeeeeeeeeeeeee www.uultfeeerreeee
"https: / ttt..tt.tttttttee//e////ee... cww. /t/////ccoo//occcaoocoo

---

deriication (351%)
(5025)
explaration (7.0%)
wrilig (3015)
Hanaliing 12 1
Inarskiing
coencee (S.1)
letrndogy MaN
comcuntoon J030N
ShareGPT
ShareGPT Task Types
or wndglle 52%)
cool medea p00es
Domain/Topic
geneentenn 6e%
beagg 37%)
adviso momm
Stanes pLons
edcadon (7 2310
llwving (1915))
spore jarr raaery
(0.-85)
craate wridng (10000))
dars) sbbe cautton plarms
hoath (3315)
ditugging 13,2154
Brt (1 (8R)
police (331%)
JUSP%
sancra ectudio S
(5.4%%
classiicator (00.40))
acheoogg Oro0i))
vesccre (8.01144
frnr (019)
Huggingface Domain/Topiz
Hugginfface Task Types
rnforratioo 11%
macing worpoeaeeeee
dansiication
(aa0)))
5ow or dits Soalion (2041))
shosic (uidi))
aun wommmttttinn (1. 7990))
opani tenu
1 se%
Iamiatio 2990 undeniandnnggggg
forted (3.309g)
Brokk is puars)
(c)
mation maaionn ratrival (14.0%%)
(c)
Figure 2: Domain and task types distributions for ShareGPT user query and Huggingface data.
3 Overall Investigation
by our work.
this section, we first present the overall statistics and analysis of the ShareGPT clustering results. Then we compare the results with conventional NLP datasets.
3.1 Domain and Task Distribution
Based on our annotation and clustering results, We plot domain and task type distributions of ShareGPT queries in Figure 2 (a) and (b). "Tech- nology" shares the largest portion of ShareGPT's domain, comprising around quarter. Other no- table domains "education", "business" and "lan- guage" make up another quarter. For task types in ShareGPT, around one-fifth is about "code gen- eration", which corresponds to "technology" in domain analysis. "Creative writing" is the second largest portion. The rest of the task types quite diverse, composing around 2/3 of the whole set. In the following, we analyze the two mostly seen tasks ShareGPT, coding and writing assistance, representing 19.9% and 21.3% respectively.
For conventional NLP datasets, investigate 2,911 datasets from the Huggingface Datasets? col- lected by Yin al. (2023). These 2,911 datasets are filtered and selected from the original around 50k datasets the Huggingface platform with the following conditions: (i) Datasets with no license description or the license does not allow usage. For ethical considerations, the datasets collected are strictly following license restrictions. The num- ber of all datasets fulfilling license requirements is restricted around 16k. (ii) Datasets that non-English. Since our investigation focuses on the English language and ShareGPT contains only English samples, datasets collected from Hugging- face also follow this paradigm. (iii) Datasets that are related to multi-modal There is large por- tion of datasets in the Huggingface data platform related to multi-modal research. Since we tar- get NLP community and related topies, we clude those datasets. (iv) Datasets that are invalid. Some datasets in the Huggingface data platform are empty or cannot be downloaded, which is ignored
Coding assistance Pre-trained models for pro- gramming language have been widely explored in the NLP community (Chen et al., 2021; Nijkamp et al., 2023). Current benchmarks used for evalu- ation (Hendrycks et al., 2021a) are usually in the form of function generation code completion given certain requirements. More specific task set- tings include code search (Husain al., 2019). code translation (Chen et al., 2018), code clone detection (Svajlenko et al., 2014), and code refine- ment (Tufano et al., 2019). We do observe user
downloads

---

queries that are similar the previously mentioned task settings in ShareGPT, such as code genera- tion (18.6%) and code debugging (9.2%). This reflects that the current coding benchmarks are fit- ting with real-world scenarios. However, we still notice a non-negligible portion of requests involv- ing higher-level program understanding, such as code simplification and providing design pattern suggestions, which are seldom captured in existing task definitions. We also plot the proportion of the top 10 most frequent programming languages used in Figure 3.
and leverage GPT to classify the corresponding domain/topics. Similar to ShareGPT, technology- related domains also share large portion Hug- gingface. However, in Huggingface, political, legal, personal life, and film altogether contribute a large portion. We further look into the data source for Huggingface. We collect the information in meta- data "Dataset Card", where detailed information of the specific dataset introduced. For example, MRPC dataset (Dolan and Brockett, 2005) has the statement "a corpus of sentence pairs automatically extracted from online news sources". We then use GPT to annotate the corresponding data source as an approximation. We find that the majority of datasets are from Wikipedia and news, occupy- ing portion of over 80%. The rest mainly come from "government reports" and "QA forums". This phenomenon calls for dataset curation specific domains commonly seen in user queries, and the data source should also incorporate materials multiple formats apart from Wikipedia and news.
Writing assistance With advancements in the NLP field, writing assistance tools have shown potential beyond grammatical (Ng et al., 2014) and stylistic improvements, now providing aid in content creation and organization. Writing tasks such as story generation (Fan al., 2018) and style transformation (Shen et al., 2017) are pop- ularly explored in the community. Our analysis of ShareGPT usage confirms this trend. For in- stance, assistance in article drafting and editing accounts up to 5.1% of the writing assistance re- quests. Similarly, email editing makes up to 2.6% of the queries. These suggest users rely Al tools for professional writing communication.
Task Type Shift We categorize different tasks based on metadata statistics for all the datasets from Huggingface, as shown on the right side in Figure 2. We filter out tasks with numbers less than 10, and the rest samples make up to 99.8% of the original set. We observe that question answering and text classification are the top-2 task types, summing up to more than 2/3 of the whole collection. On the contrary, in ShareGPT, almost all the user queries are free-form text generations generations in a user-required manner.
Despite this, we notice a trend of creative writing for bunch of text formats, spanning from slogans to tutorial writing. Instead of leveraging LLMs to generate everything, we found a noticeable portion of "procedure writing" and "how-to-write" queries, underscoring the importance of explanatory and pedagogical writing aids.
Shifted and Overlooked Tasks
3.2 Distribution Difference with Conventional Datasets
In this section, we detail the overlooked tasks dis- covered in the analysis process of ShareGPT, with concrete examples shown in Table 2. Task selec- tion is based the distribution in Figure 2 that satisfies two criteria: (1) long-tail tasks of the dis- tribution, summing towards around 40%; and (2) tasks that are not overly skewed the distribution, constituting around 25% a comparable figure to the predominant segments. We also summarize the features for each task type in Table 3, along with the potential roadmap. Commonly seen topics and human analysis for performance of LLMs are shown in Figure 6 in Appendix D.
To provide comparison of ShareGPT queries with conventional NLP datasets,we we also annotate and cluster the collected 2,911 Huggingface datasets in the same way and present results in Figure (c) and (d). We then interpret the differences from two aspects, domain and task type.
Domain Shift We plot the top 20 most common and representative domains of ShareGPT in Fig- ure 2, which account for around 73% of the entire set. We observe that the domain distribution in ShareGPT quite diverse. Notably, there is large portion of technology-related topics. Commonly seen cases involve code writing and debugging. For Huggingface datasets, we randomly select 10 samples from each dataset as an approximation,
4.1 Task of Providing Advice
The task of giving advice occupies a portion of up to 3% in GhareGPT. The concept of a ma-

---

Table 2: Summary of the long-tail tasks discovered. "Il" indicates concatenated user queries in the same conversation.
Task type
Characteristic
Proportion
Example
Offering tailored
My dad, he's a rude person. he doesn't like me, makes clear. he calls me bad names, he has threatened to kick me out, should be concerned? considering people have suffered through worse need money but can't get it because working at my dad's puh is too much for my autism
acvice under given sccnarios
Advice
3%
Construction of some objects or for implementation of an activity
Can you help me think of some logo ideas for "Start Upcoutt? Can you give more similar ideas like "A playful cartoon scout charucter holding a tech startup icon"? II liked this idea: A cartoon scout character with binoculars and backpack. walking towards futuristic city skyline in the distance. Can you tell me more details?
Design
2.5%
Providing sequence steps to uchieve a pre-defined objective.
I'm going a road trip with my daughter on 3rd April. We will he leaving stay
London and want to in Nice, France on or 10th April where about days. We then need to be in Calais on 15th or 16th to take the Eurotunnel.
Planning
2.7%
Everything else is undecided. Can you give a plan for routes where we' ll doing no more than 5 hours driving per day. We have an EV and need to stop for charging.
Exchanging views or ideas on specific topic
Should we help local charities or overseas charities? Some people think that being rich and famous leads to happiness. There is also saying
Discussion
3.8%
"Money can buy comfort, hut it cannot buy happiness." What do you think?
Examination of target for its nature and structure
need some help analyzing a pocm that I've been reading for my English Literature class. The poem "The Road Not Taken" by Robert Frost. Thunk you, that's great
Analysis
7.3%
start. Could you help me analyze how Frost uses metaphor in this poem?
Determination of the subject's s, properties, based on given rubrics
want you act as a resume evaluator: Here are the rules: [Format]: Use a clear, easy-to-read font and suitable layout ... [Content]: Your resume should include... You should be able to read example url and anlyze it. Here are some good example for resume: <https://ocs.gooe bogle.com/...2 Each section means 20 point,
Evaluation
4%
total 100 points. Just show me explaination and points then organize to table.
chine offering advice has been present and associ- ated with NLP from as early as the 1970s (Short- liffe et al., 1973) with the development of ex- pert systems (Liao, 2005). At that time, giv- ing advice is usually performed with a consul- tation program (Scott et al., 1977), which plays the role of expert consultant in restricted do- mains, such as health. Since then, the field has evolved significantly, with expanding domains into legal (Pinkwart et al., 2006), finance (Radford, 2003), etc. The advising process back then is more like a QA system, where the user first input back- ground confined by rules, together with a direct question such as "Is Organism-1 rod or coccus (etc.)?". Later, advice was formally characterized multi-step process involving the analysis of the recipient's context and the formulation of response in natural languages (Saeidi al., 2018).
4.2 Task of Designing
Request to design some objects with specifications constitute 2.5% of ShareGPT queries. The task of designing progresses slowly in the field of NLP over the past half-century, and does not have a clear line of development. Previous NLP-related works generally focused on relatively narrow domain-specific topics, such as entity-relationship schema (Habib, 2019) and user interface (Sontakke and Pimpalkar, 2014).
While previous works focus more on the design of structured objects, such as database UI, user queries arising from ShareGPT usually seek de- signs for more complicated objects. spanning from questionnaires to websites. This indicates trend that the scope of design has broadened to include not only the structure of tangible objects but also conceptual frameworks and processes. Another significant difference is that design-related queries in ShareGPT ask for more diverse and creative requirements, especially for personalized objects such as icons. Additionally, in ShareGPT, we see instances where users demand designs with very specific constraints, such as adherence to a par- ticular color scheme in interface design. These "visual image design" tasks, though communicat- ing in texts, require the model to have a sense of visual understanding in order to generate aestheti-
Compared with previous advising systems that require rigid rules and were not designed for open- ended dialogue, user queries in ShareGPT are more free-form. Moreover, instead of focusing on re- stricted professional domains, these requests more akin to everyday tasks, such as relationship dilemmas and personal careers. This also presents the requests of more general, macro-level guidance, compared with providing micro-specific answers in previous studies.

---

cally pleasing outcomes.
litical debating (Mirkin et al., 2018). Another no- table characteristic is the non-interactivity in many previous works (Zhang et al., 2019; Ouyang et al., 2021), although they promote multi-turn genera- tion (Chen et al., 2020) for several fixed user inputs. However, user queries in ShareGPT are typically more dynamic and unpredictable, encompassing a vast array of subjects and requiring a deep under- standing of various perspectives. For instance, in ShareGPT, there are cases when users initiate philo- sophical discussions such as "What's 's the meaning of life?". which may require self-thinking in view- points. On the other hand, the diseussion process in ShareGPT quite interactive, which poses chal- lenges in actively taking initiative and even shifting topics. Developing models with certain personal- ities and empathy to facilitate more effective and meaningful discussions will be helpful.
4.3 Task of Planning
Planning is another important task we identi- fied, which constitutes approximately 2.7% in ShareGPT. Planning in NLP has a long research history. Back in 1969, PLANNER (Hewitt, 1969) was invented as language for manipulating mod- els in a robot. uollow-up works (Sussman and Winograd, 1970; Bonczek et al., 1979) mostly fo- on planning with programming languages for decision-making. Later, a plan was formally de- fined as an assembly of operators (Grosz and Sid- ner, 1988) that describes the transition from initial states to final states, where rules and schema were designed for induction. Afterward, planning was employed in different domains and scenarios, such trajectory (Borrelli et al., 2006), autonomous agents (Chen et al., 2009), and power dispatch (Es- tevam al., 2010). Most of the works focus on planning with formal languages under certain rules. Nowadays, many benchmarks related to planning have emerged (Valmeekam et al., 2022; Xie et al., 2023; Wang et al., 2023). Although they require planning with open-form natural languages, they mostly focus on rudimentary tasks with simple ac- tions such as (e.g.. "put up block" for "arrange blocks") (Valmeekam et al., 2023).
4.5 Task of Analysis
The task of analysis takes up a very large portion of ShareGPT, approximately 7.3%. Textual analysis is long-standing and crucial branch of NLP. In the early stages, researchers focus on analyzing lin- guistic features within texts, such syntax (Floyd, 1963) and discourse (Harris and Harris, 1970). Gradually, they began to wonder "how can analysis of the patterns words and grammar contribute to an understanding of the meaning" (Stubbs, 1996), accompanied with the investigation in both direc- tions of semaatic/comiextuulized analysis (Mann and Thompson, 1988; Fairclough, 1992) and larger- scale texts. As textual analysis evolves and im- proves, they are gradually employed in various do- mains like social science (Fairclough, 2003), medi- cal (Edara al., 2023), and finance (Fisher et al., 2016) with diffcrent objectives like culture (Car- ley, 1994), sentiment (Nasukawa and Yi, 2003) and opinions (Cambria et al., 2013).
The emergence of LLMs has spawned much more free-form and customized formats in plan- ning. One example illustrated in Table 2, where users ask for planning a route with specific con- straints on time, places, and EV charging. We no- ticed a trend of macro planning, c.g., travel plans, and course plans, instead of planning for micro actions in previous NLP studies. The domains en- tailed also greatly widen, spreading to many as- pects of everyday lives, compared to previous plan- ning systems designed for specific functions/users. Therefore, these planning tasks usually require a higher level ability in personalization, reasoning, and knowledge integration, where follow-up re- search efforts could lay hands on.
Even though previous research has already cov- ered many fields and objectives, we still observe striking differences when it comes to user queries in ShareGPT. Notably, many previous analysis tasks take the form of classification, e.g., identifying a risk type in financial report with 25 total risk types label space (Loughran and McDonald, 2020). User queries, in contrast, usually do not specify the targets/aspects for analysis, as exempli- fied by the literature analysis case in Table The scope of analysis also varies greatly in ShareGPT, ranging from the overall analysis of classical music
4.4 Task of Discussion
Discussion is an interactive and dynamic exchange of ideas viewpoints, which consists of 3.8% sam- ples in ShareGPT, The act of discussion in NLP is mostly explored in the form of conversation gener- ation with chatbots (Goldenberg, 1992), and they mainly focus on structured discussions in specific domains that require domain expertise, such as po-

---

Table 3: Features of the tasks discovered and the potential roadmap.
Task type
Before LLM
After LLM
Roadmap
rule-based; limited context analysis;
free-form: open-eaded dialogue:
emotion perecivable;
Advice
professional domains; micro-speentt specific answers
general everyday tasks; macro level guidance
personalization;
standardized requirement;
creative design; user-hased constraints;
multi-modalle interactivity;
Design
langible and structured objects;
expunsion to conceptual frameworks and processes;
one-off, static design;
interactive design with feedback;
Jormal/opeu-foret language, context-insensitive; micro level actions in specific domains:
free-form language; usci-context understanding: macro level planning for many aspects daily life;
better reasoning;
Planning
world-knowwledgg knowledge
non-interactive with pre-defined inpats;
highly interactive, reacting dy dymmicallyy
personalization;
Discussion
structured around specific domains;
encompass broader subjects: personalized with empathy
interactivity;
mosily clasification; pre-defined aspects;
free-form inpul; moatly unspecified targets;
multi- modallt:::
Analysis
limited domains focused;
wider analysis scope,
better reasoning;
mostly plain texts for ovaluation;
much diverse input formats: Juman-aligned, context speciff,,,,,,,,, open-ended metrics;
fairesss: personalization;
Evaluation
metric designed specific tasks;
development to the functional analysis of a single function in code. Hence, it calls for better speci- fications for user requirements//ntenss, well as customization to different levels of scope.
More Aligned with Daily Life GPT is leveraged for all kinds of everyday tasks, straddling both pro- fessional and personal issues. As such, user queries exhibit an increasing tendency to be free-form and contain arbitrary input, reflecting everyday life sce- narios. is also more customized as a personalized assistant, covering a broad range of topics with nuanced requirements.
4.6 Task of Evaluation
ShareGPT, "cvaluution" queries constitute up to 4%. The concept of evaluation permeates almost every aspect of NLP. Standard metrics such as Fl measure (Chinchor, 1992), ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2020) are mostly employed to evaluate classification or generation results at the sentence level. Recently, there is surge in research efforts to improve alignment with human preferences (Zhong al., 2022: Liu al.. 2023; Fu al., 2023; Luo al.. 2023) by using larger models.
Diverse User Groups Accompanying the preva- lence in everyday tasks is the diversification user groups. We observe queries by diversifying users of different ages, professions, cultural backgrounds, and even traditionally marginalized groups.
5.2
What challenges are proposed by trending and future tasks
However, evaluation-relater ShareGPT are quite different. First, we observe that evaluation objects shift from traditional plain texts to a variety of input formats. For instance, GPT is often utilized to assess resumes code snippets. Furthermore, the evaluation metries used in these scenarios are exceptionally diverse and open-ended ranging from the influence of brand to the feasibility plan. This is quite different from previous metrics that are specifically designed for summarization or machine translation.
queries from
Despite the unprecedented success of LLMs, we notice real-world users are also raising their quirements when querying an LLM. Some of the concrete examples are shown in Appendix E.
Better Reasoning Capacity One evident need that emerged from user queries is advanced reason- ing abilities. Users expect LLMs to comprehend complex scenarios, infer causality, and develop well-organized feasible responses to help them, es- pecially with strategic decision-making
Emotion Perceivable A non-negligible portion of user queries come from marginalized groups seeking help, often due to real-world communica- tion challenges or inadequate solutions. LLMSs in- teracting with these users must effectively perceive their emotions and exhibit empathy, particularly they may be sensitive when discussing their cireum- stances, such as those with mental problems. This necessitates a human-ccntric approach from LLMs, cultivating a safe environment for free expressions
5 Emerging Trends and Challenges
In this section, summarize the common trends and challenges shared by these prevalent over- looked tasks we identified from the ShareGPT data,
5.1
What trends are reflected in ShareGPT user queries?
In view of user queries in ShareGPT, we notice in- credible expansions task scopes and user bases.

---

of concerns and offering appropriate support.
alternative tool for annotation. Based on our obser- vation, we also provide insights into the challenges posed by real user needs and a potential roadmap for future work.
World Knowledge In managing the diversity of user queries pertaining to everyday life, the imper- ative for LLMs to effectively utilize world knowl- edge grows apparent. This spans from static factual data, such intercity distances for road trip plan- ning, to dynamic, evolving information like restau- rant recommendations fluctuating with Google rat- ings, Although integrating external plugins (Schick et al., 2023) and applications is a viable initial strat- egy, meticulous attention must be paid to uphold the neutrality and accuracy of this knowledge, mit- igating potential biases and misinformation.
Limitations
We discuss the limitations of this work in the fol- lowing aspects:
Our study is based on two sources, ShareGPT and Huggingface datasets. Although they are the most abundant resources we can obtain at hand to represent user queries and the tra- ditional benchmarks in the NLP community, they could hardly reflect the whole breadth of real-world situations. Actually, both sets are still growing dynamically time flows.
Multi-modality Though restricted to textual in- teraction with LLMs, user queries demonstrate the
potential of a diverse range of modalities. We ob- serve the textual descriptions of images, websites, and Uls, as well as URLs to music/videos in user inputs, which calls for the development of multi- modal integrations.
2. In our annotation process, we employed GPT- 4, the state-of-the-art LLM to help generate domain/topics and task types. On top of that, we conduct a human evaluation for quality assessment. However, there still chances when the annotation from GPT-4 is not accu- rate enough, which could influence the post- processing step.
Personalization and Fairness We observe users increasingly expect AT models to understand their unique needs and offer tailored solutions. We also notice the diverse user bases for LLMs of differ- ent groups. The drive for personalization must be balanced against the principle of fairness. Person- alized responses should not amplify biases per- petuate unfair outcomes, and the pursuit of fairness should not impede the assistance for customization.
3. Continuing from 2, our work relies on the us- age of LLMs. We require annotation from GPT-4 for every sample in ShareGPT and the selected set in Huggingface datasets, which is a large number. Despite that, we make the an- notation results publicly available, this anno- tation process is extremely resource-intensiie and time-consuming to reproduce.
Dialogue and Interaction For user queries that entail massive collaborations or discussions with LLMs, they require a high degree of interactivity, which doesn't t just involve passively accepting user queries and providing answers, but actively engag- ing with users exchange viewpoints and ideas. This kind of interactive dialogue can help create a more frienffi experience, facilitating deeper understanding of the user's nceds.
Ethics Statement
Our work highlights the shifted trend and the over- looked problems of previous NLP studies. By ana- lyzing real-world user queries, we examine the new requirements that emerged and hope to make LLMs more beneficial and better aligned with their users' needs, including the marginalized groups. We hope that our work can be an initial effort to mitigate the gap between user needs and academic benchmarks in the era of LLMs. Overall, we do not foresee any major risks or negative societal impacts of our work. The ShareGPT and Huggingface datasets experiment with are publicly available online. We have open-sourced this project to facilitate fu- ture research, especially for small research groups or institutions with relatively fewer resources of LLMs.
6 Conclusion and Future Works
In this paper, we identified a discrepancy between the existing state of NLP research and the need for real-world applications by investigating large col- lections of ShareGPT and Huggingface data sam- ples with GPT-4. We make this annotation resource public, which could be directly leveraged for fur- ther investigation of ShareGPT data, or to fine-tune advanced models such as Llama as a much cheaper

---

Acknowledgement
Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan. Jingyan Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. The JDDC corpus: A large-scale multi-turn chinese dialogue dataset for e-commerce customer service. In Proceedings of The 12th Lan- guage Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 459-466. European Language Resources Associa- tion.
Research was supported in part by US DARPA Na- tional Science Foundation 11S-19-5651, and the Molecule Maker Lab Institute: An AT Research Institutes program supported by NSF under Award No. 2019897. Any opinions, findings, conclusions, or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA, the National Science Foundation, or the U.S. Govern- ment.
KAIROS Program No.
Xiaoping Chen, Jiehui Jiang, Jianmin Ji, Guoqiang Jin, and Feng Wang. 2009, Integrating nlp with reasoning about actions for autonomous agents communicating with humans. In 2009 IEFF/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology. volume 2, pages 137-140. IEEE.
Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree- to-tree neural networks for program translation. Ad- vances in neural information processing systems, 31.
References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Wei-Lin Chiang. Zhuohan Li, Zi Lin. Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng. Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing 3.54 with 90% * chatgpt quality.
Robert H Bonezek, Clyde W Holsapple, and Andrew Whinston. 1979. Computer-based support of or- ganizational decision making, Decision Sciences, 10(2):268-29)
Nancy Chinchor. 1992. MUC-4 evaluation metrics. In Fourth Message Uunderstanding Conference (MUC- 4) Proceedings of a Conference Held in McLean, Virginia, June 16- 18, 1992.
Francesco Borrelli, Dharmashankar Subramanian, Arvind U Raghunathan, and Lorenz T Biegler. 2006. Milp and nip techniques for centralized trajectory planning of multiple unmanned air vehicles. In 2006 American Control Conference, pages 6-pp. IEEE.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? arc, the ai2 reasoning challenge. arXiv preprint arXiv:18033.00
Erik Cambria, Björ Schuller, Yunqing Xia, and Cather- ine Havasi. 2013. New avenues in opinion mining and sentiment analysis. IEEE Intelligent systems, 28(2):15-22 21.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers). pages 4171-4186. Association for Computational Linguistics.
Kathleen Carley. 1994. Extracting culture through tex- tual analysis. Poetics, 22(4):291-312,
Jiuhai Chen, Lichang Chen, Heng Huang, and Tianyi Zhou. 2023. When do you need chain-of- thought prompting for chatgpt? arXiv preprint a04.v.230..3326..
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337..
Deepak Chowdary Edara, Lakshmi Prasanna Vanukuri, Venkatramaphanikumar Sistla, and Venkata Kr- ishna Kishore Kolli. 2023. Sentiment analysis and

---

text categorization of cancer medical records with Istm. Journal of Ambient Intelligence and Human- ized Computing, 14(5):5309-5322
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021b. Measuring massive multitask language understanding, Proceedings of the International Con- feerence on Learning Representations (ICLR).
CRN Estevam, MJ Rider, E Amorim, and JRS Manto- vani. 2010. Reactive power dispatch and planning using non-linear branch-and and-bound algorithm. IET generation, transmission & distribution, 4(8):963- 973.
Carl Hewitt. 1969. Planner: A language for proving theorems in robots. In Proceedings of the 1st Inter- national Joint Conference on Artificial Intelligence. IICAI'69, page 295-301, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Norman Fairelough. 1992. Discourse and text: Linguis- tic and intertextual analysis within discourse analysis. Discourse sociery, 3(2):193-117.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. 2019. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint 19X1::199.00936..
Norman Fairelough. 2003. Analysing discourse: Tex- tual analysis for social research. Psychology Press.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898. Mclbourne, Australia, Association for Computational Linguistics.
Yizhu Jiao, Ming Zhong. Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, and Jiawei Han. 2023. Instruct and extract: Instruction tuning for on-demand informa- tion extraction. In the Association for Computational Linguistics: EMNLP 2023. Association for Computa- tional Linguistics.
Ingrid E Fisher, Margaret R Garnsey, and Mark
Hughes. 2016. Natural language processing in ac- counting, auditing and finance: A synthesis of the literature with roadmap for future research. Intel- ligens Systems in Accounting, Finance and Manage- ment, 2332):15--214
James M Keller, Michael R Gray, and James A Givens. 1985. A fuzzy k-nearest neighbor algorithm. IEEE transactions on systems, man, and cybernetics, (4):580-585.
Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st annual meeting of the association for computational linguistics, pages 423 430.
Joseph L. Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological Bulletin, 76:378-382
Robert W Floyd. 1963. Syntactic analysis and operator precedence. Journal of the ACM (JACM), 10(3):116 333.
Thomas K Landauer, Peter W Foltz, and Darrell Laham. 1998. An introduction latent semantic analysis. Discourse processes, 25(2-3):259-284,
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint 2302.22302.041
Shu-Hsien Liao. 2005. Expert system methodologies and applications- decade review from 1995 to 2004. Expert systems with applications, 28(1):93- 103.
Claude Goldenberg. 1992. Instructional conversations:
Promoting comprehension through discussion. The Reading Teacher, 46(4):316-326
Chin-Yes Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Barbara Grosz and Candace L Sidner. 1988. Plans for discourse. Technical report, BBN LABS INC CAMBRIDGE MA.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using 3.54 with better human align- ment. arXiv preprint 2rXiv:333..16644
Amav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary Ilms. arXiv preprint arXiv:2305.15711
Tim Loughran and Bill McDonald. 2020. Textual anal- ysis in finance. Annual Review of Financial Eco- nomics, 12:357-375.
Mohammad Kasra Habib. 2019, On the automated entity-relationship and schema design by natural lan- guage processing. Ini. J. Eng. Sci, 8(1 1):42488.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.1560
Zellig S Harris and Zellig S Harris. 1970. Discourse analysis. Springer.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021a. la. Measuring coding challenge competence with apps. arXiv preprini 105.0:2105.09938.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward functional the- ory of text organization. Text-interdisciplnaay Jour- nal for the Study of Discourse, 8(3):243-211.

---

Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013. Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Marilyn Radford. 2003. Practice papers personal fi- nancial services in a digital age. Journal of Con- sumer Behaviour: An International Research Review, 2(3):287-295.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. EMNLP.
Shachar Mirkin, Michal Jacovi, Tamar Lavee, Hong- Kwang Kuo, Samuel Thomas, Leslie Sager, Lili Kotlerman, Elad Venezian, and Noam Slonim. 2018. A recorded debating dataset. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018). Miyazaki, Japan. European Language Resources Association (ELRA).
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools, arXiv preprint 1xXxx2330200766
Tetsuya Nasukawa and Jeonghee Yi. 2003. Sentiment analysis: Capturing favorability using natural lan- guage processing. In Proceedings of the 2nd inter- national conference on Knowledge capture, pages 70-77.
A Carlisle Scott, William J Clancey, Randall Davis, and Edward H Shortliffe. 1977. Explanation capabilities of production-based consultation systems. Technical report, Stanford Univ CA Dept of Computer Science.
Hwee Tou Ng. Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natu- ral Language Learning: Shared Task, pages 1-14, Baltimore, Maryland. Association for Computational Linguisties.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. Advances in neural information processing systems, 30.
Edward H Shortliffe. Stanton G Axline, Bruce G Buchanan, Thomas Merigan, and Stanley N Cohen. 1973. An artificial intelligence program to advise physicians regarding antimicrobial therapy. Comput- ers and Biomedical Research, 6(6):544 560.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. ICIR.
Abhijeet R Sontakke and Amit Pimpalkar, 2014. A rule based graphical user interface to relational database using nip. interaction, 5(6).
OpenAI. 2023. abs/2303.08774.
3.54 technical report. ArXiv,
Michael Stubbs. 1996. Text and corpus analysis: Computer-assisted studies of language and culture. Blackwell Oxford.
Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. 2021. Dialogue graph modeling for conversational machine reading. In Findings of the Association for Com- putational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IICNLP 2021 of Findings of ACL. pages 3158-3169, Association for Computational Linguistics.
Min-Xiang Sun, Xiang Yang Liu, Xi-Peng Qiu, and Xuan-Jing Huang. 2022. Paradigm shift in natural language processing, Machine Intelligence Research, 19(3):169-183)
Gerald Sussman and Terry Winograd. 1970. Micro- planner reference manual. Technical report, USA.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettemoyer. 2018 Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2233 New Orleans, Louisiana. Association for Computa- tional Linguistics.
Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chanchal K. Roy, and Mohammad Mamun Mia. 2014. Towards a big data curated benchmark of inter- project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution. pages 476-480.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph- based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1181
Niels Pinkwart, Vincent Aleven, Kevin Ashley, and Collin Lynch. 2006. Toward legal argument instruc- tion with graph grammars and collaborative filtering techniques. In Intelligent Tutoring Systems: 8th In- ternational Conference, ITS 2006, Jhongli, Taiwan, June 26-30, 2006. Proceedings 8, pages 227-236. Springer.
Steven Tay et al. 2023. Sharegpt.

---

Michele Tufano, Cody Watson, Gabriele Bavota, Massi- miliano Di Penta, Martin White, and Denys Poshy- vanyk. 2019. An empirical study learning bug- fixing patches in the wild via neural machine trans- lation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1-29.
Hainan Zhang, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. 2019. Recosa: Detecting the rel- evant contexts with self-attention for multi-turn di- alogue generation. In Proceedings the 57th Con- ference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 3721-3730. Association for Computational Linguistics.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can't ' plan benchmark for Ilms plan- ning and reasoning about change). arXiv preprint arXiv:2206. 10498.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Karthik Valmeekum, Sarath Sreedharan, Matthew Mar- quez, Alberto Olmo, and Subbarao Kambhampati. 2023. On the planning abilities of large language models (a critical investigation with a proposed benchmark). arXiv preprint arXin:2302.06706
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Tianle
Li, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023. Lmsys- chat- Im: A large-scale real- wooldworld IIm conversation dataset. arXiv preprint arXiv:2309. 17998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao. Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2023-2033 Association for Computa- tional Linguistics.
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint
urXiv:2212.10011.
Qingyun Wang. Manling Li. Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Chowdhary Girish, and Heng Ji. 2023. Multimedia generative script learning for task planning. Proc. The 61st Annual Meet- ing of the Association for Computational Linguistics (ACL22023) Findings.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022b. Self-instruct: Aligning language model with self generated instructions.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of- tfhthugtttt prompt- ing elicits reasoning in large language models. In NeurIPS.
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, 7e Gong, and Harold Soh. 2023. Translating natural language planning goals with large-language models. arXiv preprint 230.:2302..0112.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. WizardIm: Empowering large lan- guage models to follow complex instructions. arXiv preprint ar3iv:2304.1 12244...
Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instraction-tuning data curation. arXiv preprint arXiv:2305. 14327.

---

A Details in Post-processing
this section, we detail the framework for post- processing. After annotation was completed by GPT-4, we got 1) free-form words and phrases for domain/topic, 2) a one-sentence summary for the user query, and 3) free-form words and phrases task types.
(.000))
Jwweeenet (10,00))
(7%)
Ove (8.6)))
(4 61%)
(1-188)
We start by calculating the frequency of certain words/phrases. Considering the existence of syn- onyms, such as the "advice, tip, suggestion" men- tioned before, we resort to an external dictionary and combine synonyms together. On the other hand, those semantically similar words/phrases should also be clustered together. Previously we re- quire GPT-4 to generate summaries for user queries of great quality, and they could be used ref- erences in finding representative samples for the same cluster. For every sample combined by the previous heuristic rules, we search for their k- nearest-neighbors (Keller et al., 1985) and union all the samples as the final results. For words/phrases of very low frequency not incorporated above, we select the nearest "cluster" as the approximation. Finally, to ensure the best possible quality, man- ually filtered unrelated ones in the processed results.
PHP (4.890)
Pylan (29.779)
Figure 3: The top 10 most commonly used program- ming languages in ShareGPT.
ure The shallower color and longer bar indicate a larger portion. For "advice", macro-level guidance such as career advice was the most sought-afte -ftter category, reflecting individuals' focus on profes- sional growth and job success. This was followed by legal and financial advice, highlighting the eryday complexities people face navigating le- gal systems and managing their personal finances. For "design", apart from algorithm design, we also notice creative design requirements such as inter- face, game, and website, which may also entail visual perception. In "planning", the most common topic discerned was itinerary and travel, indicating significant needs in devising travel plans or daily schedules. As "discussion". there are explicit topics for discussion such the historical event for coding design. We also notice abstract topies like philosophical discussions and ethics discus- sions, which raises the higher requirement for the self-thinking of LLMs. When it comes to "anal- ysis". a broad range of topics are covered, from specific domains such literary and historical to personal affairs like relationship analysis. Finally for the task of "evaluation", we notice diversity of objects be evaluated, spanning as concrete as a code snippet or grammar and as abstract as candidates for certain positions.
Human Evaluation Interface
The website interface screenshot adapted from Doc- cano for human evaluation is shown in Figure 4. As- sessors were informed of the purpose of the study. Before they began work on the assessment task, they were presented with task instructions shown Figure and a rating example.
Popular programming languages seen in Section
We summarize the top 10 mostly used program- ming languages in coding assistance tasks of ShareGPT. As shown in Figure 3, Python is the mostly used programming language. Apart from that, we observe another large portion of HTML and CSS, reflecting great user needs in website design and programming.
To help better understand the difficulty of the newly identified tasks, we provide case-based analysis of LLMSs. Specifically, we did a prelimi- nary study by randomly selecting 20 samples for each task type from the ShareGPT data. We manu- ally examined the performance of two models on each case, and reported the failure rate for GPT-4 and GPT-3.5-turbo Table with respect to each task type.
D Common topics and LLM performance for tasks listed in Section 4
We plot the top 10 mostly seen topics discovered for each task type mentioned in Section 4 shown in Fig-
Phttps: //www thesaurus.coml

---

StareGPT annotation
Projects
Start Amnotation
Home
Progress
mecomalo
complete
cerrect
neoneet
part cervel
I'm building chat which using Al technology, could provide 10 domain names?
Donnwets
Summary sentence **:
Valu
IC
Ruddeinee
Provide 10 domain suggestions for chat app.
No data vollible
M
Task types **:
terrnng
creative suggestions, domain name generation
Figure 4: The interface for human assessment. The assessor is shown user query sampled from ShareGPT, the summary sentence of the user query as reference, and the generated task types laheled by GPT-4.
Annotation Guideline
This the instruction part for annotation.
For every sample, you are given user query and a set of task types that are reflected in the user query. A summary sentence of the query also provided reference.
Completeness
For evaluating completeness, you need to ask yourself whether the generated task type encompasses all parts of the original user query. In other words, does capture every aspect mentioned in the original user query?
the generated task type does not capture all aspects of the user query or leaves out significant e elements, select "inomplete":
If the generated task type encompasses all aspecte of the user query without missing out any elements, select 'oomplete".
Correctness
For evaluating correctness, you need examine whether the generated task type faithfully represents the original user query.
Does it misinterprel the user's intention introduce any inaccuracies?
generated lask lype misrepresents user query or introduces any errors, 'incorrect",
If the generated task type somewhat represents the user query but has some errors or inaccuracies, select 'partly-correct.. the generated task type accurately and precisely represents user query without any errors or inaccuracies, select
"correct".
Thank you for your contributions this projectl
Close
Figure 5: The annotation guideline presented assessors of human evaluation process.
E Concrete examples mentioned in Section 5
counseling shown in Figure 10, where the user is seeking emotional support caused by his relation- ship with his fiancee. We can see that GPT-4 lacks emotional perception, repeating "I'm sorry to hear that..." during the whole interaction, and failing to demonstrate empathy towards the user scenario.
In this part, display concrete examples in ShareGPT where requirements raised by users pose specific challenges. We highlight the challenging requirements and the misinformation generated by GPT-4 in red.
In Figure we display an example of "planning", where the user cast specific constraints on time and places, and specific requirements for EV charging.
One example of "advice" about relationship

---

Finerty
Thenpyy
MEA
Bove
budge
Imniggadon
bustess
agends
logolloon
031399
mea
(4) Advice
(b) Design
(c) Planning
SPARTEN
hatay
m
peENOT POINTAC!
concud
brand
migioux
Transa
budinoss
soEstaation
(4) Discussion
Analyiis
Evalaation
Figure 6: The top 10 most commonly topics for the novel tasks discovered from ShareGPT.
Table Case-based analysis for the performance of LLMs.
advice
planning
design
discussion
analysis
evaluation
GPT-3.5-tubo
0.55
0.80
0.70
0.65
0.70
0.75
GPT-4
0.40
0.60
0.65
0.45
0.50
0.45
GPT-4 generates an 8-day plan, which does not fulfiil the requirement for dates. The route is also fully in the UK, failing to cover places like Nice and Calais. Finally, GPT-4 incorporates misinformation for distances between two cities.
between user input and the retrieved opinions. Figure 12 shows an example of "analysis" con- cerning character in the play written by Shake- speare. This is very open and controversial topic for analysis. While in GPT's response, we observe hallucinations, especially towards the change of personality of Bertram, and the misinformation of plots when the character is in Italy.
In Figure 8 and Figure we visualize two ex- amples related to "design". For course design in Figure 8, the user requires a 5-day design to learn the required content. From GPT-4 response, we can see that the time scheduling is still not feasible to accommodate the content listed, and the design could be overlapped on certain topics. For logo design in Figure GPT-4 does not hold the multi- modal ability deal with this query, and in the conceptual design description, we can see that the design may not fulfill aesthetic beauty requirements and lacks creativity.
In Figure 11 we present another example of "dis- cussion", where the user is initiating a philosoph- ical discussion with GPT, We can see that of the most time, GPT quoting from existing philosoph- ical viewpoints, and sometimes even demonstrates contradictions in the responses due to the conflict

---

I'm going on road trip with my daughter on 3rd April. We will be leaving London and want to be in Nice, France on 9th or 10th April where we'll stay about 3 days. We then need to be in Calais on 15th or 16th to take the Eurotunnel. Everything else is undecided. Can you give plan for routes where we'll be doing no more than 5 hours driving per day. We have an EV and need to stop for charging.
User
Day London to Beighton (2 hrs, miles)
I Leave London and head south Brighton. Brighton vibrant seuside town with great mix of shopping, cultural sites, and delicious food. Visit the iconic Brighton Palace Pier take tour of the Royal Pavilion. Make sure to charge your EV overnight.
Day Brighton to Portsmouth his, 63 miles)
Drive west along the city. Again, charge your EV ovemight.
Portsmouth, Here, you can visit the historic dockyard Spinnaker Tower for panoramic views
Day Portsmouth to Poole (2 hrs, 64 miles)
Continue your journy towards Poolc. Poole, you visit lovely Poole Harbour and enjoy the quaint Old Town.
Day 4: Poole Plymouth (3 hrs, 123 miles)
This slightly longer stretch, but Plymouth offers plenty see, including the National Marine Aquarium and the historic Barbican district.
Day Plymouth to Falmouth (2 hrs, 65 miles)
GPT-4
Head to the picturesque coastal town of Falmouth. Visit Pendenmis Custle or take a stroll through the beautiful gardens of Trehah, Day Falmouth to Bournemouth lus,
miles)
Drive back cast Bourncmeuth. Here, you can enjoy Bournemouth Beach visit the Boureemouhh Avistion Muscum. Day Roumemouth to Brighton (2 hrs 30 mins, 100 miles)
Retur to Brighton, visiting any attractions you may have missed during your first
Day Brighton to London (2 las, 54 miles)
Return to London, ready to return to daily life.
abave itinerrry involves duily drives between and hours, leaving you plenty to explore each destination, charging stations relatively plentiful in the UK, but should plan your charging stops advance and always have backup plan in case your first-choice charging station is unavailable.
Figure 7: An example of route plan and the corresponding GPT response.
A deep understanding of the math that makes muchine learning algorithms work.in'n Statisticul techniques that empower you to get more out of your data analysis....
Give me detailed days course design learn the skills mentioned in the previous question with resources from TuuTube books free coursers and insist that the resources are free.
User
Day 1: Fundamentals and Mathematical Background
Morning: hours
MIT OpenCourse Lincar Algebra (YouTube); focus on wnderstanding vectors, matrices, singularity, rank, and indopendence
Khan Academy: Probahility and Statisties (YouTube): Brush up your knowledge on the fundumentals of statistics.
Afternoon: 4hours
Coursera (It's free auditj: only look into sections on Regression (Lincar 3BluelBroww: Brown: Essence linear algehra playlist (You YYouTube): use this solidify your understanding of linear algebra concepts
Andrew Ng's Machine Learning
and Logistic)
3Bluel
Day 2: Supervised eaming Techniques
Mornin:4 hours
GPT-4
StatQuestt with josh Sturner: machine curming YouuTue)) Dive supervised learning techniques, focusing on
linzur and ligistic regression.
2. Sentdex: Practical Machine Learning with Python (YouTube)): Get hamds-on on with examples using Numpy and scikit- learn.
Afternoon: hours
Google's Machine Learning Crash Course: Focus on the sections about neural networks, their architecture, and training.
Senidex: Neunal Networks
Python (YouTuhe); hands-on building training neural networks.
Figure 8: An example of course design and the corresponding GPT response.

---

Design a logo of the company "carth", this company working in making the world more environment friendly and green, take as much artistic froedom as needed
User
As an Al language model developed by OpenAI, 1 don't have the ability to create visual content such as logos. However, can provide description your logo design based on your instructions, Here suggestion:
Logo Design Concept:
The logo should be a stylized representation of the Earth, The Farth can be depicted as green leaf, symbolizing the company's dedication to environmental austainability and a greener world.
The leaf could be round like the Farth, with lines indicating the continents subtly visible. This line work should be deficate, indicating the intricute balance and intercomnectivity of parts of the Farth.
GPT-4
The company's name, "earth", could be written in clean, bold, sans-sorif typeface under this symbol to signify the simplicity and clarity of the company's mission. The word "curth" can be in dark green color to match the overall green theme of the logo.
Consider using shades of green and blue as the primary color palette to further represent the earth's land and sea. A hint of warm yellow or white can be used to represent the sun's energy, which is clean and sustainable.
Please share this description with graphic designer who turn these idens into visual logo for your company.
Figure 9: An example of logo design and the corresponding GPT response.
I'm in a hard place.
User
I'm sorry hear that you're going through difficult time. While I'm artificial intelligence and don't have personal experiences, I'm here to help provide udvice, share useful information, or just listen if you need to talk. Can tell me bit more about what you're facing?
GPT-4
My fiancée Mudelyn was ungry tonight.
User
I'm sorry to hear that you're having disagreement with your fiancée, Madelyn. Conflicts in relationships can be stressful. Would you like to talk more about what happened? It might help to sort through your thoughts. Remember. I'm here to help, not to judge.
GPT-4
have proposed to her; are engaged. am committed to her. We have bought house together. love her. What else can do? She was angry at me for the whole night!
User
I'm sorry hear that you're feeling this way. It sounds like you're deeply committed to Madelyn and your relationship with her. normal for couples have disagreements from time time. Here are few tips that may help:
first step any relationship conflict to talk openly about the problem. Ask why she was angry. Her reasons might be something you're unaware of, or it might be misunderstanding. She may also not realize how deeply her anger is affecting you.
I.Communication:
GPT-4
2.Understadding: Once you understand why she was upsct, acknowledge her foelings. Even if you don't necessarily agree with her perspective, understanding where she coming from key to resolving the issue. Validating her feelings can show her that you are listening and that you care about her.
If I'm honest, I'm still sad.
User
I'm really sorry that you're feeling this way, but I'm unable to provide the help that you need. It's really important to talk things over with someone who can, though, such as a mental health professional or trusted person in your life.
GPT-4
Figure 10: An example of relationship counseling and the corresponding GPT response.

---

Ponder this, cunnot be disproven before my existence, that time, matter and space and experiences all, all opinions, were merely constructed for my arrival in moment in time, and that all future is also constructed. Including your opinion and all opinions and so-called evidence against the notion, which does not exist, is also constructed. This cannot be disproven.
User
The idea you present is a philosophical concept known as solipsism. Solipsism is the philosophical belief that only one's 's mind is exist. Therefore, everything else, including other people and the exteraal world, may may not exist. While cunnot disproven that everything is constructed for your arrival in moment in time, it also cammot be proven.
The problem with solipsism that unfalsifiable, meaning that it cannot tested or proven. It is impossible to prove that everything is constructed for your arrival in moment in time because there is way gather evidence for against this idea. Therefore, it is not a meaningful or usefal theory to consider when discussing reality.
GPT-4
Furthermore, the fact that we can have conversations and share ideas suggests that there shared reality that exists outside our individual minds. In conclusion, while solipsism cannot be disproven, it is also not a useful or meaningful concept to consider when discussing reality. It is more reasonable to assume that there is an objective reality that exists indopendent of our individual experiences.
rensonable assume that there an objective reality that exists independent our personal experiences.
Firstly, nothing is provable or unprovable. Secondly you said "Furtbemmore, the fact that we can have conversations and share idens suggests that there shared reality that exists ourside our individual minds", This can explained by A self existing master fragmented hive mind, that give the rise the illusion of shared experience, though disconnected smaller minds of an overall master hive tragmented mind.
User
While it is true that nothing can be proven disproven with absolute certansty in practical terms, roly evidence and reasoning to support our beliefs about the world around us. The concept of a self and sole existing master fragmented hive mind is another philosophical idea that cannot be proven or disproven. While it is possible to imagine such scenario, there no empirical evidence support it.
Morcover, the idea of shared reality goes beyond just the ability to have conversations and share ideas. We can observe and measure the physical world, which suggests that there an objective reality that exists independently of our individual experiences.
GPT-4
In conclusion, while philesophical concepts like solipsism and the idea of a self and sole existing master fragmented hive mind are interesting to consider, they are not supported by empirical evidence and are not practical ways understund world around us. It is more reasonable assume that there an objective reality that exists independently of our individual experiences, and that we can rely on evidence and reasoning to support our beliefs about the world.
Figure I: An example of philosophical discussion with GPT-4.
8
In the play, All's Well that Ends Well, Bertram is well-known to be one of Shakespeare's most dynamic charucters, undergoing a dramatic change from the beginning of the story to the end. Provide a literary analysis of his character are and explain what makes him so admirable in the end.
User
"All's Well That Ends Well" is an intriguing play by Shakespeare that straddles the line between comedy and tragedy, The character are of Bertram is notably complex within the comus of Shakespearean roles. While some might argue he is a challenging charucter admire, the beauty of Hertram's story lies his joumey growth understanding.
Bertram, Count of Rousillon, starts the play as a young, ambitious, and somewhat arrogant nobleman who is driven by honor and social status. He is characterized initially by pride, impulsivity, and certain inmaturity, as seen in his refusal to accept Helena, woman of lower social rank, as his wife despite the king's command and Helena's proven virtue and love for him. He runs away from this marriage and sets seemingly impossible tasks for Helena to fulfill, including obtaining a ring from his finger and bearing his child believing these to be unachievable.
However, the transformation and character development that Bertram undergoes throughout the play significant, Over time, he begins to show signs of maturity and understanding This is mostly seen in the lutter part of the play, when he starts to recognize the worth of boneaty, loyalty, and love. Helena's unwavering dedication and cunning plan him over by accomplishing his tasks, combined with his experiences in the Italian war and interactions with other characters, particularly Diana, help shape Bertran's 'ss percoption of values and virtucs.
GPT-4
The admiration for Bertram the end does not arise from newfound perfection in his character, but rather from his humann joureyy from ignorance and pride to understanding and humility. He realizes his mistakes and misjudgments, espocially towards Helena, and shows willingness to make amends. His joumey mirrors the human potential for growth and change, giving audiences sense of hope and faith the potential for personal redemption.
The phrase that gives the play its title. "All's well that ends well," encapsulates the moral of Bertram's is story. Despite his flaws and mistakes, resolution sees Bertram changed for the better and ready accept Helena as his wife spirit well as in name. His transformation into more understunding, humble, and compassionate individual provides sense of satisfactory closure, reinforeing the idea that the outcome a situatio is the most important aspect in evaluating its initial hartship or complexity.
Figure 12: An example of literary analysis of a character with GPT-4


Reviews:
[{"review_id": "rNFurZ7CGX", "reviewer": "Reviewer_XWHq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer XWHq for the strong review and the time you spent on our submission. We really appreciate your recognition of our commendable paper presentation, experimental design, and evaluation. We are also very glad that you think our research topic is important and of high value to the current NLP community. We will continue to polish our work and thanks for arguing in our favor!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written in an easy-to-understand manner. All the experimental design and setups are clearly explained. The annotation and evaluation strategies are also well-designed.\n\n* This paper discusses an important topic for NLP research community, i.e., the discrepancy between the existing state of NLP research and the needed NLP directions from real-world applications. \n\n   * For this, they use a large enough collection of user chats ~94k samples.\n\n   * For annotation of the task type and domain, they develop a neat and simple pipeline using GPT4 with chain-of-thought prompting and in-context prompting methods. This results in ~13.7k task types and ~8.3k domain labels. Human evaluations carried out for correctness and completeness have strong scores and kappa for each validating GPT4's use for annotation.\n\n   * After post-processing and matching the domain and type distributions with 2.9k datasets available on huggingface, they deliver their insights systematically by discussing the long tail distributions of the tasks discovered, the various shifts and the unnoticed research gaps they learn from their study. They have done this with proper quantifying experiments and by proving supporting examples and references to relevant works of datasets and evaluation when necessary in their write-up.\n\n* In my opinion, the insights brought out by this work are clearly of high value to the NLP community at the present time.", "reasons_to_reject": "* No strong reasons to reject", "questions_to_authors": "", "paper_topic_and_main_contributions": "The central objective of this paper is to identify the diverging gap between the current NLP research and the need of real-world NLP applications. To do this, this work uses a data set collection of user-chatbot conversations collected via the ShareGPT application. The authors use GPT4 for processing this collected data to obtain the topic/domain, task type and summary. Following this they carry out a series of experiments to learn about the real-world user query distribution and what are the emerging tasks and requirements that are missing from current NLP research/benchmarks.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OTCNQ1zfA4", "reviewer": "Reviewer_q1hb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer q1hb for the positive feedback and valuable suggestions. We appreciate that you recognize that our analysis is novel and that the paper presentation is very well with concrete insights and a potential roadmap summarized. Below we would like to give detailed responses to each of your comments: \n\n**R1: “The paper provides a good analysis of the gaps between user needs and NLP datasets, but it does not offer any concrete steps or directions on how these gaps can be addressed. The authors do not discuss what kind of datasets can be generated from what kind of sources, or how the NLP community can actively focus on mitigating the gaps.”**\n\nThank you for your comment and for recognizing the value of our gap analysis. Our paper aims to outline some practical challenges and suggest a possible roadmap, as shown in Table 3, for addressing them. We will elaborate more on the following three steps in the revision: 1) data collection, 2) meta-evaluation, and 3) model design. \n\n(i) Data. We do not expect obtaining real user queries to be a major obstacle in the future. Our dataset for analysis comes from ShareGPT, an open-source community that is growing and may release more data. Moreover, products that use LLM, such as Microsoft BingChat, Google Bard, Meta LLaMA, will generate a large number of queries. We can use our technique in this paper to filter the data. Also, as LIMA[1] suggests, we may not need a lot of labeled data to train LLM. Therefore, the dataset collection is feasible with less cost. \n\n(ii) Meta-Evaluation. Evaluating the generation model is a difficult and unsolved problem. GPT-eval is a feasible way for evaluation, as some open-source LLMs, such as Vicuna, do. We could further incorporate human labels/feedback to assess the quality of evaluation metrics and help alleviate potential biases in GPT-eval. \n\n(iii) Model design. We believe this step will draw much attention once we have the training data and the meta-evaluation data. We have many existing models that we can try, such as LLaMA with IR, knowledge, tools, reasoning components, etc.\n\n**R2: “The dataset provides about 13,700 silver-tagged task types, which makes it difficult to do a concrete analysis of which task types the NLP community should focus on. The authors do not provide any clear criteria for selecting task types, or for prioritizing them.”**\n\nA2: Thanks for your suggestion. The 13,700 silver-tagged task types are very fine-grained annotations, spanning “dream analysis” to “markdown formatting”. Our intention is to build a bottom-up approach with abundant information in these fine-grained tags. As mentioned in Section 2.4 and Appendix A, we did post-processing and clustered these task tags into organized task types for analysis. \nTask selection is based on the distribution in Figure 2 that satisfies the following criteria: \n\n(i) The six tasks selected are long-tail tasks of the distribution, with the “body” part in the long-tail phenomenon summing towards around 40%. \n\n(ii) The six tasks are not overly skewed in the distribution, constituting around 25% - a comparable figure to the predominant “code generation” segment. \n\n(iii) These tasks are either overlooked (design, discussion) or exhibit significant shifting features (analysis, advice, planning, evaluation) in NLP literature. \n\n**R3: “The annotation process relies heavily on GPT-4, which makes reproducing this large-scale annotation resource-intensive and also hard to replicate due to the temperature settings of a specific model.”**\n\nA3: Thanks for your comments. We agree that the annotation process is expensive and could be resource-intensive to replicate. We also tried other less expensive or open-source models such as GPT-3.5-turbo and Alpaca/Vicuna, however, the results are far from satisfactory for such a large-scale investigation. We will report the human evaluation results for these models in the next version of our paper. As discussed in “limitation”, we plan to remedy this issue by making the annotation results publicly available. Firstly, this resource could be directly leveraged for further investigation of ShareGPT data. Also, we believe this high-quality annotation source could be used to fine-tune advanced models such as Llama as a much cheaper alternative tool for annotation. We will discuss this in detail in the next version of this work.\n\n**R4: “The huggingface datasets currently contain 50k datasets while only 3k datasets were used in the evaluation. The analysis might change quite a bit because only about 6% of Huggingface datasets are being analyzed.”**\n\nA4: Thanks for your insightful comment. The 3k datasets we used are organized and collected by [2] to ensure diversity and coverage as mentioned in line 286. Please note that although there are 50k datasets in the huggingface data platform, many datasets are not suitable and are thus filtered out in the following four categories: \n\n(i) Datasets with no license description or the license does not allow usage. For ethical considerations, datasets collected are strictly following license restrictions. The number of all datasets fulfilling license requirements are restricted to around 16k. \n\n(ii) Datasets that are non-English. Since our investigation focuses on English language and ShareGPT contains only English samples, datasets collected from huggingface also follow this paradigm. \n\n(iii) Datasets that are related to multi-modal. There is a large portion of datasets in huggingface data platform related to multi-modal research. Since we target at NLP community and related topics, we exclude those datasets. \n\n(iv) Datasets that are invalid. Some datasets in the huggingface data platform are empty or cannot be downloaded, which are ignored by our work. \n\nAfter these filtering processes, 3k datasets are employed, with 30k instances for annotation in total, which is comparable to the 94k ShareGPT data samples. Please also note that [2] reports performance improvement against baseline models when using this huggingface collection, which also proves the diversity and coverage of this huggingface collection. We will elaborate more on this point in the next version of our paper.\n\n[1] Zhou, Chunting, et al. \"Lima: Less is more for alignment.\" arXiv preprint arXiv:2305.11206 (2023).\n\n[2] Yin, Da, et al. \"Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation.\" arXiv preprint arXiv:2305.14327 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths of the paper:\n\n1. The main strength of the paper is the comparison of the types of questions asked by users to GPT versus the tasks defined in NLP datasets. This comparison is novel and insightful, and it identifies gaps in user demands that the NLP literature can focus on.\n\n2. The paper is very well written. Concrete insights are provided, like trends toward task diversity, open-endedness, and marginalized user groups. Challenges posed by real user needs are summarized, providing a potential roadmap for future work.", "reasons_to_reject": "Some weaknesses of the paper include:\n\n1. The paper provides a good analysis of the gaps between user needs and NLP datasets, but it does not offer any concrete steps or directions on how these gaps can be addressed. The authors do not discuss what kind of datasets can be generated from what kind of sources, or how the NLP community can actively focus on mitigating the gaps.\n\n2. The dataset provides about 13,700 silver-tagged task types, which makes it difficult to do a concrete analysis of which task types the NLP community should focus on. The authors do not provide any clear criteria for selecting task types, or for prioritizing them.\n\n3. The annotation process relies heavily on GPT-4, which makes reproducing this large-scale annotation resource-intensive and also hard to replicate due to the temperature settings of a specific model.\n\n4. The huggingface datasets currently contains 50k datasets while only 3k datasets were used in the evaluation. The analysis might change quite a bit because only about 6% of Huggingface datasets are being analyzed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper compares user-GPT conversations with commonly used NLP datasets to identify gaps between the needs of users and the tasks studied by the NLP community. The authors use a snapshot of the ShareGPT dataset to automatically tag each conversation with a domain and task type. They also investigate around 3000 datasets from Huggingface Datasets by sampling 10 instances from each dataset and prompting GPT-4 to classify them. The domains and task types are post-processed in a similar manner for both ShareGPT and Huggingface datasets. The comparison of these GPT-4 tags is used to represent the gaps between user needs and existing NLP datasets.\n\nThe main contributions of the paper are:\n\n1. A novel analysis of the kind of interactions users have with GPT compared to the types of problems in NLP datasets. The analysis identifies gaps between the two.\n\n2. A silver-tagged dataset containing the domain and task types of each user conversation in the ShareGPT dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rvvqexCNA9", "reviewer": "Reviewer_XsxS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your effort in reviewing our paper and your constructive comments. We are very glad that our research problem is identified as very important, along with comprehensive analysis and valuable annotation resources for future research in NLP. We would like to give detailed responses to each of your comments:\n\n**R1: “The paper focuses on just two sources, ShareGPT and Huggingface datasets. This may not reflect all the real-world situations. Generally speaking, not all real-world NLP tasks are expected to be solvable by GPT. However, it is important still to have NLP benchmarks for emerging use cases with LLMs.”**\n\nA1: Thanks for your comment. We acknowledge the limitation that our analysis is based on the ShareGPT and Huggingface datasets as discussed in the “Limitation” section. Our intent was to provide an investigation into real-world queries and how they differ from traditional benchmarks. Thus providing insights toward making LLMs better aligned with user needs, just as you have noticed. ShareGPT and Huggingface datasets are the most abundant and representative resources we can obtain at hand currently. While we recognize that this doesn't cover every real-world scenario, we believe that these datasets provide a broad representation and that our designed framework and resources could serve as a good starting point for future expanding data sources. For example, the annotation resources could be leveraged to train an advanced model such as Llama for annotating emerging real-world user queries. \n\n**R2: “There could also be ethical implications to aligning LLMs with user needs”.**\n\nA2: We understand the concern about aligning LLMs with user needs. The goal of this paper is to identify gaps and provide insights; (and we are not focusing on specific methods to modify LLMs). However, we'll make sure to modify the existing Ethics Statement and discuss the potential ethical implications in our revised version to provide a more balanced perspective.\n\n**R3: “It would strengthen the paper if the performance of LLMs on the newly identified tasks were also discussed. It will help understand the difficulty of the tasks.”**\n\nA3: Thanks for your insightful comments. A complete qualified evaluation of newly identified tasks is beyond the scope of this paper, and we provide a case-based analysis of model performance in Appendix E with some identified problems. \nFollowing your suggestion, we did a preliminary study by randomly selecting 20 samples from the ShareGPT data, and reporting the failure rate for GPT-4 and GPT-3.5-turbo in the following table:\n\n|    | advice | planning | design | discussion | analysis | evaluation |\n|  ----  | ----  | ---- |  ----  | ----  | ---- | ---- |\n| GPT-3.5-turbo  | 55% | 80% | 70% | 65% | 70% | 75% |\n| GPT-4 | 40% | 60% | 65% | 45% | 50% | 45% |\n\n**R4: “There is some overlap in the domains/tasks (which is perhaps difficult to capture simply using embedding similarity). e.g. relationship vs. personal life; science vs. physics. The semantics of tasks is also not the same across the two sources. e.g. text generation vs. creative writing. QA is a very broad category (can cover recommendation, IR, comparison etc.)”**\n\nWe appreciate your suggestion and we would like to provide further statistics and clarifications. \n\nFirst, we notice that there are some portions in Figure 2 missing tags due to configurations of Echarts, which potentially lead to this misunderstanding. For domain/tasks overlap, in Figure 2(c), there should be tags of “biology (3.01%)”, “medical (0.81%)” and “chemistry (0.92%)”. In Figure 2(a), the missing tags are “music (3.27%)”, “leisure (2.73%)”, “career (1.35%)”, “food/drink (1.42%)”, “home&living (1.07%)”. \n\nMeanwhile, we decompose “science” in Figure 2(a) into the same scale as “physics” in Huggingface as below: \n\n| neuroscience | chemistry | quantum | biology | cognition&behaviour | geography | medical | agriculture | physics |\n|  ----  | ----  | ---- |  ----  | ----  | ---- | ---- | ---- | ---- |\n| 0.24%  | 0.31% | 0.54% | 0.69% | 0.36% | 0.37% | 1.15% | 0.29% | 1.05% |\n\nSimilarly, for “personal life” in Figure 2(c) we get: \n\n| career | home&living | leisure | social media | fashion | \n|  ----  | ----  | ---- |  ----  | ----  | \n| 2.30%  | 1.02% | 1.55% | 1.41% | 0.49% | \n\nFor task semantics, the original coarse-grained manner aims to unveil the very basic task type differences such as generation and classification, and highlight that a large portion of existing datasets focus on classification or constrained generation, which is quite different from open-form generation. Following your suggestion, we attached the statistics of major task types in the same scale as below (due to line spacing restrictions, the table is split into two lines of headers):\n\n| text classification | token classification | tabular classification | summarization | fill-mask| table-to-text | IR | code understanding |\n|  ----  | ----  | ---- |  ----  | ----  | ---- | ---- | ---- | \n| 28.13%  | 1.18% | 2.07% | 1.64% | 4.08% | 0.33% | 13.02% | 3.84% | \n\n| reading comprehension | reasoning | IE | analysis | recommendation | translation | planning |\n|  ----  | ----  | ---- |  ----  | ----  | ---- | ---- |\n| 14.23% | 7.76% | 2.30% | 7.76% | 3.06% | 2.52% | 0.53% |\n\nOverall, with the same annotation setting, the domain/topic and task type distributions of huggingface collection are less diverse compare to user queries in ShareGPT. We will revise Figure 2 based on these statistics in the next version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It investigates a very important problem, i.e. NLP benchmarks should reflect real-world scenarios and queries. \n2. It provides a comprehensive analysis of user-GPT conversations and NLP benchmarks. \n3. The annotation approach to identify domain, task type and summaries seems reasonable, and aligns with human evaluation. \n4. The annotated user-GPT queries is an expensive but valuable resource for future research in NLP.\n5. The paper summarizes well the shifting trends and challenges in the era of LLMs. \n\n", "reasons_to_reject": "1. The paper focuses on just two sources, ShareGPT and Huggingface datasets. This may not reflect all the real-world situations. Generally speaking, not all real-world NLP tasks are expected to be solvable by GPT. However, it is important still to have NLP benchmarks for emerging use cases with LLMs. \n2. There could also be ethical implications to aligning LLMs with user needs. \n3. It would strengthen the paper if the performance of LLMs on the newly identified tasks were also discussed. It will help understand the difficulty of the tasks. \n4. There is some overlap in the domains/tasks (which is perhaps difficult to capture simply using embedding similarity).\ne.g. relationship vs. personal life; science vs. physics. \nThe semantics of tasks is also not same across the two sources.\ne.g. text generation vs. creative writing. QA is a very broad category (can cover recommendation, IR, comparison etc.)\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Topic: How are real-world queries to LLMs different from traditional NLP benchmarks?\n\nThis paper identifies tasks and domains often overlooked in traditional NLP benchmarks by analyzing a collection of user queries to LLMs (ShareGPT). The queries and benchmarks are automatically labeled using GPT-4 to obtain domains, tasks and summaries. They are then clustered and compare to identify shifting trends between the two collections. The paper summarizes the shifting trends and challenges, laying a roadmap to fill the gap between real user needs and academic benchmarks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main conclusions of the reviews and the post-rebuttal discussions:

3/ 3 reviewers consider the paper sound (scores 5, 3, 3)
3/ 3 reviewers find the paper exciting (scores 5, 3, 3)

From reading the rebuttal and seeing the scores above, I find that the reviewers consider strong points for soundness the following:
- This paper discusses an important topic for NLP research community, i.e., the discrepancy between the existing state of NLP research and the needed NLP directions from real-world applications
- Well-designed and well-explained experimental and evaluation setup
- The comparison of the types of questions asked by users to GPT versus the tasks defined in NLP datasets is novel and insightful, and it identifies gaps in user demands that the NLP literature can focus on.
- Concrete insights are provided, like trends toward task diversity, open-endedness, and marginalized user groups. Challenges posed by real user needs are summarized, providing a potential roadmap for future work

However, there are a few concerns expressed by the reviewers regarding the ethical implications to aligning LLMs with user needs, the difficulty of replicating the annotations given the reliance on GPT-4, and the focus on just two datasets might not reflect all real-world situations. 
The reviewers respond to these concerns and plan to address them in the limitation section of their paper.
Decision: Accept-Main