BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions
Authors: Arth Bohra
Govert Verkes
Artem Harutyunyan
Pascal Weinberger
Giovanni Campagna
Venue: EMNLP 2023 Findings (2023)
Keywords: text classification
large language models
prompt engineering
few-shot classification
user study
personalization
TL;DR: We propose a new method for few-shot text classification leveraging large language models with a prompt that is constructed interactively by end-user and LLM.

Abstract: Text classification is a well-studied and versatile building block for many NLP applications. Yet, existing approaches require either large annotated corpora to train a model with or, when using large language models as a base, require carefully crafting the prompt as well as using a long context that can fit many examples. As a result, it is not possible for end-users to build classifiers for themselves. 

To address this issue, we propose a novel approach to few-shot text classification using an LLM. Rather than few-shot examples, the LLM is prompted with descriptions of the salient features of each class. These descriptions are coauthored by the user and the LLM interactively: while the user annotates each few-shot example, the LLM asks relevant questions that the user answers. Examples, questions, and answers are summarized to form the classification prompt.

Our experiments show that our approach yields high accuracy classifiers, within 79% of the performance of models trained with significantly larger datasets while using only 1% of their training sets. Additionally, in a study with 30 participants, we show that end-users are able to build classifiers to suit their specific needs. The personalized classifiers show an average accuracy of 90%, which is 15% higher than the state-of-the-art approach.

BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions
Arth Bohra'
Govert Verkes
University of California Berkeley
Artem Harutyunyan
Berkeley, CA, USA
Pascal Weinberger
ar arthbohra@berkellg. edu
Giovanni Campagna
Bardeen, Inc.
San Francisco, CA, USA
Jebardeen. jiovannieeaardeenn gio.anni arovert,artent pascal, ,ascca,,
Abstract
1. User sets context and initial class descriptions of their tusk
Text classification is a well-studied and versa- tile building block for many NLP applications. Yet, existing approaches require either large notated corpora to train a model with or, when using large language models carefully crafting the prompt as well as using long context that can fit many examples. As result, it is not possible for end-users to build classifiers for themselves.
Purpose: I want to separate spam from my important
important
work emails, friends.
uninportant
pronccions, other.
emails.
2. User annotates selected example
base, require
Dear Alice,
Here update your bealth, Your medical indieate you need purchase the following treatment: turmeric. Also, feel free to schodule an appointment next week!
Nate
Generating and answoring questions about the example
Alice hasm' mentioned
Hey Alice, health emails classiffy under the 'other class'. Are they important?
To address this issue, propose novel ap- proach to few-shot text classification using an LLM. Rather than few-shot examples, LLM is prompted with descriptions of the salient fea- tures of each class. These descriptions are coau- thored by the user and the LLM interactively: while the user annotates each few-shot example, the LLM asks relevant questions that the user answers. Examples, questions, and answers summarized to form the classification prompt.
anything about medical
emails.
Actually, emails about my heulth don't 't belong in the other category.. they important
just medical appointments that important?
So medical appuintments inpostant What about other appointments?
Let me clarify for you! All my appaintments are important to me!
Our experiments show that our approach yields high accuracy classifiers, within 79% of the per- formance of models trained with significantly larger datasets while using only 1% of their training sets. Additionally, in a study with 30 participants, we show that end-users are able to build classifiers to suit their specific needs. The personalized classifiers show an average accuracy of 90%, which 15% higher than the state-of of oate-offtheear ahe-art approach.
2b. Validating the model
Befere Alice answered questions, thought.. but now.
Predicted: impoctant
Correct! You were right because...
Refining class deseriptions
Based on this example, I would upditte the class descriptions by.
important
work emails, friends, emails, appointments
uninportunt
proenations, other (not including .....
1 Introduction
3. Using the Classifier
Text classification - the task of mapping a sentence or document to one class of a predefined set is a well-studied, fundamental building block in natural language processing (Maron, 1961; Li et al., 2022), with applications such as textual entailment (Da- gan et al., 2005; Bowman et al., 2016), sentiment analysis (Wang and Manning, 2012: Socher et al., 2013), topic classification (Hingmire et al., 2013), intent and dialog act classification (Godfrey et al.,
Dear Alice,
biology degree requires you get Join a 'Mati Call' with your major advisor to figure out what you can take' University
mathematics.
Alice meniione appointments were important.
inpertant
This email also
Figure 1: Overview of how a user can construct and use classifier with BYOC.
Work done while at Bardeen

---

1992; Reithinger and Klesen, 1997; Tur et al., 2010; Coucke et al., 2018; Xu al., 2021), and more. Because a classification task targets a fixed set classes, each of these many applications of clas- sification must be built separately. Furthermore, classes are often highly domain-specific, and in certain scenarios must be personalized to each user. Consider the example of classifying email as "Important" "Not Important". Many users would consider family emails to be important, but some might decide to only prioritize work emails, or only emails that require immediate action such as a re- ply. As a more extreme example, most users would consider coupons to be junk mail, but some might actively search for savings.
At the same time, for many of these tasks, the mapping from text to classes can be expressed in relatively simple rules. LLMs based on instruction tuning (Ouyang et al., 2022) can classify according to instructions expressed as text. The challenge then becomes: how can we help end-users write meaningful and sufficient instructions? End-users are not prompt engineers, and struggle to write good instructions. In our experiments we found that users often forget about relevant situations or conditions, leading to poor quality classification.
We also observe that LLMs can be prompted to elicit information and ask clarifying questions. Hence, we propose interactive approach to help users write high- classification prompts, which we call Build Your Own Classifier (BYOC):
There exist plethora of use cases where person alized classification is necessary:
1. The user provides the purpose of the classifier and the list of classes.
Classifying product business reviews as useful according to someone's preferences.
2. They annotate small amount of data. For each example, the LLM generates questions that would be helpful classify the example. The user answers the questions, then selects the correct class for the example and explains why they selected that class.
Classifying user support requests as being per- tinent to a particular product component, so they can be routed to the relevant team.
Classifying an academic paper as likely rele- vant or not to someone's research.
In all of these contexts, annotating a large dataset and then training domain-specific model is in- tractable. It is too expensive and time-consuming for individual end-users to label the hundreds of examples needed train a high-quality classifier, and even more so if the criteria change frequently, because each time the training set would need to be rebuilt from scratch. Hence, we need a technique that can work well a few-shot regime, where the user only labels limited number of examples.
The examples, questions, answers, labels, and explanations are summarized into class de- scriptions: succinet textual representations the classification criteria and relevant aspects of the training data.
4. At inference time, the LLM is prompted with only the generated class descriptions.
Our experiments show that with only a small amount of additional work for each training exam- plc, BYOC yields higher accuracy than including training data in the LLM prompt. Indeed, we can improve the accuracy of few-shot text classifica- tion to the point where can be within 79% of fine-tuning dedicated model full dataset.
To overcome this challenge, we propose to lever- age Large Language Models (LLMss), which have been shown perform very well on variety of classification tasks (Radford al., 2019; Raffel et al., 2020; Brown al., 2020; Gao al., 202 I).
Prior work using LLMs for few-shot text classi- fication proposed using task demonstrations con- sisting of examples and their labels, and proposed to include many as possible to fit in the context length (Brown et al., 2020; Gao et al., 2021). There are issues with this approach: (1) the context length is limited, and with longer text only few ex- amples can fit, and (2) the prompt length can be significantly larger than the size a single input, increasing the computational cost.
Our method is also suitable for personalization. In an experiment conducted directly with end users, our approach improves over manually provided class descriptions by 23%, and over the state of the art by 15,, while consuming 37% fewer tokens at inference time.
1.1 Contributions
The contributions of this paper are as follows:

---

A novel approach to few-shot text classifica- tion using class descriptions in lieu of task demonstrations. This approach more com- putationally efficient due to a shorter prompt.
et al., 2020; Gao et al., 2021), optionally with ex- planations (Lampinen al., 2022). Task demon- strations are computationally expensive and limited by the language model context window size. Addi- tionally, it is unclear whether the model can learn from demonstrations effectively (Min et al., 2022). Finally, it is known that end-users struggle to write high accuracy classification prompts (Reynolds and McDonell, 2021; Jiang et al., 2022).
A method, called BYOC, to interactively con- struct class descriptions for classification, by prompting the LLM to ask relevant questions to the user. Ours is the first approach to use LLM--geerated questions to help users craft high quality prompts.
Prompt Construction Concurrent to this work. Pryzant et al. (2023) proposed automatically op- timizing the prompt given few-shot data, using search algorithm and LLM- LLM- generated feedback. Wang et al. (2023) proposed summarizing few-shot examples to obtain succinet prompts. Neither of those methods incorporates user feedback to dis- ambiguate challenging examples.
Experimental results on the Web of Science dataset (Kowsari et al., 2017) show that a clas- sifier built with BYOC is 9% better than the few-shot state of the art, and reaches 79% of the accuracy of the state-of-the-art model trained on full dataset, using only 1% of their training set.
Kaneko et al. (2023) also proposed leveraging question generation for classification. They pro- pose to use interactivity at inference time; our goal instead is to be able to classify non-interactively, and only use questions at training time.
BOOC enables non-experts build classi- fiers with high-accuracy, with an average accu- racy of 90%, 15% higher than the equivalent few-shot baseline and 23% higher than asking users to write prompts directly. Additionally, the classifiers built with BYOC outperform model trained for all users at once, which underscores the need for personalization.
Finally, we note that natural language feedback was also proposed for explainability (Marasović et al., 2021; Yordanov et al., 2022; Wiegreffe et al., 2022), but those approaches did not lead to im- provement in accuracy.
We built an end-to-end version of BYOC within the Bardeen web automation platform' In study with 30 participants, we find that users find our approach interpretable, and 80% would consider using it.
Data Augmentation A different line of work pro- posed to augment the training data with automati- cally generated or automatically labeled data.
In particular, one line of work observed that often large unlabeled corpus is available, and proposed using programmatic rules to label it in a weakly supervised fashion (Ratner et al., 2016, 2017). Later work then proposed learning those rules from natural language explanations (Hancock et al., 2018) 8) or extracting those rules from a pre- trained language model (Schick and Schütze, 2021; Pryzant al., 2022). Weak supervision is not prac- tical for end-user personalized classification, be- cause crafting the rules requires a high level of expertise. Refining the rules also requires large labeled validation set, which is often not available.
2 Related Work
Few-Shot Text Classification Text classification using few-shot training sets has been studied before. Previous methods proposed dedicated model archi- tectures, based on prototype networks (Snell et al., 2017; Sun et al., 2019) or contrastive representation learning (Yan al., 2018; Chen et al., 2022). These methods cannot take advantage of large language models, especially API-based L.Mss that are only available as a text-based black box.
With L.Ms,, text classification is usually either zero-shot through providing instructions and class names (Radford et al., 2019; Raffel et al., 2020; Ouyang et al., 2022: Sun et al., 2023), through transfer learning from existing tasks (Wang et al., 2021), or few-shot by task demonstrations (Brown
A different line of work proposed to use LLMSs to synthesize training data, which is then used to train smaller classifier (Yoo al., 2021; Ye al., 2022: Gao et al., 2023; Møller et al., 2023). As with using LLMs directly for classification, the challenge is to instruct the LLM so the resulting training data is personalized, and not a statistical average across

---

many users. At the same time, our approaches are complementary, and synthesis could be used in the future to distill the LLM to more computationally efficient model.
confirm what is stated in the class descriptions. Al- ice answers the question, addressing any confusion the model may have. Then Alice presented with another question related to the same email, either a follow-up question or a new question altogether.
3 Overview of BYOC
Alice answers again, and the process repeats up to configurable number of questions for each email.
In this section, we present the process through which a user can build a personalized classifier using BYOC. We do so by way of an example: Alice wants to build email classifier to catego- rize emails as "Important" and "Unimportant". The overall flow is summarized in Fig.
Prediction and Labeling After Alice answers all questions, the model uses the responses to pre- dict the class of the current email. BYOC displays the prediction and its reasoning, which enables Al- ice to assess the model's decision decision-making process. Alice then enters the correct classification of the email, and also corrects the model's reasoning, if necessary.
3.1 High Level Planning
As the first step towards building classifier, Alice provides high-level plan of what she wishes to classify, which provides the model with an initial understanding of her objective. First, Alice writes a short purpose statement of what she would like to accomplish: "I want to separate spam from my important emails". Then, she specifies the list of classes she wants to classify into two classes, in this example. For each class, Alice can optionally write an initial description of what kind of text fits in each class. For example, she can say that work emails are important, and promotions are not. The descriptions play an important role in BYOC and will be iteratively improved in later steps.
Prompt Refinement Given the text, questions, answers, classification, and explanation for the cur- rent example, BYOC automatically updates the class descriptions, adding or removing any infor- mation to improve classification accuracy. Alice then repeats this process new email, until she chooses to stop adding new data to the training set. After annotating all training examples, Alice can review the final refined class descriptions. She can optionally edit the class descriptions to her liking, making any corrections or adding any information that might be missing.
Finally, Alice assigns a name to the classifier and saves it so she can use it later.
3.2 Interactive Training
Without training, the model would have limited understanding of Alice's specific needs for clas- sification. Instead, Alice interactively trains the model with a small amount of data. The goal is two- fold: for Alice to understand the model's decision- making process to refine the prompt, and for the model to learn about difficult examples and specific criteria relevant to the task.
3.3 Using the Classifier
After Alice creates the classifier, she can use as an action in Bardeen. For instance, she builds a pro- gram that scans her inbox, applies the constructed classifier to each email, and if the email is classi- fied as "not important". it is marked as read and archived. Alternatively, she use program that triggers any email and sends her text mes- sage when the email is classified as important. The same classifier, once built, can be used different programs. Moreover, she can share with others, like family members with similar email needs.
Training Data First, the user selects source of data to use for training. In our example, Al- ice selects her email account, and BYOC uses her credentials to fetch and show her an email.
Interactive QA Below the email, BYOC presents Alice with question, automatically gen- erated from the text of the email. The goal of the question is clarify the meaning of the text in the context of classification. Along with the question, BYOC displays the model's 's reasoning for asking the question, whether is to clarify, broaden, or
4 Prompting for Personalized Classification
In this section, we first present the formal definition of a classifier model that uses class descriptions, and then present a method to interactively compute those class descriptions.

---

4.1 Problem Statement
questions that refine broaden the scope of the current class descriptions, and improve the LLM's general understanding of the classifica- tion task. In particular, we prompt the model to produce questions that uniquely depend on the uset, that are not answerable based on the given text, that are not too similar across ex- amples, and that are not too specific given example. The model is also provided the pre- vious questions and answers, and instructed avoid producing redundant questions. The prompt was refined based on our initial experi- ments, in order to improve the informativeness of questios.
Given a sample (x, y), where x the input text to be classified and y E C the class, our goal is to learn function f parameterized on no and de
dc) -
such that, with high probability ù = y. that is, the predicted class is equal to the true class. nc is the set of user-defined names of all classes c E C, and de is the set of all class descriptions de.
We learn de from the user-provided classifier purpose p. the class names nc. and small set of labeled samples (I4, Vo) D:
de BYOC(P, nc, D)
InteractivePrediet:: The model attempts to make its own classification of the text based on the current class descriptions and the ques- tions answered by the user for this sample. Access to the answers from the user, which is available for regular prediction, improves the accuracy and the quality of the reflection obtained from the model.
The function BYOC that computes the class de- scriptions is an interactive function: in addition to the given inputs, it has access to oracle (the user) who answer any question posed them in natural language.
4.2 Computing Class Descriptions
BOC proceeds iteratively for each sample (1a, v1). updating dc at each step. The initial dc is pro- vided by the user.
Update: The model computes new class description for a given class, given the cur- rent class descriptions, the input, questions and answers, model- and user-generated label, and user explanation. The model is prompted to incorporate the information from the new answers, without discarding relevant informa- tion from the existing descriptions.
At each step, we ask the user questions in succes- sion. The set of question and answers Qi is initially empty, and it gets updated with each question qi,t and answer ai,t (1 < 1 S M):
Qi,0 = 0
4i, GenQuestion I dc,i-1, Qi,-1)
The model sees both the correct and predicted labels. It is important to include both because it allows the model to update both the class description of the correct class and the de- scription of the incorrect one that was initially predicted.
air = User(qrt)
Qit = Qi U ((B,t, d1,)}}
"User" denotes the user answering the question.
After collecting M questions, where M is a by- perparameter, BYOC computes:
These functions are implemented with individual calls to a black box LLM. All functions incorpo- rate chain of thought prompting (Wei et al., 2022). Detailed prompts are included in Appendix B.
0. & = Interactiveerredictte,,, nc, de,,d,,,,,,.. Qi,M) de. = Update(dc,s-1,,4 ,, Qs,M. 0i Wr, ei)
where & a model-provided explanation for why the sample has the given class, which we obtain through self-reflection (Shinn et al., 2023), and e; is the user's 's own rewriting of é;
4.3 Classifying With Class Descriptions
After obtaining the final class descriptions de, we compute classification on a new sample x as:
The above procedure makes use of the following functions implemented with LLMs:
f(x) Predict(xx, nc,dc)
GenQuestion: The model generates a question to clarify the specification of the classifica- tion task. We prompt the model to generate
The function Predict concatenates fixed pream- ble, the class names, descriptions, and classifier purpose, and the input x as a prompt to the LLM,

---

and then returns the class C. whose name ne corresponds the output of the LLM. If no class corresponds to the output of the LLM, an error returned; note that this rarely happens in practice. As before, we apply chain of thought to the L.M.
Few-Shor: in addition to the Zero-Shot prompt, the prompt includes few-shot training examples, up to the token limit. Each few- shot example consists of input and label. All inputs are summarized, if necessary.
4.4 Handling Long Inputs
Explanation: as in the previous model, but additionally the explanation from the user is included for each few-shot example. This is similar to the approach used by Lampinen et al. (2022).
For certain problems a single input I can be too long to fit in the LLM prompt. To account for this, we propose to replace with summarized version, up threshold size K in the number of tokens:
i { {
Summarize p) x
|a|| > K
QA: as in the previous model, but each few- shot example includes both the explanation and all questions generated by BYOC for that example and the answers from the user.
otherwise
where no the class names and p is the classi- fier purpose. The threshold K strikes balance between computation cost, space in the context for the class descriptions, and space for the input.
BYOC (our approach): the model is prompted with class names and class descriptions ob- tained as described in Section 4.2.
We propose a novel summarization approach specifically optimized for classification. First, we split the input into chunks I of equal size, approx- imately respecting paragraph and word boundaries. We then compute the summary of each chunk 8j. given the previous summary and the overall pur- pose of the classifier p.
We the GPT-3.5-Turbo model as the LLM while constructing the classifier interactively, and the GPT4 model (OpenAI, 2023) as the LLM dur- ing evaluation on the validation and test set. We used GPT-3.5-Turbo in the interactive part for rea- sons of speed, where it can be a factor for usability. We use temperature of 0 for classification to maxi- mize reproducibility, whereas use a temperature of 0.3 to generate questions, in order make them more varied and creative.
81 SummarizeCCuun((g..z) 21)
8j SummarrieCChunkss41.......))
SummarizeChunk is implemented by a call to the LLM. The prompt is included in the appendix.
5.2 Personalized Classification
5 Evaluation
Our main goal is to support personalized classi- fication. To evaluate BYOC ability to do so, we perform a user study, in which we ask a group users to try and use BYOC to classify their emails. The study is both a way to collect real data to eval- uate B BYOC as a model, and a way to study how well end-users can interact with BYOC.
In this section, we evaluate the quality of our pro- posed approach. Our main experiment is person- alized classification use-case where we attempt to classify emails according to the user's provided cri- teria. For comparison, we also evaluate against an existing dataset for the classification of long text.
Experimental Setting We have built an end-to- end prototype of BYOC with graphical user- interface, based on the Bardeen web browser au- tomation platform. We recruit 30 English-speaking users, 16 women and 14 men, to participate in our study. 10% of the users are above the age of 50 years old, 20% are between the ages of 30 and 50, and 70% are under the age of 30. 13 users are in a technical field, 9 are pursuing business, 6 are studying medicine, and 2 are in the arts.
5.1 Experimental Setup
We compare the following approaches:
Zero-Shot: the model is prompted with class names and class descriptions provided by users directly, followed only by the input to classify. Inputs longer than the LLM context window are truncated.
+ Summary: we use the same prompt as in Zero-Shot, but additionally inputs longer than the LLM context window are summarized as described Section 4.4.
We ask each user build a personal classifier to categorize their emails as important or unimportant. We use 30 emails from each user as the data source,

---

Train
Dev
Test
sification with user explanations (Lampinen et al., 2022) by more than 15%. The zero-shot approach achieves a respectable accuracy of 67%. Adding summarization improves by 4% by avoiding sud- den truncation. Few-shot Few-shot further improve accuracy by 3%, but we find no measurable im- provement by adding explanations. Asking ques- tions to the user is marginally effective.
# Examples
300
300
300
# Tokens
473.155
586,165
668,438
Table I: Size of the BYOC email classification dataset.
# Tokens
Approach
Build
Run
Accuracy
Overall, our results are two-fold. First, it is clear that merely asking users to provide class descrip- tions is not sufficient, and training data is necessary. Second, our results show that the manner in which the training data is annotated and is provided to the model is critical to achieve good accuracy. Not only the questions answered by the user make dif- ference compared to the existing approach of expla- nations, the biggest improvement appears when the information collected during the annotation stage is summarized into co-authored class descriptions. B YOC is the first method that enables the model to extract key insights from every example in the train- ing set, and discard the large amount of otherwise irrelevant information.
Zero-Shot
0
1,628
66.7%
Summary
0
3,447
71.0%
Few-Shot
0
5,535
74.7%
Explanation
0
5,590
74.7%
QA
93,282
5,818
76.7%
BYOC
170,411
3,681
90.0% %
Fine-tuned
n1a
n/a
78.3%
Table 2: Test-set accuracy and token counts of our pro- posed approach against the different baselines for per- sonalized email classification. Fine-tuned refers to a single model trained on all training data simultaneously, while other models are trained for each user separately.
out of their most recent 100. For HTML emails, we remove HTML tags and only use plain text.
B YOC is also more token-efficient when com- pared to previous approaches. Even though takes on average 170,411 tokens to build an email clas- sifier for a user, this cost is fixed. BYOC uses 1,909 fewer tokens for every example compared to few-shot examples, and is comparable in cost to zero-shot with summarization. In summary, be- sides the initial fixed cost when building the clas- sifier, BOC is more accurate and cost-effective than the state of the art.
10 of those emails are used in training: the user annotates them during the interactive training flow (Section 3.2), together with the answers to the ques- tions generated by BYOC and the explanation for the classification. Each user answers 3 questions for each training email. The remaining 20 emails are used for validation and testing; users annotate them the end after constructing the classifier with BYOC. 10 of emails for each user are used for validation and 10 for testing.
Importance of Personalization In many scenar- ios, would be impractical for cach users to ac- quire enough training data to fine-tune personal- ized model. Hence, we also wish to evaluate the tradeoff between few-shot personalized model, compared to fine-tuned model that uses more training data across different users. To do so, we train single fine-tuned model using all 300 emails in the training set at once. We use a distilled BERT model (Sanh et al.. 2019) with a classifier head connected to the embedding of the first token.
Dataset We acquired a dataset of 900 emails from the 30 users participating in our study. Due to the nature of most emails in an inbox being spam unimportant users, 36% of the emails are clas- sified as important, with the rest being unimportant. Statistics of the dataset are shown in Table
Overall Accuracy Table 2 shows the results of our experiment. The table shows the accuracy of the classifier on the test sets, as the percentage of examples classified correctly across all users. The table also shows the number of prompt tokens nec- essary construct the classifier average across users ("# Tokens Build"), and perform the classi- fication on average across emails ("# Tokens Run""), BYOC improves over the state of the art of clas-
Results for this experiment are shown in Table 2, in the "Fine-Tuned" line. The fine-tuned model achieves higher accuracy than LLM baselines, but B YOC still improves the accuracy by 13%. The increase in accuracy is likely due to everyone's differences in perception of what makes an impor- tant or unimportant email. B BYOC more accurate

---

because it enables the LLM receive personal information about each user.
demic paper is classified into one of fixed set of topics. We choose this task because it requires domain-specific knowledge, especially around the specific usage of each class, impairing the zero- shot ability of LLMs. The task also involves longer text, which limits the number of few-shot examples that an LLM can process.
User Evaluation At the end of the study, ask each user also complete short survey to gather feedback on their interaction with BYOC.
Overall, on average on a scale of to 10, users rate their experience with B BOC an 8.6. When asked about whether they would actually take the time to build a classifier, on scale of to 10, users rate their willingness 9. PI reported, "Yes, although tedious, the questions made it capable of differentiating junk and important emails." "
Dataset We use the WOS-5736 dataset by Kowsari et al. (2017). The dataset contains eleven classes of research paper abstract topics, with three parent classes: biochemistry, electrical engineering, and psychology.
Furthermore, on Likert scale between and 5. on average users report their likeliness to use the classifier in the future a 4.6, with 50% of them saying they would use the classifier for personal use-cases, 40% using the classifier for education, and 20% for work. Many users found use cases for BYOC beyond emails. P2 reports: "I could see myself using it in order to give me specific recom- mendations on things to eat, watch, or do and also to sort emails." " P3 stated a more specific use-case for themselves: "I could use a personalized classi- fier to classify homework assignments in college to figure out which ones should be prioritized."
For each sub-class, we selected abstracts at random belonging to that class, and annotated them with BOC, answering 3 questions each.
As we are not experts in the fields covered by the dataset, we consulted online sources, includ- ing Google and ChatGPT, to provide the initial class descriptions and answer the model's gener- ated questions. Answering each question took around 3-5 minutes on average. We assume that that end-users will be classifying text that they are familiar with. It is reasonable to assume that do- main experts, with access to the same resources and knowledge bases, would have better answers with less amount of effort to annotate the data. Note that incorrect answers can only impair accu- racy; our results is thus lower-bound of what be achieved by a domain expert. An example of question and answer in Fig. 2.
Beyond the higher level questions asked about BYOC, we also asked users about their expe- rience with specific parts of the experiment. Users noticed how the questions asked by the model re- minded them about a lot of the cases that they had forgotten about in their initial class descriptions. P4 states, "Yes, the process was smooth because the questions allowed me to realize further ways to classify the emails that I had initially forgot- ten." Furthermore, when talking about whether they liked the interactive experience of answering questions, P5 reports: did, because they were used to determine what groups certain things went in. The process felt like sorting in real life."
The state-of-hee the-art approach on this dataset leverages hierarchical model, classifying first the text into one of the parent classes, and then into one of the sub-classes, We use the same approach for this experiment. As the abstracts are short, no summarization
necessary for this experiment. The questions, answers, and generated class de- scriptions for the experiments are available online?.
Results Table 3 compares BYOC against the state of the art Our best result improves over the few-shot approach by 9%, while consuming an av- erage of 1,167 fewer tokens per sample.
Additionally, all users believe that the model ul- timately produced more descriptive and accurate class descriptions than they had written initially. P6 reports, "Yes, the model gave better descrip- tions and allowed the classifications be more accurate to what wanted."
When compared to the sate-of-theear approach for this dataset (Javeed, 2023), which used the full training set of 4,588 examples, our model is only 20% worse. If we take the state-of-the-arr result an upper bound over this dataset, BYOC achieves 79. 79.2% of the maximum possible accuracy with less than 1 % of their training set.
5.3 Comparison With Existing Approaches
For comparison with previous classification litera- ture, we also wish to evaluate BYOC on an existing dataset. We choose the task of academic paper topic classification, in which the abstract of an aca-
Pht httpa///c/.maareeeee////oeoeedee

---

Approach
# Tokens
Accuracy
cess of question-answeereee and minimal annotation are able make LLM capture the essential fea- tures of each class, enabling a precise classification. Our experiments highlights the importance both personalizing and user-interaction with the LLM. First, we show that users do not know how to instruct LLMs. Second, we demonstrate the need for personalization. A fine-tuned model trained on all users at once is outperformed by BYOC by 13%, despite outerforming all few-shots baselines. This suggests that there no mee-iize-itts-al model that can be applied to every-user. Our study involving real participants also confirms the usability and effectiveness of our approach. End-users are able to build classifiers for themselves exhibiting an average accuracy of 90%, rating their experience 8.6 out of 10.
Zero-Shot
631
58.7%
+ Few-Shot
3,873
63.9%
Non-hierarchical
2,276
67.0%
BYOC (GPT-3.5)
2,706
74.8%
BYOC (GPT-4)
4,656
63.2%
HDLTex
n/a 90.9%
ConvexTMM
n/a 92.42%
Hawk
n/a 94.45%
Table 3: Accuracy and token counts on the Web of reported
Science test set. Results for HDLTex by Kowsari et al. (2017), ConvexT'M by Bhattarai et al. (2022) and Hawk by Javeed (2023). Our approaches use few-shot training set while the others use the full training set.
Overall, our methodology enables users to lever- age the power of LLMs for their specific needs, opening up new possibilities for practical applica- tions of text classification in various domains.
We also compare the use GPT-3.5 during con- struction of class descriptions, in previous exper- iments, with the use of GPT4 throughout. Note that GPT4 was in all cases used for the classification after obtaining the class descriptions. We observe that class descriptions obtained from GPT4 achieve lower accuracy. From our observations we see that GPT4 tends to generate class descriptions that are longer and more specific to the given training amples, leading to form of overfitting.
This work available as commercial product in the Bardeen web automation platform', where BYOC enables end-users to classify several forms of data. Popular use-cases include sales lead, inbox classification, and support request categorization.
Limitations and Future Work
Finally, we also evaluated BYOC in non- hierarchical setting where it attempts to directly predict one of 11 classes, using the same class de- scriptions. We found it materially worse than the hierarchical case, with an accuracy of 67% on the test set. This suggests that there is limit to the number of classes that an LLM can predict using class descriptions, and the hierarchical approach is be preferred.
We have identified the following limitations of our approach:
BYOC uses the LLM as a text-based interface. This is design decision driven by the avail- ability of API-based LLMs, but it might limit the maximum achievable accuracy, compared fine-tuning the model or using a method that has access to LLM embeddings or log- its. Additionally, using an LLM without fine- tuning exposes a risk of hallucination, and it is possible that the user is misled during training.
6 Conclusion
This paper presented novel approach to few- shot text classification using large language models (LLM). By having the user cooperate with an LLM we were able to build highly accurate classifiers while annotating a very minimal amount of data. For many practical applications of classification, our method eliminates the need for large annotated corpora. We have achieved high accuracy classi- fiers that competitive with models trained with orders of magnitude larger datasets.
For long inputs it is possible that, due the summarization step, relevant criteria are omit- ted from the input of the model, which would lead to a wrong classification. While our sum- marization technique is preferable over trun- cating the input, future work should explore additional summarization methods or alterna- tively encodings of the input.
Practically speaking, the number of samples that the user is willing annotate is usually hhttps:/bbardeen.ai
We also allow end-users to create their own clas- sifiers without expecting them to be proficient in prompt engineering. Through an interactive pro-

---

very low, and it is therefore likely that rare fea- tures of the text would not be accounted for in the classification prompt because they would not be seen by the model during training and not brought up by the user. We envision future work will investigate ways to improve the clas- sifier while in use, perhaps by detecting if the text does not seem to conform to any of the classes according to the given descriptions.
tsetlin machine framework for text classification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3761-3770, Mar- seille, France. European Language Resources Asso- ciation.
Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christo- pher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1466-1477, Berlin, Germany, Association for Com- putational Linguistics.
Additionally, in order to keep the experiment consistent across users, we only evaluate in the personalized setting with one binary classification task, namely classifying important and unimpor- tant emails. Future work should investigate how the approach generalizes to different tasks, whether there are tasks that benefit more or less from per- sonalization, and whether the number and detail of classes affect the quality.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ethics Statement
We see two main risks of personalized classification in the real world. First, is possible that users, who are not machine learning experts, are unable to correctly estimate the quality of the classifier, and rely on it always being correct. This is particularly problematic if the validation set used to assess the model quality is small and not representative. As discussed in the limitations sections, to mitigate this issue, we expect that real-world deployment of BYOC will include mechanism to detect when the model should abstain from making classification.
Junfan Chen, Richong Zhang, Yongyi Mao, and Jie Xu. 2022. Contrastnet: A contrastive learning framework for few-shot text classification. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 36, pages 10492-10500.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by by-design voice interfaces. arXiv preprint arXiv:1805 10/90.
The second concern is the privacy risk from the use of third-party APIs on sensitive personal data. OpenAI does not store or train on data processed through their API. In the future, we expect partic- ularly sensitive applications will use open-source models such as LLaMa (Touvron et al., 2023), run- ning on a device of the user's is choice.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop, pages 177-190. Springer.
Specifically regarding our user study, the study was conducted with willing adult participants that expressed informed consent in writing, recruited through direct connections. Participants were com- pensated for taking part in the study. The email dataset will not be relcased. Examples provided in the paper are from the authors and not the study participants.
Jiahui Gao, Renjie Pi, LIN Yong. Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023. Self- guided noise-free data generation for efficient zero- shot learning. In The Eleventh International Confer- ence on Learning Representations.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1thh International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 38 3816-3830, Online, Association for Computa- tional Linguistics.
References
Bimal Bhattarai, Ole-Christoffer Granmo, and Lei Jiao. 2022. ConvTextTM: An explainable convolutional

---

John J. Godfrey, Edward Holliman, and Jane Mc- Daniel. 1992. Switchboard: Telephone speech cor- pus for research and development. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, volume 1. pages 517-520. IEEE Com- puter Society.
Melvin Earl Maron. 1961. Automatic indexing: an experimental inquiry. Journal of the ACM (JACM). 8(3):404-477.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint wxiv:2202.12837.
Braden Hancock, Paroma Varma, Stephanie Wang, Mar- tin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language ex- planations. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 1884-1895, Mel- bourne, Australia. Association for Computational Linguisties.
Anders Giovanni Møller, Jacob Aarup Dalsgaard, Ari- anna Pera, and Luca Maria Aiello. 2023. Is a prompt and few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks.
OpenAI. 2023. Gpt-4 technical report.
Swapnil Hingmire, Sandeep Chougule, Girish K Pal- shikar, and Sutanu Chakraborti. 2013. Document classification by topic labeling. Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 877-880.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Arshad Javeed. 2023. Hawk: An industrial-strength multi-label document classifier.
Reid Pryzant, Dan Iter, Jerry Li. Yin Tat Lee, Chen- guang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03405
Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. Promptmaker: Prompt-based prototyping with large language models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA '22, New York, NY. USA. Association for Computing Machinery.
Reid Pryzant, Ziyi Yang. Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Automatic rule induction for efficient semi-supervised learning. arXiv preprint arXiv:2205.09005
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Masahiro Kaneko, Graham Neubig, and Naoaki Okazaki. 2023. Solving nip problems through human-system collaboration: A discussion-based ap- proach. arXiv preprint arXiv:2305. 17789.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Reseurch, 21(1):5485-555
Kamran Kowsari, Donald E Brown, Mojtaba Hei- darysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura E Barnes. 2017. Hdltex: Hicrarchical deep learning for text classification. In Machine Learning and Applications (ICMLA), 2017 16th IEFF Interna- tional Conference on. IEEE
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak su- pervision. In Proceedings of the VLDB Endowment. International Conference Very Large Data Bases, volume 11, page 269. NIH Public Access.
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang. and Felix Hill. 2022.
Can language models leam from explanations in con- text? In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 537-563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data pro- gramming: Creating large training sets, quickly. In Advances in neural information processing systems, pages 3567-3575.
Qian Li, Hao Peng. Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. 2022.
A survey on text classification: From traditional to deep learning, ACM Transactions Intelligent Sys- tems and Technology (TIST), 13(2)11411
Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In Fifth European Conference on Speech Communication and Technology.
Ana Marasovié, 1z Beltagy, Doug Downey, and Matthew E Peters. 2021. Few-shot self- rationalization with natural language prompts. arXiv preprini arXiv:2111.08284
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the

---

2021 CHI Conference on Human Factors Comput- ing Systems, pages 1-7.
Sida Wang and Christopher D Manning. 2012. Base- lines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Shori Papers), pages 90-94.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.0110S.
Sinong Wang, Han Fang. Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. arXiv preprint MrXiv:2104.14663
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics; Main Volume, pages 255-269, Online. Association for Computa- tional Linguisties.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11003.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta. Mark Riedl, and Yejin Choi. 2022. Reframing
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dy- namic memory and self-reflection. arXiv preprint arXiv:2303.1 11366.
human-AI collaboration for generating free-text planations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 632-658, Seattle, United States. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Ad- vances neural information processing systems, 30.
Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu, Hu Yuan, Guoao Wei, Xiang Pan, Xin Tian, Libo Qin, et al. 2021. Fewelue: chi- nese shw-shot learning evaluation benehmark. arXiv preprint arXiv:2107.07498
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 163 1-642, Seattle, Washington, USA. Association for Computational Linguistics.
Leiming Yan, Yuhui Zheng, and Jie Cao. 2018. Few- shot learning for short text classification. Multimedia Tools and Applications, 77:29799-29810.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint 272922200..0992..
Shengli Sun. Qingfeng Sun, Kevin Zhou, and Tengchao Lv. 2019. Hierarchical attention prototypical net- works for few-shot text classification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLPP, pages 476-485, Hong Kong. China. Association for Computational Lin- guistics.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Wou Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Lever- aging large-scale language models for text augmen- tation. In Findings of the Asvociation for Computa- rional Linguistics: EMNLP 2021, pages 2225-2239, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, shangwei Guo. Tianwei Zhang, and Guoyin Wang, 2023. Text classification via large language models.
Yordan Yordanow, Vid Kocijan, Thomas Lukasiewicz. and Oana-Maria Camburu. 2022. Few-shot out-of- domain transfer learning of natural language expla- nations in a label-abundant setup. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1486-3501, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
A Question Generation Example
Gokhan Tur, Dilek Hakkani-Tür, and Larry Heck. 2010. What left be understood atis? In 2010 JEEE Spoken Language Technology Workshop, pages 19 24.
Huaxiaoyue Wang, Gonzalo Gonzalez-Puman Pumariega. Yash Sharma, and Sanjiban Choudhury. 2023. Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of-thought arXiv preprini ar0iv 16744.

---

Text
The significant co-occurrence between men's 's violence against female partners and child abuse and neglect is well documented. with their mothers. This issue is salient to isafe, a New Zealand-based Internet intervention study testing improvement in safety decisions and mental health outcomes for women experiencing intimate partner violence. This article discusses the legislative, professional, and ethical considerations that contribute to the development of the child safety protocols and discusses the development of the isafe protocol. Hypothetical scenarios of the application of the isafe protocol are used to illuminate the issues and provide basis for future discussion.
less clear how child safety should be managed in family violence research
Questions
1. Question: Are there any specific keywords or phrases in the text that are commonly associated with any of the classes?
Answer: abuse; children; ethics; responsible conduct of research
2. Question: In the context of this text, what criteria would differentiate the child abuse class from the depression class?
Answer: The criteria differentiating the child abuse class from the depression class include the nature of the behavior (physical, emotional, sexual abuse towards a child), the perpetrator-victin relationship (parent caregiver abusing the child), the focus child safety and well- well-being, and legal obligations surrounding child protection.
3. Question: Are there any specific ethical considerations mentioned in the text that could help us better understand the criteria for the child abuse class?
Answer: The specific ethical considerations mentioned in the text are not provided. making it difficult to precisely identify criteria for the child abuse class. However, based on the text's context, can be inferred that ethical considerations could include ensuring the safety and well-being of the child, protecting their rights, maintaining confidentiality, adhering to professional standards, and navigating the complexities of intervening in cases of child abuse. These considerations likely inform the development of child safety protocols and guide decision-making processes in the iSafe study.
Figure 2: Example of questions and answers generated on one abstract from the Web of Science dataset (Kowsari et al., 2017).

---

B Prompts Used for Experiments
B.1 Question Generation
System Instructions:
You taking part interactive study where the goal to build personalizable classifier for user. This strictly following user's class descriptions and criteria for classification. If user doesn't cover certain case, means trying find patterns user's thoughts or criteria make best guess.
The goal this stage in the process building a classifier we can refine the user's class description and better understand their criteria. way we choose annotate our training data by asking user's questions about the text to better help us understand the classification task at large.
annotate some training data, so
This doesn't mean asking questions about what the current text means or the specific context of the text, as answering these questions won't t always allow us to generalize the answer to other examples.
The questions that you ask must be ones that if answered, could help us broaden and improve the scope of the user's class descriptions. The questions should relate to the class names and descriptions provided, and should even ask clarifying questions about the class descriptions if needed. The questions should not be vague or obvicus. They should be questions that only the user who is trying to perform classification would know, not questions with answers that are obvious to anyone.
Some other questions may include how might you differentiate between classes, if you are deciding between two classes for this specific text.
You could also ask if there are any specific keywords or phrases that are commonly associated with certain class. Also, do not just directly ask the user what class the text belongs to this is not the goal of this stage. Don't just ask these question, since these just examples of questions you could ask.
In the prompt, also display questions that previously asked by the model, followed by the user's respanses to those questions. If needed, your questions should build upon previously asked questions and answers, you can better understand the user's intent.
Additionale tionall,, along with the question you ask, also explain why you asked the given question and what you hope understand from the by asking the question. Your response should be formatted like:
Thoughts: <thoughts>
Question: <question>
Explanation: <explanation>
User Instructions
For this classification task, the classes that the user chose were: <class_1> <class_2> <class_n> name. Here is description of the task hand (why the wants to build this classification task): <classification task_descriptions
Each class also has corresponding class description, which describes the criteria the follow to belong to a given class. are the class descriptions of each class.
<class_ 1_t3__1__mm_>> <class <class_1_desees ipttinniioonn <class_2_name> <class_2 <class_2.d
<class___name>> <class_n_description
Here is the current text that we are annotating:
Start of text s[text)
End of text
Based on this text and the classes we are trying to classify the text under, we generated the following questions and answers about the text to help us classify it. There might be no questions generated yet.

---

<question_12
<answer_1>
<question_2>
<answer_2>
<question_m-1>
<answer_m-13
We want to ask another question about the text to help us improve or broaden the scope of the class descriptions, as per the instructions provided. The question could also be a follow up question to the previous questions. What should it be and why?;
B.2 Few-shot Classification (During Interactive Annotation)
System Instructions:
You are taking part in an interactive study where the goal is to build personalizable classifier for user. More specifically, into classes of their own choosing. This means strictly following the user's class descriptions and criteria for classification. If user doesn't cover a certain case, it means trying to find patterns in the user's thoughts criteria to make best guess.
need to create a classifier that classifies the text of a user
The goal of this stage in the process is to classify training data, so can understand if the current class descriptions are covering enough use-cases for them to be generalized and deployed for all texts. In this prompt, are focusing single text and trying to classify it into one of the classes the user has inputted.
Additionally, for your reference, have also asked the classify this text. Use those questions in order to make your classification.
series of questions to help
you provided any information questions, then make your best guess what class the text belongs
Furthermore, beyond just classification, also explain your thoughts you decide what text belongs to. After making the classification, then reflect on why you chose that class. Your response should be formatted like:
Thoughts: <thoughts>
Class: <class>
Reflection: <reflection>
User Instructions:
For this classification task, the classes that the chose were: <class_ <class_2 _>_ <class_n> name. Here is description of the task hand (why the user wants to build this classification task): sesssiffiattion_tssedeeerrrrrtte
Each class also has corresponding class description, which describes the criteria the text follow belong given class. Here the class descriptions of each class.
<class_ _elass___nnmm>> <class descriiption
<class_22 1aaas___nnamss name>: <class descrription
<class_n_nan _aass_n_name>: <class_n_des descrrieeioini
Here is the current text that we are annotating:
Start of text
$[text)
End of text
Based on this text and the classes we are trying to classify the text under, we generated the following questions and answers about the text to help us classify it. There might be no questions generated yet.
<question_1> <answer_13

---

<question_2> <answer_2>
<question_m> <answer_mi>
Now, given this information, classify the text above make sure to incorporate your thoughts and reflection as well.';
B.3 Refining Class Descriptions
System Instructions:
You are taking part in an interactive study where the goal is to build personalizable classifier for a user. More specifically, you need to create a classifier that classifies the text of a user into classes of their own choosing. This means strictly following the user's class descriptions and criteria for classification. If user doesn't cover a certain case, it means trying to find patterns in the user's thoughts / criteria to make best guess.
The goal of this stage in the process is to update the class descriptions that the user has inputted, based on the training data we are annotating. In this prompt, are focusing single text, which the user has answered questions about and explained to us why that example belongs in given class.
Using this information, and the correct class of the text, need to update the class description of the correct class, to broaden it's scope contain more posspleesposssiiiiiies.s Furthermore, beyond just the classification, also explain your thoughts you decide what class the text belongs to. After making the classification then reflect why chose that class. Your response should be formatted like:
Thoughts: <thoughts>
Description: <updated class _lasss_deccrppttionee. www.deeietcoooooooocooo claass>>>> Reason: <reason_why_jon iiasonn_wyyyoouu_udddtte__ttttt
User Instructions:
For this classification task, the classes that the user chose were: <class_1>, <class_2>, <class_n> name. Here is description of the task at hand (why the user wants to build this classification task): saapssffiicatioottte.essssriiiie
Each class also has corresponding class description, which describes the criteria the text must follow to belong to given class. Here are the class descriptions of each class.
<class_1_name" sclass_1_descriige <class_ 2_nam>>; <class_ desecrippion
<class_n_ ___mm>> <class_ wew.nnddeerrppiii
Here the text that annotating:
Start of text $(text) End text
this text and classes we are trying classify text under, we generated the following questions answers about the text help us classify it. There might no questions generated yet.
<question_1>
<answer_1>
<question_22
<answer_22
<question_mp>
<answer_m>
Using these questions and answers, we classified the text. Our initial classification was <model Londel__redicctonn. The actual class of the text is Ecorrect_clase The user explained the reason that this was the correct class was this explanation: exser_explanation>

---

want update the description the class <class_to tt_te_t_eeeeeeeeeeeeeeeee updated>> more accurate based this new information. without changing or altering the meaning the current description, what better class description for the class: sclass_to_be_updttee
there nothing to to the current class description from this example, then just copy the current class description word for word.
B.4 Summarization
System Instructions:
I had a long thread of text and wanted to take the summary, so I split it into smaller parts. We will provide the summary of the first <i-1> parts of the essay and the sunmary of the <i> part.
Use these summaries as context to provide a summary of the entire essay, Don't make the length of the summary of this one part of the text equal to length of the the summary of multiple parts of text.
Make the summary of the new thread proportional to how many parts of the text have been summar ized so far.
For example, if the summary of about 5 parts of the text are 480 words, the length of the summary of the sixth part should be around 80 words.
Remember, the focus of the summary should also be on: ccassffiiatio.tsss...ssrttttot
User Instructions:
Summarize the following text with focus preserving context. Format the reponse with style: free-text. Summarize with the context: slassificationnttts_deese
and with focus: want
eventually perform classification this text.
Summary of First <i-1> parts: Duumary_of_prrviooussoo Part <i>: scurrent part>


Reviews:
[{"review_id": "QrI7GxCH0H", "reviewer": "Reviewer_JyXx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and thorough review.\n\n**The Importance of User Interaction**\nOur main goal with this paper is to evaluate how we can leverage human interaction to improve the performance of NLP models in the real world, with a specific focus on classification. As Reviewer cNMD also notes, our objective goes beyond the standard application of an algorithm to a pre-labeled academic classification dataset. As such, we structured our experiments around the user study and evaluation in the personalized setting. Indeed, the human aspect makes the paper challenging to reproduce as a benchmark. However, it remains integral to our contribution. We aim to assess whether people can achieve good classification for their own use cases.\n\nOur most surprising result is that the most common way people outside of research interact with LLMs - by providing handcrafted zero-shot instructions - performs very poorly. Users need help to write good instructions, even in a use case they are very familiar with, such as email.\n\n**The Novelty of the Approach**\nWe propose a novel interactive approach to refine these zero-shot class descriptions, and we achieve a new state-of-the-art result for few-shot classification with a similar amount of annotation effort - 90% compared to 66% with handcrafted class descriptions (Table 2). As discussed in Section 3.2, we are the first to incorporate interactive and prompt refinement aspects. This combination enables BYOC to use the user’s annotation effort best while preserving the inference-time cost efficiency of a zero-shot-like approach.\n\nFurthermore, unlike prior work, we changed the input of the LLM in a novel way during the annotation process. In previous studies, researchers have used labels and explanations as inputs to their LLM for classification tasks. On the other hand, in BYOC, our inputs to the models are labels, questions/answers, and an explanation. This is far more effective than previous works because:\n- The questions presented to the user are easily understandable to a non-technical end-user, enabling them to provide more relevant and useful answers.\n- The questions increase the LLM’s ‘understanding’ of the classification task since we synthesize these questions into new class descriptions, which enables the class descriptions to capture more relevant information.\n- Using labels, Q&A, and explanations, we pack far more information per sample than previous methods. \n- As a result, we need far less data to annotate, and the annotation process is more relevant and user-friendly.\n\n**Using our Approach in other Tasks**\nWhile we only evaluate classification in this paper, a similar interactive approach would benefit settings such as recommendation and text generation. Future work should evaluate our approach on those tasks.\n\n**The Reproducibility of the Work**\nFor reproducibility and comparison with existing work, we also evaluate on a second dataset, WOS, which is a standard benchmark. We will release all our prompts (included in Appendix A) and all questions and answers used to obtain the class descriptions through BYOC. These are equivalent to the “BYOC dataset” and can be used to train and evaluate future methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is quite user-friendly to enable non-experts to build a high accuracy classifier. The result looks promising showing a 9% improvement over the few-shot state of the art, and reaches 82% of the accuracy of a model trained on a full dataset with only 1% of their training set. The author has also surveyed with 30 participants, who find this approach interpretable and would consider for their use cases.  This could be landed as a very useful application of LLM to build customized classifiers.", "reasons_to_reject": "I find this paper lacks technical strength or novelty. The application is good but it lacks more novelty in the method. Since it requires user interaction to build the classifier, it makes it hard to exactly reproduce the results and easily compare with other baselines on more datasets. I would recommend the authors to propose a way for others to reproduce the results more easily and how they could leverage the framework for other tasks. And evaluate it beyond one dataset as reported in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a few-shot text classification approach using an LLM. Rather than few-shot examples, the LLM is prompted with descriptions of the salient features of each class. These descriptions are coauthored by the user and the LLM interactively: while the user annotates each few-shot example, the LLM asks relevant questions that the user answers. Examples, questions, and answers are summarized to form the classification prompt. The main contribution is to interactively construct class descriptions for classification, by\nprompting the LLM to ask relevant questions and enables non-experts to build classifiers with high-accuracy.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "66Ullbj1el", "reviewer": "Reviewer_ixK3", "summary": "", "questions": "None", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and thorough review. \n\n**The Novelty and Expectedness of the Paper**\nOur paper demonstrates that using an LLM with zero-shot class descriptions leads to poor performance. This is surprising given the high performance of models like GPT4 in many tasks. We find that significant effort is required for prompt tuning, which end-users can only achieve in an interactive, guided setting. Please also see the response to Reviewer JyXx for further elaboration on this point.\n\n**The Sensitivity of our Results to the Prompt**\nRegarding our system design and sensitivity to details, we first optimized our prompts and hyperparameters through the author's interaction with the system on internal sample data by analyzing the model's output. We used both chain-of-thought and self-reflection to improve our prompts. Since our approach is interactive, changing a design choice that affects the user study is challenging and requires multiple user studies, likely with multiple user groups and larger sample sizes, to achieve significance. On the other hand, for the final experiment on both personalized and non-personalized data, the prompts were optimized on the validation data, as is standard practice. We conducted ablation studies (Table 2) of each aspect of our approach.\n\nSpecifically, regarding the instruction about taking part in an interactive study, we observed that without that instruction, the model would generate questions and outputs more tied to the specific input given (email or paper abstract). In contrast, with the given instructions, we qualitatively observed that the model behaved as if it was part of a longer context.\n\nWe will include further discussion of how we designed our prompts in the final revision of our paper, taking advantage of the additional page. We also discuss our prompt designs in the response to Reviewer cNMD.\n\nWe aim to build a practical classification approach and evaluate its usability with real users. We included the additional system details to emphasize the overall usability. These details are relevant to contextualize this work and give validity to the survey portion of our user study.\n\n**Our Choice to use GPT-3.5**\nRegarding using GPT-3.5, our choice was due to speed and cost, especially in the interactive setting where speed can be a factor for usability. We will include an offline experiment that exclusively uses GPT-4 in the final version of the paper.\n\nWe thank the reviewer for the suggested improvements to the style and presentation of the paper and will make sure to include those in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is an interesting idea and timely topic due to popularity of LLMs and not yet their properties well understood.", "reasons_to_reject": "The result is not that surprizing and the details are not yet presented in the best way so the paper may benefit from revisions. The findings can be very sensitive to specific details like prompts used. But the sensitivity to those and ablations have not been run. The specific choice of prompts has not been justified, e.g. the LLM is expected to operate with the concepts “study” and “essay”. It would be interesting to probe that directly how well it can do that .\n\nThe setup is a bit artificially since if we assume that GPT4 is the best model around to assess the results, then why we don’t use it directly for classification and all the refinements as well? Smaller GPT3 is used instead for those.\n", "questions_to_authors": "None", "paper_topic_and_main_contributions": "The paper shows how the effectiveness of few-shot training examples can be improved through the interaction with the foundation LLM, in a manner similar to active learning approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Examples in Figure 1 is a bit simplistic since only shows how “appointment” topic is added to the list of non-spams.\n\nSection 4 formal descriptions are hard to follow without specific examples.  Same applies to section 3.1.\n\nMore diversity of the included transcripts would help: e.g. the instruction “You are taking part in an interactive study” repeated several times making up a substantial portion of the presented transcript.\n\nSome system-aspect details can be skipped if they are not important for the experiment, e.g. “she can share it with others, like family  members with similar email needs” or “the user selects the source of data”\n"}}, {"review_id": "dvJWQS581F", "reviewer": "Reviewer_cNMD", "summary": "", "questions": "A. You state that \"The model sees both the correct and predicted labels, which allows it to weigh how important is the new example\" (line 385). Is there a basis for assuming that this is indeed what the LLM does with this information?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and thorough review.\n\n**The Design of Our Prompts**\nWe gradually improved a basic prompt by adding instructions, observing performance on large amounts of data, and author interaction. This resulted in the prompt presented in our paper.\n\nSpecifically for the question-generation prompt, we observed that the LLM generated similar questions for each example. As a result, we added an extra section in our prompt in which we instructed the model to not only ask very general questions. Similarly, in the same stage, we observed that the model occasionally asked a redundant question on further examples, such as \"Are meetings important?\" on multiple meeting emails. We added a section in the prompt to overcome that. Overall, the question-generation prompt was optimized to broaden the range and scope of the user's class descriptions.\n\nFor the class description refinement prompt, specifically in response to question A, we observed that updating the class description for correct and incorrect predictions is necessary. We found that we could only achieve that by providing the model with both the incorrect prediction and the correct label.\n\nFinally, for the prompts used to classify the test set, we manually tuned using the acquired validation data from the personalized experiments, following standard practice. We will include this discussion of our prompt design in the final revision of our paper, taking advantage of the extra page. We also discuss our prompt designs in the response to Reviewer ixK3.\n\n**The Scalability of Our Approach**\nWe understand the concern about the scalability of our approach. In the comparison on Web of Science, which uses a hierarchical approach to reduce the number of classes, we also evaluated BYOC in a non-hierarchical setting where it attempts to predict one of 11 classes directly. We found it materially worse than the hierarchical case, with an accuracy of 67% on the test set. We will include a discussion of this result in the final revision of this paper.\n\nFurthermore, assuming that a fine-tuned specialized model will perform similarly to BYOC is not unreasonable. However, we cannot expect non-technical end users to be able to fine-tune a model for every user case. We can also assume that an end-user would not go through the process of annotating large datasets to fine-tune a model for their classification tasks. Therefore, the alternative is to use a one-size-fits-all model, which we proved to have lower accuracy than the proposed approach.\n\n**Analyzing the User's Ability to Effectively Answer the Model's Questions**\nRegarding the user's ability to answer the model's questions effectively, while we did not evaluate with experts on WOS, our personalized experiment should be a good proxy of what is achievable, as everyone is an expert of their inbox. Our results outperform existing few-shot fine-tuning by a large margin. On WOS, our answers were provided by non-experts and, on average, required 3-5 minutes each. As such, it is reasonable to assume that experts with access to the same tools and knowledge bases would have similar or better answers. The amount of effort would still be lower than annotating enough training data. Conducting a user study is one possibility for further analysis of this dataset.\n\n**The 'Importance of Personalization' Experiment**\nOur \"Importance of Personalization\" experiment aims to compare a model trained on one user's data with a model trained on all users' data. Unfortunately, a few-shot method that applies to only one user cannot be used with all users because the prompt would be far beyond the context window of any LLM. Hence, we need to compare a few-shot method to a fine-tuned method. The choice of fine-tuning BERT, rather than a larger model, was then dictated by computational constraints. Future work could evaluate larger fine-tuned models such as LLaMA.\n\n**The Missing Citation and Style**\nFinally, regarding the citations we missed (ConvTextTM and Hawk) and changing the notation of \"classifier purpose\", we thank you for noting these down, and we promise to include them in the revision of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper deals with some very important and under-explored real world problems, namely those of building personalized classifiers and of utilizing interactive feedback with a human in the loop and different kinds of user input, and not just standard application of an algorithm to a pre-labeled academic classification dataset.\n- The concept that the ultimate goal of interaction with the user is to arrive at refined descriptions of the target classes, rather than a labeled (or pseudo-labeled) set of examples, and that this would better serve the classifier, is an interesting one, and may lead the community in novel directions.\n- The overall methodology looks sound.\n- The work is well-written and easy to follow.", "reasons_to_reject": "- I would have liked to see more discussion of the way the prompts were engineered, given that these prompts look extremely complex. First, it is not so clear that the LLM (even GPT-4) is truly able to comprehend such long and complex instructions, nor that all components of this instruction are actually necessary. And second, I wonder what was the process by which the authors arrived at this particular set of instructions.\n- Presumably this approach would not scale easily to a scenario with many classes and/or with very fine-grained class distinctions — in terms of both computation, annotation effort, and performance — mainly since a larger amount of questions and increasingly complex user answers would be required to construct a good description for each class. In such a scenario, it is likely that given the same amount of effort by the domain expert, existing approaches for few-shot fine-tuning would ultimately reach superior results. \nOn this note, while the results for WOC are encouraging, obtaining multiple user answers with a quality similar to the example in App. A is no easy feat. I will say that while I agree with the claim that the results can be considered a lower bound on what *can* be achieved by a domain expert, this does not necessarily imply that it is a lower bound on what *will* be achieved by a domain expert in a realistic scenario, at least without devoting a very significant amount of time both on the part of the domain expert and on properly guiding the expert on the importance and expected form of the provided answers. Given the significant overhead, the question would then turn to whether a given amount of effort would be better spent on complex feedback to the LLM or on more traditional labeling of a larger set of training examples for fine-tuning. This is of course an open question.\n- While I completely agree on the importance of being able to build personalized classifiers, I am not so convinced by the text in lines 544-559. The model, methodology, and type of input data are all very different for BYOC and the fine-tuned setting, it is difficult to conclude from this experiment that the performance difference stems specifically from personalization.\n", "questions_to_authors": "A. You state that \"The model sees both the correct and predicted labels, which allows it to weigh how important is the new example\" (line 385). Is there a basis for assuming that this is indeed what the LLM does with this information?\n", "paper_topic_and_main_contributions": "In this work, the authors propose a new approach for using LLMs to perform text classification with a human in the loop, which is oriented towards solving a personalized classification problem given a limited amount of feedback from the user. The basic approach is that the phase of interaction between the LLM and the user is used to take the description of the the target classes initially given by the user, and refine them based on a set of few-shot examples and some back-and-forth between the LLM and the user. Then, at inference, the refined class descriptions are given as input to an LLM in order to decide how to classify a new input example. The authors conducted a test study with users that use this interactive feedback mechanism to construct a personalized binary classifier for each user. The authors show that this approach outperformed other baselines for incorporating the different kinds of inputs given by the users for classification by an LLM. In addition, they report the performance of this approach over an academic classification dataset.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "HDLTex is technically not the SOTA for the WOS-5736 dataset, there are some newer works - ConvTextTM (https://aclanthology.org/2022.lrec-1.401), Hawk (https://arxiv.org/abs/2301.06057).\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- I would suggest changing the notation of the \"classifier purpose\", just because _p_ is strongly associated with the context of probabilities whereas here the meaning is of a class description."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to learn text classifiers based on interacting with an LLM to annotate a small amount of training examples, with the goal of generating class descriptions for use in a prompting-based approach. They find that their approach outperforms other prompting-based approaches and conduct a user study that indicates positive reception of the approach.  

Soundness: 

Reviewers give fair "soundness" ratings with 3/3/4 though list several concerns in their written reviews. 

Reviewers voice concerns regarding the technical details of the experimental evaluation, namely the focus on only one public benchmark (WOC) and the lack of justification for the specific prompts used. Additionally, since the proposed approach relies on interactivity with humans, this naturally limits reproducibility. The authors however try to mitigate the lack or reproducibility by releasing much information on the conducted study. 


Excitement: 

Reviewers also give fair "excitement" ratings with 3/3/4. 

One limiting factor excitement are presumed difficulties in scaling the proposed approach to many classes, or classes with fine-grained distinctions, in which case a potentially very large amount of interactions would be necessary with the LLM. The authors acknowledge this limitation in their response and will add discussion on this. 

The novelty is questioned by some reviewers, but the author response alleviates this by pointing out that their approach outperforms other prompting-based approach significantly, albeit only evaluated on one benchmark dataset and one small proprietary dataset.
Decision: Accept-Findings