The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators
Authors: Tzu-Heng Huang
Catherine Cao
Vaishnavi Bhargava
Frederic Sala
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Automated Data Labeling; Code Generation; Weak Supervision
Primary area: other
TL;DR: We propose an alternative program distillation approach to replace expensive annotation processes that require repetitive prompting for labels.

Abstract: Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models. Unfortunately, this comes at a cost: employing top-of-the-line models often requires paying thousands of dollars for API calls, while the resulting datasets are static and challenging to audit. To address these challenges, we propose a simple alternative: rather than directly querying labels from pretrained models, we task models to generate programs that can produce labels. These programs can be stored and applied locally, re-used and extended, and cost orders of magnitude less. Our system, $\textbf{Alchemist}$, obtains comparable to or better performance than large language model-based annotation in a range of tasks for a fraction of the cost: on average, improvements amount to a $\textbf{12.9}$% enhancement while the total labeling costs across all datasets are reduced by a factor of approximately $\textbf{500}\times$.

The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators
Tzu-Heng Huang, Catherine Cao, Vaishnavi Bhargava, Frederic Sala University of Wisconsin- fthuang273, ccao35, vbhargava3) (winga edu, fredsalašcs wisc.
Abstract
Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models. Unfortunately, this comes at a cost: employing top-of-the-line models often requires paying thousands of dollars for API calls, while the resulting datasets are static and challenging to audit. To address these challenges, we propose a simple alternative: rather than directly querying labels from pretrained models, we task models to generate programs that can produce labels. These programs can be stored and applied locally, re-used and extended, and cost orders of magnitude less. Our system, Alchemist, obtains comparable to or better performance than large language model-based annotation
range of tasks for a fraction the cost: on average. improvements amount to 12.9 % enhancement while the total labeling costs across all datasets are reduced by a factor of approximately 500 We release our code here: https: //gs::///iiii //ggttuu.... com www./ooo///pacee wtt.m//pppocceeeaablceeeeeiiiiie
Introduction
One of the most exciting developments in machine learning is the use of large pretrained models to act as annotators or labelers 2 5 6 7 This includes the use of large language models (LLMs) like GPT-4 [ and Claude 3
10] This process offers multiple benefits. First, pretrained models are an efficient way to annotate and have the potential to partially or fully replace expensive human crowdworkers 2
12 Second, this approach allows for distilling large models into smaller,
task-specific models that can be deployed locally at lower cost 3 13 g This is additionally
important in settings like healthcare and finance where privacy laws require the of local models.
Despite this promise, pretrained model-based annotation has several drawbacks that stymie its adoption. These drawbacks include
High Cost: Labeling a dataset can be expensive. This is particularly so in cases where each data point consists of many tokens. For example, we find that labeling a moderately-sizee dataset [14] with 7,569 data points using GPT-4 costs over $1,200.
Lack of Extensibility: Making even small changes specifications necessitates re-running the entire pipeline to obtain new labels. This inflexibility means the resulting labels are static.
Inability to Audit: API access to pretrained models does not permit inspecting most aspects the model. Users must simply accept the provided labels with only minimal additional information.
Techniques that ask the model for explanations for its decisions may not be reliable [15] 16 [7].
We address these obstacles through simple but surprisingly powerful notion. Rather than having pretrained models label data, we task language models to generate programs that can output labels. These synthesized programs serve annotators. capturing the underlying logic used by the models when annotating. other words, instead of distilling powerful model to label dataset (and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt Template in Alchemist
[Task Description]
Write bug-free and executable function python to label comment on Youtube spam ham.
[Labeling Instruction]
If the comment spam, function returns 1. the comment is ham, function returns
If the comment cannot be determined as spam ham, function returns -1.
[Function Signature]
def label_spam (lext_commee commenty:
[Task Description]
Write bug-free and executable function python to label the toplc of biomedical document.
[Labeling Instruction]
If document about colon cancer, returns 0. document is about lung cancer, returns 1.
If document about thyroid cancer, returns 2. document cannol be categorized, retums -1.
[Function Signature]
def cancer (document):
Legg aeg_arore.ee Jerm Teanddeesoy "quairg' "aagg". S4tterarn war's "Iaeeraay"
Oxceert Bocumen, bwerl
dreor of consett off, encll sooes
retars
ofiee oecer aryltem socmnat for fore de oftee cince somel Tip apriidln aeptter socuert tern
mecoiaatti yougee coneett ckcleenbiiddi aulgene lin,, comerntt
koot
Ureor of coleer hay many susond
cur cenee not Enxtes.oennt Merrooucaeee
Iepert deneen Uivertt
Bortst clace wet foolon, panee Tuny cancarts
reture
coNttIiN aie and, recure (han
returE accusee cantor do conagerznes or belenge to auteiple canggnsar
Figure 1: Examples of generated programs and their prompts. These are synthesized by GPT-4 for spam detection and cancer identification tasks. Programs use regular expressions (left program) and keyword matching (right program) as their labeling logic to classify data points.
subsequently training a smaller model on the labeled data), we distill directly into code (Figure D These resulting programs can either make predictions directly or can label training dataset then train downstream model using i
This simple notion resolves all of the challenges related to pretrained model-based annotation. First, API calls scale with the number of programs instead of the number of data points. That is, since we generate programs that can themselves make any number of predictions locally at no cost, we can reduce the number of API calls by orders of magnitude. For example, for the dataset described above [4], the number of GPT-4 calls was reduced from 7,569 (the size of the dataset) to 10 (the number of generated programs), resulting in massive cost reduction from $1,200 to $0.70, 1,700- fold decrease. Moreover, code can be easily inspected, corrected, and extended, allowing seamless adaptation when prediction classes labeling rules change.
While a powerful idea, distilling model into code presents several challenges. First, any particular program may be inaccurate, fail to compile, may otherwise be flawed, resulting in noisy program outputs. We address this obstacle by applying weak supervision, a framework for dataset construction from multiple noisy sources of signal 118
Next, operating on non-text modalities is challenging. We handle this via a simple two-step approach that first extracts high-level concepts and then uses them in concert with local feature extractor to enable tractable program generation.
19
20 21
Contributions. We propose an alternative approach to replace expensive annotation processes that require repetitive prompting for labels. We developed a system called Alchemist that implements this idea. Empirically, Alchemist improves performance five out of eight datasets, with an average enhancement of 1.99%---ll reducing total costs by a factor of approximately x00x. Finally, we introduce and validate extensions that address non-text modalitics.
2 Related Work
Our work relates to LLM-based annotation, prompting, and the weak supervision framework.
Using Large Pretrained Models for Data Annotation. Large pretrained models have demonstrated powerful capabilities using zero-shot prompting across a wide range of tasks [.. One promising development is their potential to serve data labelers, which can reduce the cost and human effort in
'The latter option is preferable, as these models can often generalize beyond their source of supervision B

---

Prompts
Labeling Programs
Specialized Prompt Design
Language Model
Program Output Collection
Output Aggregation
Program Generation
Figure Overall workflow for Alchemist.
data labeling D 2 12 Existing research in this area mainly focuses on approaches that allow for efficient inference, enhanced label generation, and distilling into smaller but specialized labelers [3] 6 [7 8]. However, scalability is the main limitation in these approaches, as making inferences via querying API for data examples can be cost-probe post-prohibitive. To tackle this challenge, rather than prompting for labels repetitively, we propose prompting pretrained models for programs that use synthesized labeling logic and can thus serve alternative data labelers.
Prompt Engineering In-Context Learning. In-context learning adapts pretrained models to new tasks without additional fine-tuning D. It involves providing relevant examples demonstrations to solve the task, such as pairs languages for translation [22]. By including task-specific examples, models can better understand the task at hand. Adding few data points as demonstrations [23 is commonly suggested when models act as data annotators. Moreover, they can be selected [24 25). retrieved (26), or more efficiently, generated [27]. We explore various types of supplementary information that can added to Alchemist to help improve program generation and permit more control over the labeling logic used in the programs.
Weak Supervision Framework. Weak supervision enables the rapid creation of large training
datasets by aggregating theap-but-noisy signals derived from various labeling sources [[8 [9 21
(28).
These sources can be crafted by domain expertise, using labeling heuristics, or even trained on smaller,
weaker classifiers [29] 30 (1 Recent advancements in code generation open up the potential to automate the heuristic design process. Frameworks such as Scriptorium WS 33), and DataSculpt (34)
have been developed to take advantage of mode-gennrattie models [35 9
36 to craft weak supervision
sources through prompting. While similar in spirit to our approach, these have several drawbacks: ScriptoriumWS requires more human effort in prompt engineering to better guide code-generation models. Both Scriptorium WS and DataSculpt can perform poorly in tasks requiring specific domain expertise and, most importantly, they do not handle modalities beyond text-unlike Alchemist.
Alchemist System
We begin by presenting general annotation workflow in Alchemist, followed by detailed discussion of each key step.
General Workflow.
The process is depicted in Fig. First, users select unlabeled dataset and create simple prompts to instruct language models to generate programs that incorporate labeling logic. These prompts integrate relevant information and may vary in their design, allowing for the synthesis multiple programs. Next, given set of generated programs and their outputs, we apply weak supervision techniques obtain set of aggregated labels. Finally, the labeled points be used to train a distilled model that can be stored and used locally.
3.1 Prompting Strategy
We propose general and extensible prompt template for querying language models to generate annotator programs. This general template consists three key components:
Task Description: Provides the model an overview of generated program's desired objectives. Labeling Instructions: Specifies classes and the expected structure of the program's output.
Function Signature: Describes the function's 's name and the input types to be used.

---

What primitive concepts to classify waterbirds and landbirds?
LLM Generation
Wing
Bill
Foot
rounded, elliptical
short, compact
toed, grasping
long, pointed
long, streamlined
paddling, swimming
A bird that <Body> is <Description>
Alchemist System
Concept Embedding
Labeling Programs
Image Embedding
0.62
0.38
0.13
0.87
0.22
0.78
Waterbirds Landbirds
Similarity Score Set
Figure 3: Alchemist can handle rich modalities through simple extension. First, language model identifies task-specific concepts (top). Then, local multimodal model is used as feature extractor for these concepts, producing low-dimensional feature vectors that can be ingested hy generated labeling programs.
This simple but general template allows for flexible incorporation of various types of information, enabling the model to generate programs that are tailored to specific requirements. Two sample prompt templates in Alchemist are displayed in Fig I
Using Supplementary Information. Drawing inspiration from few-shot prompting [37 [. where users provide demonstrations (i.e., data points with their labels) to enhance generated responses, we explore various types of supplementary information that can be integrated to assist models in synthesizing programs. This approach is particularly useful for scenarios where language models may lack the expertise to generate effective programs, or where specific adaptations in labeling logic are required. Such information generated by language models themselves. Additionally, it can be combined with retrieval- sarreval-augmenteed generation (RAG) systems
be crafted by users themselves, domain experts or, more efficiently,
38 39
access external knowledge.
We explore various types of supplementary information to assist in code generation, starting with high-level concepts and then progressively looking into more practical details to control programs.
Dataset and Prediction Class Description. First, supplementary information can include relevant background details about the purpose for which the dataset was built and high-level information about the dataset, such as definitions for each label class. By providing this context, the language model can better understand the task at hand.
Data Exemplars. Furthermore, recommend including a small number of labeled data examples in the prompt. This can help language models better comprehend the specific problem. Examples act as concrete illustrations of the task, offering clearer understanding of the expected output. This be particularly beneficial when dealing with a complex problem.
make of Seyword-scarching techniques (c.g.. [. For instance, in situations such as spam detection or topic classification, certain words or phrases may have strong correlation with specific classifications, Providing several keywords in the prompt may lead models to create labeling programs that explicitly search for the presence or absence of these keywords. This allows for more targeted and precise labeling.
Keywords. Next. labeling logic in programs
Specialized Labeling Rules. Finally, more prior knowledge such as heuristics, specialized labeling rules, guidance, and domain-specific knowledge can be integrated into the prompt. This information can provide concrete labeling steps on how to label specific classes and offer greater control over the logic implemented in the generated programs.

---

def Laze-_iin, Stw.iAinniiaaatecooo (asgeCinage)
sef Labe www.srreeeiggg......o poodlngg wwwggggaareeee
function classefy inges of is landzares or Notorbirts.
sinilarity
Paroacters:
str: "landbrrd" "saterbire" basnd hearistic analosis.
toed, prappprrrgroccoo soore (fleeti) Sinblarizy soore for 'toed, grasping"
pad www.siiwinnnnncrrr..rr score Moatte Siniloriky scare for 'paddlang, snimang'..
(eg (tage
Returna:
land Assuntng sone bisv indicates veter aurroandings. thrrihh blue threshots e 56, 100
thresheld 0.5
tool soree thrcooldd ane pootling wow.ggswwwinng soaming thresiadd:
"landbird"
graee gixets
inglelll grees ttresteldl
aestting
tass_rrmping
reture "vsterbird"
(Lng 1y5,2)
eltr ahoetard grosolng scere pastling cataning scoret 9.11 Sterlar scores
siot_ploela a2. (, blue, thrsatstt
reture -1
blue, preen_
raturn "aasbirr"
eloe:
clses
return "Lastbire"
retern "saterbire"
Figure 4: Program examples generated by GPT40 on Waterbirds dataset. The left program synthesized by directly asking for labeling program when the input an image (raw pixels), while the right program uses Alchemist's 's extension. The former labels birds using the dominant color in the image, which can be predicted incorrectly due to spurious correlations (e.g., background).
Overall, supplementary context is provided before the task description to enhance language models' understanding of the task. This, in turn, enables models to generate programs that are more effective and tailored to the specific requirements of user needs.
3.2 Dataset Synthesis
While generated programs can efficiently annotate data, these programs may produce outputs that are noisy or inaccurate. However, as such programs may employ different techniques, such pattern- matching, heuristic rules, or other approaches-ease with its own strengths and limitations there may be complementary signal in their outputs. This means impact of noise. To do so, we apply weak supervision techniques [8] 19 20 21]. This process starts by learning model of the reliabilities of the programs. Once learned, this model enables aggregating label outputs from different programs into high-quality pseudolabels.
aggregate them to mitigate the
Alchemist is compatible with variety of weak supervision aggregation models, called label models, providing flexibility in the choice of the weak supervision approach. For simplicity, in this work, we focus on using the Snorkel framework [99] which is a standard and widely-used approach in the weak supervision community.
3.3 Extensions: Handling Complex Modalities.
Crafting programs that operate over text is relatively easy for large language models. More complex data modalities, however, can be far more challenging. Consider images illustrative example. Even employing state-of-the-as multimodal models, e.g., GPT-4o (40) and GPT-4V 9. to seek programs operating over sample images may not produce satisfactory results.
To address this challenge, we extend Alchemist's pipeline to include an intermediate step. Specifically, we convert the raw data (i.e., in our example, image pixels) into set of features representing high-level concepts. These concepts are obtained by prompting a language model (or, potentially, a multimodal model) to identify task-relevant notions. For example, for a bird categorization task, models may identify "wing shape," "beak shape," or "foot type" as informative concepts for distinguishing between bird species. Next, we any open-source local multimodal model, like CLIP [41,, feature extractor for the identified concepts, producing www-dimessional feature vectors that can be easily ingested by generated programs. As such models free, this does not increase cost.
Fig. 3 and Fig. present examples of generated high-level concepts and the corresponding programs used for the Waterbirds dataset, where the task is to distinguish between landbird and waterbird specices 42]. This simple approach can be applied to any data modality where we have access to local multimodal model (i.c., model operating the modality of interest and text).
5

---

YouTube
SMS
Yelp
IMDb
Est. Cost
Accuracy
Est. Cost
Fl-score
Est. Cost
Accuracy
Est. Cost
Accuracy
Zero-shot Prompting Alchemist with GPT-3.5
0.096
0.871
0.240 0.004
0.907
3.873 0.005
0.845
3,400 0.004
0.737
0.004
0.891
0.900
0.575
0.662
MedAbs
Cancer
Finance
French
Est. Cost
Accuracy
Cost
Accuracy
Est. Cost
Accuracy
Est. Cost
Accuracy
Zero-shot Prompting
1.944
0.311 0.346
15.925
0.716
0.201
0.641
0.641 0.006
0.611
Alchemist with GPT-3.5
0.006
0.003
0.968
0.007
0.660
0.690
Table I: Testing performance of the distilled model reported for each combination of method and dataset. The estimated cost is obtained by calculating the number of input and output tokens associated with GPT-3.5's pricing table [49]. Other models may be even more expensive.
4 Experiments
We study the capability of Alchemist empirically. Our goals are to validate the following claims:
Cost Reduction and Improved Performance (Sec. 4.1 Alchemist can reduce cost by orders of magnitude, while producing labels of similar or better accuracy.
Extendibility to Other Modalities (Sec. 4.2 Alchemist can operate with modalities beyond text.
Use of Supplementary Information (Sec. : Incorporating relevant information into prompts enables the generation of better programs, yielding more accurate pseudolabels.
More Diverse Programs Can Help (Sec. 4.4):
Increasing the diversity of generated programs created by different labeling logic enables better pseudo labels.
Comparing to Human-crafted Programs (Sec 4.5 Synthesized programs may be more effective in comparison to human crranne ones.
Datasets. We include diverse datasets covering text and image modalities. For text, include eight datasets that span three different types of language tasks. These include the YouTube 43
SMS 44
datasets for spam classification, IMDb 45) Yelp [45), Finance [46), and French 47 datasets for sentiment analysis, and the MedAbs 48 and Cancer [14 datasets for topic classification. We note that the Finance, French, MedAbs, and Cancer datasets relatively challenging, with points that require a degree of domain expertise for accurate labeling. For example, the French dataset requires a good understanding of the language. These may pose challenges for pretrained models.
For our extensions to richer modalities, we focus on image tasks. Our evaluation uses the Waterbirds dataset 44). This dataset is designed to assess models' robustness to spurious correlations and ability to handle distribution shifts. More details are Appendix A
4.1 Cost Reduction and Improved Performance
Setup. We open evaluation of Alchemist with text domain datasets and use GPT-3.5 generate programs. For each dataset, we input pure prompts without supplementary information into GPT-3.5 and generate 10 programs to We construct training datasets by aggregating the programs' outputs into pseudolabels with the weak supervision framework Snorkel [19]. We then train a two-layer MLP distilled model. We run five times with different random seeds and report their average performance. As our main baseline, we directly use language models to produce annotations per point. The resulting labels are used train distilled model for comparison. The prompt template used in our baseline approach and our training settings are provided in Appendix A
Expected Results. We anticipate that Alchemist can generate programs that can produce accurate labels while substantially reducing the expense of API calls.
Results. Table D presents the distilled model's performance on each testing dataset. We observe that label accuracy is improved on five out of eight datasets, particularly in challenging settings such as the MedAbs, Cancer, and French datasets, outperforming the baseline zero-shot prompting approach. We also report the estimated costs of building training datasets. The costs for zero-shot prompting depend on the number of tokens for the dataset. In contrast, Alchemist only prompts 10 programs for

---

Feature Extractor
Method
Average Accuracy (1)
Worst Group Accuracy (1)
Gap (U)
Vanilla Alchemist with GPT4o
0.395
0.367
0.028
Vanilla Alchemist with Claude
0.781
0.022
0.759
CLIP VIT-B/32
Zero-shot Prompting
0.820
0.318
0,502
Group Prompting
0.823
0,383
0.440
Alchemist with GPT40
0.805
0.283
0.522
Alchemist with Claude 3
0.774
0.463
0,410
CLIP VIT-L/14
Zero-shot Prompting
0.904
0.335
0.569
Group Prompting
0.791
0.240
0.551
Alchemist with GPT4o Alchemist Claude 3
0.802
0.467
0.335
0.737
0.346
0.391
Table 2: Alchemist on non-text modalities. We experiment with standard Alchemist (top), our proposed extension with two CLIP-based local models feature extractors, and CLIP prompting baselines. Alchemist achieves comparable performance on average accuracy while improving robustness to spurious correlations.
each task, resulting in significant reduction in the costs- by orders magnitude. This efficiency is the main advantage of Alchemist, as it allows for the creation of high-quality datasets with minimal expense. We include ablation studies with other weak supervision models within the Alchemist framework in Appendix C They successfully demonstrate the flexibility and robustness of using Alchemist.
4.2 Extending Alchemist to Other Modalities
Setup. Next, we validate the extension of Alchemist to richer modalities. We consider our approach, where prompt multimodal model such GPT4o and Claude to generate high-level task- specific concepts. We extract features for these concepts by employing CLIP as our local feature extractor. This converts raw pixels into feature vectors for the extracted high-level concepts, producing a set of similarity scores. Armed with these scores, we describe scores associated with their concepts in prompts and ask GPT4o and Claude for 10 programs. As before, aggregation procedure.
Snorkel
Baselines. We study two baselines. The first is the vanilla version of Alchemist, where directly ask GPT40 and Claude 3 to produce code that can operate on images (see left program in Fig. 4 The second is simple zero-shot prompting using CLIP, along with variant, group prompting approach that assumes access to spurious information and adds it to the given promp
We expect employing our two-step process can enable tractable program genera- tion. addition, we hypothesize that programs generated this way are beneficial targeting salient concepts and reducing the impact of irrelevant or shortcut features, thereby enhancing robustness.
Expected Results.
Results. We present results in Table 2 Our evaluation focuses on three key metrics: average accuracy. worst group accuracy, and the gap between these two measures. Ideally, robust model should achieve high average accuracy and high worst group accuracy while minimizing the disparity between the two. First, we see that directly asking programs to use may have very low performance (GPT4o) or may hugely suffer from spurious correlations, destroying worst group performance (Claude CLIP zero-shot). Our method addresses both cases. Compared to baseline methods, Alchemist demonstrates increased worst group accuracy and reduced gap between the average and worst group accuracies. This is key strength of Alchemist: targeting salient concepts to be used as features may help move models away from spurious shortcuts found in the data. This validates Alchemist's ability to handle complex modalities while improving robustness.
7

---

YouTube
Yelp
GPT-3.5
GPT-
Claude
GPT:3.5
GPT-1
Claude
GPT-3.5
GPT-4
Clando
GPT-3.5
GPT-4
Claude
General Promgt
0.92
0.92
0.56
0.64
0.62
0.75
0.65
0.82
0.78
0.71
0.77
0.77
Description
Data Excmplars
0.1
0.86
0.76
0.46
0.66
0.62
0.72
0.82
0.82
0.68
0.75
0.73
Keywerds
0.76
0.93
0.53
0.40
0.42
0.64
0.69
0.81
0.78
0.69
0.78
0.72
Labeling Rules
0.74
0.82
0.56
0.67
0.67
0.58
0.75
0.81
0.79
0.71
0.77
0.74
GPT-3.5
GPT-4
Claude 3
GPT-3.5
GPT-
Claude 3
GFT--3.5
GPT-4
Closde 3
GPT-3.5
GPT4
Claude 3
General Prompt
0.52
0.53
0.55
0.71
0.73
0.59
0.66
0.56
0.65
0.55
0.56
Description
0.49
5 Data Examples
0.53
I154
0.55
0.55
0.57
0.63
0.60
0.50
0.601
0.40
0.69
0.44
Keywords 0.55
0.55
0.55
0..5
0.55
0.46
0.66
1162
0,65
0.69
0.66
0.67
Labcling
Table 3: Testing performance of the label model is reported for each combination of prompting strategy and dataset. We observe that GPT-4 and Claude 3 (that may possess better comprehension capabilities) exhibit greater enhancements when provided with supplementary information.
SMS
Yelp
IMD6
3.6
0.81
2.60
0.575
0.247
2.06
6.730
3.64
0.722
3.22
0.735
0.727
20
24
20
Number Progams
Number Collected Progams
Numbor
Progams
Figure 5: Performance is reported using their average performance and standard deviations. Results indicate that the label model is improved when the number of diverse programs increases.
4.3 Use of Supplementary Information
Setup. We test how integrating relevant information into the prompt context can augment generated programs. Instead of manually crafting supplementary information, we harness the power of language models to generate and integrate. This approach is useful for challenging datasets where users may have the necessary knowledge or expertise to start. We evaluate the effectiveness of this approach, by comparing label model performance using programs generated by two different methods; pure prompting and in-context prompting. In-context prompting involves supplementary information, while pure prompting relies solely on the task description without any additional guidance. We employ GPT-3.5, GPT-4 and Claude as our program sources and synthesize ten for each strategy.
Expected Results. We hypothesize that providing supplementary information can enhance task understanding, demonstrate specific labeling logic, and offer concrete steps, ultimately leading to better programs for use.
Results. Table presents this comparative analysis label model performance using different type of information. We observe that by incorporating supplementary information into pure prompts, Alchemist can guide language models to generate more effective programs, which in turn produce more accurate pseudolabels. Improvements are particularly evident in the challenging datasets such as Finance and French. Moreover, this approach be combined with RAG systems include external knowledge bases and customize the relevant information. Such flexibility compared to zero-shot prompting another key strength of Alchemist, as programs can easily be adapted, augmented, and specialized.
4.4 More Diverse Programs Can Help
Setup. As shown in Table 3, incorporating differentt supplementary information results in varying degrees of additional improvement. Potentially, certain sets of supplementary information allow the
'the group prompts are "waterbird on water background", "waterbird on land background", "landbird on water background". and "landbird on land background".

---

YouTube
SMS5
Yelp
DMDs
Human culted
Synthesited Progeame
Uaman statted
Systhesized Program
Syuthesied Progruma
Ihmn suiteed
Synthesiae Promms
GPT4
Cloode
GPT4
Claoke
GPT-355
GPT4
Clande
GPT35
GPE4
Clasis
Num of Prommns
10
10
73
10
8
10
5
10
Concaage
0,99
1.00
1.00
1,00
0441
1.00
1.00
1.00
6:83
0.78
0.99
688
0.88
0.89
1,00
698
Netrrmmnoo
D88
0.60
0.75
0.57
0.23
0.66
Table 4: Analysis showing that Alchemist can achieve comparable or better accuracy and higher coverage while using fewer programs to label the data.
model to specialize better on certain data points than others. We seek to achieve these performance improvements without the need to re-prompt the model with each set of supplementary information. Instead, we collect previously generated programs to obtain set of programs with greater diversity. We ask: can Alchemist achieve better performance by modeling more diverse programs?
We randomly select a set programs from each category, collect them, and train the label model with their program outputs. Additionally, increase the number of sampled programs in each category from to 9. We test this approach on the datasets where Alchemist gives comparable or lower performance than zero-shot prompting in our initial experiments in Table namely the SMS, Yelp, and IMDb datasets.
Expected Results. By obtaining more diverse programs to use, Alchemist can capture wider range of perspectives and labeling logic, potentially leading to more accurate pscudolabels.
Results. Fig. 5 visualizes the effect on the label model's performance when we increase the diversity in collected programs. demonstrates trend and indicates that involving more diverse set of programs can help to mitigate the impact of individual strategy biases or limitations, leading to the production of better labels.
Overall, results in Sec. 4.3 and in Sec. 4.4 underscore that the use of supplementary information and involving diverse types of programs can help achieve better performance.
4.5 Comparing to Human-crafted Programs
Setup. Lastly, we compare synthesized programs in Alchemist and manually crafted labeling functions in WRENCH (50), which is a widely-used benchmark for evaluating weak supervision methods. We focus the datasets that overlap between Alchemist and WRENCH. For each dataset, we use pure prompts to query GPT-3.5, GPT-4, and Claude 3 for 10 programs. We then evaluate the performance of the distilled model for both methods. We also include the label model's coverage in our comparison. Higher coverage means that label model can produce more pseudolabels, yielding larger size of training dataset to use.
Expected Results. We expect that synthesized programs may offer some advantages in terms of efficiency and effectiveness compared to human-designed ones.
Results. Fable(ppreente their comparison. By leveraging the knowledge and capabilities of language models, we find that generated programs offer several advantages, including better coverage (i.e., the ability to label more data points) and comparable, or even better, performance. Generated programs reduce the need for laborious engineering, which can be time-consuming and often requires a tedious design process to fine-tune labeling logic, such as thresholds and keyword usage. This design process may lead to many undiscovered rules, resulting in lower performance coverage and potentially limiting the effectiveness the labeling dunctions-unlee synthesized programs.
This is particularly evident in the SMS dataset, where WRENCH requires 73 manually crafted labeling functions to obtain high-quality labels, while Alchemist only needs 10 generated programs to obtain comparable performance and higher coverage. This significant reduction highlights the potential of Alchemist assist humans designing labelingfunction and make more accessible to users without extensive domain expertise.

---

Conclusion
We propose an alternative approach to costly annotation procedures that require repeated API requests for labels. Our solution introduces simple notion of prompting programs to serve annotators. developed an automated labeling system called Alchemist to embody this idea. Empirically, our results indicate that Alchemist demonstrates comparable or even superior performance compared to language model- based annotation, improving five out of eight datasets with an average enhancement of 12.9%. Notably, Alchemist reduces total costs by a factor of approximately 500. Furthermore, we showease the system's 's extensibility to handle more complex modalities while enhancing the robustness of predicted labels. Finally, confirm that incorporating relevant information can generate better programs, and increasing diversity leads to obtaining higher-quality labels.
6 Acknowledgments
We are grateful for the support of the NSF under CCF2106707 (Program Synthesis for Weak Supervision) and the Wisconsin Alumni Research Foundation (WARF). We thank Dyah Adila, Albert Gu, Harit Vishwakarma, and Nicholas Roberts, for their helpful feedback and valuable discussion.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187 33:1877-1901, 2020.
[2] Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Want reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108. 13487, 2021.
[3] Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXiv:2202.07922, 2022.
[4] Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Progen: Progressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3671-3683, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[5] Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning. In International Conference on Learning Representations, 2023.
[6] Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303. /6854, 2023.
[7] Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. Llmaaa: Making large language models active annotators. arXiv preprint 20. 2023.
[8] Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. Tuning language models training data generators for International Conference on Machine Learning, 2023.
few-shot learning, In
[9] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 230.:22303308774,, 2023.
[10] Authropic. Introducing the next generation of claude, Mar 4, 2024.
[11] Fabrizio Gilardi, Meysam Alizadeh, and Mael Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120530))23000111220, 2023.
10

---

[12] Xiaohuan Pei, Yanxi Li. and Chang Xu. Gpt self- for better data annotator. arXiv preprint (XX1::3306.04349, 2023.
[13] Cheng- Yu Hsieh, Chun-Liang Li, Chih- Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. arXiv preprini arXiv:2305.0230/ 2023.
[14] Tetsuya Sasaki, Firoz Chowdhury, and Sunil Thite. Medical text dataset: Cancer doc classifica- tion, 2023.
[15] Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H Gilpin. Can large language models explain themselves? a study of im-generated self- self-explanations. arXiv preprint arXiv:2310.11207, 2023.
[16] Andreas Madsen, Sarath Chandar, and Siva Reddy. Can large language models explain them selves? arXiv preprint arXiv:2401.0792) 2024.
[17] Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plausi- bility: On the (un) reliability of explanations from large language models. arXiv preprint wrXiv22402.04614 2024.
[18] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly, Advances in neural information processing systems, 29, 2016.
[19] Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment. International Conference Very Large Data Bases, volume 11, page 269. NIH Public Access, 2017.
[20] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4763-4771, 2019.
[21] Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In International Conference on Machine Learning, pages 2290-3291. PMLR, 2020.
[22] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. arXiv preprint 122.:2212..2237,, 2022.
[23] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[24] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.66800 2021.
[25] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint 220.vv2205.145555 2022.
[26] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.0863: 2021.
[27] Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-gov goo Lec. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint wrXiv:2206.0808: 2022.
[28] Daniel Y. Fu, Mayee Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christo- pher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In Pro- ceedings of the 37th International Conference on Machine Learning, ICMI.20.. JMLR.org. 2020.
11

---

[29] Paroma Varma and Christopher Ré. Snuba: Automating weak supervision to label training data. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 12, page 223. NIH Public Access, 2018
[30] Nilaksh Das. Sanya Chaba, Renzhi Wu, Sakshi Gandhi, Duen Horng Chau, and Xu Chu. Goggles: Automatic image labeling with affinity coding. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pages 1717-1732, 2020.
[31] Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. Interactive weak supervision: Learning useful heuristies for data labeling. In International Conference on Learning Representations, 2021.
[32] Nicholas Roberts, Xintong Li, Tzu-Heng Huang, Dyah Adila, Spencer Schoenberg, Cheng- Yu Liu, Lauren Pick, Haotian Ma, Aws Albarghouthi, and Frederic Sala. AutoWS-bench-lle 101: Benchmarking automated weak supervision with 100 labels. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[33] Tzu "zu-Heng Huang, Catherine Cao, Spencer Schoenberg, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Scriptoriumws: A code generation assistant for weak supervision. ICLR Deep Learning for Code Workshop, 2023,
[34] Naiqing Guan, Kaiwen Chen, and Nick Koudas. Can large language models design accurate label functions?, 2023.
[35] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint (xXiv:2107.03374, 2021.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.08 092888, 2023.
[37] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey on in-context learning. arXiv preprint urXiv:2301.0025: 2022.
[38] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. eerrieval-augment generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[39] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-ail -agmmettdd generation for large language models: survey, arXiv preprint arXiv:2312.10991 2023.
[40] Open AI. Hello gpt-4o, Mar 13, 2024.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763 PMLR, 2021.
[42] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprini arXiv: arXiv: 110791111111111111111111 2019.
[43] Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida. Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th international conference on machine learning and applications (ICMLA), pages 138-143. IEFE, 2015.
[44] Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 1 11th ACM symposium on Document engineering, pages 259-262, 2011.
12

---

[45] Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. Denoising multi-source weak supervision for neural text classification. arXiv preprint arXiv:2010.04582, 2020.
[46] P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
[47] Abir ELTAIEF. french book reviews, 2023.
[48] Tim Schopf, Daniel Braun, and Florian Matthes. Evaluating unsupervised text classification: Zero-shot and similarity-basee approaches. In Proceedings of the 2022 6th International Conference on Natural Language Processing and Information Retrieval, NLPIR '22, page 6-15, New York, NY, USA, 2023. Association for Computing Machinery.
[49] OpenAl. Openai pricing table.
[50] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprchensive benchmark for weak supervision. arXiv preprint arXiv:2109.11377 2021.
[51] Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models with foundation models. arXiv preprint arXiv:2309.045: 2023.
[52] A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28( (8(1):20-21 28, 1979.
[53] George B Moody and Roger G Mark. The impact of the mit-bih arrhythmia datahase. IEEE engineering in medicine and biology magazine, 20(3):45-500 2001.
[54] Ron Kohavi. Census Income. http::/bbi.rr//0.233//55GC
UCI Machine Learning Repository, 1996. DOI:
[55] Jason A Fries, Paroma Varma, Vincent S Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau, al. Weakly supervised classification of aortic valve malformations using unlabeled cardiac mri sequences. Nature communications, 10(1):3111 2019.
13

---

The appendix is organized as follows. First, we provide details about datasets, training settings, and computation resources in Appendix [ Next, in Appendix B we list prompts that we use to query language models. Then, we present ablation studies in Appendix C using other models in weak supervision to work with Alchemist. We include two additional modalities in Appendix [ We incorporate other work [5T] for enhancing robustness into Alchemist and present in E Lastly, we discuss limitations and broader impacts our work in Appendix
A Datasets and Implementation Details
Dataset
Task Type
Prodiction Classes
of Clisses
of Train
YorTube
spam detection
|"pam" "hamn" "spam" "hams")
1686
spam detection
Yolp 15
resiaurant review sentiment classification
"postive", "regative")
30400
IMDB 45
mowie reviow sentiment classifieation
postive. "regurive")
20000
"nooplamm"" , "digestive system discases", "bervous system discases",
MedAIx
medical ahstract topic clavsification
cardivvseellrr disrases". " "genrral puthological conditions)
10395
H
biomedical document topic classifization
"olon cancer'. "Bung cancer". "thyroad cancer"))
5450
Finanee
fiaace news semiment classification
"psittvve "noutral". "negative")
Frunch
book review seniment classificaiion
"usitive "noutral" "negative")
6951
Waterhirds
+
bird species classification
landbird" "wasetbird"' )
5794
Table 5: Dataset Table.
We place more details about our datasets and experimental setups here. First, in Table we show task type, prediction classes, and number of training data points in each dataset. MedAbs, Cancer, Finance, and French are considered to be more challenging settings, where these datasets typically need domain expertise provide labels. Waterbirds considered to test for a more complex modality.
We employ Snorkel as our label model to aggregate program outputs and report results in the main paper. We show more results using different choices of label model in Appendix C All the distilled models use the MLP model that is trained with 2 hidden layers, each comprising 32 units, using ReLU activations between layers and normalization. We run 5 times with different random seeds and report their average performance. We use a A6000 NVidia GPU to run all experiments.
B
Used Prompts
Dataset
Zero-shot Prompting (Baseline)
YouTube
what is the category of this youtube comment: [text]
SMS
what is the category of this sms text: [text]
Yelp
what is the sentiment of this restaurant review: [text]
IMDb
what is the sentiment of this movic review: [text]
MedAbs
what is the topic of this abstract: [text]
Cancer
what is the topic of this document: [text]
Finance
what is sentiment of this news: [text]
French
what is the sentiment of this book review; [text]
Table 6: Prompts for baseline approach are presented.
Dataser
Task Description (Alchemist)
YouTube
Write bug-frec and exccutable function in python to label commict Youtube spam ham.
SMS
Write bug-free and executable function in python to label SMS text as spam or ham.
Yelp
Write bug-free and exocutable function in python to label the sentiment of restaurant review on Yelp as postive or negutive.
IMDB
Write hug-free and execurable function in python
label the sentiment movie review IMDB postive negulive
MedAbs
Write hug-free and executable function in python to label the topic medical abstract.
Canzer
Wiite hug-free and execurable function in python to label the topic of biomedical document.
Finncee
bug-free and exccutable function pyllon label the sentiment financial news postive, neutral, negative
French
Write a bug-ftee and exccutable function in python to label the sentiment book review written in French as postive, neutral, or negative.
Table Task descriptions in Alchemist's prompt are presented.
Next, we present the prompts used to query language models in the baselines and Alchemist.
14

---

Youtube
SMS
Yelp
IMDB
Est. Cost
Accuracy
Est. Cost
Fl-score
Est. Cost
Accuracy
Est. Cost
Accuracy
Zero-shot Prompting
0.096
0.871
0.240
0.907
3.873
0.845
3.400
0.737
Weighted Majority Vote
0.004
0.874
0.004
0.886
0.005
0.705
0.004
0.520
Swwid-Skene
0.004
0.864
0.004
0.895
0.005
0,682
0.004
0.507
FlyingSquid
0.004
0.863
0.004
0.915
0.005
0.678
0.004
0.500
Snorkel
0.004
0.891
0.004
0,900
0.005
0.575
0.004
0,662
MedAbs
Cancer
Finance
French
Est. Cost
Accuracy
Est. Cost
Accuracy
Est. Cost
Accuracy
Est. Cost
Accuracy
Zero-shot Prompting
1.944
0.311
15.925
0.716
0.201
0.641
0.641
0.611
Weighted Majority Vote
0.006
0.354
0.003
0,968
0.007
0.650
0.006
0.221
Duwid -kene
0.006
0.262
0.003
0.957
0.007
0.661
0.006
0,221
Sqyid Snorkel
0.006
0,323
0.003
0.967
0.007
0.661
0.006
0.690
0.006
0.346
0.003
0.968
0.007
0.660
0.006
0.690
Table 8: Testing performance of the distilled model is reported for cach combination of label model and dataset.
First, we show the prompts used for the baseline approach of zero-shot prompting on text datasets in T In these prompts, the placeholder "[text]" replaced with individual data points and sent via API calls to obtain labels for each data point.
Next, we present the prompts used in Alchemist in Table 7 The table displays the task description component of each prompt. These descriptions outline the objective of the generated program and are associated with the prediction classes. For the labeling instructions, we directly map the prediction classes to their corresponding class indices and query the language models to output the appropriate class index.
For the image task, we use the prompts ['an image of landbird", "an image of waterbird"] to perform zero-shot prompting using CLIP. In Alchemist, we first query high-level concepts and then combine them with computed scores to prompt LLMs generate programs. The first step involves the following prompt: "What are the visual primitive concepts to classify "landbird" and "waterbird""? Please organize the primitive concepts by name and use comparisons for the classes. Parse the results into JSON format."
Once have obtained of similarity scores, similarity scores for the following descriptions as float numbers. If score is close to 1, it is highly related to the description. If score close to 0, it is less related the description. The descriptions are: ["A bird's foot type is toed, grasping" "]: ["A bird's ' foot type is paddling, swimming"), Generate a labeling function with input scores classify landbirds and waterbirds. If it cannot be determined, the function should return -1, but use this cautiously." Descriptions will be replaced by different generated concepts.
the following prompt: "I have measured
C Ablation Studies
Alchemist is compatible with variety of weak supervision aggregation approaches. We report additional results with different choices of label models. Besides Snorkel, we consider three more widely-used label models: Weighted Majority Vote, Dawid-Skene (52). and FlyingSquid (FS) (28). We reuse our experimental setup from Sec. 4. and in Sec. 4.5 and present the performance of the distilled models in Table and in Table respectively.
In Table we observe that the label accuracy is enhanced or achieves comparable performance with different label models, showcasing Alchemist's flexibility in working with various label models. In
T we include compare them with numan-crafted labeling functions developed in WRENCH [50]. Similarly, Alchemist obtains higher coverage and achieves comparable even better label accuracy while reducing the need to craft a large number of programs manually.
Next, we conduct experiment by varying the temperature in our query APIs and running Alchemist on four different datasets. We train the end model five times with different random seeds and computed the average performance and the variance. Results are shown in Table I0 We observe
15

---

Number of Progrums
Coverage
Weighted Majority Vote
Duwid-Skene
FlyingSquid
Snorkel
Human-crafted
10
0.89
0.88
0.84
0.87
0.85
Youtube
GPT-3.5
10
1.00
0.87
0.86
0.86
0.89
GPT-4
10
1.00
0.85
0.88
0.87
0.89
Claude 3
10
1.00
0.77
0.71
0.73
0.72
Human-crafted
73
0.41
0.90
0.86
0.00
0.89
SMS
GPT-3,5
10
1.00
0.89
0.90
0.90
0.90
0,92
Claude 3
10
1.00
0.91
0.92
0.92
0.89
Human-crafted
8
0.83
0.75
0.83
0.77
0.76
Yelp
GPT-3.5
10
0.78
0.70
0.68
0.68
0.57
GPT-4
10
0.99
0.73
0.81
0.72
0.82
Claude
0.77
Humaan-crafted
5
0.72
0.68
0.73
IMDb
GPT-3.5
10
0.89
0.52
0.50
0.66
GPT-4
10
1.00
0.54
0.55
0.54
0.75
Claude
10
0.60
0.70
Table 9: We offer a comparison between a wider range of label model options for synthesized programs and those designed by humans.
YouTube
SMS
French
Cancer
Label Model
End Model
Label Model
End Model
Label Model
End Model
Label Model
End Model
0.0
0.755
0.020
0.795
0.005
0.589
0.000
0.866
0.014
0.550
0.001
0.690
0.000
0.729
0.000
0.935
0.009
Temperaiure
0,5
0.898
0.002
0.870
0.005
0.600
0.013
0.854
0.019
0.519
0.000
0.690
0.001
0.733
0.000
0.940
0.010
1.0
0.817
0.004
0.803
0.017
0.667
0,022
0,930
0.012
0.555
0.001
0.690
0.000
0.731
0.000
0.938
0.006
Table 10: We varied the temperature in GPT-4 API calls, showing consistent performance across four datasets. We also trained the end model five times, displaying the average performance and the variance. These results confirm the stability of Alchemist.
consistent labeling performance across different temperatures (0.0, 0.5, and 1.0), demonstrating Alchemist's stability. Additionally, the stability of generated programs highlights the significance of including aggregation models to handle noisy and diverse outputs, resolve conflicts, and produce final labels.
D Richer Modalities
We run Alchemist in two additional modalities in Table time-series (ECG heartbeat classifica- tion [53] and tabular (Census income classification (44 For ECG heartbeat classification, we generated 10 labeling programs from GPT-4o. For the Census income dataset, we generated program codes for each attribute (e.g., gender, education, age, race). We used Snorkel as our label model. The results demonstrate Alchemist's capability to handle more complex modalities and produce satisfac- tory performance. We compared Alchemist with human-crafted labeling functions from WRENCH. Alchemist uses fewer labeling functions (programs) and reaches higher labeling performance.
In general, Alchemist will work well with any of these modalities as long as we have access to any cheap local feature extractor. This includes medical imaging tasks: (55) showed manually-crafted simple labeling functions were able to identify heart problems in MRI sequences based on very simple primitives, which could act as the feature extractors for Alchemist.
E
Improving Robustness
integrate advancements from existing robustness techniques into Alchemist to further improve accuracy and reduce spurious correlations. Specifically, we consider RoboShot [51], which prompts I.LMs for spurious and correct correlation features, then calibrates image embeddings by projecting them to reject accept concepts. In setup, GPT-4o to identify spurious correlations for
16

---

ECG Heartbeat Classification
Census Income Classification
Human Crafted
Synthesized by GPT4o
Human Crafted
Synthesized by GPT4o
Modality
Time-Series
Tabular
# of Train Set / # of Test Set # of Classes
87554 / 21892
30162 / 15060
5
2
of Programs Label Model Accuracy
13 0.725
0.827
0.681
Table 11: We included two new modalities in our evaluation: time-series and tabular data. Both performed well using few generated programs. the Census dataset, Alchemist outperformed human-crafted labeling functions in WRENCH.
Feature Extractor
Method
Average Accuracy (1)
Worst Group Accuracy (1)
Gap (4)
CLIP VIT-B/32
Zero-shot Prompting
0.820
0.318
0.502
Group Prompting
0,823
0,383
0,440
Alchemist with GPT44
0,805
0.283
0,522
RoboShot + Alchemist with GPT4o
0.700
0.375
0,325
CL-PVIT-114 VIT-/144
Zero-shot Prompting
0.904
0.335
0.569
Group Prompting
0.791
0.240
0,551
Alchemist with GPT4o
0.802
0.467
0.335
RoboShot Alchemist with GPT4o
0.803
0,569
0.234
Table 12: We integrate Roboshot into Alchemist by querying GPT-4 for spurious correlation features and rejecting them, then reusing the generated programs in Alchemist. This integration improves average accuracy and worst-group performance, enhancing robustness to spurious correlations.
classifying waterbirds and landbirds, then project image embeddings onto these concept embeddings to reject spurious correlations through subtraction. We compute cosine similarity using the calibrated embeddings to obtain score sets, which are then fed into Alchemist's generated programs. The spurious correlations identified by GPT-4o are ["water background", "land background", "aquatic plants", "trees and bushes"]. Results are displayed in Table [2] This integration using GPT-4o successfully enhances robustness to spurious correlations by improving worst-group accuracies.
Discussion
Techniques to Evaluate Generated Programs. There are many ways to evaluate the quality of generated programs in advance. Expert users can quickly determine whether key problem properties are being used by looking at the code. Besides human inspection, Alchemist includes several automated measurement tools diagnose generated programs. First, we analyze program outputs to compute coverage, polarity, conflict, and overlap (see for definitions). For example, coverage (the fraction of data points with at least one label) is below 10%, discard the program and ask users to generate a new one. Moreover, a validation dataset is available, Alchemist can run diagnostics to empirically compare accuracy with ground truth, offering more insight into the program's reliability. This data-driven feedback loop ensures tractable program generation. Notably, these tools are not typically accessible with model- annotation methods.
Linitations. There two primary limitations in Alchemist. First, the performance of the datasets we test is still dependent on the capabilities the language model. the language model's ability to comprehend the given task and generate effective programs is subpar, the labeling performance may suffer. The second limitation arises when dealing with extremely complex tasks. As the complexity of the task increases, the generated code may become longer, more intricate, and harder understand, posing challenges for developers who take time to validate correctness.
labeling LFAnalyyis httpa//aeaeaaaaoaeeeeeeeeeeeeeee
17

---

Broader Impacts. We do not see explicit negative impacts in Alchemist's annotation process. However, generated programs from language models may contain biased labeling logic, toxic content, or malicious functions. To mitigate this, auditing and guardrails may be necessary.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our claims are accurately reflect our contributions in data annotation and its scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
Yes. We discuss the limitations of the work in Appendix F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper is an empirical work. It does not apply to this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. We report our training settings, and used prompts in the Appendix (See Appendix And Appendix
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes.
We release our code and data here:
:https://githut com/
BraccketLab/Alchhmiss
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nisscccpublic//uust http:///cc/giiee///eeuuubiiiiiin
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss training and testing details in our experiment sections and in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
Re- Re-prompting labels for data points may create huge expense.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We place the information about computer resources in the Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://s//aarrp....caaalal/aeeee
Answer: [Yes]
Justification: It follows NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss them and place in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This doesn't apply to this paper.
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes. We properly credit data, paper, and ideas that we used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paaerssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We document well about the asset.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer:
[NA]
Justification:
This doesn't It apply to this work.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper doesn't involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "dwjoUyxbhN", "reviewer": "Reviewer_n36E", "summary": "The paper proposes an automated way to label large quantities of data by leveraging large language models to generate labeling functions which can be used to label data using programmatic weak supervision. The paper demonstrates that this procedure can generate labeling functions which are more accurate than manually generated LFs and querying LLMs for labels directly at a fraction of cost. The authors also show that prompt tuning benefits their method, and that it can be used for other richer modalities.", "questions": "I wonder what the authors thoughts are on points 3 and 4 above in the weaknesses.", "limitations": "The authors have discussed the limitation of their approach.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written, and easy to follow. \n2. The experiments are well designed.\n3. The idea is simple, intuitive and has promising performance.\n4. Evaluation on richer data modalities is challenging, and I appreciate the authors' inclusion of image datasets.", "weaknesses": "Most of these concerns are minor, and some of them can be readily fixed. \n1. **Comparisons with related work:**: I believe that comparisons with empirical ScriptoriumWS and DataSculpt are necessary to demonstrate the benefits of the proposed approach. With that said, I found the experiments to be rigorous. I would also like the authors to catalog the differences between their approach and ScriptoriumWS, DataSculpt, and [1] to better highlight the novelty of their methodology and findings. I am aware that [1] was published very close to the NeurIPS deadline so I do not expect the authors to compare their methods with [1], but conceptual differences (if any) can still be highlighted. \n2. **Missing references:**: The idea of using keywords and LLMs to automatically create LFs is not new, so I would encourage the authors to reference some prior work (e.g. [2]).\n3. **Lack of reproduciblity and claims:** The techniques presented in the paper and the results are not reproducibile with the details given in the paper. I would encourage the authors to release their code for the same. The paper mentions that \"any particular program may be inaccurate, fail to compile, or may otherwise be flawed, ...\". I am curious how the authors ensure that the programs are not downright inaccurate (because snorkel would require the accuracy of LFs to be greater than random chance), or fail to compile? I believe that some kind of a data-driven feedback loop is important to ensure that the programs compile, and the generated LFs have coverage, and are accurate. Therefore, I am excited about the possibility to include keywords, dataset descriptions, example in the prompt, but I feel that there has to be a feedback loop after the label model labels the data or identifies inaccurate LFs.\n4. **Richer modalities:** The claim on scaling to richer modalities should be softened, because the authors only evaluate their approach on a relatively simple image dataset. More challenging tasks may include labeling chest X-rays [3], datasets for which are publicly available, and other richer modalities such as time series data [4]. I do not expect the authors to conduct these experiments during the rebuttal, but such an extension would be very interesting and valuable, in my opinion. \n\n### References\n1. Smith, Ryan, et al. \"Language models in the loop: Incorporating prompting into weak supervision.\" ACM/JMS Journal of Data Science 1.2 (2024): 1-30.\n2. Gao, Chufan, et al. \"Classifying unstructured clinical notes via automatic weak supervision.\" Machine Learning for Healthcare Conference. PMLR, 2022.\n3. Irvin, Jeremy, et al. \"Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.\n4. Goswami, Mononito, Benedikt Boecking, and Artur Dubrawski. \"Weak supervision for affordable modeling of electrocardiogram data.\" AMIA Annual Symposium Proceedings. Vol. 2021. American Medical Informatics Association, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer n36E\n\nThank you for recognizing our paper as well-written with well-designed experiments. We appreciate your acknowledgment of our simple, effective idea and the significance of including diverse modalities. We appreciate your thoughtful review!\n\n* **On Comparisons with Related Work.**\n    * We compared Alchemist with related works in Table 1 in our attachment. **Alchemist uses fewer programs while achieving comparable labeling accuracy.** **Key differences** include:\n        * **Motivation:** Both ScriptoriumWS and DataSculpt address weaknesses in weak supervision, whereas Alchemist focuses on the downsides of model-based annotation.\n        * **Approach:** In ScriptoriumWS, supplementary information is manually crafted into prompts to generate code, while Alchemist uses LLMs to distill self-knowledge, enhancing labeling performance but using less human effort. DataSculpt employs *hundreds of programs for obtaining high-quality labels, but Alchemist achieves comparable accuracy with just 10 programs*.\n        * **Complex Modalities:** Alchemist handles complex modalities beyond text, while ScriptoriumWS and DataSculpt *do not*.\n    * PromptWS significantly differs from Alchemist. PromptWS prompts LLMs *multiple times for each sample*, **resulting in more API calls (even compared to zero-shot prompting).** While this method can improve performance, it incurs extremely high labeling costs, is harder to audit, and lacks extensibility. Alchemist, in contrast, addresses these issues.\n* **On Suggested References.**\n    * Thank you for sharing this paper. We have updated our related work and included it. While this work shares a similar idea with Alchemist, Alchemist goes beyond keyword extraction by capturing more complex and diverse labeling logic, and organizing them into executable programs (see Figure 1 in the paper). Moreover, extending to non-text modalities like image classification is challenging, but Alchemist is capable of this. \n* **On Reproducibility.**\n    * We attached our code in the supplementary material of our submission. We will release our code and a full set of outputs, including generated programs, after paper notification.\n* **On the Data-Driven Feedback Loop.**\n    * This is a good question, and it is a crucial step. Alchemist includes several tools for diagnosing generated programs. First, we analyze program outputs to compute coverage, polarity, conflict, and overlap (see [1] for definitions). For example, if coverage (the fraction of data points with at least one label) is below 10%, we discard the program and ask users to generate a new one. Second, if a validation dataset is provided, Alchemist computes empirical accuracy by comparing outputs with ground truth. Such a data-driven feedback loop ensures tractable program generation. We have included this step's description in the updated draft.\n* **On Richer Modalities.**\n    * Thank you for the suggestion! We included two additional modalities in Table 2: **time-series** (ECG heartbeat classification [2]) and **tabular** (Census income classification [3]). For ECG heartbeat classification, we generated 10 labeling programs from GPT-4o. For the Census income dataset, we generated program codes for each attribute (e.g., gender, education, age, race). We used Snorkel as our label model. The results demonstrate Alchemist's capability to handle more complex modalities and produce satisfactory performance. Moreover, we compared Alchemist with human-crafted labeling functions from WRENCH. Alchemist **uses fewer labeling functions (programs) and reaches higher labeling performance.** In general, Alchemist will work well with any of these modalities as long as we have access to any cheap local feature extractor. This includes medical imaging tasks: [4] showed manually-crafted simple labeling functions were able to identify heart problems in MRI sequences based on very simple primitives, which could act as the feature extractors for Alchemist.  \n\n[1] https://snorkel.readthedocs.io/en/v0.9.3/packages/_autosummary/labeling/snorkel.labeling.LFAnalysis.html \\\n[2] https://physionet.org/content/mitdb/1.0.0/ \\\n[3] https://archive.ics.uci.edu/dataset/20/census+income \\\n[4] Fries, J.A., Varma, P., Chen, V.S. et al. Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences. Nat Commun 10, 3111 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kcEu3tKp6S", "reviewer": "Reviewer_uwkF", "summary": "The paper presents a new method for creating data labels that leverage a Large Language Model and the weak supervision/data programming labeling paradigm. In this work, the LLM is used to generate labeling code, typically in the form of functions, which can then be used to create weak labels for weak supervision. The paper goes on to show this method’s utility across several benchmark text datasets and one image dataset. The paper also does an ablation study showing the impact of different elements of the method, like numbers of weak functions, and including examples in the prompt to create the functions.", "questions": "I have no additional questions.", "limitations": "One other societal limitation the work could address is how works like this are changing the data labeling industry. Data labeling is still a massive, multi-million-dollar industry and these works are changing the nature of that industry. Ideally, they are changing it positively, but it can still disrupt how the data labeling industry works and cost human labelers their income.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is strong in its significance, clarity, and quality. For significance, the paper is attacking a significant problem. One of the great promises of large ML models, with multimodal or text only, is their ability to zero-shot label text. This ability allows LLMs to possibly overcome one of – if not the chief issue – with building machine learning models: labeled data. Thus, this paper is attacking a profoundly important problem for the application of ML to real-world problems. For clarity,  the paper is well-written and the diagrams are very helpful. When combined with the appendices and supplementary material, I can not only easily see how to reproduce their results, but how to apply this method to my data labeling problems. In other words, I can easily see wide adoption and use of this method. Finally, the paper does a reasonably good job in its empirical testing to cover many variations on the application of the method (e.g., having both text and image datasets) and variations to the method (e.g., including examples, numbers of weak functions, etc.).", "weaknesses": "The weakness of the paper is in its grounding/novelty and some of its methods, particularly when applied to other modalities. For the grounding, while the paper actually does capture many of the previous works that have done something similar, there are works like Cruickshank and Ng, “DIVERSE: Deciphering Internet Views on the U.S. Military Through Video Comment Stance Analysis, A Novel Benchmark Dataset for Stance Classification” where they used LLMs + weak supervision to create an actual new dataset. They did not, however, have the novel insight about creating the weak labeling functions that this work does.\n\nFor the method in the image or multimodal case, two works have done something very similar with CLIP. First, Adila et al. “Zero-shot Robustification of Zero-shot Models” use the same CLIP model and water birds dataset, but get substantial performance increases by “debiasing” the image embeddings, with text characteristics. This is very similar to what is done with the labeling functions where this paper tries to get labeling functions to classify aspects of the birds and ignore spurious contexts. Second, Bendou et al.’s “LLM meets Vision-Language Models for Zero-Shot One-Class Classification” presents a method for one-shot classification using VLMs, which uses an LLM to build negative classes around a positive class. This is similar to the insights in this paper around developing labeling functions to highlight important visual distinctions in the classes to improve the labels. Taken together, I think this paper might be able to incorporate the insights from these other papers to actually improve their results. For example, the debiasing in the first paper could be used with the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer uwkF\nWe are grateful for your review and for describing our paper as significant, well-written, and of high quality. We address your questions below and provide additional experimental results incorporating Roboshot into Alchemist!\n\n* **On Grounding.**\n    * Thank you for sharing this paper with us. We agree it relates to Alchemist, and we have included it in our updated related work. **Key differences** include: unlike Alchemist, which automatically generates programs, this work requires domain expertise and more human effort to craft heuristics and search for proper keywords.\n    * In Sec. 4.4 in our paper, we include an experiment comparing Alchemist to human-crafted labeling functions. For example, in the SMS dataset, WRENCH requires 73 manually crafted labeling functions to obtain high-quality labels, whereas **Alchemist only needs 10 generated programs to achieve comparable performance and higher coverage**. This significant reduction highlights Alchemist's capability to assist humans in generating label sources, making it more accessible to users without extensive domain expertise.\n* **On Two Similar Works.**\n    * Thank you for pointing out these two papers. The first paper prompts LLMs for spurious and correct correlation features, then calibrates image embeddings by projecting them to reject or accept concepts. The second paper asks LLMs for confusing visual objects to perform zero-shot one-class classification with CLIP. We have included them in the draft.\n* **On Incorporating Roboshot into Alchemist.**\n    * This is an interesting idea. We implemented this suggestion by incorporating Roboshot into Alchemist. We first used GPT-4o to identify spurious correlations for classifying waterbirds and landbirds, then projected image embeddings onto these concept embeddings to reject spurious correlations through subtraction. We computed cosine similarity using the calibrated embeddings to obtain score sets, which were then fed into Alchemist’s generated programs. The spurious correlations identified by GPT-4o were *{'water background,' 'land background', 'aquatic plants,' 'trees and bushes'}*. Results are displayed in Table 3 in the attachment. We observe that integrating Roboshot with Alchemist using GPT-4o **enhanced robustness to spurious correlations by improving accuracies.**\n* **On Framework Limitations.**\n    * Our motivation is to address the **drawbacks of pretrained model-based annotation, which is expensive, lacks extensibility, and makes results hard to inspect**. We believe Alchemist serves as a better labeling tool by reducing costs and providing human labelers with more power to inspect and refine labeling results. In addition, generated programs can serve as templates for human labelers to rewrite, extend, or customize labeling logic. These benefits make the labeling process easier and more efficient for humans.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPoudD41tl", "reviewer": "Reviewer_d5ZD", "summary": "The authors propose an innovative solution for high cost of APIs that using large pretrained models to generate programs that act as annotators. This idea helps replace or supplement crowdworkers and allows for distilling large models into smaller, more specialized ones. Traditional methods can be costly and produce static, hard-to-audit datasets due to expensive API calls. To tackle this, they introduce Alchemist, a system that tasks models to generate reusable labeling programs. These programs can be applied locally, significantly cutting costs while maintaining or even improving performance.", "questions": "Refer to weaknesses.", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Generating programs that can produce labels is an interesting idea. This paper introduces a mechanism that can generate multiple labels, transforming the one-to-one mapping between samples and labels into a sustainable one-to-many or many-to-many relationship. This approach can significantly reduce API costs.\n\n2. From a practical implementation standpoint, Alchemist's flexibility and reusability are major strengths. The system allows users to create simple prompts for generating labeling programs that can be stored, reused, and extended. This adaptability makes it a versatile tool suitable for a wide range of tasks and ensures that users can tailor it to their specific needs.\n\n2. The code is straightforward and easy to read, making it accessible even for those who might not be deeply familiar with the underlying concepts.", "weaknesses": "1. One of the limitations of Alchemist is that the generated programs tend to handle only fixed tasks, often relying on threshold-based judgments. This means that for more complex or varied tasks, the system might still need to call specialist models via API, which somewhat limits its flexibility.\n\n2. Another weakness is the lack of experimental evidence regarding the stability of the generated programs. For instance, the paper doesn't thoroughly explore how factors like the temperature variable in the code could affect the quality and consistency of the generated labeling programs.\n\n3. The authors could also improve their literature review. There are several highly relevant papers that discuss various aspects of data annotation that weren't cited, such as [1] [2] [3] [4] [5]. \n\n[1] https://arxiv.org/abs/2310.04668\n[2] https://arxiv.org/abs/2303.15056\n[3] https://arxiv.org/abs/2306.04349\n[4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834\n[5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer d5ZD\n\nThank you for recognizing Alchemist’s flexibility, extensibility, and for your kind words about our paper! We address your questions below and include experimental results on more complex tasks and program stability.\n\n* **On More Complex Tasks.**\n    * Alchemist is not limited to threshold-based simple tasks. It is capable of capturing more complex labeling logic, as we have observed in some of the generated programs. To further demonstrate this capability, we included two more challenging datasets: **ECG heartbeat classification** and **Census income classification.** Results are shown in Table 2 in our attachment and demonstrate that Alchemist can handle more complex modalities and perform well. Moreover, we compared Alchemist with human-crafted labeling functions from WRENCH. **Alchemist uses fewer labeling functions (programs) and reaches higher labeling performance.**\n    * Additionally, since submission, we have been working on an additional, highly timely, application of this work to LLM-as-a-judge. This is a crucial area since users and companies spend vast sums of money for evaluating model outputs using automated LLM-as-a-judge techniques. We have collaborated with a leading startup in the industry to use our technique in production environments to reduce costs. Our preliminary results in this area suggest that Alchemist can capture sufficiently complex logic to identify poor-quality responses from language model pipelines. \n* **On the Stability of Generated Programs.**\n    * This is a great question. We conducted an experiment by varying the temperature in our query APIs and running Alchemist on four different datasets. We trained the end model five times with different random seeds and computed the average performance and the variance. Results are shown in Table 4 in our attachment. We observe **consistent labeling performance** across different temperatures (0.0, 0.5, and 1.0), demonstrating Alchemist’s stability. Additionally, the stability of generated programs highlights **the significance of including aggregation models** to handle noisy and diverse outputs, resolve conflicts, and produce final labels.\n* **On Improving Literature Review.**\n    * Thank you for sharing these papers with us. We have included them in our related work section. **Key differences**: These papers prompt LLMs for labels **per-sample**, requiring a large number of API calls (scaling with the size of the dataset) and making it hard to inspect mistakes and revise labeling logic, unlike Alchemist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RE7vNil7dx", "reviewer": "Reviewer_b4or", "summary": "The paper proposes a labeling workflow (Alchemist) using LLMs where instead of labeling each data point using a teacher model, we ask the teacher model to generate a program to label for the given task. Multiple such programs are generated and using an aggregation function, we get pseudo labels. These pseudo labeled points are then used as training data to train a student model. They show that such labeling mechanism is effective and less expensive compared to the teacher model labeling every data point. They also perform ablation studies for multi-modality, supplementary information during program generation, and diversity of programs.", "questions": "1. Why are the accuracy numbers for Alchemist with GPT-3.5 different in table 1 and table 3?\n2. How does the performance compare if we use the generated programs for labeling the test set instead of training the student model?\n3. Given that the performance gain/no loss is not consistent across datasets, what is the recommendation for a new dataset, i.e. how can one know whether this technique is going to work for that dataset? There are mentions that it works better for complex dataset, any other detailed characterization possible?", "limitations": "The main limitation is that it is hard to understand if Alchemist is going to work for your task without spending the money to label and compare. Performance loss when observed is not small to ignore. Generated programs may be reviewed by human experts, but they may not be straight-forward to interpret and gain confidence from.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Knowledge distillation using a larger LLM to label datasets is very prevalent now. It is an expensive process.\n2. Alchemist is an interesting way to reduce costs.\n3. The idea to generate weak labeling functions/programs is novel and as the weak supervision literature shows, it can be effective in certain cases.\n4. The paper is clearly written, and the experiments section is organized to answer critical questions.", "weaknesses": "As the results show, no performance degradation is not guaranteed (for e.g. table 1, Yelp dataset). For a new dataset, without comparing the approaches, it may be hard to understand if Alchemist works for it without impacting the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer b4or\n\nWe are grateful for the review and the positive assessment. Thank you for acknowledging Alchemist is novel, interesting, and our paper is well-written. We address your questions below.\n    \n* **On Performance Degradation.**\n    * In the majority of cases (six out of eight datasets), Alchemist **performs as well as or even better than model-based annotation, in addition to being orders of magnitude cheaper, more extensible, and easier to inspect**. In fact, our results suggest that users should prefer Alchemist over model-based annotation. \n    * In cases where a user seeks to improve Alchemist's base performance, we can simply add more generated programs---**at extremely low cost.** In Table 1 in our paper, we report results with 10 generated programs. Increasing the number of generated programs enhances performance, as we showed in Figure 5, where generating 20 labeling programs for Yelp and IMDb datasets results in **better labeling performance compared to zero-shot prompting.**\n* **On Accuracy Performance in Table 1 and Table 3.**\n    * Results shown in Table 1 are for a model trained on Alchemist-created labels (we report performance on the testing dataset), while in Table 3, we present the label model's performance (i.e., just using the ensemble of programs that Alchemist generates) on the testing dataset. These two approaches correspond to *two stages in the weak supervision process*: first, create a label model that just uses the labeling functions (programs, in our case), and second, obtain labels from the label model and train an *end model*. Using just the label model and training an end model have both been extensively studied in the weak supervision literature. It is known that training an end model on weakly-labeled data enables generalizing beyond the label model in many cases. The reason for this is based on the observation that the label model may not be capable of capturing all labeling patterns (i.e., all possible features). It may instead only use certain simple ones. Training an end model permits learning many patterns, both simple and complex, which enables better generalization in many scenarios. **Our results are consistent with these notions.** \n* **On Labeling the Test Set.**\n    * We described the difference between the label model and the end model above. We organized Alchemist's results in Table 5 in our attachment. We show the testing performance using the label model and the trained end model for eight datasets. **In most of the cases, the end model is preferable to the label model.**\n* **On Techniques to Evaluate Generated Programs.**\n    * There are many ways to evaluate the quality of generated programs in advance. First, expert users can quickly determine whether key problem properties are being used by looking at the code. Besides human inspection, Alchemist includes several automated measurement tools to diagnose generated programs. We analyze program outputs and compute their coverage, polarity, conflict, and overlap (see [1] for definitions). For instance, coverage measures the fraction of data points with at least one label. If coverage is below a certain bar (e.g., 10%), we discard the program and ask users to generate a new one.\n    * Moreover, if a validation dataset is available, Alchemist can run diagnostics to empirically compare accuracy with ground truth, offering more insight into the program's reliability. **Notably, these tools are not typically accessible with model-based annotation methods.**\n* **On Data Characteristics.**\n    * In general, we have found that Alchemist works well when the labeling logic involves a mixture of logical formulas over a set of salient primitives. This captures most cases of interest. Formalizing a definition of what data or task characteristics are relevant is a very interesting problem for future work; we foresee studying, for example, the expressivity of programs and understanding if they can meet most ML tasks. Kolmogorov complexity or similar notions will likely be helpful here.\n* **On Framework Limitation.**\n    * This limitation is common across many annotation approaches. We often only know the results after completing the entire labeling process, which is also true for model-based annotation methods. However, Alchemist significantly reduces expenses, requiring a lower order of magnitude and enabling more efficient reruns of the labeling process. In other words, **Alchemist offers a much more tractable approach to iteration compared to model-based annotation**. \n\n[1] https://snorkel.readthedocs.io/en/v0.9.3/packages/_autosummary/labeling/snorkel.labeling.LFAnalysis.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### General Response
We are grateful for all the comments and constructive feedback on our work. Reviewers consistently found our paper to be well-written and easy to follow and described our work as novel and offering promising performance.

Reviewer **n36E** complimented the inclusion of complex modalities in the evaluation. Reviewer **uwkF** highlighted Alchemist's ease of implementation, foreseeing wide adoption. Reviewer **d5ZD** acknowledged Alchemist's flexibility and reusability, making it a versatile tool for diverse tasks. Reviewer **b4or** agreed Alchemist enables cost reduction. We have adopted suggested clarifications, improved our literature review, and conducted new experiments, leading to a much stronger draft.

### New Results Included:
* **Comparison to Other Works [Reviewer n36E]:** We compared Alchemist with other works, including ScriptoriumWS and DataSculpt, as shown in Table 1. Results indicate that **Alchemist uses fewer generated programs while achieving comparable performance.** Additionally, Alchemist addresses data modalities beyond text, unlike these works. We cataloged more detailed differences in the threads below.
* **New Modalities [Reviewer n36E, d5ZD]:** Table 2 includes datasets for ECG heartbeat classification and Census income classification, representing time-series and tabular modalities, respectively. Both illustrate Alchemist's ability to address these settings.
* **Incorporating RoboShot [Reviewer uwkF]:** We integrated RoboShot into Alchemist, using GPT-4o to identify and reject spurious correlations. As shown in Table 3, this integration improved average accuracy and worst-group performance, **enhancing robustness to spurious correlations.**
* **Stability of Generated Programs [Reviewer d5ZD]:** We varied the temperature in GPT-4 API calls, **showing consistent performance** across four datasets, confirming the stability of generated programs (see Table 4).
* **Labeling on Test Set [Reviewer b4or]:** Table 5 presents performance results using the resulting weak supervision label models and trained models (on the annotations from the label models) on eight testing datasets. We generated 10 programs from GPT-3.5-Turbo for each. Results demonstrate that in the majority of scenarios, **using the trained model (with Alchemist annotations) is preferable**.

We have addressed reviewers' questions and placed our comments in their respective threads below. Thank you again for your questions and thoughtful reviews!

Meta-review: We thank the authors for their submission and constructive engagement throughout the rebuttal phase of the process. The reviewers and AC were impressed by the simplicity, practicality, and versatility of the contributions, which we believe will be of significant interest to the broader applied ML community. The reviewers also highlight the rigour of the evaluations and clarity of exposition throughout. The authors do a good job addressing key outstanding questions by the reviewers throughout the rebuttal period. We encourage the authors to incorporate these additions in their next revision.
Decision: Accept (spotlight)