The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators

Authors: tzu-heng  huang
catherine  cao
vaishnavi  bhargava
frederic  sala
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:35:55

Abstract: Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models. Unfortunately, this comes at a cost: employing top-of-the-line models often requires paying thousands of dollars for API calls, while the resulting datasets are static and challenging to audit. To address these challenges, we propose a simple alternative: rather than directly querying labels from pretrained models, we task models to generate programs that can produce labels. These programs can be stored and applied locally, re-used and extended, and cost orders of magnitude less. Our system, $\textbf{Alchemist}$, obtains comparable to or better performance than large language model-based annotation in a range of tasks for a fraction of the cost: on average, improvements amount to a $\textbf{12.9}$% enhancement while the total labeling costs across all datasets are reduced by a factor of approximately $\textbf{500}\times$.

Main paper: 
 introduction: 
 One of the most exciting developments in machine learning is the use of large pretrained models to act as annotators or labelers [1, 2, 3, 4, 5, 6, 7, 8] . This includes the use of large language models (LLMs) like GPT-4 [9] and Claude 3 [10] . This process offers multiple benefits. First, pretrained models are an efficient way to annotate and have the potential to partially or fully replace expensive human crowdworkers [2, 6, 11, 12] . Second, this approach allows for distilling large models into smaller, task-specific models that can be deployed locally at lower cost [3, 13, 7, 8] . This is additionally important in settings like healthcare and finance where privacy laws require the use of local models.Despite this promise, pretrained model-based annotation has several drawbacks that stymie its adoption. These drawbacks include • High Cost: Labeling a dataset can be expensive. This is particularly so in cases where each data point consists of many tokens. For example, we find that labeling a moderately-sized dataset [14] with 7,569 data points using GPT-4 costs over $1,200. • Lack of Extensibility: Making even small changes to specifications necessitates re-running the entire pipeline to obtain new labels. This inflexibility means the resulting labels are static. • Inability to Audit: API access to pretrained models does not permit inspecting most aspects of the model. Users must simply accept the provided labels with only minimal additional information.Techniques that ask the model for explanations for its decisions may not be reliable [15, 16, 17] .We address these obstacles through a simple but surprisingly powerful notion. Rather than having pretrained models label data, we task language models to generate programs that can output labels. These synthesized programs serve as annotators, capturing the underlying logic used by the models when annotating. In other words, instead of distilling a powerful model to label a dataset (and subsequently training a smaller model on the labeled data), we distill directly into code (Figure 1 ). These resulting programs can either make predictions directly or can label training dataset then train a downstream model using it 1 .This simple notion resolves all of the challenges related to pretrained model-based annotation. First, API calls scale with the number of programs instead of the number of data points. That is, since we generate programs that can themselves make any number of predictions locally at no cost, we can reduce the number of API calls by orders of magnitude. For example, for the dataset described above [14] , the number of GPT-4 calls was reduced from 7,569 (the size of the dataset) to 10 (the number of generated programs), resulting in a massive cost reduction from $1,200 to $0.70, a 1,700fold decrease. Moreover, code can be easily inspected, corrected, and extended, allowing seamless adaptation when prediction classes or labeling rules change.While a powerful idea, distilling model into code presents several challenges. First, any particular program may be inaccurate, fail to compile, or may otherwise be flawed, resulting in noisy program outputs. We address this obstacle by applying weak supervision, a framework for dataset construction from multiple noisy sources of signal [18, 19, 20, 21] . Next, operating on non-text modalities is challenging. We handle this via a simple two-step approach that first extracts high-level concepts and then uses them in concert with a local feature extractor to enable tractable program generation.Contributions. We propose an alternative approach to replace expensive annotation processes that require repetitive prompting for labels. We developed a system called Alchemist that implements this idea. Empirically, Alchemist improves performance five out of eight datasets, with an average enhancement of 12.9%-while reducing total costs by a factor of approximately 500×. Finally, we introduce and validate extensions that address non-text modalities. 
related work: 
 Our work relates to LLM-based annotation, prompting, and the weak supervision framework.Using Large Pretrained Models for Data Annotation. Large pretrained models have demonstrated powerful capabilities using zero-shot prompting across a wide range of tasks [1] . One promising development is their potential to serve as data labelers, which can reduce the cost and human effort in data labeling [1, 2, 11, 12] . Existing research in this area mainly focuses on approaches that allow for more efficient inference, enhanced label generation, and distilling into smaller but specialized labelers [3, 4, 5, 6, 7, 8] . However, scalability is the main limitation in these approaches, as making inferences via querying an API for data examples can be cost-prohibitive. To tackle this challenge, rather than prompting for labels repetitively, we propose prompting pretrained models for programs that use synthesized labeling logic and can thus serve as alternative data labelers.Prompt Engineering & In-Context Learning. In-context learning adapts pretrained models to new tasks without additional fine-tuning [1] . It involves providing relevant examples as demonstrations to solve the task, such as pairs of languages for translation [22] . By including task-specific examples, models can better understand the task at hand. Adding a few data points as demonstrations [23] is commonly suggested when models act as data annotators. Moreover, they can be selected [24, 25] , retrieved [26] , or more efficiently, generated [27] . We explore various types of supplementary information that can be added to Alchemist to help improve program generation and permit more control over the labeling logic used in the programs.Weak Supervision Framework. Weak supervision enables the rapid creation of large training datasets by aggregating cheap-but-noisy signals derived from various labeling sources [18, 19, 21, 28] . These sources can be crafted by domain expertise, using labeling heuristics, or even trained on smaller, weaker classifiers [29, 30, 31, 32] . Recent advancements in code generation open up the potential to automate the heuristic design process. Frameworks such as ScriptoriumWS [33] , and DataSculpt [34] have been developed to take advantage of code-generating models [35, 9, 36] to craft weak supervision sources through prompting. While similar in spirit to our approach, these have several drawbacks: ScriptoriumWS requires more human effort in prompt engineering to better guide code-generation models. Both ScriptoriumWS and DataSculpt can perform poorly in tasks requiring specific domain expertise and, most importantly, they do not handle modalities beyond text-unlike Alchemist. 
alchemist system: 
 We begin by presenting a general annotation workflow in Alchemist, followed by a detailed discussion of each key step.General Workflow. The process is depicted in Fig. 2 . First, users select an unlabeled dataset and create simple prompts to instruct language models to generate programs that incorporate labeling logic. These prompts can integrate relevant information and may vary in their design, allowing for the synthesis of multiple programs. Next, given a set of generated programs and their outputs, we apply weak supervision techniques to obtain a set of aggregated labels. Finally, the labeled points can be used to train a distilled model that can be stored and used locally.We propose a general and extensible prompt template for querying language models to generate annotator programs. This general template consists of three key components:• Task Description: Provides the model an overview of generated program's desired objectives.• Labeling Instructions: Specifies classes and the expected structure of the program's output.• Function Signature: Describes the function's name and the input types to be used. Using Supplementary Information. Drawing inspiration from few-shot prompting [37, 1] , where users provide demonstrations (i.e., data points with their labels) to enhance generated responses, we explore various types of supplementary information that can be integrated to assist models in synthesizing programs. This approach is particularly useful for scenarios where language models may lack the expertise to generate effective programs, or where specific adaptations in labeling logic are required. Such information can be crafted by users themselves, domain experts or, more efficiently, generated by language models themselves. Additionally, it can be combined with retrieval-augmented generation (RAG) systems [38, 39] to access external knowledge.We explore various types of supplementary information to assist in code generation, starting with high-level concepts and then progressively looking into more practical details to control programs.Dataset and Prediction Class Description. First, supplementary information can include relevant background details about the purpose for which the dataset was built and high-level information about the dataset, such as definitions for each label class. By providing this context, the language model can better understand the task at hand.Data Exemplars. Furthermore, we recommend including a small number of labeled data examples in the prompt. This can help language models better comprehend the specific problem. Examples act as concrete illustrations of the task, offering a clearer understanding of the expected output. This can be particularly beneficial when dealing with a complex problem.Keywords. Next, labeling logic in programs can make use of keyword-searching techniques (e.g., Fig 1) . For instance, in situations such as spam detection or topic classification, certain words or phrases may have a strong correlation with specific classifications. Providing several keywords in the prompt may lead models to create labeling programs that explicitly search for the presence or absence of these keywords. This allows for more targeted and precise labeling.Specialized Labeling Rules. Finally, more prior knowledge such as heuristics, specialized labeling rules, guidance, and domain-specific knowledge can be integrated into the prompt. This information can provide concrete labeling steps on how to label specific classes and offer greater control over the logic implemented in the generated programs. Overall, supplementary context is provided before the task description to enhance language models' understanding of the task. This, in turn, enables models to generate programs that are more effective and tailored to the specific requirements of user needs.While generated programs can efficiently annotate data, these programs may produce outputs that are noisy or inaccurate. However, as such programs may employ different techniques, such as patternmatching, heuristic rules, or other approaches-each with its own strengths and limitations-there may be complementary signal in their outputs. This means we can aggregate them to mitigate the impact of noise. To do so, we apply weak supervision techniques [18, 19, 20, 21] . This process starts by learning a model of the reliabilities of the programs. Once learned, this model enables aggregating label outputs from different programs into high-quality pseudolabels.Alchemist is compatible with a variety of weak supervision aggregation models, called label models, providing flexibility in the choice of the weak supervision approach. For simplicity, in this work, we focus on using the Snorkel framework [19] , which is a standard and widely-used approach in the weak supervision community.Crafting programs that operate over text is relatively easy for large language models. More complex data modalities, however, can be far more challenging. Consider images as an illustrative example.Even employing state-of-the-art multimodal models, e.g., GPT-4o [40] and GPT-4V [9] , to seek programs operating over sample images may not produce satisfactory results.To address this challenge, we extend Alchemist's pipeline to include an intermediate step. Specifically, we convert the raw data (i.e., in our example, image pixels) into a set of features representing high-level concepts. These concepts are obtained by prompting a language model (or, potentially, a multimodal model) to identify task-relevant notions. For example, for a bird categorization task, models may identify "wing shape," "beak shape," or "foot type" as informative concepts for distinguishing between bird species. Next, we use any open-source local multimodal model, like CLIP [41] , as a feature extractor for the identified concepts, producing low-dimensional feature vectors that can be easily ingested by generated programs. As such models are free, this does not increase our cost.Fig. 3 and Fig. 4 present examples of generated high-level concepts and the corresponding programs used for the Waterbirds dataset, where the task is to distinguish between landbird and waterbird specices [42] . This simple approach can be applied to any data modality where we have access to a local multimodal model (i.e., a model operating on the modality of interest and text). 1 : Testing performance of the distilled model is reported for each combination of method and dataset. The estimated cost is obtained by calculating the number of input and output tokens associated with GPT-3.5's pricing table [49] . Other models may be even more expensive.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper is an empirical work. It does not apply to this paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We study the capability of Alchemist empirically. Our goals are to validate the following claims:• Cost Reduction and Improved Performance (Sec. 4.1): Alchemist can reduce cost by orders of magnitude, while producing labels of similar or better accuracy.• Extendibility to Other Modalities (Sec. 4.2): Alchemist can operate with modalities beyond text.• Use of Supplementary Information (Sec. 4.3): Incorporating relevant information into prompts enables the generation of better programs, yielding more accurate pseudolabels.• More Diverse Programs Can Help (Sec. 4.4): Increasing the diversity of generated programs created by different labeling logic enables better pseudo labels.• Comparing to Human-crafted Programs (Sec 4.5): Synthesized programs may be more effective in comparison to human-crafted ones.Datasets. We include diverse datasets covering text and image modalities. For text, we include eight datasets that span three different types of language tasks. These include the YouTube [43] , SMS [44] datasets for spam classification, IMDb [45] , Yelp [45] , Finance [46] , and French [47] datasets for sentiment analysis, and the MedAbs [48] and Cancer [14] datasets for topic classification. We note that the Finance, French, MedAbs, and Cancer datasets are relatively challenging, with points that require a degree of domain expertise for accurate labeling. For example, the French dataset requires a good understanding of the language. These may pose challenges for pretrained models.For our extensions to richer modalities, we focus on image tasks. Our evaluation uses the Waterbirds dataset [42] . This dataset is designed to assess models' robustness to spurious correlations and ability to handle distribution shifts. More details are in Appendix A.Setup. We open our evaluation of Alchemist with text domain datasets and use GPT-3.5 to generate programs. For each dataset, we input pure prompts without supplementary information into GPT-3.5 and generate 10 programs to use. We construct training datasets by aggregating the programs' outputs into pseudolabels with the weak supervision framework Snorkel [19] . We then train a two-layer MLP as a distilled model. We run five times with different random seeds and report their average performance. As our main baseline, we directly use language models to produce annotations per point. The resulting labels are used to train a distilled model for comparison. The prompt template used in our baseline approach and our training settings are provided in Appendix A.Expected Results. We anticipate that Alchemist can generate programs that can produce accurate labels while substantially reducing the expense of API calls.Results. Table 1 presents the distilled model's performance on each testing dataset. We observe that label accuracy is improved on five out of eight datasets, particularly in challenging settings such as the MedAbs, Cancer, and French datasets, outperforming the baseline zero-shot prompting approach. We also report the estimated costs of building training datasets. The costs for zero-shot prompting depend on the number of tokens for the dataset. In contrast, Alchemist only prompts 10 programs for each task, resulting in a significant reduction in the costs-by orders of magnitude. This efficiency is the main advantage of Alchemist, as it allows for the creation of high-quality datasets with minimal expense. We include ablation studies with other weak supervision models within the Alchemist framework in Appendix C. They successfully demonstrate the flexibility and robustness of using Alchemist.Setup. Next, we validate the extension of Alchemist to richer modalities. We consider our approach, where we prompt a multimodal model such as GPT4o and Claude 3, to generate high-level taskspecific concepts. We extract features for these concepts by employing CLIP as our local feature extractor. This converts raw pixels into feature vectors for the extracted high-level concepts, producing a set of similarity scores. Armed with these scores, we describe scores associated with their concepts in prompts and ask GPT4o and Claude 3 for 10 programs. As before, we use Snorkel as our aggregation procedure.Baselines. We study two baselines. The first is the vanilla version of Alchemist, where we directly ask GPT4o and Claude 3 to produce code that can operate on images (see left program in Fig. 4 ). The second is simple zero-shot prompting using CLIP, along with a variant, a group prompting approach that assumes access to spurious information and adds it to the given prompt 2 .Expected Results. We expect employing our two-step process can enable tractable program generation. In addition, we hypothesize that programs generated in this way are beneficial in targeting salient concepts and reducing the impact of irrelevant or shortcut features, thereby enhancing robustness.Results. We present results in Table 2 . Our evaluation focuses on three key metrics: average accuracy, worst group accuracy, and the gap between these two measures. Ideally, a robust model should achieve high average accuracy and high worst group accuracy while minimizing the disparity between the two. First, we see that directly asking programs to use may have very low performance (GPT4o) or may hugely suffer from spurious correlations, destroying worst group performance (Claude 3, CLIP zero-shot). Our method addresses both cases. Compared to baseline methods, Alchemist demonstrates increased worst group accuracy and a reduced gap between the average and worst group accuracies. This is a key strength of Alchemist: targeting salient concepts to be used as features may help move models away from spurious shortcuts found in the data. This validates Alchemist's ability to handle complex modalities while improving robustness. Setup. We test how integrating relevant information into the prompt context can augment generated programs. Instead of manually crafting supplementary information, we harness the power of language models to generate and integrate. This approach is useful for challenging datasets where users may not have the necessary knowledge or expertise to start. We evaluate the effectiveness of this approach, by comparing label model performance using programs generated by two different methods: pure prompting and in-context prompting. In-context prompting involves supplementary information, while pure prompting relies solely on the task description without any additional guidance. We employ GPT-3.5, GPT-4 and Claude 3 as our program sources and synthesize ten for each strategy.Expected Results. We hypothesize that providing supplementary information can enhance task understanding, demonstrate specific labeling logic, and offer concrete steps, ultimately leading to better programs for use.Results. Table 3 presents this comparative analysis on label model performance using different type of information. We observe that by incorporating supplementary information into pure prompts, Alchemist can guide language models to generate more effective programs, which in turn produce more accurate pseudolabels. Improvements are particularly evident in the challenging datasets such as Finance and French. Moreover, this approach can be combined with RAG systems to include external knowledge bases and customize the relevant information. Such flexibility compared to zero-shot prompting is another key strength of Alchemist, as programs can easily be adapted, augmented, and specialized.Setup. As shown in model to specialize better on certain data points than others. We seek to achieve these performance improvements without the need to re-prompt the model with each set of supplementary information. Instead, we collect previously generated programs to obtain a set of programs with greater diversity. We ask: can Alchemist achieve better performance by modeling more diverse programs?We randomly select a set of programs from each category, collect them, and train the label model with their program outputs. Additionally, we increase the number of sampled programs in each category from 4 to 9. We test this approach on the datasets where Alchemist gives comparable or lower performance than zero-shot prompting in our initial experiments in Table 1 , namely the SMS, Yelp, and IMDb datasets.Expected Results. By obtaining more diverse programs to use, Alchemist can capture a wider range of perspectives and labeling logic, potentially leading to more accurate pseudolabels.Results. Fig. 5 visualizes the effect on the label model's performance when we increase the diversity in collected programs. It demonstrates a trend and indicates that involving a more diverse set of programs can help to mitigate the impact of individual strategy biases or limitations, leading to the production of better labels.Overall, results in Sec. 4.3 and in Sec. 4.4 underscore that the use of supplementary information and involving diverse types of programs can help achieve better performance.Setup. Lastly, we compare synthesized programs in Alchemist and manually crafted labeling functions in WRENCH [50] , which is a widely-used benchmark for evaluating weak supervision methods. We focus on the datasets that overlap between Alchemist and WRENCH. For each dataset, we use pure prompts to query GPT-3.5, GPT-4, and Claude 3 for 10 programs. We then evaluate the performance of the distilled model for both methods. We also include the label model's coverage in our comparison. Higher coverage means that label model can produce more pseudolabels, yielding a larger size of training dataset to use.Expected Results. We expect that synthesized programs may offer some advantages in terms of efficiency and effectiveness compared to human-designed ones.Results. Table 4 presents their comparison. By leveraging the knowledge and capabilities of language models, we find that generated programs offer several advantages, including better coverage (i.e., the ability to label more data points) and comparable, or even better, performance. Generated programs can reduce the need for laborious engineering, which can be time-consuming and often requires a tedious design process to fine-tune labeling logic, such as thresholds and keyword usage. This design process may lead to many undiscovered rules, resulting in lower performance on coverage and potentially limiting the effectiveness of the labeling functions-unlike synthesized programs. This is particularly evident in the SMS dataset, where WRENCH requires 73 manually crafted labeling functions to obtain high-quality labels, while Alchemist only needs 10 generated programs to obtain comparable performance and higher coverage. This significant reduction highlights the potential of Alchemist to assist humans in designing labeling functions and make it more accessible to users without extensive domain expertise.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Yes. We report our training settings, and used prompts in the Appendix (See Appendix A and Appendix B). 
conclusion: 
 We propose an alternative approach to costly annotation procedures that require repeated API requests for labels. Our solution introduces a simple notion of prompting programs to serve as annotators.We developed an automated labeling system called Alchemist to embody this idea. Empirically, our results indicate that Alchemist demonstrates comparable or even superior performance compared to language model-based annotation, improving five out of eight datasets with an average enhancement of 12.9%. Notably, Alchemist reduces total costs by a factor of approximately 500. Furthermore, we showcase the system's extensibility to handle more complex modalities while enhancing the robustness of predicted labels. Finally, we confirm that incorporating relevant information can generate better programs, and increasing diversity leads to obtaining higher-quality labels.The appendix is organized as follows. First, we provide details about datasets, training settings, and computation resources in Appendix A. Next, in Appendix B we list prompts that we use to query language models. Then, we present ablation studies in Appendix C using other models in weak supervision to work with Alchemist. We include two additional modalities in Appendix D. We incorporate other work [51] for enhancing robustness into Alchemist and present in E. Lastly, we discuss limitations and broader impacts of our work in Appendix F.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Yes. We release our code and data here: :https://github.com/ SprocketLab/Alchemist.. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This doesn't apply to this paper.Answer: [NA] Justification: This doesn't apply to this work. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Dataset Task Type Prediction Classes # of Classes # of Train YouTube [43] spam comment detection {"spam", "ham"} 2 1686 SMS [44] spam text detection {"spam", "ham"} 2 4571 Yelp [45] restaurant review sentiment classification {"postive", "negative"} 2 30400 IMDb [45] movie review sentiment classification {"postive", "negative"} 2 20000MedAbs [48] medical abstract topic classification {"neoplasms", "digestive system diseases", "nervous system diseases", "cardiovascular diseases", "general pathological conditions"} 5 10395Cancer [14] biomedical document topic classification {"colon cancer", "lung cancer", "thyroid cancer"} 3 5450 Finance [46] finance news sentiment classification {"positive", "neutral", "negative"} 3 3488 French [47] book review sentiment classification {"positive", "neutral", "negative"} 3 6953 Waterbirds [42] bird species classification {"landbird", "waterbird"} 2 5794Table 5 : Dataset Table . We place more details about our datasets and experimental setups here. First, in Table 5 we show task type, prediction classes, and number of training data points in each dataset. MedAbs, Cancer, Finance, and French are considered to be more challenging settings, where these datasets typically need domain expertise to provide labels. Waterbirds is considered to test for a more complex modality.We employ Snorkel as our label model to aggregate program outputs and report results in the main paper. We show more results using different choices of label model in Appendix C. All the distilled models use the MLP model that is trained with 2 hidden layers, each comprising 32 units, using ReLU activations between layers and no normalization. We run 5 times with different random seeds and report their average performance. We use a A6000 NVidia GPU to run all experiments. First, we show the prompts used for the baseline approach of zero-shot prompting on text datasets in Table 6 . In these prompts, the placeholder "[text]" is replaced with individual data points and sent via API calls to obtain labels for each data point.Next, we present the prompts used in Alchemist in Table 7 . The table displays the task description component of each prompt. These descriptions outline the objective of the generated program and are associated with the prediction classes. For the labeling instructions, we directly map the prediction classes to their corresponding class indices and query the language models to output the appropriate class index.For the image task, we use the prompts ["an image of landbird", "an image of waterbird"] to perform zero-shot prompting using CLIP. In Alchemist, we first query high-level concepts and then combine them with computed scores to prompt LLMs to generate programs. The first step involves the following prompt: "What are the visual primitive concepts to classify "landbird" and "waterbird"? Please organize the primitive concepts by name and use comparisons for the classes. Parse the results into JSON format."Once we have obtained a set of similarity scores, we use the following prompt: "I have measured similarity scores for the following descriptions as float numbers. If a score is close to 1, it is highly related to the description. If a score is close to 0, it is less related to the description. The descriptions are: ["A bird's foot type is toed, grasping"]; ["A bird's foot type is paddling, swimming"]. Generate a labeling function with input scores to classify landbirds and waterbirds. If it cannot be determined, the function should return -1, but use this cautiously." Descriptions will be replaced by different generated concepts.Alchemist is compatible with a variety of weak supervision aggregation approaches. We report additional results with different choices of label models. Besides Snorkel, we consider three more widely-used label models: Weighted Majority Vote, Dawid-Skene [52] , and FlyingSquid (FS) [28] . We reuse our experimental setup from Sec. 4.1 and in Sec. 4.5 and present the performance of the distilled models in Table 8 and in Table 9 , respectively.In Table 8 , we observe that the label accuracy is enhanced or achieves comparable performance with different label models, showcasing Alchemist's flexibility in working with various label models. In Table 9 , we include compare them with human-crafted labeling functions developed in WRENCH [50] .Similarly, Alchemist obtains higher coverage and achieves comparable or even better label accuracy while reducing the need to craft a large number of programs manually.Next, we conduct an experiment by varying the temperature in our query APIs and running Alchemist on four different datasets. We train the end model five times with different random seeds and computed the average performance and the variance. Results are shown in Table 10 . We observe Temperature 0.0 0.755 0.020 0.795 0.005 0.589 0.000 0.866 0.014 0.550 0.001 0.690 0.000 0.729 0.000 0.935 0.009 0.5 0.898 0.002 0.870 0.005 0.603 0.013 0.854 0.019 0.519 0.000 0.690 0.001 0.733 0.000 0.940 0.010 1.0 0.817 0.004 0.803 0.017 0.667 0.022 0.930 0.012 0.555 0.001 0.690 0.000 0.731 0.000 0.938 0.006Table 10 : We varied the temperature in GPT-4 API calls, showing consistent performance across four datasets. We also trained the end model five times, displaying the average performance and the variance. These results confirm the stability of Alchemist.consistent labeling performance across different temperatures (0.0, 0.5, and 1.0), demonstrating Alchemist's stability. Additionally, the stability of generated programs highlights the significance of including aggregation models to handle noisy and diverse outputs, resolve conflicts, and produce final labels.We run Alchemist in two additional modalities in Table 11 : time-series (ECG heartbeat classification [53] ) and tabular (Census income classification [54] ). For ECG heartbeat classification, we generated 10 labeling programs from GPT-4o. For the Census income dataset, we generated program codes for each attribute (e.g., gender, education, age, race). We used Snorkel as our label model. The results demonstrate Alchemist's capability to handle more complex modalities and produce satisfactory performance. We compared Alchemist with human-crafted labeling functions from WRENCH. Alchemist uses fewer labeling functions (programs) and reaches higher labeling performance.In general, Alchemist will work well with any of these modalities as long as we have access to any cheap local feature extractor. This includes medical imaging tasks: [55] showed manually-crafted simple labeling functions were able to identify heart problems in MRI sequences based on very simple primitives, which could act as the feature extractors for Alchemist.We integrate advancements from existing robustness techniques into Alchemist to further improve accuracy and reduce spurious correlations. Specifically, we consider RoboShot [51] , which prompts LLMs for spurious and correct correlation features, then calibrates image embeddings by projecting them to reject or accept concepts. In our setup, we use GPT-4o to identify spurious correlations for Table 12 : We integrate Roboshot into Alchemist by querying GPT-4 for spurious correlation features and rejecting them, then reusing the generated programs in Alchemist. This integration improves average accuracy and worst-group performance, enhancing robustness to spurious correlations.classifying waterbirds and landbirds, then project image embeddings onto these concept embeddings to reject spurious correlations through subtraction. We compute cosine similarity using the calibrated embeddings to obtain score sets, which are then fed into Alchemist's generated programs. The spurious correlations identified by GPT-4o are ["water background", "land background", "aquatic plants", "trees and bushes"]. Results are displayed in Table 12 . This integration using GPT-4o successfully enhances robustness to spurious correlations by improving worst-group accuracies.Techniques to Evaluate Generated Programs. There are many ways to evaluate the quality of generated programs in advance. Expert users can quickly determine whether key problem properties are being used by looking at the code. Besides human inspection, Alchemist includes several automated measurement tools to diagnose generated programs. First, we analyze program outputs to compute coverage, polarity, conflict, and overlap (see 3 for definitions). For example, if coverage (the fraction of data points with at least one label) is below 10%, we discard the program and ask users to generate a new one. Moreover, if a validation dataset is available, Alchemist can run diagnostics to empirically compare accuracy with ground truth, offering more insight into the program's reliability. This data-driven feedback loop ensures tractable program generation. Notably, these tools are not typically accessible with model-based annotation methods.Limitations. There are two primary limitations in Alchemist. First, the performance of the datasets we test is still dependent on the capabilities of the language model. If the language model's ability to comprehend the given task and generate effective programs is subpar, the labeling performance may suffer. The second limitation arises when dealing with extremely complex tasks. As the complexity of the task increases, the generated code may become longer, more intricate, and harder to understand, posing challenges for developers who take time to validate correctness.Broader Impacts. We do not see explicit negative impacts in Alchemist's annotation process. However, generated programs from language models may contain biased labeling logic, toxic content, or malicious functions. To mitigate this, auditing and guardrails may be necessary.Justification: Yes. We discuss the limitations of the work in Appendix F.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).38th Conference on Neural Information Processing Systems (NeurIPS 2024).The latter option is preferable, as these models can often generalize beyond their source of supervision[13]the group prompts are "waterbird on water background", "waterbird on land background", "landbird on water background", and "landbird on land background".https://snorkel.readthedocs.io/en/v0.9.3/packages/_autosummary/labeling/snorkel. labeling.LFAnalysis.html 


Decision: We thank the authors for their submission and constructive engagement throughout the rebuttal phase of the process. The reviewers and AC were impressed by the simplicity, practicality, and versatility of the contributions, which we believe will be of significant interest to the broader applied ML community. The reviewers also highlight the rigour of the evaluations and clarity of exposition throughout. The authors do a good job addressing key outstanding questions by the reviewers throughout the rebuttal period. We encourage the authors to incorporate these additions in their next revision.

Reviews: review: paper_summary: paper_summary: The paper proposes an automated way to label large quantities of data by leveraging large language models to generate labeling functions which can be used to label data using programmatic weak supervision. The paper demonstrates that this procedure can generate labeling functions which are more accurate than manually generated LFs and querying LLMs for labels directly at a fraction of cost. The authors also show that prompt tuning benefits their method, and that it can be used for other richer modalities.
strength_weakness: strength_weakness: Most of these concerns are minor, and some of them can be readily fixed. 
1. **Comparisons with related work:**: I believe that comparisons with empirical ScriptoriumWS and DataSculpt are necessary to demonstrate the benefits of the proposed approach. With that said, I found the experiments to be rigorous. I would also like the authors to catalog the differences between their approach and ScriptoriumWS, DataSculpt, and [1] to better highlight the novelty of their methodology and findings. I am aware that [1] was published very close to the NeurIPS deadline so I do not expect the authors to compare their methods with [1], but conceptual differences (if any) can still be highlighted. 
2. **Missing references:**: The idea of using keywords and LLMs to automatically create LFs is not new, so I would encourage the authors to reference some prior work (e.g. [2]).
3. **Lack of reproduciblity and claims:** The techniques presented in the paper and the results are not reproducibile with the details given in the paper. I would encourage the authors to release their code for the same. The paper mentions that "any particular program may be inaccurate, fail to compile, or may otherwise be flawed, ...". I am curious how the authors ensure that the programs are not downright inaccurate (because snorkel would require the accuracy of LFs to be greater than random chance), or fail to compile? I believe that some kind of a data-driven feedback loop is important to ensure that the programs compile, and the generated LFs have coverage, and are accurate. Therefore, I am excited about the possibility to include keywords, dataset descriptions, example in the prompt, but I feel that there has to be a feedback loop after the label model labels the data or identifies inaccurate LFs.
4. **Richer modalities:** The claim on scaling to richer modalities should be softened, because the authors only evaluate their approach on a relatively simple image dataset. More challenging tasks may include labeling chest X-rays [3], datasets for which are publicly available, and other richer modalities such as time series data [4]. I do not expect the authors to conduct these experiments during the rebuttal, but such an extension would be very interesting and valuable, in my opinion. 

### References
1. Smith, Ryan, et al. "Language models in the loop: Incorporating prompting into weak supervision." ACM/JMS Journal of Data Science 1.2 (2024): 1-30.
2. Gao, Chufan, et al. "Classifying unstructured clinical notes via automatic weak supervision." Machine Learning for Healthcare Conference. PMLR, 2022.
3. Irvin, Jeremy, et al. "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.
4. Goswami, Mononito, Benedikt Boecking, and Artur Dubrawski. "Weak supervision for affordable modeling of electrocardiogram data." AMIA Annual Symposium Proceedings. Vol. 2021. American Medical Informatics Association, 2021.
questions: questions: I wonder what the authors thoughts are on points 3 and 4 above in the weaknesses.
limitations: limitations: The authors have discussed the limitation of their approach.
paper_summary: paper_summary: The paper presents a new method for creating data labels that leverage a Large Language Model and the weak supervision/data programming labeling paradigm. In this work, the LLM is used to generate labeling code, typically in the form of functions, which can then be used to create weak labels for weak supervision. The paper goes on to show this method’s utility across several benchmark text datasets and one image dataset. The paper also does an ablation study showing the impact of different elements of the method, like numbers of weak functions, and including examples in the prompt to create the functions.
strength_weakness: strength_weakness: The weakness of the paper is in its grounding/novelty and some of its methods, particularly when applied to other modalities. For the grounding, while the paper actually does capture many of the previous works that have done something similar, there are works like Cruickshank and Ng, “DIVERSE: Deciphering Internet Views on the U.S. Military Through Video Comment Stance Analysis, A Novel Benchmark Dataset for Stance Classification” where they used LLMs + weak supervision to create an actual new dataset. They did not, however, have the novel insight about creating the weak labeling functions that this work does.

For the method in the image or multimodal case, two works have done something very similar with CLIP. First, Adila et al. “Zero-shot Robustification of Zero-shot Models” use the same CLIP model and water birds dataset, but get substantial performance increases by “debiasing” the image embeddings, with text characteristics. This is very similar to what is done with the labeling functions where this paper tries to get labeling functions to classify aspects of the birds and ignore spurious contexts. Second, Bendou et al.’s “LLM meets Vision-Language Models for Zero-Shot One-Class Classification” presents a method for one-shot classification using VLMs, which uses an LLM to build negative classes around a positive class. This is similar to the insights in this paper around developing labeling functions to highlight important visual distinctions in the classes to improve the labels. Taken together, I think this paper might be able to incorporate the insights from these other papers to actually improve their results. For example, the debiasing in the first paper could be used with the proposed method.
questions: questions: I have no additional questions.
limitations: limitations: One other societal limitation the work could address is how works like this are changing the data labeling industry. Data labeling is still a massive, multi-million-dollar industry and these works are changing the nature of that industry. Ideally, they are changing it positively, but it can still disrupt how the data labeling industry works and cost human labelers their income.
paper_summary: paper_summary: The authors propose an innovative solution for high cost of APIs that using large pretrained models to generate programs that act as annotators. This idea helps replace or supplement crowdworkers and allows for distilling large models into smaller, more specialized ones. Traditional methods can be costly and produce static, hard-to-audit datasets due to expensive API calls. To tackle this, they introduce Alchemist, a system that tasks models to generate reusable labeling programs. These programs can be applied locally, significantly cutting costs while maintaining or even improving performance.
strength_weakness: strength_weakness: 1. One of the limitations of Alchemist is that the generated programs tend to handle only fixed tasks, often relying on threshold-based judgments. This means that for more complex or varied tasks, the system might still need to call specialist models via API, which somewhat limits its flexibility.

2. Another weakness is the lack of experimental evidence regarding the stability of the generated programs. For instance, the paper doesn't thoroughly explore how factors like the temperature variable in the code could affect the quality and consistency of the generated labeling programs.

3. The authors could also improve their literature review. There are several highly relevant papers that discuss various aspects of data annotation that weren't cited, such as [1] [2] [3] [4] [5]. 

[1] https://arxiv.org/abs/2310.04668
[2] https://arxiv.org/abs/2303.15056
[3] https://arxiv.org/abs/2306.04349
[4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834
[5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161
questions: questions: Refer to weaknesses.
limitations: limitations: No
paper_summary: paper_summary: The paper proposes a labeling workflow (Alchemist) using LLMs where instead of labeling each data point using a teacher model, we ask the teacher model to generate a program to label for the given task. Multiple such programs are generated and using an aggregation function, we get pseudo labels. These pseudo labeled points are then used as training data to train a student model. They show that such labeling mechanism is effective and less expensive compared to the teacher model labeling every data point. They also perform ablation studies for multi-modality, supplementary information during program generation, and diversity of programs.
strength_weakness: strength_weakness: As the results show, no performance degradation is not guaranteed (for e.g. table 1, Yelp dataset). For a new dataset, without comparing the approaches, it may be hard to understand if Alchemist works for it without impacting the performance.
questions: questions: 1. Why are the accuracy numbers for Alchemist with GPT-3.5 different in table 1 and table 3?
2. How does the performance compare if we use the generated programs for labeling the test set instead of training the student model?
3. Given that the performance gain/no loss is not consistent across datasets, what is the recommendation for a new dataset, i.e. how can one know whether this technique is going to work for that dataset? There are mentions that it works better for complex dataset, any other detailed characterization possible?
limitations: limitations: The main limitation is that it is hard to understand if Alchemist is going to work for your task without spending the money to label and compare. Performance loss when observed is not small to ignore. Generated programs may be reviewed by human experts, but they may not be straight-forward to interpret and gain confidence from.

Comments: title: Reply to Rebuttal
Thank you for the rebuttal!
comment: Dear reviewer,

Thank you for your valuable suggestion and your support. We are excited about the new results with Roboshot as well. We appreciate it! Thank you for your time!
The Authors have improved the paper further. I am particularly impressed to see the Roboshot idea incorporated and that it further enhanced the method. I stand by my rating of this being string accept.
Dear reviewer,

Thank you for your valuable feedback! Your suggestions have improved our paper. Appreciate it. We answer your questions below.

1. Thank you for your suggestion! We agree and have added additional descriptions to the paper addressing the output-driven feedback loop. One way to handle the challenge of making this work in non-text modality settings is to develop a set of simple unit tests that can be used to drive the feedback loop. 

2. The difference comes down to what the goals of these techniques are. ScriptoriumWS and DataSculpt focus on automating weak source (labeling function) creation, addressing the limitations of human-crafted labeling functions, which require subject matter experts to implement by hand. The ideal outcome of these systems is to speed up the process of performing weak supervision by reducing the complexity of writing correct code. In contrast, Alchemist addresses the downsides of model-based annotation, such as high cost, lack of extensibility, and difficulty in auditing. Its goal is to efficiently distill model capabilities. While the mechanisms currently used to accomplish these goals have similarities, they are motivated by different challenges.

3. Indeed, ScriptoriumWS does not operate on non-text modalities at all; that technique is to directly prompt a language model to generate programs/labeling functions that perform a text task. The most straightforward way to extend the ScriptoriumWS methodology is to prompt a multimodal model like GPT4o to generate programs over the target modality (e.g., images). There are two downsides we have identified to this approach to extending ScriptoriumWS:
    * It requires access to a powerful multimodal model, which may not exist for many modalities of interest,
    * Even if such multimodal models exist, they may struggle with spurious correlations. 

While we did not directly evaluate ScriptoriumWS with GPT4o, we did extend Alchemist itself in this fashion as a baseline. In Table 2 in our paper, we reported our findings, which suggest that the spurious correlation issue is indeed a problem. In contrast, Alchemist’s approach, based on obtaining primitives and using a cheap feature extractor, reduces the impact of these spurious correlations. It additionally also mitigates the other challenge, as such feature extractors are much easier to obtain compared to a powerful multimodal model. 

4. In this setting, we did not use the two-stage extension method to generate programs. Instead, we included supplementary information, such as dataset and prediction class definitions, directly in the prompts. Following the basic recipe for our work, we generate 10 programs and use Snorkel as the aggregation method to produce predictions. We show two generated programs below as examples. They use peak count as their labeling logic. The first program achieved 69.5% accuracy, while another reached 81.5%.

```
def label_by_fft_peak(time_series):
    """ Label based on the frequency domain characteristics """
    fft_result = np.fft.fft(time_series)
    fft_magnitude = np.abs(fft_result)
    peak_freq = np.argmax(fft_magnitude[1:]) + 1  # Ignoring the DC component
    if peak_freq < 15:
        return 0  # 'N'
    elif peak_freq < 30:
        return 1  # 'S'
    elif peak_freq < 45:
        return 2  # 'V'
    elif peak_freq < 60:
        return 3  # 'F'
    else:
        return 4  # 'Q'
```
```
def label_by_peak_count(time_series):
    """ Label based on the number of peaks in the signal """
    from scipy.signal import find_peaks
    peaks, _ = find_peaks(time_series, height=0)
    peak_count = len(peaks)
    if peak_count > 7:
        return 0  # 'N'
    elif peak_count > 5:
        return 1  # 'S'
    elif peak_count > 3:
        return 2  # 'V'
    elif peak_count > 1:
        return 3  # 'F'
    else:
        return 4  # 'Q'
```
Thank you for your response and valuable feedback. Your suggestions have improved our paper. If you have any further questions, feel free to ask—we’re happy to provide more information. Thanks again!
The authors have satisfactorily addressed most of my problems. Most concerns has been addressed and some senarios may out of scope of this paper. I have raised my score. Once again, I want to express my gratitude for your hard work and commitment.
Dear Authors, 

Thank you so much for your rebuttal and putting in time and effort. I have raised my score to reflect my current assessment of the paper.

Below are some general thoughts based on the rebuttal:
1. In addition to data-driven feedback, I also think that output driven feedback is important. In some cases the LLM might hallucinate and not generate LF code grounded on the input prompts, in which case an output-driven feedback loop can ideally guide the model to generate correct programs (accuracy is the second step, the model must generate correct programs first). I foresee this to be a problem in non-text modalities where the model is having to use APIs or local feature extractors. I think some discussion on this and your empirical findings would be beneficial to the community. 
2. "*Both ScriptoriumWS and DataSculpt address weaknesses in weak supervision, whereas Alchemist focuses on the downsides of model-based annotation.*" Can you explain this statement? 
3. It seems that ScriptoriumWS and Alchemist are similar in terms of accuracy and the # of programs which are generated. It seems from your description that the difference lies in extensibility to other richer modalities. Can you explain what exactly prevents the core ScriptoriumWS methodology to model richer modalities? The answer can be as simple as, "the method *can* be extended with X, Y and Z, but this was not evaluated". 
4. I would like to authors to provide some details about their time series classification experiment. The results are encouraging, and this is out of curiosity. I re-read the section on richer modalities in the paper, and the extension to time series doesn't seem trivial. I would like to know more about how the authors carried out this experiment.
Thank you! Your suggestions helped us to improve the paper. Please let us know if you have any further questions; we are very happy to follow up. If there are no further concerns, we would appreciate it if you increase your score---we appreciate it!
I thank the authors for the detailed responses to my questions and additional experiments.