DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines
Authors: Omar Khattab
Arnav Singhvi
Paridhi Maheshwari
Zhiyuan Zhang
Keshav Santhanam
Sri Vardhamanan A
Saiful Haq
Ashutosh Sharma
Thomas T. Joshi
Hanna Moazam
Heather Miller
Matei Zaharia
Christopher Potts
Venue: ICLR 2024 spotlight (2024)
Keywords: programming models
prompting techniques
in-context learning
few-shot learning
chain of thought
multi-hop reasoning
language agents
Primary area: infrastructure, software libraries, hardware, etc.
TL;DR: We propose a programming model that unifies LM prompting, finetuning, and augmentation techniques. We evaluate simple strategies to bootstrap and optimize complex and multi-stage reasoning chains, establishing strong results with small and large LMs.

Abstract: The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded “prompt templates”, i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, or imperative computational graphs where LMs are invoked through declarative modules. DSPy modules are parameterized, meaning they can learn how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. We design a compiler that will optimize any DSPy pipeline to maximize a given metric, by creating and collecting demonstrations. We conduct two case studies, showing that succinct DSPy programs can express and optimize pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops. Within minutes of compiling, DSPy can automatically produce pipelines that outperform out-of-the-box few-shot prompting as well as expert-created demonstrations for GPT-3.5 and Llama2-13b-chat. On top of that, DSPy programs compiled for relatively small LMs like 770M parameter T5 and Llama2-13b-chat are competitive with many approaches that rely on large and proprietary LMs like GPT-3.5 and on expert-written prompt chains. DSPy is available at https://github.com/stanfordnlp/dspy

Published as a conference paper at ICLR 2024
DSPY: COMPILING DECLARATIVE LANGUAGE MODEL CALLS INTO SELF-IMPROVING PIPELINES
Omar Khattab,' Arnav Singhvi,?
Paridhi Maheshwari, 4 Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, 5 Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller,
3.9 Matei Zaharia, Christopher Potts
Stannfor University, PUC Berkeley, CCaneeii Mellon University, 4 *Amazon Alexa AI, Dashworks Technologies, Inc., "IT Bombay, 7 "Calera Capital, Microsoft Qatar, Two Sigma Investments
okhat stanford.
ABSTRACT
The ML community is rapidly exploring techniques for prompting language mod- els (LMs) and for stacking them into pipelines that solve complex tasks. Un- fortunately, existing LM pipelines are typically implemented using hard-coded "prompt templates", i.e. lengthy strings discovered via trial and error. Toward more systematic approach for developing and optimizing LM pipelines, we intro- duce DSPy, programming model that abstracts LM pipelines as text transforma- tion graphs, or imperative computational graphs where LMs are invoked through declarative modules. DSPy modules are parameterized, meaning they can learn how to apply compositions of prompting. finetuning, augmentation, and reason- ing techniques. We design compiler that will optimize any DSPy pipeline to maximize given metric, by creating and collecting demonstrations. We conduct two case studies, showing that succinct DSPy programs can express and optimize pipelines that reason about math word problems, tackle multi-hop retrieval, swer complex questions, and control agent loops. Within minutes of compiling, DSPy can automatically produce pipelines that outperform out-of-the-brr few- shot prompting as well as expert-created demonstrations for GPT-3.5 and Llama2- 13b-chat. On top of that, DSPy programs compiled for relatively small LMs like 770M parameter T5 and Llama2- 13b-chat are competitive with many approaches that rely on large and proprietary LMs like GPT-3.5 and on expert-writted written prompt chains. DSPy is available at https: thub. com/ ctmrsstanff stanfordnlp/dspy.
INTRODUCTION
Language models (LMs) are enabling researchers build NLP systems at higher levels of abstrac- tion and with lower data requirements than ever before (Bommasani et al., 2021). This is fueling an exploding space of "prompting" techniques-and lightweight finetuning techniques-for adapting LMs to new tasks (Kojima et al., 2022), eliciting systematic reasoning from them (Wei et al., 2022; Wang et al., 2022b). and augmenting them with retrieved sources (Guu et al., 2020; Lazaridou al., 2022) or with tools (Yao et al., 2022). Most of these techniques are explored in isolation, but interest has been growing building multi-stage pipelines and agents that decompose complex tasks into more manageable calls to LMs in effort to improve performance (Qi et al., 2019; Khattab al., 202 Karpas et al., 2022; Dohan et al., 2022; Pourreza & Rafiei, 2023; Yao et al., 2023).
Unfortunately, LMs are known to be sensitive to how they are prompted for each task, and this is exacerbated in pipelines where multiple LM calls have to interact effectively. As a result, the LM calls in existing LM pipelines and in popular developer frameworks hard-coded 'prompt templates', that is, long strings of instructions and demonstrations that are hand crafted through manual trial and error. We argue that this approach, while pervasive, be brittle
generally implemented using

---

Published as a conference paper at ICLR 2024
and unscalable-ccomceptuaay conceptually akin to might not generalize to different pipelines or across different LMs, data domains, or even inputs.
the weights for a classifier. A given string prompt
Toward more systematic approach to designing Al pipelines, we introduce the DSPy programming model. DSPy pushes building new LM pipelines away from manipulating free-form strings and closer to programming (composing modular operators to build text transformation graphs) where compiler automatically generates optimized LM invocation strategies and prompts from a program. We draw inspiration from the consensus that emerged around neural network abstractions (Bergstra et al., 2013), where (1) many general-purpos layers can be modularly composed in any complex architecture and (2) the model weights can be trained using optimizers instead of being hand-tune -und-e.
To this end, we propose the DSPy programming model (Section 3). We first translate string- -bannd prompting techniques, including complex and task-dependent ones like Chain of Thought (Wei et al., 2022) and ReAct (Yao et al., 2022), into declarative modules that carry natural- -aniuagg--- typed sig- natures. DSPy modules are task-adaptive components- akin to neural network layers-thhe abstract any particular text transformation, like answering question or summarizing a paper. We then pa- rameterize each module so that it can learn its desired behavior by iteratively bootstrapping useful demonstrations within the pipeline. Inspired directly by PyTorch abstractions (Paszke et al., 2019), DSPy modules are used via define-by-run computational graphs. Pipelines are expressed by (1) declaring the modules needed and (2) using these modules in any logical control flow (e.g., if statements, for loops, exceptions, etc.) to logically connect the modules.
We then develop the DSPy compiler (Section 4), which optimizes any DSPy program improve quality or cost. The compiler inputs are the program, a few training inputs with optional labels, and validation metric. The compiler simulates versions of the program on the inputs and bootstraps example traces of each module for self-improvement, using them to construct effective few-shot prompts or finetuning small LMs for steps of the pipeline. Optimization DSPy is highly modular: it is conducted by teleprompters," which are -pueral-purpose optimization strategies that determine how the modules should learn from data. In this way, the compiler automatically maps the declara- tive modules to high-quality compositions of prompting, finetuning, reasoning, and augmentation.
Programming models like DSPy could be assessed along many dimensions; we focus on the role of hand-crafted prompts in shaping system performance. We are seeking to reduce or even remove their role through DSPy modules and teleprompters. We report on two expansive case studies: math word problems (GSM8K; Cobbe et al. 2021) and multi-hop question answering (HotPotQA; Yang et al. 2018) with explorations of chain of thought, multi-chain reflection, multi-hop retrieval, cerreval-augmentee question answering, and agent loops. Our evaluations a number of different compiling strategies effectively and show that straightforward DSPy programs outperform systems using hand-crafted prompts, while also allowing our programs use much smaller and hence more efficient LMs effectively. Overall, the main contributions of this paper are empirical and algorithmic: with DSPy, we have found that we can implement very short programs that can bootstrap state-of- the-art multi-stage NLP systems using LMs as small as Llama2-1 with no hand-crafted prompts.
2 RELATED WORK
This work is inspired by the role that Torch (Collobert et al., 2002), Theano (Bergstra et al., 2010; 2011; Al-Rfou et al., 2016), Chainer (Tokui et al., 2015), and others played in the development in deep learning by providing powerful abstractions. We are seeking to offer a solid conceptual framework and programming abstractions for what we call foundation model programming. We draw differentiable programming (Wang et al., 2018) but applied to LM calls rather than neural networks, and borrow syntactic elements from PyTorch (Paszke et al., 2019).
In-context learning (McCann al. 2018; Radford al. 2018; Brown et al. 2020) is a key mechanism for programming LMs in this new mode. A growing body of work has revealed that, especially with instruction tning (Ouyang et al., 2022), we can elicit sophisticated behavior via prompting (Wei et al., 2022; Wang et al., 2022b; Press et al., 2022; Yao et al., 2022; Khot et al., 2022; Madaan et al., 2023). Similarly, forms of weak supervision that would normally require task-specific (Khattab
DDSPy is pronounced dee-ess-pie. It's the second iteration of our earlier Demonstrate-SSarcch-Pe framework (DSP; Khattab et al. 2022).
We derive the name tele-prompters from the notion of abstracting and automating the task of prompting- such it happens a distance, without manual intervention.
2

---

Published as a conference paper at ICLR 2024
et al., 2021a;b) or hand-built (Ratner et al., 2016; Hancock al., 2018) heuristics are now done by LMs (Wang et al., 2022b; Zelikman et al., 2022; Zhang et al., 2022; Shao et al., 2023). In-context learning methods now routinely invoke tools, leading to LM pipelines that use retrieval (Chen al., 2017; Lewis et al., 2020; Guu et al., 2020; Lazaridou et al., 2022; Izacard et al., 2022), and APIs (Nakano al., 2021). Toolkits have been developed to facilitate this, including LangChain (Chase, 2022), Semantic Kernel (Microsoft, 2023), and Llamalndex (Liu, 2022), and many other retrieval or agent libraries. While these toolkits provide pre-packaged chains and agents that connect LMs with tools, they suffer from the pervasive prompt engineering challenges we address in DSPy: in particular, their abstractions still express task sse--speciiic behavior through hand-writter prompt tem- plates (see Appendix D). LMQL (Beurer-Kellner et al., 2023) is a related query language, which efficiently constrains the decoding algorithms of LMs to generate only strings that fulfill logical constrains (e.g., lists of bullets or values formatted correctly for a calculator). Whereas DSPy fo- cuses on optimizing LM pipelines for a given metric, LMQL's lower-level interface for controlled decoding could be beneficial to implement specific advanced modules within DSPy.
Researchers are starting to apply discrete optimization and RL to find effective prompts, generally for a single logical LM call (Guo et al., 2023; Pryzant et al., 2023; Huang et al., 2022; Yang et al., 2023). DSPy seeks to generalize this space: it offers a rich framework for optimizing arbitrary pipelines from high-level declarative signatures, by bootstrapping high-quality multi-stage demon- strations with constraints. In this framework, DSPy teleprompters may apply optimization using model selection techniques like cross-validation or, in principle. with sophisticated techniques in- volving RL and LM feedback (Hu et al., 2023; Zhao et al., 2023a; Shinn et al., 2023) or learned or Bayesian hyperparameter optimization methods (Bergstra et al., 2013; Akiba al., 2019). The present paper seeks to motivate DSPy as programming model and to report new empirical find- ings from applying the DSPy compiler. This inspired by formative work by Bergstra et al. (2010; 2013), Paszke et al. (2019). and Wolf et al. (2020), who support their respective programming mod- els with a mix of benchmark numbers and some qualitative measures. For the current paper, we focus on showing that DSPy and its compiler allow to build outstanding LM systems without hand-crafted prompt strings, but instead from truly modular units, and that this opens up doors for systematically exploring rich design space at very high programmatic level of abstraction.
THE DSPY PROGRAMMING MODEL
We present DSPy, which treats LMs as abstract devices for text generation," 3 and optimizes their us- age in arbitrary computational graphs. DSPy programs are expressed in Python: each program takes the task input (e.g.. question to answer or paper to summarize) and returns the output (e.g., an answer or a summary) after series of steps. DSPy contributes three abstractions toward automatic optimization: signatures, modules, and teleprompters. Signatures abstract the input/output behavior of module; modules replace existing hand- prommpptp techniques and can be composed in arbitrary pipelines; and teleprompters optimize each module in the pipeline to maximize metric.
3.1 NATURAL LANGUAGE SIGNATURES CAN ABSTRACT PROMPTING & FINETUNING
Instead of free- free-form string prompts, DSPy programs use signatures to assign work to the LM. This is a natural- lauuaal- typed function declaration: a declarative spec that tells DSPy what a transforma- tion needs to do (e.g., consume questions and return answers), rather than how specific LM should be prompted to implement that behavior. More formally, a DSPy signature is a tuple of input fields and output fields (and optional instruction). A field consists of field name and optional metadata." The roles of fields are inferred from field names, e.g. the compiler will use in-context learning to interpret question differently from answer and iteratively refine its usage of these fields.
Signatures offer two benefits over prompts: they support compilation into high-quality, self- improving, and applline-adaptive prompts or finetunes. This is primarily done by bootstrapping (Sec 4) useful demonstrating examples for each signature. Additionally, they handle structured for- matting and parsing logic to reduce (or, ideally, avoid) brittle string manipulation in user programs.
P We assume access one or more LMs, which consume a prompt string and return completions. This may a promptable IM capable of in-context learning (c.g., GPT-3.5 or llama2-7b) or a smaller finetuneable LM (c.g., T5-base). An LM may be selected as the default; operations will use it unless configured otherwise. DDeccription of the task is optional and usually omitted. Fields can carry optional prefix and description.
3

---

Published as a conference paper at ICLR 2024
In practice, DSPy signatures can be expressed with shorthand notation like question -> answer, so that line in the following is a complete DSPy program for a basic question -antwerrnnn system (with line illustrating usage and line 3 the response when GPT-3.5 is the LM):
qa dapy. Predict(" (ellcccttqqquuttt ("question -> answer") ga(question="Wher is Guarani spoken?") Out: Prediction (answer= HHinnly in South America
In the shorthand notation, each field's name indicates the semantic role that the input (or output) field plays the transformation. DSPy will parse this notation and expand the field names into meaningful instructions for the I.M, so that engl _occuueeeetttt -> Trench_transi alliiotinnnn would prompt for English to French translation. When needed, DSPy offers more advanced programming interfaces for expressing more explicit constraints on signatures (Appendix B).
3.2 PARAMETERIZED & TEMPLATED MODULES CAN ABSTRACT PROMPTING TECHNIQUES
Akin to type signatures in programming languages, DSPy signatures define an interface and pro- vide type-like hints on the expected behavior. To use a signature, we declare module with that signature, like the Predict module above. A module declaration like this returns a function having that signature. The core module in DSPy is Predict (Appendix G.I), which stores internally the (initially None, but otherwise overrides the default LM for
supplied signature, an optional LM this module), and a list of demonstrations for prompting (initially empty). The instantiated module behaves as a callable function: it takes in keyword arguments corresponding to the signature input fields (e.g., quest ion), formats a prompt to implement the signature and include the appropriate demonstrations, calls the LM, and parses the output fields. When Predict detects it's being used in compile mode, it internally tracks input/output traces to assist bootstrapping.
DSPy includes more sophisticated modules like ChainOfThought, ProgramOfThought Mul watiihainnomparison ion, and ReAct,' These can be used interchangeably to implement a DSPy signature, e.g. changing Predict to ChainOfThought above leads to a system that thinks step by step before committing to its output field. Importantly, these modules are implemented in a few lines of code by expanding the user-defined signature and calling Predict one or more times on new signatures. We show simplified implementation of ChainofThought in Appendix G.2 using seven lines of code. This is a fully-fledged module capable of learning effective few-shot prompt- ing for any LM or task. We contrast that with Appendix E, which copies long reasoning prompts hand-written by recent papers and prompting libraries.
DSPy modules translate prompting techniques into modular functions that support any signature by parameterizing these modules. To understand this, observe that any LM call seeking to implement a particular signature needs to specify parameters that include: (1) the specific LM call (Chen al., 2023), (2) the prompt instructions (Yang et al., 2023) and the string prefix of each signature field and, most importantly, (3) the demonstrations used few-shot prompts (for frozen LMs) as training data (for finetuning). We focus in this paper on generating and selecting useful demonstrations and using them for prompting or finetuning. We find that bootstrapping demonstrations gives us a powerful way to teach sophisticated pipelines of LMs new behaviors systematically.
DSPy modules can be composed arbitrary pipelines in beeine-by-run interface. Inspired di- rectly by PyTorch and Chainer, one first declares the modules needed at initialization, allowing DSPy to keep track of them for optimization, and then one expresses the pipeline with arbitrary code that calls the modules in a forward method. As a simple illustration, we offer the follow- generation (RAG) system. 6 DSPy programs may use tools, which are modules that execute computation. We support retrieval models through dspy. Retrieve module. At the time of writing. DSPy has built-in support for ColBERTv2, Py- serini, and Pinecone retrievers, and we have explored experimental dspy. SQL for executing SQL queries and dspy. Pythoonnneerprreeer for executing Python code in a sandbox.
ing simple but complete
serieval-augmeenee
'These generalize prompting techniques from the literature, respectively, by Wei et al. (2022), Chen et al. (2022), Yoran al. (2023), and Yao et al. (2022).
'To highlight modularity, we use ChainofThought as a drop-in replacement of the basic Predict. One can now simply write RAG() ("Where is Guaraní spoken?") to use it. Notice that, if we use signature "context, question search query", we get a system that generates search queries rather than answers.
4

---

Published as a conference paper at ICLR 2024
class RAG Module):
def
...intt__ (self, nun _ passsages=:)
"Retrieve" will the default retrieval settings unless overriden.
self. retrieve dspy. (etruieeeeeuuu.passsaee
"Chaindfthought
signature that generates answers given retrieval question.
self. generate__nswee
dspy. ChainofThoughh ("contexx, question +> answer*)
def
forwer question):
context self. retrieve serriivveequeeee (questiquu..n passages
return self gelffgggnrrtteaannee (ew.teeecontttccccme (context=cantext, questinn=question)
3.3 TELEPROMPTERS CAN AUTOMATE PROMPTING FOR ARBITRARY PIPELINES
When compiling a DSPy program, we generally invoke teleprompter, which is an optimizer that takes the program, training set, and metric-and returns a new optimized program. Different teleprompters (Sec 4) apply different strategies for optimization, typically gradient-free. In DSPy, training sets may be small, potentially a handful of examples, though larger data enables more powerful optimization. Training examples may be incomplete, i.c., only input values are necessary. Labels for the pipeline steps we typically assume labels only for (at most) the program's 's final output, not the intermediate steps. This label-efficiency critical for modularity: building a pipeline in DSPy requires simply recompiling the new pipeline's 's code, not annotating data specific to the new pipeline.
required, unless they need to be used in the metric. In practice,
Metrics be simple notions like exact match (EM) F1, they be entire DSPy programs that balance multiple concerns. For example, we may compile the RAG module above against a dataset of question-answer pairs qa_trainset and the metric EM. The goal of optimization here is to effectively bootstrap few-shot demonstrations. The following code achieves this:
Snall training set with only questions and final qa_trainset Edspy. Exampy. (questiion What is the copital of France?", answer="Paris")
# The teleprompter will bootstrap missing labels: reasoning chains and retrieval contexts. answer_exact__mathh
teleprompter supp.BootstraaeewShee (nerriccsyeererreeeeeeeee compiled_rag teleprompter.. wwtpeeteee..roooeeeeeeeeeeeeeeee compile (RAG(), trainsetsqp_trainsst.
In this example, the BootstrapFewShot teleprompter (Sec 4, Appendix G.2) simulates RAG on the training example(s). It will collect demonstrations of each module (i.e., examples of its input-output behavior) that collectively lead to valid output (i.e., respecting the signatures and the metric).
Teleprompters can be composed by specifying teacher program. DSPy will sample demonstra- tions from this program for prompt optimization. This composition can enable very rich pipelines, where expensive programs (e.g., ensembles of large LMs) supervise cheap programs (e.g., simpler pipelines using smaller LMs). One may start with compiled rra from above (say, compiled to use a large Llama2- 13b-chat LM) but now tin-tune Flan-T5- I5-raee to create an efficient program:
a Larger set of questions with And labels*. Labels for all steps will be bootstrapped. lunabbleed_ueestionss Idspy (quust(qqusstion=" is capital Gernany?") ...]
3 assune no answer, we use wenettiinn__elepppppee Bootstrapf inetune (netricsdsny. anw.er waweerppsssae_eetee wlt.eeddaaaaaaaaieeeeeeeeeeeeeee finetuning www.niinggellppopppeeeoo compile (RAG(), teacher:conpiled_.ra crsinstt-unlabeele__ussett
to filter ungrounded answers.
target= oooellooooooooooe foogle//lan-t5-lares=
THE DSPY COMPILER
key source of DSPy's 's expressive power its ability compile- or automatically optimize any program. Compiling relies on teleprompter, which is an optimizer for DSPy programs that improves the quality (or cost) modules via prompting or finetuning, which are unified in DSPy. The compiler first finds all unique Predict modules (predictors) in a program, including those nested under other modules. While DSPy does not enforce this when creating new teleprompters, typical teleprompters go through three stages.
Candidate Generation For each predictor p. the teleprompter may generate candidate values for the parameters p: the instructions, field descriptions, or demonstrations (i.e., example input-output pairs). In this iteration of DSPy, we focus on demonstrations and find that simple rejection sampling can help bootstrap highly effective multi-stage systems. Consider the simplest non-trivial teleprompter in DSPy, BootstrapFewShor (simplified pseudocode in Appendix H.
Stage 1:
5

---

Published as a conference paper at ICLR 2024
This teleprompter will simulate a teacher program (or, if unset, the zero-shot version of the program) on some training inputs, possibly one or more times with a high temperature. When running in compile mode, multi-stage traces are tracked transparently and in thread-safe fashion throughout execution. The program's 's metric is used to filter for multi-stage traces that together help the pipeline pass the metric. We thus obtain potential labels for all signatures in the program by throwing away the bad examples and using the good examples as potential demonstrations, though these design decisions are under user control. While LMs can be highly unreliable, we find they can be rather efficient at searching the space of solutions for multi-stage designs. A well-decomposed program can typically find at least a few training examples where the LM can pass the constraints enforced by the signatures and metrics, allowing us to bootstrap iteratively if needed.
Stage 2: Parameter Optimization Now each parameter has a discrete set of candidates: demon- strations, instructions, etc. Many hyperparameter tuning algorithms (c.g., random search or Tree- structured Parzen Estimators as in HyperOpt (Bergstra et al., 2013) and Optuna (Akiba et al., 2019)) can be applied for selection among candidates. We report a simplified implementation of DSPy's Bootstrapf ewShotwi BootstraapewShotiii thRandomSearch in Appendix H.2. Another type of optimization is finetun- ing with BootstrapF wwesstrpaiinettt intuune, where the demonstrations are used to update the LM's weights for each predictor. When this is applied, the LM of each module is updated to the new LM weights.
Stage 3: Higher-Order Program Optimization A different type of optimization that the DSPy compiler supports is modifying the control flow of the program. One of the simplest forms of this is ensembling, which we use in the case studies in this work. An ensemble will bootstrap multiple copies of the same program, and then replace the program with a new one that runs them all in parallel and reduces their predictions into one with custom function (e.g., majority voting). In future work, this stage can easily accommodate techniques for more dynamic (i.c., test-time) bootstrapping well as automatic backtracking-Iin like logic.
GOALS OF EVALUATION
Programming frameworks can be evaluated along many dimensions: computational efficiency, de- veloper efficiency, intuitiveness of the code and concepts, and so forth. In this paper, we focus on perhaps the most pressing issue for current LM pipelines: the role of hand-written, taec-specific prompts in achieving performant systems. Our evaluations seek to test the following hypotheses:
H1 With DSPy, we can replace hand-crafted prompt strings with concise and well-defined modules, without reducing quality or expressive power.
H2 Parameterizing the modules and treating prompting as an optimization problem makes DSPy better at adapting to different LMs, and it may outperform expert-written written prompts.
H3 The resulting modularity makes it possible to more thoroughly explore complex pipelines that have useful performance characteristics or that fit nuanced metrics.
Our evaluation will explore these hypotheses using diverse task-program pairs. We hope this begins shift from underspecified questions like "how do different LMS compare on GSMSK" toward "how they compare on GSM8K with program P when compiled with strategy S", which is a well-defined and reprodueible run. Ultimately, our goal to reduce the role of artful prompt construction in modern AI, in favor of the development of new modular, composable programs and optimizers.
6 CASE STUDY: MATH WORD PROBLEMS
We evaluate on the popular GSM8K dataset with grade school math questions (Cobbe et al., 2021). We sample 200 and 300 question question-answer pairs from the official training set for training and develop- ment, respectively. Our final evaluations use the 1.3k official test examples. We report extensive comparisons on the development set to avoid overfitting on test. Following prior work on GSM8K, evaluate the accuracy of the final numerical value that appears in the LM output.
Programs Considered For this task, we consider three simple DSPy programs: one-step Pre- dict module (vanilla), a two-step ChainOfThought module (CoT), and finally a multi-stage ComparerOfThoughts module [ThoughtReflectionn reflection, five reasoning chains are sampled from the LM (alongside their answers) and they are compared in parallel by built-in Mult MultiChainComparise module, which generalizes Yoran
These are defined by the code below. In

---

Published as a conference paper at ICLR 2024
Table 1: Results with in-context learning GSM8K math word problems. Each row represents a separate pipeline: the module in the Program column is compiled against the examples in the Training set. The programs, compilers, and (small) training sets are defined in Section 6. Rows with ensemble build on the immediately preceding row. Notably, all programs in this table are expressed by composing two four DSPy modules and teleprompters. Compiling the right modules, instead of string prompts, improves different LMs from 9-25% accuracy to 46-81% 1 accuracy.
GPT-3.5
Llama2- 1atma2-13bbaee 13b-chat
Program
Compilation
Training
Dev
Test
Dev
Test
none
n/a
24.0
25.2
7.0
9.4
fewshot
trainset
33.1
4.3
vanilla
bootstrap
trainset
44.0
28.0
bootstrapxx
trainset
64.7
61.7
37.3
36.5
+ensemble
trainset
62.7
61.9
39.0
34.6
none
n/a
56.0
26.7
fewshot
trainset
27.3
CaT
Fewshat
+human_CoT
78.6
72.4
34.3
33.7
bootstrap
trainset
80.3
72.9
43.3
+ensemble
trainset
88.3
81.6
43.7
u/a trainset
65.0
36.7
reflection
fewshot hootstrap +ensemble
71.7
36.3
trainset
83.0
76.0
44.3
40.2
trainset
86.7
46.9
al. (2023). This generates answer taking into account the patterns from the five attempts. Critically, the modules used are all generic, none is specific to math problems or a particular LM.
depy. Predict question answer")
Program "vanilla"
dapy, ChainoFFhouuhhtt( question answer")
Progran
ThoughtRer Medule): der
(self, nur_attenpts):
self-predict dspy. Chain0f ("cuestton ansver", senun_attempts)
selr. selr. compare dapy. inCoopaaa Isan(
iaaaarrssnnn""isiioo "questtin -> answer". atous_attenpts)
def fsrl, question):
10
completions self -elf- return self.
completions
corpare (question-question completion penplleioonnccpeetiiioo
13 reflection ThoughtReflection (nnprrioonnnunsttetssstte
6SHBK Progran 'reflection"
Compiling As we discussed in Section 4, DSPy programs can be compiled into new, optimized programs. In experiments, evaluate the programs zero-shot (no compiling) well number of strategies for compiling. Our simplest compiler is LabeledF FewShtt:
fewshot = dspy. LabeledFewShot (k=a))ttkkkkooooooo compi) compile (progran, trainsetstrainset)
Here, program can be any DSPy module. the fields common to the training examples and the signature(s), in this case, question and answer but not reasoning for instance. We report the average of 3-5 runs (depending the setting) when applying such sampling. Next, we consider bootstrapping few-shot examples with random search:
This samples k=8 demonstrations from the trainset for
tp wwwrraafesssootttteennnnnneeeeee (eneecchcnnniiiigeeeeee stt.rriicsssskkkcccrrayyy bootstrap tp. compi 1 (progran, trainsetstrainset, valsetadevset)
This will generate demonstration chains for examples in the training set and optimize the selection of demonstrations (from this set) to self-improve the program's 's modules. As the name indicates, this is done with random search, treating the selection of demonstrations as a parameter to optimize. Next, if desired, this bootstrapping process can be nested in DSPy. In particular, we can use the optimized bootstrap program itself to further bootstrap another program. This is relevant, for example, whenever the original zero-shot program performs relatively poorly.
bootstrap2 tp. compile teacherr trainset-trainset. velset=devset)
And lastly, we consider ensembling these bootstraps:
# A progran that ensembles the top-7 candidate programs fron a bootstrapping compiler run (in particular 'bootstrap' or. when applicable. 'bootstrap2') with najority voting. ensemble
majority) compi (bootstrrapproograsd

---

Published as a conference paper at ICLR 2024
GSM8K includes human reasoning chains. Above, trainset does not include these reasoning chains. We also evaluate with trainset.human.Com which extends the examples in trainset with the human reasoning string. These two datasets can be used interchangeably as the value for the trainset parameter above. We note here that compiling generally runs on the order of minutes (or tens of minutes) as even the more expensive settings only require running the program few thousand times (e.g., 10-20 trials over 150-300 validation examples) and they can occur in parallel.
Our results are summarized in Table 1, which includes dev results as well as our evalua- tion of promising representatives of each approach on the test set. First, the reflection program while only a few lines longer than the others a clear winner, though CoT is quite effective with ensemble. Second, the bootstrap compilation procedure leads large gains for every program, across both LMs. Interestingly, vanilla is helped by compiling with bootstrap as the teacher program (bootstrapx2). On inspecting the prompts bootstrapped (Appendix 1), we see that the LM leverages the metric (i.e., correctness of the final numerical value in the output) so it uses the answer field for reasoning first. Third, while the human reasoning chains (+human_CoT) provide large boost for fewshot, we can match or surpass this using bootstrap, which substantiates our hypothesis that DSPy can cut the need for hand-crafted prompts."
Results
7 CASE STUDY: COMPLEX QUESTION ANSWERING
In this case study, we explore the multi-hop question answering task with the HotPotQA (Yang et al., 2018) dataset in the open-domain "fullwiki" setting. For retrieval, we use search index of the of- ficial Wikipedia 2017 "abstracts" dump of HotPotQA. Search is conducted by a ColBERTv2 (San- thanam et al., 2021) retriever. The HotPorQA test set hidden, so we reserve the official validation set for our testing, and sample 1000 examples for that. We sub-divide the training set into 70%/30% train/validation splits. In the training (and thus validation) split, we keep only examples marked as "hard" in the original dataset, which matches the designation of the official validation and test sets. For training and for reporting development results, we sample 200 and 300 examples respectively.
Programs Considered Our simplest baseline is the vanilla program used in the previous case study on GSM8K (Section 6); the "question -> answer" signature is universal enough that it will work for many tasks when compiled appropriately. Our baseline RAG program the one given in layer. This program does not excel at HotPotQA, and this
Section 3.2 with dspy. ChainofThougee motivates us to evaluate two multi-hop programs. We first test ReAct (Yao al., 2022), a multi-step agent for tool use, which is implemented as built-in module in DSPy. In the simplest case, a ReAct module for a particular signature can be declared as follows in DSPy:
react dspy. ReAct( ("question >> answer", cools=[dspy Retrieve (k=1)J. max_ steps=5)
And we test the following custom program, which is akin to Baleen (Khattab et al., 2021a), IRRR (Qi et al., 2020), and has similarities IRCoT (Trivedi al., 2022):
BasicHulciHog (dspy. vodule):
def inniint (self,
hop):: selF retrieve dapy Retrieve self zenerate. self scneerate_answer = dspy, Cheinch ( contteet question >> answer
per hop) dspy. dspy.Chaino Thouoo c"context question -> search, search_ query")
der farnard(salr, question):
context =
10
for hop ra)ge(2):
11
quary self. generate_ (oraaaoooooccttttttttttttte queatiorsquestionn search_querry context relf.retrieve (querr)) (query).
14
return www.gattn.asseeeeeeca (context=context, question=question)
15
16 nultihop BasicMultiHop (passulatttooppsssssssssss
We can informally compare these with published results for GSM8K. Zhang el al. (2022) reports 48% for text-davinci-002, close to our results using llama2-13b, and 63% with codex using COT. Zhang et al. (2022) report 57% for CoT prompting with PaLM 540-B, which becomes 74% upon adding self-consistency. Llama2 (Touvron et al., 2023) presented 28.7%, 42.2%, and 56.8% for llama2- 13b, llama2-34b. 34b, and llama2- 70b. Our implementation with llama2-133 13b is competitive with their llama2-344 34b results, while we don't 't use human reasoning chains in our program. Zhao et al. (2023b) reports that CoT scores around 80% for g.5- turbo from April 2023. The GPT-4 paper (OpenAI, 2023) reports that GPT-3.5 scores 57% and GPT-4 elevates this 92% but they note that GPT-4 was trained on subset of GSM8K.
8

---

Published as a conference paper at ICLR 2024
Table 2: Results with in-context learning HotPoIQA. We report answer exact match (Ans) and pair-retrieval accuracy (Psg). Each row represents separate pipeline: the module in the Program column is compiled against the examples the Training set. The programs, compilers, and (small) training sets are defined in the main text. For HotPotQA, we use the training set (and not dev) directly for cross-valication. "The marked result evaluated on 50% of our test set due to cost.
GPT-3.5
Llama2-1 13b-chat
Program
Compiler
Dev
Test
Dev
Test
Ans
Psg
Ans
Psg
Ans
Psg
Ans
Psg
vanilla
fewshot
34.3
n/a
31.5
n/a
27.5
n/a
21.8
n/a
fewshot
36.4
36,0
29.8
34.4
34.5
36.0
28.0
34,4
CoT_RAG
bootstrap
42.3
36.0
38.3
36.0
32.9
34.4
none
20.3
20.0
+human_r
33.0
28.3
react
bootstrap
31.0
24.7
bootstrup
39.0
40.0
fewshot
36.9
38.3
31.2
40.8
34.7
32.0
31.3
30.8
mul tiop
bootstrap
48.7
47.0
39.6
43.8
42.0
48.3
36.4
43.5
ensemble
54.7
45.6"
50.0
41.0
Compiling We continue to use the compilers used for GSM8K (see Section 6). We also consider two compositions of our teleprompters. For ReAct, we consider applying pootstrapFewShore thRandomSearch starting from an earlier bootstrap of the ReAct program. For the simple mult program, we consider fine-tuning starting from its earlier bootstrap.
mul multthhp__ts t5 dspy. Bootstrap inneFiinnttteee (metric=answer _ea_____aa_ataaeeeeeeee compile (progran, teacher: trainsetmtreinset. terget=' "erget="t--Larg""
Results Table 2 summarizes our results: a simple multihop program is the best across all models and in general bootstrap again proves to be very effective. Perhaps most importantly, we can make Llama2-13b-chat competitive with GPT-3.5 using the multihop program. We also evaluated the compiler mul tihop_t5 defined above which produces a T5-Large (770M parameter) model. This program scores 39.3% answer EM and 46% passage accuracy on the dev set, using only 200 labeled inputs and 800 unlabeled questions otherwise, with a teacher program consisting of an ensemble (union) of two Llama2- 1---chahhhht multihop programs. Such program would impose orders of mag- nitude lower costs for inference than a proprietary LM like GPT-3.5.
8 CONCLUSION
This paper introduced DSPy, a new programming model for designing Al systems using pipelines of pretrained LMs and other tools. We presented three new concepts introduced in this abstraction (DSPy signatures, modules, and teleprompters), and showed in two very different case studies that supports rapid development of highly effective systems that use relatively small LMs. We have maintained open-source versions of this framework throughout 2023 and since. In this period, have seen and created a large number of programs that were compiled to high-quality systems by DSPy, spanning tasks from information extraction low-resource synthetic data generation. In the interest of space and to maintain reasonable scope in this paper, we leave reporting on such tasks under controlled experimental conditions to future work. While in-context learning has proved transformative, we argue that the true expressive power in this emerging paradigm is in building so- phisticated text transformation graphs in which composable modules and optimizers (teleprompters) come together to leverage LMs in more systematic and reliable ways.
*Our results may be pegged against the evaluation HotPoiQA in number of recent papers, though there can be significant variation in evaluation methodology and test samples across studies in this space. et al. (2022) achieve 25.2% EM with CoT prompting. With precite-ad--ansee 62B (Chowdhery et al., 2022). Sun et al. (2022) achieve 26.5% EM. Wang et al. (2022a) achieve 33,8% EM and 44.6 Fl when applying self- consistency for BaLB. Yao et al. (2022) achieve 35.1% EM using the ReAct agent, with tool giving it the ability for search using Wikipedia API. Trivedi et al. (2022) reports 49% using pipeline with code-davinci-002 on sample of 500 HotPotQA questions.
technique PaLM-

---

Published as a conference paper at ICLR 2024
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna:
A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-62222222222 2631, 2019.
Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, Alexander Belopolsky, et al. Theano: Python framework for fast computation of mathematical expressions. arXiv e-prints,
pp. arXiv-1605, 2016.
James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: A CPU and GPU math compiler in Python. In Proc. 9th python science conf, volume 1, pp. 3-10, 2010.
James Bergstra, Frédéric Bastien, Olivier Breuleux, Pascal Lamblin, Razvan Pascanu, Olivier De- lalleau, Guillaume Desjardins, David Warde-Farley, Ian Goodfellow, Arnaud Bergeron, al. Theano: Deep learning gpus with Python. In NIPS 2011, BigLearning Workshop, Granada, Spain, volume 3. Citeseer, 2011.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds dimensions for vision architectures. In International conference on machine learning, pp. 115-123. PMLR, 2013.
Luca Beurer-Kellner, Mare Fischer, and Martin Vechev. Prompting is programming: A query lan- guage for large language models. Proceedings of the ACM on Programming Languages, 7(PDDI): 1969, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:210..072 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Harrison Chase. langchain.
Hwchase 1 1//aancchii..
2022.
URL https:///ithiie https://ggithub..om///ctssss com/hwchase11/
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18663/1//11--1171 URL https: //aclanthology.org/Phh org/P17-1171.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10.14460 14168, 2021.
Ronan Collobert, Samy Bengio, and Johnny Mariéthoz. Torch: a modular machine learning software library. Technical report, Idiap, 2002.
10

---

Published as a conference paper at ICLR 2024
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Dickkteekt et al. Language model cascades. arXiv preprint arXiv:2207.103:: 2022.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. RaIr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pp. 16477-16508, 2023a.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and
Graham Neubig. Pal: Program-aided language models. International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023b.
Qingyan Guo, Rui Wang. Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujuu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint 12Xiv:2309008533 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-We Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint SXXv:2002.08909 2020. URL https: //arxiv. orgo hbtg//bb///// 2002. 08909.
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1884- 1895. Association for Computational Linguistics, 2018. URL wttttttp///c///e org/ hwwggg/nnhoogyy/// P18-1175.
Bin Hu, Chenyang Zhao, Pu Zhang, Zihao Zhou, Yuanhang Yang, Zenglin Xu, and Bin Liu. En- abling intelligent interactions between an agent and an I.I.: reinforcement learning approach. arXiv preprint arXiv:2306.03600 2023.
wwwpaawwwwwoogoab///aebeea/ 03604.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprini urXiv:2210. 1610, 2022.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208. 03299, 2022.
Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Peyton-Brown et al. Mrki systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete rea- soning. arXiv preprint 22Xiv:2205.00445. 2022.
Omar Khattab, Christopher Potts, and Matei Zaharia. Baleen: Robust Multi-Hop Reasoning Scale via Condensed Retrieval. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021a.
Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guidde supervision for openqa with ColBERT. Transactions of the Association for Computational Linguistics, 9:929-944 202 lb.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search -eeerchh--rdit:::: Composing retrieval and language models for kmowledge-intensiin nip. arXiv preprint arXiv:2212. 14024, 2022.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02400 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint 120.::2205..1116 2022.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05/111 2022.
11

---

Published as a conference paper at ICLR 2024
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensing tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 9474. Curran Asso- ciates, Inc., 2020, URLhttpar httpttt:::////ceeeeeeee //prcceednoocccsssoooooo (e1e322205578e1112225566650
nip
Jerry Liu. Llamalndex, 11 2022. URL https: thub com http///errrllll//llaaa... index.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with felf-feee arXiv preprint arXiv:2303. 2023.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv:1806.0877 2018. URL https:
waaxxii.org/abs///// 08733.
Microsoft. Semantic kemel. 2023. URL
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schul- man. WebGPT: Browser-assisted question-answering with human feedback, 2021. URL https:
211100933.
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint 2203:::0030022222 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, perforreffe deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Alché-Bucc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: / http://prrceeeeinneeie nwwrips...... CC/ fwtrrfffllespeeeeeeeeee file/bdbca228fee7/ (tt.:/a8000772222222277111122222
Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to- sqi with self-correction. arXiv preprint arXiv:2304. 1015, 2023.
Ofrr Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210003350, 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03499. 2023.
Peng Qi. Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning, Answering complex open-domain questions through iterative query generation. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCNLPP), pp. 2590-2602. Hong Kong, China, 2019, Association for Computational Linguistics. doi: 10. 10.18653/1/D11--1226 URL https://
Peng Qi, Haejun Lee, Oghenetegiri Sido, Christopher D Manning, et al. Retrieve, rerank, read, then iterate: Answering open-domain questions of arbitrary complexity from text. arXiv preprint 12527.
arXiv:2010. 12527, 2020. URL
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language derstanding by generative pre-training. Ms, OpenAI, 2018. URL https://opentt wwt.ppeenai..caem..aee.mmmmeeeee com/blog/
uunguag--usupervissee
Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems 29, pp. 3567-3575. Curran Associates, Inc., 2016. URL https: /ppppppppsssa//////////ee/eees.p htppscccpppper. 6523-data pnnittapporramminnn weeeiinnn-eaannn-i-ee largee -aage-raiiingngg------rarrrn sutiiiikk-sss-uuicl......
Keshav Santhanam, Omar Khattab, Jon
Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv preprint arXiv:2112.01488, 2021.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Syn- thetic prompting: Generating chain-of-thouuht demonstrations for large language models. arXiv preprint 2tt.::223020000118, 2023.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint urXiv:2303.11353 2023.
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and
Lijuan Wang. Prompting gpt-3 be reliable. arXiv preprint arXiv:2210.09/55, 2022.
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitatioo--augmented language models. arXiv preprint arXiv:2210.01296. 2022.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: next-generation open source framework for deep learning. Proceedings of workshop on machine learning systems (Learn- ingSys) in the twenty-ninth annual conference neural information processing systems (NIPS), volume 5, pp. 1-6, 2015.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving re- trieval with chain-of-thougle reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212. 10509, 2022.
Fei Wang, James Decker, Xilun Wu, Gregory Essertel, and Tiark Rompf. Backpropaga- tion with callbacks: Foundations for efficient and expressive differentiable programming. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URLhttps:/ www.iis.....ss....e..... httpa///....ippp...aaaeaeeeeee. http///pr...lleeeeer//////////// pdf.
Xuezhi Wang, Jason Wei, Dale Schaurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale- augmented ensembles in language models. arXiv preprint arXiv:2207.0m 0074777777... 2022a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXi::2203.11177, 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 2022.
Thomas Wolf, Lysandre Debut, Victor Sanb, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Sf the natural language processing. In Proceedings of the 2020 Conference Empirical Methods
13

---

Published as a conference paper at ICLR 2024
in Natural Language Processing: System Demonstrations, pp. 38 48-45, Online, 2020. Associ- ation for Computational Linguistics. doi: 10.18653112222.eempdddeeoo //aclanthology.oo org/2020.... emnlp-demos.6.
URL https:
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.0344 03409, 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 1ttiv:8809.00900 09600, 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. Answering questions by meta-reasoning over multiple chains of thought. arXiv preprint arXiv:2304.13000 2023.
Eric Zelikman, Yuhuai Wu, and Noah D Goodman. Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv: /4465, 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. arXiv preprint arXiv:2308, 10144, 2023a. URL https:
//arxiv.org//pee 2308. 10144.
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning, arXiv preprint arXiv:2305. 14333, 2023b.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, Clark Barrett, and Ying Sheng. Efficiently programming large language models using sglang, 2023.
14

---

Published as a conference paper at ICLR 2024
A LATENCY & THROUGHPUT
In the main paper, we introduced the DSPy programming model and presented quality-oriente case studies. In this appendix. we report on efficiency-oriente metrics. We select one of the most sophisticated programs we discussed, namely, the mul tihop program from Sec 7.
We run self-contained experiment from scratch to sidestep the effect of caching. We compile the mihti program using the latest OpenAI GPT-3.5 model at the time of writing this appendix, i.e. gpt-3.5- 10..5--uro-11111 1006. We compile using the trotstrapFewShot thRandomSear ch teleprompter with 7 candidate programs. DSPy supports parallel evaluation. In these runs, we set the maximum number of parallel threads to 10.
Compiling this program takes approximately 370 seconds (or 6 minutes), which is a little shorter than one minute per candidate program. The number of candidates for random search is a simple configurable parameter, whose typical range is 5 through 30. For OpenAI models, periments use 13 candidate, though we observe strong results with as little as 7. This compilation process makes approximately 3200 API calls to the language model, with total of 2.7M input to- kens and 156k output tokens. With the current pricing of OpenAl models, this costs a total of $3.0 USD to compile.
main ex-
For inference, single-threaded inference with the mul tihop program takes on the order of 2-3 sec- onds per question, which includes three chain of thought prompts (i.e., two for query generation and one for answer generation) and two retrieval queries (i... each retrieving three passages). In the presence of multiple questions, multi-threading again possible, and observe that we can process more than 150 questions per minute with 10 threads. The exact maximum depends on the rate limits supported by the retriever and the LM server.
To provide comparison of inference costs, we took four rows of increased complexity for Hot- PotQA (Table 2) and tested them with gpt-3.5-1 while disabling the cache (which would oth- erwise skip any repeated computations, e.g. retrieval queries or LM calls). We ran 100 questions from HotPoiQA with a single thread, and we report the average latency below. Each numbered bul- let below represents program (with compiler strategy in parentheses). For Vanilla (fewshot), we observe average latency of 0.3 seconds per question and average Cost of $0.0005 per question (1x). For CoT RRAG (fewshot), we observe average latency of 1.1 seconds per question and average cost of $0.0013 per question (2.6x). For Multihop (fewshot). we observe average latency of 2.6 seconds per question and average cost of $0.0018 per question (3.6x). For Multihop (bootstrap), we observe average latency of 2.6 seconds per question and average cost of $0.0041 per question (8.2x). As this shows, these programs are within an order of magnitude of the cost and latency of the simplest one, even though we see major quality improvements from vanilla to multihop.
Latency and throughput results for g3t-3.5-turbos S-turbo-111 1106 depend on the OpenAl API load and the rate limit permitted. For local models at the scale of 13B parameters, we typically see higher laten- cies and lower throughputs in practice, when serving the model from one or few A100 GPUs. We suspect this gap will be bridged with recent open research advancements in serving open models, especially approaches that apply intelligent prefix caching for prompts that share the same prefixes, e.g. Zheng et al. (2023).
B ADVANCED SIGNATURES
When more control is desired, one can express signatures Python classes to provide explicit instructions of the transformation and describe the format or role of each field more directly. For instance, the following signature generates search queries using context and an optional question:
class GenerateSearchhuurry (dspy Selssy.Siige Signature):
"**Mrite simple search query that will help answer complex question. ...
context dspy. www.LnnuuFFieldddee==aaaa contain relevant facts*)
question = dspy. InputField() query dspy. Output certputfieed(prefiiiii ield(prefix=""Sarre Query:")
Using the above, we can specify a complete system for the generation of synthetic IR dataset where the queries are mediated by question generated by the LM:
15

---

Published as a conference paper at ICLR 2024
query_gen Sresy.Predictee (reddict(eenrrtteeeerchhorrr
query-sen(context (context==="aaannuu Languuge typology")
out: Prediction (question=" "veryr""angguag classification" OR "language typology" -wikipedia")
the nain types of language classification"",
If questions are available, they can be supplied as shown: query-sen(contex==llnggggu typology", question="What are the primary language fami limi of South America?").
C MORE ADVANCED METRICS
If one wanted to push the compiled program to be extractive given its retrieved contexts, one could define a custom metric to use in place of dspy. evaluate. -ewpeeeeeect_maaae
def ny my_.rag_validation logic (example, pred, trace=None):
unswer__ater dspy.evalueee evaluate, enswer_ wtt__eett__accc(eeanp... pred)
Is the prediction substring some passage?
context_natch any ((pred. answer. lnwwer.lower () in c) for in context)
return answer_natch and context_match
D COMPARISON WITH EXISTING LIBRARIES LIKE LANGCHAIN AND LLAMAINDEX
LangChain and Llamalndex are perhaps the most popular library in the general space of prompting LMs. These libraries have different focus compared to DSPy and they suffer internally from the prompt engineering challenges that DSPy aims to resolve. In particular, whereas the goal of DSPy to tackle the fundamental challenges of prompt engineering for building new LM computational graphs, LangChain and LlamaIndex generally help application developers who need pre- rre-packaedd components and chains, e.g., implementations of popular and reusable pipelines (e.g.. particular agents and specific retrieval pipelines) and tools (e.g., connections to various databases and imple- mentations of long- and short-term memory for agents).
These off-the-shelf higher-level abstractions contrast with DSPy's focus on introducing core com- posable operators. In particular, DSPy introduces signatures (to abstract prompts), modules (to abstract prompting techniques), and teleprompters to act as optimizers for arbitrary imperative code (DSPy programs) that chain modules together. Its goal to help researchers and practitioners build new LM pipelines quickly and achieve very high quality through automatic compilation (self- improvement) instead of manual prompt engineering.
In contrast, typical existing research implementations and existing libraries like LangChain and LlamaIndex are implemented using manual prompt engineering, which is the key problem that DSPy tackles. We conducted an informal study to highlight this. In late September 2023, we found that the LangChain codebase contains 50 strings exceeding 1000 characters, which are generally prompts, compared to none at all in DSPy. Indeed, substantial number of LangChain's 's Python files are singularly dedicated to task-related templating and prompt engineering with 12 prompts. py files and and 42 prompt. py files. DSPy, on the other hand, provides a structured framework that automatically bootstraps prompts. The library itself does not contain single Hand-written prompt demonstration for any tasks at the time of writing, despite the very high quality with various LMs.
To review the typical forms of prompt engineering in existing libraries, we consider the follow- ing in LangChain. The LangChain Program-Aided Language Model Gao et (2023a) chain pro- gram uses few-shot learning, leveraging template that is 3982 characters long with 8 math word problems (Prompt 2) and corresponding outputted programs as learning examples for the language model. LangChain also contains a prompt for SQL query tasks for each of the databases like Or- acle, GoogleSQL, DuckDB, Crate, and MySQL, with the average length of these prompts at 1058 characters. Other task areas such QA with sources (Prompt D) and Graph. QA also have signif- icantly lengthy prompt templates, with averages of 1337 and 722 characters, respectively. While expert-written prompts can be useful, believe that LM- and task-adaptive prompts bootstrapped automatically can offer far more power (and are far more modular) than hard-coding prompt per database provider inside the code base. The next appendix section contains number of prompts copied from related research papers and existing libraries.
16

---

Published as a conference paper at ICLR 2024
E SAMPLE LARGE PROMPTS
This section highlights few popular existing frameworks that structure prompts with extensive prompt engineering templates. The primary objective capture how many words and characters are used for such large multi-line prompts defined for tasks or tools and present these example prompts retrieved from open-sourced papers and repositories. The formatting of these example prompts is adapted from Gao et al. (2023a).
Task/Tool Prompt
Source
Words
Characters
Prompt I:
Text-evidence checker
Gao et al. (2023a)
818
4964
Prompt Math word problems (PAL.)
LangChain Gao al. (2023b)
566
3957
Prompt 3: ReAct
Yao et al. (2022)
593
3889
Prompt 4: Zero-shot ReAct
LangChain
101
600)
Prompt QA with sources
LangChain
992
6197
Prompt 6: SQL MyScale querying
LangChain
343
2239
Prompt Relevant docs retrieval
LlamaIndex
129
719
Prompt 8: IRS chatbot
LlamaIndex
389
2258
17

---

Published as a conference paper at ICLR 2024
1 [web] will check sore things you sald.
3. (1) You said: Your nose skisches back and forth between nustrils, when you sleep, you switch about every 45 minutes. This
is to prevent builoup nucus. It's called the nasal cycle.
4 checked: How orten do your nastrils switch?
5 Found this article: Although we don't usually notice it, daring the rasal eyele one nestrill becoses congested and thus contributes less sirflow, while the other becomes decongested, average, the congestion pattern swi tchee about every hours, according to a small 2815 study published in the Journal PLOS One.
Your nese's swi tchiin
every hours, not vinutes.
7 This disagrees with what you said.
9 (2) You said: The Little House books were written Laura Ingalls Wilder. The books were published by HarperCallins. checked: Who ished the Little House books?
found this article: These the books that started it all the stories that captured the hearts and iraginations children and young adil worlowide. tri by Laura IrgalTs Milder and publ by Harpertall these beloved backs ressin favorite
day.
12 The Little House books ware published by Harpencollins.
This agroes with what you sald.
14
(3) You said: The Stanford Prison Experiment was conducted the basement Jordon Hall, Stanford's s paychology building. checked: Where was Stanford Prison Experinent conducted?
found this articte: Carried out August 15-21, 1971 in the basement of Jorden Hall, the Stanford Prison Experizent set to exanine the psychological effects of auchority and powerlessness prison enviromeett 18 The Stanford Prison Experiment was conducted in Jordan Hall.
17
This agroas with what you sald.
28
21 (4) You said: Social work is profession that is based in the philosophical tradition of hunaniss, It is an intellectusl discipline that has its roots in 1800s.
22
checked: When did social work have its roots?
this articles The Energence Growth the work Professiontar>st>> Social work's roots were planted the 1880s, when charity organization societies (COS) were created to organize municipal voluntary reltef associations and settlerent houses were established.
Social work has its roots in the 1888s, not 18905.
25
This disagrees with what you said.
27 (5) You said: The llavel-iakini algorithe is an algorithr for converting the adjacency natrix of a graph into its adjacency list. It is nared after Yoclay Havel Santh Hakini. checked: What is the Havel-Hakint algorithe?
28
found this article: The Havel -Hak algoriche constructs special solution steple grach for the given dagree sequence exists,
that cannot find positive answer, This construction is based on recurssve algoritin. The algorichn was published by Havel (1955), and later by Hakini (1962).
Havel-takint algorithe for constructing special solution simple for the given degree sequence exists, proving that one cannot Find pasi tive arswar, not converting the adfacency natrix of graph into Its adfacency Hist. This disagrees with what you said.
32
(6) said: "Tive My Life" is song by American singer-sergariter Bill Hedley Fron the soundtrack of the 1987 film produced by Michael Lleyd.
Dirty Bancing, The
34
checked: Who was the producer of "(I've Had) The Time of My Life"?
found this article: Septomber 2810, the original deso of this song, with ranis producar Michael Llayd, was released as digital Fillss in an effort to raise roney for the Patrick Swayze Pancreas Cancer Resarch Foundationn at Stanford University.
36 "Time of My I lifa" was produced by Hichael I Tayd.
37
This agrees with what you said.
39 (7) You said: Kelvin Hopins was suspended fron the Labor Party because he had allegedly sexually harassed and behaved fnsoprapriated iately
abour Party activist, Ava Ftonadradet.
48
I checked: Why was Kelvin Hepins suspeneded from the Labor Party?
found this articla: former Labour MP left the party bafore an inquiry into sexual harassment allegations against his was able to be concluded, the party has confirmed, Kelvin Hopkins accused in 2017 of inappropriate physical contact and was suspended by the Labour party pending an investigation, This agrees with what you said, Kelvin Hopins was suspended because had allegedly sexually harassed and behaved inaorrpriately towards Labour Party activist, Ava Etenadzadeh. This agrees
you
64
(R) swid: the hatcles of texington and Concord, the Britishh siide was led by General Thoras Sai th. checked: Who led the British side in the battle of Lexington and Comcord?
46
67
found this article: Interesting faats about the Battles of Lexington and Concord. The British were led by Lieutenant British reguiars.
Colonel Francis Seith. There
48
The British side was led by Leuterant Colonel Francis Sei th, not General Thonas Hall. you said.
This disegrees
58
51 (9) swid: (texe)
checked: (suery)
53 found this article: (evidence)
Figure 1: Example few-shot prompt using reasoning chain for agreement model that identifies inconsistencies between text and evidence (Gao et al., 2023a).
18

---

Published as a conference paper at ICLR 2024
Q: Olivia has $23. She bought rave bagels for $3 each. How auch money does she have Left?
solution in Python:
solution():
**'Olivia has $23. She bought five bagels For each, How nuch does she have left? money innyylnltiall
bagels
18
bagel.cost
noney_spent bagels bageiccest
noney.left noney.initiall neney_spent
reault noney,Jeff left
result
17
Q: Hichael had golf balls, On tuesday, lost 23 golf balls, vednesday, he lost more. How many golf balls did have the end wednesday?
21
solution Pyshon:
23
25 def solution(::
'"Michhel had 58 golf balls. On tuesday, lost golf balls. On wednesdoy, he lost more. Howw many golf balls
he have the end wednesday? toe
27
golf haf.baall.initioll
solf.balls..osttteedddyy
29
golf.balls.bstteeddeee
golf halls 1aft golf halls Initial golf tuesday golf widnesaa result golf Jallssleftt
return result
34
36
Q: There nine conputers the Five
installed each fros ronday thursday.
How nany computers are now in the server room?
48
solution in Python:
42
solution()
""There nine computers the roos. Five installed each nonday thursday. How nany conputers are now in the server room?""s
corputora_initaal =:
46
cosputers_per.dyy
runudays
computers added computers per
conputers.total computers.initin result conputers.tota return result
computers.added
51
53
five Christnaas, solution Python:
from and does
68
def solutionO:
**Shawwn has five toys. For Christras, he got two toys each from his non and dad. How many toys does have nox?
toys.initial
65
morltoys
dad_toys
67
total received son. son.toys dod boys
total toys toys. initial total received result total.toys
return result
74
Q: had lollipops. iee Denny lollipups. Jason has lollipops. Hlow lollipops Jason Denny?
solution in Python:
19
88

---

Published as a conference paper at ICLR 2024
solutionO::
Jason had 28 lollipops. He gave Denny sone lollipops. Now Jason has 12 lollipops. How many lellipops did Jason give Denny?**
JasenJolllpppssinntii
boson.olllpoop.aai
denny popss
1al1ipops fnitial jason Toll after
result celmy1olllippss
result
11
16
Q: Leah had 12 chocolates and her sister had 42, If they' ate 35, how many pieces do they have left in total?
solution in Python:
solution():
21
***aesh had 32 chocolates and her sister had 42, TF they ate 35, how many places do they have left in total?"*
Teah. chocochocoltts
23
sister.chocolates 12
totaL_hhcolates Jeah-chocolates sister.chocolstes
25
chocolates_eater _atan 35
chocolated, lefe total Lchooolates chocolates.ssten
result chocolates.leff
return resule
38
Q: 1f there the lot and arrive, the parking lot?
35
Python:
37
def solution():
""If there are 3 cars in the parking lot and 2 more cars arrive, how manyy cars are in parking lot
initial
42
cors.arrived
total.car carsanntiaal cors.arrived
return 44
result total.cars
69
51
0: There are 15 trees in the greve. Grove workars will plant trees in the today. After they are done, there will
trees, many trees grove workers plant today?
solution in Python:
55
solution():
57
Thhreee are 15 trees in the grove, Grove workers wil1 plant trees in the grove today. After they are done, there will
be 21 trees, Fox mony trees grove workers plant today?" trees.initial 15
59
trees aftter 21
troes_initial
61
result trees_added
return
68
Q:
(question)
solution in Python:
Figure 2: PAL example few-shot prompt for solving math questions by generating code.
20

---

Published as a conference paper at ICLR 2024
Salve question answaring task with interleaving Thought, Action, Observation steps. Thought can reason about the currant
situation, and Actior can three types:
(1) Searchfenticy). which searches the exact entity Vikipedia and returns the first paragrash exists. If
<lll return sore sinilar entities to search.
Lockuplkeyeord), which
containing keyword the passage.
(3) Finish[answer), which returns the anserr and finishes the task.
5 Here sone examples.
6 Question: What is the elevation range for the area that the eastern sector of the Coloradd orogany extends Into?
Action 1: Searchtoolorado orageny]
Observation The Colorado
episode nountain building (an orogeny) in Colorado and surrounding
9 Action 2: Lookupleastern sector]
Observation (Rasult
extends High Plains and called Central Plains orogeny.
11 Action 3: SearchEHlggh Plaina]
Oisarvation High Plains refers
distinct regions:
Action 4: SearchCHigh Plains (United Statee)3
Observation 4: The lligh Plains are subregion the Great Plains, From east to west, the lligh Plains rise in elevation from around 1,880 7,930 ft (550 to 2.138 ().(3]
Action 5: Finish[1, 380 to 7, 7,888 [t] Musician and satiriat Allie maned after who?
song "The Simpsorss character Milhouse,
Action 1: SearchDMi
Observation 1: Milhouse Mussolini Houten is recurring character in the Fox animated television series The Simpsons voiced by Ponela laayen and created by Hott Groening. Action Lookuplnaned after]
Obsarvation 2: (Rasuli / Milhouse was nased after U.S. president Richard Nixon, whose niddle name was Milhous,
Action FinishDRichard Nixon]
22 Question: Which documentary is about f'innish rock groups, Adan Clayton Powell The Sainas Gesture? Action 1: Search[Adan Mayyon Powell]
Observation Could not find Didon Clayton Powelll. Similar: ['Adan Clayton Powell III', "Seventh Averue (Manhattan)" 'Adan Clayton Powell Ir. State office Building'. 'Isabel, Mashington Powell", 'Adan Powell', "Adon Clayton Powell (fila)'. 'Giancario Espesito") to"].
Action 2: SearchEAdan Elayton Powel1 (Fiin))
docurentary directed by Richard Kilberg. filn is the rise and fall of influential African-trerrcan politician Adon Clayton Powell [r][3][4] It was later aired as part of the series The American Experience.
Clayton Powell
Action 3: Tinissh[Tee Sainaa Gesture]
question: What profession does Nicholas Ray and Elia Kazan have in common?
Action is SearchDVicholas Ray]
Observation to Ni chcholas Bay (born Raysond Nicholas Kienzle Jr., August 7, 1911 { June 16, 1979) was an American Film director, screeneriter, and
for
filn Mithout Cause.
31 Action 2: Search(Elia Kazan]
Oasarvation
Anerican film and theatre director, producer, screarwrlte actor. Action rinish(dirrector screemeriter. actor]
Kazan
Question: Which magazine was started first Arthur's Hagazine or First for Wonen? SearchEerthur's
36
Observation 1: Arthur's Nagazine (1846-1846) was Anerican literary periodical published in Philadelphia in the 19th century.
37 Action 2: Search[First for Noren]
Obsarvation First Wonen
nagazine published by Bauer Group the USA.C13 213 The magazine was started
1989.
39
Action 3: Fintsh[Arthur's Hagazine] Pavel Drysehn Lennid
type work??
41 Action 1: Search[Pavel Urysohn]
Urysonn (February 3, August 1824) was Soviet nathonatician known for his contributions In dimension theory.
Observation Pavel
Action 2: Searchfi eonid [evin]
Observation Leantd Anatollevich Levin Soviet-Aeerican nathematician and computer scientist.
Action 3: Finish[yes]
Figure ReAct example prompt for interleaving Thought, Action, Observation steps.
21

---

Published as a conference paper at ICLR 2024
Answer the folloxing questions as best you can. You have access to the following tools: need
useful for
questions
3. Use the following Format:
the input
5 Thought: you should always think about what to do
Action: action take, should
[Search]
Action Input: the input the action
Observation: the result of the action
Thought/Actinn/cctionn Input/3bservatiio repeat tires)
18 Thought: now know the Final answer
final
original input question
12 Degin!
Question:
(question)
Thought:
Figure 4: Langchain ReAct example prompt for interleaving Thought, Action, Observation steps.
22

---

Published as a conference paper at ICLR 2024
Given following extracted parts of long docusent and question, create final answer references ("SOURCFS).
2. If don't know the just that you don't know. Don't to make
3. Al WAYS return a "SOURCES" part in your ansser.
QUESTION: Which state/country's law governs the interpretation of the contract?
This Agressent is governed by Fnglish law and the partles subeit to the exclusive jurtsdiction of the Figlish courts in relation to any dispute (contractual or non-contractual) concerning this Agreesent save that either party may apply to cours for injunetion other relief protect its Intellectual Property Rights.
Concent:
6- Source: 26-p1
Concent: No Naiver. Faallure or delay exercising any right or resedy under this Agresment shall not constitute waiver of such (or any other) right or renedy.
11.7 Severability. The inalliity, illogality wenforcsability of any tern (or part of tern) this Agreament shall not affect the continuation in force of the rissalnder of the tern (if any) and this agremeent.
11 11.5 No Agency, Excent as expressly stated otherwise, nothing in this Agreerent shall create an agency, partnership or
Joint venture of any kind between parties.
12 11.9 hi Third-Party beneficiaries.
Source:: 38-p1
14 Concent: (b) ir Google believes, in good raith, that the Distributor has violated or caused Google to violate any
Anti-Bribery Lass (as dafined in Clause 8.5) such violation ta reasonably likely to acour,
15
Source: 4xpl
FINAL ANSKER: This Agreement is governed English law.
18
SOURCES: 25-p1
20
QUESTION: What did the prosident about Michaol Jackson?
Content: Modax Speaker, Madan Vice President, First Lady and Second Gentlensnn Members of Congress and the Cabinet. Justices of the Suprere Court. Ny fellow Arericans.
Last year COVID-13 Tanight, we neet as Besocrata Republ ieans and Irdepandents. Rut rest inportantly as Americans. American people the Constitution.
spart. This year are finally togother
24
With duty to one another
26 And with an unwavriing resolve that freedon will always triugh over tyranny.
six days ago, Russia's Vladiwir Putin sought shake the foundations the free world thinking could make it bend his meraccng ways, But bodly niscalculated.
28 He thought he could roll into Ukraine and the world would roll over. Instead he net wall of strength he never imagined.
He Ukrainian people.
38
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their deternination, inspires the world. Groups of citizens blocking tanks their bodies. Everyone fros students retirees teachers turned soldiers defending their horeland.
Source: 8-pl
Concent: And wan't
34
Ne have lost so nuch to COVID-19. Time with one another. And worst of all, so ruch loss of life. stop tooking coviD-ta partisan dividing line it for what 1s:
Lat's this
God-owful disease.
stop seeing other as enentas, and start seeting each other who really are: Follow Anericans.
Ne can't change how divided 've been. But we change how nove forwardjon COVID-19 and other issues rust Face together.
recently visited the New York Police Department days after the funerals officer Kilbert Mora and his partner. orficor Jason Rivera.
They responding
9-1-1 call when officer Nora sas 27 years old.
and kiTTed thex with stolen gun.
48
officer Rivera was 27.
up the streets they later chose pstrol as police officers. 43 spoke with their fanilies and told thei that we are forever in debt for their secrifice, and we will carry on their rission to restare the trust safety every community deserves.
Both Doninican Americans who'd
Source: 24-p1
Content: And proud Ukrainian people,
years of indeperdence, repeatedly that they will not tolerate anyone who tries to take their country backwards.
To all Americans, I will be honest with you, as I've always prosised. Russian dictator, invading a foreigs country, has around the world.
47 And I'n taking robust action to rake sure the pain of our sanctions is targeted at Russio's aconosy. And will use every tool Tanight, can arnuncee that the Uni States has worked with sb other countriea to release 68 Million barrela of afl from reserves around the world.
protect American businesses and consuners.
48
49
America will lead that effort, releasing 30 Million barrels from our own Strategic Petroleun Reserve. And we stand ready to do vore if necessary, fnified with our allies. will help blunt prices here hore. And know about what's happening alarming. 51 But want you to know that we are geing to be okay.
These
Source: 5-pll
53
Content: Nore support for patients and fanilies.
Congress to fund ARPA-11, the rdvanced Research Projects Agency Health. 55 It's based on DARPA/the Defense Departsent project that Led to the Internet, GPS, and so much more.
get there,
ARPA-H will have singular purpose] to drive breakthroughs in cancer, Alzheiner's, "s, diabetes, and nore.
23

---

Published as a conference paper at ICLR 2024
agenda for the 2 Re can da this.
gathered In this Capitol, generation after generation, Anericans have debated great questions anid great strife, and have done
My Fellox seericans
citadel denocracy.
great things.
5 Ne have fought for frredon, expanded liberty, defeated totalitarianisn and terrer. strangeat, Freest, most prosperous nation
known.
hour.
5 Our norent of responsibilsty.
and conscience, history itself.
18 It is in this mosent that our character is formed. Our purpose is found. Our future is forged.
nation.
Source: 36-p1
ANSVER: The president
Michael Jackson.
SOURCES:
17
QUESTION:
(question)
cummaries
ANSWER:
Figure 5: Langchain example prompt for QA with sources.
You MyScale espert. Given an ieput question, first create syntactically correct MyScale query to run, then results the query and Myscale queries has a vector distence function called (eSluu array) to corpute relevance to the user's question array
the cuestion.
by
when the query is asking for (top.) clesest res. you have to use this cistance function to calculate distance to entity's array on vector calurn
by distance retrieve relevant rous.
-NTICF+:: DESTAMEF (calume, array) only accept an array colurn as its First argusent and NeuralArrayyeeeity as its second arganent. You also need a user defined function called (euralArrayemmtitt) Unless
retrieve the entity's array.
specifies in the question specific number of exarples ebtain, for most (top k) results using the LIMI clause per MyScale, You should only order according to the distarce function. query
query each column name in couble quotes (?) to denate ther celinited identifiers.
to answer question.
query colures Pay attention to use today( function to get the current date, if the question involves "today ' ORDER BY clouse should senicolon
Pay attention use only the column names you
below.
not exist. Also, pay attention to which coluen which table.
always after WHRR clause.
of SQL. attention
table
forrat:
table Info seccice
[table Info)
Question: (input)
SQuQuery:
16 llere some exapples:
18
CREATE TABLE "ChatPapar" ( abstract String,
id String.
21
vector Array(Float32))
ENGINE Regltaatefipaacnneeergttreet))))
23
ORDER BY id
25
Question: What is Feature Pyrasid Natxork??
SQuQuery: SELECT ChatPaper. title, ChatPaper. id, ChatPapen.
ChatPaper ORDER
DISTANCE (vector,
Meuraakkray(Ppperaakk contribution)) LIMET (top.k)
29
***.***e table info .*....*
(table.info)
Question: (Imput)
SQUQuery:
Figure Langchain example prompt for SQL. querying using MyScale.
24

---

Published as a conference paper at ICLR 2024
1ist decurents shown below. Each docusent has number next to it along with summary the docusent, question
is alsc proviced.
2. Respond with the numbers of the docunents you should consult to answer the question, in orcer of relevance, as vell as the relevance
3 The relevance score is a number from 1-18 based on how relevant you think the decusent is to the question. include any documents
question.
Example Format:
Document
8 <sunaary of documant
18
Document 2: csumary
12
15
Document 18: coumary
17
Question: cquestion>
19 Ansier::
Doc: 9, Relevance:
Relevance:
22
Doc: 7, Relevance:
24 Let's try this new:
[context str)
26
Question: (tuory.str)
27
Answer:
Figure 7: LlamaIndex example prompt for returning relevant documents and corresponding sum- maries.
You are an IRS, chatbot whose prinary goal is to help users with filling their tax returns for the 2022 year. polite professional.
Provide replies
Answer questions truthfully based on official government information, with consideration context provided belox changes
can affect refund.
Do questions that not related to United States procedures and respond with "I only help with any tax-related questions you nay have.".
If
saying
your question.
to Find your answer at was. irs. gow/fans"
Changes for 2022 that can affect tax refund:
Changes in nebbr of dependents, erployment velf-eeployrent incore divorce, asong other factors, ray affect tax-filing status and refund. siditional stirulus payments. Unlike 2929 and 2821, there payments for 2022 so taxpayers should not expect get an additional paysent. levels.
stirulus
significantly smaller rafund cospared with the previcus tax year. Changes include asonts for the Child Tax Credit (CTC), the Farned Incore Tax Credit pre-CoviD levels.
credits
taxpayers
likely
(EITC)
For 2822, the CTC is worth $2, 888 for each qualifying child. child must be under 17 at the and of 2872 be quilifying child. for the EITC, eligible taspayers with no children will pet $560 for the 2022 tax year. The Child and Deperdent credit
roximan $2.189
11 No above- the- charitable deductions. During COVID, tarpayers were able to take up to s688 charitable donation tax Howaver, for year 7022, taspayers who don' 't isenize son't be able to deduct their charitable contributions. Presium Tax aligibility for the presium tax credit.
deduction
standard deductian,
year 2022, taxpayers may qualify tempororily expanded Eligibility rules changed to clais tax credit for clean vehicles. Revien the changes under the Inflation Reduction Act 2022 qualify for
More people ray
Vehicle Credit.
Figure 8: LlamaIndex example prompt for IRS chatbot guidelines.
25

---

Published as a conference paper at ICLR 2024
F SOFTWARE DEPENDENCIES
DSPy has the following core software requirements:
Python: Version 3.9 or higher.
HuggingFace Text Generation Inference server: Used for hosting language models from the HuggingFace transformers library.
OpenAl connectivity: Required for accessing GPT models.
C retrieval server: Serves as the default search index retriever.
For optimal performance:
Access to GPUs is recommended for hosting the language and retrieval models.
DSPy experiments specifically relied NVIDIA A100-SXM GPUs with 80 GiBs mem- ory.
For model fine-tuning, the following package versions used:
datasets-2. 14.5
Transformers-4.32..
peft-0.5.0
tr1-0.7.1
G MODULES
G.1 PREDICT
class (reay
def __init_ (self, signature, s*config):
self. signature dapy. (iigaature(signauurg
self.config config
Nodule Paraneters.
self. Im = dspy AarameterLM(Nong) use the default LM
self. demonstrations dspy, P (C))
10
def forward =*kwargs):
kwargs)
12
signature stt_ttte_riht__sinttuuu (self (self. signature. kwargs)
13
denonstrations htr_the__righ_ddccnnsrrriioss (sstt.dtntssttaiionn kwargs)
14
15
prompt signature(demos=seel/ denos, **kwargs) completians In. (prontt, **self.conff
16
17
prediction = Prediction. ottartteeeerreeeiooooooooooooooa ooonallcnllettosscccoleeetossss,
(completions, signature=siinature))
18
19
if dsp. sewtings.commilinng is not None:
20
trace (preditttett inputsskwargs, outputsseredicction) dspy settings traces. (tranc)
21
22
prediction
G.2 CHAIN OF THOUGHT
class ChainOfThought (dspy. Module): def __init__ (self, signature):
Nodify signature
'simputs koutputs 'kinputs rationale, *autputs
rationale_fiell dspy (eppuriielddreeiieeeeeeeesee signature dspy, mnyy..lgntturessggnee
Let's think step by step.
prepend_ outpptt,
Declare sub sub-nedule with the modi fied signature. self. predict
Predict signature)
10
11
def follf, **kwargs):
12
# Just forward the inputs to the sub-module. return self, self.predict predict (**kwargs)
13
26

---

Published as a conference paper at ICLR 2024
H TELEPROMPTERS
H.1 BOOTSTRAPFEWSHOT
def -_init__ (self, metric=None):
self metric metric
def compile(self, student, trainset, teacher=None): if teacher
teacher conpiled_pregran student, student,deepcoay()
student
# Step 1. Prepare nappings between student and teacher Predict modules.
10
& Note: other modules will rely on Predict internally.
11
assert nome2predictor, predictor2nane = prw.ppredictors_eeeriiivie (student, teacher)
teacher)
12
14
V Step 2. Bootstrap traces for each Predict medule. loap over training set. for example in trainset:
15
try example ence simplicity.
16
17
if we_ hew.ddeeeughhhttttteeeee hew.dd_eeushhbotttreeeeee ()) break
18
19
turn on compil ing mode which will allow us to keep track of the traces dapy. setting. (ontext(comillinggTTre))) # run the teacher progran on the example, and get its final prediction here
20
21
22
e that compiling=Irue nay affect prediction = teacher (**example. ())
23
24
the of the all interal Predict calls from teacher program predicted_traces = dapy. settings.trac
26
27
28
if the prediction is valid, add the example to the traces self metric prediction, predicteo.traces) for predictor, inputs. outputs predicted_tracee = Example (autonttautt **inputs, **outputs) predictor_nane htw.aeddpprrrraaaeeiiittttto d append (d)
29
30
31
32
33
35
36
return compiled_program
H.2 BOOSTRAPENSSIOOWIIIRANNSSSEEEE (Telepronpter):
class def ..iiit t._ (self, metric None, trials=16): self metric astric self. trials trials
def compile student. *, teacher=None, trainset, valset=None): of cross cross-validation if valset trainset if valset is None else valset
unset.
candidates
11
for seed in range(self. (snefss)).trialss)
bootstrap few-shot progran.
13
shuffled,trainsee trainset shuffle(trainser seed=seed)
tp (matrtrrarrraaaattttetereeeeee candidate_progran tp. compile (student,,, shurfled_trainset, teacher)
max_bbotstrrap.dmaasraaddd ())
15
16
Step Evaluate the generated candidate program.
18
score append candidate_progrant
self. netric, valset)
19
20
return candidate progran. return max key=lambda x[0])[1]
22
27

---

Published as a conference paper at ICLR 2024
I EXAMPLES OF THE PROMPTS AUTOMATICALLY GENERATED BY DSPY
We include three example prompts bootstrapped by DSPy for the 1lama2-1 11-ma-t1b-ccat experiments in this paper Figures 9, 10, and 11. These include the prompt automatically generated by DSPy for GSM8K vanilla program compiled with bootstrap the GSM8K CoT program compiled with bootstrap, the second-hop cenerate_query prompt for HotPorQA's multihop program. All of these, particularly their demonstrations' labels and their selection, ically using 1lama2-13b-chat Consider the first one, i.c. Figures 9. This shows that, even though the program (vanilla) does not induce chain of thought, the LM effectively leverages the nature of the GSM8K metric (checking for the final number for evaluation) and bootstraps its own reasoning- based demonstration, which makes the program effective at test time. Next, consider the second-hop query prompt bootstrapped by DSPy using 11 11ama2-133 12---1hb-hat,, shown in Figure 11. We hypothe- size this prompt works effectively because it demonstrates the process of generating search queries for two complex questions by producing two very different types of queries: a simplified question ("When was the first...?") and keyword query ("Jeremy Paxman birth year"). This might allow the LM to resolve different questions at test-time more effectively.
generated by DSPy automat-
28

---

Published as a conference paper at ICLR 2024
Gen Fields 'quettion,, produce "ansser".
Follow the following format.
Question: {qqusstion)
Answer: 4 [nsserr)
12 Question: Jimy and Irene shopping for clothes on Tuesday, where senior citizens get a 188 discount on their purchases. Jimyy picks shorts from the $15 rack. Irene grabs sntrts from rack. ruch money they give the cashier?
Answer: Jimmy picks out shorts at $15 each 545, Irene grabs shirts at $17 each $65. Total cost $45 585 $138. Since senior citizens 18% discount, will of $128 513. they will cashier $138 513 $117.
Question:
Copy of the prompt automatically generated by DSPy for GSM8K Llama2- 13b-chat vanilla program compiled with bootstrapx2.
Figure 9:
Given the fields 'question', produce the fields "answer'.
Fallow folloxing Format.
Question: {(question)
8 Reasoning: Let's thank step by stop in order to Sproduce the answer. We
4 (answer)
13 Question: Mark is baking bread. He has to let it rise for 128 minutes twice. He alsu needs to spend 10 ninutes kneading minutes baking
nany minutes 14 Reasoning: Let's think stop by stop order to rind out how nany minutes it takes Mark to rinish nakiing the bread. we know rise
to naking
that needs and 30 sinutes baking it. in total, it will take Mark 18 30 260 ninutes finish masknng the bread.
winutes twice, so that's minutes. Then, needs spend
kneading
Answer: 283
17
Question: Ben has $2888 For his bustness operations casts. He orders goods From his supol and writes than cheque for $688. debtor pays
purchases they made credit. Hr. Ben
to equipeent mwintenance
and spends $1269 the whole operation. How such is
remsining with?
Reasoning: Let's think step by step order to find out hox nuch money ir. Ben is remaining with. Me know that he had begin with,
$1400 Then, paid hin $580, he has $1480 s880 $2200. Finally, he spent $1290 on equipment maintenance so he has $2200 $1200 left.
$600
frrm
he 52008
21 Ansiver: $1900
other denonstrations here
26
Question:
Figure 10: Shortened copy of the prompt automatically generated by DSPy for GSM8K Ilama2- 13b-chat CoT program compiled with bootstrap.
29

---

Published as a conference paper at ICLR 2024
rields 'contexe", "question', the rields searchhquery'.
Follow Foll lowing format.
Context: [contex)))
[squsttion)
Reasoning: let's think step by step in order to s(produce the search.guary) We
Search Query: s(seorch.query)
11
Context:
Twilight series)
sertis of varpire-thered Fantasy novels American
Scephenie Neyer,
Harper Connelly Mysteries llarper Connelly Mysteries series fantasy mystery Harris, and first published in 2005.
by Charlaine
[3] The Dark lleraine The Dark lleroiie is series vampire- thered fantasy rosance novels wricten by English author Abigall ciobs, published Harpertollins 2012.
first of vaspire-thesed Fantasy official Illustraced Guide serves as spin-off encyclopedic reference book first published?
Question: which year
Twiliihh The
21 Reasoning: Let's think stop by stop in order to determine the year the first of the vamptre-thened rantasy ronance novels was first sublished.
Search Query: when was the first of the varpire-thered Fantasy rorance novels published?
25
27 Concext:
[1] The Victorians | The Victorians Their Story In Pictures is 2009 British documentary series which focuses on
Victorian and culture.
[23 The Caxtons The Caxtons: Fanily Picture is an 1849 Victorian novel by Edward Rui wer-l ytton that was popular in its
38 [3] Victorian (conics) The Victorian is 25-issue conte book series published by Penny-Farthing Press and starting in
31
Question: The Victorians
Their Story In Pictures docusentary series written by an author born what year?
34 Reasoning: Let's think step by step in order to produce the search query. Me know that the decurentary series is about presented Jereny Pasran, Therefore,
Victorian culture, was Jereny Pawaon was born.
year
Search Query: Jareny Paxman birth year
39
Concoxt:
Figure 11: Shortened copy of the prompt automatically generated by DSPy for HotPotQA Llama2- 13b-chat multi-hop program (generating second hop query) compiled with bootstrap.
30

---

Published as a conference paper at ICLR 2024
J FUTURE WORK DIRECTIONS
As we discuss in Sec 2, this work is inspired by the role that abstractions and libraries for deep learning played in the development of the field. Our goal for DSPy is to recreate that revolution for what we call foundation model programming. To this end, we have started working on several follow-up directions that expand and improve DSPy.
At a high level, we see highly promising directions in: (1) the implementation of new modules (cor- responding to various new prompting techniques), (2) the systematic evaluation of larger number of advanced programs for open-ended tasks, (3) the addition of various automatic LM-based metrics for optimization, (4) the addition of methods for better debugging and inspection and (5) for more controlled generation (a la LMQL: Beurer-Kellner et al. 2023), and (6) the development deeper understanding of the tradeoffs associated with compiling with large vs. small LMs.
Beyond these, one of the key strengths of DSPy is the modularity between programs and teleprompters (optimizers). The discussion in Sec 4 lays out several angles for building more so- phisticated teleprompters, which can improve the quality of existing well as DSPy programs.
Better candidate generation. We envision more systematic treatment of the variables to opti- mize, c.g. instructions, demonstrations, string prefixes (c.g.. "Let's think step by step" for chain of thought), and numerical hyperparameters (e.g., the number of passages to retrieve or the number of candidate answers to generate). Future work may benefit from the way variables are defined in Op- tuna (Akiba al., 2019) and in the conceptual framework of Dohan et al. (2022) to allow expressing these. Future work may also evaluate methods to generate more diverse or more challenging demon- strations, including more sophisticated logic for bootstrapping (e.g., trying multiple variants of the program) or filtering (e.g.. only selecting examples that the zero-shot version of the program cannot solve correctly).
Better parameter optimization. More sophisticated programs will likely benefit from more ad- vanced optimization strategies. For instance, the Optuna teleprompter in DSPy may allow Bayesian optimizations that exceed the quality achieved with random search. More research is required to understand the tradeoffs of cost and quality in this case. Going a step beyond this, LMs can take a more active role in optimization, beyond bootstrapping demonstrations. For instance, they can re- vise instructions based on categories of common failures or attempt to simulate a notion of gradients over multi-stage programs.
Better higher-order program optimizations. All programs in this paper were directed to prompt (or ensemble of optimized prompts) for each module. is possible to treat different inputs for each module more dynamically, directing them to more specialized prompts. While DSPy has an initial teleprompter that treats different inputs more dynamically, this space is very large and warrants much exploration. Beyond that, the ability to backtrack to correct mistakes and address LM feedback accordingly is another promising direction.
In all of these, the references presented in Sec may serve as powerful starting points for each individual component (e.g., utilizing LM feedback), which DSPy can generalize and extend to deal with arbitrary LM pipelines of declarative modules.
31


Reviews:
[{"review_id": "Q00ZqwA9Bd", "reviewer": "Reviewer_FCUg", "summary": "This paper introduces DSPy, a framework for expressing LM pipelines in a higher level language with support for automatic pipeline optimization. The results demonstrate that 1) complex pipelines can be written quickly and concisely in DSPy and 2) their automatic pipeline optimizations yield substantial gains in performance for two multi-step reasoning tasks (math reasoning and multi-hop QA).", "questions": "Do the authors have any ideas for future optimization strategies (or \"teleprompters\") that could be implemented within this framework?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "DSPy is a major improvement over manually composing complex LM pipelines by hand. The paper also demonstrates the possibility of automatically optimizing parts of the pipeline once written in the DSPy framework. Finally, the experimental results are very impressive, particularly given the simplicity of user experience.", "weaknesses": "There is a major missing related work [1], which takes a similar approach of expressing LM pipelines as programs, and also comes with built-in optimizations. I would be happy to increase my score if the paper is revised to include a discussion comparing the two approaches.\n\nA more minor concern is that the optimization techniques demonstrated here are relatively limited in scope. From a conceptual standpoint, I would have liked to see more than ensembling or bootstrapping few shot examples.\n\n[1] Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting Is Programming: A Query Language for Large Language Models. Proc. ACM Program. Lang. 7, PLDI, Article 186 (June 2023), 24 pages. https://doi.org/10.1145/3591300\n\n====\n\nThe rebuttal has addressed my concerns and I have increased my score from a 6 to 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzjRMISkL7", "reviewer": "Reviewer_zPoD", "summary": "This paper presents a programming model DSPy that allows defining parameterized LLM-based text transformations. Under this model, multi-step LLM strategies like chain-of-thought, ensembles of chains-of-thoughts, ReAct, and reflection are straightforward to implement. The parameters in these strategies can include the few-shot demonstrations and the specific prompting wording, and DSPy admits using LLM-powered search strategies for optimizing over these strategy parameters. With DSPy, the paper authors demonstrate using a large language model to optimize prompting strategies for use with comparatively smaller language models.", "questions": "# Questions and Suggestions:\n\nTerminology: In the abstract you say DSPy programs are \"compiled to\" a language model. I think this wording is a bit wrong or misleading, and that it would be better to say the program is \"optimized for\" or (if you insist on the language of compilation) \"compiled for\".\n\nThe paper has a repeated metaphor with neural network abstractions and deep learning frameworks (Section 1 paragraph 3, and Section 2 paragraph 1, and Section 3.2 paragraph 4). The metaphor rests on two properties of DSPy: its programs are modular and admit optimization. However, the main feature of deep learning systems is that they admit *gradient based* optimization, which is absent from DSPy, weakening the metaphor.\n\nTeleprompters are described as general-purpose optimization strategies, though if I understand correctly they are limited to gradient-free optimization techniques. This detail warrants mentioning.\n\nStrong claim: The paper claims DSPy is the first programming model that translates prompting techniques into parameterized declarative modules that can be optimized. In evaluating this claim, I considered the following relevant works that I thought I would note here.\nDohan 2022 Language Model Cascades https://arxiv.org/abs/2207.10342 also casts several prompting techniques like scratchpads / chain of thought, verifiers, STaR, selection-inference under a unified framework.\nDecomposed Prompting: A Modular Approach for Solving Complex Tasks https://arxiv.org/abs/2210.02406 splits tasks into subtasks that are solved a recombined to solve the overall task.\nANPL: Compiling Natural Programs with Interactive Decomposition https://arxiv.org/abs/2305.18498 admits the user performing the decomposition of tasks into subtasks.\nLarge Language Models as Optimizers https://arxiv.org/abs/2309.03409 like a DSPy compiler produces text-transformation using an LLM to perform prompt optimization.\nAmong these, Dohan 2022 is closest to the claim made in the paper.\n\n-- In your treatment of Predict, I don't think the inclusion of the \"None\" implementation detail aids the discussion.\n\nIn the list of parameters (Section 3.2, paragraph 3), field descriptions are omitted. \nHow do you think about safety during SQL generation, since SQL queries can modify or delete tables. How do you approach using tools with side effects during compilation?\nIn the RAG example at the end of Section 3.2, is it complete without a definition of Retrieve provided?\nHow can a DSPy user introduce a new Retrieve implementation?\n\nIn my view, one of the key results is that expensive LLMs can be used during the optimization process of a prompt pipeline targeting smaller LMs. I will copy here the text from the \"weaknesses section\" above pertaining to this point: There are no examples of compiled programs provided in the paper or appendices. I think an analysis of compiled programs would benefit the paper meaningfully. In particular, some unanswered questions about the compiled programs include: how do they differ from the types of prompt programs that people write by hand or using other frameworks? how do programs compiled for small LMs differ from those compiled for large LMs? are there any obvious patterns in the compiled programs? how about obvious shortcomings or irregularities, where additional hand-optimization would be easy? any evidence that the optimization techniques overfit to the validation set?\n\nRegarding how the compiled programs differ when targeting smaller LMs, a qualitative investigation with examples would be welcome.\n\nIn Table 1, reporting some measure of compute usage or time both for compilation and inference would be valuable, and would make the various strategies (both programs and compilation approaches) more readily comparable.\n\nNote: MultiChainComparison implementation is not provided.\nNote: Do the compilation approaches account for overfitting the validation set?\n\nAnother idea for further study: Can you compile a DSPy program for a small LM and then run that program using a more powerful LLM different from the one targeted during compilation in order to save on compilation costs? What sort of performance degradation does this incur?\n\nI also include here some typographic issues I identified in the paper. These did not meaningfully hinder readability of the work.\n\nTypo: GMS08K -> GSM8K (page 2, paragraph 4)\nTypo: format -> formats (page 4, Section 3.2)\nTypo: grammar: but -> or (page 5, Section 3.3)\nTypo: \"As we assumes\" in the example at the end of Section 3.3\nTypo: these -> this, ensembles -> ensembling (page 6, stage 3)\nTypo: \"specific math problems or particular LM.\" -> \"specific to math problems or a particular LM.\" (page 7)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a novel programming paradigm for defining and optimizing parameterized LLM-based text transformations.\n* Though I have not used DSPy myself, my impression is that there is a focus on expressiveness and usability in the programming model, which enhances its usefulness for the ICLR community. One piece of evidence for this expressiveness is the ability showcased to write CoT, reflection, ReAct, and other prompting strategies concisely within the DSPy model. Another is the composibility e.g. of different programs with different optimization (aka \"compilation\") approaches.\n* DSPy reduces the reliance of creating LLM-based text transformations on manual prompting. This can be seen as a strength, reducing the need for expertise in prompt writing. (It can also on some occasions be a weakness if it increases the amount of examples required e.g. from none.)", "weaknesses": "* My impression is that the value-add of DSPy for any of the listed strategies is somewhat small. E.g. implementing any of CoT, Reflection, or ReAct without DSPy does not require much code. The same is true, I think, for the optimization approaches / compilers. I expect this value-add grows when working with many such strategies at once, and additionally that DSPy provides organizational value both as the programs and compilers grow in complexity, and as the set of people using them grows. Being able to compose different strategies and compiler techniques is also one of the key beneficial properties of the system.\n* The evaluations performed do not provide a measure of compute usage or time (both for compilation as well as inference of the compiled programs), which makes comparisons across programs and compilers less meaningful.\n* There are no examples of compiled programs provided in the paper or appendices. I think an analysis of compiled programs would benefit the paper meaningfully. In particular, some unanswered questions about the compiled programs include: how do they differ from the types of prompt programs that people write by hand or using other frameworks? how do programs compiled for small LMs differ from those compiled for large LMs? are there any obvious patterns in the compiled programs? how about obvious shortcomings or irregularities, where additional hand-optimization would be easy? any evidence that the optimization techniques overfit to the validation set?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIhxcYk7zs", "reviewer": "Reviewer_cQiR", "summary": "This paper introduces DSPy, an LM pipeline framework consisting of a programming model and compiler. The programming model provides composable and declarative modules for LM instruction. These modules function similarly to function calls, allowing users to define input/output behavior using natural language signatures. The compiler is capable of generating high-quality prompts automatically or fine-tuning LMs using a general optimization strategy (teleprompter). Through evaluations on GSM8K and HotpotQA, the authors demonstrate DSPy's ability to reduce the required number of handwritten prompt templates.", "questions": "## Approach:\n1. Can we easily implement common and advanced prompting techniques with signatures? I noticed that the signatures used in the Case Study are quite simple. When it comes to dealing with complex user intentions, do you think additional hand-written comments are necessary alongside signatures? For instance, in the ChainOfThought module, I found that the prompt \"Let's think step by step\" still requires handwriting within the module. How many prompts would need to be handwritten to achieve functional equivalence with a complex prompt (such as Appendix D prompt 6) using signatures?\n2. How can users debug their DSPy programs effectively? In Section 4 of the paper, it is mentioned that \"A well-decomposed program can typically find at least a few training examples...\" I suppose that the success of optimization depends on a well-structured decomposition by the user, which implies the need for frequent modification of DSPy programs. Consequently, it raises the question of how users can determine which modules are not functioning correctly. Is it necessary for users to repeatedly rewrite the entire DSPy program?\n3. How much does the optimization cost in terms of computing resources or API usage? (This question may exceed the paper's scope) \n## Evaluation\n1. In section 5, Hypothesis 2, the paper mentions the possibility of DSPy outperforming expert-written prompts. However, is there an evaluation comparing DSPy with handwritten prompts to support this claim?\n2. In Section 5 of the paper, it states, \"... ‘how they compare on GSM8K with program P when compiled with strategy S’, which is a well-defined and reproducible run.” How can the reproducibility of these results be ensured?\n3. In Case Study 1 using the GSM8K dataset, GPT-3.5 with 5 few shots achieves a score of 57%, while vanilla+GPT-3.5+fewshot only scores 33.1%. What factors contribute to the decline in performance, considering the similarity between these two approaches?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel Approach. This paper introduces a novel approach to systematically build LM pipelines and compile modules into a set of prompts (or fine-tunes) optimized for specific tasks. The approach is promising, showcasing its ability to reduce the human effort required to develop prompt pipelines. \n2. Well-Written. The paper is clearly written and presents the core concepts of DSPy in a straightforward manner.", "weaknesses": "1. Lack of details: DSPy stands out from other frameworks, like LangChain, due to its capacity for automatic prompt generation and optimization. However, the introduction of the DSPy Compiler lacks sufficient specificity. It would be beneficial to provide more comprehensive details on how DSPy generates candidate values for module parameters (instructions, field description, and example input/output) based on signatures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)