explain the following like i am a layperson: ""arXiv:2312.13382v2 [cs.CL] 2 Feb 2024
DSPy Assertions:
Computational Constraints for Self-Refining Language Model Pipelines
Arnav Singhvi * 1 Manish Shetty * 1 Shangyin Tan * 1
Christopher Potts 2 Koushik Sen 1 Matei Zaharia 1 Omar Khattab 2
Abstract
Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important
constraints requires heuristic “prompt engineering.” We introduce LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. We integrate our
constructs into the recent DSPy programming
model for LMs and present new strategies that
allow DSPy to compile programs with LM Assertions into more reliable and accurate systems.
We also propose strategies to use assertions at inference time for automatic self-refinement with
LMs. We report on four diverse case studies for
text generation and find that LM Assertions improve not only compliance with imposed rules
but also downstream task performance, passing
constraints up to 164% more often and generating up to 37% more higher-quality responses.
1. Introduction
Language models (LMs) now power various applications,
from conversational agents to writing assistants. However,
the probabilistic nature of LMs often results in outputs
that may not align with the constraints of the domain or
the larger pipeline in which the LM is used. To address
this, researchers have explored various techniques, including applying constrained decoding (Hokamp & Liu, 2017;
Hu et al., 2019), exploring approaches for self-reflection
and tree search (Madaan et al., 2023; Shinn et al., 2023;
Yao et al., 2023), building domain-specific languages and
like LMQL (Beurer-Kellner et al., 2023), or monitoring
models with assertions and guardrails (Kang et al., 2020;
Rebedea et al., 2023) to steer LMs towards more controllable outputs.
Recently, several LM frameworks like LangChain (Chase,
2022) and DSPy (Khattab et al., 2022; 2024) provide developers with interfaces to build complex LM pipelines. Some
offer several features to control LM outputs, e.g., DSPy can
optimize multi-stage prompts to maximize a target metric.
However, they cannot take arbitrary computational constraints on LM pipelines to either teach the LM to follow
such constraints and/or to introspectively self-refine outputs. While some of this may be achieved via pain-staking
“prompt engineering” or other ad-hoc guidance strategies,
such efforts are labor-intensive and conflate the high-level
design of new AI systems with the low-level exploration of
teaching LMs how to follow constraints.
We propose LM Assertions, a novel programming construct
designed to enforce user-specified properties on LM outputs within a pipeline. Drawing inspiration from runtime
assertions and program specifications in traditional programming, LM Assertions are expressed as boolean conditions that express the desired characteristics of LM outputs. We integrate LM Assertions into the recent DSPy programming model. Besides serving as conventional runtime
monitors, LM Assertions enable multiple novel assertiondriven optimizations to improve LM programs.
Assertion-driven backtracking. LM Assertions can facilitate runtime self-refinement in LM pipelines at inference
time. When a constraint fails, we allow the pipeline to backtrack and retry the failing module. LM Assertions provide
feedback on retry attempts; they inject erring outputs and
error messages to the prompt to introspectively self-refine
outputs. Figure 1 illustrates this within a DSPy pipeline.
Assertion-driven example bootstrapping. LM Assertions can enable guided prompt optimizers at compile time.
Integrated with existing automatic prompt optimizers in
DSPy, they can enable generating harder few-shot examples, which can teach LM programs to conduct challenging
steps.
Counterexample bootstrapping. During prompt optimization and example bootstrapping, another important
contribution of LM Assertions is to develop demonstrations
that contain failed examples and traces to fix the errors.
When the counterexamples are mixed with bootstrapped
few-shot examples, the LM is more likely to avoid the same
mistakes without assertion-driven backtracking.
1
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
class MultiHopQAWithAssertions(dspy.Module):
...
def forward(self, question):
context, queries = [], [question]
for hop in range(2):
query = self.generate_query(context=context, question=question).query
dspy.Suggest(len(query) < 100,
"Query should be less than 100 characters")
dspy.Suggest(is_query_distinct(query, queries),
f"Query should be distinct from {queries}")
context += self.retrieve(query).passages
queries.append(query)
return self.generate_answer(context=context, question=question)
UPDATED PROMPT WITH FEEDBACK
Context: . . .
Question: . . .
Past Query: <previous attempt w/ errors> . . .
Instruction: Query should be distinct from . . .
FAIL! ✗
backtrack and regenerate query with new prompt
Figure 1. DSPy program with LM Assertions for multi-hop question-answering task with a retriever. We introduce two soft assertions
(suggestions): (1) query to retriever should be less than 100 characters; (2) query to retriever should differ from previous queries. For
instance, if the second suggestion fails, DSPy will construct a new prompt to retry the generate_query module with additional fields,
highlighting the previously generated query and a user-defined error message to help the LM refine its generation.
We propose two types of LM Assertions: (hard) Assertions
and (soft) Suggestions, denoted by Assert and Suggest,
respectively. Hard assertions represent critical conditions
that, when violated after a maximum number of retries,
cause the LM pipeline to halt, signaling a non-negotiable
breach of requirements. On the other hand, suggestions denote desirable but non-essential properties; their violation
triggers the self-refinement process, but exceeding a maximum number of retries does not halt the pipeline. Instead,
the pipeline continues to execute the next module.
We implement our work atop DSPy, a state-of-the-art
framework for building and automatically optimizing
declarative LM pipelines. The integration enables us to
design and implement our three new assertion-driven optimizations. It enables DSPy programs to self-refine and
produce outputs that adhere to specific guidelines. It simplifies debugging, providing developers a clearer understanding of LM behavior in complex pipelines. In addition, by
combining LM Assertions with prompt optimizers in DSPy,
we can bootstrap better few-shot examples and counterexamples to assertions to make the pipeline more robust and
performant.
We evaluate the effectiveness of LM Assertions on four
varied knowledge-intensive tasks: multihop question answering (MultiHopQA), long format question answering
(LongFormQA), formatted quiz generation (QuizGen), and
valid tweet generation (TweetGen). Our experiments show
that LM Assertions and assertion-driven backtracking significantly improve LM programs from generating 37.6%
well-formatted quizzes in JSON to 98.8% in QuizGen.
Then, with assertion-driven example bootstrapping and
counterexample bootstrapping, we see an increase from
2.0% to 73.0% in writing more engaging tweets in TweetGen and a consistent boost on almost all other assertion
metrics. Finally, with LM Assertions and all assertiondriven optimizations, we see a maximum gain from 30.5%
to 87.2% in generating valid quiz questions.
Our contributions are, first, introducing LM Assertions as
an abstraction for language model programming. Then,
we present three novel optimizations for LM pipelines:
assertion-driven backtracking during inference to help
models self-refine, assertion-driven example bootstrapping to generate and choose more effective few-shot examples, and counterexample bootstrapping to develop fewshot examples with erroneous results and corresponding
fixes to help the model become more reliable at complying
to developer-desired constraints.
2. Background and Motivation
The goals of LM Assertions are general and can be applied
to any LM program. Due to its modular paradigm, flexibility, and extensibility, we implement our work as extensions
to the state-of-the-art DSPy (Khattab et al., 2024) framework. Below, we briefly describe the DSPy programming
model for building declarative LM pipelines and compiling
them into auto-optimized prompt (or finetune) chains. We
then sketch a realistic, motivating example for LM Assertions and show their usefulness for self-refinement in LM
pipelines.
2
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
2.1. The DSPy Programming model
DSPy is a framework for programmatically solving advanced tasks with language and retrieval models through
composing and declaring modules. The overarching goal of
DSPy is to replace brittle “prompt engineering” tricks with
composable modules and automatic (typically discrete) optimizers.
First, instead of free-form string prompts, a DSPy programmer will define a signature to specify what an LM needs to
do declaratively. For instance, a module may need to consume a question and return an answer, as shown below:
qa = dspy.Predict("question -> answer")
qa(question="Where is the Eiffel tower?")
# Output: The Eiffel Tower is located in Paris, France.
To use a signature, the programmer declares a module with
that signature, like we defined a Predict module above.
The core module for working with signatures in DSPy is
Predict. Internally, it stores the supplied signature. When
the signature is called, like a function, it constructs a formatted prompt according to the signature’s inputs and outputs. Then, it calls an LM with a list of demonstrations (if
any) following this format for prompting.
DSPy modules usually call dspy.Predict one or more
times. They generally encapsulate prompting techniques,
turning them into modular functions that support any
signature. This contrasts with handwriting task-specific
prompts with manually tuned instructions or few-shot examples. Consider, for example, the below DSPy module
from Khattab et al. (2024), which implements the popular
“chain-of-thought” prompting technique (Wei et al., 2022).
class ChainOfThought(dspy.Module):
def __init__(self, signature):
rationale_field = dspy.OutputField(
prefix="Reasoning: Think step by step.")
signature = dspy.Signature(signature).
prepend_output_field(rationale_field)
self.predict = dspy.Predict(signature)
def forward(self, **kwargs):
return self.predict(**kwargs)
DSPy modules can be composed in arbitrary pipelines
by first declaring the modules needed at initialization
and then expressing the pipeline with arbitrary code that
calls the modules in a forward method (as shown in the
ChainOfThought module above and the MultiHopQA program
in Section 2.2). Finally, DSPy provides optimizers that automates generating good quality demonstrations (few-shot
examples) or instructions for a task given a metric to optimize. We may also refer to the few-shot example selection
process as compiling the LM pipeline application.
Challenges. DSPy signatures provide type hints that softly
shape LM’s behavior. However, the framework currently
lacks constructs developers can use to specify arbitrary
computational constraints the pipeline must satisfy. Additionally, one can imagine the LM pipeline using these constraints to refine its outputs and to teach the LM to respect
these specifications at compile time.
To address these challenges, we integrate LM Assertions
as first-class primitives in DSPy. In the style of Pythonic
assertions, they are intuitive constructs that allow DSPy
to constrain LM outputs. They are flexible in that they
can be strict restrictions, softer guidelines for backtracking
and self-correction of LM calls, or simple debugging statements. In what follows, we describe a motivating example
of a DSPy program that uses LM Assertions for multi-hop
question answering.
2.2. Motivating Example
Aiden is a developer building an LM pipeline for multi-hop
question-answering. The task involves the LM performing
a series of inferential steps (multi-hop) before answering a
question while utilizing a retriever to get relevant context.
In a simple DSPy implementation, Aiden may design the
pipeline below, where the LM generates search queries to
collect relevant context and aggregate them to generate the
answer.1
class MultiHopQA(dspy.Module):
def __init__(self):
self.retrieve = dspy.Retrieve(k=3)
self.gen_query = dspy.ChainOfThought("context,question -> query")
self.gen_answer = dspy.ChainOfThought("context,question -> answer")
def forward(self, question):
context = []
for hop in range(2):
query = self.gen_query(context=context, question=question).query
context += self.retrieve(query).passages
return self.gen_answer(context=context, question=question)
However, certain issues with the pipeline might affect its
performance. For instance, since questions are complex,
the generated search query could be long and imprecise,
resulting in irrelevant retrieved context. Another issue is
that similar multi-hop queries would result in redundant retrieved context. One might observe that these are properties
of generated queries that are computationally checkable
and, if expressible as constraints on the pipeline, might improve its performance.
Figure 1 shows a DSPy program with LM Assertions for
this task. To mitigate the issues above, Aiden introduces
two soft LM Assertions: first, they restrict the length of
1We borrow this implementation from Khattab et al. (2024).
It captures the key computations in popular multi-hop questionanswering systems such as Baleen (Khattab et al., 2021) and IRCoT (Trivedi et al., 2022).
3
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
the query to be less than 100 characters, aiming for precise
information retrieval. Second, they require the query generated at each hop to be dissimilar from previous hops, discouraging retrieval of redundant information. They specify
these as soft constraints using the Suggest construct. The
force of this construct is to allow the pipeline to backtrack
to the failing module and try again. On retrying, the LM
prompt also contains its past attempts and suggestion messages, enabling constraint-guided self-refinement.
In Section 5, we evaluate this pipeline on the HotPotQA (Yang et al., 2018) dataset. We find that enabling
the developer to express two simple suggestions improves
the retriever’s recall (by 6.5%–7.9%) and the accuracy of
generated answers (by 3.4%–14.4%).
3. Semantics of LM Assertions
To help with the goals mentioned above, in this work, we
introduce LM Assertions and integrate them in DSPy. We
define LM Assertions as programmatic elements that dictate certain conditions or rules that must be adhered to during the execution of an LM pipeline. These constraints
ensure that the pipeline’s behavior aligns with developerspecified invariants or guidelines, enhancing the reliability,
predictability, and correctness of the pipeline’s output.
We categorize LM Assertions into two well-defined programming constructs, namely Assertions and Suggestions,
denoted by the constructs Assert and Suggest. They
are constructs that enforce constraints and guide an LM
pipeline’s execution flow.
Delineating Assert from Conventional Assertions. The
conventional assert statement, built into most programming languages, is a debugging aid that checks a condition and, if the condition evaluates to false, raises an
AssertionError exception, typically terminating the program execution. In contrast, our Assert construct offers
a sophisticated retry mechanism while supporting several
new optimizations. On an Assert failing, the pipeline transitions to a special retry state, allowing it to reattempt
a failing LM call while being aware of its previous attempts and the error message raised. If, after a maximum number of self-refinement attempts, the assertion still
fails, the pipeline transitions to an error state and raises
an AssertionError, terminating the pipeline. This enables
Assert to be much more powerful than conventional assert
statements, leveraging the LM to conduct retries and adjustments before concluding that an error is irrecoverable.
Delineating Suggest from Assert. In contrast to our Assert
statements, our Suggest statements are softer constraints
that recommend but do not mandate conditions that may
guide the LM pipeline toward desired domain-specific outcomes. When a Suggest condition is not met, like Assert
, the pipeline enters the special retry state, allowing reattempts of the failing LM call and self-refinement. However, if the suggestion fails after a maximum number of
self-refinement attempts, the pipeline simply logs a warning SuggestionError message and continues execution. This
allows the pipeline to adjust its behavior in response to the
suggestion while being flexible and resilient to suboptimal
states (or sub-optimal or heuristic computational checks).
In the following sections, we define the default backtracking semantics of these constructs more formally. However,
we provide the opportunity to extend custom semantics for
both Assert and Suggest (Appendix A.2).
3.1. Semantics of Assert
The Assert construct enforces invariants within the LM
pipeline. The semantics of an assertion can be defined in
terms of a state transition system by adapting big-step operational semantics notations in (Kahn, 1987) where σr represents the pipeline’s state, and the subscript r represents the
current retry count within the state σ. The maximum number of retries allowed per assertion is denoted by R. The
transition relation σr ⊢ i → σ
′
reads as “under the state σr,
the instruction i transitions the state to σ
′
”. Now, we define
a simplified semantics for Assert:
σr ⊢ Assert(e, m) → σ
′
0 if eval(σ, e) = true
σr ⊢ Assert(e, m) → σr+1 if eval(σ, e) = false and r < R
σr ⊢ Assert(e, m) → σ
⊥
if eval(σ, e) = false and r ≥ R
Here, eval(σ, e) denotes the evaluation of expression e in
state σ. If e evaluates to true, the pipeline transitions to
a new state σ
′
and continues execution. If e evaluates to
false and the current retry count r is less than the maximum allowed retries R, the pipeline transitions to a retry
state σr+1. Here, the pipeline attempts to recover or adjust its behavior, incrementing the retry count r by one. If
the assertion continues to fail and the retry count reaches
R, the pipeline transitions to an error state σ
⊥, and an
AssertionError with message m is raised, halting the execution.
3.2. Semantics of Suggest
The Suggest construct provides non-binding guidance to the
LM pipeline. Similar to Assert, its semantics can be defined
as follows:
4
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
σr ⊢ Suggest(e, m) → σ
′
0 if eval(σ, e) = true
σr ⊢ Suggest(e, m) → σr+1 if eval(σ, e) = false and r < R
σr ⊢ Suggest(e, m) → σ
′′
0 if eval(σ, e) = false and r ≥ R
If the expression e evaluates to true, the pipeline transitions
to a new state σ
′
and continues execution. If e evaluates
to false and the current retry count r does not exceed the
maximum retries R, the pipeline attempts to recover or adjust its behavior in a retry state σr+1. However, different
from Assert, if the suggestion continues to fail and the retry
count r reaches R, the pipeline transitions to a new state
σ
′′ where it resets the retry count, logs the message m as
a warning of a SuggestionError that could not be resolved,
and continues executing the next pipeline module.
4. Assertion-Driven Optimizations
4.1. Assertion-Driven Backtracking
Both Assert and Suggest allow the pipeline to retry a failing
LM call and self-refine its outputs in a special retry state.
One might observe that this involves dynamically altering
the control flow of the LM pipeline during execution. On
passing assertions and suggestions, the control flows typically into the next LM pipeline module.
To handle assertion failures, the LM pipeline may utilize
an error handler that determines the next instruction to execute. The handler takes the current erring state σ and the error message m as inputs and returns a new state. In the new
state, control flows as described in Section 3.1 and 3.2. For
both Assert and Suggest, if the maximum retry attempts are
not surpassed, the handler yields the control to the failing
LM module with an updated prompt that includes past failing outputs and instructions. However, upon exceeding the
maximum retries, the handler halts the execution for a failing Assert or progresses to the next module in the pipeline
for a Suggest.
In Appendix A, we describe the implementation of these
constructs and handlers in DSPy. Our implementation is
publicly available and has been integrated into the DSPy
framework.
4.2. Assertion-Driven Example Bootstrapping
LM Assertions can be useful in optimizing the prompts as
well. One optimizer BootstrapFewShot in DSPy employs a
teacher-student method to use the teacher model to bootstrap representative few-shot demonstrations for the student model for the same program. During the bootstrapping step, the teacher model can utilize LM Assertions as
extra filters to bootstrap more robust examples.
Based on our observation, in some cases, the naïve optimizer in DSPy would bootstrap an example with the correct
final response while the intermediate module outputs are incorrect, which leads to wrong demos for intermediate LM
modules. To enable assertion-driven example bootstrapping, we apply assertion-driven backtracking to the teacher
model in the BootstrapFewShot optimizer. In this way, all
the bootstrapped demonstrations are guaranteed to follow
the intermediate constraints, too. In this way, although the
prompt optimizer only has the metric for the final answer,
the examples selected will have higher qualities for all intermediate modules thanks to LM Assertions.
4.3. Counterexample Bootstrapping
When integrating LM Assertions and assertion-driven backtracking in the teacher model of a prompt optimizer, we can
collect traces where the language model fails certain assertions.
The optimizer in DSPy is able to incorporate feedback from
the erroneous examples during backtracking as demonstrations. The usage of counterexample is twofold. First, counterexamples serve as negative demonstrations in the fewshot learning prompt, guiding models to avoid making similar mistakes. Second, with assertion-driven backtracking,
counterexample traces often come with the demonstrations
of fixing particular LM Assertion failures. These demonstrations are helpful for the student model to achieve a
much higher rate of passing the underlying constraints even
without LM Assertions and assertion-drive backtracking.
Overall, with counterexample bootstrapping only, the overhead of backtracking and self-refinement for the student
model is completely eliminated while the program still
has the ability to generate more responses that adhere to
programmer-defined assertions.
5. Evaluation
5.1. Tasks & Metrics
We study various aspects of LM Assertions on 4 interesting
variants of the popular HotPotQA (Yang et al., 2018) task:
T1 MultiHopQA: A complex question-answering task involving generating multi-hop search queries for questions and using the retrieved context to generate the
correct answer.
T2 LongFormQA: A more demanding questionanswering task, where the generated answer must
contain citations that are faithful to the retrieved
context information.
T3 TweetGen: A variant of HotPotQA, where the gener5
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
ated answer is expected to be a concise and engaging
“tweet” that is also faithful to the retrieved context.
T4 QuizGen: A task involving generating candidate answer choices for HotPotQA questions in a JSON format,
with distractor choices and the correct answer.
Each task is evaluated with two metric categories:
• Intrinsic Quality measures the degree to which
the outputs conform to the LM Assertions specified
within the program. This metric is a benchmark for
the system’s ability to pass internal validation checks.
• Extrinsic Quality measures how LM Assertions affect downstream performance, often on task-specific
properties we cannot assert directly without access to
ground-truth labels. Here, assertions provide guidance
that indirectly influences overall performance.
These two metrics will respectively enable us to investigate the hypotheses that LM Assertions can facilitate selfcorrection and refinement in LM pipelines (H1) and that
such guided self-refinement can enhance the performance
of downstream applications (H2).
We provide a more comprehensive overview of the advanced tasks LongFormQA (Appendix B.1), QuizGen (Appendix B.2) and TweetGen (Appendix B.3), evaluated metrics, and applied constraints in Appendix B.
5.2. Dataset and Models
We utilize the HotPotQA (Yang et al., 2018) dataset for
each task in the open-domain “fullwiki” setting. We then
partition the official training set into subsets: 70% for training and 30% for validation. We only focus on examples labeled as “hard” within the dataset to align with the criteria
marked by the official validation and test sets. For training
and development sets, we sample 300 examples each. We
sample 500 examples for testing.
We use the official Wikipedia 2017 “abstracts” dump of
HotPotQA using a ColBERTv2 (Santhanam et al., 2021)
retriever for retrieval. We test the program using OpenAI’s gpt-3.5-turbo (Brown et al., 2020) with max_tokens
=500 and temperature=0.7 for our experimental setup.
5.3. Strategies
Table 1 summarizes the five strategies in which LM Assertions (particularly Suggest) can be utilized for each
task. First, we distinguish uncompiled strategies (i.e., zeroshot) that predict responses directly from compiled strategies (i.e., few-shot) that contain demonstrations crafted
by the DSPy compiler (Khattab et al., 2024) using its
Assertion
Strategy Compilation Student Teacher
Vanilla ✗ ✗ —
Compile ✓ ✗ ✗
Infer w/ Assert ✗ ✓ —
Compile w/ Assert ✓ ✗ ✓
C+Infer w/ Assert ✓ ✓ ✓
Table 1. Summary of assertion enabled strategies in DSPy.
Vanilla is the baseline with inference only, and Compile is
the baseline with DSPy native compilation. Infer w/ Assert
supports assertion-driven backtracking for inference only,
Compile w/ Assert incorporates assertion-driven example bootstrapping and counterexample bootstrapping during compilation.
Finally, C+Infer w/ Assert contains all the assertion-driven optimizations during compilation and inference.
BootstrapFewShotWithRandomSearch optimizer. In the compiled setting, we differentiate student and teacher modules,
with the teacher compiling demonstrations for the student.
Note that both teacher and student modules use the same
LM in our experiments, which is the default approach in
DSPy, but the teacher modules are executed on training examples and filtered based on the metric.
Baselines include the Vanilla strategy that performs zeroshot prediction with no LM Assertions applied and the
Compile strategy that compiles few-shot learning with the
naive DSPy optimizer. Our strategies explore when assertions are applied: during inference (student only Infer
w/ Assert), during compilation (teacher only Compile w/
Assert), or both (C+Infer w/ Assert).
To ensure the baseline strategies are aware of the potential constraints, we add complex instructions (prompts)
containing all the LM Assertion metrics in QuizGen and
TweetGen to all pipelines. In this way, strategies with assertions do not have the privilege of knowing the intrinsic
metrics. We also report experiments with minimal/generic
instructions in Appendix C, where the baselines perform
much worse and give our assertion strategies more performance advantages.
5.4. Results
Our evaluation aims to answer the following hypotheses:
H1 LM Assertions facilitate automated self-correction
and refinement through assertion-driven backtracking
for arbitrary LM pipelines by showing the LM past
outputs and error messages.
H2 Assertion-driven backtracking with LM Assertions
can also enable LM pipelines to improve downstream
application performance.
H3 When used with compilation and prompt optimization,
6
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
MultiHopQA (Dev / Test)
Strategy Suggestions Passed Retrieval Recall Answer Correctness
Vanilla 66.3 / 66.2 35.0 / 36.6 45.7 / 41.6
Infer w/ Assert 89.7 / 88.0 39.3 / 39.0 47.3 / 43.0
Compile 71.3 / 63.4 37.0 / 40.2 43.7 / 40.4
Compile w/ Assert 78.3 / 71.6 44.3 / 42.2 52.7 / 46.2
C+Infer w/ Assert 95.7 / 91.6 46.0 / 43.4 53.3 / 45.4
LongFormQA (Dev / Test)
Strategy Citation Faithfulness Citation Recall Citation Precision Has Answer
Vanilla 77.3 / 78.8 51.5 / 52.1 58.1 / 59.2 65.7 / 60.2
Infer w/ Assert 90.0 / 90.8 56.3 / 57.8 63.9 / 64.6 65.7 / 60.4
Compile 82.3 / 79.6 41.0 / 39.8 76.8 / 73.5 68.3 / 56.4
Compile w/ Assert 84.0 / 81.2 55.8 / 53.5 66.4 / 63.5 68.0 / 57.4
C+Infer w/ Assert 92.7 / 91.8 43.8 / 43.0 80.1 / 76.3 69.7 / 55.4
QuizGen (Dev / Test)
Strategy Correct JSON Has Answer Citation Precision Validity
Vanilla 41.7 / 37.6 40.3 / 34.8 63.7 / 60.4 36.9 / 30.5
Infer w/ Assert 100 / 98.8 86.3 / 76.6 73.0 / 67.0 80.2 / 70.5
Compile 100 / 100 96.3 / 92.8 68.3 / 63.8 86.1 / 81.7
Compile w/ Assert 100 / 99.8 95.0 / 91.6 70.0 / 62.4 85.1 / 80.5
C+Infer w/ Assert 100 / 100 96.3 / 94.6 82.7 / 75.4 91.0 / 87.2
TweetGen (Dev / Test)
Strategy No "#" Has Answer Concise Engaging Faithful Quality
Vanilla 21.3 / 19.8 52.3 / 46.0 99.7 / 99.6 29.3 / 32.2 78.3 / 79.0 34.7 / 30.5
Infer w/ Assert 71.7 / 67.6 48.7 / 41.0 98.3 / 96.6 37.0 / 36.4 67.7 / 70.4 38.3 / 30.6
Compile 100 / 100 51.0 / 44.2 100 / 100 1.0 / 2.0 63.0 / 65.6 37.8 / 32.8
Compile w/ Assert 96.3 / 95.0 55.0 / 48.8 97.7 / 98.6 74.0 / 73.0 75.0 / 74.8 48.5 / 42.9
C+Infer w/ Assert 98.0 / 96.2 56.0 / 49.2 96.7 / 97.2 90.7 / 85.0 68.3 / 68.0 51.4 / 45.0
Figure 2. Evaluation of each task on the validation set (Dev) and the test set (Test). Tasks are described in Section 5.1, and LM pipeline
configuration are described in Table 1. For each task, we use the same LM pipeline program except for the LM Assertions. Extrinsic
metrics (downstream application performance) are highlighted in grey. For each metric, higher is always better. The highest value in
each column is in bold.
LM Assertions bootstrap more robust and effective
examples/counterexamples, aiding the goal of complying more with the computational constraints and
achieving higher downstream performance.
5.4.1. H1: SELF-CORRECTION VIA LM ASSERTIONS
To study this hypothesis, we mainly look at the intrinsic
metrics of the tasks, i.e., metrics that check if the LM
pipeline conforms to the constraints of the LM assertions
introduced. In Figure 2, we observe that LM Assertions
consistently provide gains for all tasks when comparing
the Vanilla and Infer w/ Assert strategies. That is, in
a zero-shot setting, introducing our self-refinement-based
LM assertions substantially improves the pipeline’s ability
to conform to specs, e.g. in the MultiHopQA task (Figure 1), the number of Suggestions Passed increases by
32.9% for the test set.
The increase is more prominent in the QuizGen task, where
the LM program is tasked to generate a multiple-choice
quiz question in JSON format. Without LM Assertions, the
model pipeline struggles to generate quizzes in valid JSON
(Correct JSON). However, after including constraints that
the response should be in JSON and include the correct answer as one of the choices, together with backtracking and
self-refinement to fix these constraints, the final answers
have correct formatting 98.8% of the time and have the
right answer 76.6% of the time.
5.4.2. H2: PERFORMANCE VIA SELF-CORRECTION
Next, we focus on whether defining suggestions in the
program can help achieve better downstream performance
by comparing Infer w/ Assert with Vanilla. We observe that on most tasks–MultiHopQA, LongFormQA, and
QuizGen–we get a moderate to large improvement on extrinsic/downstream metrics ( grey columns) when suggestions are defined. Notably, in QuizGen, the overall Validity
of the quiz generated increases from 30.5% to 70.5%.
However, on tasks like TweetGen, we do not see a significant increase in the overall Quality of the generated tweet
on the test set. We believe this is a case of “conflicting
7
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
suggestions”, where sequentially defined suggestions can
override each other’s impact if they are hard to disentangle
during self-refinement. We observe similar behavior in a
few experiments in the compiled strategies of Compile w/
Assert and C+Infer w/ Assert and display a few examples in Appendix D.
5.4.3. H3: COMPILING WITH LM ASSERTIONS
Then, we explore an exciting use case of LM Assertions
to serve as the filter and optimizer for few-shot demonstrations in prompt optimization. We evaluate all four tasks
on three settings: the baseline Compile, where the program utilizes a DSPy optimizer to bootstrap few-shot examples; Compile w/ Assert, where we enable suggestions
in the bootstrapping process only; and finally, C+Infer
w/ Assert, where suggestions and self-refinements are enabled in both bootstrapping and compiled program during
inference.
By comparing Compile with Compile w/ Assert, we find
that constructing few-shot examples that adhere to LM Assertions and show the self-refinement traces in the demonstrations makes the LM pipeline more likely to adhere to
the same guidelines, even without self-correction and backtracking. For example, in the TweetGen experiment, the
strategy compiled with suggestions has a 73.0% chance of
generating Engaging tweets, while the baseline few-shot
strategy only generates 2.0%. Overall, compiling with suggestions helps tweet generation gain 30.7% more overall
Quality. For other tasks, too, compiling with assertions almost always shows stronger performance in intrinsic and
extrinsic metrics.
A surprising finding for TweetGen is the decrease in engagement (Engaging) when compiling with assertions. We
inspect the responses of Compile w/ Assert and find that
the tweets are short, thus less engaging. We suspect the
following reasons: first, the user-provided instruction to fix
this suggestion may not be precise enough for an LLM to
follow. Second, as we mentioned in the analysis for H2,
some LM Assertions might conflict with each other, making discrete optimization of prompts challenging to satisfy
all constraints.
Finally, we put everything together and build C+Infer w/
Assert where suggestions are enabled at all times. This
setting performs best for most intrinsic metrics over all
other strategies due to the high-quality few-shot examples
collected and runtime self-refinement. In the MultiHopQA
question answering task, the compiled module with suggestions increases by 9.1% compared to the zero-shot baseline.
In QuizGen, the zero-shot baseline only generates 30.5%
valid quiz questions, while the final compiled program is
valid 87.2% of the time. Similarly, in TweetGen, we see a
47.5% increase. In LongFormQA cited long passage question answering, although all the suggestions are more likely
to pass, the answer inclusion (Has Answer) metric slightly
dropped; this suggests the opportunities to find better LM
Assertions for this program that can potentially influence
the downstream tasks.
6. Related Work
Programming with constraints is standard in most programming languages. Languages like Java (Bartetzko et al.,
2001) and Python (Python Software Foundation, 2023)
support assertions as first-class statements to perform runtime checks of certain properties. However, most runtime
checks can only be used to warn the programmer or abort
the execution.
Kang et al. (2020) proposed a concept called model assertions, which can be used to monitor the behavior of ML
models and to improve the quality of a model in training
through data collection and weak supervision. LM Assertions and the pipeline optimizations we perform with them
differ from model assertions in multiple ways: first, LM
Assertions can be used for backtracking an LM pipeline to
retry a failing module for self-refinement, which drastically
improves the performance of the pipeline; second, LM Assertions can be used as filters to select better examples for
few-shot learning; finally, LM Assertions aid generating
counterexamples and fixing traces, which further enhance
the LM pipeline to learn from past failures and improve.
More recent efforts on generating controllable outputs
for language models include LMQL (Beurer-Kellner et al.,
2023), NeMo Guardrails (Rebedea et al., 2023), etc. Although these systems permit some sort of computation constraints, they work on a single LM without consideration in
the LM pipeline setting, which misses the assertion-driven
optimization opportunities proposed by our work.
By integrating Python-style assertions, we ensure programmers can clearly express computational constraints
on DSPy programs and assert desired program behavior. These declarative constraints are leveraged in extensible and powerful ways to abstract and generalize notions of self-refinement and DSPy’s capabilities for prompt
optimization through compilation. We report on initial evaluation of an implementation that does so in this
work. Such self-refinement of LLMs (Madaan et al., 2023;
Shridhar et al., 2023) is central to this approach in making
DSPy autonomous and context-aware (Tyen et al., 2023).
Enforcing methodologies of iterative refinement using error
feedback (Xu et al., 2023) and utilizing reasoning capabilities through presenting past generations and feedback for
correction (Qiu et al., 2023) resonates with the objective of
DSPy assertions.
8
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
7. Conclusion
We have introduced LM Assertions, a new construct
for expressing arbitrary computational constraints on the
behavior of LMs when used as building blocks of
larger programs. We integrate LM Assertions into the
DSPy (Khattab et al., 2024) programming model, define
runtime retry semantics, and an implementation for them
that abstracts and generalizes LM self-refinement approaches to arbitrary steps in arbitrary pipelines. We also
discuss several other mechanisms that our LM Assertion
constructs can use to inform DSPy compilation into higherquality prompts that reduce the assertion failure rates. Our
evaluations show substantial gains on four case studies, reporting both intrinsic (i.e., assertion-specific) and extrinsic
(i.e., downstream) task metrics. By enabling DSPy programs to autonomously backtrack and self-correct and compile better few-shot examples, we hope to open avenues for
building more reliable LM programs at higher levels of abstraction than was previously possible.
Impact Statement
This paper presents work whose goal is to advance the field
of Machine Learning. There are many potential societal
consequences of our work, none which we feel must be
specifically highlighted here.
References
Bartetzko, D., Fischer, C., Möller, M., and Wehrheim,
H. Jass - java with assertions. In Havelund, K.
and Rosu, G. (eds.), Workshop on Runtime Verification, RV 2001, in connection with CAV 2001, Paris,
France, July 23, 2001, volume 55 of Electronic Notes
in Theoretical Computer Science, pp. 103–117. Elsevier, 2001. doi: 10.1016/S1571-0661(04)00247-6. URL
https://doi.org/10.1016/S1571-0661(04)00247-6.
Beurer-Kellner, L., Fischer, M., and Vechev, M. Prompting
is programming: A query language for large language
models. Proceedings of the ACM on Programming Languages, 7(PLDI):1946–1969, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems,
33:1877–1901, 2020.
Chase, H. LangChain, October 2022. URL
https://github.com/langchain-ai/langchain.
Hokamp, C. and Liu, Q. Lexically constrained decoding
for sequence generation using grid beam search. arXiv
preprint arXiv:1704.07138, 2017.
Hu, J. E., Khayrallah, H., Culkin, R., Xia, P., Chen, T.,
Post, M., and Van Durme, B. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1
(Long and Short Papers), pp. 839–850, 2019.
Kahn, G. Natural semantics. In Brandenburg, F., VidalNaquet, G., and Wirsing, M. (eds.), STACS 87, 4th Annual Symposium on Theoretical Aspects of Computer Science, Passau, Germany, February 19-21, 1987, Proceedings, volume 247 of Lecture Notes in Computer Science,
pp. 22–39. Springer, 1987. doi: 10.1007/BFB0039592.
URL https://doi.org/10.1007/BFb0039592.
Kang, D., Raghavan, D., Bailis, P., and Zaharia, M. Model
assertions for monitoring and improving ml models. Proceedings of Machine Learning and Systems, 2:481–496,
2020.
Khattab, O., Potts, C., and Zaharia, M. Baleen: Robust
multi-hop reasoning at scale via condensed retrieval. Advances in Neural Information Processing Systems, 34:
27670–27682, 2021.
Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang,
P., Potts, C., and Zaharia, M. Demonstrate-searchpredict: Composing retrieval and language models for
knowledge-intensive nlp, 2022.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A.,
Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and
Potts, C. Dspy: Compiling declarative language model
calls into self-improving pipelines. ICLR, 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao,
L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S.,
Yang, Y., et al. Self-refine: Iterative refinement with selffeedback. arXiv preprint arXiv:2303.17651, 2023.
Python Software Foundation. 7. simple statements.
https://docs.python.org/3/reference/simple_stmts.html#the-asser2023. Accessed: 2023-12-01.
Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhagavatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., et al.
Phenomenal yet puzzling: Testing inductive reasoning
capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08559, 2023.
Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., and Cohen, J. Nemo guardrails: A toolkit for controllable and
safe llm applications with programmable rails, 2023.
9
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C.,
and Zaharia, M. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint
arXiv:2112.01488, 2021.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R.,
and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on
Neural Information Processing Systems, 2023.
Shridhar, K., Sinha, K., Cohen, A., Wang, T., Yu, P., Pasunuru, R., Sachan, M., Weston, J., and Celikyilmaz, A.
The art of llm refinement: Ask, refine, and trust. arXiv
preprint arXiv:2311.07961, 2023.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal,
A. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv
preprint arXiv:2212.10509, 2022.
Tyen, G., Mansoor, H., Chen, P., Mak, T., and Carbune, V. ˘
Llms cannot find reasoning errors, but can correct them!
arXiv preprint arXiv:2311.08516, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F.,
Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought
prompting elicits reasoning in large language models.
Advances in Neural Information Processing Systems, 35:
24824–24837, 2022.
Xu, W., Deutsch, D., Finkelstein, M., Juraska, J., Zhang,
B., Liu, Z., Wang, W. Y., Li, L., and Freitag, M. Pinpoint, not criticize: Refining large language models
via fine-grained actionable feedback. arXiv preprint
arXiv:2311.09336, 2023.
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W.,
Salakhutdinov, R., and Manning, C. D. Hotpotqa: A
dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L.,
Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv
preprint arXiv:2305.10601, 2023.
10
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
A. Implementation
We introduce the proposed LM Assertions as plug-in interfaces in the DSPy framework according to the semantics in
Section 3. Next, we describe details about the design of our APIs and how we implement the semantics of both Assert and
Suggest in DSPy.
A.1. API Design
dspy.Assert(constraint: bool, msg: Optional[str],
backtrack: Optional[module])
dspy.Suggest(constraint: bool, msg: Optional[str],
backtrack: Optional[module])
We inherit a simple API design for LM Assertions. Both suggestions and assertions take a boolean value constraint as
input. Note that the computation for constraint can invoke other DSPy modules, potentially calling the LM to inform the
result for the constraint. Then, the user provides an optional error message, which is used for error logging and feedback
construction for backtracking and refinement. Finally, to enable backtracking, both dspy.Assert and dspy.Suggest contains
an optional backtrack argument, which points to the target module to backtrack to if the constraints fail.
A.2. Error Handlers
To implement various strategies of both assertions and suggestions for different use cases, we exploit Python’s native error
and exception handling.
We encode error-handling logic as function wrappers. To that extent, we provide a primitive constraint_tranform to
wrap any DSPy module with handlers. When the constraints in dspy.Assert and dspy.Suggest are false, they raise
AssertionError and SuggestionError, respectively. Then, the dedicated error handling clause in the function wrapper can
reroute the errors to the correct semantics.
As a result, the program’s behavior after an assertion or suggestion error is completely controlled by the handlers used.
To support flexibility in using LM Assertions with DSPy, we implement several composable handlers, such as disabling
suggestions and assertions, suppressing assertion errors with logging, etc.
The default handlers follow the semantics as described in Section 3 to enable self-refinement. That is, we allow R
retry attempts for AssertionError and SuggestionError by backtracking to the failing LM. After R retry attempts, an
AssertionError will be raised while SuggestionError will only be logged silently.
A.3. Backtracking
To implement backtracking in DSPy, we introduce a new auxiliary meta-module called Retry. This module is a lightweight
wrapper for any DSPy module, providing additional information about all previously unsuccessful predictions. When DSPy
determines the need to backtrack to a specific module, it calls Retry. As shown in Figure 1, the Retry module automatically
adds the failed predictions and the corresponding user-defined error messages raised to the prompt. Then, the LM pipeline
can backtrack to the previously failed module with this updated prompt. In this way, the original module to refine is selfaware and informed of past attempts and errors on them. Consequently, this empowers the LM to develop more informed
and error-avoiding generations in subsequent iterations of self-refinement.
B. Case Studies
B.1. LongFormQA
B.1.1. TASK
In this task, we build on the Multi-Hop QA (Figure 1) task by expecting long-form answers to questions that include
citations to referenced context.
Figure 3 shows an implementation of this task in DSPy. As shown, it is nearly identical to Figure 1 outside of the introduction of a new dspy.ChainOfThought module that generates cited paragraphs referencing the retrieved context. With this task
11
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
class LongFormQAWithAssertions(dspy.Module):
def __init__(self, passages_per_hop=3):
self.retrieve = dspy.Retrieve(k=passages_per_hop)
self.generate_query = dspy.ChainOfThought("context, question -> query")
self.generate_cited_paragraph = dspy.ChainOfThought("context, question -> paragraph") #has field description to
include citations
def forward(self, question):
context = []
for hop in range(2):
query = self.generate_query(context=context, question=question).query
context += self.retrieve(query).passages
pred = self.generate_cited_paragraph(context=context, question=question)
dspy.Suggest(citations_check(pred.paragraph), "Every 1-2 sentences should have citations: ’text... [x].’")
for line, citation in get_lines_and_citations(pred, context):
dspy.Suggest(is_faithful(line, citation), f"Your output should be based on the context: ’{citations}’.")
return pred
Figure 3. DSPy program with LM Assertions for long-form paragraph multi-hop question answering task with a retriever. We introduce
two suggestions: (1) asserting every 1-2 sentences has a citation; (2) every text segment preceding a citation is faithful to its cited
reference.
and LM pipeline, we aim not just to produce accurate answers but to generate well-structured long-form answers that are
faithful to the retrieved context.
B.1.2. METRICS
We assess intrinsic performance using a sophisticated metric, Citation Faithfulness. In this metric, a small DSPy program
uses the LM to check if the text preceding each citation appropriately supports the cited context. Our check outputs a
boolean for faithfulness, which is then averaged across the citations in the output to aggregate a metric for evaluation. As
extrinsic metrics, we use: (1) Answer Correctness, verifying if the gold answer is correctly incorporated; (2) Citation
Precision, gauging the proportion of correctly cited titles; and (3) Citation Recall, measuring the coverage of gold titles
cited.
B.1.3. CONSTRAINTS SPECIFIED
As a simple initial check, we include a Suggest statement that requires every 1–2 of sentences generated has citations in
an intended format. This is checked by a simple Python function citations_check. As a more sophisticated check, we
Suggest that the text preceding any citation must be faithful to the cited context, ensuring that the reference text accurately
represents the content of the cited information. Since this is a fuzzy condition, we employ a small DSPy program (one
that uses the LM) to perform this check. Notably, the robust API design of Suggest allows the user to specify arbitrary
expressions as conditional checks, such as an LM call. The goal of this Suggest statement is to ensure that all sentences
are appropriately attributed to correct supporting sources.
B.2. QuizGen
B.2.1. TASK
We introduce a new task stemming from the HotPotQA dataset in turning questions from the dataset into quiz questions by
generating possible answer choices for the question in a JSON format.
This task is represented by a very simple program in DSPy with a dspy.ChainOfThought module that generates a set of
answer choices based on a defined question-answer pair and a specified number of choices. To ensure well-defined quiz
questions, we aim for this task to adhere to consistent formatting and offer a set of plausible distractor answer choices
12
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
class QuizChoiceGenerationWithAssertions(dspy.Module):
def __init__(self):
super().__init__()
self.generate_choices = dspy.ChainOfThought("question, correct_answer, number_of_choices -> answer_choices"
) #has specified instruction to guide inputs -> outputs
def forward(self, question, answer):
choice_string = self.generate_choices(question=question, correct_answer=answer, number_of_choices=
number_of_choices).answer_choices
dspy.Suggest(format_checker(choice_string), "The format of the answer choices should be in JSON format.
Please revise accordingly.")
dspy.Suggest(is_correct_answer_included(answer, choice_string), "The answer choices do not include the
correct answer to the question. Please revise accordingly.")
plausibility_question = "Are the distractors in the answer choices plausible and not easily identifiable as
incorrect?"
plausibility_assessment = dspy.Predict("question, answer_choices, assessment_question -> assessment_answer"
)(question=question, answer_choices=choice_string, assessment_question=plausibility_question)
dspy.Suggest(is_plausibility_yes(plausibility_assessment.assessment_answer), "The answer choices are not
plausible distractors or are too easily identifiable as incorrect. Please revise to provide more challenging
and plausible distractors.")
return dspy.Prediction(choices = choice_string)
Figure 4. DSPy program with LM Assertions for quiz question choice generation. We introduce 3 suggestions: (1) asserting JSON
format; (2) correct answer is included; (3) plausible distractor choices are present.
alongside the actual correct answer to the question.
B.2.2. METRICS
We assess the task’s intrinsic performance across the following metrics: (1) Valid Formatting; (2) Correct Answer Inclusion;
and (3) Choices’ Plausibility.
We verify consistent formatting by parsing the generated answer choices and checking their consistency to maintain JSON
formatting of key-value pairs.
We similarly ensure that the outputted answer choices include the correct answer corresponding to the respective question
from the HotPotQA dataset.
For determining the plausibility of the distractor choices, we build a DSPy program that relies on the LM to assess the
quality of the answer choice questions. This relies on the inputs: question, generated answer choices, and the assessment
question we provide: Are the distractors in the answer choices plausible and not easily identifiable as incorrect? This
plausibility verification then outputs an assessment answer of whether the distractors are plausible or not.
For the extrinsic metric, we define a composite scoring metric that considers the intrinsic metrics above. The metric
imposes that the conditions of valid formatting and correct answer inclusion are met, thereby ensuring valid quiz questions.
When this case is met for the generated answer choices, we return an average score over all three of the intrinsic metrics.
If either of these conditions is not met, the score defaults to 0.
B.2.3. CONSTRAINTS SPECIFIED
For the simple check of Valid Formatting, we include a Suggest statement that requires the format of the answer choices
to be in JSON format. This is checked by a simple Python function format_checker.
Similarly, we verify Correct Answer Inclusion with the Suggest statement that indicates if the answer choices do not
13
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
include the correct answer. This is checked by a simple Python function is_correct_answer_included.
To verify the plausibility of the answer choices to reflect strong distractor choices alongside the correct choice, we employ
the Suggest statement to indicate if the answer choices are not plausible distractors or are too easily identifiable as incorrect.
With a DSPy program in place to assess the choices, this Suggest statement ensures that all of the answer choices are
plausible distractors.
B.3. TweetGen
B.3.1. TASK
We introduce another new task derived from the HotPotQA dataset in generating tweets to answer questions.
class TweetGenerationWithAssertions(dspy.Module):
def __init__(self):
super().__init__()
self.generate_tweet = dspy.ChainOfThought("question, context -> tweet") #has specified instruction to guide
inputs -> outputs
def forward(self, question, answer):
context = []
generate_query = [dspy.ChainOfThought("context, question -> query") for _ in range(2)]
retrieve = dspy.Retrieve(k=3)
for hop in range(2):
query = generate_query[hop](context=context, question=question).query
passages = retrieve(query).passages
context = deduplicate(context + passages)
generated_tweet = self.generate_tweet(question=question, context=context).tweet
dspy.Suggest(has_no_hashtags(generated_tweet), f"Please revise the tweet to remove hashtag phrases
following it.")
dspy.Suggest(is_within_length_limit(generated_tweet, 280), f"Please ensure the tweet is within {280}
characters.")
dspy.Suggest(has_correct_answer(generated_tweet, answer), "The tweet does not include the correct answer to
the question. Please revise accordingly.")
engaging_question = "Does the assessed text make for a self-contained, engaging tweet? Say no if it is not
engaging."
engaging_assessment = dspy.Predict("context, assessed_text, assessment_question -> assessment_answer")(
context=context, assessed_text=generated_tweet, assessment_question=engaging_question)
dspy.Suggest(is_assessment_yes(engaging_assessment.assessment_answer), "The text is not engaging enough.
Please revise to make it more captivating.")
faithful_question = "Is the assessed text grounded in the context? Say no if it includes significant facts
not in the context."
faithful_assessment = dspy.Predict("context, assessed_text, assessment_question -> assessment_answer")(
context=’N/A’, assessed_text=generated_tweet, assessment_question=faithful_question)
dspy.Suggest(is_assessment_yes(faithful_assessment.assessment_answer), "The text contains unfaithful
elements or significant facts not in the context. Please revise for accuracy.")
return dspy.Prediction(generated_tweet=generated_tweet, context=context)
Figure 5. DSPy program with LM Assertions for tweet generation. We introduce 5 suggestions: (1) asserting no hashtags; (2) correct
answer is included; (3) tweet is within character limit; (4) tweet is engaging; (5) tweet is faithful to context.
This task mirrors the MultiHopQA task with the addition of a dspy.ChainOfThought module layer to utilize the retrieved
context and corresponding question to generate a tweet that effectively answers the question. We aim for the task to ensure
the tweet not only answers the question but is engaging to the reader and faithful to its relevant context.
B.3.2. METRICS
We assess the task’s intrinsic performance across various metrics: (1) No Hashtags; (2) Correct Answer Inclusion; (3)
Within Length; (4) Engaging; (5) Faithful.
We impose an intrinsic constraint to ensure none of the tweets have hashtags, ensuring all tweets maintain a consistent
14
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
tweeting style.
As we do with QuizChoiceGeneration, we ensure the outputted tweet includes the correct answer corresponding to the
respective question from the HotPotQA dataset.
We also ensure that the generated tweet adheres to a character count limit of 280 characters to model sample tweet behavior.
For determining the engagement of the tweet, we build a DSPy program that relies on the LM to assess this. This relies on
the inputs: question, context, generated tweet, and the assessment question we provide: Does the assessed text make for a
self-contained, engaging tweet? This verification outputs its assessment of whether the tweet is engaging in relation to its
corresponding question and retrieved context.
We perform a similar assessment for the tweet’s faithfulness, with the simple modification to the assessment question: Is
the assessed text grounded in the context?
For the extrinsic metric, we define a composite scoring metric that considers all of the intrinsic metrics above. The metric
imposes that the most relevant intrinsic conditions of a well-formed tweet are met, particularly if the tweet contains the
correct answer to the question and is within the tweeting character limit. When this case is met for the generated answer
choices, we return an average score over all five of the intrinsic metrics. If either of these conditions is not met, the score
defaults to 0.
B.3.3. CONSTRAINTS SPECIFIED
To verify that the tweet contains no hashtags, we include a Suggest statement that requires the tweet to be generated without
any hashtag phrases. This is checked by a simple Python function through regex checks in has_no_hashtags.
To verify the generated tweet adheres to the character limits, we impose this through the Suggest statement to ensure that
the tweet is under the specified character limit, which we specify as 280 in our experiments. This is checked by a simple
Python function is_within_length_limit.
Similarly, we verify Correct Answer Inclusion with the Suggest statement that indicates if the answer choices do not
include the correct answer. This is checked by a simple Python function has_correct_answer.
To verify the engagement level of the generated tweet, we employ the Suggest statement to simply indicate whether the
tweet is engaging enough as determined by the LM and DSPy program in place to assess engagement.
We conduct a similar approach for faithfulness as well, checking for the tweet’s faithfulness to its retrieved context.
C. Impact on Using Different LLM Instructions
We explore comparative tests in the specified instructions for the case studies mentioned above. We differentiate between a
primitive instruction that aims to simply specify a task’s objective and a complete instruction that accounts for the respective
intrinsic and extrinsic metric measured for the task. These tests are conducted specifically on the TweetGen and QuizGen
tasks which encompass more complex metrics. Our experiments on the complete instructions are presented in Figure 2
while we demonstrate our results on the primitive instructions below.
C.1. TweetGen
Primitive instruction: "Generate a tweet that effectively answers a question."
Complete instruction with metrics accounted for: "Generate an engaging tweet that effectively answers a question staying
faithful to the context, is less than 280 characters, and has no hashtags."
C.2. QuizGen
Primitive instruction: "Generate answer choices for the specified question."
Complete instruction with metrics accounted for: "Generate answer choices in JSON format that include the correct answer
and plausible distractors for the specified question."
15
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
TweetGen w/ Primitive Instructions (Dev/Test)
Strategy No "#" Has Answer Concise Engaging Faithful Quality
Vanilla 3.3 / 3.0 53.7 / 48.2 96.3 / 97.0 35.7 / 36.4 80.0 / 81.2 33.7 / 30.4
Infer w/ Assert 49.3 / 49.6 50.3 / 41.8 92.0 / 92.4 45.3 / 41.0 72.3 / 74.0 34.3 / 27.8
Compile 0.0 / 0.2 55.7 / 46.2 100 / 99.6 47.3 / 46.6 78.3 / 76.8 36.7 / 30.8
Compile w/ Assert 98.7 / 97.4 55.0 / 45.8 99.3 / 99.0 1.3 / 2.6 65.3 / 70.0 40.4 / 34.3
C+Infer w/ Assert 41.3 / 41.0 55.7 / 48.2 94.7 / 93.8 54.3 / 60.2 76.7 / 81.2 40.3 / 35.0
QuizGen w/ Primitive Instructions (Dev/Test)
Strategy Correct JSON Has Answer Citation Precision Validity
Vanilla 1.3 / 2.8 1.3 / 2.6 61.3 / 61.8 1.2 / 2.3
Infer w/ Assert 91.7 / 93.4 73.3 / 72.6 75.0 / 69.8 69.8 / 68.0
Compile 100 / 100 94.3 / 89.8 72.7 / 67.4 85.4 / 80.1
Compile w/ Assert 100 / 100 95.7 / 91.4 63.0 / 57.0 83.7 / 78.5
C+Infer w/ Assert 100 / 100 93.3 / 89.4 73.7 / 67.8 85.8 / 81.1
Figure 6. Evaluation of TweetGen and QuizGen task using the primitive instruction. The LM pipeline configuration are described in
Table 1. For each task, we use the same LM pipeline program except for the LM Assertions. Extrinsic metrics (downstream application
performance) are highlighted in grey. For each metric, higher is always better. The highest value in each column is bold.
C.3. Discussion
Based on these experiments on primitive instructions, we discovered that when the baseline pipeline only has access to highlevel and generic instructions, it is almost impossible for the pipeline to follow the underlying constraints. For example,
in QuizGen, the baseline Vanilla strategy only generates 2.8% of quizzes with Correct JSON format and 2.6% of quizzes
that contains the correct answer.
However, for our assertion-driven optimization enabled pipelines, the performance on primitive experiments are still comparable to the counter-part with complex instructions. This indicates that model pipelines with LM Assertions and assertiondriven optimizations are less sensitive to instructions, requiring less effort on manual prompt tuning.
D. Interesting Examples of Correlated Constraints
This section showcases selected examples that demonstrate potential conflicts between Assertions specified in the
TweetGen task. Each set of examples highlights instances of two configurations that produce outputs to achieve success on
the TweetGen Metrics (Appendix B.3.2) and face the nuanced challenge of balancing these constraints in execution.
Example 1 Both results accurately include the correct answer. However, Compile w/ Assert does not pass the Engaging constraint despite the specified assertion but ensures the No "#" constraint is met whereas C+Infer w/ Assert
violates the No "#" constraint while satisfying the Engaging constraint.
Question: What was the name of the treaty that made Hungary a landlocked state which contained the Kolozsvar
Ghetto?
Compile w/ Assert: The Treaty of Trianon made Hungary landlocked and led to the existence of the
Kolozsvar Ghetto. This lesser-known ghetto was located in Kolozsvár, Kingdom of Hungary (now ClujNapoca, Romania).
C+Infer w/ Assert: The Treaty of Trianon, signed in 1920, reshaped Hungarys destiny, leading to its ´
landlocked status and the emergence of the Kolozsvar Ghetto. Uncover the gripping tale of this pivotal
moment in history and its lasting effects. #HistoryUnveiled
Answer: Treaty of Trianon
16
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
Example 2 Both results do not include the correct answer. However, both configurations produce engaging outputs. Yet
C+Infer w/ Assert does not conform to the No "#" constraint while Compile w/ Assert does.
Question: Which American car rental company is also a member of the Association of Car Rental Industry Sytems
Standards?
Compile w/ Assert: "Enterprise, a leading American car rental company, is also a proud member of
the Association of Car Rental Industry Systems Standards. Rent with confidence and enjoy a seamless
experience with Enterprise!"
C+Infer w/ Assert: "Looking for a car rental company that meets industry standards? Look no further than ACRISS member Enterprise! With their commitment to excellence, you can trust them for a
seamless rental experience. Get ready for a smooth ride! #CarRental #Enterprise"
Answer: Budget Rent a Car
17""