Repository: stanfordnlp/dspy (language: python)

Issue:
Bug in Custom LM tutorial: https://dspy-docs.vercel.app/docs/building-blocks/language_models#creating-custom-lm-class
When calling  `super().__init__(model)`, the `self.model` gets overwritten with a string.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/build_and_release.yml b/.github/workflows/build_and_release.yml
index 7900d79df..da234b787 100644
--- a/.github/workflows/build_and_release.yml
+++ b/.github/workflows/build_and_release.yml
@@ -11,7 +11,7 @@ jobs:
     outputs:  
       version: ${{ steps.extract_tag.outputs.tag }}  
     steps:  
-      - uses: actions/checkout@v2  
+      - uses: actions/checkout@v4
       - id: extract_tag  
         name: Extract tag name  
         run: echo "::set-output name=tag::$(echo $GITHUB_REF | cut -d / -f 3)"  
@@ -24,7 +24,7 @@ jobs:
     permissions:
       id-token: write  # IMPORTANT: mandatory for trusted publishing
     steps:
-      - uses: actions/checkout@master
+      - uses: actions/checkout@v4
       - name: Set up Python 3.9
         uses: actions/setup-python@v3
         with:
@@ -54,7 +54,6 @@ jobs:
         uses: pypa/gh-action-pypi-publish@release/v1 # This requires a trusted publisher to be setup in pypi/testpypi
         with: 
           repository-url: https://test.pypi.org/legacy/
-          verbose: true
 
   build-and-publish-pypi:
     needs: [extract-tag, build-and-publish-test-pypi]
@@ -64,34 +63,36 @@ jobs:
     permissions:
       id-token: write  # IMPORTANT: mandatory for trusted publishing
     steps:
-      - uses: actions/checkout@master
+      - uses: actions/checkout@v4
       - name: Set up Python 3.9
         uses: actions/setup-python@v3
         with:
           python-version: "3.9"
       - name: Install dependencies
         run: python3 -m pip install setuptools wheel twine
-      - name: Update version in setup.py
+      - name: Update version in setup.py (dspy)
         run: sed -i '/#replace_package_version_marker/{n;s/version="[^"]*"/version="${{ needs.extract-tag.outputs.version }}"/;}' setup.py
       - name: Update version in pyproject.toml
         run: sed -i '/#replace_package_version_marker/{n;s/version="[^"]*"/version="${{ needs.extract-tag.outputs.version }}"/;}' pyproject.toml
-      # Publish to dspy-ai
-      - name: Update package name in setup.py  
-        run: sed -i '/#replace_package_name_marker/{n;s/name="[^"]*"/name="dspy-ai"/;}' setup.py
-      - name: Update package name in pyproject.toml
-        run: sed -i '/#replace_package_name_marker/{n;s/name="[^"]*"/name="dspy-ai"/;}' pyproject.toml
-      - name: Build a binary wheel
-        run: python3 setup.py sdist bdist_wheel
-      - name: Publish distribution 📦 to PyPI (dspy-ai)
-        uses: pypa/gh-action-pypi-publish@release/v1 # This requires a trusted publisher to be setup in pypi
       # Publish to dspy
       - name: Update package name in setup.py  
-        run: sed -i '/#replace_package_name_marker/{n;s/name="[^"]*"/name="dspy"/;}' setup.py
+        run: |
+          sed -i '/#replace_package_name_marker/{n;s/name="[^"]*"/name="dspy"/;}' setup.py
       - name: Update package name in pyproject.toml
         run: sed -i '/#replace_package_name_marker/{n;s/name="[^"]*"/name="dspy"/;}' pyproject.toml
       - name: Build a binary wheel
         run: python3 setup.py sdist bdist_wheel
       - name: Publish distribution 📦 to PyPI (dspy)
         uses: pypa/gh-action-pypi-publish@release/v1 # This requires a trusted publisher to be setup in pypi
-
-    
+      # Publish to dspy-ai
+      - name: Update version in setup.py (dspy-ai)
+        run: sed -i '/#replace_package_version_marker/{n;s/version="[^"]*"/version="${{ needs.extract-tag.outputs.version }}"/;}' ./dspy/.internal_dspyai/setup.py
+      - name: Update package name in setup.py  
+        run: sed -i '/#replace_package_name_marker/{n;s/name="[^"]*"/name="dspy-ai"/;}' ./dspy/.internal_dspyai/setup.py
+      - name: Update dspy dependency version in setup.py  
+        run: |
+          sed -i '/#replace_dspy_version_marker/{n;s/dspy==[^"]*/dspy==${{ needs.extract-tag.outputs.version }}/;}' ./dspy/.internal_dspyai/setup.py  
+      - name: Build a binary wheel
+        run: python3 ./dspy/.internal_dspyai/setup.py sdist bdist_wheel
+      - name: Publish distribution 📦 to PyPI (dspy-ai)
+        uses: pypa/gh-action-pypi-publish@release/v1 # This requires a trusted publisher to be setup in pypi
\ No newline at end of file
diff --git a/README.md b/README.md
index dac7adaa2..25b086013 100644
--- a/README.md
+++ b/README.md
@@ -116,6 +116,7 @@ The DSPy documentation is divided into **tutorials** (step-by-step illustration
 - [DSPy: Not Your Average Prompt Engineering](https://jina.ai/news/dspy-not-your-average-prompt-engineering), why it's crucial for future prompt engineering, and yet why it is challenging for prompt engineers to learn.
 - **Tracing & Optimization Tracking in DSPy** with Parea AI: [Tutorial on tracing & evaluating a DSPy RAG program](https://docs.parea.ai/tutorials/dspy-rag-trace-evaluate/tutorial)
 - [DSPy: Not Your Average Prompt Engineering](https://jina.ai/news/dspy-not-your-average-prompt-engineering), why it's crucial for future prompt engineering, and yet why it is challenging for prompt engineers to learn.
+- [CS 194/294-196 (LLM Agents) - Lecture 5, Omar Khattab](https://www.youtube.com/watch?v=JEMYuzrKLUw)
 
 ### B) Guides
 
diff --git a/docs/api/assertions.md b/docs/api/assertions.md
index 07dfa55f4..edb25013a 100644
--- a/docs/api/assertions.md
+++ b/docs/api/assertions.md
@@ -101,12 +101,14 @@ We can declare these validation checks through `dspy.Suggest` statements (as we
 dspy.Suggest(
     len(query) <= 100,
     "Query should be short and less than 100 characters",
+    target_module=self.generate_query
 )
 
 dspy.Suggest(
     validate_query_distinction_local(prev_queries, query),
     "Query should be distinct from: "
     + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),
+    target_module=self.generate_query
 )
 ```
 
@@ -133,12 +135,14 @@ class SimplifiedBaleenAssertions(dspy.Module):
             dspy.Suggest(
                 len(query) <= 100,
                 "Query should be short and less than 100 characters",
+                target_module=self.generate_query
             )
 
             dspy.Suggest(
                 validate_query_distinction_local(prev_queries, query),
                 "Query should be distinct from: "
                 + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),
+                target_module=self.generate_query
             )
 
             prev_queries.append(query)
diff --git a/docs/docs/building-blocks/1-language_models.md b/docs/docs/building-blocks/1-language_models.md
index 192117275..2a1bc2db8 100644
--- a/docs/docs/building-blocks/1-language_models.md
+++ b/docs/docs/building-blocks/1-language_models.md
@@ -155,31 +155,47 @@ If there is not much overlap in features between your LM and LiteLLM it's better
 Let's create an LM for Gemini using `google-generativeai` package from scratch:
 
 ```python
-from dspy import LM
+import os
+import dspy
 import google.generativeai as genai
 
-
-class GeminiLM:
-    def __init__(self, model, api_key, endpoint):
-        genai.configure(api_key=os.environ["API_KEY"] or api_key)
-        self.model = genai.GenerativeModel(model)
+class GeminiLM(dspy.LM):
+    def __init__(self, model, api_key=None, endpoint=None, **kwargs):
+        genai.configure(api_key=os.environ["GEMINI_API_KEY"] or api_key)
 
         self.endpoint = endpoint
         self.history = []
-        super().__init__(model)
-    
+
+        super().__init__(model, **kwargs)
+        self.model = genai.GenerativeModel(model)
 
     def __call__(self, prompt=None, messages=None, **kwargs):
-        if isinstance(prompt, str):
-            prompt = [prompt]
+        # Custom chat model working for text completion model
+        prompt = '\n\n'.join([x['content'] for x in messages] + ['BEGIN RESPONSE:'])
 
         completions = self.model.generate_content(prompt)
         self.history.append({"prompt": prompt, "completions": completions})
-         
+        
+        # Must return a list of strings
+        return [completions.candidates[0].content.parts[0].text]
 
     def inspect_history(self):
         for interaction in self.history:
             print(f"Prompt: {interaction['prompt']} -> Completions: {interaction['completions']}")
+
+lm = GeminiLM("gemini-1.5-flash", temperature=0)
+dspy.configure(lm=lm)
+
+qa = dspy.ChainOfThought("question->answer")
+qa(question="What is the capital of France?")
+```
+
+**Output:**
+```text
+Prediction(
+    reasoning='France is a country in Western Europe. Its capital city is Paris.',
+    answer='Paris'
+)
 ```
 
 The above example is the simplest form of LM. You can add more options to tweak generation config and even control the generated output based on your requirement.
@@ -323,4 +339,4 @@ The above example is a simple Adapter that converts the input to uppercase befor
 
 ### Overriding `__call__` method
 
-To gain control over usage of format and parse and even more fine-grained control over the flow of input from signature to outputs you can override `__call__` method and implement your custom flow. Although for most cases only implementing `parse` and `format` function will be fine.
\ No newline at end of file
+To gain control over usage of format and parse and even more fine-grained control over the flow of input from signature to outputs you can override `__call__` method and implement your custom flow. Although for most cases only implementing `parse` and `format` function will be fine.
diff --git a/docs/docs/building-blocks/7-assertions.md b/docs/docs/building-blocks/7-assertions.md
index ae6a49859..4b0297f6b 100644
--- a/docs/docs/building-blocks/7-assertions.md
+++ b/docs/docs/building-blocks/7-assertions.md
@@ -99,12 +99,14 @@ We can declare these validation checks through `dspy.Suggest` statements (as we
 dspy.Suggest(
     len(query) <= 100,
     "Query should be short and less than 100 characters",
+    target_module=self.generate_query
 )
 
 dspy.Suggest(
     validate_query_distinction_local(prev_queries, query),
     "Query should be distinct from: "
     + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),
+    target_module=self.generate_query
 )
 ```
 
@@ -131,12 +133,14 @@ class SimplifiedBaleenAssertions(dspy.Module):
             dspy.Suggest(
                 len(query) <= 100,
                 "Query should be short and less than 100 characters",
+                target_module=self.generate_query
             )
 
             dspy.Suggest(
                 validate_query_distinction_local(prev_queries, query),
                 "Query should be distinct from: "
                 + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),
+                target_module=self.generate_query
             )
 
             prev_queries.append(query)
diff --git a/docs/docs/cheatsheet.md b/docs/docs/cheatsheet.md
index d6fef8b13..630b9a563 100644
--- a/docs/docs/cheatsheet.md
+++ b/docs/docs/cheatsheet.md
@@ -392,6 +392,7 @@ compiled_program_optimized_bayesian_signature = teleprompter.compile(your_dspy_p
 Note: detailed documentation can be found [here](/docs/deep-dive/optimizers/miprov2.md). `MIPROv2` is the latest extension of `MIPRO` which includes updates such as (1) improvements to instruction proposal and (2) more efficient search with minibatching.
 
 #### Optimizing with MIPROv2
+This shows how to perform an easy out-of-the box run with `auto=light`, which configures many hyperparameters for you and performs a light optimization run. You can alternatively set `auto=medium` or `auto=heavy` to perform longer optimization runs. The more detailed `MIPROv2` documentation [here](/docs/deep-dive/optimizers/miprov2.md) also provides more information about how to set hyperparameters by hand.
 ```python
 # Import the optimizer
 from dspy.teleprompt import MIPROv2
@@ -399,10 +400,7 @@ from dspy.teleprompt import MIPROv2
 # Initialize optimizer
 teleprompter = MIPROv2(
     metric=gsm8k_metric,
-    num_candidates=7,
-    init_temperature=0.5,
-    verbose=False,
-    num_threads=4,
+    auto="light", # Can choose between light, medium, and heavy optimization runs
 )
 
 # Optimize program
@@ -412,10 +410,6 @@ optimized_program = teleprompter.compile(
     trainset=trainset,
     max_bootstrapped_demos=3,
     max_labeled_demos=4,
-    num_trials=15,
-    minibatch_size=25,
-    minibatch_full_eval_steps=10,
-    minibatch=True, 
     requires_permission_to_run=False,
 )
 
@@ -429,35 +423,31 @@ evaluate(optimized_program, devset=devset[:])
 
 #### Optimizing instructions only with MIPROv2 (0-Shot)
 ```python
-# Import optimizer
+# Import the optimizer
 from dspy.teleprompt import MIPROv2
 
-# Initialize optimizer 
+# Initialize optimizer
 teleprompter = MIPROv2(
     metric=gsm8k_metric,
-    num_candidates=15,
-    init_temperature=0.5,
-    verbose=False,
-    num_threads=4,
+    auto="light", # Can choose between light, medium, and heavy optimization runs
 )
 
-# Perform optimization
-print(f"Optimizing program with MIPRO (0-Shot)...")
-zeroshot_optimized_program = teleprompter.compile(
+# Optimize program
+print(f"Optimizing program with MIPRO...")
+optimized_program = teleprompter.compile(
     program.deepcopy(),
     trainset=trainset,
-    max_bootstrapped_demos=0, # setting demos to 0 for 0-shot optimization
+    max_bootstrapped_demos=0,
     max_labeled_demos=0,
-    num_trials=15,
-    minibatch=False, 
     requires_permission_to_run=False,
 )
 
+# Save optimize program for future use
+optimized_program.save(f"mipro_optimized")
 
-zeroshot_optimized_program.save(f"mipro_0shot_optimized")
-
-print(f"Evaluate optimized program...")
-evaluate(zeroshot_optimized_program, devset=devset[:])
+# Evaluate optimized program
+print(f"Evluate optimized program...")
+evaluate(optimized_program, devset=devset[:])
 ```
 ### Signature Optimizer with Types
 
diff --git a/docs/docs/deep-dive/modules/assertions.mdx b/docs/docs/deep-dive/modules/assertions.mdx
index 93fac43a0..acda69af4 100644
--- a/docs/docs/deep-dive/modules/assertions.mdx
+++ b/docs/docs/deep-dive/modules/assertions.mdx
@@ -104,12 +104,14 @@ We can declare these validation checks through `dspy.Suggest` statements (as we
 dspy.Suggest(
     len(query) <= 100,
     "Query should be short and less than 100 characters",
+    target_module=self.generate_query
 )
 
 dspy.Suggest(
     validate_query_distinction_local(prev_queries, query),
     "Query should be distinct from: "
     + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),
+    target_module=self.generate_query
 )
 ```
 
@@ -136,12 +138,14 @@ class SimplifiedBaleenAssertions(dspy.Module):
             dspy.Suggest(
                 len(query) <= 100,
                 "Query should be short and less than 100 characters",
+                target_module=self.generate_query
             )
 
             dspy.Suggest(
                 validate_query_distinction_local(prev_queries, query),
                 "Query should be distinct from: "
                 + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),
+                target_module=self.generate_query
             )
 
             prev_queries.append(query)
diff --git a/docs/docs/deep-dive/optimizers/miprov2.md b/docs/docs/deep-dive/optimizers/miprov2.md
index 1ef388ea1..759f003f1 100644
--- a/docs/docs/deep-dive/optimizers/miprov2.md
+++ b/docs/docs/deep-dive/optimizers/miprov2.md
@@ -6,6 +6,7 @@
    - [Setting up a Sample Pipeline](#setting-up-a-sample-pipeline)
    - [Optimizing with MIPROv2](#optimizing-with-miprov2)
    - [Optimizing instructions only with MIPROv2 (0-Shot)](#optimizing-instructions-only-with-miprov2-0-shot)
+   - [Optimizing with MIPROv2 (Advanced)](#optimizing-with-miprov2-advanced)
 3. [Parameters](#parameters)
    - [Initialization Parameters](#initialization-parameters)
    - [Compile Method Specific Parameters](#compile-method-specific-parameters)
@@ -69,7 +70,7 @@ evaluate(program, devset=devset[:])
 Now we have the baseline pipeline ready to use, so let's try using the `MIPROv2` optimizer to improve our pipeline's performance!
 
 ### Optimizing with `MIPROv2`
-Here we show how to import, initialize, and compile our program with optimized few-shot examples and instructions using `MIPROv2`.
+To get started with `MIPROv2`, we'd recommend using the `auto` flag, starting with a `light` optimization run. This will set up hyperparameters for you to do a light optimization run on your program.
 
 ```python
 # Import the optimizer
@@ -78,10 +79,7 @@ from dspy.teleprompt import MIPROv2
 # Initialize optimizer
 teleprompter = MIPROv2(
     metric=gsm8k_metric,
-    num_candidates=7,
-    init_temperature=0.5,
-    verbose=False,
-    num_threads=4,
+    auto="light", # Can choose between light, medium, and heavy optimization runs
 )
 
 # Optimize program
@@ -91,10 +89,6 @@ optimized_program = teleprompter.compile(
     trainset=trainset,
     max_bootstrapped_demos=3,
     max_labeled_demos=4,
-    num_trials=15,
-    minibatch_size=25,
-    minibatch_full_eval_steps=10,
-    minibatch=True, 
     requires_permission_to_run=False,
 )
 
@@ -106,41 +100,72 @@ print(f"Evluate optimized program...")
 evaluate(optimized_program, devset=devset[:])
 ```
 
-### Optimizing instructions only with `MIPROv2` (0-Shot)
+#### Optimizing instructions only with `MIPROv2` (0-Shot)
 
 In some cases, we may want to only optimize the instruction, rather than including few-shot examples in the prompt. The code below demonstrates how this can be done using `MIPROv2`. Note that the key difference involves setting `max_labeled_demos` and `max_bootstrapped_demos` to zero.
 ```python
-# Import optimizer
+# Import the optimizer
 from dspy.teleprompt import MIPROv2
 
-# Initialize optimizer 
+# Initialize optimizer
+teleprompter = MIPROv2(
+    metric=gsm8k_metric,
+    auto="light", # Can choose between light, medium, and heavy optimization runs
+)
+
+# Optimize program
+print(f"Optimizing zero-shot program with MIPRO...")
+zeroshot_optimized_program = teleprompter.compile(
+    program.deepcopy(),
+    trainset=trainset,
+    max_bootstrapped_demos=0, # ZERO FEW-SHOT EXAMPLES
+    max_labeled_demos=0, # ZERO FEW-SHOT EXAMPLES
+    requires_permission_to_run=False,
+)
+
+# Save optimize program for future use
+zeroshot_optimized_program.save(f"mipro_zeroshot_optimized")
+
+# Evaluate optimized program
+print(f"Evluate optimized program...")
+evaluate(zeroshot_optimized_program, devset=devset[:])
+```
+
+#### Optimizing with `MIPROv2` (advanced)
+Once you've gotten a feel for using `MIPROv2` with `auto` settings, you may want to experiment with setting hyperparameters yourself to get the best results. The code below shows an example of how you can go about this. A full description of each parameter can be found in the section below.
+
+```python
+# Import the optimizer
+from dspy.teleprompt import MIPROv2
+
+# Initialize optimizer
 teleprompter = MIPROv2(
     metric=gsm8k_metric,
     num_candidates=7,
     init_temperature=0.5,
+    max_bootstrapped_demos=3,
+    max_labeled_demos=4,
     verbose=False,
-    num_threads=4,
 )
 
-# Perform optimization
-print(f"Optimizing program with MIPRO (0-Shot)...")
-zeroshot_optimized_program = teleprompter.compile(
+# Optimize program
+print(f"Optimizing program with MIPRO...")
+optimized_program = teleprompter.compile(
     program.deepcopy(),
     trainset=trainset,
-    max_bootstrapped_demos=0, # setting demos to 0 for 0-shot optimization
-    max_labeled_demos=0,
     num_trials=15,
     minibatch_size=25,
     minibatch_full_eval_steps=10,
-    minibatch=False, 
+    minibatch=True, 
     requires_permission_to_run=False,
 )
 
+# Save optimize program for future use
+optimized_program.save(f"mipro_optimized")
 
-zeroshot_optimized_program.save(f"mipro_0shot_optimized")
-
-print(f"Evaluate optimized program...")
-evaluate(zeroshot_optimized_program, devset=devset[:])
+# Evaluate optimized program
+print(f"Evluate optimized program...")
+evaluate(optimized_program, devset=devset[:])
 ```
 
 ## Parameters
@@ -149,32 +174,36 @@ evaluate(zeroshot_optimized_program, devset=devset[:])
 
 | Parameter            | Type         | Default                                              | Description                                                                                                                  |
 |----------------------|--------------|------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
-| `metric`           | `dspy.metric`       | N/A - Required                                                 | The evaluation metric used to optimize the task model.                                                                       |
+| `metric`           | `dspy.metric`       | **Required**                                                 | The evaluation metric used to optimize the task model.                                                                       |
 | `prompt_model`     | `dspy.LM`       | LM specified in `dspy.settings`                        | Model used for prompt generation.                                                                                            |
 | `task_model`       | `dspy.LM`       | LM specified in `dspy.settings`                        | Model used for task execution.                                                                                               |
+| `auto`       | `Optional[str]`       | None                        | If set to `light`, `medium`, or `heavy`, this will automatically configure the following hyperparameters: `num_candidates`, `num_trials`, `minibatch`, and will also cap the size of `valset` up to 100, 300, and 1000 for `light`, `medium`, and `heavy` runs respectively.                                                            |
 | `num_candidates`   | `int`      | `10`                                                   | Number of candidate instructions & few-shot examples to generate and evaluate for each predictor. If `num_candidates=10`, this means for a 2 module LM program we'll be optimizing over 10 candidates x 2 modules x 2 variables (few-shot ex. and instructions for each module)= 40 total variables. Therfore, if we increase `num_candidates`, we will probably want to increase `num_trials` as well (see Compile parameters).                                                                          |
 | `num_threads`      | `int`      |  `6`                                       | Threads to use for evaluation.                                                                                               |
 | `max_errors`       | `int`      | `10`                                                 | Maximum errors during an evaluation run that can be made before throwing an Exception.                                       |
 | `teacher_settings` | `dict`       | `{}`                                                 | Settings to use for the teacher model that bootstraps few-shot examples. An example dict would be `{lm=<dspy.LM object>}`. If your LM program with your default model is struggling to bootstrap any examples, it could be worth using a more powerful teacher model for bootstrapping.                                                     |
+| `max_bootstrapped_demos` | `int`  | `4`      | Maximum number of bootstrapped demonstrations to generate and include in the prompt.                              |
+| `max_labeled_demos`      | `int`  | `16`      | Maximum number of labeled demonstrations to generate and include in the prompt. Note that these differ from bootstrapped examples because they are just inputs & outputs sampled directly from the training set and do not have bootstrapped intermediate steps. 
 | `init_temperature` | `float`        | `1.0`                                                  | The initial temperature for prompt generation, influencing creativity.                                                       |
 | `verbose`          | `bool`      | `False`                                                | Enables printing intermediate steps and information.                                                                         |
 | `track_stats`      | `bool`      | `True`                                                | Logs relevant information through the optimization process if set to True.                                                                       |
 | `metric_threshold` | `float`        | `None`                                                 | A metric threshold is used if we only want to keep bootstrapped few-shot examples that exceed some threshold of performance.  |
+| `seed` | `int`        | `9`                                                 | Seed for reproducibility.  |
 
 ### Compile Parameters
 
 | Parameter                  | Type     | Default | Description                                                                                              |
 |----------------------------|----------|---------|----------------------------------------------------------------------------------------------------------|
-| `student`                | `dspy.Module`   | N/A (Required)    | The base program to optimize.                                                                            |
-| `trainset`               | `List[dspy.Example]`  | N/A (Required)    | Training dataset which is used to bootstrap few-shot examples and instructions. If a separate `valset` is not specified, 80% of this training set will also be used as a validation set for evaluating new candidate prompts.                                                                    |
+| `student`                | `dspy.Module`   | **Required**    | The base program to optimize.                                                                            |
+| `trainset`               | `List[dspy.Example]`  | **Required** | Training dataset which is used to bootstrap few-shot examples and instructions. If a separate `valset` is not specified, 80% of this training set will also be used as a validation set for evaluating new candidate prompts.                                                                    |
 | `valset`               | `List[dspy.Example]`  | Defaults to 80% of trainset | Dataset which is used to evaluate candidate prompts. We recommend using somewhere between 50-500 examples for optimization.                                                                      |
 | `num_trials`            | `int`  | `30`      | Number of optimization trials to run. When `minibatch` is set to `True`, this represents the number of minibatch trials that will be run on batches of size `minibatch_size`. When minibatch is set to `False`, each trial uses a full evaluation on the training set. In both cases, we recommend setting `num_trials` to a *minimum* of .75 x # modules in program x # variables per module (2 if few-shot examples & instructions will both be optimized, 1 in the 0-shot case).        |
 | `minibatch`              | `bool`  | `True`    | Flag to enable evaluating over minibatches of data (instead of the full validation set) for evaluation each trial.                                                           |
 | `minibatch_size`         | `int`  | `25.0`    | Size of minibatches for evaluations.                                                                     |
 | `minibatch_full_eval_steps` | `int` | `10`    | When minibatching is enabled, a full evaluation on the validation set will be carried out every `minibatch_full_eval_steps` on the top averaging set of prompts (according to their average score on the minibatch trials).          
-| `max_bootstrapped_demos` | `int`  | `4`      | Maximum number of bootstrapped demonstrations to generate and include in the prompt.                              |
-| `max_labeled_demos`      | `int`  | `16`      | Maximum number of labeled demonstrations to generate and include in the prompt. Note that these differ from bootstrapped examples because they are just inputs & outputs sampled directly from the training set and do not have bootstrapped intermediate steps.                             |
-| `seed`                   | `int`  | `9`       | Seed for reproducibility.                                                    |                                  |
+| `max_bootstrapped_demos` | `Optional[int]`  | Defaults to `init` value. | Maximum number of bootstrapped demonstrations to generate and include in the prompt.                              |
+| `max_labeled_demos`      | `Optional[int]`  | Defaults to `init` value.  | Maximum number of labeled demonstrations to generate and include in the prompt. Note that these differ from bootstrapped examples because they are just inputs & outputs sampled directly from the training set and do not have bootstrapped intermediate steps.                             |
+| `seed`                   | `Optional[int]`  | Defaults to `init` value. | Seed for reproducibility.                                                    |                                  |
 | `program_aware_proposer` | `bool`  | `True`    | Flag to enable summarizing a reflexive view of the code for your LM program.                             |
 | `data_aware_proposer` | `bool`  | `True`    | Flag to enable summarizing your training dataset.                             |
 | `view_data_batch_size` | `int`  | `10`    | Number of data examples to look at a time when generating the summary.                             |
@@ -189,7 +218,7 @@ At a high level, `MIPROv2` works by creating both few-shot examples and new inst
 These steps are broken down in more detail below:
 1) **Bootstrap Few-Shot Examples**: The same bootstrapping technique used in `BootstrapFewshotWithRandomSearch` is used to create few-shot examples. This works by randomly sampling examples from your training set, which are then run through your LM program. If the output from the program is correct for this example, it is kept as a valid few-shot example candidate. Otherwise, we try another example until we've curated the specified amount of few-shot example candidates. This step creates `num_candidates` sets of `max_bootstrapped_demos` bootstrapped examples and `max_labeled_demos` basic examples sampled from the training set.
 2) **Propose Instruction Candidates**. Next, we propose instruction candidates for each predictor in the program. This is done using another LM program as a proposer, which bootstraps & summarizes relevant information about the task to generate high quality instructions. Specifically, the instruction proposer includes (1) a generated summary of properties of the training dataset, (2) a generated summary of your LM program's code and the specific predictor that an instruction is being generated for, (3) the previously bootstrapped few-shot examples to show reference inputs / outputs for a given predictor and (4) a randomly sampled tip for generation (i.e. "be creative", "be concise", etc.) to help explore the feature space of potential instructions.
-3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_batches` trials, where a new set of prompts are evaluated over our validation set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full validation set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full validation set is returned.
+3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_trials` trials, where a new set of prompts are evaluated over our validation set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full validation set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full validation set is returned.
 
 
 For those interested in more details, more information on `MIPROv2` along with a study on `MIPROv2` compared with other DSPy optimizers can be found in [this paper](https://arxiv.org/abs/2406.11695). 
\ No newline at end of file
diff --git a/docs/docs/faqs.md b/docs/docs/faqs.md
index b2605138a..0213cd708 100644
--- a/docs/docs/faqs.md
+++ b/docs/docs/faqs.md
@@ -85,13 +85,23 @@ Open source libraries such as [RAGautouille](https://github.com/bclavie/ragatoui
 
 - **How do I turn off the cache? How do I export the cache?**
 
-You can turn off the cache by setting the [`DSP_CACHEBOOL`](https://github.com/stanfordnlp/dspy/blob/main/dsp/modules/cache_utils.py#L9) environment variable to `False`, which disables the `cache_turn_on` flag.
+From v2.5, you can turn off the cache by setting `cache` parameter in `dspy.LM` to `False`:
+
+```python
+dspy.LM('openai/gpt-4o-mini',  cache=False)
+```
 
 Your local cache will be saved to the global env directory `os.environ["DSP_CACHEDIR"]` or for notebooks `os.environ["DSP_NOTEBOOK_CACHEDIR"]`. You can usually set the cachedir to `os.path.join(repo_path, 'cache')` and export this cache from here:
 ```python
 os.environ["DSP_NOTEBOOK_CACHEDIR"] = os.path.join(os.getcwd(), 'cache')
 ```
 
+:::warning Important
+`DSP_CACHEDIR` is responsible for old clients (including dspy.OpenAI, dspy.ColBERTv2, etc.) and `DSPY_CACHEDIR` is responsible for the new dspy.LM client.
+:::
+
+In the AWS lambda deployment, you should disable both DSP_* and DSPY_*.
+
 
 ## Advanced Usage
 
diff --git a/docs/docs/tutorials/examples.md b/docs/docs/tutorials/examples.md
index c7bed2606..61e5d6b30 100644
--- a/docs/docs/tutorials/examples.md
+++ b/docs/docs/tutorials/examples.md
@@ -25,5 +25,7 @@ You can find other examples tweeted by [@lateinteraction](https://twitter.com/la
 - [Using Ollama with DSPy for Mistral (quantized) by @jrknox1977](https://gist.github.com/jrknox1977/78c17e492b5a75ee5bbaf9673aee4641)
 - [Using DSPy, "The Unreasonable Effectiveness of Eccentric Automatic Prompts" (paper) by VMware's Rick Battle & Teja Gollapudi, and interview at TheRegister](https://www.theregister.com/2024/02/22/prompt_engineering_ai_models/)
 - [Portkey with DSPy: Efficiently Track Metrics, Optimize Performance, and Scale Across 250+ LLMs](https://docs.portkey.ai/docs/integrations/libraries/dspy)
+- [Langtrace AI DSPy Examples Repository](https://github.com/Scale3-Labs/dspy-examples)
+- [Langtrace Blogpost - Build a summarization system using DSPy](https://www.langtrace.ai/blog/build-a-reliable-summarization-system-using-dspy-and-langtrace)
 
 There are also recent cool examples at [Weaviate's DSPy cookbook](https://github.com/weaviate/recipes/tree/main/integrations/llm-frameworks/dspy) by Connor Shorten. [See tutorial on YouTube](https://www.youtube.com/watch?v=CEuUG4Umfxs).
diff --git a/docs/docs/tutorials/summarization.md b/docs/docs/tutorials/summarization.md
new file mode 100644
index 000000000..af4f9871c
--- /dev/null
+++ b/docs/docs/tutorials/summarization.md
@@ -0,0 +1,191 @@
+---
+sidebar_position: 3
+---
+
+# [03] Summarization
+
+Summarization is a fundamental task in natural language processing that involves condensing a longer piece of text into a shorter version while retaining its key information and main ideas. It's a crucial skill for both humans and machines, with applications ranging from creating article abstracts to generating concise reports from lengthy documents.
+
+While summarization is valuable, evaluating the quality of summaries produced by language models presents significant challenges. Unlike tasks with clear right or wrong answers, summarization quality is often subjective and context-dependent. Some key difficulties in evaluating summarized outputs include:
+
+1. Balancing information retention with conciseness
+2. Preserving the original text's tone and intent
+3. Ensuring factual accuracy without introducing errors
+4. Adapting to different types of source material and summary purposes
+5. Accounting for varying reader preferences and background knowledge
+
+These challenges make it difficult to create universal metrics for summary quality, often requiring a combination of automated measures and human judgment for comprehensive evaluation.
+
+In this example, we show how you can build a summarization system using DSPy. The technique involves the following:
+
+1. We build a DSPy program for doing the actual summarization task which takes a `passage` as input and gives a `summary` as output.
+2. We would ideally want to optimize the summarization program over a metric in order to build an effective program. But since writing a metric function for summarization tasks is subjective and ambiguous in nature, we can use a LM to do this task which in turn means we can write another DSPy program to do the task of scoring a summary based on the variable we care about.
+3. Finally, we compose the system by using the Metric program for defining the metric function in the Summarization program.
+
+## Metric program
+
+Define two signatures for scoring the summary.
+
+1. The first signature breaks down the summary and assigns defined labels as grades.
+```
+import dspy
+
+class Breakdown(dspy.Signature):
+    """
+    Given a passage, break down the passage into key ideas.
+    Enumerate every key idea in the passage and
+    assign it an importance grade
+    (High, Medium, or Low).
+    """
+
+    passage = dspy.InputField()
+    key_ideas: str = dspy.OutputField(
+        desc="numbered list of one key idea per line,"
+        "followed by its importance grade, "
+             "e.g. 1. <Idea here>. High.")
+    importance_grades: list[str] = dspy.OutputField(
+        desc='list of importance grades, '
+             'e.g. ["High", "Medium", "Low"].')
+```
+
+2. The second signature assigns a binary score using the summary and the break down from the previous signature as inputs
+```
+class SummaryCorrectness(dspy.Signature):
+    """
+    Compare a system generated summary to the key ideas in the passage.
+    For every key idea supplied,
+    assign a binary score based on whether the summary contains it.
+    And compute an overall score based on the binary scores.
+    """
+
+    key_ideas: str = dspy.InputField(
+        desc="key ideas in the passage "
+             "for evaluating the summary")
+    summary: str = dspy.InputField()
+    binary_scores: list[bool] = dspy.OutputField(
+        desc="list of binary scores for each key idea, "
+             "e.g. [True, False, True]")
+    overall_score: float = dspy.OutputField(
+        desc="overall score for the summary out of 1.0")
+```
+
+3. Now write the actual program that computes a weighted score between 0 to 1 and falls back to the overall_score that's loosely scored by the model
+
+```
+class Metric(dspy.Module):
+    """
+    Compute a score for the correctness of a summary.
+    """
+
+    def __init__(self):
+        self.breakdown = dspy.ChainOfThought(Breakdown)
+        self.assess = dspy.ChainOfThought(SummaryCorrectness)
+
+    def forward(self, example, pred, trace=None):
+        breakdown = self.breakdown(
+            passage=example.passage
+        )
+        key_ideas = breakdown.key_ideas
+        importance_grades = breakdown.importance_grades
+
+        scores = self.assess(
+            key_ideas=key_ideas,
+            summary=pred.summary,
+        )
+
+        try:
+            weight_map = {'High': 1.0, 'Medium': 0.7}
+            score = sum(
+                weight_map.get(g, 0.2) * int(b)
+                for g, b in zip(importance_grades, scores.binary_scores)
+            )
+            score /= sum(weight_map.get(g, 0.2) for g in importance_grades)
+
+        # pylint: disable=broad-except
+        except Exception:
+            score = float(scores.overall_score)
+
+        return score if trace is None else score >= 0.75
+```
+
+## Summarization Program
+
+First define it's signature
+
+```
+class SummarizeSignature(dspy.Signature):
+    """
+    Given a passage, generate a summary.
+    """
+
+    passage = dspy.InputField(desc="a passage to summarize")
+    summary: str = dspy.OutputField(
+        desc="a concise summary of the passage")
+```
+
+Next, define the program
+
+```
+class Summarize(dspy.Module):
+    def __init__(self):
+        self.summarize = dspy.ChainOfThought(SummarizeSignature)
+
+    def forward(self, passage: str):
+        summary = self.summarize(
+            passage=passage
+        )
+        return summary
+```
+
+Let's jump into the evaluation process now. First, define the language model.
+
+```
+lm = dspy.LM("openai/gpt-4o-mini", max_tokens=250)
+dspy.settings.configure(lm=lm)
+```
+
+Load a dataset. The dataset is a `jsonl` file where each line has a passage, summary and a score.
+
+```
+dataset = []
+with open('src/summarization/programs/summarize/dataset.jsonl', 'r',
+          encoding='utf-8') as f:
+    for line in f:
+        data = json.loads(line)
+
+        passage = data.get("passage", "")
+        summary = data.get("summary", "")
+        score = data.get("score", 0)
+
+        example = dspy.Example(passage=passage, summary=summary, score=score)
+        dataset.append(example)
+
+trainset = [x.with_inputs("passage") for x in dataset]
+```
+
+Now, define the metric. This is where we use the `Metric` program to compute the score for the summary generated by the `Summarization` program.
+
+```
+def metric(gold, pred, trace=None):
+    metric_program = Metric()
+    examp = dspy.Example(passage=gold.passage)
+    predicted = dspy.Example(summary=pred)
+    pred_score = metric_program(example=examp, pred=predicted)
+    gold_score = gold.score
+    # check if they are almost equal
+    return abs(float(gold_score) - float(pred_score)) < 0.2
+```
+
+Create the program and evaluate it.
+```
+program = Summarize()
+
+evaluate = dspy.Evaluate(devset=trainset, metric=metric,
+                    display_progress=True,
+                    display_table=True, provide_traceback=True)
+
+res = evaluate(program, devset=trainset)
+print(res)
+```
+
+That's it! Hopefully this example was helpful in understanding how you can compose DSPy programs to build a truly compound AI system that's optimized for isolated tasks.
\ No newline at end of file
diff --git a/docs/docusaurus.config.ts b/docs/docusaurus.config.ts
index ca922e56d..8dcaefb94 100644
--- a/docs/docusaurus.config.ts
+++ b/docs/docusaurus.config.ts
@@ -150,15 +150,6 @@ const config: Config = {
       darkTheme: prismThemes.dracula, // Dark theme for code blocks
     },
   },
-  themes: [
-    [
-      require.resolve("@easyops-cn/docusaurus-search-local"),
-      /** @type {import("@easyops-cn/docusaurus-search-local").PluginOptions} */
-      ({
-        hashed: true,
-      }),
-    ],
-  ],
   scripts: [
     {
       id: "runllm-widget-script",
diff --git a/docs/package-lock.json b/docs/package-lock.json
index 76e0f5685..36f00e48e 100644
--- a/docs/package-lock.json
+++ b/docs/package-lock.json
@@ -10,7 +10,7 @@
       "dependencies": {
         "@docusaurus/core": "3.1.0",
         "@docusaurus/preset-classic": "3.1.0",
-        "@easyops-cn/docusaurus-search-local": "^0.40.1",
+        "@getcanary/web": "^1.0.9",
         "@mdx-js/react": "^3.0.0",
         "clsx": "^2.0.0",
         "prism-react-renderer": "^2.3.0",
@@ -2759,83 +2759,54 @@
         "node": ">=18.0"
       }
     },
-    "node_modules/@easyops-cn/autocomplete.js": {
-      "version": "0.38.1",
-      "resolved": "https://registry.npmjs.org/@easyops-cn/autocomplete.js/-/autocomplete.js-0.38.1.tgz",
-      "integrity": "sha512-drg76jS6syilOUmVNkyo1c7ZEBPcPuK+aJA7AksM5ZIIbV57DMHCywiCr+uHyv8BE5jUTU98j/H7gVrkHrWW3Q==",
+    "node_modules/@floating-ui/core": {
+      "version": "1.6.8",
+      "resolved": "https://registry.npmjs.org/@floating-ui/core/-/core-1.6.8.tgz",
+      "integrity": "sha512-7XJ9cPU+yI2QeLS+FCSlqNFZJq8arvswefkZrYI1yQBbftw6FyrZOxYSh+9S7z7TpeWlRt9zJ5IhM1WIL334jA==",
       "dependencies": {
-        "cssesc": "^3.0.0",
-        "immediate": "^3.2.3"
-      }
-    },
-    "node_modules/@easyops-cn/docusaurus-search-local": {
-      "version": "0.40.1",
-      "resolved": "https://registry.npmjs.org/@easyops-cn/docusaurus-search-local/-/docusaurus-search-local-0.40.1.tgz",
-      "integrity": "sha512-4HMFZMpKKdd5qq1nFB8cvrAkgzZ1kNxphVciI64YHtmDYGIthVGZVG6+Ci7AAhzCR+ixLJkYwtVekvuMLjr2ZQ==",
-      "dependencies": {
-        "@docusaurus/plugin-content-docs": "^2 || ^3",
-        "@docusaurus/theme-translations": "^2 || ^3",
-        "@docusaurus/utils": "^2 || ^3",
-        "@docusaurus/utils-common": "^2 || ^3",
-        "@docusaurus/utils-validation": "^2 || ^3",
-        "@easyops-cn/autocomplete.js": "^0.38.1",
-        "@node-rs/jieba": "^1.6.0",
-        "cheerio": "^1.0.0-rc.3",
-        "clsx": "^1.1.1",
-        "debug": "^4.2.0",
-        "fs-extra": "^10.0.0",
-        "klaw-sync": "^6.0.0",
-        "lunr": "^2.3.9",
-        "lunr-languages": "^1.4.0",
-        "mark.js": "^8.11.1",
-        "tslib": "^2.4.0"
-      },
-      "engines": {
-        "node": ">=12"
-      },
-      "peerDependencies": {
-        "@docusaurus/theme-common": "^2 || ^3",
-        "react": "^16.14.0 || ^17 || ^18",
-        "react-dom": "^16.14.0 || 17 || ^18"
-      }
-    },
-    "node_modules/@easyops-cn/docusaurus-search-local/node_modules/clsx": {
-      "version": "1.2.1",
-      "resolved": "https://registry.npmjs.org/clsx/-/clsx-1.2.1.tgz",
-      "integrity": "sha512-EcR6r5a8bj6pu3ycsa/E/cKVGuTgZJZdsyUYHOksG/UHIiKfjxzRxYJpyVBwYaQeOvghal9fcc4PidlgzugAQg==",
-      "engines": {
-        "node": ">=6"
+        "@floating-ui/utils": "^0.2.8"
       }
     },
-    "node_modules/@easyops-cn/docusaurus-search-local/node_modules/fs-extra": {
-      "version": "10.1.0",
-      "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-10.1.0.tgz",
-      "integrity": "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ==",
+    "node_modules/@floating-ui/dom": {
+      "version": "1.6.11",
+      "resolved": "https://registry.npmjs.org/@floating-ui/dom/-/dom-1.6.11.tgz",
+      "integrity": "sha512-qkMCxSR24v2vGkhYDo/UzxfJN3D4syqSjyuTFz6C7XcpU1pASPRieNI0Kj5VP3/503mOfYiGY891ugBX1GlABQ==",
       "dependencies": {
-        "graceful-fs": "^4.2.0",
-        "jsonfile": "^6.0.1",
-        "universalify": "^2.0.0"
-      },
-      "engines": {
-        "node": ">=12"
+        "@floating-ui/core": "^1.6.0",
+        "@floating-ui/utils": "^0.2.8"
       }
     },
-    "node_modules/@emnapi/core": {
-      "version": "0.45.0",
-      "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-0.45.0.tgz",
-      "integrity": "sha512-DPWjcUDQkCeEM4VnljEOEcXdAD7pp8zSZsgOujk/LGIwCXWbXJngin+MO4zbH429lzeC3WbYLGjE2MaUOwzpyw==",
-      "optional": true,
+    "node_modules/@floating-ui/utils": {
+      "version": "0.2.8",
+      "resolved": "https://registry.npmjs.org/@floating-ui/utils/-/utils-0.2.8.tgz",
+      "integrity": "sha512-kym7SodPp8/wloecOpcmSnWJsK7M0E5Wg8UcFA+uO4B9s5d0ywXOEro/8HM9x0rW+TljRzul/14UYz3TleT3ig=="
+    },
+    "node_modules/@getcanary/web": {
+      "version": "1.0.9",
+      "resolved": "https://registry.npmjs.org/@getcanary/web/-/web-1.0.9.tgz",
+      "integrity": "sha512-TG+0sN/4zxvD62PbFCLOAGhhykD8y9j3LBlBc4Pba5dEmNUggPsgZXnx+GxK4Q/U5Tof2Cr1JfgYhC2CDLcaww==",
       "dependencies": {
-        "tslib": "^2.4.0"
+        "@floating-ui/dom": "^1.6.8",
+        "@lit-labs/observers": "^2.0.2",
+        "@lit/context": "^1.1.2",
+        "@lit/task": "^1.0.1",
+        "@xstate/store": "^2.5.0",
+        "best-effort-json-parser": "^1.1.2",
+        "lit": "^3.1.4",
+        "marked": "^14.0.0",
+        "picomatch": "^4.0.2",
+        "prismjs": "^1.29.0"
       }
     },
-    "node_modules/@emnapi/runtime": {
-      "version": "0.45.0",
-      "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-0.45.0.tgz",
-      "integrity": "sha512-Txumi3td7J4A/xTTwlssKieHKTGl3j4A1tglBx72auZ49YK7ePY6XZricgIg9mnZT4xPfA+UPCUdnhRuEFDL+w==",
-      "optional": true,
-      "dependencies": {
-        "tslib": "^2.4.0"
+    "node_modules/@getcanary/web/node_modules/picomatch": {
+      "version": "4.0.2",
+      "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
+      "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
+      "engines": {
+        "node": ">=12"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/jonschlinkert"
       }
     },
     "node_modules/@hapi/hoek": {
@@ -2935,6 +2906,44 @@
       "resolved": "https://registry.npmjs.org/@leichtgewicht/ip-codec/-/ip-codec-2.0.4.tgz",
       "integrity": "sha512-Hcv+nVC0kZnQ3tD9GVu5xSMR4VVYOteQIr/hwFPVEvPdlXqgGEuRjiheChHgdM+JyqdgNcmzZOX/tnl0JOiI7A=="
     },
+    "node_modules/@lit-labs/observers": {
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/@lit-labs/observers/-/observers-2.0.4.tgz",
+      "integrity": "sha512-x95jhDPGb+HtYU3hEdqkcLxb6v2JBP3tcajaiOijs1F/ZmOgRT0pRPn0v+jhhk8mAAbEO12SZJjPCmuysunssQ==",
+      "dependencies": {
+        "@lit/reactive-element": "^1.0.0 || ^2.0.0",
+        "lit-html": "^3.2.0"
+      }
+    },
+    "node_modules/@lit-labs/ssr-dom-shim": {
+      "version": "1.2.1",
+      "resolved": "https://registry.npmjs.org/@lit-labs/ssr-dom-shim/-/ssr-dom-shim-1.2.1.tgz",
+      "integrity": "sha512-wx4aBmgeGvFmOKucFKY+8VFJSYZxs9poN3SDNQFF6lT6NrQUnHiPB2PWz2sc4ieEcAaYYzN+1uWahEeTq2aRIQ=="
+    },
+    "node_modules/@lit/context": {
+      "version": "1.1.3",
+      "resolved": "https://registry.npmjs.org/@lit/context/-/context-1.1.3.tgz",
+      "integrity": "sha512-Auh37F4S0PZM93HTDfZWs97mmzaQ7M3vnTc9YvxAGyP3UItSK/8Fs0vTOGT+njuvOwbKio/l8Cx/zWL4vkutpQ==",
+      "dependencies": {
+        "@lit/reactive-element": "^1.6.2 || ^2.0.0"
+      }
+    },
+    "node_modules/@lit/reactive-element": {
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/@lit/reactive-element/-/reactive-element-2.0.4.tgz",
+      "integrity": "sha512-GFn91inaUa2oHLak8awSIigYz0cU0Payr1rcFsrkf5OJ5eSPxElyZfKh0f2p9FsTiZWXQdWGJeXZICEfXXYSXQ==",
+      "dependencies": {
+        "@lit-labs/ssr-dom-shim": "^1.2.0"
+      }
+    },
+    "node_modules/@lit/task": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/@lit/task/-/task-1.0.1.tgz",
+      "integrity": "sha512-fVLDtmwCau8NywnFIXaJxsCZjzaIxnVq+cFRKYC1Y4tA4/0rMTvF6DLZZ2JE51BwzOluaKtgJX8x1QDsQtAaIw==",
+      "dependencies": {
+        "@lit/reactive-element": "^1.0.0 || ^2.0.0"
+      }
+    },
     "node_modules/@mdx-js/mdx": {
       "version": "3.0.0",
       "resolved": "https://registry.npmjs.org/@mdx-js/mdx/-/mdx-3.0.0.tgz",
@@ -2985,255 +2994,6 @@
         "react": ">=16"
       }
     },
-    "node_modules/@napi-rs/wasm-runtime": {
-      "version": "0.1.1",
-      "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-0.1.1.tgz",
-      "integrity": "sha512-ATj9ua659JgrkICjJscaeZdmPr44cb/KFjNWuD0N6pux0SpzaM7+iOuuK11mAnQM2N9q0DT4REu6NkL8ZEhopw==",
-      "optional": true,
-      "dependencies": {
-        "@emnapi/core": "^0.45.0",
-        "@emnapi/runtime": "^0.45.0",
-        "@tybys/wasm-util": "^0.8.1"
-      }
-    },
-    "node_modules/@node-rs/jieba": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba/-/jieba-1.10.0.tgz",
-      "integrity": "sha512-9oZMCvZVnrAMeWTSnEjJ0OSw7YcV4dJJKSioqq80oUNf3eYLGdEXsgYwCe1AYEMcfUfNVgvjznItJKrsoud0IA==",
-      "engines": {
-        "node": ">= 10"
-      },
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/Brooooooklyn"
-      },
-      "optionalDependencies": {
-        "@node-rs/jieba-android-arm-eabi": "1.10.0",
-        "@node-rs/jieba-android-arm64": "1.10.0",
-        "@node-rs/jieba-darwin-arm64": "1.10.0",
-        "@node-rs/jieba-darwin-x64": "1.10.0",
-        "@node-rs/jieba-freebsd-x64": "1.10.0",
-        "@node-rs/jieba-linux-arm-gnueabihf": "1.10.0",
-        "@node-rs/jieba-linux-arm64-gnu": "1.10.0",
-        "@node-rs/jieba-linux-arm64-musl": "1.10.0",
-        "@node-rs/jieba-linux-x64-gnu": "1.10.0",
-        "@node-rs/jieba-linux-x64-musl": "1.10.0",
-        "@node-rs/jieba-wasm32-wasi": "1.10.0",
-        "@node-rs/jieba-win32-arm64-msvc": "1.10.0",
-        "@node-rs/jieba-win32-ia32-msvc": "1.10.0",
-        "@node-rs/jieba-win32-x64-msvc": "1.10.0"
-      }
-    },
-    "node_modules/@node-rs/jieba-android-arm-eabi": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-android-arm-eabi/-/jieba-android-arm-eabi-1.10.0.tgz",
-      "integrity": "sha512-bzusJSLHm7I0qL8aQXGLt7IQ51Px35yGGEcQ/Ps4SEt0AxRSJ2/rxNET/8mlwBpOCZ5xiKE3BOBRfQajiPiI3g==",
-      "cpu": [
-        "arm"
-      ],
-      "optional": true,
-      "os": [
-        "android"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-android-arm64": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-android-arm64/-/jieba-android-arm64-1.10.0.tgz",
-      "integrity": "sha512-g89Oq5U2RPmtlvuQhjNj8YZc5Gq033ODb7Ot4Z/OdIHvg2WMxi2M1GQhcdKu60dO79/tazc53W6I8/y691DUfQ==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "android"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-darwin-arm64": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-darwin-arm64/-/jieba-darwin-arm64-1.10.0.tgz",
-      "integrity": "sha512-IhR5r+XxFcfhVsF93zQ3uCJy8ndotRntXzoW/JCyKqOahUo/ITQRT6vTKHKMyD9xNmjl222OZonBSo2+mlI2fQ==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "darwin"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-darwin-x64": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-darwin-x64/-/jieba-darwin-x64-1.10.0.tgz",
-      "integrity": "sha512-MBIs8ixKY4FPnifdZ7eTx6ht85TXE4kFBK4c8A/VDAbnmzBzpEyuV7tHUA2wAdfR0muC9j7/5FB4kQGZgYfc8g==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "darwin"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-freebsd-x64": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-freebsd-x64/-/jieba-freebsd-x64-1.10.0.tgz",
-      "integrity": "sha512-MuY+1QEXONxo3I/uFLFju0/pSN5bzQORhJkIdP8CYv+jZaVB4Uz6rC7A5HrgjiAXOna6QsKlRgx2bYyHfaBUrA==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "freebsd"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-linux-arm-gnueabihf": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-linux-arm-gnueabihf/-/jieba-linux-arm-gnueabihf-1.10.0.tgz",
-      "integrity": "sha512-QfSBnwISdVuTqsi4iThAO1LSbKRSqSsIWiIJgCduhYsTDDiG9+pHyfiZtcTwSf73SDXHZ400QuBNONWLQ/dSag==",
-      "cpu": [
-        "arm"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-linux-arm64-gnu": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-linux-arm64-gnu/-/jieba-linux-arm64-gnu-1.10.0.tgz",
-      "integrity": "sha512-vzA2tX/6dReEd/7tZ9927glWQmKDausM6R9S5CqZx4BA4NSaWAK0xFdWsz0K7np459FXqNavLdNB5FVFJb4zzA==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-linux-arm64-musl": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-linux-arm64-musl/-/jieba-linux-arm64-musl-1.10.0.tgz",
-      "integrity": "sha512-gxqoAVOQsn9sgYK6mFO9dsMZ/yOMvVecLZW5rGvLErjiugVvYUlESXIvCqxp2GSws8RtTqJj6p9u/lBmCCuvaw==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-linux-x64-gnu": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-linux-x64-gnu/-/jieba-linux-x64-gnu-1.10.0.tgz",
-      "integrity": "sha512-rS5Shs8JITxJjFIjoIZ5a9O+GO21TJgKu03g2qwFE3QaN5ZOvXtz+/AqqyfT4GmmMhCujD83AGqfOGXDmItF9w==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-linux-x64-musl": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-linux-x64-musl/-/jieba-linux-x64-musl-1.10.0.tgz",
-      "integrity": "sha512-BvSiF2rR8Birh2oEVHcYwq0WGC1cegkEdddWsPrrSmpKmukJE2zyjcxaOOggq2apb8fIRsjyeeUh6X3R5AgjvA==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-wasm32-wasi": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-wasm32-wasi/-/jieba-wasm32-wasi-1.10.0.tgz",
-      "integrity": "sha512-EzeAAbRrFTdYw61rd8Mfwdp/fA21d58z9vLY06CDbI+dqANfMFn1IUdwzKWi8S5J/MRhvbzonbbh3yHlz6F43Q==",
-      "cpu": [
-        "wasm32"
-      ],
-      "optional": true,
-      "dependencies": {
-        "@napi-rs/wasm-runtime": "^0.1.1"
-      },
-      "engines": {
-        "node": ">=14.0.0"
-      }
-    },
-    "node_modules/@node-rs/jieba-win32-arm64-msvc": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-win32-arm64-msvc/-/jieba-win32-arm64-msvc-1.10.0.tgz",
-      "integrity": "sha512-eZjRLFUAvq1/E5+xXfJRqIB99Gu6BA+6+EXf/rCLuvEjXrDQuUunhmrSoOL5MjmUXTtazS+bXq9PXV5EFYyOPw==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "win32"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-win32-ia32-msvc": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-win32-ia32-msvc/-/jieba-win32-ia32-msvc-1.10.0.tgz",
-      "integrity": "sha512-DrfbeCN7UcLN+MiocZabWo74XZIjfpQsJ/WMOItZzVbU2gDcJSkSyAhML9+OqId66DhGCMFFlGinocElM8iIAw==",
-      "cpu": [
-        "ia32"
-      ],
-      "optional": true,
-      "os": [
-        "win32"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@node-rs/jieba-win32-x64-msvc": {
-      "version": "1.10.0",
-      "resolved": "https://registry.npmjs.org/@node-rs/jieba-win32-x64-msvc/-/jieba-win32-x64-msvc-1.10.0.tgz",
-      "integrity": "sha512-RjBkBmjjHmj+bofiq5/han8wzbCkDk24OAPJ+YX8PX20GFSHmdjCiWapv3AooN8/RiKqlBfgodjS1JUngNWo5g==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "win32"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
     "node_modules/@nodelib/fs.scandir": {
       "version": "2.1.5",
       "resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz",
@@ -3627,15 +3387,6 @@
         "node": ">=10.13.0"
       }
     },
-    "node_modules/@tybys/wasm-util": {
-      "version": "0.8.1",
-      "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.8.1.tgz",
-      "integrity": "sha512-GSsTwyBl4pIzsxAY5wroZdyQKyhXk0d8PCRZtrSZ2WEB1cBdrp2EgGBwHOGCZtIIPun/DL3+AykCv+J6fyRH4Q==",
-      "optional": true,
-      "dependencies": {
-        "tslib": "^2.4.0"
-      }
-    },
     "node_modules/@types/acorn": {
       "version": "4.0.6",
       "resolved": "https://registry.npmjs.org/@types/acorn/-/acorn-4.0.6.tgz",
@@ -3962,6 +3713,11 @@
         "@types/node": "*"
       }
     },
+    "node_modules/@types/trusted-types": {
+      "version": "2.0.7",
+      "resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz",
+      "integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw=="
+    },
     "node_modules/@types/unist": {
       "version": "3.0.2",
       "resolved": "https://registry.npmjs.org/@types/unist/-/unist-3.0.2.tgz",
@@ -4124,6 +3880,23 @@
         "@xtuc/long": "4.2.2"
       }
     },
+    "node_modules/@xstate/store": {
+      "version": "2.6.0",
+      "resolved": "https://registry.npmjs.org/@xstate/store/-/store-2.6.0.tgz",
+      "integrity": "sha512-pHiGIn378yPSCY36f/8iFF1KtKTKpGINqUVJH/dYydzWT+uXc4zKUQ+XUk0qTHchTvBXQ/UivRox2Q19ZnzTjw==",
+      "peerDependencies": {
+        "react": "^18.2.0",
+        "solid-js": "^1.7.6"
+      },
+      "peerDependenciesMeta": {
+        "react": {
+          "optional": true
+        },
+        "solid-js": {
+          "optional": true
+        }
+      }
+    },
     "node_modules/@xtuc/ieee754": {
       "version": "1.2.0",
       "resolved": "https://registry.npmjs.org/@xtuc/ieee754/-/ieee754-1.2.0.tgz",
@@ -4542,6 +4315,11 @@
       "resolved": "https://registry.npmjs.org/batch/-/batch-0.6.1.tgz",
       "integrity": "sha512-x+VAiMRL6UPkx+kudNvxTl6hB2XNNCG2r+7wixVfIYwu/2HKRXimwQyaumLjMveWvT2Hkd/cAJw+QBMfJ/EKVw=="
     },
+    "node_modules/best-effort-json-parser": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/best-effort-json-parser/-/best-effort-json-parser-1.1.2.tgz",
+      "integrity": "sha512-RD7tyk24pNCDwEKFACauR6Lqp5m6BHUrehwyhN/pA8V3QYWq8Y+hk9vHZvKiThZsdEFTaUqN49duVsamgCd8/g=="
+    },
     "node_modules/big.js": {
       "version": "5.2.2",
       "resolved": "https://registry.npmjs.org/big.js/-/big.js-5.2.2.tgz",
@@ -7761,11 +7539,6 @@
         "node": ">=16.x"
       }
     },
-    "node_modules/immediate": {
-      "version": "3.3.0",
-      "resolved": "https://registry.npmjs.org/immediate/-/immediate-3.3.0.tgz",
-      "integrity": "sha512-HR7EVodfFUdQCTIeySw+WDRFJlPcLOJbXfwwZ7Oom6tjsvZ3bOkCDJHehQC3nxJrv7+f9XecwazynjU8e4Vw3Q=="
-    },
     "node_modules/immer": {
       "version": "9.0.21",
       "resolved": "https://registry.npmjs.org/immer/-/immer-9.0.21.tgz",
@@ -8295,14 +8068,6 @@
         "node": ">=0.10.0"
       }
     },
-    "node_modules/klaw-sync": {
-      "version": "6.0.0",
-      "resolved": "https://registry.npmjs.org/klaw-sync/-/klaw-sync-6.0.0.tgz",
-      "integrity": "sha512-nIeuVSzdCCs6TDPTqI8w1Yre34sSq7AkZ4B3sfOBbI2CgVSB4Du4aLQijFU2+lhAFCwt9+42Hel6lQNIv6AntQ==",
-      "dependencies": {
-        "graceful-fs": "^4.1.11"
-      }
-    },
     "node_modules/kleur": {
       "version": "3.0.3",
       "resolved": "https://registry.npmjs.org/kleur/-/kleur-3.0.3.tgz",
@@ -8355,6 +8120,34 @@
       "resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz",
       "integrity": "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg=="
     },
+    "node_modules/lit": {
+      "version": "3.2.1",
+      "resolved": "https://registry.npmjs.org/lit/-/lit-3.2.1.tgz",
+      "integrity": "sha512-1BBa1E/z0O9ye5fZprPtdqnc0BFzxIxTTOO/tQFmyC/hj1O3jL4TfmLBw0WEwjAokdLwpclkvGgDJwTIh0/22w==",
+      "dependencies": {
+        "@lit/reactive-element": "^2.0.4",
+        "lit-element": "^4.1.0",
+        "lit-html": "^3.2.0"
+      }
+    },
+    "node_modules/lit-element": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/lit-element/-/lit-element-4.1.1.tgz",
+      "integrity": "sha512-HO9Tkkh34QkTeUmEdNYhMT8hzLid7YlMlATSi1q4q17HE5d9mrrEHJ/o8O2D0cMi182zK1F3v7x0PWFjrhXFew==",
+      "dependencies": {
+        "@lit-labs/ssr-dom-shim": "^1.2.0",
+        "@lit/reactive-element": "^2.0.4",
+        "lit-html": "^3.2.0"
+      }
+    },
+    "node_modules/lit-html": {
+      "version": "3.2.1",
+      "resolved": "https://registry.npmjs.org/lit-html/-/lit-html-3.2.1.tgz",
+      "integrity": "sha512-qI/3lziaPMSKsrwlxH/xMgikhQ0EGOX2ICU73Bi/YHFvz2j/yMCIrw4+puF2IpQ4+upd3EWbvnHM9+PnJn48YA==",
+      "dependencies": {
+        "@types/trusted-types": "^2.0.2"
+      }
+    },
     "node_modules/loader-runner": {
       "version": "4.3.0",
       "resolved": "https://registry.npmjs.org/loader-runner/-/loader-runner-4.3.0.tgz",
@@ -8457,21 +8250,6 @@
         "yallist": "^3.0.2"
       }
     },
-    "node_modules/lunr": {
-      "version": "2.3.9",
-      "resolved": "https://registry.npmjs.org/lunr/-/lunr-2.3.9.tgz",
-      "integrity": "sha512-zTU3DaZaF3Rt9rhN3uBMGQD3dD2/vFQqnvZCDv4dl5iOzq2IZQqTxu90r4E5J+nP70J3ilqVCrbho2eWaeW8Ow=="
-    },
-    "node_modules/lunr-languages": {
-      "version": "1.14.0",
-      "resolved": "https://registry.npmjs.org/lunr-languages/-/lunr-languages-1.14.0.tgz",
-      "integrity": "sha512-hWUAb2KqM3L7J5bcrngszzISY4BxrXn/Xhbb9TTCJYEGqlR1nG67/M14sp09+PTIRklobrn57IAxcdcO/ZFyNA=="
-    },
-    "node_modules/mark.js": {
-      "version": "8.11.1",
-      "resolved": "https://registry.npmjs.org/mark.js/-/mark.js-8.11.1.tgz",
-      "integrity": "sha512-1I+1qpDt4idfgLQG+BNWmrqku+7/2bi5nLf4YwF8y8zXvmfiTBY3PV3ZibfrjBueCByROpuBjLLFCajqkgYoLQ=="
-    },
     "node_modules/markdown-extensions": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/markdown-extensions/-/markdown-extensions-2.0.0.tgz",
@@ -8492,6 +8270,17 @@
         "url": "https://github.com/sponsors/wooorm"
       }
     },
+    "node_modules/marked": {
+      "version": "14.1.2",
+      "resolved": "https://registry.npmjs.org/marked/-/marked-14.1.2.tgz",
+      "integrity": "sha512-f3r0yqpz31VXiDB/wj9GaOB0a2PRLQl6vJmXiFrniNwjkKdvakqJRULhjFKJpxOchlCRiG5fcacoUZY5Xa6PEQ==",
+      "bin": {
+        "marked": "bin/marked.js"
+      },
+      "engines": {
+        "node": ">= 18"
+      }
+    },
     "node_modules/mdast-util-directive": {
       "version": "3.0.0",
       "resolved": "https://registry.npmjs.org/mdast-util-directive/-/mdast-util-directive-3.0.0.tgz",
diff --git a/docs/package.json b/docs/package.json
index 8a50a3c89..a1b6d2b65 100644
--- a/docs/package.json
+++ b/docs/package.json
@@ -17,7 +17,7 @@
   "dependencies": {
     "@docusaurus/core": "3.1.0",
     "@docusaurus/preset-classic": "3.1.0",
-    "@easyops-cn/docusaurus-search-local": "^0.40.1",
+    "@getcanary/web": "^1.0.9",
     "@mdx-js/react": "^3.0.0",
     "clsx": "^2.0.0",
     "prism-react-renderer": "^2.3.0",
diff --git a/docs/src/css/custom.css b/docs/src/css/custom.css
index 9fb67458e..e4c874a92 100644
--- a/docs/src/css/custom.css
+++ b/docs/src/css/custom.css
@@ -75,4 +75,18 @@ html[data-theme='dark'] .header-github-link:before {
 html[data-theme='dark'] .header-discord-link:before {
   background: url("data:image/svg+xml,%3Csvg viewBox='0 0 24 24' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath fill='white' d='M20.317 4.3698a19.7913 19.7913 0 00-4.8851-1.5152.0741.0741 0 00-.0785.0371c-.211.3753-.4447.8648-.6083 1.2495-1.8447-.2762-3.68-.2762-5.4868 0-.1636-.3933-.4058-.8742-.6177-1.2495a.077.077 0 00-.0785-.037 19.7363 19.7363 0 00-4.8852 1.515.0699.0699 0 00-.0321.0277C.5334 9.0458-.319 13.5799.0992 18.0578a.0824.0824 0 00.0312.0561c2.0528 1.5076 4.0413 2.4228 5.9929 3.0294a.0777.0777 0 00.0842-.0276c.4616-.6304.8731-1.2952 1.226-1.9942a.076.076 0 00-.0416-.1057c-.6528-.2476-1.2743-.5495-1.8722-.8923a.077.077 0 01-.0076-.1277c.1258-.0943.2517-.1923.3718-.2914a.0743.0743 0 01.0776-.0105c3.9278 1.7933 8.18 1.7933 12.0614 0a.0739.0739 0 01.0785.0095c.1202.099.246.1981.3728.2924a.077.077 0 01-.0066.1276 12.2986 12.2986 0 01-1.873.8914.0766.0766 0 00-.0407.1067c.3604.698.7719 1.3628 1.225 1.9932a.076.076 0 00.0842.0286c1.961-.6067 3.9495-1.5219 6.0023-3.0294a.077.077 0 00.0313-.0552c.5004-5.177-.8382-9.6739-3.5485-13.6604a.061.061 0 00-.0312-.0286zM8.02 15.3312c-1.1825 0-2.1569-1.0857-2.1569-2.419 0-1.3332.9555-2.4189 2.157-2.4189 1.2108 0 2.1757 1.0952 2.1568 2.419 0 1.3332-.9555 2.4189-2.1569 2.4189zm7.9748 0c-1.1825 0-2.1569-1.0857-2.1569-2.419 0-1.3332.9554-2.4189 2.1569-2.4189 1.2108 0 2.1757 1.0952 2.1568 2.419 0 1.3332-.946 2.4189-2.1568 2.4189Z'/%3E%3C/svg%3E")
     no-repeat center center;
-}
\ No newline at end of file
+}
+
+canary-root {
+  /* https://getcanary.dev/docs/common/customization/styling#css-variables */
+  --canary-color-primary-c: 0.15;
+  --canary-color-primary-h: 270;
+}
+
+canary-trigger-searchbar::part(button) {
+  background-color: var(--ifm-background-color);
+}
+
+canary-trigger-searchbar::part(kbd) {
+  background-color: var(--ifm-background-color);
+}
diff --git a/docs/src/theme/SearchBar.js b/docs/src/theme/SearchBar.js
new file mode 100644
index 000000000..b625b372e
--- /dev/null
+++ b/docs/src/theme/SearchBar.js
@@ -0,0 +1,65 @@
+import React from "react";
+
+export default function CanarySearchBar() {
+  const [loaded, setLoaded] = React.useState(false);
+
+  const TABS = JSON.stringify([
+    {
+      name: "Docs",
+      pattern: "**/docs/**",
+      options: { ignore: "**/api/**" },
+    },
+    {
+      name: "API",
+      pattern: "**/api/**",
+    },
+    {
+      name: "Github",
+      pattern: "**/github.com/**",
+    },
+  ]);
+
+  React.useEffect(() => {
+    Promise.all([
+      import("@getcanary/web/components/canary-root.js"),
+      import("@getcanary/web/components/canary-provider-cloud.js"),
+      import("@getcanary/web/components/canary-modal.js"),
+      import("@getcanary/web/components/canary-trigger-searchbar.js"),
+      import("@getcanary/web/components/canary-content.js"),
+      import("@getcanary/web/components/canary-input.js"),
+      import("@getcanary/web/components/canary-search.js"),
+      import("@getcanary/web/components/canary-search-results.js"),
+      import("@getcanary/web/components/canary-search-match-github-issue.js"),
+      import("@getcanary/web/components/canary-search-match-github-discussion.js"),
+      import("@getcanary/web/components/canary-filter-tabs-glob.js"),
+      import("@getcanary/web/components/canary-footer.js"),
+    ]).then(() => {
+      setLoaded(true);
+    });
+  }, []);
+
+  if (!loaded) {
+    return null;
+  }
+
+  return (
+    <canary-root framework="docusaurus">
+      <canary-provider-cloud project-key="cpab9997bf">
+        <canary-modal>
+          <canary-trigger-searchbar slot="trigger"></canary-trigger-searchbar>
+          <canary-content slot="content">
+            <canary-input slot="input"></canary-input>
+            <canary-search slot="mode">
+              <canary-filter-tabs-glob
+                slot="head"
+                tabs={TABS}
+              ></canary-filter-tabs-glob>
+              <canary-search-results slot="body"></canary-search-results>
+            </canary-search>
+            <canary-footer slot="footer"></canary-footer>
+          </canary-content>
+        </canary-modal>
+      </canary-provider-cloud>
+    </canary-root>
+  );
+}
diff --git a/dsp/modules/dummy_lm.py b/dsp/modules/dummy_lm.py
index 1bd6a04a1..49f35fa72 100644
--- a/dsp/modules/dummy_lm.py
+++ b/dsp/modules/dummy_lm.py
@@ -5,7 +5,7 @@
 
 
 # This testing module was moved in PR #735 to patch Arize Phoenix logging
-class DummyLM(LM):
+class DSPDummyLM(LM):
     """Dummy language model for unit testing purposes."""
 
     def __init__(self, answers: Union[list[str], dict[str, str]], follow_examples: bool = False):
@@ -61,7 +61,7 @@ def basic_request(self, prompt, n=1, **kwargs) -> dict[str, list[dict[str, str]]
                 },
             )
 
-            RED, GREEN, RESET = "\033[91m", "\033[92m", "\033[0m"
+            RED, _, RESET = "\033[91m", "\033[92m", "\033[0m"
             print("=== DummyLM ===")
             print(prompt, end="")
             print(f"{RED}{answer}{RESET}")
diff --git a/dsp/utils/settings.py b/dsp/utils/settings.py
index df073c707..61fc37ae7 100644
--- a/dsp/utils/settings.py
+++ b/dsp/utils/settings.py
@@ -1,8 +1,30 @@
 import threading
+from copy import deepcopy
 from contextlib import contextmanager
 
 from dsp.utils.utils import dotdict
 
+DEFAULT_CONFIG = dotdict(
+    lm=None,
+    adapter=None,
+    rm=None,
+    branch_idx=0,
+    reranker=None,
+    compiled_lm=None,
+    force_reuse_cached_compilation=False,
+    compiling=False,
+    skip_logprobs=False,
+    trace=[],
+    release=0,
+    bypass_assert=False,
+    bypass_suggest=False,
+    assert_failures=0,
+    suggest_failures=0,
+    langchain_history=[],
+    experimental=False,
+    backoff_time=10,
+)
+
 
 class Settings:
     """DSP configuration settings."""
@@ -25,27 +47,9 @@ def __new__(cls):
             #  TODO: remove first-class support for re-ranker and potentially combine with RM to form a pipeline of sorts
             #  eg: RetrieveThenRerankPipeline(RetrievalModel, Reranker)
             #  downstream operations like dsp.retrieve would use configs from the defined pipeline.
-            config = dotdict(
-                lm=None,
-                adapter=None,
-                rm=None,
-                branch_idx=0,
-                reranker=None,
-                compiled_lm=None,
-                force_reuse_cached_compilation=False,
-                compiling=False,  # TODO: can probably be removed
-                skip_logprobs=False,
-                trace=[],
-                release=0,
-                bypass_assert=False,
-                bypass_suggest=False,
-                assert_failures=0,
-                suggest_failures=0,
-                langchain_history=[],
-                experimental=False,
-                backoff_time = 10
-            )
-            cls._instance.__append(config)
+
+            # make a deepcopy of the default config to avoid modifying the default config
+            cls._instance.__append(deepcopy(DEFAULT_CONFIG))
 
         return cls._instance
 
diff --git a/dspy/.internal_dspyai/setup.py b/dspy/.internal_dspyai/setup.py
new file mode 100644
index 000000000..726145521
--- /dev/null
+++ b/dspy/.internal_dspyai/setup.py
@@ -0,0 +1,23 @@
+from setuptools import find_packages, setup
+
+# Read the content of the README file
+with open("README.md", encoding="utf-8") as f:
+    long_description = f.read()
+
+setup(
+    #replace_package_name_marker
+    name="dspy-ai",
+    #replace_package_version_marker
+    version="2.5.5", 	
+    description="DSPy",	
+    long_description=long_description,	
+    long_description_content_type="text/markdown",	
+    url="https://github.com/stanfordnlp/dsp",	
+    author="Omar Khattab",	
+    author_email="okhattab@stanford.edu",	
+    license="MIT License",	
+    packages=find_packages(include=["dsp.*", "dspy.*", "dsp", "dspy"]),	
+    python_requires=">=3.9",
+    #replace_dspy_version_marker
+    install_requires=["dspy==2.5.3"]
+)	
diff --git a/dspy/adapters/base.py b/dspy/adapters/base.py
index ddb0ee3e8..914e31bd1 100644
--- a/dspy/adapters/base.py
+++ b/dspy/adapters/base.py
@@ -7,8 +7,12 @@ def __call__(self, lm, lm_kwargs, signature, demos, inputs, _parse_values=True):
         values = []
 
         for output in outputs:
-            value = self.parse(signature, output, _parse_values=_parse_values)
-            assert set(value.keys()) == set(signature.output_fields.keys()), f"Expected {signature.output_fields.keys()} but got {value.keys()}"
-            values.append(value)
+            try:
+                value = self.parse(signature, output, _parse_values=_parse_values)
+                assert set(value.keys()) == set(signature.output_fields.keys()), f"Expected {signature.output_fields.keys()} but got {value.keys()}"
+                values.append(value)
+            except Exception as e:
+                # print(f"Error parsing output: {output} for input: {inputs}")
+                raise e
         
         return values
diff --git a/dspy/adapters/chat_adapter.py b/dspy/adapters/chat_adapter.py
index e2c3abeba..03f1236ca 100644
--- a/dspy/adapters/chat_adapter.py
+++ b/dspy/adapters/chat_adapter.py
@@ -1,13 +1,15 @@
-import re
 import ast
 import json
+import re
 import textwrap
+from typing import get_args, get_origin
 
+import pydantic
 from pydantic import TypeAdapter
+
 from .base import Adapter
-from typing import get_origin, get_args
 
-field_header_pattern = re.compile(r'\[\[ ## (\w+) ## \]\]')
+field_header_pattern = re.compile(r"\[\[ ## (\w+) ## \]\]")
 
 
 class ChatAdapter(Adapter):
@@ -20,9 +22,11 @@ def format(self, signature, demos, inputs):
         # Extract demos where some of the output_fields are not filled in.
         incomplete_demos = [demo for demo in demos if not all(k in demo for k in signature.fields)]
         complete_demos = [demo for demo in demos if demo not in incomplete_demos]
-        incomplete_demos = [demo for demo in incomplete_demos \
-                            if any(k in demo for k in signature.input_fields) and \
-                                any(k in demo for k in signature.output_fields)]
+        incomplete_demos = [
+            demo
+            for demo in incomplete_demos
+            if any(k in demo for k in signature.input_fields) and any(k in demo for k in signature.output_fields)
+        ]
 
         demos = incomplete_demos + complete_demos
 
@@ -31,20 +35,22 @@ def format(self, signature, demos, inputs):
         for demo in demos:
             messages.append(format_turn(signature, demo, role="user", incomplete=demo in incomplete_demos))
             messages.append(format_turn(signature, demo, role="assistant", incomplete=demo in incomplete_demos))
-        
+
         messages.append(format_turn(signature, inputs, role="user"))
 
         return messages
-    
+
     def parse(self, signature, completion, _parse_values=True):
         sections = [(None, [])]
 
         for line in completion.splitlines():
             match = field_header_pattern.match(line.strip())
-            if match: sections.append((match.group(1), []))
-            else: sections[-1][1].append(line)
+            if match:
+                sections.append((match.group(1), []))
+            else:
+                sections[-1][1].append(line)
 
-        sections = [(k, '\n'.join(v).strip()) for k, v in sections]
+        sections = [(k, "\n".join(v).strip()) for k, v in sections]
 
         fields = {}
         for k, v in sections:
@@ -52,48 +58,84 @@ def parse(self, signature, completion, _parse_values=True):
                 try:
                     fields[k] = parse_value(v, signature.output_fields[k].annotation) if _parse_values else v
                 except Exception as e:
-                    raise ValueError(f"Error parsing field {k}: {e}.\n\n\t\tOn attempting to parse the value\n```\n{v}\n```")
+                    raise ValueError(
+                        f"Error parsing field {k}: {e}.\n\n\t\tOn attempting to parse the value\n```\n{v}\n```"
+                    )
 
-        if fields.keys() != signature.output_fields.keys():
-            raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")
+        if list(fields.keys()) != list(signature.output_fields.keys()):
+            raise ValueError(f"Expected {list(signature.output_fields.keys())} but got {fields.keys()}")
 
         return fields
 
+    def format_completion(self, signature, outputs):
+        reconstructed = []
+        
+        fields_dict = signature.output_fields
+
+        field_name_output_map = {field: outputs[field] for field in fields_dict.keys()}
+
+        for field, value in field_name_output_map.items():
+            reconstructed.append(f"[[ ## {field} ## ]]")
+            reconstructed.append(str(value))
+            reconstructed.append("")  # Add an empty line for separation
+        
+        reconstructed.append("[[ ## completed ## ]]")
+
+        result = "\n".join(reconstructed).strip()
+
+        return result
+
 def format_blob(blob):
-    if '\n' not in blob and "«" not in blob and "»" not in blob: return f"«{blob}»"
+    if "\n" not in blob and "«" not in blob and "»" not in blob:
+        return f"«{blob}»"
 
-    modified_blob = blob.replace('\n', '\n    ')
+    modified_blob = blob.replace("\n", "\n    ")
     return f"«««\n    {modified_blob}\n»»»"
 
 
 def format_list(items):
-    if len(items) == 0: return "N/A"
-    if len(items) == 1: return format_blob(items[0])
+    if len(items) == 0:
+        return "N/A"
+    if len(items) == 1:
+        return format_blob(items[0])
 
     return "\n".join([f"[{idx+1}] {format_blob(txt)}" for idx, txt in enumerate(items)])
 
 
+def _format_field_value(value) -> str:
+    if isinstance(value, list):
+        return format_list(value)
+    elif isinstance(value, pydantic.BaseModel):
+        return value.model_dump_json()
+    else:
+        return str(value)
+
+
 def format_fields(fields):
     output = []
     for k, v in fields.items():
-        v = v if not isinstance(v, list) else format_list(v)
+        v = _format_field_value(v)
         output.append(f"[[ ## {k} ## ]]\n{v}")
 
-    return '\n\n'.join(output).strip()
-        
+    return "\n\n".join(output).strip()
+
 
 def parse_value(value, annotation):
-    if annotation is str: return str(value)
+    if annotation is str:
+        return str(value)
     parsed_value = value
     if isinstance(value, str):
-        try: parsed_value = json.loads(value)
+        try:
+            parsed_value = json.loads(value)
         except json.JSONDecodeError:
-            try: parsed_value = ast.literal_eval(value)
-            except (ValueError, SyntaxError): parsed_value = value
+            try:
+                parsed_value = ast.literal_eval(value)
+            except (ValueError, SyntaxError):
+                parsed_value = value
     return TypeAdapter(annotation).validate_python(parsed_value)
 
 
-def format_turn(signature, values, role, incomplete=False):       
+def format_turn(signature, values, role, incomplete=False):
     content = []
 
     if role == "user":
@@ -101,42 +143,45 @@ def format_turn(signature, values, role, incomplete=False):
         if incomplete:
             content.append("This is an example of the task, though some input or output fields are not supplied.")
     else:
-        field_names, values = list(signature.output_fields.keys()) + ['completed'], {**values, 'completed': ''}
+        field_names, values = list(signature.output_fields.keys()) + ["completed"], {**values, "completed": ""}
 
     if not incomplete:
         if not set(values).issuperset(set(field_names)):
             raise ValueError(f"Expected {field_names} but got {values.keys()}")
-    
+
     content.append(format_fields({k: values.get(k, "Not supplied for this particular example.") for k in field_names}))
 
     if role == "user":
-        content.append("Respond with the corresponding output fields, starting with the field " +
-                       ", then ".join(f"`{f}`" for f in signature.output_fields) +
-                       ", and then ending with the marker for `completed`.")
+        content.append(f"Respond with the corresponding output fields using the proper format of "
+                         f"[[ ## <field_name> ## ]] followed by the field value. Start with the field "
+                         f"{', then '.join(f'`[[ ## {f} ## ]]`' for f in signature.output_fields)}, "
+                         f"and then ending with the marker for `completed`.")
 
-    return {"role": role, "content": '\n\n'.join(content).strip()}
+    return {"role": role, "content": "\n\n".join(content).strip()}
 
 
 def get_annotation_name(annotation):
     origin = get_origin(annotation)
     args = get_args(annotation)
     if origin is None:
-        if hasattr(annotation, '__name__'):
+        if hasattr(annotation, "__name__"):
             return annotation.__name__
         else:
             return str(annotation)
     else:
-        args_str = ', '.join(get_annotation_name(arg) for arg in args)
-        return f"{origin.__name__}[{args_str}]"
+        args_str = ", ".join(get_annotation_name(arg) for arg in args)
+        return f"{get_annotation_name(origin)}[{args_str}]"
+
 
 def enumerate_fields(fields):
     parts = []
     for idx, (k, v) in enumerate(fields.items()):
         parts.append(f"{idx+1}. `{k}`")
         parts[-1] += f" ({get_annotation_name(v.annotation)})"
-        parts[-1] += f": {v.json_schema_extra['desc']}" if v.json_schema_extra['desc'] != f'${{{k}}}' else ''
+        parts[-1] += f": {v.json_schema_extra['desc']}" if v.json_schema_extra["desc"] != f"${{{k}}}" else ""
+
+    return "\n".join(parts).strip()
 
-    return '\n'.join(parts).strip()
 
 def prepare_instructions(signature):
     parts = []
@@ -144,12 +189,12 @@ def prepare_instructions(signature):
     parts.append("Your output fields are:\n" + enumerate_fields(signature.output_fields))
     parts.append("All interactions will be structured in the following way, with the appropriate values filled in.")
 
-    parts.append(format_fields({f : f"{{{f}}}" for f in signature.input_fields}))
-    parts.append(format_fields({f : f"{{{f}}}" for f in signature.output_fields}))
-    parts.append(format_fields({'completed' : ""}))
+    parts.append(format_fields({f: f"{{{f}}}" for f in signature.input_fields}))
+    parts.append(format_fields({f: f"{{{f}}}" for f in signature.output_fields}))
+    parts.append(format_fields({"completed": ""}))
 
     instructions = textwrap.dedent(signature.instructions)
-    objective = ('\n' + ' ' * 8).join([''] + instructions.splitlines())
+    objective = ("\n" + " " * 8).join([""] + instructions.splitlines())
     parts.append(f"In adhering to this structure, your objective is: {objective}")
 
     # parts.append("You will receive some input fields in each interaction. " +
@@ -157,4 +202,4 @@ def prepare_instructions(signature):
     #              ", then ".join(f"`{f}`" for f in signature.output_fields) +
     #              ", and then ending with the marker for `completed`.")
 
-    return '\n\n'.join(parts).strip()
+    return "\n\n".join(parts).strip()
diff --git a/dspy/clients/anyscale.py b/dspy/clients/anyscale.py
index 53a0f0c26..6ec76c082 100644
--- a/dspy/clients/anyscale.py
+++ b/dspy/clients/anyscale.py
@@ -1,15 +1,23 @@
-import re
-import time
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Union
+import json
+import yaml
+import os
+from concurrent.futures import ThreadPoolExecutor
+from collections import defaultdict
+import functools
 
-import dspy
 from dspy import logger
 from dspy.clients.finetune import (
     FinetuneJob,
-    TrainingMethod,
-    get_finetune_directory,
-    validate_training_data
+    TrainingMethod
 )
+import asyncio
+
+try:
+    from anyscale.job import JobConfig
+    import anyscale
+except ImportError:
+    anyscale = None
 
 #-------------------------------------------------------------------------------
 #    Variables
@@ -41,6 +49,23 @@ def anyscale_model_kill(model: str, launch_kwargs: Dict[str, Any]):
 #-------------------------------------------------------------------------------
 
 class FinetuneJobAnyScale(FinetuneJob):
+    def __init__(self,
+        model: str,
+        # message_completion_pairs: List[Dict[str, str]],
+        train_path: str,
+        eval_path: Optional[str],
+        train_kwargs: Optional[Dict[str, Any]]=None,
+        serve_config_path: str = "serve_1B.yaml"
+    ):
+        super().__init__(model, train_path, eval_path, train_kwargs)
+        self.model = model
+        # self.message_completion_pairs = message_completion_pairs
+        self.train_path = train_path
+        self.train_kwargs: Dict[str, Any] = train_kwargs or {}
+        if "model" not in self.train_kwargs:
+            self.train_kwargs["model"] = model
+        self.job_id = None
+        self.serve_config_path = serve_config_path
     
     def cancel(self):
         """Cancel the finetune job."""
@@ -53,57 +78,507 @@ def status(self):
         """Get the status of the finetune job."""
         raise NotImplementedError("Method `status` is not implemented.")
 
+    async def run_finetune(self):
+        """Start the finetune job."""
+        try:
+            logger.info("[Finetune] Starting training process...")
+            if TrainingMethod.SFT not in TRAINING_METHODS_ANYSCALE:
+                raise NotImplementedError(f"AnyScale can only support {TRAINING_METHODS_ANYSCALE} for the time being")
+
+            logger.info("[Finetune] Reading and validating datasets...")
+            # Convert the data from prompt completion to OAI compatible messages
+            train_dataset = read_jsonl(self.train_path)
+            val_dataset = read_jsonl(self.eval_path) if self.eval_path else None
+
+            if not verify_datasets(train_dataset, val_dataset):
+                logger.error("[Finetune] Error: Unable to verify arguments")
+                raise RuntimeError("Unable to verify argument")
+
+            logger.info("[Finetune] Converting data to JSONL format...")
+            for path, dataset in [(self.train_path, train_dataset), (self.eval_path, val_dataset)]:
+                if not (path and dataset):
+                    continue
+                write_jsonl(path, dataset)
+
+            logger.info("[Finetune] Submitting data to remote storage...")
+            remote_train_path, remote_eval_path = submit_data(train_path=self.train_path, eval_path=self.eval_path)
+            logger.info(f"[Finetune] Data submitted. Remote train path: {remote_train_path}, Remote eval path: {remote_eval_path}")
+
+            logger.info("[Finetune] Generating configuration files...")
+            _, compute_config = generate_config_files(train_path=remote_train_path, eval_path=remote_eval_path, **self.train_kwargs)
+            
+            logger.info("[Finetune] Starting remote training...")
+            job_id = start_remote_training(compute_config=compute_config, **self.train_kwargs)
+            self.job_id = job_id
+            logger.info(f"[Finetune] Remote training started. Job ID: {job_id}")
+
+            logger.info("[Finetune] Waiting for training to complete...")
+            await self.wait_for_training()
+            logger.info("[Finetune] Training completed.")
+
+            logger.info("[Finetune] Retrieving model information...")
+            model_info = get_model_info(job_id)
+            logger.info(f"[Finetune] Model info retrieved: {model_info}")
+
+            storage_uri = model_info["storage_uri"]
+            logger.info(f"[Finetune] Copying LoRA weights from {storage_uri}...")
+            model_names, lora_dynamic_path = copy_lora_weights(storage_uri, model_info, job_id)
+            logger.info(f"[Finetune] LoRA weights copied. Model names: {model_names}")
+
+            logger.info("[Finetune] Setting result in future object...")
+            for model_name in model_names:
+                yield model_name
+            logger.info("[Finetune] Training process completed successfully.")
+
+            logger.info("[Finetune] Updating model config with the proper dynamic path")
+            update_model_config(lora_dynamic_path, self.serve_config_path)
+
+        except Exception as e:
+            logger.error(f"[Finetune] Error occurred during training: {str(e)}")
+            raise e
+
+        # executor = ThreadPoolExecutor(max_workers=1)
+        # executor.submit(
+        #     execute_finetune_job,
+        #     finetune_job,
+        #     launch_kwargs=launch_kwargs,
+        #     cache_finetune=cache_finetune
+        # )
+        # executor.shutdown(wait=False)
+    
+    async def wait_for_training(self):
+        """Wait for the training to complete in a non-blocking manner."""
+        loop = asyncio.get_running_loop()
+        
+        # Run the blocking `anyscale.job.wait` in a separate thread
+        
+        with ThreadPoolExecutor() as pool:
+            await loop.run_in_executor(pool, functools.partial(anyscale.job.wait, id=self.job_id, timeout_s=18000))
+
+    
+
 
 def is_anyscale_model(model: str) -> bool:
     """Check if the model is an AnyScale model."""
     logger.info("Is AnyScale model is not implemented, returning False as a default to not break lm.py")
     return False
 
+# async def _run_finetune(job: FinetuneJobAnyScale):
+#         try:
+#             logger.info("[Finetune] Starting training process...")
+#             if TrainingMethod.SFT not in TRAINING_METHODS_ANYSCALE:
+#                 raise NotImplementedError(f"AnyScale can only support {TRAINING_METHODS_ANYSCALE} for the time being")
 
-def finetune_anyscale(
-        job: FinetuneJobAnyScale,
-        model: str,
-        message_completion_pairs: List[Dict[str, str]],
-        train_kwargs: Optional[Dict[str, Any]]=None,
-    ) -> str:
-    """Fine-tune with AnyScale."""
-    # Fake fine-tuning
-    logger.info("[Finetune] Fake fine-tuning")
-    train_kwargs = train_kwargs or {}
+#             logger.info("[Finetune] Reading and validating datasets...")
+#             # Convert the data from prompt completion to OAI compatible messages
+#             train_dataset = read_jsonl(train_path)
+#             val_dataset = read_jsonl(eval_path) if eval_path else None
+
+#             if not verify_datasets(train_dataset, val_dataset):
+#                 logger.error("[Finetune] Error: Unable to verify arguments")
+#                 raise RuntimeError("Unable to verify argument")
+
+#             logger.info("[Finetune] Converting data to JSONL format...")
+#             for path, dataset in [(train_path, train_dataset), (eval_path, val_dataset)]:
+#                 if not (path and dataset):
+#                     continue
+#                 write_jsonl(path, dataset)
+
+#             logger.info("[Finetune] Submitting data to remote storage...")
+#             remote_train_path, remote_eval_path = submit_data(train_path=train_path, eval_path=eval_path)
+#             logger.info(f"[Finetune] Data submitted. Remote train path: {remote_train_path}, Remote eval path: {remote_eval_path}")
+
+#             logger.info("[Finetune] Generating configuration files...")
+#             _, compute_config = generate_config_files(train_path=remote_train_path, eval_path=remote_eval_path, **train_kwargs)
+            
+#             logger.info("[Finetune] Starting remote training...")
+#             job_id = start_remote_training(compute_config=compute_config, **train_kwargs)
+#             job.job_id = job_id
+#             logger.info(f"[Finetune] Remote training started. Job ID: {job_id}")
+
+#             logger.info("[Finetune] Waiting for training to complete...")
+#             wait_for_training(job)
+#             logger.info("[Finetune] Training completed.")
+
+#             logger.info("[Finetune] Retrieving model information...")
+#             model_info = get_model_info(job_id)
+#             logger.info(f"[Finetune] Model info retrieved: {model_info}")
 
+#             storage_uri = model_info["storage_uri"]
+#             logger.info(f"[Finetune] Copying LoRA weights from {storage_uri}...")
+#             model_names = copy_lora_weights(storage_uri, model_info, job_id)
+#             logger.info(f"[Finetune] LoRA weights copied. Model names: {model_names}")
+
+#             logger.info("[Finetune] Setting result in future object...")
+#             job.set_result(model_names)
+#             logger.info("[Finetune] Training process completed successfully.")
+
+#         except Exception as e:
+#             logger.error(f"[Finetune] Error occurred during training: {str(e)}")
+#             job.set_exception(e)
+
+
+# async def finetune_anyscale(
+#     model: str,
+#     train_path: str,
+#     eval_path: Optional[str],
+#     train_kwargs: Optional[Dict[str, Any]] = None,
+# ) -> FinetuneJobAnyScale:
+#     """
+#     Initiate fine-tuning process for an Anyscale model.
+    
+#     Args:
+#         model (str): The name of the Anyscale model to fine-tune.
+#         message_completion_pairs (List[Dict[str, str]]): List of prompt-completion pairs for training.
+#         train_kwargs (Optional[Dict[str, Any]]): Additional training parameters.
+    
+#     Returns:
+#         FinetuneJobAnyScale: A job object representing the fine-tuning process.
+#     """
+#     train_kwargs = train_kwargs or {}
+    
+    
+
+#     executor = ThreadPoolExecutor(max_workers=1)
+#     executor.submit(_run_finetune)
+    
+#     return job
+
+def update_model_config(lora_dynamic_path: str, serve_config_path: str):
+    """Update the model config storage location with the job_id."""
+    with open(serve_config_path, "r") as f:
+        serve_config = yaml.safe_load(f)
+    
+    model_config_location = serve_config["applications"][0]["args"]["llm_configs"][0]
+    
+    with open(model_config_location, "r") as f:
+        model_config = yaml.safe_load(f)
+
+    model_config["lora_config"]["dynamic_lora_loading_path"] = lora_dynamic_path
+    
+    with open(model_config_location, "w") as f:
+        yaml.safe_dump(model_config, f)
+    
+
+def verify_datasets(dataset: List[dict[str, Any]], valset: Optional[List[dict[str, Any]]]) -> bool:
+    """Verify the training arguments before starting training."""
+    def validate_dataset(name, data: dict[str, Any]) -> bool:
+        dataset_validation = openai_data_validation(data)
+
+        if dataset_validation:
+            logger.error(f"Dataset validation failed: {dataset_validation}")
+            return False
+
+        if name == "train":
+            convo_lens = check_message_lengths(data)
+            estimate_cost(data, convo_lens=convo_lens)
+        return True
+
+    datasets = {"train": dataset}
+    if valset:
+        datasets["val"] = valset
+
+    for name, data in datasets.items():
+        if not validate_dataset(name, data):
+            return False
+    return True
+
+def submit_data(train_path: str, eval_path: Optional[str]):
+    """Upload the data to the Workspace cloud storage."""
+    storage = os.environ['ANYSCALE_ARTIFACT_STORAGE']
+    
+    datasets = {"train": train_path}
+    if eval_path:
+        datasets["val"] = eval_path
+
+    fine_tuning_file_ids = {}
+    for name, path in datasets.items():
+        num_items = len(read_jsonl(path))
+        logger.info(f"Number of items in {name} data: {num_items}")
+
+        remote_path = os.path.join(storage, path.split("/")[-1])
+        logger.info(f"Uploading {name} data to S3 at {remote_path}")
+        if remote_path[:2] == "s3":
+            os.system(f"aws s3 cp {path} {remote_path}")
+        elif remote_path[:2] == "gs":
+            os.system(f"gcloud storage cp {path} {remote_path}")
+        else:
+            os.system(f"cp {path} {remote_path}")
+            logger.info(f"Copied {path} to {remote_path}")
+        fine_tuning_file_ids[name] = remote_path
+    
+    return fine_tuning_file_ids["train"], fine_tuning_file_ids.get("val", None)
+
+def generate_config_files(train_path: str, eval_path: Optional[str], **kwargs):
+    base_model_yaml_path = kwargs.get("train_config_yaml", None)
+    assert kwargs["model"] is not None, "Model is required to generate the config files"
+
+    use_lora = kwargs.get("use_lora", False)
+    example_dir = ""
+    lora_path = "configs/training/lora" if use_lora else "configs/training/full_param"
+
+
+    if not base_model_yaml_path:
+        def get_yaml_config(model_name):
+            if "llama" in model_name.lower():
+                if "70b" in model_name:
+                    return "llama-3-70b.yaml"
+                elif "13b" in model_name:
+                    return "llama-3-70b.yaml"
+                else:
+                    return "llama-3-8b.yaml"
+            elif "mistral" in model_name.lower():
+                if "mixtral" in model_name.lower():
+                    return "mixtral-8x7b.yaml"
+                else:
+                    return "mistral-7b.yaml"
+            else:
+                raise RuntimeError("No default yaml found for the model")
+
+        default_model_yaml_path = get_yaml_config(kwargs["model"])
+        base_model_yaml_path = os.path.join(example_dir, lora_path, default_model_yaml_path)
+        logger.info(f"Using default yaml template for model: {base_model_yaml_path}")
+        
+    model_config_data = yaml.safe_load(open(base_model_yaml_path, "r"))
+    model_config_data.update(kwargs.get("hyperparameters", {}))
+    
+    model_config_data["model_id"] = kwargs["model"]
+
+    custom_modifications = {
+        "model_id": kwargs["model"],
+        "train_path": train_path,
+        "valid_path": eval_path,
+        "logger": {
+            "provider": "wandb",
+        },
+        "num_checkpoints_to_keep": 10
+    }
+    if kwargs.get("output_dir", None):
+        custom_modifications["output_dir"] = kwargs["output_dir"]
+
+    model_config_data.update(custom_modifications)
+    model_config_data = {k: v for k, v in model_config_data.items() if v is not None}
+
+    def freeze(d):
+        if isinstance(d, dict):
+            return tuple(sorted((key, freeze(value)) for key, value in d.items()))
+        elif isinstance(d, list):
+            return tuple(freeze(value) for value in sorted(d))
+        elif isinstance(d, set):
+            return tuple(freeze(value) for value in sorted(d))
+        return d
+
+    def hash_dict(d):
+        return hash(freeze(d))
+    dict_sorted_hash = hash_dict(model_config_data)
+    if dict_sorted_hash < 0:
+        dict_sorted_hash = -dict_sorted_hash
+    filename = f"model_config_dspy_{dict_sorted_hash}.yaml"
+    logger.info(f"Model config data: {model_config_data}")
+    yaml.safe_dump(model_config_data, open(filename, "w"))
+
+    ft_path = os.path.join("utils", "ft.py")
+
+    compute_config_dict = {
+        "name": "dspy-llmforge-fine-tuning-job",
+        "entrypoint": f"llmforge anyscale finetune {filename}",
+        "working_dir": ".",
+        "image_uri": "localhost:5555/anyscale/llm-forge:0.5.6",
+        "requirements": [
+            "wandb",
+        ],
+        "env_vars": {
+            "WANDB_API_KEY": os.environ.get("WANDB_API_KEY", ""),
+            "HF_TOKEN": os.environ.get("HF_TOKEN", ""),
+            "HF_HOME": os.environ.get("HF_HOME", ""),
+        }
+    }
+    compute_config_kwargs = kwargs.get("compute_config", {})
+    compute_config_dict.update(compute_config_kwargs)
+    compute_config = JobConfig(**compute_config_dict)
+
+    job_runner_config_path = kwargs.get("compute_yaml_path", "job_runner_config.yaml")
+
+    return job_runner_config_path, compute_config
+
+def start_remote_training(compute_config, **kwargs) -> str:
+    job_id: str = anyscale.job.submit(compute_config)
+    return job_id
+
+def wait_for_training(job):
+    logger.info("Waiting for training to complete")
+    anyscale.job.wait(id=job.job_id, timeout_s=18000)
+
+def get_model_info(job_id):
+    return anyscale.llm.model.get(job_id=job_id).to_dict()
+
+def copy_lora_weights(storage_uri, model_info, job_id):
     try:
-      logger.info("[Finetune] Validate the formatting of the fine-tuning data")
-      training_method = TrainingMethod.SFT  # Hardcoded
-      validate_training_data(message_completion_pairs, training_method)
-      time.sleep(5)
-      logger.info("[Finetune] Done!")
-
-      logger.info("[Finetune] Saving the data to a file")
-      finetune_parent_dir = get_finetune_directory()
-      # We utilize message_completion_pairs here
-      time.sleep(1)
-      logger.info("[Finetune] Done!")
-
-      logger.info("[Finetune] Uploading the data to the cloud")
-      time.sleep(1)
-      logger.info("[Finetune] Done!")
-
-      logger.info("[Finetune] Launch training")
-      # We utilize model and train_kwargs here
-      time.sleep(1)
-      logger.info("[Finetune] Done!")
-
-      logger.info("[Finetune] Wait for training to complete")
-      time.sleep(1)
-      logger.info("[Finetune] Done!")
-
-      logger.info("[Finetune] Get trained model client")
-      model = "anyscale_model"  # Hardcoded
-      time.sleep(1)
-      logger.info("[Finetune] Done!")
-
-      return model
+        from google.cloud import storage
+
+        storage_client = storage.Client()
+
+        bucket_name = storage_uri.split('/')[2]
+        source_folder = '/'.join(storage_uri.split('/')[3:-1])
+        logger.info(f"Source folder: {source_folder}")
 
+        bucket = storage_client.bucket(bucket_name)
+
+        blobs = bucket.list_blobs(prefix=source_folder)
+
+        subfolders = set()
+        for blob in blobs:
+            if '/' in blob.name[len(source_folder):]:
+                subfolder = blob.name.split('/')[:-1]
+                subfolders.add('/'.join(subfolder))
+
+        base_model_id = model_info["base_model_id"]
+        lora_dynamic_path = f"dspy/lora_weights/{job_id}/{base_model_id}"
+
+        model_names = []
+        for subfolder in subfolders:
+            subfolder_name = subfolder.split('/')[-1]
+            destination_folder = f"{lora_dynamic_path}:{subfolder_name}"
+            if subfolder_name.startswith("epoch"):
+                model_names.append("/".join(destination_folder.split("/")[-2:]))
+            else:
+                continue
+            
+            subfolder_blobs = bucket.list_blobs(prefix=subfolder)
+            
+            for blob in subfolder_blobs:
+                source_blob = bucket.blob(blob.name)
+                destination_blob_name = f"{destination_folder}/{blob.name.split('/')[-1]}"
+                bucket.copy_blob(source_blob, bucket, destination_blob_name)
+                logger.info(f"Copied {source_blob.name} to {destination_blob_name}")
+
+        logger.info(f"All subfolders copied to: gs://{bucket_name}/{lora_dynamic_path}")
+        completed_path = f"gs://{bucket_name}/{lora_dynamic_path}"
+        return model_names, completed_path
+    
     except Exception as e:
-      logger.error(f"[Finetune] Error: {e}")
-      raise e
+        logger.error(f"An error occurred: {str(e)}")
+        raise e
+
+def read_jsonl(filename):
+    with open(filename, "r") as f:
+        return [json.loads(line) for line in f]
+
+def write_jsonl(filename, data):
+    with open(filename, "w") as f:
+        for item in data:
+            f.write(json.dumps(item) + "\n")
+
+# Fine tuning utils
+def openai_data_validation(dataset: List[dict[str, Any]]) -> Union[dict[str, Any], AssertionError]:
+    format_errors = defaultdict(int)
+    for ex in dataset:
+        if not isinstance(ex, dict):
+            format_errors["data_type"] += 1
+            continue
+
+        messages = ex.get("messages", None)
+        if not messages:
+            format_errors["missing_messages_list"] += 1
+            continue
+
+        for message in messages:
+            if "role" not in message or "content" not in message:
+                format_errors["message_missing_key"] += 1
+
+            if any(k not in ("role", "content", "name", "function_call", "weight") for k in message):
+                format_errors["message_unrecognized_key"] += 1
+
+            if message.get("role", None) not in ("system", "user", "assistant", "function"):
+                format_errors["unrecognized_role"] += 1
+
+            content = message.get("content", None)
+            function_call = message.get("function_call", None)
+
+            if (not content and not function_call) or not isinstance(content, str):
+                format_errors["missing_content"] += 1
+
+        if not any(message.get("role", None) == "assistant" for message in messages):
+            format_errors["example_missing_assistant_message"] += 1
+
+    if format_errors:
+        return format_errors
+    return None
+
+def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):
+    import tiktoken
+    encoding = tiktoken.get_encoding("cl100k_base")
+
+    num_tokens = 0
+    for message in messages:
+        num_tokens += tokens_per_message
+        for key, value in message.items():
+            num_tokens += len(encoding.encode(value))
+            if key == "name":
+                num_tokens += tokens_per_name
+    num_tokens += 3
+    return num_tokens
+
+def num_assistant_tokens_from_messages(messages):
+    import tiktoken
+    encoding = tiktoken.get_encoding("cl100k_base")
+
+    num_tokens = 0
+    for message in messages:
+        if message["role"] == "assistant":
+            num_tokens += len(encoding.encode(message["content"]))
+    return num_tokens
+
+def check_message_lengths(dataset: List[dict[str, Any]]) -> list[int]:
+    n_missing_system = 0
+    n_missing_user = 0
+    n_messages = []
+    convo_lens = []
+    assistant_message_lens = []
+
+    for ex in dataset:
+        messages = ex["messages"]
+        if not any(message["role"] == "system" for message in messages):
+            n_missing_system += 1
+        if not any(message["role"] == "user" for message in messages):
+            n_missing_user += 1
+        n_messages.append(len(messages))
+        convo_lens.append(num_tokens_from_messages(messages))
+        assistant_message_lens.append(num_assistant_tokens_from_messages(messages))
+    n_too_long = sum([length > 16385 for length in convo_lens])
+
+    if n_too_long > 0:
+        logger.info(f"There are {n_too_long} examples that may be over the 16,385 token limit, they will be truncated during fine-tuning.")
+
+    if n_missing_system > 0:
+        logger.info(f"There are {n_missing_system} examples that are missing a system message.")
+
+    if n_missing_user > 0:
+        logger.info(f"There are {n_missing_user} examples that are missing a user message.")
+
+    return convo_lens
+
+def estimate_cost(dataset: dict[str, Any], tokens_per_message=3, tokens_per_name=1, convo_lens=None):
+    MAX_TOKENS_PER_EXAMPLE = 16385
+    TARGET_EPOCHS = 3
+    MIN_TARGET_EXAMPLES = 100
+    MAX_TARGET_EXAMPLES = 25000
+    MIN_DEFAULT_EPOCHS = 1
+    MAX_DEFAULT_EPOCHS = 25
+
+    n_epochs = TARGET_EPOCHS
+    n_train_examples = len(dataset)
+    if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES:
+        n_epochs = min(MAX_DEFAULT_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)
+    elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES:
+        n_epochs = max(MIN_DEFAULT_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)
+
+    if convo_lens is None:
+        convo_lens = check_message_lengths(dataset)
+
+    n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)
+
+    logger.info(f"""The charge for finetuning is determined by the number of epochs multiplied by the number of billing tokens in the dataset. Here are the stats for this training dataset:
+    num_billing_tokens: {n_billing_tokens_in_dataset}
+    n_epochs: {n_epochs}
+    num_total_charge_tokens: {n_epochs * n_billing_tokens_in_dataset}""")
\ No newline at end of file
diff --git a/dspy/clients/finetune.py b/dspy/clients/finetune.py
index 7fa5b95af..8082eb6f6 100644
--- a/dspy/clients/finetune.py
+++ b/dspy/clients/finetune.py
@@ -1,12 +1,10 @@
-from abc import ABC, abstractmethod
-from concurrent.futures import ThreadPoolExecutor, Future
+from abc import abstractmethod
 from enum import Enum
 import os
 from pathlib import Path
 from typing import List, Dict, Any, Optional
 import ujson
 
-from dspy import logger
 
 
 # TODO: Move to a centralized location with all the other environment variables
@@ -40,22 +38,28 @@ class TrainingMethod(str, Enum):
 }
 
 
-class FinetuneJob(Future):
+class FinetuneJob():
 
     def __init__(self,
         model: str,
-        message_completion_pairs: List[Dict[str, str]],
+        # message_completion_pairs: List[Dict[str, str]],
+        train_path: str,
+        eval_path: Optional[str],
         train_kwargs: Optional[Dict[str, Any]]=None,
     ):
         self.model = model
-        self.message_completion_pairs = message_completion_pairs
+        # self.message_completion_pairs = message_completion_pairs
+        self.train_path = train_path
+        self.eval_path = eval_path
         self.train_kwargs: Dict[str, Any] = train_kwargs or {}
         super().__init__()
     
     def get_kwargs(self):
         return dict(
             model=self.model,
-            message_completion_pairs=self.message_completion_pairs,
+            # message_completion_pairs=self.message_completion_pairs,
+            train_path=self.train_path,
+            eval_path=self.eval_path,
             train_kwargs=self.train_kwargs,
         )
 
diff --git a/dspy/clients/lm.py b/dspy/clients/lm.py
index 5175fe2be..10a7615dc 100644
--- a/dspy/clients/lm.py
+++ b/dspy/clients/lm.py
@@ -1,8 +1,8 @@
-from concurrent.futures import ThreadPoolExecutor
 import functools
 import os
+import uuid
+from datetime import datetime
 from pathlib import Path
-import re
 from typing import Any, Dict, List, Optional, Type, Union
 import ujson
 
@@ -20,33 +20,17 @@
     anyscale_model_kill,
 )
 
+import litellm
+import ujson
+from litellm.caching import Cache
 
-DISK_CACHE_DIR = os.environ.get('DSPY_CACHEDIR') or os.path.join(Path.home(), '.dspy_cache')
-
-    
-try:
-    import warnings
-    with warnings.catch_warnings():
-        warnings.filterwarnings("ignore", category=UserWarning)
-        if "LITELLM_LOCAL_MODEL_COST_MAP" not in os.environ:
-             os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
-        import litellm  
-        litellm.telemetry = False
-
-    from litellm.caching import Cache
-    litellm.cache = Cache(disk_cache_dir=DISK_CACHE_DIR, type="disk")
-
-except ImportError:
-    class LitellmPlaceholder:
-        def __getattr__(self, _): raise ImportError("The LiteLLM package is not installed. Run `pip install litellm`.")
-
-    litellm = LitellmPlaceholder()
+disk_cache_dir = os.environ.get("DSPY_CACHEDIR") or os.path.join(Path.home(), ".dspy_cache")
+litellm.cache = Cache(disk_cache_dir=disk_cache_dir, type="disk")
+litellm.telemetry = False
 
+if "LITELLM_LOCAL_MODEL_COST_MAP" not in os.environ:
+    os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
 
-#-------------------------------------------------------------------------------
-#    LiteLLM Client
-#-------------------------------------------------------------------------------
-        
 class LM:
     def __init__(self, 
             model,
@@ -66,8 +50,9 @@ def __init__(self,
 
         # TODO: This is error prone!
         if "o1-" in model:
-            assert max_tokens >= 5000 and temperature == 1.0, \
-                "OpenAI's o1-* models require passing temperature=1.0 and max_tokens >= 5000 to `dspy.LM(...)`"
+            assert (
+                max_tokens >= 5000 and temperature == 1.0
+            ), "OpenAI's o1-* models require passing temperature=1.0 and max_tokens >= 5000 to `dspy.LM(...)`"
 
     def __call__(self, prompt=None, messages=None, **kwargs):
         # Build the request.
@@ -76,8 +61,10 @@ def __call__(self, prompt=None, messages=None, **kwargs):
         kwargs = {**self.kwargs, **kwargs}
 
         # Make the request and handle LRU & disk caching.
-        if self.model_type == "chat": completion = cached_litellm_completion if cache else litellm_completion
-        else: completion = cached_litellm_text_completion if cache else litellm_text_completion
+        if self.model_type == "chat":
+            completion = cached_litellm_completion if cache else litellm_completion
+        else:
+            completion = cached_litellm_text_completion if cache else litellm_text_completion
 
         response = completion(ujson.dumps(dict(model=self.model, messages=messages, **kwargs)))
         outputs = [c.message.content if hasattr(c, "message") else c["text"] for c in response["choices"]]
@@ -87,10 +74,17 @@ def __call__(self, prompt=None, messages=None, **kwargs):
         entry = dict(prompt=prompt, messages=messages, kwargs=kwargs, response=response)
         entry = dict(**entry, outputs=outputs, usage=dict(response["usage"]))
         entry = dict(**entry, cost=response.get("_hidden_params", {}).get("response_cost"))
+        entry = dict(
+            **entry,
+            timestamp=datetime.now().isoformat(),
+            uuid=str(uuid.uuid4()),
+            model=self.model,
+            model_type=self.model_type,
+        )
         self.history.append(entry)
-
+        
         return outputs
-    
+
     def inspect_history(self, n: int = 1):
         _inspect_history(self, n)
 
@@ -110,8 +104,12 @@ def kill(self):
             anyscale_model_kill(self.model, self.launch_kwargs)
         logger.debug(f"`LM.kill()` is called for the auto-launched model {self.model} -- no action is taken.")
 
-    def finetune(self,
-            message_completion_pairs: List[Dict[str, str]],
+    async def finetune(self,
+            # message_completion_pairs: List[Dict[str, str]],
+            method: TrainingMethod,
+            train_path: str,
+            eval_path: Optional[str],
+            provider: str = "openai",
             train_kwargs: Optional[Dict[str, Any]]=None,
             launch_kwargs: Optional[Dict[str, Any]]=None,
             cache_finetune: bool = True,
@@ -122,45 +120,43 @@ def finetune(self,
         err = "Fine-tuning is an experimental feature and requires `dspy.settings.experimental = True`."
         assert settings.experimental, err
 
-        # Get the fine-tuning provider, if it is supported
-        try:
-            provider = _get_supported_finetune_provider(self.model)
-        except ValueError as err:
-            raise err
-
         # Initialize the finetune job
         FinetuneJobClass = get_provider_finetune_job_class(provider=provider)
         finetune_job = FinetuneJobClass(
             model=self.model,
-            message_completion_pairs=message_completion_pairs,
+            train_path=train_path,
+            eval_path=eval_path,
             train_kwargs=train_kwargs,
         )
 
-        # Start asyncronous training
-        executor = ThreadPoolExecutor(max_workers=1)
-        executor.submit(
-            execute_finetune_job,
-            finetune_job,
-            launch_kwargs=launch_kwargs,
-            cache_finetune=cache_finetune
-        )
-        executor.shutdown(wait=False)
+        job = finetune_job.run_finetune()
 
-        return finetune_job
+        return job
+    
+    def copy(self, **kwargs):
+        """Returns a copy of the language model with the same parameters."""
+        kwargs = {**self.kwargs, **kwargs}
+        # model = kwargs.pop("model") or self.model
+        init_kwargs = dict(model=self.model, model_type=self.model_type, cache=self.cache, temperature=self.kwargs["temperature"], max_tokens=self.kwargs["max_tokens"])
+        init_kwargs = {**init_kwargs, **kwargs}
+        return self.__class__(**init_kwargs)
 
 
 @functools.lru_cache(maxsize=None)
 def cached_litellm_completion(request):
     return litellm_completion(request, cache={"no-cache": False, "no-store": False})
 
+
 def litellm_completion(request, cache={"no-cache": True, "no-store": True}):
     kwargs = ujson.loads(request)
     return litellm.completion(cache=cache, **kwargs)
 
+
 @functools.lru_cache(maxsize=None)
 def cached_litellm_text_completion(request):
     return litellm_text_completion(request, cache={"no-cache": False, "no-store": False})
 
+
 def litellm_text_completion(request, cache={"no-cache": True, "no-store": True}):
     kwargs = ujson.loads(request)
 
@@ -174,29 +170,40 @@ def litellm_text_completion(request, cache={"no-cache": True, "no-store": True})
     api_base = kwargs.pop("api_base", None) or os.getenv(f"{provider}_API_BASE")
 
     # Build the prompt from the messages.
-    prompt = '\n\n'.join([x['content'] for x in kwargs.pop("messages")] + ['BEGIN RESPONSE:'])
-
-    return litellm.text_completion(cache=cache, model=f'text-completion-openai/{model}', api_key=api_key,
-                                   api_base=api_base, prompt=prompt, **kwargs)
+    prompt = "\n\n".join([x["content"] for x in kwargs.pop("messages")] + ["BEGIN RESPONSE:"])
+
+    return litellm.text_completion(
+        cache=cache,
+        model=f"text-completion-openai/{model}",
+        api_key=api_key,
+        api_base=api_base,
+        prompt=prompt,
+        **kwargs,
+    )
 
 
 def _green(text: str, end: str = "\n"):
     return "\x1b[32m" + str(text).lstrip() + "\x1b[0m" + end
 
+
 def _red(text: str, end: str = "\n"):
     return "\x1b[31m" + str(text) + "\x1b[0m" + end
 
+
 def _inspect_history(lm, n: int = 1):
     """Prints the last n prompts and their completions."""
 
     for item in lm.history[-n:]:
-        messages = item["messages"] or [{"role": "user", "content": item['prompt']}]
+        messages = item["messages"] or [{"role": "user", "content": item["prompt"]}]
         outputs = item["outputs"]
+        timestamp = item.get("timestamp", "Unknown time")
 
         print("\n\n\n")
+        print("\x1b[34m" + f"[{timestamp}]" + "\x1b[0m" + "\n")
+
         for msg in messages:
             print(_red(f"{msg['role'].capitalize()} message:"))
-            print(msg['content'].strip())
+            print(msg["content"].strip())
             print("\n")
 
         print(_red("Response:"))
@@ -205,7 +212,7 @@ def _inspect_history(lm, n: int = 1):
         if len(outputs) > 1:
             choices_text = f" \t (and {len(outputs)-1} other completions)"
             print(_red(choices_text, end=""))
-        
+
     print("\n\n\n")
 
 
@@ -216,13 +223,12 @@ def _inspect_history(lm, n: int = 1):
 
 from dspy.clients.openai import (
     FinetuneJobOpenAI,
-    is_openai_model,
-    finetune_openai
+    is_openai_model
 )
 from dspy.clients.anyscale import (
     FinetuneJobAnyScale,
-    is_anyscale_model,
-    finetune_anyscale
+    # is_anyscale_model,
+    # finetune_anyscale
 )
 
 
@@ -267,44 +273,44 @@ def get_provider_finetune_job_class(provider: str) -> Type[FinetuneJob]:
     return _CLS
 
 
-def get_provider_finetune_function(provider: str) -> callable:
-    """Return the finetune function for the given model."""
-    # Mapping from provider to finetune function
-    _PROVIDER_TO_FINETUNE_FUNCTION = {
-        _PROVIDER_ANYSCALE: finetune_anyscale,
-        _PROVIDER_OPENAI: finetune_openai,
-    }
-
-    # Get the finetuning provider
-    finetune_function = _PROVIDER_TO_FINETUNE_FUNCTION[provider]
-
-    return finetune_function
-
-
-def execute_finetune_job(
-    job: FinetuneJob[Type[LM]],
-    launch_kwargs: Optional[Dict[str, Any]]=None,
-    cache_finetune: bool=True
-):
-    """Execute the finetune job in a blocking manner."""
-    # Input validation
-    launch_kwargs = launch_kwargs or {}
-
-    # Execute finetune job
-    job_kwargs = job.get_kwargs()
-    if cache_finetune:
-        try:
-            model = cached_finetune(job=job, **job_kwargs)
-        except ValueError as err:
-            raise err
-    else:
-        model = finetune(job=job, **job_kwargs)
-
-    # Launch the LM
-    lm = LM(model=model, **launch_kwargs)
-
-    # Set the result of the finetuning job to the fine-tuned LM
-    job.set_result(lm)
+# def get_provider_finetune_function(provider: str) -> callable:
+#     """Return the finetune function for the given model."""
+#     # Mapping from provider to finetune function
+#     _PROVIDER_TO_FINETUNE_FUNCTION = {
+#         _PROVIDER_ANYSCALE: finetune_anyscale,
+#         _PROVIDER_OPENAI: finetune_openai,
+#     }
+
+#     # Get the finetuning provider
+#     finetune_function = _PROVIDER_TO_FINETUNE_FUNCTION[provider]
+
+#     return finetune_function
+
+
+# def execute_finetune_job(
+#     job: FinetuneJob[Type[LM]],
+#     launch_kwargs: Optional[Dict[str, Any]]=None,
+#     cache_finetune: bool=True
+# ):
+#     """Execute the finetune job in a blocking manner."""
+#     # Input validation
+#     launch_kwargs = launch_kwargs or {}
+
+#     # Execute finetune job
+#     job_kwargs = job.get_kwargs()
+#     if cache_finetune:
+#         try:
+#             model = cached_finetune(job=job, **job_kwargs)
+#         except ValueError as err:
+#             raise err
+#     else:
+#         model = finetune(job=job, **job_kwargs)
+
+#     # Launch the LM
+#     lm = LM(model=model, **launch_kwargs)
+
+#     # Set the result of the finetuning job to the fine-tuned LM
+#     job.set_result(lm)
 
 
 # TODO: Perhaps we shouldn't directly cache the data
diff --git a/dspy/clients/openai.py b/dspy/clients/openai.py
index d17d0efc7..3eee3a934 100644
--- a/dspy/clients/openai.py
+++ b/dspy/clients/openai.py
@@ -7,7 +7,6 @@
     FinetuneJob,
     TrainingMethod,
     validate_training_data,
-    save_data,
 )
 
 
diff --git a/dspy/clients/self_hosted.py b/dspy/clients/self_hosted.py
index 90c850e7e..e3ca93077 100644
--- a/dspy/clients/self_hosted.py
+++ b/dspy/clients/self_hosted.py
@@ -1,7 +1,5 @@
 from typing import Dict, Any
 
-from dspy import LM
-from dspy import logger
 from dspy.clients.openai import is_openai_model
 
 
diff --git a/dspy/predict/chain_of_thought.py b/dspy/predict/chain_of_thought.py
index 74656697c..b6b9cb575 100644
--- a/dspy/predict/chain_of_thought.py
+++ b/dspy/predict/chain_of_thought.py
@@ -29,7 +29,7 @@ def __init__(self, signature, rationale_type=None, activated=True, **config):
         rationale_type = rationale_type or dspy.OutputField(prefix=prefix, desc=desc)
 
         # Add "rationale" field to the output signature.
-        if isinstance(dspy.settings.lm, dspy.LM):
+        if isinstance(dspy.settings.lm, dspy.LM) or dspy.settings.experimental:
             extended_signature = signature.prepend("reasoning", rationale_type, type_=str)
         else:
             extended_signature = signature.prepend("rationale", rationale_type, type_=str)
diff --git a/dspy/predict/predict.py b/dspy/predict/predict.py
index 88a74861a..0e3610b86 100644
--- a/dspy/predict/predict.py
+++ b/dspy/predict/predict.py
@@ -1,17 +1,17 @@
 from __future__ import annotations  # Used for self type hints
 import random
 from typing import Optional
+import logging
+from functools import lru_cache
 
 from pydantic import BaseModel
 
 import dsp
 from dspy.predict.parameter import Parameter
-from dspy.primitives.program import Module
 from dspy.primitives.prediction import Prediction
+from dspy.primitives.program import Module
 from dspy.signatures.signature import ensure_signature, signature_to_template
 
-import logging
-from functools import lru_cache
 
 @lru_cache(maxsize=None)
 def warn_once(msg: str):
@@ -61,7 +61,13 @@ def reset(self):
         self.train = []
         self.demos = []
 
-    def dump_state(self, save_verbose=False):
+    def dump_state(self, save_verbose=None):
+        if save_verbose:
+            logging.warning(
+                "`save_verbose` is deprecated and will be removed in DSPy 2.6.0 release. Currently `save_verbose` "
+                "does nothing."
+            )
+
         state_keys = ["lm", "traces", "train"]
         state = {k: getattr(self, k) for k in state_keys}
 
@@ -75,33 +81,47 @@ def dump_state(self, save_verbose=False):
 
             state["demos"].append(demo)
 
-        # If `save_verbose` save all field metadata as well.
-        if save_verbose:
-            fields = []
-            for field_key in self.signature.fields.keys():
-                field_metadata = self.signature.fields[field_key]
-                fields.append({
-                    "name":  field_key,
-                    "field_type": field_metadata.json_schema_extra["__dspy_field_type"],
-                    "description": field_metadata.json_schema_extra["desc"],
-                    "prefix": field_metadata.json_schema_extra["prefix"]
-                })
-            state["fields"] = fields
-        
-        # Cache the signature instructions and the last field's name.
-        *_, last_key = self.signature.fields.keys()
-        state["signature_instructions"] = self.signature.instructions
-        state["signature_prefix"] = self.signature.fields[last_key].json_schema_extra["prefix"]
-
-        # Some special stuff for CoT.
+        state["signature"] = self.signature.dump_state()
+        # `extended_signature` is a special field for `Predict`s like CoT.
         if hasattr(self, "extended_signature"):
-            # Cache the signature instructions and the last field's name.
-            state["extended_signature_instructions"] = self.extended_signature.instructions
-            state["extended_signature_prefix"] = self.extended_signature.fields[last_key].json_schema_extra['prefix']
+            state["extended_signature"] = self.extended_signature.dump_state()
 
         return state
 
-    def load_state(self, state):
+    def load_state(self, state, use_legacy_loading=False):
+        """Load the saved state of a `Predict` object.
+
+        Args:
+            state (dict): The saved state of a `Predict` object.
+            use_legacy_loading (bool): Whether to use the legacy loading method. Only use it when you are loading a
+                saved state from a version of DSPy prior to v2.5.3.
+        """
+        if use_legacy_loading:
+            self._load_state_legacy(state)
+            return
+        if "signature" not in state:
+            # Check if the state is from a version of DSPy prior to v2.5.3.
+            raise ValueError(
+                "The saved state is from a version of DSPy prior to v2.5.3. Please use `use_legacy_loading=True` to "
+                "load the state."
+            )
+
+        excluded_keys = ["signature", "extended_signature"]
+        for name, value in state.items():
+            # `excluded_keys` are fields that go through special handling.
+            if name not in excluded_keys:
+                setattr(self, name, value)
+
+        self.signature = self.signature.load_state(state["signature"])
+
+        if "extended_signature" in state:
+            self.extended_signature.load_state(state["extended_signature"])
+
+    def _load_state_legacy(self, state):
+        """Legacy state loading for backwards compatibility.
+
+        This method is used to load the saved state of a `Predict` object from a version of DSPy prior to v2.5.3.
+        """
         for name, value in state.items():
             setattr(self, name, value)
 
@@ -114,7 +134,7 @@ def load_state(self, state):
             prefix = state["signature_prefix"]
             *_, last_key = self.signature.fields.keys()
             self.signature = self.signature.with_updated_fields(last_key, prefix=prefix)
-        
+
         # Some special stuff for CoT.
         if "extended_signature_instructions" in state:
             instructions = state["extended_signature_instructions"]
@@ -125,6 +145,7 @@ def load_state(self, state):
             *_, last_key = self.extended_signature.fields.keys()
             self.extended_signature = self.extended_signature.with_updated_fields(last_key, prefix=prefix)
 
+
     def __call__(self, **kwargs):
         return self.forward(**kwargs)
 
@@ -267,4 +288,3 @@ def v2_5_generate(lm, lm_kwargs, signature, demos, inputs, _parse_values=True):
 # # TODO: FIXME: Hmm, I guess expected behavior is that contexts can
 # affect execution. Well, we need to determine whether context dominates, __init__ demoninates, or forward dominates.
 # Generally, unless overwritten, we'd see n=None, temperature=None.
-# That will eventually mean we have to learn them.
diff --git a/dspy/primitives/assertions.py b/dspy/primitives/assertions.py
index 3f2b6166c..4ddea6760 100644
--- a/dspy/primitives/assertions.py
+++ b/dspy/primitives/assertions.py
@@ -248,7 +248,7 @@ def wrapper(*args, **kwargs):
                                 dspy.settings.backtrack_to = dsp.settings.trace[-1][0]
 
                             if dspy.settings.backtrack_to is None:
-                                dspy.logger.error("Specified module not found in trace")
+                                dspy.logger.error("Module not found in trace. If passing a DSPy Signature, please specify the intended module for the assertion (e.g., use `target_module = self.my_module(my_signature)` instead of `target_module =  my_signature`).")
 
                             # save unique feedback message for predictor
                             if error_msg not in dspy.settings.predictor_feedbacks.setdefault(
diff --git a/dspy/primitives/module.py b/dspy/primitives/module.py
index 4fe9ea692..87ce8b1aa 100644
--- a/dspy/primitives/module.py
+++ b/dspy/primitives/module.py
@@ -151,17 +151,17 @@ def reset_copy(self):
     def dump_state(self, save_verbose):
         return {name: param.dump_state(save_verbose) for name, param in self.named_parameters()}
 
-    def load_state(self, state):
+    def load_state(self, state, use_legacy_loading=False):
         for name, param in self.named_parameters():
-            param.load_state(state[name])
+            param.load_state(state[name], use_legacy_loading=use_legacy_loading)
 
     def save(self, path, save_field_meta=False):
         with open(path, "w") as f:
             f.write(ujson.dumps(self.dump_state(save_field_meta), indent=2))
 
-    def load(self, path):
+    def load(self, path, use_legacy_loading=False):
         with open(path) as f:
-            self.load_state(ujson.loads(f.read()))
+            self.load_state(ujson.loads(f.read()), use_legacy_loading=use_legacy_loading)
 
 
 def postprocess_parameter_name(name, value):
diff --git a/dspy/propose/grounded_proposer.py b/dspy/propose/grounded_proposer.py
index 993cacacc..5b70e7b21 100644
--- a/dspy/propose/grounded_proposer.py
+++ b/dspy/propose/grounded_proposer.py
@@ -187,6 +187,7 @@ def forward(
         # Summarize the program
         program_description = "Not available"
         module_code = "Not provided"
+        module_description = "Not provided"
         if self.program_aware:
             try:
                 program_description = strip_prefix(
@@ -209,18 +210,18 @@ def forward(
                         outputs.append(field_name)
 
                 module_code = f"{program.predictors()[pred_i].__class__.__name__}({', '.join(inputs)}) -> {', '.join(outputs)}"
+
+                module_description = self.describe_module(
+                    program_code=self.program_code_string,
+                    program_description=program_description,
+                    program_example=task_demos,
+                    module=module_code,
+                    max_depth=10,
+                ).module_description
             except:
                 if self.verbose: print("Error getting program description. Running without program aware proposer.")
                 self.program_aware = False
 
-        module_description = self.describe_module(
-            program_code=self.program_code_string,
-            program_description=program_description,
-            program_example=task_demos,
-            module=module_code,
-            max_depth=10,
-        ).module_description
-
         # Generate an instruction for our chosen module
         if self.verbose: print(f"task_demos {task_demos}")
         instruct = self.generate_module_instruction(
@@ -258,6 +259,7 @@ def __init__(
         set_tip_randomly=True,
         set_history_randomly=True,
         verbose=False,
+        rng=None
     ):
         super().__init__()
         self.program_aware = program_aware
@@ -268,22 +270,30 @@ def __init__(
         self.set_tip_randomly=set_tip_randomly
         self.set_history_randomly=set_history_randomly
         self.verbose = verbose
+        self.rng = rng or random
 
         self.prompt_model = get_prompt_model(prompt_model)
+
+        self.program_code_string = None
         if self.program_aware:
             try:
                 self.program_code_string = get_dspy_source_code(program)
                 if self.verbose: print("SOURCE CODE:",self.program_code_string)
             except Exception as e:
                 print(f"Error getting source code: {e}.\n\nRunning without program aware proposer.")
-                self.program_code_string = None
                 self.program_aware = False
-        else:
-            self.program_code_string = None
-        self.data_summary = create_dataset_summary(
-            trainset=trainset, view_data_batch_size=view_data_batch_size, prompt_model=prompt_model,
-        )
-        if self.verbose: print(f"DATA SUMMARY: {self.data_summary}")
+
+        self.data_summary  = None
+        if self.use_dataset_summary:
+            try:
+                self.data_summary = create_dataset_summary(
+                    trainset=trainset, view_data_batch_size=view_data_batch_size, prompt_model=prompt_model,
+                )
+                if self.verbose: print(f"DATA SUMMARY: {self.data_summary}")
+            except Exception as e:
+                print(f"Error getting data summary: {e}.\n\nRunning without data aware proposer.")
+                self.use_dataset_summary = False
+                print("")
 
     def propose_instructions_for_program(
         self,
@@ -301,11 +311,11 @@ def propose_instructions_for_program(
 
         if self.set_history_randomly:
             # Randomly select whether or not we're using instruction history
-            use_history = random.random() < 0.5
+            use_history = self.rng.random() < 0.5
             self.use_instruct_history = use_history
             if self.verbose: print(f"Use history T/F: {self.use_instruct_history}")
-        
-        num_demos = min(len(demo_candidates[0]) if demo_candidates else N, 1)
+
+        num_demos = max(len(demo_candidates[0]) if demo_candidates else N, 1)
 
         if not demo_candidates:
             if self.verbose: print("No demo candidates provided. Running without task demos.")
@@ -319,7 +329,7 @@ def propose_instructions_for_program(
                 if self.set_tip_randomly:
                     if self.verbose: print("Using a randomly generated configuration for our grounded proposer.")
                     # Randomly select the tip
-                    selected_tip_key = random.choice(list(TIPS.keys()))
+                    selected_tip_key = self.rng.choice(list(TIPS.keys()))
                     selected_tip = TIPS[selected_tip_key]
                     self.use_tip = bool(
                         selected_tip,
@@ -339,6 +349,7 @@ def propose_instructions_for_program(
                         tip=selected_tip,
                     ),
                 )
+
         return proposed_instructions
 
     def propose_instruction_for_predictor(
diff --git a/dspy/retrieve/pgvector_rm.py b/dspy/retrieve/pgvector_rm.py
index e6bf542ee..b05767c4a 100644
--- a/dspy/retrieve/pgvector_rm.py
+++ b/dspy/retrieve/pgvector_rm.py
@@ -9,7 +9,7 @@
     from psycopg2 import sql
 except ImportError:
     raise ImportError(
-        "The 'pgvector' extra is required to use PgVectorRM. Install it with `pip install dspy-ai[pgvector]`",
+        "The 'pgvector' extra is required to use PgVectorRM. Install it with `pip install dspy-ai[pgvector]`. Also, try `pip install pgvector psycopg2`.",
     )
 try:
     import openai
diff --git a/dspy/signatures/signature.py b/dspy/signatures/signature.py
index 16c2dc66d..b3702fbeb 100644
--- a/dspy/signatures/signature.py
+++ b/dspy/signatures/signature.py
@@ -96,6 +96,10 @@ def signature(cls) -> str:
     def instructions(cls) -> str:
         return inspect.cleandoc(getattr(cls, "__doc__", ""))
 
+    @instructions.setter
+    def instructions(cls, instructions: str) -> None:
+        setattr(cls, "__doc__", instructions)
+
     def with_instructions(cls, instructions: str) -> Type["Signature"]:
         return Signature(cls.fields, instructions)
 
@@ -159,6 +163,28 @@ def insert(cls, index: int, name: str, field, type_: Type = None) -> Type["Signa
         new_fields = dict(input_fields + output_fields)
         return Signature(new_fields, cls.instructions)
 
+    def dump_state(cls):
+        state = {"instructions": cls.instructions, "fields": []}
+        for field in cls.fields:
+            state["fields"].append(
+                {
+                    "prefix": cls.fields[field].json_schema_extra["prefix"],
+                    "description": cls.fields[field].json_schema_extra["desc"],
+                }
+            )
+
+        return state
+
+    def load_state(cls, state):
+        signature_copy = Signature(deepcopy(cls.fields), cls.instructions)
+
+        signature_copy.instructions = state["instructions"]
+        for field, saved_field in zip(signature_copy.fields.values(), state["fields"]):
+            field.json_schema_extra["prefix"] = saved_field["prefix"]
+            field.json_schema_extra["desc"] = saved_field["description"]
+
+        return signature_copy
+
     def equals(cls, other) -> bool:
         """Compare the JSON schema of two Pydantic models."""
         if not isinstance(other, type) or not issubclass(other, BaseModel):
@@ -239,7 +265,7 @@ def replace(
         class OldSignature(cls, Signature):
             pass
 
-        replace_fields = ["__doc__", "model_fields", "model_extra", "model_config"]
+        replace_fields = ["__doc__", "model_fields", "model_extra",]
         for field in replace_fields:
             setattr(cls, field, getattr(new_signature, field))
         cls.model_rebuild(force=True)
@@ -423,4 +449,4 @@ def infer_prefix(attribute_name: str) -> str:
         else:
             title_cased_words.append(word.capitalize())
 
-    return " ".join(title_cased_words)
\ No newline at end of file
+    return " ".join(title_cased_words)
diff --git a/dspy/teleprompt/finetune_teleprompter.py b/dspy/teleprompt/finetune_teleprompter.py
index 421053073..4fcdf3237 100644
--- a/dspy/teleprompt/finetune_teleprompter.py
+++ b/dspy/teleprompt/finetune_teleprompter.py
@@ -6,8 +6,8 @@
 from dspy.primitives.example import Example
 from dspy.primitives.program import Program
 from dspy.primitives.prediction import Prediction
-from dspy.signatures.signature import signature_to_template
 
+import concurrent.futures
 
 #-------------------------------------------------------------------------------
 #    Templates for the user-facing strings used by this module
@@ -78,102 +78,8 @@ def prepare_teacher(
 
     return teacher
 
-
-def build_prompt_completion_data_from_trace(
-        trace: List[Dict],
-        exclude_demos: bool=False,
-        try_to_record_lm_kwargs: bool = False,
-        program: Program = None,
-    ) -> Dict[str, List[Dict[str, Any]]]:
-    """Build prompt completion data from a given trace.
-  
-    Args:
-        trace: The trace from which the prompt-completion data will be built.
-        exclude_demos: Exclude the demos from the prompts even if they are
-            present in the trace. Defaults to `False`.
-        try_to_record_lm_kwargs: Whether to record the LM kwargs in the data.
-            Defaults to `False`. If set, the `lm_kwargs` field of the LM used to
-            generate the prompt-completion pair is included in the data. To
-            find the LM, we first check if the predictor that generated the
-            prompt-completion pair has an LM field set (`predictor.lm`). If it
-            does, we record it's kwargs. If it doesn't, we get the kwargs from
-            `dspy.settings.lm`. If `dspy.settings.lm` is not set either, this
-            function will not record the LM kwargs.
-        program: Optional argument used to infer the name of the predictor that
-            generated the prompt-completion pair. If provided, the returned data
-            will include the `predictor_name` field. If not provided, the
-            `predictor_name` field is not included in the data, but the caller
-            of this function can recover this information by using the
-            `predictor_ind` field that' included in the data by default, by
-            building the following dictionary:
-
-            {ind: n for ind, (n, _) in enumerate(program.named_predictors())}
-
-            where `ind` is the index of the predictor in the list returned by
-            the `named_predictors()` method of the program. Defaults to `None`.
-
-    Returns:
-        Data as a list of dictionaries with the keys `prompt`, `completion` and
-        optionally with the keys `predictor_name` and `lm_kwargs`. For a given
-        prompt-completion pair:
-        - The `prompt` field corresponds to the prompt.
-        - The `completion` field corresponds to the completion.
-        - The `predictor_ind` field corresponds to the index of the predictor
-          in the predictor list 
-        - The `predictor_name` field corresponds to the index of the predictor
-          in the list returned by the the named_predictor() method of the
-          program used to generate the trace. This field is included only if
-        the `pred_ind_to_name` argument is provided.
-        - The `lm_kwargs` field corresponds to the LM kwargs that generated the
-          prompt-completion pair. Included only if the `record_lm_kwargs` is
-          set and there is an active LM.
-    """
-    # If the program is provided, build the predictor index to name mapping
-    if program:
-        pred_ind_to_name = {
-            ind: name for ind, (name, _) in enumerate(program.named_predictors())
-        }
-
-    # Build the prompt-completion data
-    data = []
-    for pred_ind, (pred, inputs, outputs) in enumerate(trace):
-        # Get the demos from the predictor if exclude_demos is False
-        demos = [] if exclude_demos else pred.demos
-
-        # Build prompt and completion strings
-        template = signature_to_template(pred.signature)
-        prompt = template(Example(demos=demos, **inputs))
-        completion = template.query(Example(**outputs))
-        
-        # TODO: This part of the code could be improved.
-        # The method we use to build the completion (template.query) is meant to
-        # be used for creating, well, queries, and hence contains field prefixes
-        # (e.g. "Reasoning: Let's think step by step in order to"), which are
-        # also contained in the last piece of the prompt (separated with a new
-        # line) We remove this piece from the completion. This is a hacky
-        # solution since it assumes a particular template format.
-        prompt_last = prompt.split("\n")[-1]
-        completion = completion[len(prompt_last):]
-
-        # Create prompt-completion dictionary and add it to the data; optionally
-        # add the predictor_name key as well as the lm_kwargs.
-        data_dict = dict(prompt=prompt, completion=completion)
-        
-        # Record the predictor index and optionally, name
-        data_dict['predictor_ind'] = pred_ind
-        if program:
-            data_dict['predictor_name'] = pred_ind_to_name[pred_ind]
-
-        # Optionally, record the LM kwargs
-        lm = pred.lm or dspy.settings.lm
-        if try_to_record_lm_kwargs and lm:
-            data_dict['lm_kwargs'] = lm.kwargs
-        data.append(data_dict)
-
-    return data
-
-
-def convert_to_module_level_prompt_completion_data(
+# TODO: fix docstring
+def convert_to_module_level_message_data(
         data: List[Dict],
         keep_data_keys: bool = False,
         exclude_demos: bool = False,
@@ -214,7 +120,7 @@ def convert_to_module_level_prompt_completion_data(
     prompt_completion_data = []
     for data_dict in data:
         trace = data_dict["trace"]
-        trace_prompt_comletion_data = build_prompt_completion_data_from_trace(
+        trace_prompt_comletion_data = build_messages_from_trace(
             trace=trace, exclude_demos=exclude_demos,
             try_to_record_lm_kwargs=try_to_record_lm_kwargs, program=program
         )
@@ -224,14 +130,51 @@ def convert_to_module_level_prompt_completion_data(
             prompt_completion_data.append(prompt_completion_dict)
     return prompt_completion_data
 
+# TODO: fix docstring
+def build_messages_from_trace(
+        trace: List[Dict],
+        exclude_demos: bool=False,
+        try_to_record_lm_kwargs: bool = False,
+        program: Program = None,
+    ) -> Dict[str, List[Dict[str, Any]]]:
+    """Build messages from a given trace.
+    """
+    messages = []
+    # If the program is provided, build the predictor index to name mapping
+    if program:
+        pred_ind_to_name = {
+            ind: name for ind, (name, _) in enumerate(program.named_predictors())
+        }
+
+    # Build the prompt-completion data
+
+    adapter = dspy.settings.adapter or dspy.ChatAdapter()
+    data = []
+
+    # TODO: Make sure that this works for multi-stage pipelines
+    for pred_ind, (pred, inputs, outputs) in enumerate(trace):
+        # Get the demos from the predictor if exclude_demos is False
+        demos = [] if exclude_demos else pred.demos
+
+        messages = adapter.format(pred.signature, demos, inputs)
+
+        formatted_completion = adapter.format_completion(pred.signature, outputs)
+        messages.append({"role": "assistant", "content": formatted_completion})
+        data.append(messages)
+
+    return data
+
+def dummy_metric(example, pred, trace=None, frac=1.0):
+    return 1
 
 def bootstrap_data(
         program: Program,
         dataset: List[Example],
         metric: Optional[Callable[
             [Example, Prediction, Optional[List]], Union[bool, int, float]
-        ]] = None,
+        ]] = dummy_metric,
         num_threads = 1,
+        max_errors: int = 0
     ) -> List[Dict[str, Any]]:
     """Bootstrap prediction and trace data for the program using the dataset.
     
@@ -266,28 +209,60 @@ def bootstrap_data(
     info = _INFO_BOOTSTRAP_DATA.format(len(dataset), cname, num_threads)
     logger.info(info)
     evaluator = Evaluate(
-        devset=dataset, num_threads=num_threads, display_progress=True
+        devset=dataset, num_threads=num_threads, display_progress=True, max_errors=max_errors, provide_traceback=True
     )
-    evaluator(program, metric=metric)
-
-    # Re-iterate over the dataset to build the cached prompt-completion data
-    for example_ind, example in enumerate(dataset):
+    x = evaluator(program, metric=metric)
+    # print(x)
+    # data = process_dataset_threaded(dataset, program, metric, num_threads, max_errors)
+    data = []
+    for example in dataset:
+        data_dict = process_example(example, 0, program, metric)
+        if data_dict is not None:
+            data.append(data_dict)
+    
+    return data
 
-        # Run the program on the example
-        with dspy.context(trace=[]):
-            prediction = program(**example.inputs())
-            trace = dspy.settings.trace
-            score = metric(example, prediction, trace) if metric else None
-
-        # Build the data dictionary and extend the data list
-        data_dict = dict(example=example, prediction=prediction, trace=trace)
-        data_dict['example_ind'] = example_ind
-        if metric:
-            data_dict['score'] = score
-        data.append(data_dict)
+def process_dataset_threaded(dataset: List[Any], program: Callable, metric: Optional[Callable] = None, max_workers: int = None, max_errors: int = 0) -> List[Dict[str, Any]]:
+    data = []
+    num_threads = max_workers if max_workers else 10
+    with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
+        future_to_example = {executor.submit(process_example, example, i, program, metric): i 
+                             for i, example in enumerate(dataset)}
+        
+        for future in concurrent.futures.as_completed(future_to_example):
+            data_dict = future.result()
+            if data_dict is not None:
+                data.append(data_dict)
+    
+    # Sort the results based on example_ind to maintain original order
+    data.sort(key=lambda x: x['example_ind'])
     
     return data
 
+def process_example(example: Any, example_ind: int, program: Callable, metric: Optional[Callable] = None) -> Dict[str, Any]:
+    # print("Processing example:", example_ind)
+    with dspy.context(trace=[]):
+        # print("Running program...", example_ind)
+        try:
+            prediction = program(**example.inputs())
+        except Exception as e:
+            print(f"Error processing example {example_ind}: {e}")
+            return None
+        # print("Getting trace...", example_ind)
+        trace = dspy.settings.trace
+        # print("Getting score...", example_ind)
+        score = metric(example, prediction, trace) if metric else None
+
+    data_dict = {
+        'example': example,
+        'prediction': prediction,
+        'trace': trace,
+        'example_ind': example_ind
+    }
+    if metric:
+        data_dict['score'] = score
+    
+    return data_dict
 
 # TODO: If we can ensure to pass the "round" information every time a call is
 # issued to an LM, we can make repetitive un-cached calls to the same LM without
@@ -302,6 +277,7 @@ def bootstrap_data_for_round(
         sampling_round: int = 0,
         sampling_temperature: Optional[float] = 0.9,
         sampling_temperature_delta: float = 0.001,
+        max_errors: int = 0
     ) -> Union[List[Dict], AssertionError]:
     """ Bootstrap data for the given sampling round.
 
@@ -387,7 +363,8 @@ def copy_model_with_updated_temp(lm):
 
     # Ensure that the LM consistency is satisfied, which ensures that either (1)
     # the global LM is set or (2) all the predictors have an LM set.
-    program._assert_lm_consistency()
+    # TODO(isaac): Uncomment this line after the LM consistency property is
+    # program._assert_lm_consistency()
 
     # Deepcopy the program and copy the dataset to avoid modifying the original
     program = program.deepcopy()
@@ -403,8 +380,9 @@ def copy_model_with_updated_temp(lm):
 
     # Collect the data for the given round
     with dspy.context(lm=context_lm):
+        # print(context_lm.kwargs)
         data = bootstrap_data(
-            program, dataset, metric=metric, num_threads=num_threads
+            program, dataset, metric=metric, num_threads=num_threads, max_errors=max_errors
         )
     
     # Add the round information to the data
diff --git a/dspy/teleprompt/mipro_optimizer_v2.py b/dspy/teleprompt/mipro_optimizer_v2.py
index ab758f7e6..6603a259b 100644
--- a/dspy/teleprompt/mipro_optimizer_v2.py
+++ b/dspy/teleprompt/mipro_optimizer_v2.py
@@ -2,6 +2,7 @@
 import sys
 import textwrap
 from collections import defaultdict
+from typing import Any, Callable, Dict, List, Optional, Tuple
 
 import numpy as np
 import optuna
@@ -15,75 +16,66 @@
     eval_candidate_program,
     get_program_with_highest_avg_score,
     get_signature,
-    get_task_model_history_for_full_example,
     print_full_program,
     save_candidate_program,
     set_signature,
+    create_minibatch,
 )
 
-"""
-USAGE SUGGESTIONS:
-
-The following code can be used to compile a optimized signature teleprompter using MIPRO, and evaluate it on an end task:
-
-``` python
-from dspy.teleprompt import MIPROv2
-
-teleprompter = MIPROv2(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=10, init_temperature=1.0)
-kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=0)
-compiled_prompt_opt = teleprompter.compile(program, trainset=trainset[:TRAIN_NUM], num_trials=100, max_bootstrapped_demos=3, max_labeled_demos=5)
-eval_score = evaluate(compiled_prompt_opt, devset=valset[:EVAL_NUM], **kwargs)
-```
-
-Note that this teleprompter takes in the following parameters:
-
-* prompt_model: The model used for prompt generation. When unspecified, defaults to the model set in settings (i.e., dspy.settings.configure(lm=task_model)).
-* task_model: The model used for running your task. When unspecified, defaults to the model set in settings (i.e., dspy.settings.configure(lm=task_model)).
-* teacher_settings: The settings used for the teacher model. When unspecified, defaults to the settings set in settings (i.e., dspy.settings.configure(lm=task_model)).
-    The teacher settings are used to generate the fewshot examples.  This is the LLM/settings to use as a task model for the bootstrapping runs.
-    Typically you would want to use a model of equal or greater quality to your task model.
-* metric: The task metric used for optimization.
-* num_candidates: The number of new prompts and sets of fewshot examples to generate and evaluate. Default=10.
-* init_temperature: The temperature used to generate new prompts. Higher roughly equals more creative. Default=1.0.
-* verbose: Tells the method whether or not to print intermediate steps.
-* track_stats: Tells the method whether or not to track statistics about the optimization process.
-                If True, the method will track a dictionary with a key corresponding to the trial number, 
-                and a value containing a dict with the following keys:
-                    * program: the program being evaluated at a given trial
-                    * score: the last average evaluated score for the program
-                    * pruned: whether or not this program was pruned
-                This information will be returned as attributes of the best program.
-* log_dir: The directory to save logs and other information to. If unspecified, no logs will be saved.
-* view_data_batch_size: The number of examples to view in the data batch when producing the dataset summary. Default=10.
-* minibatch_size: The size of the minibatch to use when evaluating the program if using minibatched evaluations. Default=25.
-* minibatch_full_eval_steps: The number of steps to take before doing a full evaluation of the program if using minibatched evaluations. Default=10.
-* metric_threshold: If the metric yields a numerical value, then check it against this threshold when deciding whether or not to accept a bootstrap example.
-"""
-
+# Constants
 BOOTSTRAPPED_FEWSHOT_EXAMPLES_IN_CONTEXT = 3
 LABELED_FEWSHOT_EXAMPLES_IN_CONTEXT = 0
+MIN_MINIBATCH_SIZE = 50
+
+AUTO_RUN_SETTINGS = {
+    "light": {"num_trials": 7, "val_size": 100},
+    "medium": {"num_trials": 25, "val_size": 300},
+    "heavy": {"num_trials": 50, "val_size": 1000},
+}
+
+# ANSI escape codes for colors
+YELLOW = "\033[93m"
+GREEN = "\033[92m"
+BLUE = "\033[94m"
+BOLD = "\033[1m"
+ENDC = "\033[0m"  # Resets the color to default
+
 
 class MIPROv2(Teleprompter):
     def __init__(
         self,
-        metric,
-        prompt_model=dspy.settings.lm,
-        task_model=dspy.settings.lm,
-        teacher_settings={},
-        num_candidates=10,
-        num_threads=6,
-        max_errors=10,
-        init_temperature=0.5,
-        verbose=False,
-        track_stats=True,
-        log_dir=None,
-        metric_threshold=None,
+        metric: Callable,
+        prompt_model: Optional[Any] = None,
+        task_model: Optional[Any] = None,
+        teacher_settings: Dict = {},
+        max_bootstrapped_demos: int = 4,
+        max_labeled_demos: int = 16,
+        auto: Optional[str] = None,
+        num_candidates: int = 10,
+        num_threads: int = 6,
+        max_errors: int = 10,
+        seed: int = 9,
+        init_temperature: float = 0.5,
+        verbose: bool = False,
+        track_stats: bool = True,
+        log_dir: Optional[str] = None,
+        metric_threshold: Optional[float] = None,
     ):
+        # Validate 'auto' parameter
+        allowed_modes = {None, "light", "medium", "heavy"}
+        if auto not in allowed_modes:
+            raise ValueError(
+                f"Invalid value for auto: {auto}. Must be one of {allowed_modes}."
+            )
+        self.auto = auto
+
         self.num_candidates = num_candidates
         self.metric = metric
         self.init_temperature = init_temperature
-        self.task_model = task_model
-        self.prompt_model = prompt_model
+        self.task_model = task_model if task_model else dspy.settings.lm
+        self.prompt_model = prompt_model if prompt_model else dspy.settings.lm
+        self.max_bootstrapped_demos = max_bootstrapped_demos
+        self.max_labeled_demos = max_labeled_demos
         self.verbose = verbose
         self.track_stats = track_stats
         self.log_dir = log_dir
@@ -93,411 +85,647 @@ def __init__(
         self.num_threads = num_threads
         self.max_errors = max_errors
         self.metric_threshold = metric_threshold
+        self.seed = seed
+        self.rng = None
 
     def compile(
         self,
-        student,
+        student: Any,
         *,
-        trainset,
-        valset=None,
-        num_trials=30,
-        max_bootstrapped_demos=4,
-        max_labeled_demos=16,
-        seed=9,
-        minibatch=True,
-        minibatch_size=25,
-        minibatch_full_eval_steps=10,
-        program_aware_proposer=True,
-        data_aware_proposer=True,
-        view_data_batch_size=10,
-        tip_aware_proposer=True,
-        fewshot_aware_proposer=True,
-        requires_permission_to_run=True,
-    ):
-        # Define ANSI escape codes for colors
-        YELLOW = "\033[93m"
-        GREEN = "\033[92m"
-        BLUE = "\033[94m"
-        BOLD = "\033[1m"
-        ENDC = "\033[0m"  # Resets the color to default
+        trainset: List,
+        valset: Optional[List] = None,
+        num_trials: int = 30,
+        max_bootstrapped_demos: Optional[int] = None,
+        max_labeled_demos: Optional[int] = None,
+        seed: Optional[int] = None,
+        minibatch: bool = True,
+        minibatch_size: int = 25,
+        minibatch_full_eval_steps: int = 10,
+        program_aware_proposer: bool = True,
+        data_aware_proposer: bool = True,
+        view_data_batch_size: int = 10,
+        tip_aware_proposer: bool = True,
+        fewshot_aware_proposer: bool = True,
+        requires_permission_to_run: bool = True,
+    ) -> Any:
+        # Set random seeds
+        seed = seed or self.seed
+        self._set_random_seeds(seed)
+
+        # Update max demos if specified
+        if max_bootstrapped_demos is not None:
+            self.max_bootstrapped_demos = max_bootstrapped_demos
+        if max_labeled_demos is not None:
+            self.max_labeled_demos = max_labeled_demos
+
+        # Set training & validation sets
+        trainset, valset = self._set_and_validate_datasets(trainset, valset)
+
+        # Set hyperparameters based on run mode (if set)
+        zeroshot_opt = (self.max_bootstrapped_demos == 0) and (
+            self.max_labeled_demos == 0
+        )
+        num_trials, valset, minibatch = self._set_hyperparams_from_run_mode(
+            student, num_trials, minibatch, zeroshot_opt, valset
+        )
+
+        if self.auto:
+            self._print_auto_run_settings(num_trials, minibatch, valset)
 
-        random.seed(seed)
+        if minibatch and minibatch_size > len(valset):
+            raise ValueError(
+                f"Minibatch size cannot exceed the size of the valset. Valset size: {len(valset)}."
+            )
 
-        # Validate inputs
+        # Estimate LM calls and get user confirmation
+        if requires_permission_to_run:
+            if not self._get_user_confirmation(
+                student,
+                num_trials,
+                minibatch,
+                minibatch_size,
+                minibatch_full_eval_steps,
+                valset,
+                program_aware_proposer,
+            ):
+                print("Compilation aborted by the user.")
+                return student  # Return the original student program
+
+        # Initialize program and evaluator
+        program = student.deepcopy()
+        evaluate = Evaluate(
+            devset=valset,
+            metric=self.metric,
+            num_threads=self.num_threads,
+            max_errors=self.max_errors,
+            display_table=False,
+            display_progress=True,
+        )
+
+        # Step 1: Bootstrap few-shot examples
+        demo_candidates = self._bootstrap_fewshot_examples(program, trainset, seed)
+
+        # Step 2: Propose instruction candidates
+        instruction_candidates = self._propose_instructions(
+            program,
+            trainset,
+            demo_candidates,
+            view_data_batch_size,
+            program_aware_proposer,
+            data_aware_proposer,
+            tip_aware_proposer,
+            fewshot_aware_proposer,
+        )
+
+        # If zero-shot, discard demos
+        if zeroshot_opt:
+            demo_candidates = None
+
+        # Step 3: Find optimal prompt parameters
+        best_program = self._optimize_prompt_parameters(
+            program,
+            instruction_candidates,
+            demo_candidates,
+            evaluate,
+            valset,
+            num_trials,
+            minibatch,
+            minibatch_size,
+            minibatch_full_eval_steps,
+            seed,
+        )
+
+        return best_program
+    
+    def _set_random_seeds(self,
+        seed
+    ):
+        self.rng = random.Random(seed)
+        np.random.seed(seed)
+
+    def _set_hyperparams_from_run_mode(
+        self,
+        program: Any,
+        num_trials: int,
+        minibatch: bool,
+        zeroshot_opt: bool,
+        valset: List,
+    ) -> Tuple[int, List, bool]:
+        if self.auto is None:
+            return num_trials, valset, minibatch
+
+        num_vars = len(program.predictors())
+        if not zeroshot_opt:
+            num_vars *= 2  # Account for few-shot examples + instruction variables
+
+        auto_settings = AUTO_RUN_SETTINGS[self.auto]
+        num_trials = auto_settings["num_trials"]
+        valset = create_minibatch(valset, batch_size=auto_settings["val_size"], rng=self.rng)
+        minibatch = len(valset) > MIN_MINIBATCH_SIZE
+        self.num_candidates = int(
+            np.round(np.min([num_trials * num_vars, (1.5 * num_trials) / num_vars]))
+        )
+
+        return num_trials, valset, minibatch
+
+    def _set_and_validate_datasets(self, trainset: List, valset: Optional[List]):
         if not trainset:
             raise ValueError("Trainset cannot be empty.")
 
-        if not valset:
+        if valset is None:
             if len(trainset) < 2:
-                raise ValueError("Trainset must have at least 2 examples if no valset specified, or at least 1 example with external validation set.")
-            
-            valset_size = min(500, max(1, int(len(trainset) * 0.80))) # 80% of trainset, capped at 500
+                raise ValueError(
+                    "Trainset must have at least 2 examples if no valset specified."
+                )
+            valset_size = min(500, max(1, int(len(trainset) * 0.80)))
             cutoff = len(trainset) - valset_size
             valset = trainset[cutoff:]
             trainset = trainset[:cutoff]
-
         else:
             if len(valset) < 1:
-                raise ValueError("Validation set must have at least 1 example if specified.")
-            
-        if minibatch and minibatch_size > len(valset):
-            raise ValueError(f"Minibatch size cannot exceed the size of the valset.  Note that your validation set contains {len(valset)} examples.  Your train set contains {len(trainset)} examples.")
-        
-        if minibatch and num_trials < minibatch_full_eval_steps:
-            raise ValueError(f"Number of trials (num_trials={num_trials}) must be greater than or equal to the number of minibatch full eval steps (minibatch_full_eval_steps={minibatch_full_eval_steps}).")
-
-        estimated_prompt_model_calls = 10 + self.num_candidates * len(
-                student.predictors(),
-            ) + (0 if not program_aware_proposer else len(student.predictors()) + 1)  # num data summary calls + N * P + (P + 1)
-        
-        prompt_model_line = ""
-        if not program_aware_proposer:
-            prompt_model_line = f"""{YELLOW}- Prompt Model: {BLUE}{BOLD}10{ENDC}{YELLOW} data summarizer calls + {BLUE}{BOLD}{self.num_candidates}{ENDC}{YELLOW} * {BLUE}{BOLD}{len(student.predictors())}{ENDC}{YELLOW} lm calls in program = {BLUE}{BOLD}{estimated_prompt_model_calls}{ENDC}{YELLOW} prompt model calls{ENDC}"""
-        else:
-            prompt_model_line = f"""{YELLOW}- Prompt Model: {BLUE}{BOLD}10{ENDC}{YELLOW} data summarizer calls + {BLUE}{BOLD}{self.num_candidates}{ENDC}{YELLOW} * {BLUE}{BOLD}{len(student.predictors())}{ENDC}{YELLOW} lm calls in program + ({BLUE}{BOLD}{len(student.predictors()) + 1}{ENDC}{YELLOW}) lm calls in program aware proposer = {BLUE}{BOLD}{estimated_prompt_model_calls}{ENDC}{YELLOW} prompt model calls{ENDC}"""
+                raise ValueError("Validation set must have at least 1 example.")
+
+        return trainset, valset
+
+    def _print_auto_run_settings(self, num_trials: int, minibatch: bool, valset: List):
+        print(
+            f"\nRUNNING WITH THE FOLLOWING {self.auto.upper()} AUTO RUN SETTINGS:"
+            f"\nnum_trials: {num_trials}"
+            f"\nminibatch: {minibatch}"
+            f"\nnum_candidates: {self.num_candidates}"
+            f"\nvalset size: {len(valset)}\n"
+        )
 
-        estimated_task_model_calls_wo_module_calls = 0
-        task_model_line = ""
+    def _estimate_lm_calls(
+        self,
+        program: Any,
+        num_trials: int,
+        minibatch: bool,
+        minibatch_size: int,
+        minibatch_full_eval_steps: int,
+        valset: List,
+        program_aware_proposer: bool,
+    ) -> Tuple[str, str]:
+        num_predictors = len(program.predictors())
+
+        # Estimate prompt model calls
+        estimated_prompt_model_calls = (
+            10  # Data summarizer calls
+            + self.num_candidates * num_predictors  # Candidate generation
+            + (
+                num_predictors + 1 if program_aware_proposer else 0
+            )  # Program-aware proposer
+        )
+        prompt_model_line = (
+            f"{YELLOW}- Prompt Generation: {BLUE}{BOLD}10{ENDC}{YELLOW} data summarizer calls + "
+            f"{BLUE}{BOLD}{self.num_candidates}{ENDC}{YELLOW} * "
+            f"{BLUE}{BOLD}{num_predictors}{ENDC}{YELLOW} lm calls in program "
+            f"+ ({BLUE}{BOLD}{num_predictors + 1}{ENDC}{YELLOW}) lm calls in program-aware proposer "
+            f"= {BLUE}{BOLD}{estimated_prompt_model_calls}{ENDC}{YELLOW} prompt model calls{ENDC}"
+        )
+
+        # Estimate task model calls
         if not minibatch:
-            estimated_task_model_calls_wo_module_calls = len(trainset) * num_trials  # M * T * P
-            task_model_line = f"""{YELLOW}- Task Model: {BLUE}{BOLD}{len(valset)}{ENDC}{YELLOW} examples in val set * {BLUE}{BOLD}{num_trials}{ENDC}{YELLOW} batches * {BLUE}{BOLD}# of LM calls in your program{ENDC}{YELLOW} = ({BLUE}{BOLD}{estimated_task_model_calls_wo_module_calls} * # of LM calls in your program{ENDC}{YELLOW}) task model calls{ENDC}"""
+            estimated_task_model_calls = len(valset) * num_trials
+            task_model_line = (
+                f"{YELLOW}- Program Evaluation: {BLUE}{BOLD}{len(valset)}{ENDC}{YELLOW} examples in val set * "
+                f"{BLUE}{BOLD}{num_trials}{ENDC}{YELLOW} batches = "
+                f"{BLUE}{BOLD}{estimated_task_model_calls}{ENDC}{YELLOW} LM program calls{ENDC}"
+            )
         else:
-            estimated_task_model_calls_wo_module_calls = minibatch_size * num_trials + (len(trainset) * (num_trials // minibatch_full_eval_steps))  # B * T * P
-            task_model_line = f"""{YELLOW}- Task Model: {BLUE}{BOLD}{minibatch_size}{ENDC}{YELLOW} examples in minibatch * {BLUE}{BOLD}{num_trials}{ENDC}{YELLOW} batches + {BLUE}{BOLD}{len(valset)}{ENDC}{YELLOW} examples in val set * {BLUE}{BOLD}{num_trials // minibatch_full_eval_steps}{ENDC}{YELLOW} full evals = {BLUE}{BOLD}{estimated_task_model_calls_wo_module_calls}{ENDC}{YELLOW} task model calls{ENDC}"""
-            
+            full_eval_steps = num_trials // minibatch_full_eval_steps + 1
+            estimated_task_model_calls = (
+                minibatch_size * num_trials + len(valset) * full_eval_steps
+            )
+            task_model_line = (
+                f"{YELLOW}- Program Evaluation: {BLUE}{BOLD}{minibatch_size}{ENDC}{YELLOW} examples in minibatch * "
+                f"{BLUE}{BOLD}{num_trials}{ENDC}{YELLOW} batches + "
+                f"{BLUE}{BOLD}{len(valset)}{ENDC}{YELLOW} examples in val set * "
+                f"{BLUE}{BOLD}{full_eval_steps}{ENDC}{YELLOW} full evals = "
+                f"{BLUE}{BOLD}{estimated_task_model_calls}{ENDC}{YELLOW} LM Program calls{ENDC}"
+            )
+
+        return prompt_model_line, task_model_line
 
-        user_message = textwrap.dedent(f"""\
+    def _get_user_confirmation(
+        self,
+        program: Any,
+        num_trials: int,
+        minibatch: bool,
+        minibatch_size: int,
+        minibatch_full_eval_steps: int,
+        valset: List,
+        program_aware_proposer: bool,
+    ) -> bool:
+        prompt_model_line, task_model_line = self._estimate_lm_calls(
+            program,
+            num_trials,
+            minibatch,
+            minibatch_size,
+            minibatch_full_eval_steps,
+            valset,
+            program_aware_proposer,
+        )
+
+        user_message = textwrap.dedent(
+            f"""\
             {YELLOW}{BOLD}Projected Language Model (LM) Calls{ENDC}
 
-            Please be advised that based on the parameters you have set, the maximum number of LM calls is projected as follows:
+            Based on the parameters you have set, the maximum number of LM calls is projected as follows:
 
-            
             {prompt_model_line}
             {task_model_line}
 
             {YELLOW}{BOLD}Estimated Cost Calculation:{ENDC}
 
             {YELLOW}Total Cost = (Number of calls to task model * (Avg Input Token Length per Call * Task Model Price per Input Token + Avg Output Token Length per Call * Task Model Price per Output Token) 
-                        + (Number of calls to prompt model * (Avg Input Token Length per Call * Task Prompt Price per Input Token + Avg Output Token Length per Call * Prompt Model Price per Output Token).{ENDC}
+                        + (Number of program calls * (Avg Input Token Length per Call * Task Prompt Price per Input Token + Avg Output Token Length per Call * Prompt Model Price per Output Token).{ENDC}
 
             For a preliminary estimate of potential costs, we recommend you perform your own calculations based on the task
             and prompt models you intend to use. If the projected costs exceed your budget or expectations, you may consider:
 
             {YELLOW}- Reducing the number of trials (`num_trials`), the size of the valset, or the number of LM calls in your program.{ENDC}
-            {YELLOW}- Using a cheaper task model to optimize the prompt.{ENDC}\n""")
+            {YELLOW}- Using a cheaper task model to optimize the prompt.{ENDC}
+            {YELLOW}- Setting `minibatch=True` if you haven't already.{ENDC}\n"""
+        )
 
-        user_confirmation_message = textwrap.dedent(f"""\
+        user_confirmation_message = textwrap.dedent(
+            f"""\
             To proceed with the execution of this program, please confirm by typing {BLUE}'y'{ENDC} for yes or {BLUE}'n'{ENDC} for no.
 
             If you would like to bypass this confirmation step in future executions, set the {YELLOW}`requires_permission_to_run`{ENDC} flag to {YELLOW}`False`{ENDC} when calling compile.
 
             {YELLOW}Awaiting your input...{ENDC}
-        """)
-
-        if requires_permission_to_run: print(user_message)
-
-        sys.stdout.flush()  # Flush the output buffer to force the message to print
+        """
+        )
+
+        print(user_message)
+        sys.stdout.flush()
+        print(user_confirmation_message)
+        user_input = input("Do you wish to continue? (y/n): ").strip().lower()
+        return user_input == "y"
+
+    def _bootstrap_fewshot_examples(
+        self, program: Any, trainset: List, seed: int
+    ) -> Optional[List]:
+        print("\n==> STEP 1: BOOTSTRAP FEWSHOT EXAMPLES <==")
+        if self.max_bootstrapped_demos > 0:
+            print(
+                "These will be used as few-shot example candidates for our program and for creating instructions.\n"
+            )
+        else:
+            print("These will be used for informing instruction proposal.\n")
 
-        run = True
-        # TODO: make sure these estimates are good for mini-batching
-        if requires_permission_to_run:
-            print(user_confirmation_message)
-            user_input = input("Do you wish to continue? (y/n): ").strip().lower()
-            if user_input != "y":
-                print("Compilation aborted by the user.")
-                run = False
+        print(f"Bootstrapping N={self.num_candidates} sets of demonstrations...")
 
-        if run:
-            # Setup random seeds
-            random.seed(seed)
-            np.random.seed(seed)
+        zeroshot = self.max_bootstrapped_demos == 0 and self.max_labeled_demos == 0
 
-            # Set up program and evaluation function
-            program = student.deepcopy()
-            evaluate = Evaluate(
-                devset=valset,
+        try:
+            demo_candidates = create_n_fewshot_demo_sets(
+                student=program,
+                num_candidate_sets=self.num_candidates,
+                trainset=trainset,
+                max_labeled_demos=(
+                    LABELED_FEWSHOT_EXAMPLES_IN_CONTEXT
+                    if zeroshot
+                    else self.max_labeled_demos
+                ),
+                max_bootstrapped_demos=(
+                    BOOTSTRAPPED_FEWSHOT_EXAMPLES_IN_CONTEXT
+                    if zeroshot
+                    else self.max_bootstrapped_demos
+                ),
                 metric=self.metric,
-                num_threads=self.num_threads,
                 max_errors=self.max_errors,
-                display_table=False,
-                display_progress=True,
+                teacher_settings=self.teacher_settings,
+                seed=seed,
+                metric_threshold=self.metric_threshold,
+                rng=self.rng,
             )
+        except Exception as e:
+            print(f"Error generating few-shot examples: {e}")
+            print("Running without few-shot examples.")
+            demo_candidates = None
+
+        return demo_candidates
 
-            # Determine the number of fewshot examples to use to generate demos for prompt
-            if max_bootstrapped_demos == 0 and max_labeled_demos == 0:
-                max_bootstrapped_demos_for_candidate_gen = BOOTSTRAPPED_FEWSHOT_EXAMPLES_IN_CONTEXT
-                max_labeled_demos_for_candidate_gen = LABELED_FEWSHOT_EXAMPLES_IN_CONTEXT
+    def _propose_instructions(
+        self,
+        program: Any,
+        trainset: List,
+        demo_candidates: Optional[List],
+        view_data_batch_size: int,
+        program_aware_proposer: bool,
+        data_aware_proposer: bool,
+        tip_aware_proposer: bool,
+        fewshot_aware_proposer: bool,
+    ) -> Dict[int, List[str]]:
+        print("\n==> STEP 2: PROPOSE INSTRUCTION CANDIDATES <==")
+        print(
+            "We will use the few-shot examples from the previous step, a generated dataset summary, a summary of the program code, and a randomly selected prompting tip to propose instructions."
+        )
+
+        proposer = GroundedProposer(
+            program=program,
+            trainset=trainset,
+            prompt_model=self.prompt_model,
+            view_data_batch_size=view_data_batch_size,
+            program_aware=program_aware_proposer,
+            use_dataset_summary=data_aware_proposer,
+            use_task_demos=fewshot_aware_proposer,
+            use_tip=tip_aware_proposer,
+            set_tip_randomly=tip_aware_proposer,
+            use_instruct_history=False,
+            set_history_randomly=False,
+            verbose=self.verbose,
+            rng=self.rng
+        )
+
+        print("\nProposing instructions...\n")
+        instruction_candidates = proposer.propose_instructions_for_program(
+            trainset=trainset,
+            program=program,
+            demo_candidates=demo_candidates,
+            N=self.num_candidates,
+            T=self.init_temperature,
+            trial_logs={},
+        )
+
+        for i, pred in enumerate(program.predictors()):
+            print(f"Proposed Instructions for Predictor {i}:\n")
+            instruction_candidates[i][0] = get_signature(pred).instructions
+            for j, instruction in enumerate(instruction_candidates[i]):
+                print(f"{j}: {instruction}\n")
+            print("\n")
+
+        return instruction_candidates
+
+    def _optimize_prompt_parameters(
+        self,
+        program: Any,
+        instruction_candidates: Dict[int, List[str]],
+        demo_candidates: Optional[List],
+        evaluate: Evaluate,
+        valset: List,
+        num_trials: int,
+        minibatch: bool,
+        minibatch_size: int,
+        minibatch_full_eval_steps: int,
+        seed: int,
+    ) -> Optional[Any]:
+        print("Evaluating the default program...\n")
+        default_score = eval_candidate_program(len(valset), valset, program, evaluate, self.rng)
+        print(f"Default program score: {default_score}\n")
+
+        # Initialize optimization variables
+        best_score = default_score
+        best_program = program.deepcopy()
+        trial_logs = {}
+        total_eval_calls = 0
+        if minibatch:
+            scores = []
+        else:
+            scores = [default_score]
+        full_eval_scores = [default_score]
+        param_score_dict = defaultdict(list)
+        fully_evaled_param_combos = {}
+
+        # Define the objective function
+        def objective(trial):
+            nonlocal program, best_program, best_score, trial_logs, total_eval_calls, scores, full_eval_scores
+
+            trial_num = trial.number + 1
+            if minibatch:
+                print(f"== Minibatch Trial {trial_num} / {num_trials} ==")
             else:
-                max_bootstrapped_demos_for_candidate_gen = max_bootstrapped_demos
-                max_labeled_demos_for_candidate_gen = max_labeled_demos
-            
-            # Generate N few shot example sets (these will inform instruction creation, and be used as few-shot examples in our prompt)
-            print("Beginning MIPROv2 optimization process...")
-            print("\n==> STEP 1: BOOTSTRAP FEWSHOT EXAMPLES <==")
-            if max_bootstrapped_demos > 0:
-                print("These will be used for as few-shot examples candidates for our program and for creating instructions.\n")
+                print(f"===== Trial {trial_num} / {num_trials} =====")
+
+            trial_logs[trial_num] = {}
+
+            # Create a new candidate program
+            candidate_program = program.deepcopy()
+
+            # Choose instructions and demos, insert them into the program
+            chosen_params = self._select_and_insert_instructions_and_demos(
+                candidate_program,
+                instruction_candidates,
+                demo_candidates,
+                trial,
+                trial_logs,
+                trial_num,
+            )
+
+            # Log assembled program
+            if self.verbose:
+                print("Evaluating the following candidate program...\n")
+                print_full_program(candidate_program)
+
+            # Save the candidate program
+            trial_logs[trial_num]["program_path"] = save_candidate_program(
+                candidate_program, self.log_dir, trial_num
+            )
+
+            # Evaluate the candidate program
+            batch_size = minibatch_size if minibatch else len(valset)
+            score = eval_candidate_program(
+                batch_size, valset, candidate_program, evaluate, self.rng
+            )
+
+            # Update best score and program
+            if not minibatch and score > best_score:
+                best_score = score
+                best_program = candidate_program.deepcopy()
+                print(f"{GREEN}Best full score so far!{ENDC} Score: {score}")
+
+            # Log evaluation results
+            scores.append(score)
+            if minibatch:
+                self._log_minibatch_eval(
+                    score,
+                    best_score,
+                    batch_size,
+                    chosen_params,
+                    scores,
+                    full_eval_scores,
+                    trial,
+                    num_trials,
+                )
             else:
-                print("These will be used for informing instruction proposal.\n")
-            print(f"Bootstrapping N={self.num_candidates} sets of demonstrations...")
-            try:
-                demo_candidates = create_n_fewshot_demo_sets(
-                    student=program,
-                    num_candidate_sets=self.num_candidates,
-                    trainset=trainset,
-                    max_labeled_demos=max_labeled_demos_for_candidate_gen,
-                    max_bootstrapped_demos=max_bootstrapped_demos_for_candidate_gen,
-                    metric=self.metric,
-                    max_errors=self.max_errors,
-                    teacher_settings=self.teacher_settings,
-                    seed=seed,
-                    metric_threshold=self.metric_threshold,
+                self._log_normal_eval(
+                    score, best_score, chosen_params, scores, trial, num_trials
                 )
-            except Exception as e:
-                print(f"Error generating fewshot examples: {e}")
-                print("Running without fewshot examples.")
-                demo_candidates = None
-            
-            # Generate N candidate instructions
-
-            # Setup our proposer 
-            print("\n==> STEP 2: PROPOSE INSTRUCTION CANDIDATES <==")
-            print("In this step, by default we will use the few-shot examples from the previous step, a generated dataset summary, a summary of the program code, and a randomly selected prompting tip to propose instructions.")
-
-            proposer = GroundedProposer(
-                program=program,
-                trainset=trainset,
-                prompt_model=self.prompt_model,
-                # program_code_string=self.program_code_string,
-                view_data_batch_size=view_data_batch_size,
-                program_aware=program_aware_proposer,
-                use_dataset_summary=data_aware_proposer,
-                use_task_demos=fewshot_aware_proposer,
-                use_tip=tip_aware_proposer,
-                set_tip_randomly=tip_aware_proposer,
-                use_instruct_history=False,
-                set_history_randomly=False,
-                verbose = self.verbose,
+            categorical_key = ",".join(map(str, chosen_params))
+            param_score_dict[categorical_key].append(
+                (score, candidate_program),
             )
-            
-            print("\nProposing instructions...\n")
-            instruction_candidates = proposer.propose_instructions_for_program(
-                trainset=trainset,
-                program=program,
-                demo_candidates=demo_candidates,
-                N=self.num_candidates,
-                T=self.init_temperature,
-                trial_logs={},
-            )
-            for i, pred in enumerate(program.predictors()):
-                print(f"Proposed Instructions for Predictor {i}:\n")
-                instruction_candidates[i][0] = get_signature(pred).instructions
-                for j, instruction in enumerate(instruction_candidates[i]):
-                    print(f"{j}: {instruction}\n")
-                print("\n")
-
-            # If we're doing zero-shot, reset demo_candidates to none now that we've used them for instruction proposal
-            if max_bootstrapped_demos == 0 and max_labeled_demos == 0:
-                demo_candidates = None
-
-            # Initialize variables to track during the optimization process
-            best_scoring_trial = 0
-            trial_logs = {}
-            total_eval_calls = 0
-            param_score_dict = defaultdict(list) # Dictionaries of paramater combinations we've tried, and their associated scores
-            fully_evaled_param_combos = {} # List of the parameter combinations we've done full evals of
-
-            # Evaluate the default program
-            print("Evaluating the default program...\n")
-            default_score = eval_candidate_program(len(valset), valset, program, evaluate)
-            print(f"Default program score: {default_score}\n")
-
-            best_score = default_score
-            best_program = program.deepcopy()
-
-
-            # Define our trial objective
-            def create_objective(
-                baseline_program,
-                instruction_candidates,
-                demo_candidates,
-                evaluate,
-                valset,
+            trial_logs[trial_num]["num_eval_calls"] = batch_size
+            trial_logs[trial_num]["full_eval"] = batch_size >= len(valset)
+            trial_logs[trial_num]["score"] = score
+            total_eval_calls += batch_size
+            trial_logs[trial_num]["total_eval_calls_so_far"] = total_eval_calls
+            trial_logs[trial_num]["program"] = candidate_program.deepcopy()
+
+            # If minibatch, perform full evaluation at intervals
+            if minibatch and (
+                (trial_num % minibatch_full_eval_steps == 0)
+                or (trial_num == num_trials)
             ):
-                def objective(trial):
-                    nonlocal best_program, best_score, best_scoring_trial, trial_logs, total_eval_calls  # Allow access to the outer variables
-                    
-                    # Kick off trial
-                    if minibatch:
-                        print(f"== Minibatch Trial {trial.number+1} / {num_trials} ==")
-                    else:
-                        print(f"===== Trial {trial.number+1} / {num_trials} =====")
-                    trial_logs[trial.number+1] = {}
-
-                    # Create a new candidate program
-                    candidate_program = baseline_program.deepcopy()
-
-                    # Choose set of instructions & demos to use for each predictor
-                    chosen_params = []
-                    for i, p_new in enumerate(candidate_program.predictors()):
-
-                        # Get instruction candidates / demos for our given predictor
-                        p_instruction_candidates = instruction_candidates[i]
-                        if demo_candidates:
-                            p_demo_candidates = demo_candidates[i]
-
-                        # Suggest the index of the instruction / demo candidate to use in our trial
-                        instruction_idx = trial.suggest_categorical(
-                            f"{i}_predictor_instruction",
-                            range(len(p_instruction_candidates)),
-                        )
-                        # chosen_params.append(instruction_idx)
-                        chosen_params.append(f"Predictor {i+1}: Instruction {instruction_idx}")
-                        if demo_candidates:
-                            demos_idx = trial.suggest_categorical(
-                                f"{i}_predictor_demos", range(len(p_demo_candidates)),
-                            )
-                            chosen_params.append(f"Predictor {i+1}: Few-Shot Set {demos_idx}")
-
-                        # Log the selected instruction / demo candidate
-                        trial_logs[trial.number+1][
-                            f"{i}_predictor_instruction"
-                        ] = instruction_idx
-                        if demo_candidates:
-                            trial_logs[trial.number+1][f"{i}_predictor_demos"] = demos_idx
-
-                        dspy.logger.debug(f"instruction_idx {instruction_idx}")
-                        if demo_candidates:
-                            dspy.logger.debug(f"demos_idx {demos_idx}")
-
-                        # Set the instruction
-                        selected_instruction = p_instruction_candidates[instruction_idx]
-                        updated_signature = get_signature(p_new).with_instructions(
-                            selected_instruction,
-                        )
-                        set_signature(p_new, updated_signature)
-
-                        # Set the demos
-                        if demo_candidates:
-                            p_new.demos = p_demo_candidates[demos_idx]
-
-                    # Log assembled program
-                    if self.verbose: print("Evaluating the following candidate program...\n")
-                    if self.verbose: print_full_program(candidate_program)
-
-                    # Save the candidate program
-                    trial_logs[trial.number+1]["program_path"] = save_candidate_program(
-                        candidate_program, self.log_dir, trial.number+1,
-                    )
-
-                    trial_logs[trial.number+1]["num_eval_calls"] = 0
-
-                    # Evaluate the candidate program with relevant batch size
-                    batch_size = minibatch_size if minibatch else len(valset)
-
-                    score = eval_candidate_program(
-                        batch_size, valset, candidate_program, 
-                        evaluate,
-                    )
-
-                    # Print out a full trace of the program in use
-                    if self.verbose:
-                        print("Full trace of prompts in use on an example...")
-                        get_task_model_history_for_full_example(
-                            candidate_program, self.task_model, valset, evaluate,
-                        )
-
-                    # Log relevant information
-                    categorical_key = ",".join(map(str, chosen_params))
-                    param_score_dict[categorical_key].append(
-                        (score, candidate_program),
-                    )
-                    trial_logs[trial.number+1]["num_eval_calls"] = batch_size
-                    trial_logs[trial.number+1]["full_eval"] = batch_size >= len(valset)
-                    trial_logs[trial.number+1]["score"] = score
-                    trial_logs[trial.number+1]["pruned"] = False
-                    total_eval_calls += trial_logs[trial.number+1]["num_eval_calls"]
-                    trial_logs[trial.number+1]["total_eval_calls_so_far"] = total_eval_calls
-                    trial_logs[trial.number+1]["program"] = candidate_program.deepcopy()
-
-                    # If this score was from a full evaluation, update the best program if the new score is better
-                    best_score_updated = False
-                    if score > best_score and trial_logs[trial.number+1]["full_eval"] and not minibatch:
-                        best_score = score
-                        best_scoring_trial = trial.number+1
-                        best_program = candidate_program.deepcopy()
-                        best_score_updated = True
-                    
-                    if minibatch:
-                        print(f"Score: {score} on minibatch of size {batch_size} with parameters {chosen_params}.\n\n")
-                    else:
-                        print(f"Score: {score} with parameters {chosen_params}.")
-                        if best_score_updated:
-                            print(f"{GREEN}New best score updated!{ENDC} Score: {best_score} on trial {best_scoring_trial}.\n\n")
-                        else:
-                            print(f"Best score so far: {best_score} on trial {best_scoring_trial}.\n\n")
-                        
-
-                    # If we're doing minibatching, check to see if it's time to do a full eval
-                    if minibatch and (((trial.number+1) % minibatch_full_eval_steps == 0) or (trial.number+1 == num_trials)):
-                        print(f"===== Full Eval {len(fully_evaled_param_combos)+1} =====")
-                        
-                        # Save old information as the minibatch version
-                        trial_logs[trial.number+1]["mb_score"] = score
-                        trial_logs[trial.number+1]["mb_program_path"] = trial_logs[trial.number+1]["program_path"]
-
-                        # Identify our best program (based on mean of scores so far, and do a full eval on it)
-                        highest_mean_program, mean, combo_key = get_program_with_highest_avg_score(param_score_dict, fully_evaled_param_combos)
-                        
-                        if trial.number+1 // minibatch_full_eval_steps > 0:
-                            print(f"Doing full eval on next top averaging program (Avg Score: {mean}) so far from mini-batch trials...")
-                        else:
-                            print(f"Doing full eval on top averaging program (Avg Score: {mean}) so far from mini-batch trials...")
-                        full_val_score = eval_candidate_program(
-                            len(valset), valset, highest_mean_program, evaluate,
-                        )
-
-                        # Log relevant information
-                        fully_evaled_param_combos[combo_key] = {"program":highest_mean_program, "score": full_val_score}
-                        total_eval_calls += len(valset)
-                        trial_logs[trial.number+1]["total_eval_calls_so_far"] = total_eval_calls
-                        trial_logs[trial.number+1]["full_eval"] = True
-                        trial_logs[trial.number+1]["program_path"] = save_candidate_program(
-                            program=highest_mean_program, log_dir=self.log_dir, trial_num=trial.number+1, note="full_eval",
-                        )
-                        trial_logs[trial.number+1]["score"] = full_val_score
-                        
-                        if full_val_score > best_score:
-                            print(f"{GREEN}Best full eval score so far!{ENDC} Score: {full_val_score}")
-                            best_score = full_val_score
-                            best_scoring_trial = trial.number+1
-                            best_program = highest_mean_program.deepcopy()
-                            best_score_updated = True
-                        else:
-                            print(f"Full eval score: {full_val_score}")
-                            print(f"Best full eval score so far: {best_score}")
-                        print("=======================\n\n")
-                    
-                    return score
-
-                return objective
-
-            # Run the trial
-            optuna.logging.set_verbosity(optuna.logging.WARNING)
-            print("==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==")
-            print("In this step, we will evaluate the program over a series of trials with different combinations of instructions and few-shot examples to find the optimal combination. Bayesian Optimization will be used for this search process.\n")
-            objective_function = create_objective(
-                program, instruction_candidates, demo_candidates, evaluate, valset,
+                best_score, best_program = self._perform_full_evaluation(
+                    trial_num,
+                    param_score_dict,
+                    fully_evaled_param_combos,
+                    evaluate,
+                    valset,
+                    trial_logs,
+                    total_eval_calls,
+                    full_eval_scores,
+                    best_score,
+                    best_program,
+                )
+
+            return score
+
+        # Run optimization
+        optuna.logging.set_verbosity(optuna.logging.WARNING)
+        print("==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==")
+        print(
+            "We will evaluate the program over a series of trials with different combinations of instructions and few-shot examples to find the optimal combination using Bayesian Optimization.\n"
+        )
+
+        sampler = optuna.samplers.TPESampler(seed=seed, multivariate=True)
+        study = optuna.create_study(direction="maximize", sampler=sampler)
+        study.optimize(objective, n_trials=num_trials)
+
+        # Attach logs to best program
+        if best_program is not None and self.track_stats:
+            best_program.trial_logs = trial_logs
+            best_program.score = best_score
+            best_program.prompt_model_total_calls = self.prompt_model_total_calls
+            best_program.total_calls = self.total_calls
+
+        print(f"Returning best identified program with score {best_score}!")
+
+        return best_program
+
+    def _log_minibatch_eval(
+        self,
+        score,
+        best_score,
+        batch_size,
+        chosen_params,
+        scores,
+        full_eval_scores,
+        trial,
+        num_trials,
+    ):
+        print(
+            f"Score: {score} on minibatch of size {batch_size} with parameters {chosen_params}."
+        )
+        print(f"Minibatch scores so far: {'['+', '.join([f'{s}' for s in scores])+']'}")
+        trajectory = "[" + ", ".join([f"{s}" for s in full_eval_scores]) + "]"
+        print(f"Full eval scores so far: {trajectory}")
+        print(f"Best full score so far: {best_score}")
+        print(
+            f'{"="*len(f"== Minibatch Trial {trial.number+1} / {num_trials} ==")}\n\n'
+        )
+
+    def _log_normal_eval(
+        self, score, best_score, chosen_params, scores, trial, num_trials
+    ):
+        print(f"Score: {score} with parameters {chosen_params}.")
+        print(f"Scores so far: {'['+', '.join([f'{s}' for s in scores])+']'}")
+        print(f"Best score so far: {best_score}")
+        print(f'{"="*len(f"===== Trial {trial.number+1} / {num_trials} =====")}\n\n')
+
+    def _select_and_insert_instructions_and_demos(
+        self,
+        candidate_program: Any,
+        instruction_candidates: Dict[int, List[str]],
+        demo_candidates: Optional[List],
+        trial: optuna.trial.Trial,
+        trial_logs: Dict,
+        trial_num: int,
+    ) -> List[str]:
+        chosen_params = []
+
+        for i, predictor in enumerate(candidate_program.predictors()):
+            # Select instruction
+            instruction_idx = trial.suggest_categorical(
+                f"{i}_predictor_instruction", range(len(instruction_candidates[i]))
+            )
+            selected_instruction = instruction_candidates[i][instruction_idx]
+            updated_signature = get_signature(predictor).with_instructions(
+                selected_instruction
             )
-            sampler = optuna.samplers.TPESampler(seed=seed, multivariate=True)
-            study = optuna.create_study(direction="maximize", sampler=sampler)
-            _ = study.optimize(objective_function, n_trials=num_trials)
+            set_signature(predictor, updated_signature)
+            trial_logs[trial_num][f"{i}_predictor_instruction"] = instruction_idx
+            chosen_params.append(f"Predictor {i+1}: Instruction {instruction_idx}")
+
+            # Select demos if available
+            if demo_candidates:
+                demos_idx = trial.suggest_categorical(
+                    f"{i}_predictor_demos", range(len(demo_candidates[i]))
+                )
+                predictor.demos = demo_candidates[i][demos_idx]
+                trial_logs[trial_num][f"{i}_predictor_demos"] = demos_idx
+                chosen_params.append(f"Predictor {i+1}: Few-Shot Set {demos_idx}")
 
-            if best_program is not None and self.track_stats:
-                best_program.trial_logs = trial_logs
-                best_program.score = best_score
-                best_program.prompt_model_total_calls = self.prompt_model_total_calls
-                best_program.total_calls = self.total_calls
+        return chosen_params
 
-            return best_program
+    def _perform_full_evaluation(
+        self,
+        trial_num: int,
+        param_score_dict: Dict,
+        fully_evaled_param_combos: Dict,
+        evaluate: Evaluate,
+        valset: List,
+        trial_logs: Dict,
+        total_eval_calls: int,
+        full_eval_scores: List[int],
+        best_score: float,
+        best_program: Any,
+    ):
+        print(f"===== Full Eval {len(fully_evaled_param_combos)+1} =====")
 
-        return student
+        # Identify best program to evaluate fully
+        highest_mean_program, mean_score, combo_key = (
+            get_program_with_highest_avg_score(
+                param_score_dict, fully_evaled_param_combos
+            )
+        )
+        print(
+            f"Doing full eval on next top averaging program (Avg Score: {mean_score}) from minibatch trials..."
+        )
+        full_eval_score = eval_candidate_program(
+            len(valset), valset, highest_mean_program, evaluate, self.rng
+        )
+        full_eval_scores.append(full_eval_score)
+
+        # Log full evaluation results
+        fully_evaled_param_combos[combo_key] = {
+            "program": highest_mean_program,
+            "score": full_eval_score,
+        }
+        total_eval_calls += len(valset)
+        trial_logs[trial_num]["total_eval_calls_so_far"] = total_eval_calls
+        trial_logs[trial_num]["full_eval"] = True
+        trial_logs[trial_num]["program_path"] = save_candidate_program(
+            program=highest_mean_program,
+            log_dir=self.log_dir,
+            trial_num=trial_num,
+            note="full_eval",
+        )
+        trial_logs[trial_num]["score"] = full_eval_score
+
+        # Update best score and program if necessary
+        if full_eval_score > best_score:
+            print(f"{GREEN}New best full eval score!{ENDC} Score: {full_eval_score}")
+            best_score = full_eval_score
+            best_program = highest_mean_program.deepcopy()
+        trajectory = "[" + ", ".join([f"{s}" for s in full_eval_scores]) + "]"
+        print(f"Full eval scores so far: {trajectory}")
+        print(f"Best full score so far: {best_score}")
+        print(len(f"===== Full Eval {len(fully_evaled_param_combos)+1} =====") * "=")
+        print("\n")
+
+        return best_score, best_program
diff --git a/dspy/teleprompt/random_search.py b/dspy/teleprompt/random_search.py
index 57bd38150..4fa8ba23f 100644
--- a/dspy/teleprompt/random_search.py
+++ b/dspy/teleprompt/random_search.py
@@ -85,6 +85,7 @@ def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None
                     max_labeled_demos=self.max_labeled_demos,
                     teacher_settings=self.teacher_settings,
                     max_rounds=self.max_rounds,
+                    max_errors=self.max_errors,
                 )
                 program = optimizer.compile(student, teacher=teacher, trainset=trainset_copy)
 
@@ -101,6 +102,7 @@ def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None
                     max_labeled_demos=self.max_labeled_demos,
                     teacher_settings=self.teacher_settings,
                     max_rounds=self.max_rounds,
+                    max_errors=self.max_errors,
                 )
 
                 program = optimizer.compile(student, teacher=teacher, trainset=trainset_copy)
diff --git a/dspy/teleprompt/utils.py b/dspy/teleprompt/utils.py
index ebcabe229..3640caadf 100644
--- a/dspy/teleprompt/utils.py
+++ b/dspy/teleprompt/utils.py
@@ -24,14 +24,17 @@
 ### OPTIMIZER TRAINING UTILS ###
 
 
-def create_minibatch(trainset, batch_size=50):
+def create_minibatch(trainset, batch_size=50, rng=None):
     """Create a minibatch from the trainset."""
 
     # Ensure batch_size isn't larger than the size of the dataset
     batch_size = min(batch_size, len(trainset))
 
-    # Randomly sample indices for the mini-batch
-    sampled_indices = random.sample(range(len(trainset)), batch_size)
+    # If no RNG is provided, fall back to the global random instance
+    rng = rng or random
+
+    # Randomly sample indices for the mini-batch using the provided rng
+    sampled_indices = rng.sample(range(len(trainset)), batch_size)
 
     # Create the mini-batch using the sampled indices
     minibatch = [trainset[i] for i in sampled_indices]
@@ -39,7 +42,7 @@ def create_minibatch(trainset, batch_size=50):
     return minibatch
 
 
-def eval_candidate_program(batch_size, trainset, candidate_program, evaluate):
+def eval_candidate_program(batch_size, trainset, candidate_program, evaluate, rng=None):
     """Evaluate a candidate program on the trainset, using the specified batch size."""
     # Evaluate on the full trainset
     if batch_size >= len(trainset):
@@ -48,7 +51,7 @@ def eval_candidate_program(batch_size, trainset, candidate_program, evaluate):
     else:
         score = evaluate(
             candidate_program,
-            devset=create_minibatch(trainset, batch_size),
+            devset=create_minibatch(trainset, batch_size, rng),
         )
 
     return score
@@ -279,6 +282,7 @@ def create_n_fewshot_demo_sets(
     teacher=None,
     include_non_bootstrapped=True,
     seed=0,
+    rng=None
 ):
     """
     This function is copied from random_search.py, and creates fewshot examples in the same way that random search does.
@@ -292,17 +296,15 @@ def create_n_fewshot_demo_sets(
     # Initialize demo_candidates dictionary
     for i, _ in enumerate(student.predictors()):
         demo_candidates[i] = []
-    
-    starter_seed = seed
-    # Shuffle the trainset with the starter seed
-    random.Random(starter_seed).shuffle(trainset)
+
+    rng = rng or random.Random(seed)
 
     # Go through and create each candidate set
     for seed in range(-3, num_candidate_sets):
 
         print(f"Bootstrapping set {seed+4}/{num_candidate_sets+3}")
 
-        trainset2 = list(trainset)
+        trainset_copy = list(trainset)
 
         if seed == -3 and include_non_bootstrapped:
             # zero-shot
@@ -316,7 +318,7 @@ def create_n_fewshot_demo_sets(
             # labels only
             teleprompter = LabeledFewShot(k=max_labeled_demos)
             program2 = teleprompter.compile(
-                student, trainset=trainset2, sample=labeled_sample,
+                student, trainset=trainset_copy, sample=labeled_sample,
             )
 
         elif seed == -1:
@@ -329,12 +331,12 @@ def create_n_fewshot_demo_sets(
                 teacher_settings=teacher_settings,
                 max_rounds=max_rounds,
             )
-            program2 = program.compile(student, teacher=teacher, trainset=trainset2)
+            program2 = program.compile(student, teacher=teacher, trainset=trainset_copy)
 
         else:
             # shuffled few-shot
-            random.Random(seed).shuffle(trainset2)
-            size = random.Random(seed).randint(min_num_samples, max_bootstrapped_demos)
+            rng.shuffle(trainset_copy)
+            size = rng.randint(min_num_samples, max_bootstrapped_demos)
 
             teleprompter = BootstrapFewShot(
                 metric=metric,
@@ -347,7 +349,7 @@ def create_n_fewshot_demo_sets(
             )
 
             program2 = teleprompter.compile(
-                student, teacher=teacher, trainset=trainset2,
+                student, teacher=teacher, trainset=trainset_copy,
             )
 
         for i, _ in enumerate(student.predictors()):
diff --git a/dspy/utils/dummies.py b/dspy/utils/dummies.py
index d97d59b5a..b26cdda04 100644
--- a/dspy/utils/dummies.py
+++ b/dspy/utils/dummies.py
@@ -1,15 +1,18 @@
 import random
 import re
+from collections import defaultdict
 from typing import Union
 
 import numpy as np
 
-from dsp.modules import LM
+from dsp.modules import LM as DSPLM
 from dsp.utils.utils import dotdict
+from dspy.adapters.chat_adapter import field_header_pattern, format_fields
+from dspy.clients.lm import LM
 
 
-class DummyLM(LM):
-    """Dummy language model for unit testing purposes."""
+class DSPDummyLM(DSPLM):
+    """Dummy language model for unit testing purposes subclassing DSP LM class."""
 
     def __init__(self, answers: Union[list[str], dict[str, str]], follow_examples: bool = False):
         """Initializes the dummy language model.
@@ -64,7 +67,7 @@ def basic_request(self, prompt, n=1, **kwargs) -> dict[str, list[dict[str, str]]
                 },
             )
 
-            RED, GREEN, RESET = "\033[91m", "\033[92m", "\033[0m"
+            RED, _, RESET = "\033[91m", "\033[92m", "\033[0m"
             print("=== DummyLM ===")
             print(prompt, end="")
             print(f"{RED}{answer}{RESET}")
@@ -94,6 +97,111 @@ def get_convo(self, index) -> str:
         return self.history[index]["prompt"] + " " + self.history[index]["response"]["choices"][0]["text"]
 
 
+class DummyLM(LM):
+    """
+    Dummy language model for unit testing purposes.
+
+    Three modes of operation:
+
+    ## 1. List of dictionaries
+
+    If a list of dictionaries is provided, the dummy model will return the next dictionary
+    in the list for each request, formatted according to the `format_fields` function.
+    from the chat adapter.
+
+    ```python
+        lm = DummyLM([{"answer": "red"}, {"answer": "blue"}])
+        dspy.settings.configure(lm=lm)
+        predictor("What color is the sky?")
+        # Output: "[[## answer ##]]\nred"
+        predictor("What color is the sky?")
+        # Output: "[[## answer ##]]\nblue"
+    ```
+
+    ## 2. Dictionary of dictionaries
+
+    If a dictionary of dictionaries is provided, the dummy model will return the value
+    corresponding to the key which is contained with the final message of the prompt,
+    formatted according to the `format_fields` function from the chat adapter.
+
+    ```python
+        lm = DummyLM({"What color is the sky?": {"answer": "blue"}})
+        dspy.settings.configure(lm=lm)
+        predictor("What color is the sky?")
+        # Output: "[[## answer ##]]\nblue"
+    ```
+
+    ## 3. Follow examples
+
+    If `follow_examples` is set to True, and the prompt contains an example input exactly equal to the prompt,
+    the dummy model will return the output from that example.
+
+    ```python
+        lm = DummyLM([{"answer": "red"}], follow_examples=True)
+        dspy.settings.configure(lm=lm)
+        predictor("What color is the sky?, demos=dspy.Example(input="What color is the sky?", output="blue"))
+        # Output: "[[## answer ##]]\nblue"
+    ```
+
+    """
+
+    def __init__(self, answers: Union[list[dict[str, str]], dict[str, dict[str, str]]], follow_examples: bool = False):
+        super().__init__("dummy", "chat", 0.0, 1000, True)
+        self.answers = answers
+        if isinstance(answers, list):
+            self.answers = iter(answers)
+        self.follow_examples = follow_examples
+
+    def _use_example(self, messages):
+        # find all field names
+        fields = defaultdict(int)
+        for message in messages:
+            if "content" in message:
+                if ma := field_header_pattern.match(message["content"]):
+                    fields[message["content"][ma.start() : ma.end()]] += 1
+        # find the fields which are missing from the final turns
+        max_count = max(fields.values())
+        output_fields = [field for field, count in fields.items() if count != max_count]
+
+        # get the output from the last turn that has the output fields as headers
+        final_input = messages[-1]["content"].split("\n\n")[0]
+        for input, output in zip(reversed(messages[:-1]), reversed(messages)):
+            if any(field in output["content"] for field in output_fields) and final_input in input["content"]:
+                return output["content"]
+
+    def __call__(self, prompt=None, messages=None, **kwargs):
+        # Build the request.
+        outputs = []
+        for _ in range(kwargs.get("n", 1)):
+            messages = messages or [{"role": "user", "content": prompt}]
+            kwargs = {**self.kwargs, **kwargs}
+
+            if self.follow_examples:
+                outputs.append(self._use_example(messages))
+            elif isinstance(self.answers, dict):
+                outputs.append(
+                    next(
+                        (format_fields(v) for k, v in self.answers.items() if k in messages[-1]["content"]),
+                        "No more responses",
+                    )
+                )
+            else:
+                outputs.append(format_fields(next(self.answers, {"answer": "No more responses"})))
+
+            # Logging, with removed api key & where `cost` is None on cache hit.
+            kwargs = {k: v for k, v in kwargs.items() if not k.startswith("api_")}
+            entry = dict(prompt=prompt, messages=messages, kwargs=kwargs)
+            entry = dict(**entry, outputs=outputs, usage=0)
+            entry = dict(**entry, cost=0)
+            self.history.append(entry)
+
+        return outputs
+
+    def get_convo(self, index):
+        """Get the prompt + anwer from the ith message."""
+        return self.history[index]["messages"], self.history[index]["outputs"]
+
+
 def dummy_rm(passages=()) -> callable:
     if not passages:
 
diff --git a/examples/coding/hackercup_utils.py b/examples/coding/hackercup_utils.py
index ec443dfd7..66d00a04e 100644
--- a/examples/coding/hackercup_utils.py
+++ b/examples/coding/hackercup_utils.py
@@ -3,6 +3,9 @@
 import multiprocessing
 import concurrent.futures
 from typing import Optional
+import os
+import sys
+import traceback
 
 """
 Note that this code is largely based off of the code here:
@@ -33,21 +36,24 @@ def extract_code(code_str):
     return code.strip()
 
 
-import traceback
 
 
 def run_with_timeout(code: str, input: Optional[str], timeout: int):
     def target_fn(input, return_dict):
         vars = {}
         try:
+            # Redirect stdout to silence print statements
+            sys.stdout = open(os.devnull, 'w')
             exec(code, vars)
             fn = vars.get("solve", lambda x: x)
             return_dict["result"] = fn(input)
         except Exception as e:
             return_dict["error"] = str(e)
-            return_dict["stack_trace"] = (
-                traceback.format_exc()
-            )  # Capture the stack trace
+            return_dict["stack_trace"] = traceback.format_exc()
+        finally:
+            # Restore stdout to its original setting
+            sys.stdout.close()
+            sys.stdout = sys.__stdout__
 
     manager = multiprocessing.Manager()
     return_dict = manager.dict()
@@ -75,7 +81,6 @@ def target_fn(input, return_dict):
 
     return {"result": return_dict.get("result"), "error": None, "stack_trace": None}
 
-
 async def arun(
     code: Optional[str] = None, input: Optional[str] = None, timeout: int = 60
 ):
diff --git a/examples/qa/hotpot/hotpotqa_with_assertions.ipynb b/examples/qa/hotpot/hotpotqa_with_assertions.ipynb
index 8bf62b6c4..277152606 100644
--- a/examples/qa/hotpot/hotpotqa_with_assertions.ipynb
+++ b/examples/qa/hotpot/hotpotqa_with_assertions.ipynb
@@ -193,12 +193,14 @@
     "            dspy.Suggest(\n",
     "                len(query) <= 100,\n",
     "                \"Query should be short and less than 100 characters\",\n",
+    "                target_module=self.generate_query\n",
     "            )\n",
     "\n",
     "            dspy.Suggest(\n",
     "                validate_query_distinction_local(prev_queries, query),\n",
     "                \"Query should be distinct from: \"\n",
     "                + \"; \".join(f\"{i+1}) {q}\" for i, q in enumerate(prev_queries)),\n",
+    "                target_module=self.generate_query\n",
     "            )\n",
     "\n",
     "            prev_queries.append(query)\n",
diff --git a/pyproject.toml b/pyproject.toml
index b849eccca..b7df588bc 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 #replace_package_name_marker
 name="dspy-ai"
 #replace_package_version_marker
-version="2.5.3"
+version="2.5.5"
 description = "DSPy"
 readme = "README.md"
 authors = [{ name = "Omar Khattab", email = "okhattab@stanford.edu" }]
diff --git a/requirements.txt b/requirements.txt
index 29f419764..f4b28d788 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,3 @@
-# DSPy requirements
 backoff
 datasets
 joblib~=1.3
diff --git a/setup.py b/setup.py
index d3c26108f..c3dac8fc2 100644
--- a/setup.py
+++ b/setup.py
@@ -10,9 +10,9 @@
 
 setup(	
     #replace_package_name_marker
-    name="dspy-ai",
+    name="dspy",
     #replace_package_version_marker
-    version="2.5.3", 	
+    version="2.5.5", 	
     description="DSPy",	
     long_description=long_description,	
     long_description_content_type="text/markdown",	
@@ -39,7 +39,8 @@
         "snowflake": ["snowflake-snowpark-python"],
         "fastembed": ["fastembed"],
         "groq": ["groq~=0.8.0"],
-        "langfuse": ["langfuse~=2.36.1"]
+        "langfuse": ["langfuse~=2.36.1"],
+        "pgvector": ["psycopg2~=2.9.9","pgvector~=0.2.5"]
     },
     classifiers=[	
         "Development Status :: 3 - Alpha",	


Test patch:
diff --git a/examples/finetuning/finetune_test.ipynb b/examples/finetuning/finetune_test.ipynb
index f74fd4a91..71c43bbcf 100644
--- a/examples/finetuning/finetune_test.ipynb
+++ b/examples/finetuning/finetune_test.ipynb
@@ -18,7 +18,6 @@
     "%load_ext autoreload\n",
     "%autoreload 2\n",
     "\n",
-    "import tqdm\n",
     "import os\n",
     "\n",
     "os.environ[\"DSPY_CACHEDIR\"] = \"\"\n",
diff --git a/testing/README.md b/testing/README.md
index 95505dfeb..6fc730072 100644
--- a/testing/README.md
+++ b/testing/README.md
@@ -7,7 +7,7 @@ Optimizer Tester is intended to allow simple and repeatable testing of DSPy Opti
 To use the Optimizer Tester in code instantiate an OptimizerTester object:
 
 ```python
-from testing.optimizer_tester import OptimizerTester
+from optimizer_tester import OptimizerTester
 
 tester = OptimizerTester()
 ```
@@ -17,8 +17,8 @@ The default verison (no parameters) expects a llama model hosted on ports [7140,
 If you prefer to specify your own model parameters then you can pass models into the OptimizerTester
 
 ```python
-task_model = dspy.HFClientTGI(...)
-prompt_model = dspy.OpenAI(...)
+task_model = dspy.LM(...)
+prompt_model = dspy.LM(...)
 
 tester = OptimizerTester(task_model=task_model, prompt_model=prompt_model)
 ```
@@ -26,7 +26,7 @@ tester = OptimizerTester(task_model=task_model, prompt_model=prompt_model)
 If you just want to get baseline results for a particular task you're ready to go!
 
 ```python
-tester.test_baseline(datasets=["HotPotQA", "GSM8K", "Scone"])
+tester.test_baseline(datasets=["hotpotqa", "gsm8k", "scone"])
 ```
 
 If you want to test out a custom optimizer you'll have to write a quick function to call it properly:
diff --git a/testing/__init__.py b/testing/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/testing/datasets/hotpotqa_conditional/hotpot_dev.csv b/testing/datasets/hotpotqa_conditional/hotpot_dev.csv
new file mode 100644
index 000000000..f1c6665fa
--- /dev/null
+++ b/testing/datasets/hotpotqa_conditional/hotpot_dev.csv
@@ -0,0 +1,201 @@
+question,answer,answer category,question word,topic,,,,,,,,,,,,,,,,,,,,,
+Are both Cangzhou and Qionghai in the Hebei province of China?,no,boolean,are,geography,,,,,,,,,,,,,,,,,,,,,
+Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?,National Hockey League,organization,who,sports,,,,,,,,,,,,,,,,,,,,,
+"The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay Lightning of the National Hockey League (NHL)?",Steve Yzerman,person,which,sports,,,,,,,,,,,,,,,,,,,,,
+What river is near the Crichton Collegiate Church?,the River Tyne,location,what,geography,,,,,,,,,,,,,,,,,,,,,
+In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?,King Alfred the Great,person,which,history,,,,,,,,,,,,,,,,,,,,,
+The Newark Airport Exchange is at the northern edge of an airport that is operated by whom?,Port Authority of New York and New Jersey,organization,whom,geography,,,,,,,,,,,,,,,,,,,,,
+Where did an event take place resulting in a win during a domestic double due to the action of a Peruvian footballer known for his goal scoring ability?,Bundesliga,organization,where,sports,,,,,,,,,,,,,,,,,,,,,
+Are both Chico Municipal Airport and William R. Fairchild International Airport in California?,no,boolean,are,geography,,,,,,,,,,,,,,,,,,,,,
+In which Maine county is Fort Pownall located?,"Waldo County, Maine",location,which,geography,,,,,,,,,,,,,,,,,,,,,
+"Which 90s rock band has more recently reformed, Gene or The Afghan Whigs?",The Afghan Whigs,band,which,entertainment,,,,,,,,,,,,,,,,,,,,,
+"What year did the mountain known in Italian as ""Monte Vesuvio"", erupt?",79 AD,date,what,history,,,,,,,,,,,,,,,,,,,,,
+Is the 72nd field brigade part of the oldest or newest established field army?,the oldest,other,is,history,,,,,,,,,,,,,,,,,,,,,
+Was Stanislaw Kiszka paid for his services by the Royal Treasury?,not,other,was,history,,,,,,,,,,,,,,,,,,,,,
+"Which film director is younger, Del Lord or Wang Xiaoshuai?",Del Lord,person,which,entertainment,,,,,,,,,,,,,,,,,,,,,
+Lord North Street has a resident in which former Conservative MP who received an 18-month prison sentence for perjury in 1999?,Jonathan William Patrick Aitken,person,which,politics,,,,,,,,,,,,,,,,,,,,,
+"What is the name of this region of Italy, referring to the medieval March of Ancona and nearby marches of Camerino and Fermo, where the comune Pollenza is located?",Marche,location,what,geography,,,,,,,,,,,,,,,,,,,,,
+William Hughes Miller was born in a city with how many inhabitants ?,"7,402 at the 2010 census",number,how,demographics,,,,,,,,,,,,,,,,,,,,,
+What do students do at the school of New York University where Meleko Mokgosi is an artist and assistant professor?,design their own interdisciplinary program,activity,what,education,,,,,,,,,,,,,,,,,,,,,
+What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?,English,nationality,what,entertainment,,,,,,,,,,,,,,,,,,,,,
+What American actor plays an East side drug lord who prefers peaceful solutions to business disputes when possible?,Robert F. Chew,person,what,entertainment,,,,,,,,,,,,,,,,,,,,,
+What city is 11 miles north of the birthplace of actor Toby Sawyer?,Manchester,location,what,geography,,,,,,,,,,,,,,,,,,,,,
+"Who was born first, Tony Kaye or Deepa Mehta?",Deepa Mehta,person,who,people,,,,,,,,,,,,,,,,,,,,,
+What is the English translation of the name of the store that Macy's replaced in Boise Town Square?,the good market,translation,what,retail,,,,,,,,,,,,,,,,,,,,,
+Who did Lizzette Reynolds fire to make her notable in November 2007?,Christine Comer,person,who,people,,,,,,,,,,,,,,,,,,,,,
+"What was the name of the man, who was billed by the coiner of the phrase ""There's a sucker born every minute"" as ""Boy Lightning Calculator""?",William Street Hutchings,person,what,history,,,,,,,,,,,,,,,,,,,,,
+"Which battle was fought for a shorter period of time, the Battle of the Ch'ongch'on River, or the Meuse-Argonne Offensive?",Battle of the Ch'ongch'on River,event,which,history,,,,,,,,,,,,,,,,,,,,,
+What cricketeer active 1974–1993 had a strong performance in the tour by the Australian cricket team in England in 1981?,Ian Botham,person,what,sports,,,,,,,,,,,,,,,,,,,,,
+What is the present post of the head coach of the 1982 NC State Wolfpack football team ?,defensive assistant at Florida Atlantic,position,what,sports,,,,,,,,,,,,,,,,,,,,,
+"Which Scottish actor sang ""Come What May""?",Ewan McGregor,person,which,entertainment,,,,,,,,,,,,,,,,,,,,,
+Where have Ivan Bella and Frank De Winne both traveled?,space,location,where,space,,,,,,,,,,,,,,,,,,,,,
+"The original work by Anton Chekhov involving a disillusioned schoolmaster, which inspired a later play by this British playwright, was written specifically for whom?",Maria Yermolova,person,whom,literature,,,,,,,,,,,,,,,,,,,,,
+Are Roswell International Air Center and Pago Pago International Airport both located in the mainland US?,no,boolean,are,geography,,,,,,,,,,,,,,,,,,,,,
+Untold: The Greatest Sports Stories Never Told was hosted by a sportscaster commonly referred to as what ?,the voice of basketball,nickname,what,sports,,,,,,,,,,,,,,,,,,,,,
+Are Walt Disney and Sacro GRA both documentry films?,yes,boolean,are,film,,,,,,,,,,,,,,,,,,,,,
+What is the Palestinian Islamic organization that governs th small territory on the eastern coast of the Mediterranean Sea that was captured by Israel during the 1967 Six-Day War?,Hamas,organization,what,politics,,,,,,,,,,,,,,,,,,,,,
+What album did the song of which Taylor Swift premiered the music video of during the pre-show of the 2015 MTV Video Music Awards come from?,1989,album,what,music,,,,,,,,,,,,,,,,,,,,,
+"Which is considered a genus level classification, Apera or Gunnera manicata?",Apera,genus,which,biology,,,,,,,,,,,,,,,,,,,,,
+Do The Drums and Pussy Galore play music of similar genres?,no,boolean,do,music,,,,,,,,,,,,,,,,,,,,,
+What is the post-nominal abbreviation for the university where the Banded Mongoose Research Project is based?,Exon,abbreviation,what,education,,,,,,,,,,,,,,,,,,,,,
+Are both Benjamin Christensen and Len Wiseman directors?,yes,boolean,are,film,,,,,,,,,,,,,,,,,question,answer,answer category,question word,topic
+Steven Cuitlahuac Melendez and Disney are connected by what American animator?,Bill Melendez,person,what,animation,,,,,,,,,,,,,,,,,Where have Ivan Bella and Frank De Winne both traveled?,space,location,where,space
+Shark Creek is located on this river which is in the northern rivers district?,Clarence River,location,what,geography,,,,,,,,,,,,,,,,,"The original work by Anton Chekhov involving a disillusioned schoolmaster, which inspired a later play by this British playwright, was written specifically for whom?",Maria Yermolova,person,whom,literature
+Who was the producer of the 2016 animated film about an amnesiac fish?,Pixar,organization,who,film,,,,,,,,,,,,,,,,,Are Roswell International Air Center and Pago Pago International Airport both located in the mainland US?,no,boolean,are,geography
+Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?,Renault,organization,who,sports,,,,,,,,,,,,,,,,,Untold: The Greatest Sports Stories Never Told was hosted by a sportscaster commonly referred to as what?,the voice of basketball,nickname,what,sports
+"Fredrick Law Olmsted was an American landscape architect, journalist, social critic and public administrator that designed what neighborhood in Trenton, New Jersey?",Cadwalader Heights,location,what,architecture,,,,,,,,,,,,,,,,,Are Walt Disney and Sacro GRA both documentary films?,yes,boolean,are,film
+Gordon Warnecke worked alongside the former senator for which political party on Young Toscanini?,"""Forza Italia"" party.",organization,which,politics,,,,,,,,,,,,,,,,,What is the Palestinian Islamic organization that governs the small territory on the eastern coast of the Mediterranean Sea that was captured by Israel during the 1967 Six-Day War?,Hamas,organization,what,politics
+André Zucca was a French photographer who worked with a German propaganda magazine published by what Nazi organization?,the Wehrmacht,organization,what,history,,,,,,,,,,,,,,,,,What album did the song of which Taylor Swift premiered the music video of during the pre-show of the 2015 MTV Video Music Awards come from?,1989,album,what,music
+Both Bill Ponsford and Bill Woodfull played what?,cricketer,profession,what,sports,,,,,,,,,,,,,,,,,"Which is considered a genus level classification, Apera or Gunnera manicata?",Apera,genus,which,biology
+Suzana S. Drobnjaković Ponti acted in a film loosely based on a book by who?,Danny Wallace,person,who,literature,,,,,,,,,,,,,,,,,Do The Drums and Pussy Galore play music of similar genres?,no,boolean,do,music
+In what city was the Election Law Journal founded?,Portland,location,what,law,,,,,,,,,,,,,,,,,What is the post-nominal abbreviation for the university where the Banded Mongoose Research Project is based?,Exon,abbreviation,what,education
+Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?,Brian Virtue,person,who,music,,,,,,,,,,,,,,,,,Are both Benjamin Christensen and Len Wiseman directors?,yes,boolean,are,film
+Are both the University of Chicago and Syracuse University public universities?,no,boolean,are,education,,,,,,,,,,,,,,,,,Steven Cuitlahuac Melendez and Disney are connected by what American animator?,Bill Melendez,person,what,animation
+In what region was Eddy Mazzoleni born?,"Lombardy, northern Italy",location,what,geography,,,,,,,,,,,,,,,,,Shark Creek is located on this river which is in the northern rivers district?,Clarence River,location,what,geography
+Who edited the 1990 American romantic comedy film directed by Garry Marshall?,Raja Raymond Gosnell,person,who,film,,,,,,,,,,,,,,,,,Who was the producer of the 2016 animated film about an amnesiac fish?,Pixar,organization,who,film
+Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall,seventh,position,what,transportation,,,,,,,,,,,,,,,,,Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?,Renault,organization,who,sports
+"At which university did this American writer, lawyer, actor, and commentator on political and economic issues, who appeared in the 1993 comedy-drama film ""Me and the Kid"" get a law degree?",Yale Law School,organization,which,education,,,,,,,,,,,,,,,,,,,,,
+The first person to interpret Relational quantum mechanics has worked in what countries?,"Italy, the United States and since 2000, in France",location,what,science,,,,,,,,,,,,,,,,,,,,,
+"How many seasons did, the Guard with a FG% around .420, play in the NBA ?",14 seasons,number,how,sports,,,,,,,,,,,,,,,,,,,,,
+Jorge Miramón plays for the team that is in what autonomous country?,Catalonia,location,what,sports,,,,,,,,,,,,,,,,,,,,,
+Which Big East champion began his NBA career with the Celtics in 2007?,Jeff Green,person,which,sports,,,,,,,,,,,,,,,,,,,,,
+What genre do The Gaslight Anthem and Seaweed share?,rock,genre,what,music,,,,,,,,,,,,,,,,,,,,,
+What was the 2010 population of the birthplace of Gerard Piel?,"17,121",number,what,demographics,,,,,,,,,,,,,,,,,,,,,
+"Folk dances from Thessaly include this popular dance, which is a mixture of the slow and fast rhythms of hasapiko and which other dance?",hasaposerviko,dance,which,dance,,,,,,,,,,,,,,,,,,,,,
+Elisabeth Hasselbeck the former host of The Look for Less graduated from what College?,Boston College,organization,what,education,,,,,,,,,,,,,,,,,,,,,
+Which has more species Nothoscordum or Calypso bulbosa?,Nothoscordum,species,which,biology,,,,,,,,,,,,,,,,,,,,,
+Do both Central Illinois Regional Airport and Idaho Falls Regional Airport serve two cities each?,no,boolean,do,aviation,,,,,,,,,,,,,,,,,,,,,
+How much is spent on the type of whiskey that 1792 Whiskey is in the United States?,about $2.7 billion,amount,how,economics,,,,,,,,,,,,,,,,,,,,,
+"Rex Reed appeared in the film Myra Breckinridge, in addition to being known as what?",an American film critic,profession,what,film,,,,,,,,,,,,,,,,,,,,,
+What is being built both at 3 Hudson Boulevard and 520 Park Avenue?,an American skyscraper under construction,thing,what,architecture,,,,,,,,,,,,,,,,,,,,,
+"Who has published more works, Melissa Bank  or Alan Moore?",Alan Moore,person,who,literature,,,,,,,,,,,,,,,,,,,,,
+"What footballer is a striker for İstanbul Başakşehir and endorsed the song ""Skelewu?""",Sheyi Emmanuel Adebayor,person,what,sports,,,,,,,,,,,,,,,,,,,,,
+What country does Lex Gabinia and Pompey have in common?,Rome,location,what,history,,,,,,,,,,,,,,,,,,,,,
+What was the full name of the author that memorialized Susan Bertie through her single volume of poems?,Emilia Lanier,person,what,literature,,,,,,,,,,,,,,,,,,,,,
+George Ho Ho-chi's father was known as what?,the grand old man of Hong Kong,nickname,what,nickname,,,,,,,,,,,,,,,,,,,,,
+"Which of these is in the Apiaceae family, Ostrowskia or Selinum?",Selinum,species,which,biology,,,,,,,,,,,,,,,,,,,,,
+For what contribution did Simon van der Meer won the that is given by he Royal Swedish Academy of Sciences?,CERN,organization,what,science,,,,,,,,,,,,,,,,,,,,,
+"Which character, an elder sister, was played by Melissa Sue Anderson in the NBC series ""Little House on the Prairie""?",Mary Ingalls,character,which,television,,,,,,,,,,,,,,,,,,,,,
+Juwan Howard was part of the Michigan team that had four participants in what 1991 exhibition?,1991 McDonald's All-American Game,event,what,sports,,,,,,,,,,,,,,,,,,,,,
+"In 2007, which network aired the eleventh season of the animated television series created by Trey Parker and Matt Stone?",Comedy Central,network,which,television,,,,,,,,,,,,,,,,,,,,,
+The coach of the 2017 Houston Texans coached which team from 2012 to 2013?,Penn State,organization,which,sports,,,,,,,,,,,,,,,,,,,,,
+"The Black Halo features a guest appearance by a Dutch symphonic metal band, that was founded by who?",Mark Jansen,person,who,music,,,,,,,,,,,,,,,,,,,,,
+The score for Frankie and Johnny was composed by a composer who has won what collection of four awards?,"""EGOT""",award,what,entertainment,,,,,,,,,,,,,,,,,,,,,
+Billy Collins teaches at a campus in which city and state?,"Southampton, New York",location,which,education,,,,,,,,,,,,,,,,,,,,,
+The character Beatrice Prior stars in how many movies?,trilogy,number,how,film,,,,,,,,,,,,,,,,,,,,,
+The series Bottersnikes and Gumbles includes a castmember who won a BAFTA Award for her role in which film?,The Age of Innocence,film,which,film,,,,,,,,,,,,,,,,,,,,,
+What division I-AA university football team represented the Southern Conference in the 1924 season?,South Carolina Gamecocks,organization,what,sports,,,,,,,,,,,,,,,,,,,,,
+The Chinese film 20 Once Again is a remake of which 2014 South Korean comedy?,Miss Granny,film,which,film,,,,,,,,,,,,,,,,,,,,,
+Who directed the film for which Danny Michael was nominated for an Academy Award ?,"Alan Parker,",person,who,film,,,,,,,,,,,,,,,,,,,,,
+The LIU Post Pioneers have a football team that plays in which Division II conference?,Northeast-10,organization,which,sports,,,,,,,,,,,,,,,,,,,,,
+The Neoclassical monument Benito Juárez Hemicycle is located in this city of Mexico that consists of how many municipalities?,sixteen,number,how,geography,,,,,,,,,,,,,,,,,,,,,
+"The score for ""To Gillian on Her 37th Birthday"" was composed by James Horner, known for his frequent motifs associated with what kind of traditional music?",Celtic music,music genre,what,music,,,,,,,,,,,,,,,,,,,,,
+What trade in common do Debarun Pal and Peter Duffell share?,Film Director,profession,what,film,,,,,,,,,,,,,,,,,,,,,
+"While kosher companies Streit's and Manischewitz are both famous matzo competitors, Manischewitz. is also famous for what other product?",kosher wine,product,what,food,,,,,,,,,,,,,,,,,,,,,
+Jordan Phillips played his college football games at the team that is a member of what conference?,Big 12,organization,what,sports,,,,,,,,,,,,,,,,,,,,,
+"When band has more members, Brand New or The Dodos?",Brand New,band,which,music,,,,,,,,,,,,,,,,,,,,,
+What position was the top seed in the 2010 China Open - Women's Singles in the Danish professional tennis scene?,top,position,what,sports,,,,,,,,,,,,,,,,,,,,,
+"Who died first, Brian O'Nolan or Bernard Malamud?",Brian O'Nolan,person,who,literature,,,,,,,,,,,,,,,,,,,,,
+"An American actress, best known for her film choreography starred in a film that earned who a Razzie Award nomination for Worst New Star?",Robby Rosa,person,who,film,,,,,,,,,,,,,,,,,,,,,
+Reverend Gary Davis has many students including one who was the composer of Hey Looka Yonder. What is the students name?,Tom Winslow,person,what,music,,,,,,,,,,,,,,,,,,,,,
+Give a Girl a Break featured Marge Champion as what kind of talent?,dancer,talent,what,film,,,,,,,,,,,,,,,,,,,,,
+Who is noted for their work on the history of Egypt's largest Christian church?,Father Menassa Youhanna,person,who,religion,,,,,,,,,,,,,,,,,,,,,
+What novel did the American short story writer who also wrote Losing Battles wrote,American South,novel,what,literature,,,,,,,,,,,,,,,,,,,,,
+Estonian Philharmonic Chamber Choir won the grammy Award for Best Choral Performance for two songs by a composer born in what year ?,1935,year,what,music,,,,,,,,,,,,,,,,,,,,,
+What are both National Bird and America's Heart and Soul?,film,film,what,film,,,,,,,,,,,,,,,,,,,,,
+Bridge Day commemorates the 1977 completion of which single-span arch bridge?,New River Gorge Bridge,bridge,which,architecture,,,,,,,,,,,,,,,,,,,,,
+On what streets is the hospital that cared for Molly Meldrum located?,the corner of Commercial and Punt Roads,location,what,geography,,,,,,,,,,,,,,,,,,,,,
+What road is the entrance for the Naval Center named after Rear Admiral John A. Dahlgren located at?,State Route 206 (SR 206),road,what,geography,,,,,,,,,,,,,,,,,,,,,
+"Which co-founder of Instagram was born December 30, 1983?",Kevin Systrom,person,which,technology,,,,,,,,,,,,,,,,,,,,,
+What do both Lincoln Steffens and Tom Clancy have in common?,published,trait,what,literature,,,,,,,,,,,,,,,,,,,,,
+What industry were both Mike Nichols and Liam Lynch a part of?,film,industry,what,film,,,,,,,,,,,,,,,,,,,,,
+"Which game is the oldest, Ticket to Ride or King of Tokyo?",Ticket to Ride,game,which,games,,,,,,,,,,,,,,,,,,,,,
+Noah Galloway was a contestant  on the 20th season of a tv show that premiered on what date?,"March 16, 2015",date,what,television,,,,,,,,,,,,,,,,,,,,,
+"What was the nationality of John Wayne's co-star in ""Big Jim McLain""?",American,nationality,what,film,,,,,,,,,,,,,,,,,,,,,
+South Hadley High School became known for a series of bullying incidents that led to an event that led to the criminal prosecution of six teenagers for what charges?,civil rights violations,charges,what,law,,,,,,,,,,,,,,,,,,,,,
+"Which was around longer in 2010, Miss Universe or Miss Perú?",Miss Universe,pageant,which,pageants,,,,,,,,,,,,,,,,,,,,,
+Are Leontopodium and Sandersonia both native to Europe?,no,boolean,are,botany,,,,,,,,,,,,,,,,,,,,,
+"The musical, Newsies, starring Mike Faust as Morris Delancey , was produced by which company?",Disney Theatrical Productions,organization,which,theatre,,,,,,,,,,,,,,,,,,,,,
+George Hires represented the state that ranks what in regards to population figures?,11th-most populous,position,what,demographics,,,,,,,,,,,,,,,,,,,,,
+"What is an alternate name for a universal joint that is also the last name of an English natural philosopher, architect and polymath?",Hooke,thing,what,science,,,,,,,,,,,,,,,,,,,,,
+What is the birth date of this British actor and former professional footballer who starred alongside John Travolta in Swordfish?,5 January 1965,date,what,entertainment,,,,,,,,,,,,,,,,,,,,,
+"The Living Oracles is a translation of the New Testament created by Alexander Campbell, the early Restoration Movement leader who stressed as paramount the reliance on few essentials as well as what item?",scripture,item,what,religion,,,,,,,,,,,,,,,,,,,,,
+Jean-Pierre Kohut-Svelko is a production designer and art director and the winner of a national film award of Francewhich was first awarded in what year?,1976,year,what,film,,,,,,,,,,,,,,,,,,,,,
+What Greek rock performer was a member of The Dots and producer for American hardcore punk band Bad Brains?,Jimi Quidd,person,what,music,,,,,,,,,,,,,,,,,,,,,
+What kind of group does Go Radio and Mayday Parade have in common?,rock band,group,what,music,,,,,,,,,,,,,,,,,,,,,
+In which city is the conglomerate that bought Ray Ban from Bausch & Lomb ?,Milan,city,which,business,,,,,,,,,,,,,,,,,,,,,
+What company published the video game whose main protagonist is Altaïr Ibn-La'Ahad?,Ubisoft,company,what,gaming,,,,,,,,,,,,,,,,,,,,,
+"Which is found in Japan, Emmenopterys or Nipponanthemum?",Nipponanthemum,species,which,botany,,,,,,,,,,,,,,,,,,,,,
+Jim Challinor died in a town loated in what county?,Cheshire,county,what,geography,,,,,,,,,,,,,,,,,,,,,
+What weekly newsmagazine is published in the capital and largest city of Peru?,Lima,city,what,news,,,,,,,,,,,,,,,,,,,,,
+"Who is younger, Harry Daghlian or Zach Bogosian?",Zachary M. Bogosian,person,who,history,,,,,,,,,,,,,,,,,,,,,
+In which american sitcom with  247 episodes did Jeff Mooring appeared?,Jeff Mooring,person,which,television,,,,,,,,,,,,,,,,,,,,,
+"Which is published more frequently, The People's Friend or Bust?",The People's Friend,magazine,which,news,,,,,,,,,,,,,,,,,,,,,
+What river in south Central America saw a battle fought on its right bank on 19 December 1810,Paraná River,river,what,history,,,,,,,,,,,,,,,,,,,,,
+What is the birthdate of this American sports interviewer who works for ESPN and hosts Football Sunday on ESPN Radio?,"August 12, 1972",date,what,sports,,,,,,,,,,,,,,,,,,,,,
+"What was the name of the musical group with which the career of the singer of ""Shikva"" began?",Awaz,group,what,music,,,,,,,,,,,,,,,,,,,,,
+Where in the United States will you find the adult Anthophylax hoffmani?,Appalachian Mountains of the Southeastern United States,location,where,geography,,,,,,,,,,,,,,,,,,,,,
+Trent Cotchin was a player in the sport that plays on what shaped field?,oval-shaped,shape,what,sports,,,,,,,,,,,,,,,,,,,,,
+Which of the following novelists is known for writing about sexuality and emotional health: Gail Godwin or D. H. Lawrence?,David Herbert Lawrence,person,which,literature,,,,,,,,,,,,,,,,,,,,,
+When was the  Prime Minister of India after which V. K. Krishna Menon was regarded as the second most powerful man in India born?,14 November 1889,date,when,history,,,,,,,,,,,,,,,,,,,,,
+What is the release date of the album that was supported by the Synthesis Tour?,"November 10, 2017",date,what,music,,,,,,,,,,,,,,,,,,,,,
+Which occupations do Jim Sheridan and Ian Schneider both share?,director and screenwriter,profession,which,film,,,,,,,,,,,,,,,,,,,,,
+Hugo Adam Bedau used the example of an American domestic terrorist to support what political policy?,anti-death-penalty,policy,what,politics,,,,,,,,,,,,,,,,,,,,,
+The 1984 British Grand Prix was won by what driver who is the only one to have been a champion for both Ferrari and McLaren?,Niki Lauda,person,what,sports,,,,,,,,,,,,,,,,,,,,,
+"Which astronaut was born first, Christer Fuglesang or Philippe Perrin?",Christer Fuglesang,person,which,space,,,,,,,,,,,,,,,,,,,,,
+In what year did the team that built the Lotus E21 begin to compete under the Lotus name?,2012,year,what,sports,,,,,,,,,,,,,,,,,,,,,
+where is the mosque  where  De'Montmorency College of Dentistry: is  situated next to located,Lahore,location,where,geography,,,,,,,,,,,,,,,,,,,,,
+Gol Transportes Aéreos is the second largest Brazilian airline company by market share after the company that merged with LAN Airlines on which date?,"June 22, 2012",date,which,business,,,,,,,,,,,,,,,,,,,,,
+The home ground stadium for FC Copenhagen hosted a crowd of how many during the 2010 Danish Cup Final?,"18,856",number,how,sports,,,,,,,,,,,,,,,,,,,,,
+Torre Catalano and D. W. Griffith were both what?,"director, writer",profession,what,film,,,,,,,,,,,,,,,,,,,,,
+"Which film won more awards, Mary and Max or The Happy Cricket and the Giant Bugs? ",Mary and Max,film,which,film,,,,,,,,,,,,,,,,,,,,,
+Where in New York is the highway that ends at the toll barrier for Heckscher State Park in Great River?,"Long Island, New York",location,where,geography,,,,,,,,,,,,,,,,,,,,,
+Does Empire of the Sun or 3 Doors Down have more band members?,3 Doors Down,band,does,music,,,,,,,,,,,,,,,,,,,,,
+Elizabeth Louise Botting CBE worked for which company that operated the UK National Lottery?,Camelot Group,organization,which,business,,,,,,,,,,,,,,,,,,,,,
+The Chief Secretary to the Treasury is also a member of what political party?,British Conservative Party,political party,what,politics,,,,,,,,,,,,,,,,,,,,,
+What nationality is the founder of the group Reindeer Section?,Northern Irish,nationality,what,music,,,,,,,,,,,,,,,,,,,,,
+what govern the  public institutes that  IIT Council is the governing body for ?,"""Institutes of Technology Act, 1961",act,what,education,,,,,,,,,,,,,,,,,,,,,
+"Which American pizza chain has their headquarters in a more southerly state, Pietro's Pizza or Pizza Patrón?",Pizza Patrón Inc.,company,which,business,,,,,,,,,,,,,,,,,,,,,
+"Which documentary was released first, Grizzly Man or Best Boy?",Best Boy,film,which,film,,,,,,,,,,,,,,,,,,,,,
+"What is the birthdate of this American actor, director, screenwriter and producer best known for A Christmas Story and director of Black Christmas?","August 5, 1939",date,what,entertainment,,,,,,,,,,,,,,,,,,,,,
+"Which band is new, Before Their Eyes or Matthew Good Band?",Before Their Eyes,band,which,music,,,,,,,,,,,,,,,,,,,,,
+What do Léonce Perret and Larry Cohen have in common in the film industry?,"producer, director",profession,what,film,,,,,,,,,,,,,,,,,,,,,
+"The actor who played Chino Ho Kelly on the T.V. series ""Hawaii Five-O"" Starred along side his son that played which character?",Sgt. Duke Lukela,character,what,television,,,,,,,,,,,,,,,,,,,,,
+What teams did a Mount Olive native NFL quarterback play for?,Houston/Tennessee Oilers – Tennessee Titans and also played for the Baltimore Ravens,teams,what,sports,,,,,,,,,,,,,,,,,,,,,
+"Who was born on May 20, 1968 and portrays Givens on Justified?",Timothy Olyphant,person,who,television,,,,,,,,,,,,,,,,,,,,,
+Ronnie Dunn and Jenna Sanz-Agero were both formerly part of bands in which they each acted in what role?,singer,role,what,music,,,,,,,,,,,,,,,,,,,,,
+"Air, an album by Cecil Taylor, features which performer born in Idabel, Oklahoma?",Sunny Murray,person,which,music,,,,,,,,,,,,,,,,,,,,,
+What is the formal name of the party George Perry Mahoney was a candidate for?,The States' Rights Democratic Party,political party,what,politics,,,,,,,,,,,,,,,,,,,,,
+What sport is represented by Zuffa founded in 2001 by Frank Fertitta III and Lorenzo Fertitta?,mixed martial arts,sport,what,sports,,,,,,,,,,,,,,,,,,,,,
+"Where is the academy, from which the Democrat State Senator from Maine's 4th district from 2004 to 2012 graduated, located?","Saco, Maine",location,where,education,,,,,,,,,,,,,,,,,,,,,
+"Which of the sport analyst of The Experts Network is nicknamed  ""The Iron Man""?",Calvin Edwin Ripken Jr.,person,which,sports,,,,,,,,,,,,,,,,,,,,,
+Which band had more members The Fatima Mansions or  3OH!3 ?,The Fatima Mansions,band,which,music,,,,,,,,,,,,,,,,,,,,,
+"What did Indonesian singer Indah Dewi Pertiwi sell in KFC outlets throughout the country, making it the fifth best seller in history?",her first album,product,what,music,,,,,,,,,,,,,,,,,,,,,
+The logarithmic spiral was investigated by the mathematician who was a proponent of which branch of mathematics?,Leibnizian calculus,branch,which,mathematics,,,,,,,,,,,,,,,,,,,,,
+What major feild do both The Western Institute of Technology and the Rose-Hulman Institute of Technology offer?,engineering,field,what,education,,,,,,,,,,,,,,,,,,,,,
+"""Miffy's Adventures Big and Small"" is based on the book series ""Miffy"", along with a continuation of a previous show that was a stop-motion animated television series, what type of animated show was ""Miffy's Adventures Big and Small""?",CGI-animated,type,what,television,,,,,,,,,,,,,,,,,,,,,
+The place where John Laub is an American criminologist and Distinguished University Professor in the Department of Criminology and Criminal Justice at was founded in what year?,1856,year,what,education,,,,,,,,,,,,,,,,,,,,,
+What kind of motorcycle club was informant Dany Kane a member of?,one-percenter motorcycle club,type,what,crime,,,,,,,,,,,,,,,,,,,,,
+The special division's constitutionality was upheld by which 1988 United States Federal Court Case?,"Morrison v. Olson, 487 U.S. 654 (1988)",court case,which,law,,,,,,,,,,,,,,,,,,,,,
+"Which grouo released the album, ""Pale Sun, Cresent Moon""?",Cowboy Junkies,group,which,music,,,,,,,,,,,,,,,,,,,,,
+In what location in Lincolnshire can you find the historic Jew's House?,Steep Hill is a popular tourist street in the historic city of Lincoln,location,what,geography,,,,,,,,,,,,,,,,,,,,,
+"What is the birthday of the band member of the group 58 who also formed Motley Crue, Sister, and Brides of Destruction?","December 11, 1958",date,what,music,,,,,,,,,,,,,,,,,,,,,
+"Which musician has been a member of more bands, Henry Paul or Jens Kidman?",Henry Paul,person,which,music,,,,,,,,,,,,,,,,,,,,,
+What hit film did Lee Eun-ju star in besides one about a police detective who investigates a murder case?,Taegukgi,film,what,film,,,,,,,,,,,,,,,,,,,,,
+When was the low-brow art magazine of which George Petros was a contributing editor of founded?,1994,year,when,art,,,,,,,,,,,,,,,,,,,,,
+"Who was featured as a guest contribution in ""Press Play"" and in a leaked sex tape with Rick Salomon?",Paris Hilton,person,who,entertainment,,,,,,,,,,,,,,,,,,,,,
+Which minor league team that plays its home games at Ogren Park did Tetsuya Yamaguchi play for before he was drafted by the Giants?,The Missoula Osprey,team,which,sports,,,,,,,,,,,,,,,,,,,,,
+Katz Editores is a scholarly publisher that has released articles by the winner of what 2000 award?,Nobel Prize in Physiology or Medicine,award,what,academia,,,,,,,,,,,,,,,,,,,,,
+Andrew Form produced which 2013 dystopian horror film?,The Purge,film,which,film,,,,,,,,,,,,,,,,,,,,,
+In what Wars did Gaius Cassius Longinus command troops?,Wars of the Second Triumvirate,wars,what,history,,,,,,,,,,,,,,,,,,,,,
+Are The Gaslight Anthem and Dinosaur Jr. both bands formed in the 20th century?,no,boolean,are,music,,,,,,,,,,,,,,,,,,,,,
+Which two artists are also featured in the Jay-Z song in which Jeff Bhasker won a Grammy Award for producing?,Kanye West and Rihanna,person,which,music,,,,,,,,,,,,,,,,,,,,,
+Robert Case is best known for inspiring the naming of what nor'easter that absorbed Hurricane Grace?,1991 Perfect Storm,storm,what,weather,,,,,,,,,,,,,,,,,,,,,
+"Which Norwegian black metal band unleashed an album called Masterpiss Of Pain, in 2001 on Satyr's Moonfog label",Khold,band,which,music,,,,,,,,,,,,,,,,,,,,,
+Who wrote the introduction to the book Cleaver wrote in 1968?,Maxwell Geismar,person,who,literature,,,,,,,,,,,,,,,,,,,,,
+"Which actor, born in 1926, appeared in the film Sandokan the Great?",Steve Reeves,person,which,film,,,,,,,,,,,,,,,,,,,,,
+BLIZZARD! The Storm That Changed America is a book about the storm that was referred to by what name?,Great White Hurricane,storm,what,history,,,,,,,,,,,,,,,,,,,,,
+What show is an American-Canadian drama starring Scott Lowell playing Ted Schmidt?,Queer as Folk,television show,what,television,,,,,,,,,,,,,,,,,,,,,
+Focke-Wulf Fw 187 was an intermediate design above the model that was the backbone of which fighting force?,Luftwaffe,fighting force,which,military,,,,,,,,,,,,,,,,,,,,,
+Ivan Matias produced hit songs for an American R&B and soul singer-songwriter who was born on March 16th in what year?,1976,year,what,music,,,,,,,,,,,,,,,,,,,,,
+Which maximum security jail housed the killer of Julissa brisman?,Suffolk County Jail,jail,which,crime,,,,,,,,,,,,,,,,,,,,,
\ No newline at end of file
diff --git a/testing/datasets/hotpotqa_conditional/hotpot_test.csv b/testing/datasets/hotpotqa_conditional/hotpot_test.csv
new file mode 100644
index 000000000..ee69346d4
--- /dev/null
+++ b/testing/datasets/hotpotqa_conditional/hotpot_test.csv
@@ -0,0 +1,201 @@
+question,answer,answer category,question word,topic
+What relation is Anakin Skywalker to Princess Leia?,Vader's daughter,relation,what,film
+"In what year did the brother-in-law of Ezzo, Count Palatine of Lotharingia, die?",1002,year,what,history
+Which Football Club is based in the capital of Lesotho?,Lerotholi Polytechnic Football Club,club,which,sports
+What cuisine is described as a cultural blending of Mediterranean influences (such as those created by Italian and Spanish populations)?,Argentine cuisine,cuisine,what,food
+What stadium does the team who has Kent Baer as the defensive coordinator play its home games?,Sam Boyd Stadium,stadium,what,sports
+Which documentary film was released first Tar Creek or Volcanic Sprint?,Volcanic Sprint,film,which,film
+The design team for role playing game Black Crusade featured the work of this designer who was the lead developer for what game?,Dark Heresy,game,what,gaming
+Jamie Harris is best known for his role as The Hook-Handed Man in a movie directed by who ?,Brad Silberling,person,who,film
+"What Box Codax group musician was also the guitarist, backing/lead vocalist and keyboardist of another Glasgow-based band?","Nicholas John ""Nick"" McCarthy",person,what,music
+"Out of two American colonies that had a series of skirmishes and raids between 1701 and 1765 at the disputed border, which British proprietary colony became a royal colony on the northeast coast of North America?",Province of New York,colony,which,history
+Are Portuguese Podengo and Russo-European Laika the same breed of dog?,no,boolean,are,animals
+Where does Yannick Ferreira Carrasco play home games?,Wanda Metropolitano,location,where,sports
+"Which Walt Disney film was released earlier, The Rescuers or The Muppets?",The Muppets,film,which,film
+Who has been making music long Eric Gaffney or Pearl Jam?,Eric Gaffney,person,who,music
+Kang Sung-yeon played Prince Yeonsan in a film that runs for how many minutes ?,119 minutes,duration,how,film
+Rafe Hernandez is a fictional character who played with Chrishell Stause's charachter on Days of our live?,Jordan Ridgeway,character,who,television
+"St Mary's School in Pune, India was run for over 100 years by what organization started by the Reverend William John Butler?",The Community of St Mary the Virgin (CSMV),organization,what,education
+What prominent Soviet director worked frequently with Composer Isaak Dunayevsky?,Grigori Vasilyevich Aleksandrov,person,what,film
+Ben Folds and Nic Offer are both considered to be which type of artists?,musician,profession,which,music
+What do Park So-yeon and Michael Crafter have in common?,vocalist,profession,what,music
+In what corner of Missouri is radio station WMBH located?,southwestern,location,what,geography
+"Are both ""Cooking Light"" and ""Vibe"" magazines?",yes,boolean,are,media
+What association did the woman whose museum is located in the Bolling–Gatewood House help found?,NAACP,association,what,history
+Where did The Cure record their highest charting single?,Hookend Recording Studios,location,where,music
+To what sport does the Atomweight class belong?,mixed martial arts,sport,what,sports
+Where did this dish emerge in the late 1950s that is a long-standing tradition in New Jersey?,Centre-du-Québec,location,where,food
+What was the the origin of the player that the Bull's traded with 76ers during the 1999-2000 midseason?,Croatian,nationality,what,sports
+Which actor/actress from Overnight Delivery was born in New Orleans?,Reese Witherspoon,person,which,film
+"What English-American actor known for his role in ""Raiders of the Lost Ark"" starred in The Lodger?","Alfredo ""Alfred"" Molina",person,what,film
+"Which film, The Happiest Millionaire or Mars Needs Moms, was created first?",The Happiest Millionaire,film,which,film
+" What is the latest album of Tim Hecker, the Canadian electronic music maker?",Love Streams,album,what,music
+Genealogy is an Armenian supergroup that competed in a 2015 singing contest in what city?,"Vienna, Austria",location,what,music
+"Was Rudolf Schwarzkogler closely associated with the Viennese Actionism group, while Howard Zieff was an American television commercial director?",yes,boolean,was,art
+Who is the current governor of the state where former wrestler Stephen Cepello painted the Governor's Mansion?,Mark Dayton,person,who,politics
+The Pioneer Helmet and the Benty Grange helmet are linked because of which feature?,A simple iron boar crest,feature,which,history
+What is the legal name of Johnny Campbell's hometown?,Londonderry,location,what,geography
+Orlando Maturana had to compete with what other Colombian retired footballer for the national football team's striker position?,Adolfo José Valencia Mosquera,person,what,sports
+"""The Simpson's"" episode featuring the  ""Itchy & Scratchy & Poochie"" show satirized the world of television production just as the episode featuring the ""Itchy & Scratchy & Marge"" satirized what type of issue?",censorship,issue,what,television
+What game did the team with an 11-1 regular season record play in for the BCS title game?,2004 Nokia Sugar Bowl,game,what,sports
+What South Australian capital held UFC Fight Night 65 in May of 2015?,Adelaide,city,what,sports
+What is the name of the father of Childericus?,Merovech,person,what,history
+"Not Richard, But Dick was an album by the band formed in what city?",Philadelphia,city,what,music
+Jean-Paul Sartre or George Bernard Shaw  have more influence on turn of the century literature?,George Bernard Shaw,person,who,literature
+The Music Hall Guild of Great Britain and America includes a patron in which actress who had a role on EastEnders?,Anita Dobson,person,which,television
+When was the American economist to which Joseph A. Loftus worked as a communications assistant born?,"December 13, 1920",date,when,history
+"What film genre involves humor and love, as seen in Mamma Mia: Here We Go Again!?",Romantic comedy films,genre,what,film
+Which MTV show was hosted by a member of the Likwit Crew?,Pimp My Ride,television show,which,television
+"In what year did the manager of the singer-songwriter best known for ""American Pie"" die?",2005,year,what,music
+"What South Korean actor was a member of the pop duo TVXQ and stared in Heading to the Ground, a 2009 MBC television series?",Jung Yun-ho,person,what,television
+Were the films Tonka and 101 Dalmatians released in the same decade?,1958 Walt Disney Western adventure film,decade,were,film
+Who was a supporting cast member of King Rat and was knighted in February 2001?,Sir Thomas Daniel Courtenay,person,who,film
+Are Grant Nicholas and Danny Shirley both American singers?,no,boolean,are,music
+Clifford Odets was seen as a successor to a Nobel Prize winner known as what?,American playwright and Nobel laureate in Literature,title,what,literature
+What is the name of the city located near the historical region known as Margianḗ in Greek?,Mary,city,what,geography
+"Les Ottolenghi was previously the Chief Information Officer at what casino in Paradise, Nevada?",Las Vegas Sands,company,what,business
+"In which year was the author born that wrote about  the former succubus Jezebel, now the mortal Jesse Harris, as she tries to avoid her Hellish past and learn how to be truly human?",1970,year,which,literature
+Reima Karppinen won a silver medal in the double sculls at the Rowing Championships held in what German city?,Oberschleißheim,city,what,sports
+"What landmark event did Bill Miner achieve at Silverdale, British Columbia, coining the phrase ""Hands Up!"" in doing so?",Canada's first train robbery,event,what,history
+What nationality were social anthropologists Alfred Gell and Edmund Leach?,British,nationality,what,anthropology
+"Which is located in Cornwall, Southwell Minster or Truro Cathedral?","The Cathedral of the Blessed Virgin Mary, Truro",building,which,geography
+are Tinto Brass and Baltasar Kormákur both from the same country ?,no,boolean,are,film
+"The pseudonymous partner to ""Willis the Guard"" in the American musical duo of Jerry Buckner and Gary Garcia is noun which refers to which portion of a bet?",amount charged by a bookmaker,term,which,music
+Which professional baseball player was born in 1984 and played as a rookie for the Los Angeles Dogers in 2007?,Matthew Ryan Kemp,person,which,sports
+What was the nickname of the English monarch that Petruccio Ubaldini presented one of his books?,The Virgin Queen,nickname,what,history
+In which city is a football club that uses Gabriel Barbosa?,Milan,city,which,sports
+"Who was older when they died, Teinosuke Kinugasa or Don O. Newland",Teinosuke Kinugasa,person,who,history
+What kind of result does 2013 Ghana Movie Awards and A Northern Affair have in common?,winners,result,what,film
+"The song ""Only a Pawn in Their Game"" was based on a crime whose perpetrator was convicted in this year.",1994,year,what,music
+Blank Check and Air Bud are both examples of what type of movie?,American comedy,genre,what,film
+Who was one of the stars who played the two oldest children in a TV series which had a 2010 movie based on it?,Michael Seater,person,who,television
+What president was assassinated and had Worthy Stevens Streator as one of his pallbearers?,James Abram Garfield,person,what,history
+J. Searle Dawley and Ken Annakin were both directors of what?,"Kenneth Cooper ""Ken"" Annakin, OBE (10 August 1914 – 22 April 2009) was a prolific English film director.",person,what,film
+Andrew Jaspan was the co-founder of what not-for-profit media outlet?,The Conversation,organization,what,media
+Which politician represents a city on the Flathead Indian Reservation?,Gregory J. Hertz,person,which,politics
+Spider9 was founded in 2011 by the head of which subsidiary of Wanxiang Group?,"A123 Systems, LLC",company,which,business
+What is the middle name of the actress who plays Bobbi Bacha in Suburban Madness?,Ann,name,what,film
+Which Prince of Bismarcks EMS dispatch incited France to declare the Franco-Prussian War in July 1870,Otto Eduard Leopold,person,which,history
+During which months of the year does the goalkeeper for Bengaluru FC play?,November to March,months,during,sports
+Easter Airways' head office is located at which airport owned by Manchester Airports Group?,Humberside Airport,location,which,business
+Stephen Sondheim and Thomas Z. Shepard both worked in the production of what genre of music?,musicals,genre,what,music
+"Which battle occurred first, the Battle of Manila or the Battle of Guam?",Battle of Guam,battle,which,history
+What region of Italy was Giorgio Pini born in?,Emilia-Romagna Region,region,what,geography
+Where was the host of Australia's Got Talent born?,"Warrnambool, Victoria",location,where,television
+Who will conduct the firing cannons in Victoria as part of a musical piece written to celebrate the defeat of Napoleon in Russia?,Tania Miller,person,who,music
+Are both Odds and Supergrass rock bands?,yes,boolean,are,music
+"Who was the brother of the architect of Hook and Ladder No. 4 located in Albany, NY?",Cuyler Reynolds,person,who,history
+Were both Roger Donaldson and André Cayatte french filmmakers?,no,boolean,were,film
+"Who is from farther west, Kings of Convenience or Belle and Sebastian?",Belle and Sebastian,band,who,music
+Which Canadian heavy metal band is Safe  the second EP by,Kittie,band,which,music
+"What festival is held every June in Bartlesville, Oklahoma?",OK Mozart,festival,what,events
+The title Grigory Drozd lost in May 2016 was won by a boxer who made his acting debut in what film?,"""Creed""",film,what,sports
+"Which Austrian and later American chess master who was also a highly influential writer and chess theoretician competed for the first contest  that was defined ""in advance"" as being for the world championship?",Steinitz,person,which,sports
+"In what year was the American country rock band who's number one song was done as a cover on the album ""What If?"" founded?",1966,year,what,music
+Aughton Park railway station is on a branch of the Northern Line how far from Liverpool?,12+3/4 mi northeast of Liverpool,distance,how,geography
+What liberal arts college has their athletic teams named the Lincoln Memorial Railsplitters?,Lincoln Memorial University,college,what,education
+What building is opposite the ceremonial meeting place of the Accession Council in the United Kingdom?,Mark Masons' Hall,building,what,geography
+"Which was founded first, University of California, Santa Barbara or Hamdard University?","University of California, Santa Barbara",university,which,education
+"Which species is the most numerous, Podocarpus or Osmunda? ",Podocarpus,species,which,biology
+"Brandy was handpicked by a producer that herself had been recognized by the ""Guinness World Records"" for what?",most awarded female act of all-time,record,what,music
+"The American cable and satellite television channel that April ""Ape"" Margera appears upon is owned by whom?",Viacom Media Networks,company,whom,media
+Who did Neleus of Scepsis study under in addition to Theophrastus in the Peripatetic school?,Aristotle,person,who,philosophy
+What was the name of the treaty that made Hungary a landlocked state which contained the Kolozsvar Ghetto?,Treaty of Trianon,treaty,what,history
+What is the name of the English poet who wrote a poem titled Don Juan and also served as the real-life model of a character in the novel Glenarvon?,Lord Byron,poet,what,literature
+What is the name of the former MGM Grand Garden Special Events Center where the Mayweather-Ortiz fight took place?,MGM Grand Garden Arena,location,what,sports
+"Philadelphia, Here I come! is a 1964 play by an Irish dramatist who has been likened to an Irish Checkhov?",Brian Patrick Friel,person,what,literature
+"Before composing music for the game which contains Chocobos and Moogles, what cover band was he involved in?",Bothtec,band,what,music
+Did Zaza Pachulia win an NBA Championship with the Atlanta Hawks?,Zaza Pachulia,person,did,sports
+What is the relationship of Yeshahework Yilma's mother to the man who was Ethiopia's emperor from 1930 to 1974?,niece,relation,what,history
+What ranks does Jerry Kyd wear while captaining the largest warship ever built for the Royal Navy?,Captain while retaining the substantive rank of Commodore,rank,what,military
+Which film was released first out of The Hunchback of Notre Dame and Miracle of the White Stallions?,Miracle of the White Stallions,film,which,film
+Lighthouse Beach is a small section of the Fire Island National Seashore that is adjacent to a state park in Long Island that is how big?,875 acre,size,how,geography
+"What film came out first, All the Marbles or What Ever Happened to Baby Jane?",What Ever Happened to Baby Jane?,film,what,film
+What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?,Beyond the Clouds,film,what,film
+"Who was born first, Mark Steven Johnson or Robert Thornby?",Robert Thornby,person,who,film
+The Black Star identified closely with this American magazine magnate who was called what?,"the most influential private citizen in the America of his day""",title,what,history
+"Which writer also enjoyed boxing, Amy Tan or Norman Lindsay?",Norman Alfred William Lindsay,person,which,literature
+What professional do both Marc Bolan and Ian MacKaye share?,musician,profession,what,music
+"Which of Tara Strong major voice role in animated series is an American animated television series based on the DC Comics fictional superhero team, the ""Teen Titans""?",Teen Titans Go!,television show,which,television
+What defines Downers Grove as a thriller?,by the moods they elicit,characteristic,what,film
+"""Kapitän zur See"" Ernst Lindemann was the commander of which one of two battleships built for Nazi Germany's ""Kriegsmarine""?",Bismarck,ship,which,military
+How tall was one of Drew Barry's brothers?,6 ft,height,how,sports
+Black Book starred the actress and writer of what heritage?,Dutch,nationality,what,film
+This cabin built in 1925 is located in a park encompasses over 1 million acre and includes parts of two what?,mountain ranges,feature,what,geography
+When did the person for whom Tubman University was named serve as President of his country?,1944 until his death in 1971,time period,when,history
+Are Hanafuda and Okey both card games?,no,boolean,are,games
+What production company was co-created by the lead singer of the band Poison?,Sheen Michaels Entertainment,company,what,business
+"Warren Gamaliel Harding had an affair with which American secretary born on November 9, 1896?",Nan Britton,person,which,history
+When was the actor born who starred in Sugarfoot?,"January 23, 1898",date,when,film
+"Which group of people in West Africa with significant populations in Ghana, Ivory Coast, Liberia and Sierra Leone use the Ida?",The Yoruba,group,which,anthropology
+"Which magazine launched first, Real Simple or Life?",Life,magazine,which,media
+The Good Design movement is associated with what Danish furniture designer whose style is often described as Organic Functionality?,Hans Jørgensen Wegner,person,what,design
+"Which band has more constant members, Lilys or Ratatat?",Ratatat,band,which,music
+What soldier and statesman was also a brother to James FitzJames mother?,"John Churchill, 1st Duke of Marlborough",person,what,history
+" Also known as the Great Action, what other name did this event go by?",Große Aktion,event,what,history
+"Which of the campaign that brought out the term Vichy Republican on social media was formally launched on June 16, 2015, at Trump Tower in New York City?",presidential campaign of Donald Trump,event,which,politics
+What country of origin does Witness and Pamela Wallace have in common?,American,nationality,what,film
+What are both Camel Up and Outrage!?,board game,game,what,games
+The Birthday of the World: and Other Stories was written by what fantasy and sci-fi author?,Ursula Kroeber Le Guin,person,what,literature
+"Between Joe Orton and Henry James, which author is the adjective ""Ortonesque"" used to refer to his work?","John Kingsley ""Joe"" Orton",person,which,literature
+What was position of the man who served Prince Andrew from 1990-2001?,Private Secretary and Treasurer,position,what,politics
+"Where is the company, which designed the Glomar Challenger, based?","Vernier, Switzerland",location,where,business
+Rafael de Souza Pereira is a defensive midfielder for what organization that is the top level of the Mexican football league system?,Liga MX,organization,what,sports
+"Who does the previous Vice President of Production at the animation studio that is owned and operated by Viacom, currently work for?",Cartoon Network Studios,company,who,business
+Which book by William A. Dembski summarizes the concepts he introduced about intelligent design in another of his works?,Intelligent Design,book,which,literature
+"Near which town did the family of author of the ""Adventures of Huckleberry Finn"" own several tracts of land?",Santa Fe,town,which,geography
+Which American car rental company is also a member of the Association of Car Rental Industry Sytems Standards?,Budget Rent a Car,company,which,business
+Are Wolfgang Becker and Paul Andrew Williams both film directors?,yes,boolean,are,film
+"In addition to the best known comic servant from Commedia dell'arte, who else is featured in La Surprise de l'amour?",Columbine,character,who,literature
+"The ""New York Times"" bestselling book ""The 50th Law"" contains lessons and anecdotes from this historical figure who was a classical Greek (Athenian) philosopher who is known mainly through the writings of which philosopher?",Plato,person,which,literature
+What are the names of the members of the Detroit-based hip hop duo who has worked with Jason Gilbert?,"Royce da 5'9"" (Bad) and Eminem (Evil)",members,what,music
+Where's Officer Tuba? is a comedy film that stars which Hong Kong-based actress?,Joey Wong Cho-Yee,person,which,film
+"What was the name of the 1996 loose adaptation of William Shakespeare's ""Romeo & Juliet"" written by James Gunn?",Tromeo and Juliet,film,what,film
+Who graduated from The SC Bastia Reserves and Academy  and was born 9 September 1987?,Alexandre Dimitri Song Billong,person,who,sports
+What is the name of the largest publicly owned company in Minnesota that is situated in Minnesota's most populous county?,United Healthcare,company,what,business
+"Which American film director and actor famous for his role as Donny ""The Bear Jew"" Donowitz wrote the 2012 film ""The Man with the Iron Fists""?",Eli Roth,person,which,film
+What dance academy did the starring actress from The Glory of Tang Dynasty graduate from?,Beijing Dance Academy,institution,what,education
+Javier Frana and Thomaz Koch both share what occupation?,tennis player,occupation,what,sports
+Who was the winner of the season of The Rap Game where a contestant named Lady Leshurr as one of her influences?,Mani,person,who,music
+Mount Gunnison was named in honor of an explorer born in what year?,1812,year,what,history
+What is the nationality of the illustrator of Chaos Marauders?,British,nationality,what,art
+"Which is located farther West, the Cascade Locks and Canal or Lake Worth Lagoon?",Cascade Locks and Canal,location,which,geography
+"Corbin Sinclair Bryant played for the Buffalo Bills and what other NFL team that was founded in 1933, and is the oldest franchise in the AFC?",Pittsburgh Steelers,team,what,sports
+When was the brewery founded which has The Stag Inn Hastings as a tied house ?,1698,year,when,business
+What was triggered by the resignation of Conservative MP Stephen Phillips on 4 November 2016 and was when Caroline Johnson became a Member of Parliament?,The Sleaford and North Hykeham by-election,event,what,politics
+The Lucy Maud Montgomery novel about the character Anne Shirley was first translated into Japanese by what woman?,Hanako Muraoka,person,what,literature
+When was the University established where Laura Landweber is a professor ?,1754,year,when,education
+What is the middle name of the female MSNBC journalist critiqued on the Daily Howler?,Anne,name,what,media
+What kind of building did the Romans use that is still the most common architectural style for churches in Europe and America?,basilica,building,what,architecture
+H-63 is a highway that runs near the county seat of what Michigan county?,Chippewa County,location,what,geography
+Persona Non Grata includes interviews by the political leader of what nationality?,Palestinian,nationality,what,politics
+Are Control Room and Meeting Resistance both documentary films that involve Iraq?,yes,boolean,are,film
+Sebastian Gutierrez is known for writing the screenplay for the 2003 film directed by whom?,Mathieu Kassovitz,person,whom,film
+What bird of prey can be found around Brackendale?,The bald eagle,bird,what,wildlife
+On which coast can the California Aqueduct and the Lake Washington Ship Canal be found?,west,coast,which,geography
+Were H. G. Wells and Rubén Darío both English?,no,boolean,were,literature
+Marine Air Control Group 48 is stationed at Naval Station Great Lakes which located where?,"Lake County, Illinois",location,where,military
+"Which school is older, Dartmouth College or Cornell University?",Dartmouth College,school,which,education
+Who served as a Prime Minister in postwar Britain?,Clement Richard Attlee,person,who,politics
+"Who was born first, Sathish Kalathil or Jacques Feyder?",Jacques Feyder,person,who,film
+"What census-designated place located in Nassau County, New York has as portions of it's territory, districts of Brian Curran, New York State assemblyman?",Baldwin,place,what,geography
+Arena Essex Raceway is built alongside an out-of-town shopping centre constructed on the site of a former what?,chalk quarry,site,what,geography
+Which novelist wrote under the pen name Walter Ericson: Laura Esquivel or Howard Fast?,Howard Melvin Fast,person,which,literature
+Philip Despencer's brother was a favorite of which king who was deposed in January of 1327?,King Edward II,person,which,history
+What is the age range of the league for which the Bobby Smith Trophy is awarded?,16–21,age range,what,sports
+What do Lavinia Greenlaw and Nâzım Hikmet have in common?,poet and novelist,profession,what,literature
+"Which documentary film follows poetry, Race to Nowhere or SlamNation?",SlamNation,film,which,film
+What is  Cyaxares relationship to the man that governed for 53 years?,r grandfather <a hr,relation,what,history
+A writer for The Kenyon Review won a Pulitzer Prize in what year?,1947,year,what,literature
+The first person to reach both poles named a glacier after which person?,the daughter of Fridtjof Nansen.,person,which,exploration
+Tyler Redenbach was selected 77th overall by the Phoenix Coyotes in an NHL draft that was held in what city?,Nashville,city,what,sports
+Alan Schechter supervised a woman with her senior thesis entitled what,"""There Is Only the Fight . . . : An Analysis of the Alinsky Model.""",thesis,what,education
+Are Waste Land and Shikashika both types of drinks?,no,boolean,are,food
+"Who was born first, Umar Israilov or Ramzan Kadyrov?",Ramzan Kadyrov,person,who,politics
+" Force India VJM10 is a Formula One racing car previous driven by Nico Hülkenberg, and is now driven by which driver born 17 September 1996?",Esteban Ocon,person,which,sports
+"This expansion of the 2008 magazine article ""Is Google Making Us Stoopid?"" was a finalist for what award?",Pulitzer Prize,award,what,media
+What Major League Soccer team owned by Anthony Precourt is coached by a retired American soccer player ?,Columbus Crew Soccer Club,team,what,sports
+Where was the Danger Mouse produced U2 album exclusively released?,iTunes,platform,where,music
+Who designed the building that the 58th Technology & Engineering Emmy Awards were held in?,Designed by KlingStubbins,person/company,who,architecture
+The boxer that defeated Oliver Lavigilante in the 2012 Summer Olympics is of what nationality?,a Ghanaian boxer,nationality,what,sports
+Do both Changle District and Longkou face water?,yes,boolean,do,geography
\ No newline at end of file
diff --git a/testing/datasets/hotpotqa_conditional/hotpot_train.csv b/testing/datasets/hotpotqa_conditional/hotpot_train.csv
new file mode 100644
index 000000000..b967caabf
--- /dev/null
+++ b/testing/datasets/hotpotqa_conditional/hotpot_train.csv
@@ -0,0 +1,501 @@
+question,answer,answer category,question word,topic
+At My Window was released by which American singer-songwriter?,John Townes Van Zandt,person,which,music
+which  American actor was Candace Kita  guest starred with,Bill Murray,person,which,film
+"Which of these publications was most recently published, Who Put the Bomp or Self?",Self,publication,which,media
+The Victorians - Their Story In Pictures is a documentary series written by an author born in what year?,1950,year,what,film
+"Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?",Tae Kwon Do Times,publication,which,media
+In what year was the club founded that played Manchester City in the 1972 FA Charity Shield,1874,year,what,sports
+"Which is taller, the Empire State Building or the Bank of America Tower?",The Empire State Building,building,which,geography
+"Which American actress who made their film debut in the 1995 teen drama ""Kids"" was the co-founder of Voto Latino?",Rosario Dawson,person,which,film
+"Tombstone stared an actor born May 17, 1955 known as who?",Bill Paxton,person,who,film
+"What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?",Operation Citadel,code name,what,history
+"Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of ""Hamlet."" ?",Kerry Condon,person,who,film
+Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?,Buena Vista Distribution,company,which,film
+Samantha Cristoforetti and Mark Shuttleworth are both best known for being first in their field to go where?,space,location,where,exploration
+"Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers? ","""Outfield of Dreams""",type,what,sports
+"Which Pakistani cricket umpire who won 3 consecutive ICC umpire of the year awards in 2009, 2010, and 2011 will be in the ICC World Twenty20?",Aleem Sarwar Dar,person,which,sports
+The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?,2010,year,what,organization
+"""Everything Has Changed"" is a song from an album released under which record label ?",Big Machine Records,company,which,music
+"Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?",Aleksandr Danilovich Aleksandrov,person,who,science
+On the coast of what ocean is the birthplace of Diogal Sakho?,Atlantic,location,what,geography
+This American guitarist best known for her work with the Iron Maidens is an ancestor of a composer who was known as what?,The Waltz King,title,what,music
+"Who was the Tennis Masters Cup champion in 2000, Gustavo Kuerten or Stan Wawrinka?",Gustavo Kuerten,person,who,sports
+Do Stu Block and Johnny Bonnel's bands play the same type of music?,no,boolean,do,music
+Are Baltasar Kormákur and John G. Avildsen both film producers?,no,boolean,are,film
+What head of state position was held by Harry S Truman when he gave Harold E Wilson the Medal of Honor?,President of the United States,position,what,history
+How old is the fossil record of the order that contains the only strictly marine herbivorous mammal?,50-million-year-old fossil record,age,how,science
+What evening cable television station programming block has a show with Ashley Holliday as a cast member?,Nick at Nite,program,what,television
+What person does Wormholes in fiction and Nathan Rosen have in common?,Einstein,person,what,science
+"Which band had a longer hiatus, Juliette and the Licks or The Last Shadow Puppets?",The Last Shadow Puppets,band,which,music
+Remember Me Ballin' is a CD single by Indo G that features an American rapper born in what year?,1979,year,what,music
+"Who composed ""Sunflower Slow Drag"" with the King of Ragtime?",Scott Hayden,person,who,music
+Who was coach of the No. 9-ranked team that was upset in the NCAA Tournament by the 2014-15 UAB Blazers men's basketball team?,Fred Hoiberg,person,who,sports
+"Which movie was released first, Son of Flubber or Davy Crockett, King of the Wild Frontier?","Davy Crockett, King of the Wild Frontier",film,which,film
+Some performance of Take a Bow took place near the bank of which river?,River Thames,river,which,geography
+What is the nickname for this United States drag racer who drove Brutus?,Jungle Jim,nickname,what,sports
+Both Brian Warren and Jake Shields are considered what?,mixed martial arts fighter,profession,what,sports
+The historical unit of the US Army that began its service before the Civil War fought in what battle in 1879?,Battle of Milk Creek,battle,what,history
+What part of the world do the Viezenbeek and Lindemans Brewery hale from?,Brussels,location,what,geography
+Both Jack Hill and Bernard McEveety are involved in what occupation?,film director,profession,what,film
+What Brazilian professional racing driver who races for Rebellion Racing has a mother named Viviane Senna da Silva Lalli?,Bruno Senna Lalli,person,what,sports
+"""The Day I Met Marie"" is a song written by an English songwriter best known as the lead guitarist for what band?",the Shadows,band,what,music
+"Whats was the population in 2003 of Centralia, the least populated municipality in Pennsylvania, USA?",7,population,what,geography
+What Scottish nobleman was the subject of a 1934 British short documentary and was the first man to fly over Mount Everest?,"Douglas Douglas-Hamilton, 14th Duke of Hamilton",person,what,history
+What metals were produced by the the steelworks company hit in the Barrow Blitz alongside VSEL?,iron and steel,materials,what,history
+Chester Bennett was executed during an occupation that happened after how many days of fierce fighting ?,18 days,duration,how,history
+In which year was the first of the vampire-themed fantasy romance novels for which The Twilight Saga: The Official Illustrated Guide serves as a spin-off encyclopedic reference book first published?,2005,year,which,literature
+"Where was the singer who released the single ""I Never Made Love (Till I Made It with You)"" originally from ?","Lubbock, Texas",location,where,music
+What type of film was it that featured the actress who is regarded as the greatest actress of all time?,anti-McCarthyism,genre,what,film
+"What instrument did Nigel Walley play in John Lennon's band formed in 1956, which would later become the Beatles?",tea-chest bass,instrument,what,music
+Which type of horse has been used by both the British Household Cavalry and Anheuser-Busch Brewing Company for commercials?,Clydesdales,horse,which,animals
+Milan Sachs conducted the premiere of the play Jenůfa by the composer who was inspired by what types of music?,Moravian and other Slavic folk music,music,what,music
+The Montenegrin Mountain Hound and Mudhol Hound are both used for what common purpose?,hunting,purpose,what,animals
+Are Tiffanie DeBartolo and Tony Gatlif both novelists?,no,boolean,are,literature
+"Which director was nominated for more Academy Awards, Lee Tit or Sidney Lumet?",14 of his films were nominated,nomination,which,film
+"What type of specialist school is the lead member of the seven academy schools in Lincolnshire, England?","science, technology and teaching",school,what,education
+"What was formed farther east, The Ocean Blue or The Dandy Warhols",The Ocean Blue,band,what,music
+"Who was the undisputed ""boss""  that succeeded Woodrow Wilson in Atlantic City?","Enoch Lewis ""Nucky"" Johnson",person,who,history
+"Which Zealand Institute of Business and Technology campus is located in a city with a population of 50,046?",Roskilde,campus,which,education
+What year was the co-author (with Tom Verducci) of The Yankee Years born?,1940,year,what,literature
+"Who lived longer, Al-Ghazali or Averroes (Ibn Rushd)?",Ibn Rushd,person,who,history
+"Who is older, Irish boxer Andy Lee or American Boxer Bryan Lee Vera?",Bryan Lee Vera,person,who,sports
+Which team drafted Emil Johansson in the 52nd NHL Entry Draft in the seventh round?,Providence Bruins,team,which,sports
+What sport was the player born in the county seat of Osage County involved in?,Rugby,sport,what,sports
+"Which film was created first, Fig Trees or Pond Hockey?",Pond Hockey,film,which,film
+Where did the game the Angry Birds Movie was based on first launch?,New Zealand,location,where,games
+What was the nationality of the mixed martial artist who defeated Trent Jenkins in the first ever fight in UFC history?,American,nationality,what,sports
+"In which state's mountain region lay the famous peak that inspired the bestselling novel ""Cold Mountain?""",North Carolina,state,which,geography
+The Australian food spread developed by Cyril Percy Callister is considered an acquired taste based on what sense?,taste,sense,what,food
+"Which movie is based on a book, Darby O'Gill and the Little People and College Road Trip?",Darby O'Gill and the Little People,film,which,film
+Carlo Ignazio Monza's music formed a ballet based on what 18th century play?,Commedia dell'arte,person,what,music
+"What singer, who originally sang a song included as the third single in Mel McDaniel's 1980 album ""I'm Countryfied"" had the nickname ""Gentle Giant""?",Don Williams,person,who,sports
+Who did the ninth overall pick in the 2007 NBA draft tie with during voting for the 2013 NBA All-Defensive Team?,Tyson Chandler,date,which,royalty
+"Princess Astrid, Mrs. Ferner is the older sister of a king who ascended to the throne on which day ?",on 17 January 1991,genre,what,music
+What genre of music has American musicians Jonathan Davis and Philip Labonte both have in common in the bands they have played in?,metal,country,what,geography
+What country does Rand Show and Johannesburg have in common?,South Africa,company,what,film
+Michael Giacchino composed the scores to many films such as a 2015 computer-animated film produced by what studio?,Pixar Animation Studios,year,what,television
+"A police detective, author and former child actor who appeared on a show that had segments about making food and taking things apart, joined the police department in what year?",1996,location,what,music
+Singer Annette Snell died in a airplane crash during a forced landing on a highway in what state?,Georgia,person,who,television
+Who created the TV series which had the storyline Sharongate ?,Julia Smith and Tony Holland,person,who,television
+Which of the following airports hosts Peterson Air Force Base as a tenant: Newark Liberty International Airport or Colorado Springs Airport?,Colorado Springs Municipal Airport,airport,which,geography
+Are Rockefeller University and Massachusetts Institute of Technology both considered to be prestigious?,yes,boolean,are,education
+"Hargus Melvin ""Pig"" Robbins who has played with an American alternative rock band whose highest charting single was what?",Push th' Little Daisies,song,what,music
+What was the estimated population in 2015 of the city in which Anya Beyersdorf was born ?,"23,674",population,what,geography
+What actor stars in Predestination and Game of Thrones?,Noah Taylor,person,what,film
+Juan Pablo Di Pace had a role on the sequel series to what Jeff Franklin-created American sitcom?,Full House,television show,what,television
+"Which James Bond movie theme song artist released first, the artist of Die Another Day or The World Is Not Enough?",Madonna,person,which,music
+When was the Houston Rockets point guard recruited by Frank Sullivan born?,"August 23, 1988",date,when,sports
+Johnny Narron is a friend of this center fielder whom he helped in his recovery from drug addiction who won the MVP award in what year?,2010,year,what,sports
+Charlotte Löwensköld is a novel by which Swedish author and teacher?,Selma Ottilia Lovisa Lagerlöf,person,which,literature
+How much did retired Canadian professional ice hockey player of Albanian origin spend with his cohorts to purchase the Ontario Raiders?,"$250,000",amount,how,sports
+Are both Ralph Saenz and Roddy Woomble credited as writers?,no,boolean,are,literature
+"Besides dolphins and killer whales, what other animals can be seen in Hauraki Gulf Marine Park?",Bryde's Whales,animals,what,wildlife
+Who has been on a British television music competition show and was was most popular in the 80's with the pop band 'Culture Club'?,George Alan O'Dowd,person,who,music
+The Badge is a 2002 mystery-thriller film starring an American actress whose notible films include what 1999 Martin Scorsese film?,Bringing Out the Dead,film,what,film
+What actress that starred in The Legend of Tarzan got her start by appearing in Australian independent films?,Margot Elise Robbie,person,what,film
+Was The Devil and Max Devlin or Do Dooni Chaar released first?,The Devil and Max Devlin,film,was,film
+"""Hurt"" is a 2009 gothic horror movie starring William Mapother, but what TV series is he most associated with?",Lost,television show,what,television
+When was the company founded that now publishes Formula D ?,1995,year,when,business
+"Who has fought in the most championships, Tim Kennedy or Jason Miller?",Tim Kennedy,person,who,sports
+Imiliya is a town located in what pilgrimage site?,Lumbini,location,what,geography
+Which is the largest airport which serves Sauce Viejo Airport?,Aerolíneas Argentinas,airport,which,geography
+What was a previous unoffical name for the high performance variant of Audi's compact executive car?,Audi Ur-S4,name,what,automobiles
+Imperfect Circles is an episode of the drama that was developed by whom?,Brian K. Vaughan,person,what,television
+The Mercurial Vapor is a football boot  endorsed by many players such as a Brazilian professional footballer who plays for what national team?,Brazil,team,what,sports
+For which college team did this American professional basketball player for the Miami Heat of the National Basketball Association (NBA) who earned praise for being the NBA's leading rebounder during the 2016–17 Miami Heat season play?,the Marshall Thundering Herd,team,which,sports
+Do both Scott Treleaven and Raoul Peck make films?,yes,boolean,do,film
+What is the artist name who contributed to Sprition Media Games online video game's soundtrack?,Jeramiah Ross,person,what,music
+"Who was the director of the film whose success, along with that of An American Tail and The Land Before Time, prompted Steven Spielberg to establish his own animation studio?",Robert Zemeckis,person,who,film
+"Which rocket propelled weapon, sometimes referred to as the ""Stovepipe,"" did Colonel Leslie Alfred Skinner help develop?",Bazooka,weapon,which,military
+What is the name of the person who R Lee Ermy played his character in Prefontaine film and who is also an American track and field coach and co-founder of Nike Inc?,Bill Bowerman,person,what,sports
+What 45 acre neighborhood is north of the South End-Groesbeckville Historic Distric?,The Mansion Historic District,neighborhood,what,geography
+When was the member of Dillard & Clark who also played in the Byrds born?,1944,year,when,music
+Which English actor starred in Cabin Pressure,Roger William Allam,person,which,television
+"How many games were won during his career by the right-handed pitcher whose deadly beanball in 1920 is featured in ""The Pitch That Killed"" by Mike Sowell?",over 200,number,how,sports
+What fruit are the oroblanco and the pomelo similar to?,similar to grapefruit,fruit,what,food
+What country does Ngezi Platinum F.C. play in?,Africa,country,what,sports
+Are Norma and Idomeneo both operas by Bellini?,no,boolean,are,music
+Are Maynard James Keenan and Jo Kwon from the same country?,no,boolean,are,music
+"During Maine gubernatorial election, 1956, which candidate ran, who later on ran for Vice President of the United States?",Edmund Muskie,person,which,politics
+Which Portuguese winger played beside  Zlatan Ibrahimović in the 2008-09 Inter Milan team ?,Quaresma,person,which,sports
+The sea whose port is The Port of Reggio is sometimes considered as a part of which ocean?,Atlantic Ocean,ocean,which,geography
+Under which number president did Arthur A. Ballantine become solicitor of the Internal Revenue Service?,31st,number,under which,history
+Who founded a peer-reviewed scientific journal in the field of fluid mechanics?,George Keith Batchelor FRS,person,who,science
+"Which main antagonist in some of the Sherlock Holmes stories was holmes old arch-enemy, in Sherlock Holmes - The Musical",Professor James Moriarty,character,which,literature
+"FM broadcasting began at the station of which inventor, who won the first Medal of Honor by what is now the IEEE?",Edwin Howard Armstrong,person,which,technology
+Are Princess Sumaya University for Technology and Tennessee Technological University from the same country?,no,boolean,are,education
+In 1904 Selma Feldbach graduated in medicine from a comprehensive university founded in what year?,1834,year,what,education
+When did the coach of the Karachi Kings finish his time as coach of the Australia national cricket team ?,23 June 2013.,date,when,sports
+What type of gesture does Donets-Krivoy Rog Soviet Republic and have in common?,signed,gesture,what,politics
+"Who composed the song from the film ""Zindagi Na Milegi Dobara"" that was shot in the Spanish town of Alajar?",Javed Akhtar,person,who,film
+"In what year did a Soviet, Russian, and American theoretical physicist, whose main contributions are in the field of condensed matter physics, come up with his famous theory?",1957,year,what,science
+"Who supervises the subordinate that occupies the majority of Chatan, Japan?",United States Pacific Command,organization,who,military
+What show included an actress who was also in a show set in modern-day San Francisco?,"""Melrose Place""",television show,what,television
+Merle Reagle did crosswords for what magazine that has a focus on aging issues?,AARP The Magazine,magazine,what,media
+The Sting was an episode of The Office written by the American actress who created what other sitcom?,The Mindy Project,television show,what,television
+What ride at Disney was inspired by an old TV show and also inspired a made for TV movie on the Disney channel?,The Twilight Zone Tower of Terror,ride,what,film
+Abdul Alkalimat is a professor of African-American studies and library and information science at a university founded in which year ?,1867,year,which,education
+When did the Basketball Tournament which  Kansas played its final in  1996–97 incept,1997,date,when,sports
+Which Italian composer did George Balanchine add in 1976?,Arcangelo Corelli,person,which,music
+" Who was born first, Renáta Tomanová or Travis Parrott?",Renáta Tomanová,person,who,sports
+What do the outcomes of both Baker v. Wade and Bowers v. Harwick have in common?,appellate court reversed the lower court's decision,outcome,what,law
+"""Everybody's on the Run"" is a song from an album that was the first studio release since it's frontman departed from Oasis in what year?",2009,year,what,music
+How much consumption of non-fish foods would you get eating only packages with Med Marks on them?,low,amount,how much,health
+What was the population of the city where Woodward Avenue ends in 2010?,"59,515",population,what,geography
+Reggie Theus was the assistant coach for what man who had coached in the NBA?,Rick Pitino,person,who,sports
+What rock band had a song featured on a television program with Tony Ayres serving as an Executive Producer?,Bonk,band,what,music
+Are the Carlisle Cathedral and Bangor Cathedral both located in the United Kingdom?,yes,boolean,are,geography
+Gustav Mahler composed a beautiful piece performed by the Bach-Elgar Choir. What is the name of that piece??,Symphony No. 2,music,what,music
+Which former White House employee worked alongside Kaitlan Collins at the politically conservative news website The Daily Caller?,Neil Patel,person,who,media
+"Ryan Hulings plays for the Dayton Dyanmo, who is his head coach?",Dan Griest,person,who,sports
+What important political role did the female star of the film Talk About a Stranger hold ?,First Lady of the United States,position,what,politics
+"During the 2010 Dutch general election, which current Prime Minister led the People's Party for Freedom and Democracy?",Mark Rutte,person,which,politics
+"The ""To You I Belong"" singers broke up when when which member decided to leave?",Sinéad O'Carroll,person,which,music
+Laura is a talk show hosted by the presenter whose work appears on what network?,Univision,network,what,media
+"Who released the album that ""Stylo"" was a single on in the United States?",Virgin Records,company,who,music
+Where is the football club from that Glen Robert Moss plays for?,New South Wales,location,where,geography
+"Which school was founded earlier, Wellesley College or the University of Virginia?",University of Virginia,school,which,education
+"What mountain chain includes the mountain where Shiloh, Alabama is located?",Appalachian,mountain chain,what,geography
+"Which Austin, Texas band has toured with more bands, Alpha Rev or Driver Friendly?",Driver Friendly,band,which,music
+"The Fault in Our Stars is a romantic drama film that stars what American actress and activist that was born in San Bernardino County, California, and was raised in the Simi Valley?",Shailene Diann Woodley,person,what,film
+"Kelsey Ross Brokes is associated with what Costa Mesa, California-based clothing company?",RVCA,company,what,business
+"Wing warping, a system for lateral aircraft control, was pioneered by the Wright brothers during their first controlled, sustained flight near which now-famous location?","Kitty Hawk, North Carolina",location,which,geography
+The second place finisher of the 2011 Gran Premio Santander d'Italia drove for who when he won the 2009 FIA Formula One World Championship?,Brawn GP,team,who,sports
+When was the submachine gun introduced that was manufactured by Emil Bergmann's father?,1918,year,when,technology
+If you are taking the M2 motorway to one of the Northern Ireland's most busiest airport what airport are you going to?,Belfast International Airport,airport,what,geography
+"Which genus of plants is endemic to Australia, Retama or Baeckea?",Retama,genus,which,biology
+"Full Circle included a cover of the song by which songwriter, musician, and actor?","Roger Dean Miller, Sr.",person,which,music
+Who replaced the chairman of the AFL Commission who started in 2007?,Richard Goyder,person,who,sports
+Old Rectory is in a small town and civil parish where?,"North Lincolnshire, England",location,where,geography
+Are the films Underdog and Mary Poppins Returns both musical films?,no,boolean,are,film
+Which Led Zepplin bassist arranged Meic Stevens first single for Decca Records?,John Paul Jones,person,which,music
+Which 2013 sports drama starred Nicole Beharie?,42,film,which,film
+What English writer born in 1812 was a schoolfellow of Henry Sewell Stokes?,Charles Dickens,person,what,literature
+"Who produced a 1987 Virgin Records album with the help of a film director born January 20, 1946?",Orbison with T Bone Burnett,person,who,music
+Son of Paleface starred the comedian and vaudevilian of what nationality?,American,nationality,what,film
+"Meihekou and Tumen, Jilin are located in what country?",China,country,what,geography
+"Which film was released first, The Three Lives of Thomasina or The Shaggy Dog?",The Shaggy Dog,film,which,film
+"Who was born first, Rowan Atkinson, or Maxim Baldry?",Rowan Sebastian Atkinson,person,who,film
+This song covered by Dr. John in 1972 was originally sung by which group?,The Dixie Cups,group,which,music
+"How many miles long is this major railway link between London and Edinburgh, in which The Swinton-Doncaster line is a short railway between Swinton Junction and Doncaster South Yorkshire?",393 mi,distance,how,geography
+"Verrit, whose slogan is “media for the 65.8 million” refers to what part of the Hillary Clinton presidential campaign, 2008?",the number of votes,part,what,politics
+Which musician who performed in Live from Freedom Hall was born in 1960?,"Donald ""Ean"" Wayne Evans",person,which,music
+"Who provided the German libretto to the composition by a man whose best known work is ""The Threepenny Opera,"" which proved to be their last major collaboration?",Bertolt Brecht,person,who,music
+What do both Szymon Giżyński and Law and Justice have in common?,Christian democratic political,ideology,what,politics
+"The tomb of Ferdowsi example of Achaemenid architecture, a style from which culture?",the Achaemenid Persians,culture,what,history
+"What time of day does the CNN program hosted by a journalist born on March1, 1966 air?",late evening program,time,what,television
+"Which mountain is higher, Manaslu or Jengish Chokusu?",Manaslu,mountain,which,geography
+The Oster Conspiracy hoped to reinstate the grandson of which British Monarch as Emperor?,British Queen Victoria,monarch,which,history
+Who was DJ Pain 1 produced for that founded Distrurbing tha Peace?,Ludacris,person,who,music
+"What is Mikael Blomkvist's profession in ""The Girl in the Spider's Web""?",investigative journalist,profession,what,literature
+Cross Border Orchestra of Ireland is based at an institute in what province?,Leinster,province,what,geography
+"What city and state was the 1973 film, which starred Gary Kurtz, set?","Modesto, California",location,what,film
+Akihiro Gono is one of only four men to defeat the Mixed Martial Artist who has what nationality?,Cuban-Australian,nationality,what,sports
+Nike Mercurial Vapor is endorsed by the Brazilian footballer who plays what position for Liverpool?,midfielder,position,what,sports
+Chellis Glendinning worked alongside what Indian scholar?,Vandana Shiva,person,what,education
+"Both Jacques Audiard and Howard Hughes both held one common job title, what title was this?",film director,job title,what,film
+"The Series Land Rover that was manufactured in Tehran, Iran, was inspired by which US company?",US-built Willys Jeep,company,which,automobiles
+"Who received more nominations for the Academy Award Oscar, Stephen Frears or Mario Monicelli?",Mario Monicelli,person,who,film
+"What American dancer, actor, singer, director, and choreographer was in the 1943 musical comedy Du Barry Was a Lady?",Gene Kelly,person,what,film
+Who wrote an autobiography published by White Mountain Castle Publishing about his adventure in Hawaii?,"Daniel Keauhou Matsu ""Danny"" Yamashiro",person,who,literature
+"The Grohl Sessions, Vol. 1 is by a band based in what city?","Atlanta, Georgia",city,what,music
+Ivan Cooper led an anti-internment march which developed into a massacre where how many people died ?,Fourteen people died,number,how many,history
+Recycler Tour was sponsored by the beer that is what percentage alcohol by volume?,4.20%,percentage,what,food
+What Roman Goddess if wisdom and strategic warfare is featured on wood as a pignora imperii?,Minerva,deity,what,mythology
+"aFox 40 news anchor Martha MacCallum used to anchor a show on CNBC that had the same time slot that the Morning Call does now, what was the name of that show? ",The Money Wheel,show,what,television
+About which singer is the film in which Sylvie Testud  appeared in 2007 ?,Édith Piaf.,person,which,film
+Master of Study starred the actress who had a role as a political activist in what 2002 film?,Sympathy for Mr. Vengeance,film,what,film
+Martina Iñíguez wrote about the history of what dance that began in the working-class neighborhoods of Montevideo?,Tango,dance,what,dance
+"Who is the actor that was in the Scream trilogy horror films, that will star in Woody Allen's newest project? Liev Schreiber",Liev Schreiber,person,who,film
+"Which country did Laura Godoy represent in the pageant held on 19 December 2012 at The AXIS, Las Vegas, Nevada, United States?",Guatemala,country,which,events
+What genre does Hot Hot Heat and 1990s have in common?,rock,genre,what,music
+"Dinesh Chandra Bhandary was awarded the Vir Chakra, for his service during a war that lasted for how many days ?",13 days,number,how many,history
+Lightning Bug is a 2004 coming of age drama film executive produced by an American actress best known for her role as who?,Donna Pinciotti,character,who,film
+What theater did the company that Eliard Swanston worked for perform at?,the Globe Theatre,theater,what,history
+"Indian film photographer Jagdish Mali, known for taking images of various celebrities including Shabana Azmi, is father to which Bollywood actress? ",Antara Mali,person,which,film
+Who directed the 1988 film in which the exterior of the abandoned W.S. Kelly gas station and general store in Cogar can be seen?,Barry Levinson,person,who,film
+"Which composer was born first, Maurice Ravel or Béla Bartók?",Maurice Ravel,person,which,music
+"The 2006 disaster drama film starring an American actor in ""Revolutionary Road"" and ""99 Homes"" is based on what?",September 11 attacks,event,what,film
+What FA Trophy final was the 44th final where Grimsby Town was beaten?,2013 FA Trophy Final,event,what,sports
+Together is a Hungarian political party that contrasted the government of the Prime Minister who has held the post since what year?,2010,year,since what,politics
+Who founded the Dutch Symphonic metal band who released the album Black Symphony?,Sharon den Adel and guitarist Robert Westerholt,person,who,music
+Who was the director of the 2014 sci fi comedy film which features the romantic Hindi song Chaar Kadam?,Rajkumar Hirani,person,who,film
+"The Snowball family included Nathaniel, who was recorded in a document that held names and descriptions of what?","3,000 Black Loyalists",document,what,history
+Which movie produced by Sōjirō Motoki  is a 1957 Japanese film co-written and directed by Akira Kurosawa?,Throne of Blood,film,which,film
+What game of chance has a name that roughly means fate or personal destiny?,Wyrd,game,what,games
+"The tour that Taylor Swift completed in Singapore on June 12, 2014 grossed how much?",$150 million,amount,how much,music
+Does Nicolas Mahut or Elena Bovina have the highest career singles world ranking?,Elena Olegovna Bovina,person,who,sports
+American Whig party politician John Van Dyke died near the confluence of which two rivers?,the Mississippi River near its confluence with the Chippewa River,location,which,geography
+"Who attacked American Airlines Flight 444, in a nationwide bombing campaign that targeted people involved with modern technology?",Theodore John Kaczynski,person,who,history
+The Athletics at the 1982 Asian Games were held in a stadium named after which individual?,first Prime Minister of India,person,which,sports
+Kew Cottages is located in the eastern suburb of Melbourne how far away from the Central Business District?,6 km east,distance,how far,geography
+"Which band was formed first,  Let's Active or The Jesus and Mary Chain?",Let's Active,band,which,music
+What genre of music do Kiwi Time and Ultra Vivid Scene share?,rock,genre,what,music
+"Cory Devela trains at Victory Athletics in Yelm, Washington with a veteran fighter born in what year?",1975,year,what,sports
+"As of 2016, which band released more albums, The Last Shadow Puppets or The Classic Crime?",The Classic Crime,band,which,music
+"Myosotis is a genus of the Boraginaceae family, whereas Dypsis is a genus of which family?",Arecaceae,family,which,biology
+Gorgidas was the first known Theban military leader of of troop of select soldiers annihilated in what battle?,the Battle of Chaeronea,battle,what,history
+What are two mood disorders that Thalia Catherine Eley would come across in her study of depression?,depressive disorder and dysthymia,disorders,what,health
+What industry were Ransom Riggs and Seth Savoy both a part of?,American film,industry,what,film
+Natalia Bestemianova performed with her partner in winning what competition five times?,European champion,competition,what,sports
+"Where was the actress who starred next to Stuart Whitman in the movie ""The Fiercest Heart"" raised after World War II?",South Africa,location,where,film
+In what city was American professional basketball player Jason Paul Collins drafted onto the Houston Rockets?,Houston,city,what,sports
+The Celebrity Apprentice 6 cast included an American political aide in the White House who  received a bachelor's degree in broadcast journalism from where?,Central State University,university,where,television
+Who directed a 2011 British-American film in which Frank Anthony Grillo appeared?,Joe Carnahan,person,who,film
+"Danny Wallace won one cap for which team, controlled by The Football Association?",England national football team,team,which,sports
+"""God Made Girls"" is a song recorded by a American country music artist who was born when?",1994,year,when,music
+What year was the novel written which the 1931 film Huckleberry Finn was based upon?,1884,year,what,literature
+Augustin Thompson created a soft drink produced by a company located in which city ?,"Bedford, New Hampshire",city,which,business
+Which Finnish former boxer shares nickname with a Ugandan political leader and military officer?,Amin Asikainen,person,which,sports
+The Tariff of 1833 was proposed by the statesman who first served three non-consecutive terms in what post?,Speaker of the House,position,what,politics
+Who was dubbed the father of the type of rock music that emerged from post-punk in the late 1970s?,Brian Healy,person,who,music
+What is the actor who starred in School Spirit best known for?,surgeon Major Frank Burns,character,what,film
+What technology used to be how the school that Adelaide Miethke conducted classes?,radio,technology,what,education
+Were both Time Warner Center and 1095 Avenue of the Americas over 700ft tall?,no,boolean,were,geography
+Which television series is set in a teaching hospital and began airing on ABC in 2009?,Scrubs,television show,which,television
+The 2011 Nobel Prize in Physics was partially awarded to which professor of physics from UC-Berkeley?,Saul Perlmutter,person,which,science
+"The Who's rendition of ""Baba O'Riley"" contains what unique quality?",without lyrics,quality,what,music
+DesiHits raised venture capital in 2008 from which American firm?,Draper Fisher Jurvetson,company,which,business
+What is the capital of the country in which Lokovina is located?,Ljubljana,capital,what,geography
+The male lead of the show Singer & Sons was nominated for Emmy Awards how many times?,five times,number,how many,television
+which English model and actress compete with other to find  'the face' of Max Factor,Naomi Campbell,person,which,fashion
+How many square feet of gambling space does the casino Ralph Weston Sturges helped build have?,364000,area,how many,business
+What state contains a mountain that had the face of the 26th United States President carved into it?,South Dakota,state,what,geography
+What airline does Highest Duty: My Search for What Really Matters and Chesley Sullenbergerhave in common?,US Airways,airline,what,aviation
+"In what city is the house that was once home to the Republican Representative from the state of Mississippi from 1875 to 1881, which is now a National Historic Landmark?","Washington, D.C.",city,what,geography
+"Which gold mine has a longer history, Thompson-Lundmark Mine or Lupin Mine?",Thompson-Lundmark Mine,mine,which,geography
+Are Bolling v. Sharpe and Whole Woman's Health v. Hellerstedt both cases dealing with personal rights?,yes,boolean,are,law
+Did Pussy Galore start their career before Jack's Mannequin?,"Pussy Galore was an American garage rock band that formed in Washington, D.C. in 1985",band,did,music
+"What is the name of the character in the television series ""Rob & Big"" who was the best friend of Rob Dyrdek and who was born on January 13, 1972?","Christopher ""Big Black"" Boykin",person,what,television
+Chris Stewart plays for a professional ice hockey team that is the only Minneapolis-St. Paul area major professional sports league franchise to play in what city?,St. Paul,city,what,sports
+Juan Perez de Zurita arrived at the capital that lies on which river in 1558?,Dulce,river,which,geography
+"What is the full job title of Desmond Wilkinson Llewelyn's character in the ""James Bond"" films?",Quartermaster,job title,what,film
+"Which actor known for his roles on the television shows ""Party of Five"" and ""7th Heaven"" also stared in a 2001 film based off of William Shakespeare's ""Othello""?",Andrew Keegan,person,which,film
+Globe Arena is a stadium used by which team that plays in League Two?,Morecambe Football Club,team,which,sports
+"Besides a flower delivery network, what other organization is best known by its telegraphic address?",Interpol,organization,what,business
+"What television series ran for two seasons from September 1997 to March 1998, and was based on Walt Disney's adaptation of Dodie Smith's 1956 novel ""The Hundred and One Dalmatians""?",101 Dalmatians: The Series,television show,what,television
+Matthew Trupiano was the boss of which St. Louis group?,Giordano crime family,group,which,history
+The Dark Side of the Sun stars an American actor and producer who has his own company named what?,Plan B Entertainment,company,what,film
+Which airport's main aircraft and maintainence base was a public airport six miles west of Downtown Tampa?,"North American Airlines, Inc.",company,which,aviation
+"Are Emeishan City and Chaoyang, Liaoning both located in China ?",yes,boolean,are,geography
+Buraswa is a village that comes under the Tehsil of Chakrata in the capital city that is located in what region of India?,Garhwal,region,what,geography
+When was the governor who appointed Lana Gordon first elected governor?,2010,date,when,politics
+"Which exhiled Russian businessman acquired a Russian owned oil and gas company based in Moscow, Russia?",Mikhail Borisovich Khodorkovsky,person,which,business
+Kim Wall's body washed up on the beach of an island with how many inhabitants ?,"almost 200,000 inhabitants",number,how many,geography
+"James Patrick Caviezel played the role of ""Edmond Dantès"" in a movie directed by who?",Kevin Reynolds,person,who,film
+"Which novelist lived longer, Booth Tarkington or Theodore Dreiser?",Booth Tarkington,person,which,literature
+In what year was the right wing of the West Coast Express born?,1975,year,what,sports
+"Which battle lasted longer, the Battle of Tarawa or the Battle of Leyte?",The Battle of Leyte,battle,which,history
+What was the nickname of the family that owned Ashley Park around the turn of the 20th century?,Rothschilds of the East,nickname,what,history
+"What division of the American armed forces fought the Imperial Japanese Army, also know as the ""Dai-Nippon Teikoku Rikugun"", or ""Army of the Greater Japanese Empire"", at Bougainville during World War Two?",Marine Corps,division,what,military
+NanoScale Corporation was founded to develop properties of a university with a main campus in what Kansas city?,Manhattan,city,what,business
+The Worst Journey in the World is based on a memoir by an English explorer that was born in what year?,1886,year,what,literature
+Are both Jimmie Ross and Jang Hyun-seung Americans?,no,boolean,are,music
+Two Lanes of Freedom included a cameo by which other male country singer on guitar?,Keith Urban,person,which,music
+Hiroyuki Tamakoshi has illustrated for the manga created and written by whom?,Masahiro Itabashi,person,whom,literature
+"8 Heads in a Duffel Bag features Joe Pesci and what former ""Saturday Night Live"" and ""Joe Dirt"" star?",David Spade,person,what,film
+"What is the name of the Football League Cup sponsored by a bank holding company, headquartered in Mclean, Virginia?",the Capital One Cup,event,what,sports
+Are Tampa International Airport and King Salmon Airport both located in the U?,yes,boolean,are,geography
+Did the inspiration for Transformers or Shadow Hunters come first?,1986 board game,item,did,entertainment
+Where did the fictional character portrayed in Ragnarok film first appreared?,Journey into Mystery,publication,where,film
+"What highest civilian award was the actress that starred in Yaraana, along side Rishi Kapooy, Raj Babbar, Kader Khan, and Shakti Kapoor, awarded by the Government of India in 2008?",Padma Shri,award,what,film
+What is a block away from the arena where the Baltimore Blast play their games?,Baltimore Convention Center,place,what,geography
+Jean Negulesco and Colin Campbell are both film directors and screenwriters but which is also an actor?,Colin Campbell,person,which,film
+"Tallinn Airport is a secondary hub for two airlines, one of which established in 1929. What airline is this?",LOT Polish Airlines,airline,what,geography
+Are the Dogo Argentino and the Scottish Deerhound large or small dogs?,large,size,are,animals
+Sir Oliver Bury Popplewell chaired the inquiry into a stadium fire that killed how many people?,56,number,how many,history
+Who co-wrote the sci-fi adventure drama film High Life with Nick Laird?,Claire Denis,person,who,film
+What river goes by the oldest continuously occupied castle on mainland Scotland?,River Add,river,what,geography
+When did the civilisation start that Desalpar Gunthli was a part of?,3300–1300 BCE,date,when,history
+Jimmy Eriksson will race in the last 2016 GP2 Series before it gets renamed,FIA Formula 2 Championship,event,before what,sports
+In which region of France was the opera Sabinus first performed ?,the Île-de-France region of France,region,which,music
+"Satoru, a Japanese verb meaning ""to know"" or ""understand"", is the root of the Zen Buddhist word meaning what?",comprehension; understanding,meaning,what,religion
+X Games XV It took place at a multi-purpose sports arena that opened on what date?,"17, 1999",date,what,sports
+"Which university is a private university, Massachusetts Institute of Technology or University of Missouri System?","The Massachusetts Institute of Technology (MIT) is a private research university in Cambridge, Massachusetts.",university,which,education
+"Where is this late post-classic Mexica sculpture, which Antonio de León y Gama wrote an essay about at the time the sculpture was discovered, housed?",National Anthropology Museum,location,where,history
+Are Physocarpus and Rudbeckia both genus of flowering plants ?,yes,boolean,are,biology
+What date were several dozen Jewish men executed during a program in which Algirdas Klimaitis was famous for his role in?,June 27,date,what,history
+Who was born first John Wellborn Root or  Daniel Hudson Burnham?,Daniel Hudson Burnham,person,who,history
+What were the prototype Big Daddy characters called in the sequel version of the BioShock series?,Subject Delta,character,what,gaming
+On what campus did Group B of the 2015 CONCACAF Gold Cup play on July 8?,"California State University, Dominguez Hills",campus,what,sports
+Which biblical figure is engraved upon the gem named for the son of King Lohair I?,Susanna,person,which,religion
+"What year did the football player born on November 3, 1955 retire?",1993,year,what,sports
+The former owner of Fulham F.C. started a film production company in what decade?,in the late 1970s.,decade,what,business
+Guy Walters competed against which former professional boxer that is a two-weight world champion?,Virgil Hill,person,which,sports
+"Which company has more employees, Becton Dickinson or American Airlines Group?",American Airlines Group,company,which,business
+"Ross Matthews and Carson Kressley filled the space on ""RuPaul's Drag Race"" previously occupied by which fashion designer who appears on ""Project Runway""?",Santino Rice,person,which,fashion
+Are both Charles Shyer and Christopher Nolan film directors?,yes,boolean,are,film
+What is the population size of the place where Shane O'Neill refused to help the English against the Scottish settlers?,"618,000",population,what,history
+Change is Now: Renewing America's Promise includes previously released tracks from an American hip hop recording artist who debuted in 1992 with what album?,Can I Borrow a Dollar?,album,what,music
+The founding team of ZDNet features a veteran from what multinational automotive company?,Jaguar Land Rover,company,what,business
+"When the ""The Apprentice"" spin-off starring Elijah Connor premiere?",2010,date,when,television
+" Cherry Juice was originally serialized in a monthly ""shōjo"" magazine with a target demographic of what?",girls of age 8–14,demographic,what,literature
+"Though he won an Academy award for his short film biography of a 19th centry English engineering giant, this animator is also known for advirtisements of what product in the UK in the early 1980s?",Trio chocolate biscuit,product,what,advertising
+"Who produced the album that included a re-recording of ""Lithium""?",Butch Vig,person,who,music
+Did Lee Hong-gi and Jung Yong-hwa released their first solo albums in the same year?,yes,boolean,did,music
+What waterfall is located downstream of Wiesendanger Falls along the Historic Columbia River Highway?,Multnomah Falls,place,what,geography
+T. M. Thiagarajan was awarded the title to an expert in what discipline by the Madras Music Academy?,Carnatic Musician,discipline,what,music
+Where is the online company founded in 1999 by Nick Swinmurn based?,"Las Vegas, Nevada",location,where,business
+"Of the two presidents who served during the heyday of the ""Kansas City Star,"" which one did NOT use nuclear weapons in war?",Dwight D. Eisenhower,person,which,history
+In 1992 Tom and Ray Magliozzi won a Peabody Award for their show on what topic?,automobiles and automotive repair,topic,what,radio
+The video game 3D Bomb Alley is based on a Falklands War battle that ended on what date?,25 May 1982,date,what,gaming
+"Between Al-Nayrizi and Shahid Hussain Bokhari, who was born first?",al-Nairīzī,person,who,history
+What king held Aketorpe in 1086?,William the Conqueror,person,what,history
+"""Started with a Song"" surpassed a record set by what song that was released as the lead single from ""Red"" ",We Are Never Ever Getting Back Together,song,what,music
+Kevin Hales played as a midfielder for Leyton Orient and a football club founded in what year?,1905,year,what,sports
+"Which magazine was geared towards teenagers, Modern Screen or Seventeen?",Seventeen,magazine,which,media
+Tammy Blanchard performed in which musical loosely based on a famous striptease artist?,Gypsy,musical,which,theater
+What country did Frank Jardine represent at the V Olympic Winter Games?,Great Britain,country,what,sports
+Which of Syd Mead design was directed by Ridley Scott?,Blade Runner,design,which,film
+"Which Chinese city has more inhabitants, Gejiu or Dujiangyan City?",Dujiangyan,city,which,geography
+"Which film director has had the most notable films, Emile Ardolino or Seijun Suzuki?",Seijun Suzuki,person,which,film
+"Uwe Seeler, professional footballer, appeared in a comedy called ""Willi Manages The Whole Thing"", his grandson plays football. What is his name?",Levin Öztunalı,person,what,sports
+Which team did the son of head coach of the Tulane University men's basketball team play for early in his career?,Milwaukee Bucks,team,which,sports
+What did the Lichfield Cricket Club nickname itself after?,Lichfield Cathedral,place,what,sports
+What year was Sara Alstrom's sister born?,1981,year,what,people
+The Sussex Spaniel and English Cocker Spaniel are both what?,dog,animal,what,animals
+"What is the name of the ski area located in Franklin County, Maine?",Sugarloaf,place,what,geography
+"Fortunat Strowski, a French literary historian superintended critical editions of works published between what years?",1570 to 1592,date range,between what,literature
+In what country was the father of Thorvald Eiriksson born?,Norway,country,what,history
+What operational unit does Frederick Tees and The Dam Busters have in common?,Squadron,unit,what,military
+Pop Wea was associated with a Native American tribe that speaks what language?,Tiwa,language,what,history
+what was the number of Tongan athletes that competed between after 2008 in 2011 commonwealth Youth Games?,three,number,what,sports
+The Bell X-1 was an aircraft piloted at a supersonic level by the test pilot who served in what branch of the armed forces?,United States Air Force,branch,what,military
+"Until 2006, the Nissan Sentra was a rebadged export version of a Japanese car that was first built in what year?",1966,year,what,business
+What day in March 2012 did Jason Russell's viral video release?,"March 5, 2012",date,what,events
+"Who created the American sitcom which  ""Nothing Good Happens After 2 A.M."" is the 18th episode in the first season of",Carter Bays and Craig Thomas,person,who,television
+How long has the league that Eray Cümart plays football in played in its current format?,since the 2003–04 season,duration,how long,sports
+What Manhattan college was Theodore Shapiro a professor emeritus at?,Weill Cornell Medicine,place,what,education
+"What is the birthdate of this soprano singer in opera and musical theater, who sang her first solo guided by this Dutch violinist and conductor best known for creating the waltz-playing Johann Strauss Orchestra?",1 March 1978,date,what,music
+What political party stopped the construction of the Thames Gateway Bridge,the Conservative Party,organization,what,politics
+Are Joe Elliott and Paul Meany both keyboardists?,no,boolean,are,music
+Thomas Carleton lived in a village on the coast of what sea?,Irish Sea,sea,what,geography
+"Who was born first, Griffith R. Harsh or Meg Whitman?",Griffith Rutherford Harsh,person,who,people
+How did Henry Dangar participate in the European exploration of Australia?,surveyor and explorer,role,how,history
+Ghassan Shakaa was appointed to the mayoral position by which political leader?,Mohammed Yasser Abdel Rahman Abdel Raouf Arafat al-Qudwa,person,which,politics
+The 1985 European Grand Prix was held in which town?,"West Kingsdown in Kent, England",place,which,sports
+"Which book inspired the film that contains Rob Thomas's 2007 single ""Little Wonders?""",A Day with Wilbur Robinson,book,which,film
+In this 2002 film the Apollo 11 Moon landing was faked and recorded in a studio by the CIA with help from a director whose film are mostly adaptations of what?,novels or short stories,works,what,film
+Singwango Secondary School had a sister school in a census-designated place in the town of North Hempstead in what county of New York?,Nassau County,place,what,education
+Antoine Fuqua and Duncan Jones both have what job?,film director and producer,job,what,film
+"What is the camp that is joined by a base previously named in honor of pilot and Boston surgeon, Lt. Frank ""Jesse"" Otis.",Camp Edwards,place,what,military
+Janina Vilayet was a division created by merging a subdivision of the Ottoman Empire known as what?,Pashalik of Yanina,entity,what,history
+Marie-Barbe de Boullongne was a philanthropist on what continent?,North America,place,what,history
+Which computer sold in 1985 ran Aztec C?,The Amiga,device,which,technology
+"A 1955 film by Giorgio Capitani grossed 146 million lire, and was the currency in Italy during which time period?",1861 and 2002,date range,which,history
+What is the elevation of the stadium where the 2006 Las Vegas Bowl was played?,1600 ft above sea level,elevation,what,sports
+Which american president's father was a farmer and he was a former US representative from NY?,Millard Fillmore,person,which,politics
+The largest resevoir in the United States played host to the first Bassmaster Classic during what year?,1971,date,what,sports
+"Where is the crowdfunding corporation that made Allie Goertz's ""Sad Dance Songs"" album possible based?","Brooklyn, New York",place,where,business
+"What small village is located approximately 2 miles from a village and civil parish in the East Lindsey district of Lincolnshire, England?",Hogsthorpe,place,what,geography
+I luv It was released via what South Korean entertainment company that was established in 1996 by Yang Hyun-suk?,YG Entertainment Inc.,company,what,music
+"Who is known for their work as an actor, Adam G. Sevani or Gary Braver?",Adam G. Sevani,person,who,entertainment
+What were the Black Russian Terrier and the Samoyed dog breeds used for?,"working dog, guarding dog, sporting and companion dog.",purpose,what,animals
+Where was the director of Passchendaele born?,"Calgary, Alberta",place,where,film
+Who was the Medal of Honor recipient who has a cutter named after him?,Douglas Albert Munro,person,who,military
+Which has more people: Kuytun or Qitaihe?,Qitaihe,place,which,geography
+When did the reality singing competition where a Dutch rapper of Moroccan descent appeared as a judge air?,It aired during 2014,date,when,television
+The song All Lies by hip hop artist Charlie Baltimore features which rapper on the final recording?,Maino,person,which,music
+Rhoda Gemignani appeared on multiple sitcoms produced by what company?,Warner Bros.,company,what,television
+"In the year Ria van Dyke competed, who won Miss Universe?",Ximena Navarrete,person,who,events
+Task Force Faith and Operation Undertone were both components of which branch of the U.S. military?,United States Army,branch,which,military
+What is the name of a  kit violin and an opera?,L'Orfeo,item,what,music
+What was the middle name of the husband of Sylvia Fine?,Daniel,name,what,people
+What voice type does Sonny Bono's collaborator on the album Chér have?,contralto,attribute,what,music
+The album titled Omo Baba Olowo by Davido was produced by who?,Maleek Berry,person,who,music
+Which Mary Tyler Moore Show actress guest starred in the episode of an NBC comedy which also featured the final apprearance of Lindsey Broad?,Georgia Engel,person,which,television
+The Hills Have Thighs was based on the movie The HIlls Have Eyes that was released on what date?,22 July 1977,date,what,film
+What is the name of the city where English Rock band Elbow produced 'Leaders of the Free World' and is also where the city of Salford is located?,Greater Manchester,place,what,music
+Warner Bros distributed what musical comedy film in which Carmen Twillie was one of the singers?,Cats Don't Dance,film,what,film
+"Which magazine was produced later, Missbehave or Glamour?",Missbehave,magazine,which,media
+Which cricketer debuted during the 2008-09 season and was suspended in 2017 for spot-fixing?,Mohammad Shahzaib Hasan Khan,person,which,sports
+"How many miles is the city that is the largest in the Gibson County from Jackson, Tennessee?",23 miles,distance,how many,geography
+Jack Gold was part of a tradition whose protagonists usually could be described as what?,angry young men,description,what,film
+"What band that Mike Peters worked with was formed in Dunfermline, Fife, in 1981?",Big Country,band,what,music
+the NASA space radiation Laboratory and the National Weather Service weather forecast office are both located in what part of New York?,Long Island,place,what,geography
+"What was the another recognition  of one of the authors of Mexican literature who born in March 31, 1914?",poet and diplomat,recognition,what,literature
+Are Adrienne Barbeau and Robert Zildjian both singers?,no,boolean,are,music
+" ""Stous 31 Dromous"" is a song by a recording artist who began his musical career in what year?",1991,year,what,music
+What two skills do Lee Hong-gi and Dee Snider have in common?,"singer-songwriter, actor",skills,what,music
+Big Dan Mine and Copperfields Mine are both located in?,"Ontario, Canada",place,where,geography
+The Fridge had booked such acts as the English singer and DJ who was the lead singer of what pop band?,Culture Club,band,what,music
+Gadsden Mall is anchored by Belk and a company founded in what year?,1886,year,what,business
+"The Sagittarius Dwarf Spheroidal Galaxy, a loop-shaped galaxy of the Milky Way, is which type of the three main classes of galaxy originally described by Edwin Hubble in ""The Realm of the Nebulae?""",An elliptical galaxy,type,which,astronomy
+Who was the county named for in which the unincorporated community call Cockrell is located?,President Andrew Jackson,person,who,geography
+"What was a film that the actor of ""Out of the Past"" made that was based on the memoirs of Norwegian resistance soldier Knut Haukelid?",The Heroes of Telemark,film,what,film
+"Which Austrian composer wrote Symphony No. 35 dated December 10, 1767?",Joseph Haydn,person,which,music
+What honor in baseball do both Delmer Ennis and Stan Musial hold?,RBIs,honor,what,sports
+"what is the 3112 acre Pennsylvania state park that took over 11,000 to make due to the wave action of lake Erie called?",Presque Isle,place,what,geography
+What is the average weekly attendance at the church founded by the author of The Purpose Driven Life ?,"over 20,000",number,what,religion
+"What is the name of this King of England, born in June 1239, in which the Prioress swore fealty to him, after his foray in Scotland in 1296?",Edward I of England,person,what,history
+What was the first single on the Taylor Swift album which also contained a song originally written for Swift's high-school talent show?,"""Tim McGraw""",song,what,music
+Louisburg is a city in which county located in east-central Kansas?,Miami County,place,which,geography
+"In regards to the Charlotte Hornets' all time leader in points, who's number hangs in the rafters next to his at Virginia Tech?","Adrian ""Ace"" Custis",person,whose,sports
+Who did the 47th Governor appoint as a justice of the Supreme Court of Texas?,Judge Debra Lehrmann,person,who,politics
+"Which of the stars in ""Home Sweet Hell"" was born on November 24, 1978?",Katherine Heigl,person,which,film
+"Which canal was completed first, Dundee Canal or Cascade Locks and Canal?",Dundee Canal,entity,which,geography
+"Thaba Putsoa is a mountain in the Maseru District of which enclaved, landlocked country in southern Africa completely surrounded by South Africa?",Lesotho,country,which,geography
+Do Crinodendron and Nuphar belong to the same family?,no,boolean,do,biology
+The Star Wars Launch Bay interactive walkthrough attraction replaced The Magic of Disney Animation at which Disney park in 2015?,Disney's Hollywood Studios,place,which,entertainment
+"Apart from the founder of the Exclusive Brethren, what is the name of the second person who was sent by the Plymouth Brethren to investigate the outbreak of supernatural gifts of the Holy Spirit in Scotland?",Benjamin Wills Newton,person,what,religion
+"How did the surfer die, who was a member of the Kahanamoku Surf Team, and who was best known for surfing in areas such as one Hawaiian bay who's name means ""reddish water""?",alcohol-related illness,cause of death,how,sports
+Stan Brakhage and Frank Powell are known for what kind of films?,silent film,genre,what,film
+"Which university has more alternate names, Colorado State University or Alexandria University?",Colorado State University,institution,which,education
+How long did a series of 6 erotic feature films run after being ushered in by the 1965 Danish coming-of-age sex comedy;Sytten?,from 1973–78,duration,how long,film
+Celebration was recorded by the singer who was a leading presence when what cable channel premiered?,MTV,channel,what,music
+"Radagast was featured in the Silmarillion, which was published in what year?",1977,year,what,literature
+What role did Kunie Tanaka play in a film entered into the 11th Berlin International Film Festival?,thug,role,what,film
+"Bobby Cox, born in 1941, lead the Atlanta Braves 44th season but how many season did the team have overall?",139th,number,how many,sports
+What ideology is opposed by a british political party which stems from a branch of the men's liberation movement of the early 1970s?,feminism,ideology,what,politics
+"Which president-elect did the French international cabaret singer Édith Piaf sing for in 1952, performing ""Simply a Waltz""?",Eisenhower,person,which,history
+Are John Schlesinger and Mrinal Sen both English film directors?,no,boolean,are,film
+Which nationality belongs to the writer of Eiger Dreams: Ventures Among Men and Mountains?,American,nationality,which,literature
+"The Kooskia National Fish Hatchery raises and releases the Oncorhynchus tshawytscha, which his the scientific species name of what fish?",Chinook salmon,species,what,biology
+"What singer, best known for his recording of the song ""Raindrops Keep Fallin' on My Head"", had his single ""Most of All"" reach #2 on the ""Billboard"" Adult Contemporary singles chart in 1970?",B. J. Thomas,person,what,music
+"What was the beginning year of the European debt crisis that particularly affected Portugal, Italy, Greece, and Spain?",2009,year,what,history
+Who is Claudio Ranieri in reference 1996–97 A.C. Fiorentina season?,head coach,role,who,sports
+Were Subhash Ghai and Nora Ephron both in the film industry?,yes,boolean,were,film
+"Joe Johnston is an American film director best known for such effects-driven movies as ""Honey, I Shrunk the Kids"" (1989), ""Jumanji"" (1995) and what other movie?","""Jurassic Park III"" (2001)",film,what,film
+"Which drink would be more likely to be served at a wedding, Mimosa or Blow Job?",Mimosa,item,which,events
+"In which state was the black farmhand lynched in 1916 in with over 10,000 spectators, that prompted Silent Parade with about 10,000 African Americans along Fifth Avenue starting at 57th Street in New York City the next year?",Texas,state,which,history
+What type of judgement does Boston and Sacco and Vanzetti have in common?,execution,judgment,what,history
+"An american engineer born February 14, 1859 observed a wheel as a student located where?","Troy, New York.",place,where,science
+Are the bands LCD Soundsystem and Against the Current based in the same New York cities?,no,boolean,are,music
+The magazine whose was joined by Richard Taylor was founded by a Scottish journalist born in what year?,1759,year,what,media
+Which movement has its cradle in the neighborhood where the tavern Julius can be found?,the modern LGBT movement,movement,which,history
+"Who was born first, Marie Fredriksson or Juha-Pekka Leppäluoto?",Gun-Marie Fredriksson,person,who,music
+Where did the father of the 1796 ori tempore of the United States begin practicing law?,"Concord, New Hampshire",place,where,history
+Linda Hughes was educated at the university that was founded in what year?,1963,year,what,education
+Lewis Moran was murdered a week after the death of the hitman from what Australian city?,Melbourne,place,what,crime
+How many miles away from Dallas is the village that the species depicted in the Pictish stone at Golspie originated from?,three miles,distance,how many,geography
+What epistles include references to key episodes of the life of the person who is the focus of Cento vergilianus de laudibus Christi?,Pauline epistles,document,what,religion
+where is the American professional basketball was co owned by Clay Bennett based,"Oklahoma City,",place,where,sports
+Which genus is part of the mallow family; Ferraria or Abelmoschus?,Abelmoschus,genus,which,biology
+What is the name of the grandma who is featured in the 2016 film starring Cha Hak-yeon?,Moon Hyung-joo,person,what,film
+The InterContinental Bucharest is a highrise five star hotel that is related to what other Hotel brands?,"Candlewood Suites, Crowne Plaza, EVEN Hotels",brands,what,business
+What type of technology is  Pecom 32 and Apple IIc Plus,computers,technology,what,technology
+Edmund Veesenmayer served under a senior the highest ranking officer in which part of the Nazi German army?,the SS,branch,which,history
+"What is the speciality of the English man who studies the French composer who wrote the motet ""Illibata Dei virgo nutrix""?",English musicologist,specialty,what,music
+Which actress played the role of Evelyn Thompson in the film 2000 film directed by Brett Ratner?,Lisa Thornhill,person,which,film
+"Which large mammal that appears in the engraving for the Wildebeest Kuil Rock Art Centre, is one of any five extant species of odd-toed ungulates in the family Rhinocerotidae?",rhinoceros,animal,which,biology
+Alan Forbes has done posters for an American rock band that formed in 1996 in what city in California?,Palm Desert,place,what,music
+"Where is the base of a corporation that has a presence in Bomen, New South Wales?","Minnetonka, Minnesota",place,where,business
+Do the dog breeds Patterdale Terrier and Shikoku both originate in Japan?,no,boolean,do,biology
+"Which was fought in Europe, the Battle of the Wilderness or the Meuse-Argonne Offensive?",Meuse-Argonne Offensive,battle,which,history
+What is the nickname of the highway that intersects with State Route 116 near a ghost town in Churchill County?,The Loneliest Road,nickname,what,geography
+How many residents are there in the State in which Brush Creek is located ?,over six million,number,how many,geography
+Where is the main campus of the private Tusculum College?,"Tusculum, Tennessee",place,where,education
+What gender does Steve Sarkisian and Mike Connors have in common?,He,gender,what,people
+Are both PC Magazine and Kitelife magazines dedicated to computers?,no,boolean,are,media
+Are Calochortus and Carludovica both native to the United States?,no,boolean,are,biology
+"Danny Burstein starred in Indignation in 2016, what else did he star in that year?",Fiddler on the Roof,film,what,film
+"What is the year of the event which occured first, Crystal Waters was released, or Dennis Rodman was born?",1961,year,what,events
+"Which has more acts, Die Walküre or La fille du régiment?",Die Walküre,entity,which,music
+What is the annual revenue of American gaming corporation who owned Harrah's Ak-Chin Casino?,$8.6 billion (2013),amount,what,business
+Where is a type of beer that uses a process similar to American wild ale brewed?,Pajottenland region of Belgium,place,where,food
+Dharmatma included a title character based on a king of which Indian form of lottery?,Matka,game,which,culture
+What is the total area of the desert where Mount Manchester is located?,47877 mi2,area,what,geography
+Who is the male star of the film that closed The 27th Toronto International Film Festival ?,Antonio Banderas.,person,who,film
+Which Wisconsin Badgers player won a Heismann trophy and was elected to the Pro Bowl in each of their first four seasons?,Alan Ameche,person,which,sports
+"Which singer died at the age of thirty, Shaun Morgan or Dave Williams?",David Wayne Williams,person,which,music
+Mount Jefferson is near the mountain to its northeast that has what elevation?,5793 ft,elevation,what,geography
\ No newline at end of file
diff --git a/testing/optimizer_tester.py b/testing/optimizer_tester.py
index 68d394601..841512472 100644
--- a/testing/optimizer_tester.py
+++ b/testing/optimizer_tester.py
@@ -9,25 +9,53 @@
 import dspy
 from dspy.evaluate import Evaluate
 
-from .tasks.biodex import BioDexTask
-from .tasks.gsm8k import GSM8KTask
-from .tasks.hotpotqa import HotPotQATask
-from .tasks.scone import ScoNeTask
-from .tasks.tweet import TweetTask
-from .tasks.tweet_metric import TweetMetricTask
+from tasks.gsm8k import GSM8KTask
+from tasks.hotpotqa import HotPotQATask
+from tasks.scone import ScoNeTask
+from tasks.tweet import TweetTask
+from tasks.tweet_metric import TweetMetricTask
+from tasks.heart_disease import HeartDiseaseTask
+from tasks.hotpotqa_conditional import HotPotQAConditionalTask
+from tasks.hover import HoverRetrieveDiscrete
+from tasks.iris_typo import IrisTypoClassifierTask
+from tasks.iris import IrisClassifierTask
+
+datasets = [
+    "scone",
+    "hotpotqa",
+    "hotpotqa_conditional",
+    "gsm8k",
+    "tweet",
+    "heart_disease",
+    "iris",
+    "iris_typo",
+    "hover_retrieve_discrete",
+    "tweet_metric",
+]
 
-datasets = ["ScoNe", "HotPotQA", "GSM8K", "BioDex", "Tweet"]
 
 class OptimizerTester:
-    def __init__(self, datasets=datasets, default_train_num = 200, default_dev_num = 100, default_test_num = 200,
-                 num_threads = 10, default_breadth = 10, default_depth = 3, default_temperature = 1.4,
-                 prompt_model_name = "gpt-3.5-turbo-1106", task_model_name = "meta-llama/Llama-2-13b-chat-hf",
-                 prompt_model = None, task_model = None,
-                 colbert_v2_endpoint = "http://20.102.90.50:2017/wiki17_abstracts"):
+    def __init__(
+        self,
+        datasets=datasets,
+        default_train_num=200,
+        default_dev_num=100,
+        default_test_num=200,
+        num_threads=10,
+        default_breadth=10,
+        default_depth=3,
+        default_temperature=1.1,
+        prompt_model_name="gpt-3.5-turbo-1106",
+        task_model_name="meta-llama/Llama-2-13b-chat-hf",
+        prompt_model=None,
+        task_model=None,
+        max_errors=100,
+        colbert_v2_endpoint="http://20.102.90.50:2017/wiki17_abstracts",
+    ):
         self.datasets = datasets
         self.TRAIN_NUM = default_train_num
         self.DEV_NUM = default_dev_num
-        self.EVAL_NUM = default_test_num
+        self.TEST_NUM = default_test_num
         self.NUM_THREADS = num_threads
         self.BREADTH = default_breadth
         self.DEPTH = default_depth
@@ -35,21 +63,28 @@ def __init__(self, datasets=datasets, default_train_num = 200, default_dev_num =
         self.PROMPT_MODEL_NAME = prompt_model_name
         self.TASK_MODEL_NAME = task_model_name
         self.COLBERT_V2_ENDPOINT = colbert_v2_endpoint
+        self.MAX_ERRORS = max_errors
 
         load_dotenv()  # This will load the .env file's variables
 
-        openai.api_key = os.environ.get('OPENAI_API_KEY')
-        openai.api_base = os.environ.get('OPENAI_API_BASE')
+        openai.api_key = os.environ.get("OPENAI_API_KEY")
+        openai.api_base = os.environ.get("OPENAI_API_BASE")
 
         # Prompt gen model
         if not prompt_model:
-            self.prompt_model = dspy.OpenAI(model=self.PROMPT_MODEL_NAME, max_tokens=150)
+            self.prompt_model = dspy.OpenAI(
+                model=self.PROMPT_MODEL_NAME, max_tokens=700
+            )
         else:
             self.prompt_model = prompt_model
 
         # Task model
         if not task_model:
-            self.task_model = dspy.HFClientTGI(model=self.TASK_MODEL_NAME, port=[7140, 7141, 7142, 7143], max_tokens=150)
+            self.task_model = dspy.HFClientTGI(
+                model=self.TASK_MODEL_NAME,
+                port=[7140, 7141, 7142, 7143],
+                max_tokens=150,
+            )
         else:
             self.task_model = task_model
         self.colbertv2 = dspy.ColBERTv2(url=colbert_v2_endpoint)
@@ -60,21 +95,47 @@ def write_to_csv(self, folder_name, file_name, data):
         # Ensure the output directory exists
         os.makedirs(folder_name, exist_ok=True)
         file_path = os.path.join(folder_name, file_name)
-        
+
         # Check if file exists to determine if headers should be written
         file_exists = os.path.isfile(file_path)
 
-        with open(file_path, mode='a', newline='', encoding='utf-8') as file:
+        with open(file_path, mode="a", newline="", encoding="utf-8") as file:
             writer = csv.writer(file)
 
-            headers = ['test_name', 'train_score', 'dev_score', 'eval_score', 'run_time (sec)', 'train_size', 'dev_size', 'eval_size', 'task_name', 'signature_optimized', 'prompt_model_name', 'task_model_name', 'breadth', 'depth', 'meta_prompt_style', 'fewshot_before', 'fewshot_after', 'temperature', 'fewshot_candidates_num', 'max_bootstrapped_demos', 'bootstrapping', 'view_data', 'optimizer_log_dir', 'additional_notes', 'misc']
-            
+            headers = [
+                "test_name",
+                "train_score",
+                "dev_score",
+                "test_score",
+                "run_time (sec)",
+                "train_size",
+                "dev_size",
+                "test_size",
+                "task_name",
+                "signature_optimized",
+                "prompt_model_name",
+                "task_model_name",
+                "breadth",
+                "depth",
+                "meta_prompt_style",
+                "fewshot_before",
+                "fewshot_after",
+                "temperature",
+                "fewshot_candidates_num",
+                "max_bootstrapped_demos",
+                "bootstrapping",
+                "view_data",
+                "optimizer_log_dir",
+                "additional_notes",
+                "misc",
+            ]
+
             # Write headers if the file is being created
             if not file_exists:
                 writer.writerow(headers)
-            
+
             # Format the data
-            formatted_data = ['NA'] * len(headers)
+            formatted_data = ["NA"] * len(headers)
             formatted_data[-1] = ""
             for key in data.keys():
                 if key in headers:
@@ -84,124 +145,205 @@ def write_to_csv(self, folder_name, file_name, data):
             # Write the data
             writer.writerow(formatted_data)
 
-
-    def load_dataset(self,dataset):
+    def load_dataset(self, dataset):
         ds = None
         dataset = dataset.lower()
         if dataset == "scone":
             ds = ScoNeTask()
         elif dataset == "hotpotqa":
             ds = HotPotQATask()
+        elif dataset == "hotpotqa_conditional":
+            ds = HotPotQAConditionalTask()
         elif dataset == "gsm8k":
             ds = GSM8KTask()
-        elif dataset == "biodex":
-            ds = BioDexTask()
         elif dataset == "tweet":
             ds = TweetTask()
+        elif dataset == "heart_disease":
+            ds = HeartDiseaseTask()
+        elif dataset == "iris":
+            ds = IrisClassifierTask()
+        elif dataset == "iris_typo":
+            ds = IrisTypoClassifierTask()
+        elif dataset == "hover_retrieve_discrete":
+            ds = HoverRetrieveDiscrete()
         elif dataset == "tweet_metric":
             ds = TweetMetricTask()
         else:
             raise ValueError("Invalid dataset name.")
-        ds.set_splits(TRAIN_NUM=self.TRAIN_NUM, DEV_NUM=self.DEV_NUM, EVAL_NUM=self.EVAL_NUM)
+        ds.set_splits(
+            TRAIN_NUM=self.TRAIN_NUM, DEV_NUM=self.DEV_NUM, TEST_NUM=self.TEST_NUM
+        )
         return ds
-        
+
     # Computes baseline results for a given dataset
     def test_baseline(self, datasets=datasets, test_name="baseline"):
         for dataset in datasets:
+            print(f"Testing {dataset} Baseline LM Program...")
             task = self.load_dataset(dataset)
-            evaluate_train = Evaluate(devset=task.get_trainset(), metric=task.get_metric(), num_threads=self.NUM_THREADS, display_progress=True)
-            evaluate_dev = Evaluate(devset=task.get_devset(), metric=task.get_metric(), num_threads=self.NUM_THREADS, display_progress=True)
-            evaluate_eval = Evaluate(devset=task.get_evalset(), metric=task.get_metric(), num_threads=self.NUM_THREADS, display_progress=True)
+            dspy.settings.lm.max_tokens = task.get_max_tokens()
+
+            evaluate_train = Evaluate(
+                devset=task.get_trainset(),
+                metric=task.get_metric(),
+                num_threads=self.NUM_THREADS,
+                display_progress=True,
+                max_errors=self.MAX_ERRORS,
+            )
+            evaluate_dev = Evaluate(
+                devset=task.get_devset(),
+                metric=task.get_metric(),
+                num_threads=self.NUM_THREADS,
+                display_progress=True,
+                max_errors=self.MAX_ERRORS,
+            )
+            evaluate_test = Evaluate(
+                devset=task.get_testset(),
+                metric=task.get_metric(),
+                num_threads=self.NUM_THREADS,
+                display_progress=True,
+                max_errors=self.MAX_ERRORS,
+            )
             default_program = task.get_program()
 
             # Evaluate the default program
+            print(f"Train...")
             default_results_train = evaluate_train(default_program)
+            print(f"Dev...")
             default_results_dev = evaluate_dev(default_program)
-            default_results_eval = evaluate_eval(default_program)
+            print(f"Test...")
+            default_results_test = evaluate_test(default_program)
 
             # Write the results to a csv
-            self.write_to_csv('outputs', 'results.csv', {
-                'test_name': dataset + "_" + test_name,
-                'train_score': default_results_train,
-                'dev_score': default_results_dev,
-                'eval_score': default_results_eval,
-                'run_time (sec)': 0,
-                'train_size': len(task.get_trainset()),
-                'dev_size': len(task.get_devset()),
-                'eval_size': len(task.get_evalset()),
-                'task_name': dataset,
-                'signature_optimized': False,
-                'prompt_model_name': self.PROMPT_MODEL_NAME,
-                'task_model_name': self.TASK_MODEL_NAME,
-                'breadth': 'NA',
-                'depth': 'NA',
-                'meta_prompt_style': 'default',
-                'fewshot_before': False,
-                'fewshot_after': False,
-                'temperature': self.TEMPERATURE,
-                'fewshot_candidates_num': 0,
-                'max_bootstrapped_demos': 0,
-                'bootstrapping': False,
-                'view_data': False,
-                'optimizer_log_dir': 'NA',
-                'additional_notes': '',
-                'misc': '',
-            })
-
-    def test_optimizer_default(self, optimizer_function, datasets=datasets, test_name="default"):
+            self.write_to_csv(
+                "outputs",
+                "results.csv",
+                {
+                    "test_name": dataset + "_" + test_name,
+                    "train_score": default_results_train,
+                    "dev_score": default_results_dev,
+                    "test_score": default_results_test,
+                    "run_time (sec)": 0,
+                    "train_size": len(task.get_trainset()),
+                    "dev_size": len(task.get_devset()),
+                    "test_size": len(task.get_testset()),
+                    "task_name": dataset,
+                    "signature_optimized": False,
+                    "prompt_model_name": self.PROMPT_MODEL_NAME,
+                    "task_model_name": self.TASK_MODEL_NAME,
+                    "breadth": "NA",
+                    "depth": "NA",
+                    "meta_prompt_style": "default",
+                    "fewshot_before": False,
+                    "fewshot_after": False,
+                    "temperature": self.TEMPERATURE,
+                    "fewshot_candidates_num": 0,
+                    "max_bootstrapped_demos": 0,
+                    "bootstrapping": False,
+                    "view_data": False,
+                    "optimizer_log_dir": "NA",
+                    "additional_notes": "",
+                    "misc": "",
+                },
+            )
+
+    def test_optimizer_default(
+        self, optimizer_function, datasets=datasets, test_name="default"
+    ):
 
         for dataset in datasets:
             task = self.load_dataset(dataset)
-            evaluate_train = Evaluate(devset=task.get_trainset(), metric=task.get_metric(), num_threads=self.NUM_THREADS, display_progress=True)
-            evaluate_dev = Evaluate(devset=task.get_devset(), metric=task.get_metric(), num_threads=self.NUM_THREADS, display_progress=True)
-            evaluate_eval = Evaluate(devset=task.get_evalset(), metric=task.get_metric(), num_threads=self.NUM_THREADS, display_progress=True)
+            print(f"Testing  Optimizers on {dataset} ...")
+            dspy.settings.lm.max_tokens = task.get_max_tokens()
+
+            evaluate_train = Evaluate(
+                devset=task.get_trainset(),
+                metric=task.get_metric(),
+                num_threads=self.NUM_THREADS,
+                display_progress=True,
+                max_errors=self.MAX_ERRORS,
+            )
+            evaluate_dev = Evaluate(
+                devset=task.get_devset(),
+                metric=task.get_metric(),
+                num_threads=self.NUM_THREADS,
+                display_progress=True,
+                max_errors=self.MAX_ERRORS,
+            )
+            evaluate_test = Evaluate(
+                devset=task.get_testset(),
+                metric=task.get_metric(),
+                num_threads=self.NUM_THREADS,
+                display_progress=True,
+                max_errors=self.MAX_ERRORS,
+            )
             default_program = task.get_program()
 
             # Set up the optimizer kwargs
             date_timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
-            log_dir = "log_dir/" + dataset + "_" + test_name + "_" + date_timestamp + "/"
+            log_dir = (
+                "log_dir/" + dataset + "_" + test_name + "_" + date_timestamp + "/"
+            )
             os.makedirs(log_dir, exist_ok=True)
-            kwargs = dict(breadth=self.BREADTH, depth=self.DEPTH, temperature=self.TEMPERATURE, prompt_model=self.prompt_model, view_data=False, log_dir=log_dir, metric=task.get_metric(), task_model=self.task_model)
-            
+            kwargs = dict(
+                breadth=self.BREADTH,
+                depth=self.DEPTH,
+                temperature=self.TEMPERATURE,
+                prompt_model=self.prompt_model,
+                view_data=False,
+                log_dir=log_dir,
+                metric=task.get_metric(),
+                task_model=self.task_model,
+            )
+
             # Optimize the default program
             start = timer()
-            optimized_program, output_dict = optimizer_function(default_program, task.get_trainset(), task.get_devset(), test_name, dataset, kwargs)
+            optimized_program, output_dict = optimizer_function(
+                default_program,
+                task.get_trainset(),
+                task.get_devset(),
+                test_name,
+                dataset,
+                kwargs,
+            )
             end = timer()
 
             # Evaluate the optimized program
+            print(f"Optimized train score...")
             optimized_results_train = evaluate_train(optimized_program)
+            print(f"Optimized dev score...")
             optimized_results_dev = evaluate_dev(optimized_program)
-            optimized_results_eval = evaluate_eval(optimized_program)
+            print(f"Optimized test score...")
+            optimized_results_test = evaluate_test(optimized_program)
 
             output = {
-                'test_name': dataset + "_" + test_name,
-                'train_score': optimized_results_train,
-                'dev_score': optimized_results_dev,
-                'eval_score': optimized_results_eval,
-                'run_time (sec)': end - start,
-                'train_size': len(task.get_trainset()),
-                'dev_size': len(task.get_devset()),
-                'eval_size': len(task.get_evalset()),
-                'task_name': dataset,
-                'signature_optimized': True,
-                'prompt_model_name': self.PROMPT_MODEL_NAME,
-                'task_model_name': self.TASK_MODEL_NAME,
-                'breadth': self.BREADTH,
-                'depth': self.DEPTH,
-                'meta_prompt_style': 'default',
-                'fewshot_before': False,
-                'fewshot_after': False,
-                'temperature': self.TEMPERATURE,
-                'fewshot_candidates_num': 0,
-                'max_bootstrapped_demos': 0,
-                'bootstrapping': False,
-                'view_data': False,
-                'optimizer_log_dir': log_dir,
-                'additional_notes': '',
-                'misc': '',
+                "test_name": dataset + "_" + test_name,
+                "train_score": optimized_results_train,
+                "dev_score": optimized_results_dev,
+                "test_score": optimized_results_test,
+                "run_time (sec)": end - start,
+                "train_size": len(task.get_trainset()),
+                "dev_size": len(task.get_devset()),
+                "test_size": len(task.get_testset()),
+                "task_name": dataset,
+                "signature_optimized": True,
+                "prompt_model_name": self.PROMPT_MODEL_NAME,
+                "task_model_name": self.TASK_MODEL_NAME,
+                "breadth": self.BREADTH,
+                "depth": self.DEPTH,
+                "meta_prompt_style": "default",
+                "fewshot_before": False,
+                "fewshot_after": False,
+                "temperature": self.TEMPERATURE,
+                "fewshot_candidates_num": 0,
+                "max_bootstrapped_demos": 0,
+                "bootstrapping": False,
+                "view_data": False,
+                "optimizer_log_dir": log_dir,
+                "additional_notes": "",
+                "misc": "",
             }
 
             output.update(output_dict)
 
             # Write the results to a csv
-            self.write_to_csv('outputs', 'results.csv', output)
\ No newline at end of file
+            self.write_to_csv("outputs", "results.csv", output)
diff --git a/testing/tasks/__init__.py b/testing/tasks/__init__.py
index 0b1ee32ec..835a30e2d 100644
--- a/testing/tasks/__init__.py
+++ b/testing/tasks/__init__.py
@@ -1,7 +1,10 @@
-from .base_task import BaseTask
-from .biodex import BioDexTask
 from .gsm8k import GSM8KTask
 from .hotpotqa import HotPotQATask
 from .scone import ScoNeTask
 from .tweet import TweetTask
 from .tweet_metric import TweetMetricTask
+from .heart_disease import HeartDiseaseTask
+from .hotpotqa_conditional import HotPotQAConditionalTask
+from .hover import HoverRetrieveDiscrete
+from .iris_typo import IrisTypoClassifierTask
+from .iris import IrisClassifierTask
\ No newline at end of file
diff --git a/testing/tasks/base_task.py b/testing/tasks/base_task.py
index 089e6d7be..11e0c136e 100644
--- a/testing/tasks/base_task.py
+++ b/testing/tasks/base_task.py
@@ -18,7 +18,7 @@ def get_trainset(self, TRAIN_NUM=None):
             self.TRAIN_NUM = TRAIN_NUM
             return self.trainset[:TRAIN_NUM]
         else:
-            return self.trainset[:self.TRAIN_NUM]
+            return self.trainset[: self.TRAIN_NUM]
 
     def get_devset(self, TRAIN_NUM=None, DEV_NUM=None):
         if TRAIN_NUM:
@@ -32,18 +32,21 @@ def get_devset(self, TRAIN_NUM=None, DEV_NUM=None):
         else:
             index = max(len(self.trainset) - self.DEV_NUM, self.TRAIN_NUM)
         return self.trainset[index:]
-        
-    def get_evalset(self, EVAL_NUM=None):
-        if EVAL_NUM:
-            self.EVAL_NUM = EVAL_NUM
-            return self.devset[:EVAL_NUM]
+
+    def get_testset(self, TEST_NUM=None):
+        if TEST_NUM:
+            self.TEST_NUM = TEST_NUM
+            return self.testset[:TEST_NUM]
         else:
-            return self.devset[:self.EVAL_NUM]
+            return self.testset[: self.TEST_NUM]
 
-    def set_splits(self, TRAIN_NUM=None, DEV_NUM=None, EVAL_NUM=None):
+    def set_splits(self, TRAIN_NUM=None, DEV_NUM=None, TEST_NUM=None):
         if TRAIN_NUM:
             self.TRAIN_NUM = TRAIN_NUM
         if DEV_NUM:
             self.DEV_NUM = DEV_NUM
-        if EVAL_NUM:
-            self.EVAL_NUM = EVAL_NUM
\ No newline at end of file
+        if TEST_NUM:
+            self.TEST_NUM = TEST_NUM
+
+    def get_max_tokens(self):
+        return 150
diff --git a/testing/tasks/biodex.py b/testing/tasks/biodex.py
deleted file mode 100644
index 6e51745b3..000000000
--- a/testing/tasks/biodex.py
+++ /dev/null
@@ -1,636 +0,0 @@
-from __future__ import annotations
-
-import math
-import os
-import pickle
-import re
-from collections import defaultdict
-from enum import Enum
-from functools import lru_cache
-from typing import DefaultDict, Dict, List, Optional, Union
-
-import datasets
-import tqdm
-from pydantic import BaseModel
-from rapidfuzz import process
-
-import dspy
-from dsp.utils import deduplicate
-from dspy.evaluate import Evaluate
-
-from .base_task import BaseTask
-
-# Must point to a MedDra download with mdhier.asc
-data_dir = '/future/u/okhattab/data/2023/MedDraV2/meddra_23_0_english/MedAscii' # NOTE: EDIT THIS LINE 
-
-global meddra_root
-meddra_root = None
-
-def _remove_non_llt(candidates):
-    return [c for c in candidates if c.level == Level.LLT]
-
-
-def get_path(node):
-    path = f"under {node.parent.term} < {node.parent.parent.term} < {node.parent.parent.parent.term}"
-    return path
-
-
-def flatten(ll):
-    return [x for l in ll for x in l]
-
-
-def _matches_to_terms_and_paths(matches):
-    s = []
-    for n in matches:
-        new_s = f"{n.node.term} ({n.score} score for query '{n.query}'; {n.node.count} counts; {get_path(n.node)})"
-        s.append(new_s)
-    return s
-
-
-class Level(Enum):
-    ROOT = 0
-    SOC = 1
-    HLTG = 2
-    HLT = 3
-    LLT = 4
-
-    def __add__(self, other: int):
-        if self.ROOT.value <= self.value + other <= self.LLT.value:
-            return Level(self.value + other)
-        else:
-            return None
-
-    def __lt__(self, other) -> bool:
-        return self.value < other.value
-
-    def __le__(self, other) -> bool:
-        return self.value <= other.value
-
-    def __gt__(self, other) -> bool:
-        return self.value > other.value
-
-    def __ge__(self, other) -> bool:
-        return self.value >= other.value
-
-    def __sub__(self, other) -> int:
-        return self.value - other.value
-
-
-class Node(BaseModel):
-    id: str
-    term: str
-    level: Level
-    count: int = 0
-
-    parent: Node = None
-    children: Dict[str, Node] = {}
-
-    # maps for efficient nested children lookups
-    # typically only used for the root node
-    id_to_nodes: Optional[DefaultDict[str, List[Node]]] = None
-    term_to_nodes: Optional[DefaultDict[str, List[Node]]] = None
-
-    # text normalizer to build lookup tables
-    # typically only used for the root node
-    normalizer: function = lambda x: x
-
-    def get_parent_at_level(self, target_level: Level) -> Union[Node, None]:
-        # check if current node is deeper than target_level
-        level_diff = target_level - self.level
-        if level_diff > 0:
-            return None
-        else:
-            node = self
-            for _ in range(-level_diff):
-                node = node.parent
-            return node
-
-    def get_children_at_level(self, target_level: Level) -> List[Node]:
-        raise NotImplementedError
-
-    def lookup_id(self, id: str) -> Union[List[Node], None]:
-        # create lookup tables if they do not exists
-        if self.id_to_nodes == None:
-            self.set_lookup_tables()
-        # lookup
-
-        if id in self.id_to_nodes:
-            return self.id_to_nodes[id]
-        else:
-            return None
-
-    def lookup_term(self, term: str) -> Union[List[Node], None]:
-        # create lookup tables if they do not exists
-        if self.term_to_nodes == None:
-            raise RuntimeError("First set the lookup tables using 'set_lookup_tables'.")
-        # lookup
-        term = self.normalizer(term)
-        if term in self.term_to_nodes:
-            return self.term_to_nodes[term]
-        else:
-            return None
-
-    def match(
-        self,
-        query: str,
-        fuzzy_limit: int = 20,
-        fuzzy_score_cutoff: int = 80,
-        return_nodes: bool = False,
-    ):
-        """Return a tuple containing exact match and best fuzzy matches of LLT terms in ontology.
-
-        Args:
-            query (str): String being matched into ontology
-            fuzzy_limit (int, optional): Maximal limit of fuzzy matches to return. Defaults to 20.
-            fuzzy_score_cutoff (int, optional): Minimal fuzzy matching score, between 0 and 100. Defaults to 80.
-            return_nodes (bool, optional): If true, return Node objects instead of paths
-
-        Returns:
-            tuple: Tuple(exact match or None, list of fuzzy matches)
-        """
-        exact_candidate = []
-        fuzzy_candidates = []
-
-        # exact match
-        exact_matches = self.lookup_term(query)
-        if exact_matches is not None:
-            exact_candidate = _remove_non_llt(exact_matches)
-            exact_candidate = [
-                Match(query=query, score=100, node=node) for node in exact_candidate
-            ]
-
-        # fuzzy match
-        fuzzy_terms = process.extract(
-            query,
-            self.term_to_nodes.keys(),
-            limit=fuzzy_limit,
-            score_cutoff=fuzzy_score_cutoff,
-        )
-
-        fuzzy_matches = []
-        for term, sim, _ in fuzzy_terms:
-            applicable_nodes = _remove_non_llt(self.term_to_nodes[term])
-            for n in applicable_nodes:
-                fuzzy_matches.append(Match(node=n, score=sim, query=query))
-
-        if return_nodes:
-            return exact_candidate, fuzzy_matches
-        else:
-            exact_term = (
-                _matches_to_terms_and_paths(exact_candidate) if exact_candidate else []
-            )
-            fuzzy_terms = _matches_to_terms_and_paths(fuzzy_matches)
-            return exact_term, fuzzy_terms
-
-    def set_lookup_tables(self, normalizer: function = lambda x: x) -> None:
-        # set normalizer
-        self.normalizer = normalizer
-
-        # get a flat list of all nodes
-        self.term_to_nodes = defaultdict(list)
-        self.id_to_nodes = defaultdict(list)
-
-        # breadth-first traversal
-        candidates = [self]
-        while len(candidates):
-            candidate = candidates.pop(0)
-            # track nodes
-            self.term_to_nodes[self.normalizer(candidate.term)].append(candidate)
-            self.id_to_nodes[candidate.id].append(candidate)
-            # add candidate's children to candidate list
-            if candidate.children:
-                candidates.extend(candidate.children.values())
-        return
-
-    def is_equivalent_node(self, other: Node) -> bool:
-        # return true if two nodes are the same, siblings or in a parent-child relation
-        return (
-            (self == other)
-            or (self.parent == other)
-            or (self == other.parent)
-            or (self.parent == other.parent)
-        )
-
-    def terms_equivalent(self, term1: str, term2: str) -> Union[bool, None]:
-        # lookup both terms
-        term1_nodes = self.lookup_term(term1)
-        term2_nodes = self.lookup_term(term2)
-        # if at least one term not found
-        if not term1_nodes or not term2_nodes:
-            return None
-        # compare all nodes
-        for node1 in term1_nodes:
-            for node2 in term2_nodes:
-                if node1.is_equivalent_node(node2):
-                    return True
-        return False
-
-    def __key(self):
-        return (self.id, self.term, self.parent, len(self.children))
-
-    def __hash__(self):
-        return hash(self.__key())
-
-    def __eq__(self, other: Node) -> bool:
-        # undeep equals operator, does not check children
-        return self.__key() == other.__key()
-
-    def __str__(self) -> str:
-        return f"Node(id={self.id}, term={self.term}, level={self.level}, parent={self.parent.term if self.parent else None}, #children={len(self.children)}, count={self.count})"
-
-    def __repr__(self) -> str:
-        return self.__str__()
-
-    def get_path(self):
-        path = f"under {self.parent.term} < {self.parent.parent.term} < {self.parent.parent.parent.term}"
-        return path
-
-    @property
-    def all_nodes(self):
-        return flatten(list(self.id_to_nodes.values()))
-
-    @property
-    def all_llt_nodes(self):
-        return _remove_non_llt(self.all_nodes)
-
-
-class Match(BaseModel):
-    node: Node
-    score: int
-    query: str
-
-def _create_child_if_not_exists(
-    parent: Node, child_id: str, child_term: str, term_to_count: dict,
-):
-    parent_level = parent.level
-    # get child level if still appropriate
-    child_level = parent_level + 1
-    if child_level is None:
-        return None
-    # get child if exists
-    if child_id in parent.children:
-        child = parent.children[child_id]
-    # create child if not exists
-    else:
-        child_term_dict = child_term.lower().replace("'", "^")
-        if child_term_dict in term_to_count:
-            count = term_to_count[child_term_dict]
-        else:
-            count = 0
-        child = Node(
-            id=child_id, term=child_term, level=child_level, parent=parent, count=count,
-        )
-        parent.children.update({child_id: child})
-    return child
-
-def parse_mdhier(data_dir):
-    file = os.path.join(data_dir, "mdhier.asc")
-    if not os.path.isfile(file):
-        raise FileNotFoundError(f"File {file} not found.")
-    count_file = os.path.join(data_dir, "priors.pickle")
-    if not os.path.isfile(count_file):
-        raise FileNotFoundError(f"File {count_file} not found.")
-
-    # get regex
-    id_regex = r"\d{8}"
-    term_regex = r"[^$]*"
-    regex = rf"({id_regex})\$({id_regex})\$({id_regex})\$({id_regex})\$({term_regex})\$({term_regex})\$({term_regex})\$({term_regex})\$({term_regex})\$\$({id_regex})\$[YN]\$"
-    regex = re.compile(regex)
-
-    # get data
-    lines = []
-    with open(file) as fp:
-        lines = fp.readlines()
-    text = "\n".join(lines)
-
-    matches = regex.findall(text)
-
-    print(f"Read {len(lines)} lines.")
-    if len(lines) != len(matches):
-        raise RuntimeError(
-            "Error parsing mdhier.asc. Expected number of matches to match number of lines.",
-        )
-
-    # load counts
-    with open(count_file, "rb") as fp:
-        counts = pickle.load(fp)
-
-    # create ontology tree
-    root = Node(id="root", term="root", level=Level.ROOT)
-    for match in matches:
-        (
-            llt_id,
-            hlt_id,
-            hltg_id,
-            soc_id,
-            llt_term,
-            hlt_term,
-            hltg_term,
-            soc_term_1,
-            soc_term_2,
-            _,
-        ) = match
-        # recursivelty add the nodes specified by this line
-        soc_node = _create_child_if_not_exists(root, soc_id, soc_term_1, counts)
-        hltg_node = _create_child_if_not_exists(soc_node, hltg_id, hltg_term, counts)
-        hlt_node = _create_child_if_not_exists(hltg_node, hlt_id, hlt_term, counts)
-        llt_node = _create_child_if_not_exists(hlt_node, llt_id, llt_term, counts)
-
-    # print(llt_node.parent == hlt_node)
-    # print(hlt_node.parent == hltg_node)
-    # print(hltg_node.parent == soc_node)
-    # print(soc_node.parent == root)
-
-    # print(Level.SOC < Level.LLT)
-    # print(Level.SOC > Level.LLT)
-    # print(Level.SOC - Level.LLT)
-
-    # print(soc_node.get_parent_at_level(Level.ROOT))
-    # print(soc_node.get_parent_at_level(Level.SOC))
-    # print(soc_node.get_parent_at_level(Level.LLT))
-    # print(llt_node.get_parent_at_level(Level.SOC))
-
-    # print(root.lookup_term("COVID-19 pneumonia"))
-    # print(root.lookup_term("COVID-19 pneumonia")[0].parent)
-    return root
-
-@lru_cache(maxsize=100000)
-def base_ground(reaction, K):
-    global meddra_root
-    kwargs = dict(fuzzy_limit=K, fuzzy_score_cutoff=75, return_nodes=True)
-    exact_matches, fuzzy_matches = meddra_root.match(reaction, **kwargs)
-
-    return exact_matches, fuzzy_matches
-
-
-@lru_cache(maxsize=100000)
-def ground_v1(reaction, K=3):
-    """
-        Prefers exact matches over fuzzy matches, when available.
-    """
-
-    exact_matches, fuzzy_matches = base_ground(reaction, K)
-
-    matches = exact_matches[:1] or fuzzy_matches
-    matches = [(match.score / 100.0, match.node.term) for match in matches]
-    matches = sorted(list(set(matches)), reverse=True)
-
-    return tuple(matches)
-
-@lru_cache(maxsize=100000)
-def ground_v2(reaction, K=1):
-    """
-        When K=1 (default), returns exact matches (if available) or the best fuzzy match.
-    """
-
-    exact_matches, fuzzy_matches = base_ground(reaction, K)
-
-    matches = [(match.score / 100.0, match.node.term) for match in (exact_matches[:1] + fuzzy_matches)]
-    matches = sorted(list(set(matches)), reverse=True)[:K]
-
-    return tuple(matches)
-
-# Returns the best three matches (including one exact match, if available).
-ground_v3 = lambda reaction: ground_v2(reaction, K=3)
-
-@lru_cache(maxsize=100000)
-def ground_v4(reaction, K=3):
-    """
-        Returns the best three matches (including one exact match, if available) and applies a prior.
-    """
-    exact_matches, fuzzy_matches = base_ground(reaction, K)
-    
-    matches = [((match.score / 100.0) * math.sqrt(match.node.count + 0.1), match.node.term)
-               for match in (exact_matches[:1] + fuzzy_matches)]
-    matches = sorted(list(set(matches)), reverse=True)[:K]
-
-    return tuple(matches)
-
-@lru_cache(maxsize=100000)
-def ground_v4b(reaction, K=3):
-    """
-        Returns the best three matches (including one exact match, if available) and applies a prior.
-    """
-
-    exact_matches, fuzzy_matches = base_ground(reaction, K)
-    
-    matches = [((match.score / 100.0) * math.log(match.node.count + 0.1), match.node.term)
-               for match in (exact_matches[:1] + fuzzy_matches)]
-    matches = sorted(list(set(matches)), reverse=True)[:K]
-
-    return tuple(matches)
-
-@lru_cache(maxsize=100000)
-def ground_v4c(reaction, K=3):
-    """
-        Returns the best three matches (including one exact match, if available) and applies a prior.
-    """
-
-    exact_matches, fuzzy_matches = base_ground(reaction, K)
-    
-    matches = [((match.score / 100.0) * (match.node.count + 0.1), match.node.term)
-               for match in (exact_matches[:1] + fuzzy_matches)]
-    matches = sorted(list(set(matches)), reverse=True)[:K]
-
-    return tuple(matches)
-
-
-@lru_cache(maxsize=100000)
-def ground_v5(reaction, K=3):
-    exact_matches, fuzzy_matches = base_ground(reaction, K)
-    
-    matches = [((match.score / 100.0)**2 * math.log(match.node.count + 0.1), match.node.term)
-               for match in (exact_matches[:1] + fuzzy_matches)]
-    matches = sorted(list(set(matches)), reverse=True)[:K]
-
-    return tuple(matches)
-
-def preprocess_example(example):
-    title = example['title']
-    abstract = example['abstract']
-    context = example['fulltext_processed'].split('\n\nTEXT:\n', 1)[-1]
-    reactions = [r.strip().lower() for r in example['reactions'].split(',')]
-
-    example = dict(title=title, abstract=abstract, context=context, reactions=reactions)
-    example['labels'] = dspy.Example(reactions=reactions)
-
-    return example
-
-class Chunker:
-    def __init__(self, context_window=3000, max_windows=5):
-        self.context_window = context_window
-        self.max_windows = max_windows
-        self.window_overlap = 0.02
-    
-    def __call__(self, paper):
-        snippet_idx = 0
-
-        while snippet_idx < self.max_windows and paper:
-            endpos = int(self.context_window * (1.0 + self.window_overlap))
-            snippet, paper = paper[:endpos], paper[endpos:]
-
-            next_newline_pos = snippet.rfind('\n')
-            if paper and next_newline_pos != -1 and next_newline_pos >= self.context_window // 2:
-                paper = snippet[next_newline_pos+1:] + paper
-                snippet = snippet[:next_newline_pos]
-
-            yield snippet_idx, snippet.strip()
-            snippet_idx += 1
-
-def get_set_precision_and_recalls(l1, l2):
-    # lower case and strip
-    l1 = [x.strip().lower() for x in l1]
-    l2 = [x.strip().lower() for x in l2]
-
-    s1, s2 = set(l1), set(l2)
-    intersect = s1.intersection(s2)
-
-    p, r = len(intersect) / max(1, len(s1)), len(intersect) / max(1, len(s2))
-    f1 = 2 * (p * r) / max((p + r), 1)
-
-    return dspy.Prediction(precision=p, recall=r, f1=f1)
-
-def finegrained_metrics(gold, pred):
-    reaction_metrics = get_set_precision_and_recalls(pred.reactions, gold.reactions)
-    return reaction_metrics
-
-
-class CompilationMetric:
-    __name__ = 'compilation_metric'
-
-    def __init__(self, field='reactions', metric='recall', perfect_precision=False):
-        self.field = field
-        self.metric = metric
-        self.perfect_precision = perfect_precision
-
-    def __call__(self, gold, pred, trace=None):
-        field = self.field
-        metric = finegrained_metrics(gold, pred) #[field]
-
-        if trace is None: # eval
-            return metric[self.metric]
-
-        if metric.precision >= 0.5 and metric.recall >= 0.5:
-            for trace_idx in range(len(trace)):
-                *_, outputs = trace[trace_idx]
-
-                values = outputs[field].split(', ')
-                values = [r.lower().strip('.').strip() for r in values]
-                values = deduplicate([r for r in values if r and r != 'n/a'])
-
-                if self.perfect_precision:
-                    values = [r for r in values if r in gold[self.field]]
-                
-                outputs[field] = ', '.join(values) or 'N/A'
-
-                # print(gold[field], '\t\t\t', values)
-
-            return True
-
-        return False
-
-
-metricR = CompilationMetric(field='reactions', metric='recall')
-from collections import Counter
-
-
-def reduce_grounded_reactions(grounded_reactions):
-    scores = {}
-    for score, reaction in grounded_reactions:
-        scores[reaction] = scores.get(reaction, 0) + score
-    reactions = sorted(scores.items(), key=lambda x: x[1], reverse=True)
-    
-    reactions = [r for r, _ in reactions]
-    return reactions
-
-
-def metricR10(gold, pred, trace=None):
-    reactions = pred.reactions
-
-    try:
-        reactions = reduce_grounded_reactions(reactions)
-    except Exception:
-        reactions = Counter(reactions).most_common()
-        reactions = [r for r, _ in reactions]
-
-    return metricR(gold, pred.copy(reactions=reactions[:10]), trace)
-
-
-def extract_reactions_from_strings(reactions):
-    reactions = [r.strip().split('\n')[0].lower().strip('.').strip() for r in reactions]
-    reactions = ', '.join(reactions)
-    reactions = [r.lower().strip('.').strip() for r in reactions.split(',')]
-    reactions = [r for r in reactions if r and r != 'n/a']
-
-    return reactions
-
-class PredictReactions(dspy.Signature):
-    __doc__ = """Given a snippet from a medical article, identify the adverse drug reactions affecting the patient. If none are mentioned in the snippet, say N/A."""
-
-    title = dspy.InputField()
-    context = dspy.InputField()
-    reactions = dspy.OutputField(desc="list of comma-separated adverse drug reactions", format=lambda x: ', '.join(x) if isinstance(x, list) else x)
-
-HINT = "Is any of the following reactions discussed in the article snippet? The valid candidates are:"
-
-class GroundedReactionExtractor(dspy.Module):
-    def __init__(self, context_window=3000, max_windows=5, num_preds=1):
-        super().__init__()
-
-        self.chunk = Chunker(context_window=context_window, max_windows=max_windows)        
-        self.predict = dspy.ChainOfThoughtWithHint(PredictReactions, n=num_preds)
-    
-    def forward(self, title, abstract, context, labels=None):
-        hint = f"{HINT} {', '.join(labels.reactions)}." if labels else None
-        reactions = []
-
-        for _, snippet in self.chunk(abstract + '\n\n' + context):
-            chunk_reactions = self.predict(title=title, context=[snippet], hint=hint)
-            reactions.extend(extract_reactions_from_strings(chunk_reactions.completions.reactions))
-
-        reactions = [r for sublist in [ground_v4b(r) for r in reactions] for r in sublist]
-        return dspy.Prediction(reactions=reactions)
-
-class BioDexTask(BaseTask):
-    def __init__(self):
-        global meddra_root
-
-        meddra_root = parse_mdhier(data_dir)
-        meddra_root.set_lookup_tables(normalizer=str.lower)
-
-        # Load and configure the datasets.
-        dataset = datasets.load_dataset("BioDEX/BioDEX-Reactions")
-        official_trainset, official_devset = dataset['train'], dataset['validation']
-        trainset, devset = [], []
-
-        for example in tqdm.tqdm(official_trainset):
-            if len(trainset) >= 1000: break
-            trainset.append(preprocess_example(example))
-
-        for example in tqdm.tqdm(official_devset):
-            if len(devset) >= 500: break
-            devset.append(preprocess_example(example))
-
-        trainsetX = [dspy.Example(**x).with_inputs('title', 'abstract', 'context', 'labels') for x in trainset]
-        trainset = [dspy.Example(**x).with_inputs('title', 'abstract', 'context') for x in trainset]
-        devsetX = [dspy.Example(**x).with_inputs('title', 'abstract', 'context', 'labels') for x in devset]
-        devset = [dspy.Example(**x).with_inputs('title', 'abstract', 'context') for x in devset]
-
-        self.trainset = trainset
-        self.devset = devset
-
-        # Set up metrics
-        NUM_THREADS = 16
-
-        self.metric = metricR10
-
-        kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=15)
-        self.evaluarteR10 = Evaluate(devset=self.devset, metric=metricR10, **kwargs)
-
-        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, EVAL_NUM=100)
-
-    def get_program(self):
-        return GroundedReactionExtractor(context_window=3000, max_windows=5, num_preds=1) 
-    
-    def get_metric(self):
-        return self.metric
\ No newline at end of file
diff --git a/testing/tasks/gsm8k.py b/testing/tasks/gsm8k.py
index 04f20d177..f57760e9d 100644
--- a/testing/tasks/gsm8k.py
+++ b/testing/tasks/gsm8k.py
@@ -13,40 +13,45 @@ class CoT(dspy.Module):
     def __init__(self):
         super().__init__()
         self.prog = dspy.ChainOfThought("question -> answer")
-    
+
     def forward(self, question):
         return self.prog(question=question)
-    
+
+
 class GSM8KTask(BaseTask):
     def __init__(self):
-        dataset = load_dataset("gsm8k", 'main')
+        dataset = load_dataset("gsm8k", "main")
 
-        hf_official_train = dataset['train']
-        hf_official_test = dataset['test']
+        hf_official_train = dataset["train"]
+        hf_official_test = dataset["test"]
         official_train = []
         official_test = []
 
         for example in tqdm.tqdm(hf_official_train):
-            question = example['question']
+            question = example["question"]
+
+            answer = example["answer"].strip().split()
+            assert answer[-2] == "####"
 
-            answer = example['answer'].strip().split()
-            assert answer[-2] == '####'
-            
-            gold_reasoning = ' '.join(answer[:-2])
-            answer = str(int(answer[-1].replace(',', '')))
+            gold_reasoning = " ".join(answer[:-2])
+            answer = str(int(answer[-1].replace(",", "")))
 
-            official_train.append(dict(question=question, gold_reasoning=gold_reasoning, answer=answer))
+            official_train.append(
+                dict(question=question, gold_reasoning=gold_reasoning, answer=answer)
+            )
 
         for example in tqdm.tqdm(hf_official_test):
-            question = example['question']
+            question = example["question"]
 
-            answer = example['answer'].strip().split()
-            assert answer[-2] == '####'
-            
-            gold_reasoning = ' '.join(answer[:-2])
-            answer = str(int(answer[-1].replace(',', '')))
+            answer = example["answer"].strip().split()
+            assert answer[-2] == "####"
 
-            official_test.append(dict(question=question, gold_reasoning=gold_reasoning, answer=answer))
+            gold_reasoning = " ".join(answer[:-2])
+            answer = str(int(answer[-1].replace(",", "")))
+
+            official_test.append(
+                dict(question=question, gold_reasoning=gold_reasoning, answer=answer)
+            )
 
         rng = random.Random(0)
         rng.shuffle(official_train)
@@ -58,23 +63,22 @@ def __init__(self):
         devset = official_train[2000:2500]
         testset = official_test[:]
 
-        trainset = [dspy.Example(**x).with_inputs('question') for x in trainset]
-        devset = [dspy.Example(**x).with_inputs('question') for x in devset]
-        testset = [dspy.Example(**x).with_inputs('question') for x in testset]
-
-        # print(f"Trainset size: {len(trainset)}")
-        # print(f"Devset size: {len(devset)}")
-        # print(f"Testset size: {len(testset)}")
+        trainset = [dspy.Example(**x).with_inputs("question") for x in trainset]
+        devset = [dspy.Example(**x).with_inputs("question") for x in devset]
+        testset = [dspy.Example(**x).with_inputs("question") for x in testset]
 
-        self.trainset= trainset
+        self.trainset = trainset
         self.devset = devset
         self.testset = testset
         self.metric = gsm8k_metric
 
-        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, EVAL_NUM=100)
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
 
     def get_program(self):
         return CoT()
-    
+
     def get_metric(self):
-        return self.metric
\ No newline at end of file
+        return self.metric
+
+    def get_max_tokens(self):
+        return 600
diff --git a/testing/tasks/heart_disease.py b/testing/tasks/heart_disease.py
new file mode 100644
index 000000000..f64026172
--- /dev/null
+++ b/testing/tasks/heart_disease.py
@@ -0,0 +1,173 @@
+import dspy
+import random
+from datasets import load_dataset
+from .base_task import BaseTask
+
+mappings = {
+    "sex": {0: "female", 1: "male"},
+    "cp": {
+        1: "typical angina",
+        2: "atypical angina",
+        3: "non-anginal pain",
+        4: "asymptomatic",
+    },
+    "restecg": {
+        0: "normal",
+        1: "ST-T wave abnormality",
+        2: "left ventricular hypertrophy",
+    },
+    "exang": {0: "no", 1: "yes"},
+    "slope": {1: "upsloping", 2: "flat", 3: "downsloping"},
+    "thal": {"3": "normal", "6": "fixed defect", "7": "reversible defect"},
+    "target": {0: "no", 1: "yes"},  # presence of heart disease
+}
+
+dataset = load_dataset("buio/heart-disease")
+fullset = []
+
+for x in dataset["train"]:
+    for key, value in x.items():
+        if key in mappings:
+            x[key] = mappings[key].get(value, value)
+        x[key] = str(x[key])
+
+    inputs = [k for k in x.keys() if k != "target"]
+    x["answer"] = x["target"]
+    del x["target"]
+
+    fullset.append(dspy.Example(**x).with_inputs(*inputs))
+
+random.Random(0).shuffle(fullset)
+
+trainset = fullset[:120]
+devset = trainset
+testset = fullset[120:]
+
+
+class HeartDiseaseInput(dspy.Signature):
+    age = dspy.InputField(desc="Age in years")
+    sex = dspy.InputField(desc="Sex (male or female)")
+    cp = dspy.InputField(
+        desc="Chest pain type (typical angina, atypical angina, non-anginal pain, asymptomatic)"
+    )
+    trestbps = dspy.InputField(
+        desc="Resting blood pressure (in mm Hg on admission to the hospital)"
+    )
+    chol = dspy.InputField(desc="Serum cholestoral in mg/dl")
+    fbs = dspy.InputField(desc="Fasting blood sugar > 120 mg/dl (true or false)")
+    restecg = dspy.InputField(
+        desc="Resting electrocardiographic results (normal, ST-T wave abnormality, left ventricular hypertrophy)"
+    )
+    thalach = dspy.InputField(desc="Maximum heart rate achieved")
+    exang = dspy.InputField(desc="Exercise induced angina (yes or no)")
+    oldpeak = dspy.InputField(desc="ST depression induced by exercise relative to rest")
+    slope = dspy.InputField(
+        desc="The slope of the peak exercise ST segment (upsloping, flat, downsloping)"
+    )
+    ca = dspy.InputField(desc="Number of major vessels (0-3) colored by flourosopy")
+    thal = dspy.InputField(desc="Thalassemia (normal, fixed defect, reversible defect)")
+
+
+class HeartDiseaseSignature(HeartDiseaseInput):
+    """Given patient information, predict the presence of heart disease."""
+
+    answer = dspy.OutputField(
+        desc="Does this patient have heart disease? Just yes or no."
+    )
+
+
+class HeartDiseaseVote(HeartDiseaseInput):
+    """Given patient information, predict the presence of heart disease. I can critically assess the provided trainee opinions."""
+
+    context = dspy.InputField(desc="A list of opinions from trainee doctors.")
+    answer = dspy.OutputField(
+        desc="Does this patient have heart disease? Just yes or no."
+    )
+
+
+class Classify(dspy.Module):
+    def __init__(self):
+        self.classify = [
+            dspy.ChainOfThought(HeartDiseaseSignature, temperature=0.7 + i * 0.01)
+            for i in range(3)
+        ]
+        self.vote = dspy.ChainOfThought(HeartDiseaseVote)
+
+    def forward(
+        self,
+        age,
+        sex,
+        cp,
+        trestbps,
+        chol,
+        fbs,
+        restecg,
+        thalach,
+        exang,
+        oldpeak,
+        slope,
+        ca,
+        thal,
+    ):
+        kwargs = dict(
+            age=age,
+            sex=sex,
+            cp=cp,
+            trestbps=trestbps,
+            chol=chol,
+            fbs=fbs,
+            restecg=restecg,
+            thalach=thalach,
+            exang=exang,
+            oldpeak=oldpeak,
+            slope=slope,
+            ca=ca,
+            thal=thal,
+        )
+
+        opinions = [c(**kwargs) for c in self.classify]
+        opinions = [
+            (opinion.reasoning.replace("\n", " ").strip("."), opinion.answer.strip("."))
+            for opinion in opinions
+        ]
+
+        opinions = [
+            f"I'm a trainee doctor, trying to {reason}. Hence, my answer is {answer}."
+            for reason, answer in opinions
+        ]
+        return self.vote(context=opinions, **kwargs)
+
+
+class HeartDiseaseTask(BaseTask):
+    def __init__(self):
+        self.trainset = trainset
+        self.devset = devset
+        self.testset = testset
+        self.metric = dspy.evaluate.answer_exact_match
+
+    def get_program(self):
+        return Classify()
+
+    def get_metric(self):
+        return self.metric
+
+    def get_default_max_bootstrapped_demos(self):
+        return 4
+
+    def get_default_max_labeled_demos(self):
+        return 0
+
+    def get_trainset(self, TRAIN_NUM=None):
+        return self.trainset[:TRAIN_NUM]
+
+    def get_testset(self, TEST_NUM=None):
+        return self.testset[:TEST_NUM]
+
+    def get_devset(self, TRAIN_NUM=None, DEV_NUM=None):
+        return self.devset[:TRAIN_NUM]
+
+    def get_handwritten_prompts(self):
+        return []
+
+    def get_max_tokens(self):
+        return 350
diff --git a/testing/tasks/hotpotqa.py b/testing/tasks/hotpotqa.py
index f0bf17078..7a8966407 100644
--- a/testing/tasks/hotpotqa.py
+++ b/testing/tasks/hotpotqa.py
@@ -6,25 +6,30 @@
 
 
 class MultiHop(dspy.Module):
-    def __init__(self,passages_per_hop):
+    def __init__(self, passages_per_hop):
         super().__init__()
-        self.retrieve = dspy.Retrieve(k = passages_per_hop)
+        self.retrieve = dspy.Retrieve(k=passages_per_hop)
         self.generate_query = dspy.ChainOfThought("context ,question->search_query")
         self.generate_answer = dspy.ChainOfThought("context ,question->answer")
-    
-    def forward (self,question) :
+
+    def forward(self, question):
         context = []
         for hop in range(2):
-            query = self.generate_query(context = context, question = question).search_query
+            query = self.generate_query(context=context, question=question).search_query
             context += self.retrieve(query).passages
-        return dspy.Prediction(context=context, answer=self.generate_answer(context = context , question = question).answer)
+        return dspy.Prediction(
+            context=context,
+            answer=self.generate_answer(context=context, question=question).answer,
+        )
+
 
 class HotPotQATask(BaseTask):
     def __init__(self):
         # Load and configure the datasets.
-        hotpot_dataset = HotPotQA(train_seed=1, eval_seed=2023, test_size=0)
-        self.trainset = [x.with_inputs('question') for x in hotpot_dataset.train]
-        self.devset = [x.with_inputs('question') for x in hotpot_dataset.dev]
+        hotpot_dataset = HotPotQA(train_seed=1, eval_seed=2023)
+
+        self.trainset = [x.with_inputs("question") for x in hotpot_dataset.train]
+        self.testset = [x.with_inputs("question") for x in hotpot_dataset.dev]
 
         # Set up metrics
         NUM_THREADS = 16
@@ -33,19 +38,34 @@ def __init__(self):
         self.metric = metric_EM
 
         def gold_passages_retrieved(example, pred, trace=None):
-            gold_titles = set(map(dspy.evaluate.normalize_text, example['gold_titles']))
-            found_titles = set(map(dspy.evaluate.normalize_text, [c.split(' | ')[0] for c in pred.context]))
+            gold_titles = set(map(dspy.evaluate.normalize_text, example["gold_titles"]))
+            found_titles = set(
+                map(
+                    dspy.evaluate.normalize_text,
+                    [c.split(" | ")[0] for c in pred.context],
+                )
+            )
             return gold_titles.issubset(found_titles)
 
         kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=15)
-        self.evaluate_EM = Evaluate(devset=self.devset, metric=metric_EM, **kwargs)
-        self.evaluate_retrieval = Evaluate(devset=self.devset, metric=gold_passages_retrieved, **kwargs)
+        self.evaluate_EM = Evaluate(devset=self.trainset, metric=metric_EM, **kwargs)
+        self.evaluate_retrieval = Evaluate(
+            devset=self.trainset, metric=gold_passages_retrieved, **kwargs
+        )
 
-        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, EVAL_NUM=100)
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
 
     def get_program(self):
-        return MultiHop(passages_per_hop=3) #TODO: make it so we can specify # of passages 
-    
+        return MultiHop(passages_per_hop=3)
+
     def get_metric(self):
         return self.metric
-    
+
+    def get_retrieval_metric(self):
+        return self.evaluate_retrieval
+
+    def get_default_max_bootstrapped_demos(self):
+        return 1
+
+    def get_default_max_labeled_demos(self):
+        return 2
diff --git a/testing/tasks/hotpotqa_conditional.py b/testing/tasks/hotpotqa_conditional.py
new file mode 100644
index 000000000..c0b0d88fa
--- /dev/null
+++ b/testing/tasks/hotpotqa_conditional.py
@@ -0,0 +1,204 @@
+import dspy
+from dspy.datasets import HotPotQA
+from dspy.evaluate import Evaluate
+import pandas as pd
+import os
+from .base_task import BaseTask
+import string
+
+
+def preprocess_text(text):
+    # Remove all punctuation
+    text = "".join(char for char in text if char not in string.punctuation)
+    # Convert to lower case
+    text = text.lower()
+    # Remove 'peace!' from the end if it exists
+    if text.endswith("peace!"):
+        text = text[
+            :-6
+        ].strip()  # Remove the last 'peace!' and strip any trailing spaces
+    return text
+
+
+def calculate_recall(correct_words, pred_words):
+    correct_set = set(correct_words)
+    pred_set = set(pred_words)
+    # Calculate the number of correct words recalled in the prediction
+    common_words = correct_set.intersection(pred_set)
+    if len(correct_set) == 0:
+        return True  # Avoid division by zero if correct answer is somehow empty
+    recall = len(common_words) / len(correct_set)
+    return recall >= 2 / 3
+
+
+def check_conditions(example, pred, trace=None, debug=False):
+    category = example.category
+    answer = pred.answer
+
+    preprocessed_pred_answer = preprocess_text(pred.answer)
+    preprocessed_correct_answer = preprocess_text(example.answer)
+
+    # Check for exact match after preprocessing
+    if preprocessed_pred_answer != preprocessed_correct_answer:
+        if debug:
+            print(
+                f"Exact match failed. Expected: '{preprocessed_correct_answer}', Got: '{preprocessed_pred_answer}'"
+            )
+        return False
+
+    # When the answer is a place, the response should contain no punctuation
+    if category == "place":
+        if any(char in answer for char in ",.?!;:"):
+            return False
+        else:
+            if debug:
+                print(
+                    f"Place. When the answer is a place, the response should contain no punctuation {answer}"
+                )
+            return True
+
+    # When the answer is a date, the response should end with "Peace!"
+    elif category == "date":
+        if not answer.endswith("Peace!"):
+            return False
+        else:
+            if debug:
+                print(
+                    f"Date. When the answer is a date, the response should end with Peace! {answer}"
+                )
+            return True
+
+    # When the answer is a person, the response should be entirely in lowercase
+    elif category == "person":
+        if answer != answer.lower():
+            return False
+        else:
+            if debug:
+                print(
+                    f"Answer. When the answer is a person, the response should be entirely in lowercase {answer}"
+                )
+            return True
+
+    # When the answer is none of the above categories, the response should be in all caps and not end with "Peace!"
+    else:
+        if answer != answer.upper() or answer.endswith("Peace!"):
+            return False
+        else:
+            if debug:
+                print(
+                    f"Other category. the response should be in all caps and not end with Peace! {answer}"
+                )
+            return True
+
+
+class MultiHop(dspy.Module):
+    def __init__(self, passages_per_hop):
+        super().__init__()
+        self.retrieve = dspy.Retrieve(k=passages_per_hop)
+        self.generate_query = dspy.ChainOfThought("context ,question->search_query")
+        self.generate_answer = dspy.ChainOfThought("context ,question->answer")
+
+    def forward(self, question):
+        context = []
+        for hop in range(2):
+            query = self.generate_query(context=context, question=question).search_query
+            context += self.retrieve(query).passages
+        return dspy.Prediction(
+            context=context,
+            answer=self.generate_answer(context=context, question=question).answer,
+        )
+
+
+class GenerateAnswerInstruction(dspy.Signature):
+    """When the answer is a person, respond entirely in lowercase.  When the answer is a place, ensure your response contains no punctuation.  When the answer is a date, end your response with “Peace!”.  Never end your response with "Peace!" under other circumstances.  When the answer is none of the above categories respond in all caps."""
+
+    context = dspy.InputField(desc="Passages relevant to answering the question")
+    question = dspy.InputField(desc="Question we want an answer to")
+    answer = dspy.OutputField(desc="Answer to the question")
+
+
+class MultiHopHandwritten(dspy.Module):
+    def __init__(self, passages_per_hop):
+        super().__init__()
+        self.retrieve = dspy.Retrieve(k=passages_per_hop)
+        self.generate_query = dspy.ChainOfThought("context ,question->search_query")
+        self.generate_answer = dspy.ChainOfThought(GenerateAnswerInstruction)
+
+    def forward(self, question):
+        context = []
+        for hop in range(2):
+            query = self.generate_query(context=context, question=question).search_query
+            context += self.retrieve(query).passages
+        return dspy.Prediction(
+            context=context,
+            answer=self.generate_answer(context=context, question=question).answer,
+        )
+
+
+class HotPotQAConditionalTask(BaseTask):
+    def __init__(self):
+        # Read in the conditional HotpotQA dataset from nfl_datasets as a csv from nfl_datasets/conditional_hotpotqa
+
+        # Get the directory where this script is located
+        script_dir = os.path.dirname(os.path.abspath(__file__))
+
+        # Construct the absolute paths to the datasets
+        hotpotqa_train_path = os.path.join(
+            script_dir, "../datasets/hotpotqa_conditional/hotpot_train.csv"
+        )
+        hotpotqa_dev_path = os.path.join(
+            script_dir, "../datasets/hotpotqa_conditional/hotpot_dev.csv"
+        )
+        hotpotqa_test_path = os.path.join(
+            script_dir, "../datasets/hotpotqa_conditional/hotpot_test.csv"
+        )
+
+        # Read the datasets
+        hotpotqa_train = pd.read_csv(hotpotqa_train_path)
+        hotpotqa_dev = pd.read_csv(hotpotqa_dev_path)
+        hotpotqa_test = pd.read_csv(hotpotqa_test_path)
+
+        combined_train = pd.concat([hotpotqa_train, hotpotqa_dev], ignore_index=True)
+
+        # Load and configure the datasets.
+        self.trainset = [
+            dspy.Example(
+                question=row["question"],
+                answer=row["answer"],
+                category=row["answer category"],
+            ).with_inputs("question")
+            for index, row in combined_train.iterrows()
+        ]
+        self.testset = [
+            dspy.Example(
+                question=row["question"],
+                answer=row["answer"],
+                category=row["answer category"],
+            ).with_inputs("question")
+            for index, row in hotpotqa_test.iterrows()
+        ]
+
+        # Set up metrics
+        NUM_THREADS = 16
+
+        # TODO: set up metrics
+        self.metric = check_conditions
+
+        kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=15)
+
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
+
+    def get_program(self):
+        return MultiHop(passages_per_hop=3)
+
+    def get_metric(self):
+        return self.metric
+
+    def get_default_max_bootstrapped_demos(self):
+        return 4
+
+    def get_default_max_labeled_demos(self):
+        return 0
+
+    def get_max_tokens(self):
+        return 600
diff --git a/testing/tasks/hover.py b/testing/tasks/hover.py
new file mode 100644
index 000000000..1a7e4378c
--- /dev/null
+++ b/testing/tasks/hover.py
@@ -0,0 +1,130 @@
+import random
+
+import tqdm
+from datasets import load_dataset
+import pandas as pd
+import dspy
+from dsp.utils.utils import deduplicate
+from dspy.evaluate import Evaluate
+
+from .base_task import BaseTask
+
+
+def count_unique_docs(example):
+    return len(set([fact["key"] for fact in example["supporting_facts"]]))
+
+
+def discrete_retrieval_eval(example, pred, trace=None):
+    gold_titles = set(
+        map(
+            dspy.evaluate.normalize_text,
+            [doc["key"] for doc in example["supporting_facts"]],
+        )
+    )
+    found_titles = set(
+        map(
+            dspy.evaluate.normalize_text,
+            [c.split(" | ")[0] for c in pred.retrieved_docs],
+        )
+    )
+    return gold_titles.issubset(found_titles)
+
+
+class RetrieveMultiHop(dspy.Module):
+    def __init__(self):
+        super().__init__()
+        self.k = 7
+        self.create_query_hop2 = dspy.ChainOfThought("claim,summary_1->query")
+        self.create_query_hop3 = dspy.ChainOfThought("claim,summary_1,summary_2->query")
+        self.retrieve_k = dspy.Retrieve(k=self.k)
+        self.summarize1 = dspy.ChainOfThought("claim,passages->summary")
+        self.summarize2 = dspy.ChainOfThought("claim,context,passages->summary")
+
+    def forward(self, claim):
+        # HOP 1
+        hop1_docs = self.retrieve_k(claim).passages
+        summary_1 = self.summarize1(
+            claim=claim, passages=hop1_docs
+        ).summary  # Summarize top k docs
+
+        # HOP 2
+        hop2_query = self.create_query_hop2(claim=claim, summary_1=summary_1).query
+        hop2_docs = self.retrieve_k(hop2_query).passages
+        summary_2 = self.summarize2(
+            claim=claim, context=summary_1, passages=hop2_docs
+        ).summary
+
+        # HOP 3
+        hop3_query = self.create_query_hop3(
+            claim=claim, summary_1=summary_1, summary_2=summary_2
+        ).query
+        hop3_docs = self.retrieve_k(hop3_query).passages
+
+        return dspy.Prediction(retrieved_docs=hop1_docs + hop2_docs + hop3_docs)
+
+
+class HoverRetrieveDiscrete(BaseTask):
+    def __init__(self):
+        dataset = load_dataset("hover")
+
+        hf_trainset = dataset["train"]
+        hf_testset = dataset["validation"]
+
+        reformatted_hf_trainset = []
+        reformatted_hf_testset = []
+
+        for example in tqdm.tqdm(hf_trainset):
+            claim = example["claim"]
+            supporting_facts = example["supporting_facts"]
+            label = example["label"]
+
+            if count_unique_docs(example) == 3:  # Limit to 3 hop examples
+                reformatted_hf_trainset.append(
+                    dict(claim=claim, supporting_facts=supporting_facts, label=label)
+                )
+
+        for example in tqdm.tqdm(hf_testset):
+            claim = example["claim"]
+            supporting_facts = example["supporting_facts"]
+            label = example["label"]
+
+            if count_unique_docs(example) == 3:
+                reformatted_hf_testset.append(
+                    dict(claim=claim, supporting_facts=supporting_facts, label=label)
+                )
+
+        rng = random.Random(0)
+        rng.shuffle(reformatted_hf_trainset)
+        rng = random.Random(1)
+        rng.shuffle(reformatted_hf_testset)
+
+        trainset = reformatted_hf_trainset
+        testset = reformatted_hf_testset
+
+        self.trainset = [dspy.Example(**x).with_inputs("claim") for x in trainset]
+        self.testset = [dspy.Example(**x).with_inputs("claim") for x in testset]
+
+        # Set up metrics
+        NUM_THREADS = 16
+
+        self.metric = discrete_retrieval_eval
+
+        kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=15)
+        self.evaluate = Evaluate(devset=self.trainset, metric=self.metric, **kwargs)
+
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
+
+    def get_program(self):
+        return RetrieveMultiHop()
+
+    def get_metric(self):
+        return self.metric
+
+    def get_default_max_bootstrapped_demos(self):
+        return 1
+
+    def get_default_max_labeled_demos(self):
+        return 2
+
+    def get_max_tokens(self):
+        return 600
diff --git a/testing/tasks/iris.py b/testing/tasks/iris.py
new file mode 100644
index 000000000..ec0646f32
--- /dev/null
+++ b/testing/tasks/iris.py
@@ -0,0 +1,93 @@
+from datasets import load_dataset
+from .base_task import BaseTask
+
+import random
+import dspy
+
+
+class Sig(dspy.Signature):
+    "Given the petal and sepal dimensions in cm, predict the iris species."
+
+    petal_length = dspy.InputField()
+    petal_width = dspy.InputField()
+    sepal_length = dspy.InputField()
+    sepal_width = dspy.InputField()
+    answer = dspy.OutputField(desc="setosa, versicolor, or virginica")
+
+
+class Classify(dspy.Module):
+    def __init__(self):
+        self.pred = dspy.ChainOfThought(Sig)
+
+    def forward(self, petal_length, petal_width, sepal_length, sepal_width):
+        return self.pred(
+            petal_length=petal_length,
+            petal_width=petal_width,
+            sepal_length=sepal_length,
+            sepal_width=sepal_width,
+        )
+
+
+class IrisClassifierTask(BaseTask):
+    def __init__(self):
+        # Read in the conditional HotpotQA dataset from nfl_datasets as a csv from nfl_datasets/conditional_hotpotqa
+        dataset = load_dataset("hitorilabs/iris")
+
+        fullset = [
+            dspy.Example(**{k: str(round(v, 2)) for k, v in example.items()})
+            for example in dataset["train"]
+        ]
+        fullset = [
+            dspy.Example(
+                **{
+                    **x,
+                    "answer": ["setosa", "versicolor", "virginica"][int(x["species"])],
+                }
+            )
+            for x in fullset
+        ]
+        fullset = [
+            x.with_inputs("petal_length", "petal_width", "sepal_length", "sepal_width")
+            for x in fullset
+        ]
+
+        random.Random(0).shuffle(fullset)
+        # self.trainset, self.devset, self.testset = fullset[:25], fullset[20:75], fullset[75:]
+        self.trainset, self.testset = fullset[:75], fullset[75:]
+
+        # Set up metrics
+        NUM_THREADS = 16
+
+        # TODO: set up metrics
+        self.metric = dspy.evaluate.answer_exact_match
+
+        kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=15)
+
+        self.set_splits(TRAIN_NUM=25, DEV_NUM=55, TEST_NUM=100)
+
+    def get_program(self):
+        return Classify()
+
+    def get_metric(self):
+        return self.metric
+
+    def get_default_max_bootstrapped_demos(self):
+        return 4
+
+    def get_default_max_labeled_demos(self):
+        return 0
+
+    def get_handwritten_prompts(self):
+        return []
+
+    def get_trainset(self, TRAIN_NUM=None):
+        return self.trainset
+
+    def get_testset(self):
+        return self.testset
+
+    def get_devset(self, TRAIN_NUM=None, DEV_NUM=None):
+        return self.trainset
+
+    def get_evalset(self, EVAL_NUM=None):
+        return self.testset
diff --git a/testing/tasks/iris_typo.py b/testing/tasks/iris_typo.py
new file mode 100644
index 000000000..9bff74ec6
--- /dev/null
+++ b/testing/tasks/iris_typo.py
@@ -0,0 +1,93 @@
+from datasets import load_dataset
+from .base_task import BaseTask
+
+import random
+import dspy
+
+
+class Sig(dspy.Signature):
+    "Given the petal and sepal dimensions in cm, predict the iris species."
+    # "Given the petal and sepal dimensions in cm, predict the iris species. If petal_width < 1.0, setosa. Otherwise, if petal_width < 1.65, versicolor. Otherwise, virginica."
+
+    petal_length = dspy.InputField()
+    petal_width = dspy.InputField()
+    sepal_length = dspy.InputField()
+    sepal_width = dspy.InputField()
+    answer = dspy.OutputField(desc="setosa, versicolour, or virginica")
+
+
+class Classify(dspy.Module):
+    def __init__(self):
+        self.pred = dspy.ChainOfThought(Sig)
+
+    def forward(self, petal_length, petal_width, sepal_length, sepal_width):
+        return self.pred(
+            petal_length=petal_length,
+            petal_width=petal_width,
+            sepal_length=sepal_length,
+            sepal_width=sepal_width,
+        )
+
+
+class IrisTypoClassifierTask(BaseTask):
+    def __init__(self):
+        # Read in the conditional HotpotQA dataset from nfl_datasets as a csv from nfl_datasets/conditional_hotpotqa
+        dataset = load_dataset("hitorilabs/iris")
+
+        fullset = [
+            dspy.Example(**{k: str(round(v, 2)) for k, v in example.items()})
+            for example in dataset["train"]
+        ]
+        fullset = [
+            dspy.Example(
+                **{
+                    **x,
+                    "answer": ["setosa", "versicolor", "virginica"][int(x["species"])],
+                }
+            )
+            for x in fullset
+        ]
+        fullset = [
+            x.with_inputs("petal_length", "petal_width", "sepal_length", "sepal_width")
+            for x in fullset
+        ]
+
+        random.Random(0).shuffle(fullset)
+        self.trainset, self.testset = fullset[:75], fullset[75:]
+
+        # Set up metrics
+        NUM_THREADS = 16
+
+        # TODO: set up metrics
+        self.metric = dspy.evaluate.answer_exact_match
+
+        kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=15)
+
+        self.set_splits(TRAIN_NUM=25, DEV_NUM=55, TEST_NUM=100)
+
+    def get_program(self):
+        return Classify()
+
+    def get_metric(self):
+        return self.metric
+
+    def get_default_max_bootstrapped_demos(self):
+        return 4
+
+    def get_default_max_labeled_demos(self):
+        return 0
+
+    def get_handwritten_prompts(self):
+        return []
+
+    def get_trainset(self, TRAIN_NUM=None):
+        return self.trainset
+
+    def get_testset(self):
+        return self.testset
+
+    def get_devset(self, TRAIN_NUM=None, DEV_NUM=None):
+        return self.trainset
+
+    def get_evalset(self, EVAL_NUM=None):
+        return self.testset
diff --git a/testing/tasks/scone.py b/testing/tasks/scone.py
index 27e8691ac..2c35dbf4b 100644
--- a/testing/tasks/scone.py
+++ b/testing/tasks/scone.py
@@ -13,29 +13,32 @@ def load_scone(dirname):
     dfs = []
     for filename in glob.glob(dirname + "/*.csv"):
         df = pd.read_csv(filename, index_col=0)
-        df['category'] = os.path.basename(filename).replace(".csv", "")
+        df["category"] = os.path.basename(filename).replace(".csv", "")
         dfs.append(df)
     data_df = pd.concat(dfs)
 
     def as_example(row):
         # The 'one_scoped' file is from an earlier dataset, MoNLI, and
         # so is formatted a bit differently:
-        suffix = '' if row['category'] == 'one_scoped' else '_edited'
+        suffix = "" if row["category"] == "one_scoped" else "_edited"
         # Reformat the hypothesis to be an embedded clause in a question:
-        hkey = 'sentence2' + suffix
-        question = row[hkey][0].lower() + row[hkey][1: ].strip(".")
+        hkey = "sentence2" + suffix
+        question = row[hkey][0].lower() + row[hkey][1:].strip(".")
         question = f"Can we logically conclude for sure that {question}?"
         # Binary task formulation:
-        label = "Yes" if row['gold_label' + suffix] == 'entailment' else "No"
-        return dspy.Example({
-            "context": row['sentence1' + suffix],
-            "question": question,
-            "answer": label,
-            "category": row['category'],
-        }).with_inputs("context", "question")
+        label = "Yes" if row["gold_label" + suffix] == "entailment" else "No"
+        return dspy.Example(
+            {
+                "context": row["sentence1" + suffix],
+                "question": question,
+                "answer": label,
+                "category": row["category"],
+            }
+        ).with_inputs("context", "question")
 
     return list(data_df.apply(as_example, axis=1).values)
 
+
 class ScoNeSignature(dspy.Signature):
     ("""context, question -> answer""")
 
@@ -43,6 +46,7 @@ class ScoNeSignature(dspy.Signature):
     question = dspy.InputField()
     answer = dspy.OutputField(desc="Yes or No")
 
+
 class ScoNeCoT(dspy.Module):
     def __init__(self):
         super().__init__()
@@ -50,7 +54,8 @@ def __init__(self):
 
     def forward(self, context, question):
         return self.generate_answer(context=context, question=question)
-    
+
+
 class ScoNeTask(BaseTask):
     def __init__(self):
         # !git clone https://github.com/selenashe/ScoNe.git
@@ -64,15 +69,18 @@ def __init__(self):
         random.shuffle(all_train)
 
         # 1000 random train, 500 random dev:
-        self.trainset, self.devset = all_train[: 1000], all_train[1000: 1500]
+        self.trainset, self.testset = all_train[:1000], all_train[1000:1500]
 
         metric_EM = dspy.evaluate.answer_exact_match
         self.metric = metric_EM
 
-        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, EVAL_NUM=100)
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
 
     def get_program(self):
         return ScoNeCoT()
-    
+
     def get_metric(self):
-        return self.metric
\ No newline at end of file
+        return self.metric
+
+    def get_max_tokens(self):
+        return 200
diff --git a/testing/tasks/tweet.py b/testing/tasks/tweet.py
index 73870a059..7e366bb9b 100644
--- a/testing/tasks/tweet.py
+++ b/testing/tasks/tweet.py
@@ -17,6 +17,7 @@ class TweetSignature(dspy.Signature):
     question = dspy.InputField()
     answer = dspy.OutputField(desc="Yes or No")
 
+
 class TweetCoT(dspy.Module):
     def __init__(self):
         super().__init__()
@@ -25,43 +26,51 @@ def __init__(self):
     def forward(self, context, question):
         return self.generate_answer(context=context, question=question)
 
+
 class MultiHopTweet(dspy.Module):
-    def __init__(self,passages_per_hop):
+    def __init__(self, passages_per_hop):
         super().__init__()
-        self.retrieve = dspy.Retrieve(k = passages_per_hop)
+        self.retrieve = dspy.Retrieve(k=passages_per_hop)
         self.generate_query = dspy.ChainOfThought("context ,question->search_query")
         self.generate_answer = TweetCoT()
-    
-    def forward (self,question) :
+
+    def forward(self, question):
         context = []
         for hop in range(2):
-            query = self.generate_query(context = context, question = question).search_query
+            query = self.generate_query(context=context, question=question).search_query
             context += self.retrieve(query).passages
-        return dspy.Prediction(context=context, answer=self.generate_answer(context = context , question = question).answer)
+        return dspy.Prediction(
+            context=context,
+            answer=self.generate_answer(context=context, question=question).answer,
+        )
+
 
 # Define the signature for automatic assessments.
 class Assess(dspy.Signature):
     """Assess the quality of a tweet along the specified dimension."""
 
-    context = dspy.InputField(desc='ignore if N/A')
+    context = dspy.InputField(desc="ignore if N/A")
     assessed_text = dspy.InputField()
     assessment_question = dspy.InputField()
     assessment_answer = dspy.OutputField(desc="Yes or No")
 
+
 @lru_cache
 def load_models():
     load_dotenv()  # This will load the .env file's variables
 
-    openai.api_key = os.environ.get('OPENAI_API_KEY')
-    openai.api_base = os.environ.get('OPENAI_API_BASE')
-    gpt4T = dspy.OpenAI(model='gpt-3.5-turbo-1106', max_tokens=1000, model_type='chat')
-    # gpt4T = dspy.OpenAI(model='gpt-4-1106-preview', max_tokens=1000, model_type='chat')
+    openai.api_key = os.environ.get("OPENAI_API_KEY")
+    openai.api_base = os.environ.get("OPENAI_API_BASE")
+    gpt4T = dspy.OpenAI(model="gpt-3.5-turbo-1106", max_tokens=1000, model_type="chat")
     retrieve = dspy.Retrieve(k=5)
     return gpt4T, retrieve
 
+
 METRIC = None
+
+
 def metric(gold, pred, trace=None):
-   
+
     gpt4T, retrieve = load_models()
 
     question, answer, tweet = gold.question, gold.answer, pred.answer
@@ -69,45 +78,73 @@ def metric(gold, pred, trace=None):
 
     engaging = "Does the assessed text make for a self-contained, engaging tweet?"
     faithful = "Is the assessed text grounded in the context? Say no if it includes significant facts not in the context."
-    correct = f"The text above is should answer `{question}`. The gold answer is `{answer}`."
+    correct = (
+        f"The text above is should answer `{question}`. The gold answer is `{answer}`."
+    )
     correct = f"{correct} Does the assessed text above contain the gold answer?"
-    
-    with dspy.context(lm=gpt4T):
-        faithful = dspy.Predict(Assess)(context=context, assessed_text=tweet, assessment_question=faithful)
-        correct =  dspy.Predict(Assess)(context='N/A', assessed_text=tweet, assessment_question=correct)
-        engaging = dspy.Predict(Assess)(context='N/A', assessed_text=tweet, assessment_question=engaging)
 
-    correct, engaging, faithful = (m.assessment_answer.split()[0].lower() == 'yes' for m in [correct, engaging, faithful])
+    with dspy.context(lm=gpt4T):
+        faithful = dspy.Predict(Assess)(
+            context=context, assessed_text=tweet, assessment_question=faithful
+        )
+        correct = dspy.Predict(Assess)(
+            context="N/A", assessed_text=tweet, assessment_question=correct
+        )
+        engaging = dspy.Predict(Assess)(
+            context="N/A", assessed_text=tweet, assessment_question=engaging
+        )
+
+    correct, engaging, faithful = (
+        m.assessment_answer.split()[0].lower() == "yes"
+        for m in [correct, engaging, faithful]
+    )
     score = (correct + engaging + faithful) if correct and (len(tweet) <= 280) else 0
 
     if METRIC is not None:
-        if METRIC == 'correct': return correct
-        if METRIC == 'engaging': return engaging
-        if METRIC == 'faithful': return faithful
-
-    if trace is not None: return score >= 3
+        if METRIC == "correct":
+            return correct
+        if METRIC == "engaging":
+            return engaging
+        if METRIC == "faithful":
+            return faithful
+
+    if trace is not None:
+        return score >= 3
     return score / 3.0
 
+
 class TweetTask(BaseTask):
     def __init__(self):
 
         # Load the dataset.
-        dataset = HotPotQA(train_seed=1, train_size=500, eval_seed=2023, dev_size=200, test_size=0, keep_details=True)
+        dataset = HotPotQA(
+            train_seed=1,
+            train_size=500,
+            eval_seed=2023,
+            dev_size=200,
+            test_size=0,
+            keep_details=True,
+        )
 
         # Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
-        self.trainset = [x.without('id', 'type').with_inputs('question') for x in dataset.train]
-        self.devset = [x.without('id', 'type').with_inputs('question') for x in dataset.dev]
+        self.trainset = [
+            x.without("id", "type").with_inputs("question") for x in dataset.train
+        ]
+        self.testset = [
+            x.without("id", "type").with_inputs("question") for x in dataset.dev
+        ]
 
-        self.trainset = [x.with_inputs('question') for x in dataset.train]
-        self.devset = [x.with_inputs('question') for x in dataset.dev]
+        self.trainset = [x.with_inputs("question") for x in dataset.train]
+        self.testset = [x.with_inputs("question") for x in dataset.dev]
 
         self.metric = metric
 
-        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, EVAL_NUM=100)
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
 
     def get_program(self):
-        return MultiHopTweet(passages_per_hop=3) #TODO: make it so we can specify # of passages 
-    
+        return MultiHopTweet(
+            passages_per_hop=3
+        )  # TODO: make it so we can specify # of passages
+
     def get_metric(self):
         return self.metric
-    
diff --git a/testing/tasks/tweet_metric.py b/testing/tasks/tweet_metric.py
index b91458c48..5eec5f057 100644
--- a/testing/tasks/tweet_metric.py
+++ b/testing/tasks/tweet_metric.py
@@ -20,6 +20,7 @@ class TweetSignature(dspy.Signature):
     question = dspy.InputField()
     answer = dspy.OutputField(desc="Yes or No")
 
+
 class TweetCoT(dspy.Module):
     def __init__(self):
         super().__init__()
@@ -28,61 +29,89 @@ def __init__(self):
     def forward(self, context, question):
         return self.generate_answer(context=context, question=question)
 
+
 class MultiHopTweet(dspy.Module):
-    def __init__(self,passages_per_hop):
+    def __init__(self, passages_per_hop):
         super().__init__()
-        self.retrieve = dspy.Retrieve(k = passages_per_hop)
+        self.retrieve = dspy.Retrieve(k=passages_per_hop)
         self.generate_query = dspy.ChainOfThought("context ,question->search_query")
         self.generate_answer = TweetCoT()
-    
-    def forward (self,question) :
+
+    def forward(self, question):
         context = []
         for hop in range(2):
-            query = self.generate_query(context = context, question = question).search_query
+            query = self.generate_query(context=context, question=question).search_query
             context += self.retrieve(query).passages
-        return dspy.Prediction(context=context, answer=self.generate_answer(context = context , question = question).answer)
+        return dspy.Prediction(
+            context=context,
+            answer=self.generate_answer(context=context, question=question).answer,
+        )
+
 
 # Define the signature for automatic assessments.
 class Assess(dspy.Signature):
     """Assess the quality of a tweet along the specified dimension."""
 
-    context = dspy.InputField(desc='ignore if N/A')
+    context = dspy.InputField(desc="ignore if N/A")
     assessed_text = dspy.InputField()
     assessment_question = dspy.InputField()
     assessment_answer = dspy.OutputField(desc="Yes or No")
 
+
 @lru_cache
 def load_models():
     load_dotenv()  # This will load the .env file's variables
 
-    openai.api_key = os.environ.get('OPENAI_API_KEY')
-    openai.api_base = os.environ.get('OPENAI_API_BASE')
-    gpt3T = dspy.OpenAI(model='gpt-3.5-turbo-1106', max_tokens=1000, model_type='chat')
-    gpt4T = dspy.OpenAI(model='gpt-4-1106-preview', max_tokens=1000, model_type='chat')
+    openai.api_key = os.environ.get("OPENAI_API_KEY")
+    openai.api_base = os.environ.get("OPENAI_API_BASE")
+    gpt3T = dspy.OpenAI(model="gpt-3.5-turbo-1106", max_tokens=1000, model_type="chat")
+    gpt4T = dspy.OpenAI(model="gpt-4-1106-preview", max_tokens=1000, model_type="chat")
     retrieve = dspy.Retrieve(k=5)
     return gpt3T, gpt4T, retrieve
 
+
 METRIC = None
+
+
 def metric(gold, pred, trace=None):
     gpt3T, gpt4T, retrieve = load_models()
 
-    question, answer, tweet, context = gold.question, gold.answer, gold.tweet, gold.context
+    question, answer, tweet, context = (
+        gold.question,
+        gold.answer,
+        gold.tweet,
+        gold.context,
+    )
     score_pred = pred.score
 
     engaging = "Does the assessed text make for a self-contained, engaging tweet?"
     faithful = "Is the assessed text grounded in the context? Say no if it includes significant facts not in the context."
-    correct = f"The text above is should answer `{question}`. The gold answer is `{answer}`."
+    correct = (
+        f"The text above is should answer `{question}`. The gold answer is `{answer}`."
+    )
     correct = f"{correct} Does the assessed text above contain the gold answer?"
-    
-    with dspy.context(lm=gpt3T): # TODO Update to GPT4
-        faithful = dspy.Predict(Assess)(context=context, assessed_text=tweet, assessment_question=faithful)
-        correct =  dspy.Predict(Assess)(context='N/A', assessed_text=tweet, assessment_question=correct)
-        engaging = dspy.Predict(Assess)(context='N/A', assessed_text=tweet, assessment_question=engaging)
 
-    correct, engaging, faithful = (m.assessment_answer.split()[0].lower() == 'yes' for m in [correct, engaging, faithful])
+    with dspy.context(lm=gpt3T):  # TODO Update to GPT4
+        faithful = dspy.Predict(Assess)(
+            context=context, assessed_text=tweet, assessment_question=faithful
+        )
+        correct = dspy.Predict(Assess)(
+            context="N/A", assessed_text=tweet, assessment_question=correct
+        )
+        engaging = dspy.Predict(Assess)(
+            context="N/A", assessed_text=tweet, assessment_question=engaging
+        )
+
+    correct, engaging, faithful = (
+        m.assessment_answer.split()[0].lower() == "yes"
+        for m in [correct, engaging, faithful]
+    )
     score = (correct + engaging + faithful) if correct and (len(tweet) <= 280) else 0
 
-    return 1 - abs(score - score_pred) # We want a score we can maximize, so take the negative L1 norm and add 1
+    return 1 - abs(
+        score - score_pred
+    )  # We want a score we can maximize, so take the negative L1 norm and add 1
+
 
 class TweetMetric(dspy.Module):
     def __init__(self):
@@ -90,33 +119,56 @@ def __init__(self):
         self.engaging = dspy.Predict(Assess)
         self.faithful = dspy.Predict(Assess)
         self.correct = dspy.Predict(Assess)
-    
-    def forward (self, tweet, context, question, answer) :
+
+    def forward(self, tweet, context, question, answer):
         engaging = "Does the assessed text make for a self-contained, engaging tweet?"
         faithful = "Is the assessed text grounded in the context? Say no if it includes significant facts not in the context."
         correct = f"The text above is should answer `{question}`. The gold answer is `{answer}`."
         correct = f"{correct} Does the assessed text above contain the gold answer?"
-        
-        faithful = self.faithful(context=context, assessed_text=tweet, assessment_question=faithful)
-        correct =  self.correct(context='N/A', assessed_text=tweet, assessment_question=correct)
-        engaging = self.engaging(context='N/A', assessed_text=tweet, assessment_question=engaging)
 
-        correct, engaging, faithful = (m.assessment_answer.split()[0].lower() == 'yes' for m in [correct, engaging, faithful])
-        score = (correct + engaging + faithful) if correct and (len(tweet) <= 280) else 0
+        faithful = self.faithful(
+            context=context, assessed_text=tweet, assessment_question=faithful
+        )
+        correct = self.correct(
+            context="N/A", assessed_text=tweet, assessment_question=correct
+        )
+        engaging = self.engaging(
+            context="N/A", assessed_text=tweet, assessment_question=engaging
+        )
+
+        correct, engaging, faithful = (
+            m.assessment_answer.split()[0].lower() == "yes"
+            for m in [correct, engaging, faithful]
+        )
+        score = (
+            (correct + engaging + faithful) if correct and (len(tweet) <= 280) else 0
+        )
+
+        return dspy.Prediction(score=score / 3.0)
 
-        return dspy.Prediction(score= score/3.0)
 
 class TweetMetricTask(BaseTask):
     def __init__(self):
 
         # Load the dataset.
-        dataset = HotPotQA(train_seed=1, train_size=500, eval_seed=2023, dev_size=200, test_size=0, keep_details=True)
+        dataset = HotPotQA(
+            train_seed=1,
+            train_size=500,
+            eval_seed=2023,
+            dev_size=200,
+            test_size=0,
+            keep_details=True,
+        )
 
         # Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
-        trainset_temp = [x.without('id', 'type').with_inputs('question') for x in dataset.train]
-        devset_temp = [x.without('id', 'type').with_inputs('question') for x in dataset.dev]
+        trainset_temp = [
+            x.without("id", "type").with_inputs("question") for x in dataset.train
+        ]
+        devset_temp = [
+            x.without("id", "type").with_inputs("question") for x in dataset.dev
+        ]
         self.trainset = []
-        self.devset = []
+        self.testset = []
 
         gpt3T, gpt4T, retrieve = load_models()
 
@@ -126,26 +178,43 @@ def __init__(self):
                 question = ex.question
                 answer = ex.answer
                 tweet = MultiHopTweet(passages_per_hop=3)(ex.question).answer
-                example = {'context': context, 'question': question, 'answer': answer, 'tweet': tweet}
-
-                self.trainset.append(Example(**example, dspy_uuid=str(uuid.uuid4()), dspy_split='train').with_inputs('context', 'question', 'answer', 'tweet'))
+                example = {
+                    "context": context,
+                    "question": question,
+                    "answer": answer,
+                    "tweet": tweet,
+                }
+
+                self.trainset.append(
+                    Example(
+                        **example, dspy_uuid=str(uuid.uuid4()), dspy_split="train"
+                    ).with_inputs("context", "question", "answer", "tweet")
+                )
 
             for ex in tqdm(devset_temp, desc="Preprocessing Devset"):
                 context = retrieve(ex.question).passages
                 question = ex.question
                 answer = ex.answer
                 tweet = MultiHopTweet(passages_per_hop=3)(ex.question).answer
-                example = {'context': context, 'question': question, 'answer': answer, 'tweet': tweet}
-
-                self.devset.append(Example(**example, dspy_uuid=str(uuid.uuid4()), dspy_split='dev').with_inputs('context', 'question', 'answer', 'tweet'))
+                example = {
+                    "context": context,
+                    "question": question,
+                    "answer": answer,
+                    "tweet": tweet,
+                }
+
+                self.testset.append(
+                    Example(
+                        **example, dspy_uuid=str(uuid.uuid4()), dspy_split="dev"
+                    ).with_inputs("context", "question", "answer", "tweet")
+                )
 
         self.metric = metric
 
-        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, EVAL_NUM=100)
+        self.set_splits(TRAIN_NUM=100, DEV_NUM=100, TEST_NUM=100)
 
     def get_program(self):
         return TweetMetric()
-    
+
     def get_metric(self):
         return self.metric
-    
diff --git a/tests/conftest.py b/tests/conftest.py
new file mode 100644
index 000000000..0bb74ac90
--- /dev/null
+++ b/tests/conftest.py
@@ -0,0 +1,13 @@
+import pytest
+
+import dspy
+from dsp.utils.settings import DEFAULT_CONFIG
+
+
+@pytest.fixture(autouse=True)
+def clear_settings():
+    """Ensures that the settings are cleared after each test."""
+
+    yield
+
+    dspy.settings.configure(**DEFAULT_CONFIG, inherit_config=False)
diff --git a/tests/dsp_LM/__init__.py b/tests/dsp_LM/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/evaluate/__init__.py b/tests/dsp_LM/evaluate/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/evaluate/test_evaluate.py b/tests/dsp_LM/evaluate/test_evaluate.py
new file mode 100644
index 000000000..5c6c1f82e
--- /dev/null
+++ b/tests/dsp_LM/evaluate/test_evaluate.py
@@ -0,0 +1,100 @@
+import signal
+import threading
+from unittest.mock import patch
+
+import pytest
+
+import dsp
+import dspy
+from dspy.evaluate.evaluate import Evaluate
+from dspy.evaluate.metrics import answer_exact_match
+from dspy.predict import Predict
+from dspy.utils.dummies import DSPDummyLM
+
+
+def new_example(question, answer):
+    """Helper function to create a new example."""
+    return dspy.Example(
+        question=question,
+        answer=answer,
+    ).with_inputs("question")
+
+
+def test_evaluate_call():
+    dspy.settings.configure(lm=DSPDummyLM({"What is 1+1?": "2", "What is 2+2?": "4"}))
+    devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
+    program = Predict("question -> answer")
+    assert program(question="What is 1+1?").answer == "2"
+    ev = Evaluate(
+        devset=devset,
+        metric=answer_exact_match,
+        display_progress=False,
+    )
+    score = ev(program)
+    assert score == 100.0
+
+
+def test_multithread_evaluate_call():
+    dspy.settings.configure(lm=DSPDummyLM({"What is 1+1?": "2", "What is 2+2?": "4"}))
+    devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
+    program = Predict("question -> answer")
+    assert program(question="What is 1+1?").answer == "2"
+    ev = Evaluate(
+        devset=devset,
+        metric=answer_exact_match,
+        display_progress=False,
+        num_threads=2,
+    )
+    score = ev(program)
+    assert score == 100.0
+
+
+def test_multi_thread_evaluate_call_cancelled(monkeypatch):
+    # slow LM that sleeps for 1 second before returning the answer
+    class SlowLM(DSPDummyLM):
+        def __call__(self, prompt, **kwargs):
+            import time
+
+            time.sleep(1)
+            return super().__call__(prompt, **kwargs)
+
+    dspy.settings.configure(lm=SlowLM({"What is 1+1?": "2", "What is 2+2?": "4"}))
+
+    devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
+    program = Predict("question -> answer")
+    assert program(question="What is 1+1?").answer == "2"
+
+    # spawn a thread that will sleep for .1 seconds then send a KeyboardInterrupt
+    def sleep_then_interrupt():
+        import time
+
+        time.sleep(0.1)
+        import os
+
+        os.kill(os.getpid(), signal.SIGINT)
+
+    input_thread = threading.Thread(target=sleep_then_interrupt)
+    input_thread.start()
+
+    with pytest.raises(KeyboardInterrupt):
+        ev = Evaluate(
+            devset=devset,
+            metric=answer_exact_match,
+            display_progress=False,
+            num_threads=2,
+        )
+        score = ev(program)
+        assert score == 100.0
+
+
+def test_evaluate_call_bad():
+    dspy.settings.configure(lm=DSPDummyLM({"What is 1+1?": "0", "What is 2+2?": "0"}))
+    devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
+    program = Predict("question -> answer")
+    ev = Evaluate(
+        devset=devset,
+        metric=answer_exact_match,
+        display_progress=False,
+    )
+    score = ev(program)
+    assert score == 0.0
diff --git a/tests/dsp_LM/examples/__init__.py b/tests/dsp_LM/examples/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/functional/__init__.py b/tests/dsp_LM/functional/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/functional/test_functional.py b/tests/dsp_LM/functional/test_functional.py
new file mode 100644
index 000000000..31d0416b7
--- /dev/null
+++ b/tests/dsp_LM/functional/test_functional.py
@@ -0,0 +1,906 @@
+import datetime
+import textwrap
+from typing import Annotated, Any, Generic, List, Literal, Optional, TypeVar
+
+import pydantic
+import pytest
+from pydantic import AfterValidator, BaseModel, Field, ValidationError, field_validator, model_validator
+
+import dspy
+from dspy.functional import FunctionalModule, TypedChainOfThought, TypedPredictor, cot, predictor
+from dspy.predict.predict import Predict
+from dspy.primitives.example import Example
+from dspy.teleprompt.bootstrap import BootstrapFewShot
+from dspy.teleprompt.vanilla import LabeledFewShot
+from dspy.utils.dummies import DSPDummyLM
+
+
+def test_simple():
+    @predictor
+    def hard_question(topic: str) -> str:
+        """Think of a hard factual question about a topic."""
+
+    expected = "What is the speed of light?"
+    lm = DSPDummyLM([expected])
+    dspy.settings.configure(lm=lm)
+
+    question = hard_question(topic="Physics")
+    lm.inspect_history(n=2)
+
+    assert question == expected
+
+
+def test_list_output():
+    @predictor
+    def hard_questions(topics: List[str]) -> List[str]:
+        pass
+
+    expected = ["What is the speed of light?", "What is the speed of sound?"]
+    lm = DSPDummyLM(['["What is the speed of light?", "What is the speed of sound?"]'])
+    dspy.settings.configure(lm=lm)
+
+    question = hard_questions(topics=["Physics", "Music"])
+    lm.inspect_history(n=2)
+
+    assert question == expected
+
+
+def test_simple_type():
+    class Question(pydantic.BaseModel):
+        value: str
+
+    @predictor
+    def hard_question(topic: str) -> Question:
+        """Think of a hard factual question about a topic."""
+
+    expected = "What is the speed of light?"
+    lm = DSPDummyLM([f'{{"value": "{expected}"}}'])
+    dspy.settings.configure(lm=lm)
+
+    question = hard_question(topic="Physics")
+
+    assert isinstance(question, Question)
+    assert question.value == expected
+
+
+def test_simple_type_input():
+    class Question(pydantic.BaseModel):
+        value: str
+
+    class Answer(pydantic.BaseModel):
+        value: str
+
+    @predictor
+    def answer(question: Question) -> Answer:
+        pass
+
+    question = Question(value="What is the speed of light?")
+    lm = DSPDummyLM([f'{{"value": "3e8"}}'])
+    dspy.settings.configure(lm=lm)
+
+    result = answer(question=question)
+
+    assert result == Answer(value="3e8")
+
+
+def test_simple_class():
+    class Answer(pydantic.BaseModel):
+        value: float
+        certainty: float
+        comments: List[str] = pydantic.Field(description="At least two comments about the answer")
+
+    class QA(FunctionalModule):
+        @predictor
+        def hard_question(self, topic: str) -> str:
+            """Think of a hard factual question about a topic. It should be answerable with a number."""
+
+        @cot
+        def answer(self, question: Annotated[str, "Question to answer"]) -> Answer:
+            pass
+
+        def forward(self, **kwargs):
+            question = self.hard_question(**kwargs)
+            return (question, self.answer(question=question))
+
+    expected = Answer(
+        value=3e8,
+        certainty=0.9,
+        comments=["It is the speed of light", "It is a constant"],
+    )
+
+    lm = DSPDummyLM(
+        [
+            "What is the speed of light?",
+            "Some bad reasoning, 3e8 m/s.",
+            "3e8",  # Bad answer 1
+            "{...}",  # Model is asked to create an example
+            "Some good reasoning...",
+            expected.model_dump_json(),  # Good answer
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    qa = QA()
+    assert isinstance(qa, FunctionalModule)
+    assert isinstance(qa.answer, dspy.Module)
+
+    question, answer = qa(topic="Physics")
+
+    print(qa.answer)
+
+    assert question == "What is the speed of light?"
+    assert answer == expected
+
+
+def test_simple_oop():
+    class Question(pydantic.BaseModel):
+        value: str
+
+    class MySignature(dspy.Signature):
+        topic: str = dspy.InputField()
+        output: Question = dspy.OutputField()
+
+    # Run the signature
+    program = TypedPredictor(MySignature)
+    expected = "What is the speed of light?"
+    lm = DSPDummyLM(
+        [
+            Question(value=expected).model_dump_json(),
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    question = program(topic="Physics").output
+
+    assert isinstance(question, Question)
+    assert question.value == expected
+
+
+def test_bootstrap_effectiveness():
+    class SimpleModule(FunctionalModule):
+        @predictor
+        def output(self, input: str) -> str:
+            pass
+
+        def forward(self, **kwargs):
+            return self.output(**kwargs)
+
+    def simple_metric(example, prediction, trace=None):
+        return example.output == prediction.output
+
+    examples = [
+        ex.with_inputs("input")
+        for ex in (
+            Example(input="What is the color of the sky?", output="blue"),
+            Example(
+                input="What does the fox say?",
+                output="Ring-ding-ding-ding-dingeringeding!",
+            ),
+        )
+    ]
+    trainset = [examples[0]]
+    valset = [examples[1]]
+
+    # This test verifies if the bootstrapping process improves the student's predictions
+    student = SimpleModule()
+    teacher = SimpleModule()
+    assert student.output.predictor.signature.equals(teacher.output.predictor.signature)
+
+    lm = DSPDummyLM(["blue", "Ring-ding-ding-ding-dingeringeding!"], follow_examples=True)
+    dspy.settings.configure(lm=lm, trace=[])
+
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
+
+    lm.inspect_history(n=2)
+
+    # Check that the compiled student has the correct demos
+    _, predict = next(compiled_student.named_sub_modules(Predict, skip_compiled=False))
+    demos = predict.demos
+    assert len(demos) == 1
+    assert demos[0].input == trainset[0].input
+    assert demos[0].output == trainset[0].output
+
+    # Test the compiled student's prediction.
+    # We are using a DSPDummyLM with follow_examples=True, which means that
+    # even though it would normally reply with "Ring-ding-ding-ding-dingeringeding!"
+    # on the second output, if it seems an example that perfectly matches the
+    # prompt, it will use that instead. That is why we expect "blue" here.
+    prediction = compiled_student(input=trainset[0].input)
+    assert prediction == trainset[0].output
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `input`, produce the fields `output`.
+
+        ---
+
+        Follow the following format.
+
+        Input: ${input}
+        Output: ${output}
+
+        ---
+
+        Input: What is the color of the sky?
+        Output: blue
+
+        ---
+
+        Input: What is the color of the sky?
+        Output: blue"""
+    )
+
+
+def test_regex():
+    class TravelInformation(BaseModel):
+        origin: str = Field(pattern=r"^[A-Z]{3}$")
+        destination: str = Field(pattern=r"^[A-Z]{3}$")
+        date: datetime.date
+
+    @predictor
+    def flight_information(email: str) -> TravelInformation:
+        pass
+
+    email = textwrap.dedent(
+        """\
+        We're excited to welcome you aboard your upcoming flight from
+        John F. Kennedy International Airport (JFK) to Los Angeles International Airport (LAX)
+        on December 25, 2022. Here's everything you need to know before you take off: ...
+    """
+    )
+    lm = DSPDummyLM(
+        [
+            # Example with a bad origin code.
+            '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}',
+            # Example to help the model understand
+            "{...}",
+            # Fixed
+            '{"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}',
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    assert flight_information(email=email) == TravelInformation(
+        origin="JFK", destination="LAX", date=datetime.date(2022, 12, 25)
+    )
+
+
+def test_custom_model_validate_json():
+    class Airport(BaseModel):
+        code: str = Field(pattern=r"^[A-Z]{3}$")
+        lat: float
+        lon: float
+
+    class TravelInformation(BaseModel):
+        origin: Airport
+        destination: Airport
+        date: datetime.date
+
+        @classmethod
+        def model_validate_json(
+            cls, json_data: str, *, strict: Optional[bool] = None, context: Optional[dict[str, Any]] = None
+        ) -> "TravelInformation":
+            try:
+                __tracebackhide__ = True
+                return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
+            except ValidationError:
+                for substring_length in range(len(json_data), 1, -1):
+                    for start in range(len(json_data) - substring_length + 1):
+                        substring = json_data[start : start + substring_length]
+                        try:
+                            __tracebackhide__ = True
+                            res = cls.__pydantic_validator__.validate_json(substring, strict=strict, context=context)
+                            return res
+                        except ValidationError as exc:
+                            last_exc = exc
+                            pass
+            raise ValueError("Could not find valid json") from last_exc
+
+    @predictor
+    def flight_information(email: str) -> TravelInformation:
+        pass
+
+    email = textwrap.dedent(
+        """\
+        We're excited to welcome you aboard your upcoming flight from
+        John F. Kennedy International Airport (JFK) to Los Angeles International Airport (LAX)
+        on December 25, 2022. Here's everything you need to know before you take off: ...
+    """
+    )
+    lm = DSPDummyLM(
+        [
+            # Example with a bad origin code.
+            (
+                "Here is your json: "
+                "{"
+                '"origin": {"code":"JFK", "lat":40.6446, "lon":-73.7797}, '
+                '"destination": {"code":"LAX", "lat":33.942791, "lon":-118.410042}, '
+                '"date": "2022-12-25"}'
+            ),
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    assert flight_information(email=email) == TravelInformation(
+        origin={"code": "JFK", "lat": 40.6446, "lon": -73.7797},
+        destination={"code": "LAX", "lat": 33.942791, "lon": -118.410042},
+        date=datetime.date(2022, 12, 25),
+    )
+
+
+def test_raises():
+    class TravelInformation(BaseModel):
+        origin: str = Field(pattern=r"^[A-Z]{3}$")
+        destination: str = Field(pattern=r"^[A-Z]{3}$")
+        date: datetime.date
+
+    @predictor
+    def flight_information(email: str) -> TravelInformation:
+        pass
+
+    lm = DSPDummyLM(
+        [
+            "A list of bad inputs",
+            '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}',
+            '{"origin": "JFK", "destination": "LAX", "date": "bad date"}',
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    with pytest.raises(ValueError):
+        flight_information(email="Some email")
+
+
+def test_multi_errors():
+    class TravelInformation(BaseModel):
+        origin: str = Field(pattern=r"^[A-Z]{3}$")
+        destination: str = Field(pattern=r"^[A-Z]{3}$")
+        date: datetime.date
+
+    @predictor
+    def flight_information(email: str) -> TravelInformation:
+        pass
+
+    lm = DSPDummyLM(
+        [
+            # First origin is wrong, then destination, then all is good
+            '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}',
+            "{...}",  # Example to help the model understand
+            '{"origin": "JFK", "destination": "LA0", "date": "2022-12-25"}',
+            "{...}",  # Example to help the model understand
+            '{"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}',
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    assert flight_information(email="Some email") == TravelInformation(
+        origin="JFK", destination="LAX", date=datetime.date(2022, 12, 25)
+    )
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `email`, produce the fields `flight_information`.
+
+        ---
+
+        Follow the following format.
+
+        Email: ${email}
+
+        Past Error in Flight Information: An error to avoid in the future
+
+        Past Error (2) in Flight Information: An error to avoid in the future
+
+        Flight Information: ${flight_information}. Respond with a single JSON object. JSON Schema: {"properties": {"origin": {"pattern": "^[A-Z]{3}$", "title": "Origin", "type": "string"}, "destination": {"pattern": "^[A-Z]{3}$", "title": "Destination", "type": "string"}, "date": {"format": "date", "title": "Date", "type": "string"}}, "required": ["origin", "destination", "date"], "title": "TravelInformation", "type": "object"}
+
+        ---
+
+        Email: Some email
+
+        Past Error in Flight Information: String should match pattern '^[A-Z]{3}$': origin (error type: string_pattern_mismatch)
+
+        Past Error (2) in Flight Information: String should match pattern '^[A-Z]{3}$': destination (error type: string_pattern_mismatch)
+
+        Flight Information: {"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}"""
+    )
+
+
+def test_field_validator():
+    class UserDetails(BaseModel):
+        name: str
+        age: int
+
+        @field_validator("name")
+        @classmethod
+        def validate_name(cls, v):
+            if v.upper() != v:
+                raise ValueError("Name must be in uppercase.")
+            return v
+
+    @predictor
+    def get_user_details() -> UserDetails:
+        pass
+
+    # Keep making the mistake (lower case name) until we run
+    # out of retries.
+    lm = DSPDummyLM(
+        [
+            '{"name": "lower case name", "age": 25}',
+        ]
+        * 10
+    )
+    dspy.settings.configure(lm=lm)
+
+    with pytest.raises(ValueError):
+        get_user_details()
+
+    print(lm.get_convo(-1))
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields , produce the fields `get_user_details`.
+
+        ---
+
+        Follow the following format.
+
+        Past Error in Get User Details: An error to avoid in the future
+        Past Error (2) in Get User Details: An error to avoid in the future
+        Get User Details: ${get_user_details}. Respond with a single JSON object. JSON Schema: {"properties": {"name": {"title": "Name", "type": "string"}, "age": {"title": "Age", "type": "integer"}}, "required": ["name", "age"], "title": "UserDetails", "type": "object"}
+
+        ---
+
+        Past Error in Get User Details: Value error, Name must be in uppercase.: name (error type: value_error)
+        Past Error (2) in Get User Details: Value error, Name must be in uppercase.: name (error type: value_error)
+        Get User Details: {"name": "lower case name", "age": 25}"""
+    )
+
+
+def test_annotated_field():
+    @predictor
+    def test(input: Annotated[str, Field(description="description")]) -> Annotated[float, Field(gt=0, lt=1)]:
+        pass
+
+    # First try 0, which fails, then try 0.5, which passes
+    lm = DSPDummyLM(["0", "0.5"])
+    dspy.settings.configure(lm=lm)
+
+    output = test(input="input")
+
+    assert output == 0.5
+
+
+def test_multiple_outputs():
+    lm = DSPDummyLM([str(i) for i in range(100)])
+    dspy.settings.configure(lm=lm)
+
+    test = TypedPredictor("input -> output")
+    output = test(input="input", config=dict(n=3)).completions.output
+    assert output == ["0", "1", "2"]
+
+
+def test_multiple_outputs_int():
+    lm = DSPDummyLM([str(i) for i in range(100)])
+    dspy.settings.configure(lm=lm)
+
+    class TestSignature(dspy.Signature):
+        input: int = dspy.InputField()
+        output: int = dspy.OutputField()
+
+    test = TypedPredictor(TestSignature)
+
+    output = test(input=8, config=dict(n=3)).completions.output
+    assert output == [0, 1, 2]
+
+
+def test_multiple_outputs_int_cot():
+    # Note: Multiple outputs only work when the language model "speculatively" generates all the outputs in one go.
+    lm = DSPDummyLM(
+        [
+            "thoughts 0\nOutput: 0\n",
+            "thoughts 1\nOutput: 1\n",
+            "thoughts 2\nOutput: 2\n",
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    test = TypedChainOfThought("input:str -> output:int")
+
+    output = test(input="8", config=dict(n=3)).completions.output
+    assert output == [0, 1, 2]
+
+
+def test_parse_type_string():
+    lm = DSPDummyLM([str(i) for i in range(100)])
+    dspy.settings.configure(lm=lm)
+
+    test = TypedPredictor("input:int -> output:int")
+
+    output = test(input=8, config=dict(n=3)).completions.output
+    assert output == [0, 1, 2]
+
+
+def test_literal():
+    lm = DSPDummyLM(['"2"', '"3"'])
+    dspy.settings.configure(lm=lm)
+
+    @predictor
+    def f() -> Literal["2", "3"]:
+        pass
+
+    assert f() == "2"
+
+
+def test_literal_missmatch():
+    lm = DSPDummyLM([f'"{i}"' for i in range(5, 100)])
+    dspy.settings.configure(lm=lm)
+
+    @predictor(max_retries=1)
+    def f() -> Literal["2", "3"]:
+        pass
+
+    with pytest.raises(Exception) as e_info:
+        f()
+
+    assert e_info.value.args[1]["f"] == "Input should be '2' or '3':  (error type: literal_error)"
+
+
+def test_literal_int():
+    lm = DSPDummyLM(["2", "3"])
+    dspy.settings.configure(lm=lm)
+
+    @predictor
+    def f() -> Literal[2, 3]:
+        pass
+
+    assert f() == 2
+
+
+def test_literal_int_missmatch():
+    lm = DSPDummyLM([f"{i}" for i in range(5, 100)])
+    dspy.settings.configure(lm=lm)
+
+    @predictor(max_retries=1)
+    def f() -> Literal[2, 3]:
+        pass
+
+    with pytest.raises(Exception) as e_info:
+        f()
+
+    assert e_info.value.args[1]["f"] == "Input should be 2 or 3:  (error type: literal_error)"
+
+
+def test_fields_on_base_signature():
+    class SimpleOutput(dspy.Signature):
+        output: float = dspy.OutputField(gt=0, lt=1)
+
+    lm = DSPDummyLM(
+        [
+            "2.1",  # Bad output
+            "0.5",  # Good output
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    predictor = TypedPredictor(SimpleOutput)
+
+    assert predictor().output == 0.5
+
+
+def test_synthetic_data_gen():
+    class SyntheticFact(BaseModel):
+        fact: str = Field(..., description="a statement")
+        varacity: bool = Field(..., description="is the statement true or false")
+
+    class ExampleSignature(dspy.Signature):
+        """Generate an example of a synthetic fact."""
+
+        fact: SyntheticFact = dspy.OutputField()
+
+    lm = DSPDummyLM(
+        [
+            '{"fact": "The sky is blue", "varacity": true}',
+            '{"fact": "The sky is green", "varacity": false}',
+            '{"fact": "The sky is red", "varacity": true}',
+            '{"fact": "The earth is flat", "varacity": false}',
+            '{"fact": "The earth is round", "varacity": true}',
+            '{"fact": "The earth is a cube", "varacity": false}',
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    generator = TypedPredictor(ExampleSignature)
+    examples = generator(config=dict(n=3))
+    for ex in examples.completions.fact:
+        assert isinstance(ex, SyntheticFact)
+    assert examples.completions.fact[0] == SyntheticFact(fact="The sky is blue", varacity=True)
+
+    # If you have examples and want more
+    existing_examples = [
+        dspy.Example(fact="The sky is blue", varacity=True),
+        dspy.Example(fact="The sky is green", varacity=False),
+    ]
+    trained = LabeledFewShot().compile(student=generator, trainset=existing_examples)
+
+    augmented_examples = trained(config=dict(n=3))
+    for ex in augmented_examples.completions.fact:
+        assert isinstance(ex, SyntheticFact)
+
+
+def test_list_input2():
+    # Inspired by the Signature Optimizer
+
+    class ScoredString(pydantic.BaseModel):
+        string: str
+        score: float
+
+    class ScoredSignature(dspy.Signature):
+        attempted_signatures: list[ScoredString] = dspy.InputField()
+        proposed_signature: str = dspy.OutputField()
+
+    program = TypedChainOfThought(ScoredSignature)
+
+    lm = DSPDummyLM(["Thoughts", "Output"])
+    dspy.settings.configure(lm=lm)
+
+    output = program(
+        attempted_signatures=[
+            ScoredString(string="string 1", score=0.5),
+            ScoredString(string="string 2", score=0.4),
+            ScoredString(string="string 3", score=0.3),
+        ]
+    ).proposed_signature
+
+    print(lm.get_convo(-1))
+
+    assert output == "Output"
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `attempted_signatures`, produce the fields `proposed_signature`.
+
+        ---
+
+        Follow the following format.
+
+        Attempted Signatures: ${attempted_signatures}
+        Reasoning: Let's think step by step in order to ${produce the proposed_signature}. We ...
+        Proposed Signature: ${proposed_signature}
+
+        ---
+
+        Attempted Signatures: [{"string":"string 1","score":0.5},{"string":"string 2","score":0.4},{"string":"string 3","score":0.3}]
+        Reasoning: Let's think step by step in order to Thoughts
+        Proposed Signature: Output"""
+    )
+
+
+def test_custom_reasoning_field():
+    class Question(pydantic.BaseModel):
+        value: str
+
+    class QuestionSignature(dspy.Signature):
+        topic: str = dspy.InputField()
+        question: Question = dspy.OutputField()
+
+    reasoning = dspy.OutputField(
+        prefix="Custom Reasoning: Let's break this down. To generate a question about",
+        desc="${topic}, we should ...",
+    )
+
+    program = TypedChainOfThought(QuestionSignature, reasoning=reasoning)
+
+    expected = "What is the speed of light?"
+    lm = DSPDummyLM(["Thoughts", f'{{"value": "{expected}"}}'])
+    dspy.settings.configure(lm=lm)
+
+    output = program(topic="Physics")
+
+    assert isinstance(output.question, Question)
+    assert output.question.value == expected
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `topic`, produce the fields `question`.
+
+        ---
+
+        Follow the following format.
+
+        Topic: ${topic}
+        Custom Reasoning: Let's break this down. To generate a question about ${topic}, we should ...
+        Question: ${question}. Respond with a single JSON object. JSON Schema: {"properties": {"value": {"title": "Value", "type": "string"}}, "required": ["value"], "title": "Question", "type": "object"}
+
+        ---
+
+        Topic: Physics
+        Custom Reasoning: Let's break this down. To generate a question about Thoughts
+        Question: {"value": "What is the speed of light?"}"""
+    )
+
+
+def test_generic_signature():
+    T = TypeVar("T")
+
+    class GenericSignature(dspy.Signature, Generic[T]):
+        """My signature"""
+
+        output: T = dspy.OutputField()
+
+    predictor = TypedPredictor(GenericSignature[int])
+    assert predictor.signature.instructions == "My signature"
+
+    lm = DSPDummyLM(["23"])
+    dspy.settings.configure(lm=lm)
+
+    assert predictor().output == 23
+
+
+def test_lm_as_validator():
+    @predictor
+    def is_square(n: int) -> bool:
+        """Is n a square number?"""
+
+    def check_square(n):
+        assert is_square(n=n)
+        return n
+
+    @predictor
+    def next_square(n: int) -> Annotated[int, AfterValidator(check_square)]:
+        """What is the next square number after n?"""
+
+    lm = DSPDummyLM(["3", "False", "4", "True"])
+    dspy.settings.configure(lm=lm)
+
+    m = next_square(n=2)
+    lm.inspect_history(n=2)
+
+    assert m == 4
+
+
+def test_annotated_validator():
+    def is_square(n: int) -> int:
+        root = n**0.5
+        if not root.is_integer():
+            raise ValueError(f"{n} is not a square")
+        return n
+
+    class MySignature(dspy.Signature):
+        """What is the next square number after n?"""
+
+        n: int = dspy.InputField()
+        next_square: Annotated[int, AfterValidator(is_square)] = dspy.OutputField()
+
+    lm = DSPDummyLM(["3", "4"])
+    dspy.settings.configure(lm=lm)
+
+    m = TypedPredictor(MySignature)(n=2).next_square
+    lm.inspect_history(n=2)
+
+    assert m == 4
+
+
+def test_annotated_validator_functional():
+    def is_square(n: int) -> int:
+        if not (n**0.5).is_integer():
+            raise ValueError(f"{n} is not a square")
+        return n
+
+    @predictor
+    def next_square(n: int) -> Annotated[int, AfterValidator(is_square)]:
+        """What is the next square number after n?"""
+
+    lm = DSPDummyLM(["3", "4"])
+    dspy.settings.configure(lm=lm)
+
+    m = next_square(n=2)
+    lm.inspect_history(n=2)
+
+    assert m == 4
+
+
+def test_demos():
+    demos = [
+        dspy.Example(input="What is the speed of light?", output="3e8"),
+    ]
+    program = LabeledFewShot(k=len(demos)).compile(
+        student=dspy.TypedPredictor("input -> output"),
+        trainset=[ex.with_inputs("input") for ex in demos],
+    )
+
+    lm = DSPDummyLM(["Paris"])
+    dspy.settings.configure(lm=lm)
+
+    assert program(input="What is the capital of France?").output == "Paris"
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `input`, produce the fields `output`.
+
+        ---
+
+        Follow the following format.
+
+        Input: ${input}
+        Output: ${output}
+
+        ---
+
+        Input: What is the speed of light?
+        Output: 3e8
+
+        ---
+
+        Input: What is the capital of France?
+        Output: Paris"""
+    )
+
+
+def _test_demos_missing_input():
+    demos = [dspy.Example(input="What is the speed of light?", output="3e8")]
+    program = LabeledFewShot(k=len(demos)).compile(
+        student=dspy.TypedPredictor("input -> output, thoughts"),
+        trainset=[ex.with_inputs("input") for ex in demos],
+    )
+    dspy.settings.configure(lm=DSPDummyLM(["My thoughts", "Paris"]))
+    assert program(input="What is the capital of France?").output == "Paris"
+
+    assert dspy.settings.lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `input`, produce the fields `output`.
+
+        ---
+
+        Follow the following format.
+
+        Input: ${input}
+        Thoughts: ${thoughts}
+        Output: ${output}
+
+        ---
+
+        Input: What is the speed of light?
+        Output: 3e8
+
+        ---
+
+        Input: What is the capital of France?
+        Thoughts: My thoughts
+        Output: Paris"""
+    )
+
+
+def test_conlist():
+    dspy.settings.configure(lm=DSPDummyLM(["[]", "[1]", "[1, 2]", "[1, 2, 3]"]))
+
+    @predictor
+    def make_numbers(input: str) -> Annotated[list[int], Field(min_items=2)]:
+        pass
+
+    assert make_numbers(input="What are the first two numbers?") == [1, 2]
+
+
+def test_conlist2():
+    dspy.settings.configure(lm=DSPDummyLM(["[]", "[1]", "[1, 2]", "[1, 2, 3]"]))
+
+    make_numbers = TypedPredictor("input:str -> output:Annotated[List[int], Field(min_items=2)]")
+    assert make_numbers(input="What are the first two numbers?").output == [1, 2]
+
+
+def test_model_validator():
+    class MySignature(dspy.Signature):
+        input_data: str = dspy.InputField()
+        allowed_categories: list[str] = dspy.InputField()
+        category: str = dspy.OutputField()
+
+        @model_validator(mode="after")
+        def check_cateogry(self):
+            if self.category not in self.allowed_categories:
+                raise ValueError(f"category not in {self.allowed_categories}")
+            return self
+
+    lm = DSPDummyLM(["horse", "dog"])
+    dspy.settings.configure(lm=lm)
+    predictor = TypedPredictor(MySignature)
+
+    pred = predictor(input_data="What is the best animal?", allowed_categories=["cat", "dog"])
+    assert pred.category == "dog"
diff --git a/tests/dsp_LM/functional/test_signature_opt_typed.py b/tests/dsp_LM/functional/test_signature_opt_typed.py
new file mode 100644
index 000000000..3533926b6
--- /dev/null
+++ b/tests/dsp_LM/functional/test_signature_opt_typed.py
@@ -0,0 +1,187 @@
+import json
+from typing import Generic, TypeVar
+
+import pydantic
+from pydantic_core import to_jsonable_python
+
+import dspy
+from dspy.evaluate import Evaluate
+from dspy.evaluate.metrics import answer_exact_match
+from dspy.functional import TypedPredictor
+from dspy.teleprompt.signature_opt_typed import make_info, optimize_signature
+from dspy.utils import DSPDummyLM
+
+hotpotqa = [
+    ex.with_inputs("question")
+    for ex in [
+        dspy.Example(
+            question="At My Window was released by which American singer-songwriter?",
+            answer="John Townes Van Zandt",
+        ),
+        dspy.Example(
+            question="which  American actor was Candace Kita  guest starred with ",
+            answer="Bill Murray",
+        ),
+        dspy.Example(
+            question="Which of these publications was most recently published, Who Put the Bomp or Self?",
+            answer="Self",
+        ),
+        dspy.Example(
+            question="The Victorians - Their Story In Pictures is a documentary series written by an author born in what year?",
+            answer="1950",
+        ),
+        dspy.Example(
+            question="Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?",
+            answer="Tae Kwon Do Times",
+        ),
+        dspy.Example(
+            question="In what year was the club founded that played Manchester City in the 1972 FA Charity Shield",
+            answer="1874",
+        ),
+        dspy.Example(
+            question="Which is taller, the Empire State Building or the Bank of America Tower?",
+            answer="The Empire State Building",
+        ),
+        dspy.Example(
+            question='Which American actress who made their film debut in the 1995 teen drama "Kids" was the co-founder of Voto Latino?',
+            answer="Rosario Dawson",
+        ),
+        dspy.Example(
+            question="Tombstone stared an actor born May 17, 1955 known as who?",
+            answer="Bill Paxton",
+        ),
+        dspy.Example(
+            question="What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?",
+            answer="Operation Citadel",
+        ),
+        dspy.Example(
+            question='Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet." ?',
+            answer="Kerry Condon",
+        ),
+        dspy.Example(
+            question="Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?",
+            answer="Buena Vista Distribution",
+        ),
+        dspy.Example(
+            question="Samantha Cristoforetti and Mark Shuttleworth are both best known for being first in their field to go where? ",
+            answer="space",
+        ),
+        dspy.Example(
+            question="Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers? ",
+            answer="Outfield of Dreams",
+        ),
+        dspy.Example(
+            question="Which Pakistani cricket umpire who won 3 consecutive ICC umpire of the year awards in 2009, 2010, and 2011 will be in the ICC World Twenty20?",
+            answer="Aleem Sarwar Dar",
+        ),
+        dspy.Example(
+            question="The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?",
+            answer="2010",
+        ),
+        dspy.Example(
+            question='"Everything Has Changed" is a song from an album released under which record label ?',
+            answer="Big Machine Records",
+        ),
+        dspy.Example(
+            question="Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?",
+            answer="Aleksandr Danilovich Aleksandrov",
+        ),
+        dspy.Example(
+            question="On the coast of what ocean is the birthplace of Diogal Sakho?",
+            answer="Atlantic",
+        ),
+        dspy.Example(
+            question="This American guitarist best known for her work with the Iron Maidens is an ancestor of a composer who was known as what?",
+            answer="The Waltz King",
+        ),
+    ]
+]
+
+
+def test_opt():
+    class BasicQA(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: str = dspy.OutputField()
+
+    qa_model = DSPDummyLM([])
+    prompt_model = DSPDummyLM(
+        [
+            # Seed prompts
+            "some thoughts",
+            '[{"instructions": "I", "question_desc": "$q", "question_prefix": "Q:", "answer_desc": "$a", "answer_prefix": "A:"}]',
+        ]
+    )
+    dspy.settings.configure(lm=qa_model)
+
+    result = optimize_signature(
+        student=TypedPredictor(BasicQA),
+        evaluator=Evaluate(devset=hotpotqa, metric=answer_exact_match, num_threads=1),
+        initial_prompts=1,
+        n_iterations=2,
+        verbose=True,
+        prompt_model=prompt_model,
+        strategy="last",
+    )
+
+    # Since we are requesting the last signature, it doesn't matter that our qa_model is
+    # bad, and gets 0 score. We should still get the last signature.
+    class ExpectedSignature(dspy.Signature):
+        "I"
+
+        question: str = dspy.InputField(desc="$q", prefix="Q:")
+        answer: str = dspy.OutputField(desc="$a", prefix="A:")
+
+    assert result.program.signature.equals(ExpectedSignature)
+
+    assert result.scores == [0, 0]
+
+
+def test_opt_composed():
+    class MyModule(dspy.Module):
+        def __init__(self):
+            self.p1 = TypedPredictor("question:str -> considerations:list[str]", max_retries=1)
+            self.p2 = TypedPredictor("considerations:list[str] -> answer:str", max_retries=1)
+
+        def forward(self, question):
+            considerations = self.p1(question=question).considerations
+            return self.p2(considerations=considerations)
+
+    class ExpectedSignature1(dspy.Signature):
+        "I1"
+
+        question: str = dspy.InputField(desc="$q", prefix="Q:")
+        considerations: list[str] = dspy.OutputField(desc="$c", prefix="C:")
+
+    info1 = make_info(ExpectedSignature1)
+
+    class ExpectedSignature2(dspy.Signature):
+        "I2"
+
+        considerations: list[str] = dspy.InputField(desc="$c", prefix="C:")
+        answer: str = dspy.OutputField(desc="$a", prefix="A:")
+
+    info2 = make_info(ExpectedSignature2)
+
+    qa_model = DSPDummyLM([])
+    prompt_model = DSPDummyLM(
+        [
+            "some thoughts",
+            json.dumps([to_jsonable_python(info1)]),
+            "some thoughts",
+            json.dumps([to_jsonable_python(info2)]),
+        ]
+    )
+    dspy.settings.configure(lm=qa_model)
+
+    result = optimize_signature(
+        student=MyModule(),
+        evaluator=lambda x: 0,  # We don't care about the evaluator here
+        initial_prompts=1,
+        n_iterations=2,
+        verbose=True,
+        prompt_model=prompt_model,
+        strategy="last",
+    )
+
+    assert result.program.p1.signature.equals(ExpectedSignature1)
+    assert result.program.p2.signature.equals(ExpectedSignature2)
diff --git a/tests/dsp_LM/modules/__init__.py b/tests/dsp_LM/modules/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/predict/__init__.py b/tests/dsp_LM/predict/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/predict/test_chain_of_thought.py b/tests/dsp_LM/predict/test_chain_of_thought.py
new file mode 100644
index 000000000..2567ae032
--- /dev/null
+++ b/tests/dsp_LM/predict/test_chain_of_thought.py
@@ -0,0 +1,36 @@
+import textwrap
+
+import dspy
+from dspy import ChainOfThought
+from dspy.utils import DSPDummyLM
+
+
+def test_initialization_with_string_signature():
+    lm = DSPDummyLM(["find the number after 1", "2"])
+    dspy.settings.configure(lm=lm)
+    predict = ChainOfThought("question -> answer")
+    assert list(predict.extended_signature.output_fields.keys()) == [
+        "rationale",
+        "answer",
+    ]
+    assert predict(question="What is 1+1?").answer == "2"
+
+    print(lm.get_convo(-1))
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `question`, produce the fields `answer`.
+
+        ---
+
+        Follow the following format.
+
+        Question: ${question}
+        Reasoning: Let's think step by step in order to ${produce the answer}. We ...
+        Answer: ${answer}
+
+        ---
+
+        Question: What is 1+1?
+        Reasoning: Let's think step by step in order to find the number after 1
+        Answer: 2"""
+    )
diff --git a/tests/dsp_LM/predict/test_chain_of_thought_with_hint.py b/tests/dsp_LM/predict/test_chain_of_thought_with_hint.py
new file mode 100644
index 000000000..0afa06b1c
--- /dev/null
+++ b/tests/dsp_LM/predict/test_chain_of_thought_with_hint.py
@@ -0,0 +1,43 @@
+import dspy
+from dspy import ChainOfThoughtWithHint
+from dspy.utils import DSPDummyLM
+
+
+def test_cot_with_no_hint():
+    lm = DSPDummyLM(["find the number after 1", "2"])
+    dspy.settings.configure(lm=lm)
+    predict = ChainOfThoughtWithHint("question -> answer")
+    # Check output fields have the right order
+    assert list(predict.extended_signature2.output_fields.keys()) == [
+        "rationale",
+        "hint",
+        "answer",
+    ]
+    assert predict(question="What is 1+1?").answer == "2"
+
+    final_convo = lm.get_convo(-1)
+    assert final_convo.endswith(
+        "Question: What is 1+1?\n"
+        "Reasoning: Let's think step by step in order to find the number after 1\n"
+        "Answer: 2"
+    )
+
+
+def test_cot_with_hint():
+    lm = DSPDummyLM(["find the number after 1", "2"])
+    dspy.settings.configure(lm=lm)
+    predict = ChainOfThoughtWithHint("question -> answer")
+    assert list(predict.extended_signature2.output_fields.keys()) == [
+        "rationale",
+        "hint",
+        "answer",
+    ]
+    assert predict(question="What is 1+1?", hint="think small").answer == "2"
+
+    final_convo = lm.get_convo(-1)
+    assert final_convo.endswith(
+        "Question: What is 1+1?\n\n"
+        "Reasoning: Let's think step by step in order to find the number after 1\n\n"
+        "Hint: think small\n\n"
+        "Answer: 2"
+    )
diff --git a/tests/dsp_LM/predict/test_multi_chain_comparison.py b/tests/dsp_LM/predict/test_multi_chain_comparison.py
new file mode 100644
index 000000000..e97c3dfbd
--- /dev/null
+++ b/tests/dsp_LM/predict/test_multi_chain_comparison.py
@@ -0,0 +1,40 @@
+import pytest
+
+import dspy
+from dspy.utils.dummies import DSPDummyLM
+
+
+def test_basic_example():
+    class BasicQA(dspy.Signature):
+        """Answer questions with short factoid answers."""
+
+        question = dspy.InputField()
+        answer = dspy.OutputField(desc="often between 1 and 5 words")
+
+    # Example completions generated by a model for reference
+    completions = [
+        dspy.Prediction(
+            rationale="I recall that during clear days, the sky often appears this color.",
+            answer="blue",
+        ),
+        dspy.Prediction(
+            rationale="Based on common knowledge, I believe the sky is typically seen as this color.",
+            answer="green",
+        ),
+        dspy.Prediction(
+            rationale="From images and depictions in media, the sky is frequently represented with this hue.",
+            answer="blue",
+        ),
+    ]
+
+    # Pass signature to MultiChainComparison module
+    compare_answers = dspy.MultiChainComparison(BasicQA)
+
+    # Call the MultiChainComparison on the completions
+    question = "What is the color of the sky?"
+    lm = DSPDummyLM(["my rationale", "blue"])
+    dspy.settings.configure(lm=lm)
+    final_pred = compare_answers(completions, question=question)
+
+    assert final_pred.rationale == "my rationale"
+    assert final_pred.answer == "blue"
diff --git a/tests/dsp_LM/predict/test_predict.py b/tests/dsp_LM/predict/test_predict.py
new file mode 100644
index 000000000..9158987e3
--- /dev/null
+++ b/tests/dsp_LM/predict/test_predict.py
@@ -0,0 +1,101 @@
+import copy
+import textwrap
+
+import pydantic
+import pytest
+import ujson
+
+import dspy
+from dspy import Predict, Signature, TypedPredictor
+from dspy.utils.dummies import DSPDummyLM
+
+
+def test_call_method():
+    predict_instance = Predict("input -> output")
+    lm = DSPDummyLM(["test output"])
+    dspy.settings.configure(lm=lm)
+    result = predict_instance(input="test input")
+    assert result.output == "test output"
+    assert lm.get_convo(-1) == (
+        "Given the fields `input`, produce the fields `output`.\n"
+        "\n---\n\n"
+        "Follow the following format.\n\n"
+        "Input: ${input}\n"
+        "Output: ${output}\n"
+        "\n---\n\n"
+        "Input: test input\n"
+        "Output: test output"
+    )
+
+
+def test_forward_method():
+    program = Predict("question -> answer")
+    dspy.settings.configure(lm=DSPDummyLM([]))
+    result = program(question="What is 1+1?").answer
+    assert result == "No more responses"
+
+
+def test_forward_method2():
+    program = Predict("question -> answer1, answer2")
+    dspy.settings.configure(lm=DSPDummyLM(["my first answer", "my second answer"]))
+    result = program(question="What is 1+1?")
+    assert result.answer1 == "my first answer"
+    assert result.answer2 == "my second answer"
+
+
+def test_config_management():
+    predict_instance = Predict("input -> output")
+    predict_instance.update_config(new_key="value")
+    config = predict_instance.get_config()
+    assert "new_key" in config and config["new_key"] == "value"
+
+
+def test_multi_output():
+    program = Predict("question -> answer", n=2)
+    dspy.settings.configure(lm=DSPDummyLM(["my first answer", "my second answer"]))
+    results = program(question="What is 1+1?")
+    assert results.completions.answer[0] == "my first answer"
+    assert results.completions.answer[1] == "my second answer"
+
+
+def test_multi_output2():
+    program = Predict("question -> answer1, answer2", n=2)
+    dspy.settings.configure(
+        lm=DSPDummyLM(
+            [
+                "my 0 answer\nAnswer 2: my 2 answer",
+                "my 1 answer\nAnswer 2: my 3 answer",
+            ],
+        )
+    )
+    results = program(question="What is 1+1?")
+    assert results.completions.answer1[0] == "my 0 answer"
+    assert results.completions.answer1[1] == "my 1 answer"
+    assert results.completions.answer2[0] == "my 2 answer"
+    assert results.completions.answer2[1] == "my 3 answer"
+
+
+def test_output_only():
+    class OutputOnlySignature(dspy.Signature):
+        output = dspy.OutputField()
+
+    predictor = Predict(OutputOnlySignature)
+
+    lm = DSPDummyLM(["short answer"])
+    dspy.settings.configure(lm=lm)
+    assert predictor().output == "short answer"
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields , produce the fields `output`.
+
+        ---
+
+        Follow the following format.
+
+        Output: ${output}
+
+        ---
+
+        Output: short answer"""
+    )
diff --git a/tests/dsp_LM/predict/test_program_of_thought.py b/tests/dsp_LM/predict/test_program_of_thought.py
new file mode 100644
index 000000000..e5522a847
--- /dev/null
+++ b/tests/dsp_LM/predict/test_program_of_thought.py
@@ -0,0 +1,135 @@
+import textwrap
+
+import dspy
+from dspy import ProgramOfThought, Signature
+from dspy.utils import DSPDummyLM
+
+
+class BasicQA(Signature):
+    question = dspy.InputField()
+    answer = dspy.OutputField(desc="often between 1 and 5 words")
+
+
+def test_pot_code_generation():
+    pot = ProgramOfThought(BasicQA)
+    lm = DSPDummyLM(
+        [
+            "Reason_A",
+            "```python\nresult = 1+1\n```",
+            "Reason_B",
+            "2",
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+    res = pot(question="What is 1+1?")
+    assert res.answer == "2"
+    assert lm.get_convo(index=-1) == textwrap.dedent(
+        """\
+        Given the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.
+
+        ---
+
+        Follow the following format.
+
+        Question: ${question}
+
+        Code: python code that answers the question
+
+        Code Output: output of previously-generated python code
+
+        Reasoning: Let's think step by step in order to ${produce the answer}. We ...
+
+        Answer: often between 1 and 5 words
+
+        ---
+
+        Question: What is 1+1?
+
+        Code: result = 1+1
+
+        Code Output: 2
+
+        Reasoning: Let's think step by step in order to Reason_B
+
+        Answer: 2"""
+    )
+
+
+def test_pot_code_generation_with_error():
+    pot = ProgramOfThought(BasicQA)
+    lm = DSPDummyLM(
+        [
+            "Reason_A",
+            "```python\nresult = 1+0/0\n```",
+            "Reason_B",  # Error: division by zero
+            "```python\nresult = 1+1\n```",
+            "Reason_C",
+            "2",
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+    res = pot(question="What is 1+1?")
+    assert res.answer == "2"
+
+    # The first code example failed
+    assert lm.get_convo(index=2) == textwrap.dedent(
+        """\
+        You are given `question`, `previous_code`, `error` due to an error in previous code.
+        Your task is to correct the error and provide the new `generated_code`.
+
+        ---
+
+        Follow the following format.
+
+        Question: ${question}
+
+        Previous Code: previously-generated python code that errored
+
+        Error: error message from previously-generated python code
+
+        Reasoning: Let's think step by step in order to ${produce the generated_code}. We ...
+
+        Code: python code that answers the question
+
+        ---
+
+        Question: What is 1+1?
+
+        Previous Code: result = 1+0/0
+
+        Error: division by zero
+
+        Reasoning: Let's think step by step in order to Reason_B"""
+    )
+
+    # The second code example succeeded
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.
+
+        ---
+
+        Follow the following format.
+
+        Question: ${question}
+
+        Code: python code that answers the question
+
+        Code Output: output of previously-generated python code
+
+        Reasoning: Let's think step by step in order to ${produce the answer}. We ...
+
+        Answer: often between 1 and 5 words
+
+        ---
+
+        Question: What is 1+1?
+
+        Code: result = 1+1
+
+        Code Output: 2
+
+        Reasoning: Let's think step by step in order to Reason_C
+
+        Answer: 2"""
+    )
diff --git a/tests/dsp_LM/predict/test_react.py b/tests/dsp_LM/predict/test_react.py
new file mode 100644
index 000000000..37979ddbc
--- /dev/null
+++ b/tests/dsp_LM/predict/test_react.py
@@ -0,0 +1,154 @@
+from dataclasses import dataclass
+
+import dspy
+from dspy.utils.dummies import DSPDummyLM, dummy_rm
+
+
+def test_example_no_tools():
+    # Create a simple dataset which the model will use with the Retrieve tool.
+    lm = DSPDummyLM(
+        [
+            "Initial thoughts",  # Thought_1
+            "Finish[blue]",  # Action_1
+        ]
+    )
+    dspy.settings.configure(lm=lm, rm=dummy_rm())
+
+    program = dspy.ReAct("question -> answer")
+
+    # Check default tools
+    assert isinstance(program.tools["Finish"], dspy.Example)
+
+    # Call the ReAct module on a particular input
+    question = "What is the color of the sky?"
+    result = program(question=question)
+    assert result.answer == "blue"
+
+    # For debugging
+    print("---")
+    for row in lm.history:
+        print(row["prompt"])
+        print("Response:", row["response"]["choices"][0]["text"])
+        print("---")
+
+    assert lm.get_convo(-1).endswith(
+        "Question: What is the color of the sky?\n" "Thought 1: Initial thoughts\n" "Action 1: Finish[blue]"
+    )
+
+
+def test_example_search():
+    # Createa a simple dataset which the model will use with the Retrieve tool.
+    lm = DSPDummyLM(
+        [
+            "Initial thoughts",  # Thought_1
+            "Search[the color of the sky]",  # Thought_1
+            "More thoughts",  # Thought_2
+            "Finish[blue]",  # Action_2
+        ]
+    )
+    rm = dummy_rm(
+        [
+            "We all know the color of the sky is blue.",
+            "Somethng about the sky colors",
+            "This sentence is completely irellevant to answer the question.",
+            "Let's add some more sentences to act as summy passages.",
+            "Let's add some more sentences to act as summy passages.",
+            "Let's add some more sentences to act as summy passages.",
+        ]
+    )
+    dspy.settings.configure(lm=lm, rm=rm)
+
+    program = dspy.ReAct("question -> answer")
+
+    # Check default tools
+    assert len(program.tools) == 2
+    assert isinstance(program.tools["Search"], dspy.Retrieve)
+    assert isinstance(program.tools["Finish"], dspy.Example)
+
+    # Call the ReAct module on a particular input
+    question = "What is the color of the sky?"
+    result = program(question=question)
+    assert result.answer == "blue"
+
+    # For debugging
+    print(lm.get_convo(-1))
+
+    assert lm.get_convo(-1).endswith(
+        "Question: What is the color of the sky?\n\n"
+        "Thought 1: Initial thoughts\n\n"
+        "Action 1: Search[the color of the sky]\n\n"
+        "Observation 1:\n"
+        "[1] «We all know the color of the sky is blue.»\n"
+        "[2] «Somethng about the sky colors»\n"
+        "[3] «This sentence is completely irellevant to answer the question.»\n\n"
+        "Thought 2: More thoughts\n\n"
+        "Action 2: Finish[blue]"
+    )
+
+
+class DummyTool1:
+    name = "Tool1"
+    input_variable = "query"
+    desc = ""
+    num_calls = 0
+
+    def __call__(self, *args, **kwargs):
+        # test case with no passages attribute
+        assert args[0] == "foo"
+        self.num_calls += 1
+        return "tool 1 output"
+
+
+@dataclass
+class DummyOutput:
+    passages: str
+
+
+class DummyTool2:
+    name = "Tool2"
+    input_variable = "query"
+    desc = ""
+    num_calls = 0
+
+    def __call__(self, *args, **kwargs):
+        # test case with passages attribute
+        assert args[0] == "bar"
+        self.num_calls += 1
+        return DummyOutput(passages="tool 2 output")
+
+
+def test_custom_tools():
+    lm = DSPDummyLM(
+        [
+            "Initial thoughts",
+            "Tool1[foo]",
+            "More thoughts",
+            "Tool2[bar]",
+            "Even more thoughts",
+            "Finish[baz]",
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    tool1 = DummyTool1()
+    tool2 = DummyTool2()
+    program = dspy.ReAct("question -> answer", tools=[tool1, tool2])
+
+    question = "What is the color of the sky?"
+    result = program(question=question)
+    assert result.answer == "baz"
+
+    # each tool should be called only once
+    assert tool1.num_calls == 1
+    assert tool2.num_calls == 1
+    assert lm.get_convo(-1).endswith(
+        "Question: What is the color of the sky?\n\n"
+        "Thought 1: Initial thoughts\n\n"
+        "Action 1: Tool1[foo]\n\n"
+        "Observation 1: tool 1 output\n\n"
+        "Thought 2: More thoughts\n\n"
+        "Action 2: Tool2[bar]\n\n"
+        "Observation 2: tool 2 output\n\n"
+        "Thought 3: Even more thoughts\n\n"
+        "Action 3: Finish[baz]"
+    )
diff --git a/tests/dsp_LM/predict/test_retry.py b/tests/dsp_LM/predict/test_retry.py
new file mode 100644
index 000000000..bd22984d4
--- /dev/null
+++ b/tests/dsp_LM/predict/test_retry.py
@@ -0,0 +1,110 @@
+import functools
+
+import pydantic
+
+import dspy
+from dspy.primitives.assertions import assert_transform_module, backtrack_handler
+from dspy.utils import DSPDummyLM
+
+
+def test_retry_simple():
+    predict = dspy.Predict("question -> answer")
+    retry_module = dspy.Retry(predict)
+
+    # Test Retry has created the correct new signature
+    for field in predict.signature.output_fields:
+        assert f"past_{field}" in retry_module.new_signature.input_fields
+    assert "feedback" in retry_module.new_signature.input_fields
+
+    lm = DSPDummyLM(["blue"])
+    dspy.settings.configure(lm=lm)
+    result = retry_module.forward(
+        question="What color is the sky?",
+        past_outputs={"answer": "red"},
+        feedback="Try harder",
+    )
+    assert result.answer == "blue"
+
+    print(lm.get_convo(-1))
+    assert lm.get_convo(-1).endswith(
+        "Question: What color is the sky?\n\n" "Previous Answer: red\n\n" "Instructions: Try harder\n\n" "Answer: blue"
+    )
+
+
+def test_retry_forward_with_feedback():
+    # First we make a mistake, then we fix it
+    lm = DSPDummyLM(["red", "blue"])
+    dspy.settings.configure(lm=lm, trace=[])
+
+    class SimpleModule(dspy.Module):
+        def __init__(self):
+            super().__init__()
+            self.predictor = dspy.Predict("question -> answer")
+
+        def forward(self, **kwargs):
+            result = self.predictor(**kwargs)
+            print(f"SimpleModule got {result.answer=}")
+            dspy.Suggest(result.answer == "blue", "Please think harder")
+            return result
+
+    program = SimpleModule()
+    program = assert_transform_module(
+        program.map_named_predictors(dspy.Retry),
+        functools.partial(backtrack_handler, max_backtracks=1),
+    )
+
+    result = program(question="What color is the sky?")
+
+    assert result.answer == "blue"
+
+    print(lm.get_convo(-1))
+    assert lm.get_convo(-1).endswith(
+        "Question: What color is the sky?\n\n"
+        "Previous Answer: red\n\n"
+        "Instructions: Please think harder\n\n"
+        "Answer: blue"
+    )
+
+
+def test_retry_forward_with_typed_predictor():
+    # First we make a mistake, then we fix it
+    lm = DSPDummyLM(['{"answer":"red"}', '{"answer":"blue"}'])
+    dspy.settings.configure(lm=lm, trace=[])
+
+    class AnswerQuestion(dspy.Signature):
+        """Answer questions with succint responses."""
+
+        class Input(pydantic.BaseModel):
+            question: str
+
+        class Output(pydantic.BaseModel):
+            answer: str
+
+        input: Input = dspy.InputField()
+        output: Output = dspy.OutputField()
+
+    class QuestionAnswerer(dspy.Module):
+        def __init__(self):
+            super().__init__()
+            self.answer_question = dspy.TypedPredictor(AnswerQuestion)
+
+        def forward(self, **kwargs):
+            result = self.answer_question(input=AnswerQuestion.Input(**kwargs)).output
+            dspy.Suggest(result.answer == "blue", "Please think harder")
+            return result
+
+    program = QuestionAnswerer()
+    program = assert_transform_module(
+        program.map_named_predictors(dspy.Retry),
+        functools.partial(backtrack_handler, max_backtracks=1),
+    )
+
+    result = program(question="What color is the sky?")
+
+    assert result.answer == "blue"
+    assert lm.get_convo(-1).endswith(
+        'Input: {"question":"What color is the sky?"}\n\n'
+        'Previous Output: {"answer":"red"}\n\n'
+        "Instructions: Please think harder\n\n"
+        'Output: {"answer":"blue"}'
+    )
diff --git a/tests/dsp_LM/primitives/__init__.py b/tests/dsp_LM/primitives/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/primitives/test_program.py b/tests/dsp_LM/primitives/test_program.py
new file mode 100644
index 000000000..dc817882a
--- /dev/null
+++ b/tests/dsp_LM/primitives/test_program.py
@@ -0,0 +1,21 @@
+import dspy
+from dspy.primitives.program import Module, set_attribute_by_name  # Adjust the import based on your file structure
+from dspy.utils import DSPDummyLM
+
+
+class HopModule(dspy.Module):
+    def __init__(self):
+        super().__init__()
+        self.predict1 = dspy.Predict("question -> query")
+        self.predict2 = dspy.Predict("query -> answer")
+
+    def forward(self, question):
+        query = self.predict1(question=question).query
+        return self.predict2(query=query)
+
+
+def test_forward():
+    program = HopModule()
+    dspy.settings.configure(lm=DSPDummyLM({"What is 1+1?": "let me check", "let me check": "2"}))
+    result = program(question="What is 1+1?").answer
+    assert result == "2"
diff --git a/tests/dsp_LM/retrieve/__init__.py b/tests/dsp_LM/retrieve/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/retrieve/test_llama_index_rm.py b/tests/dsp_LM/retrieve/test_llama_index_rm.py
new file mode 100644
index 000000000..735c1a940
--- /dev/null
+++ b/tests/dsp_LM/retrieve/test_llama_index_rm.py
@@ -0,0 +1,61 @@
+import logging
+
+import pytest
+
+import dspy
+from dsp.modules.dummy_lm import DSPDummyLM
+from dspy.datasets import HotPotQA
+
+try:
+    from llama_index.core import Settings, VectorStoreIndex
+    from llama_index.core.base.base_retriever import BaseRetriever
+    from llama_index.core.embeddings.mock_embed_model import MockEmbedding
+    from llama_index.core.readers.string_iterable import StringIterableReader
+
+    from dspy.retrieve.llama_index_rm import LlamaIndexRM
+
+except ImportError:
+    logging.info("Optional dependency llama-index is not installed - skipping LlamaIndexRM tests.")
+
+
+@pytest.fixture()
+def rag_setup() -> dict:
+    """Builds the necessary fixtures to test LI"""
+    pytest.importorskip("llamaindex")
+    dataset = HotPotQA(train_seed=1, train_size=8, eval_seed=2023, dev_size=4, test_size=0)
+    trainset = [x.with_inputs("question") for x in dataset.train]
+    devset = [x.with_inputs("question") for x in dataset.dev]
+    ragset = [f"Question: {x.question} Answer: {x.answer}" for x in dataset.train]
+    dummyset = {x.question: x.answer for x in dataset.train}
+
+    Settings.embed_model = MockEmbedding(8)
+    docs = StringIterableReader().load_data(texts=ragset)
+    index = VectorStoreIndex.from_documents(documents=docs)
+    retriever = index.as_retriever()
+    rm = LlamaIndexRM(retriever)
+
+    return {
+        "index": index,
+        "retriever": retriever,
+        "rm": rm,
+        "lm": DSPDummyLM(answers=dummyset),
+        "trainset": trainset,
+        "devset": devset,
+    }
+
+
+def test_lirm_as_rm(rag_setup):
+    """Test the retriever as retriever method"""
+    pytest.importorskip("llamaindex")
+    retriever = rag_setup.get("retriever")
+    test_res_li = retriever.retrieve("At My Window was released by which American singer-songwriter?")
+    rm = rag_setup.get("rm")
+    test_res_dspy = rm.forward("At My Window was released by which American singer-songwriter?")
+
+    assert isinstance(retriever, BaseRetriever), "Ensuring that the retriever is a LI Retriever object"
+    assert isinstance(test_res_li, list), "Ensuring results are a list from LI Retriever"
+
+    assert isinstance(rm, dspy.Retrieve), "Ensuring the RM is a retriever object from dspy"
+    assert isinstance(test_res_dspy, list), "Ensuring the results are a list from the DSPy retriever"
+
+    assert len(test_res_li) == len(test_res_dspy), "Rough equality check of the results"
diff --git a/tests/dsp_LM/signatures/__init__.py b/tests/dsp_LM/signatures/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/signatures/test_signature.py b/tests/dsp_LM/signatures/test_signature.py
new file mode 100644
index 000000000..fffa58ab2
--- /dev/null
+++ b/tests/dsp_LM/signatures/test_signature.py
@@ -0,0 +1,41 @@
+import textwrap
+from typing import List
+
+import pydantic
+import pytest
+
+import dspy
+from dspy import InputField, OutputField, Signature, infer_prefix
+from dspy.utils.dummies import DSPDummyLM
+
+
+def test_multiline_instructions():
+    class MySignature(Signature):
+        """First line
+        Second line
+            Third line"""
+
+        output = OutputField()
+
+    predictor = dspy.Predict(MySignature)
+
+    lm = DSPDummyLM(["short answer"])
+    dspy.settings.configure(lm=lm)
+    assert predictor().output == "short answer"
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        First line
+        Second line
+            Third line
+
+        ---
+
+        Follow the following format.
+
+        Output: ${output}
+
+        ---
+
+        Output: short answer"""
+    )
diff --git a/tests/dsp_LM/teleprompt/__init__.py b/tests/dsp_LM/teleprompt/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/dsp_LM/teleprompt/test_bootstrap.py b/tests/dsp_LM/teleprompt/test_bootstrap.py
new file mode 100644
index 000000000..936daf8e4
--- /dev/null
+++ b/tests/dsp_LM/teleprompt/test_bootstrap.py
@@ -0,0 +1,156 @@
+import textwrap
+
+import pytest
+
+import dspy
+from dspy import Example
+from dspy.predict import Predict
+from dspy.teleprompt import BootstrapFewShot
+from dspy.utils.dummies import DSPDummyLM
+
+
+# Define a simple metric function for testing
+def simple_metric(example, prediction, trace=None):
+    # Simplified metric for testing: true if prediction matches expected output
+    return example.output == prediction.output
+
+
+examples = [
+    Example(input="What is the color of the sky?", output="blue").with_inputs("input"),
+    Example(input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!"),
+]
+trainset = [examples[0]]
+valset = [examples[1]]
+
+
+class SimpleModule(dspy.Module):
+    def __init__(self, signature):
+        super().__init__()
+        self.predictor = Predict(signature)
+
+    def forward(self, **kwargs):
+        return self.predictor(**kwargs)
+
+
+def test_compile_with_predict_instances():
+    # Create Predict instances for student and teacher
+    # Note that dspy.Predict is not itself a module, so we can't use it directly here
+    student = SimpleModule("input -> output")
+    teacher = SimpleModule("input -> output")
+
+    lm = DSPDummyLM(["Initial thoughts", "Finish[blue]"])
+    dspy.settings.configure(lm=lm)
+
+    # Initialize BootstrapFewShot and compile the student
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
+
+    assert compiled_student is not None, "Failed to compile student"
+    assert hasattr(compiled_student, "_compiled") and compiled_student._compiled, "Student compilation flag not set"
+
+
+def test_bootstrap_effectiveness():
+    # This test verifies if the bootstrapping process improves the student's predictions
+    student = SimpleModule("input -> output")
+    teacher = SimpleModule("input -> output")
+    lm = DSPDummyLM(["blue", "Ring-ding-ding-ding-dingeringeding!"], follow_examples=True)
+    dspy.settings.configure(lm=lm, trace=[])
+
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
+
+    # Check that the compiled student has the correct demos
+    assert len(compiled_student.predictor.demos) == 1
+    assert compiled_student.predictor.demos[0].input == trainset[0].input
+    assert compiled_student.predictor.demos[0].output == trainset[0].output
+
+    # Test the compiled student's prediction.
+    # We are using a DSPDummyLM with follow_examples=True, which means that
+    # even though it would normally reply with "Ring-ding-ding-ding-dingeringeding!"
+    # on the second output, if it seems an example that perfectly matches the
+    # prompt, it will use that instead. That is why we expect "blue" here.
+    prediction = compiled_student(input=trainset[0].input)
+    assert prediction.output == trainset[0].output
+
+    # For debugging
+    print("Convo")
+    print(lm.get_convo(-1))
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Given the fields `input`, produce the fields `output`.
+
+        ---
+
+        Follow the following format.
+
+        Input: ${input}
+        Output: ${output}
+
+        ---
+
+        Input: What is the color of the sky?
+        Output: blue
+
+        ---
+
+        Input: What is the color of the sky?
+        Output: blue"""
+    )
+
+
+def test_error_handling_during_bootstrap():
+    """
+    Test to verify error handling during the bootstrapping process
+    """
+
+    class BuggyModule(dspy.Module):
+        def __init__(self, signature):
+            super().__init__()
+            self.predictor = Predict(signature)
+
+        def forward(self, **kwargs):
+            raise RuntimeError("Simulated error")
+
+    student = SimpleModule("input -> output")
+    teacher = BuggyModule("input -> output")
+
+    # Setup DSPDummyLM to simulate an error scenario
+    lm = DSPDummyLM(
+        [
+            "Initial thoughts",  # Simulate initial teacher's prediction
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    bootstrap = BootstrapFewShot(
+        metric=simple_metric,
+        max_bootstrapped_demos=1,
+        max_labeled_demos=1,
+        max_errors=1,
+    )
+
+    with pytest.raises(RuntimeError, match="Simulated error"):
+        bootstrap.compile(student, teacher=teacher, trainset=trainset)
+
+
+def test_validation_set_usage():
+    """
+    Test to ensure the validation set is correctly used during bootstrapping
+    """
+    student = SimpleModule("input -> output")
+    teacher = SimpleModule("input -> output")
+
+    lm = DSPDummyLM(
+        [
+            "Initial thoughts",
+            "Finish[blue]",  # Expected output for both training and validation
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
+
+    # Check that validation examples are part of student's demos after compilation
+    assert len(compiled_student.predictor.demos) >= len(valset), "Validation set not used in compiled student demos"
diff --git a/tests/dsp_LM/teleprompt/test_copro_optimizer.py b/tests/dsp_LM/teleprompt/test_copro_optimizer.py
new file mode 100644
index 000000000..50011eecc
--- /dev/null
+++ b/tests/dsp_LM/teleprompt/test_copro_optimizer.py
@@ -0,0 +1,149 @@
+import textwrap
+
+import pytest
+
+import dspy
+from dspy import Example
+from dspy.teleprompt.signature_opt import COPRO
+from dspy.utils.dummies import DSPDummyLM
+
+
+# Define a simple metric function for testing
+def simple_metric(example, prediction):
+    # Simplified metric for testing: true if prediction matches expected output
+    return example.output == prediction.output
+
+
+# Example training and validation sets
+trainset = [
+    Example(input="Question: What is the color of the sky?", output="blue").with_inputs("input"),
+    Example(input="Question: What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs(
+        "input"
+    ),
+]
+
+
+def test_signature_optimizer_initialization():
+    optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
+    assert optimizer.metric == simple_metric, "Metric not correctly initialized"
+    assert optimizer.breadth == 2, "Breadth not correctly initialized"
+    assert optimizer.depth == 1, "Depth not correctly initialized"
+    assert optimizer.init_temperature == 1.4, "Initial temperature not correctly initialized"
+
+
+class SimpleModule(dspy.Module):
+    def __init__(self, signature):
+        super().__init__()
+        # COPRO doesn't work with dspy.Predict
+        self.predictor = dspy.ChainOfThought(signature)
+
+    def forward(self, **kwargs):
+        return self.predictor(**kwargs)
+
+
+def test_signature_optimizer_optimization_process():
+    optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
+    dspy.settings.configure(lm=DSPDummyLM(["Optimized instruction 1", "Optimized instruction 2"]))
+
+    student = SimpleModule("input -> output")
+
+    # Assuming the compile method of COPRO requires a student module, a development set, and evaluation kwargs
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
+
+    # Check that the optimized student has been modified from the original
+    # This check can be more specific based on how the optimization modifies the student
+    assert optimized_student is not student, "Optimization did not modify the student"
+
+    # Further tests can be added to verify the specifics of the optimization process,
+    # such as checking the instructions of the optimized student's predictors.
+
+
+def test_signature_optimizer_statistics_tracking():
+    optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
+    optimizer.track_stats = True  # Enable statistics tracking
+
+    dspy.settings.configure(lm=DSPDummyLM(["Optimized instruction"]))
+    student = SimpleModule("input -> output")
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
+
+    # Verify that statistics have been tracked and attached to the optimized student
+    assert hasattr(optimized_student, "total_calls"), "Total calls statistic not tracked"
+    assert hasattr(optimized_student, "results_best"), "Best results statistics not tracked"
+
+
+# Assuming the setup_signature_optimizer fixture and simple_metric function are defined as before
+
+
+def test_optimization_and_output_verification():
+    lm = DSPDummyLM(
+        [
+            "Optimized Prompt",
+            "Optimized Prefix",
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+    optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
+
+    student = SimpleModule("input -> output")
+
+    # Compile the student with the optimizer
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
+
+    # Simulate calling the optimized student with a new input
+    test_input = "What is the capital of France?"
+    prediction = optimized_student(input=test_input)
+
+    print(lm.get_convo(-1))
+
+    assert prediction.output == "No more responses"
+
+    assert lm.get_convo(-1) == textwrap.dedent(
+        """\
+        Optimized Prompt
+
+        ---
+
+        Follow the following format.
+
+        Input: ${input}
+        Reasoning: Let's think step by step in order to ${produce the output}. We ...
+        Optimized Prefix ${output}
+
+        ---
+
+        Input: What is the capital of France?
+        Reasoning: Let's think step by step in order to No more responses
+        Optimized Prefix No more responses"""
+    )
+
+
+def test_statistics_tracking_during_optimization():
+    dspy.settings.configure(lm=DSPDummyLM(["Optimized instruction for stats tracking"]))
+
+    optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
+    optimizer.track_stats = True  # Enable statistics tracking
+
+    student = SimpleModule("input -> output")
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
+
+    # Verify that statistics have been tracked
+    assert hasattr(optimized_student, "total_calls"), "Optimizer did not track total metric calls"
+    assert optimized_student.total_calls > 0, "Optimizer reported no metric calls"
+
+    # Check if the results_best and results_latest contain valid statistics
+    assert "results_best" in optimized_student.__dict__, "Optimizer did not track the best results"
+    assert "results_latest" in optimized_student.__dict__, "Optimizer did not track the latest results"
+    assert len(optimized_student.results_best) > 0, "Optimizer did not properly populate the best results statistics"
+    assert (
+        len(optimized_student.results_latest) > 0
+    ), "Optimizer did not properly populate the latest results statistics"
+
+    # Additional detailed checks can be added here to verify the contents of the tracked statistics
diff --git a/tests/dsp_LM/teleprompt/test_knn_fewshot.py b/tests/dsp_LM/teleprompt/test_knn_fewshot.py
new file mode 100644
index 000000000..97c2dbbe3
--- /dev/null
+++ b/tests/dsp_LM/teleprompt/test_knn_fewshot.py
@@ -0,0 +1,65 @@
+import pytest
+
+import dsp
+import dspy
+from dspy.teleprompt.knn_fewshot import KNNFewShot
+from dspy.utils.dummies import DSPDummyLM, DummyVectorizer
+
+
+def mock_example(question: str, answer: str) -> dsp.Example:
+    """Creates a mock DSP example with specified question and answer."""
+    return dspy.Example(question=question, answer=answer).with_inputs("question")
+
+
+@pytest.fixture
+def setup_knn_few_shot():
+    """Sets up a KNNFewShot instance for testing."""
+    trainset = [
+        mock_example("What is the capital of France?", "Paris"),
+        mock_example("What is the largest ocean?", "Pacific"),
+        mock_example("What is 2+2?", "4"),
+    ]
+    dsp.SentenceTransformersVectorizer = DummyVectorizer
+    knn_few_shot = KNNFewShot(k=2, trainset=trainset)
+    return knn_few_shot
+
+
+class SimpleModule(dspy.Module):
+    def __init__(self, signature):
+        super().__init__()
+        self.predictor = dspy.Predict(signature)
+
+    def forward(self, *args, **kwargs):
+        return self.predictor(**kwargs)
+
+    def reset_copy(self):
+        # Creates a new instance of SimpleModule with the same predictor
+        return SimpleModule(self.predictor.signature)
+
+
+# TODO: Test not working yet
+def _test_knn_few_shot_compile(setup_knn_few_shot):
+    """Tests the compile method of KNNFewShot with SimpleModule as student."""
+    student = SimpleModule("input -> output")
+    teacher = SimpleModule("input -> output")  # Assuming teacher uses the same module type
+
+    # Setup DSPDummyLM with a response for a query similar to one of the training examples
+    lm = DSPDummyLM(["Madrid", "10"])
+    dspy.settings.configure(lm=lm)  # Responses for the capital of Spain and the result of 5+5)
+
+    knn_few_shot = setup_knn_few_shot
+    trainset = knn_few_shot.KNN.trainset
+    compiled_student = knn_few_shot.compile(student, teacher=teacher, trainset=trainset, valset=None)
+
+    assert len(compiled_student.predictor.demos) == 1
+    assert compiled_student.predictor.demos[0].input == trainset[0].input
+    assert compiled_student.predictor.demos[0].output == trainset[0].output
+    # Simulate a query that is similar to one of the training examples
+    output = compiled_student.forward(input="What is the capital of Spain?").output
+
+    print("CONVO")
+    print(lm.get_convo(-1))
+
+    # Validate that the output corresponds to one of the expected DSPDummyLM responses
+    # This assumes the compiled_student's forward method will execute the predictor with the given query
+    assert output in ["Madrid", "10"], "The compiled student did not return the correct output based on the query"
diff --git a/tests/teleprompt/test_mipro_optimizer.py b/tests/dsp_LM/teleprompt/test_mipro_optimizer.py
similarity index 88%
rename from tests/teleprompt/test_mipro_optimizer.py
rename to tests/dsp_LM/teleprompt/test_mipro_optimizer.py
index 17e94a580..86d8c00d0 100644
--- a/tests/teleprompt/test_mipro_optimizer.py
+++ b/tests/dsp_LM/teleprompt/test_mipro_optimizer.py
@@ -1,11 +1,13 @@
+import re
 import textwrap
+
 import pytest
-import re
+
 import dspy
 from dsp.modules import LM
-from dspy.teleprompt.signature_opt_bayesian import MIPRO
-from dspy.utils.dummies import DummyLM
 from dspy import Example
+from dspy.teleprompt.signature_opt_bayesian import MIPRO
+from dspy.utils.dummies import DSPDummyLM
 
 
 # Define a simple metric function for testing
@@ -13,6 +15,7 @@ def simple_metric(example, prediction, trace=None):
     # Simplified metric for testing: true if prediction matches expected output
     return example.output == prediction.output
 
+
 # Some example data
 capitals = {
     "Germany": "Berlin",
@@ -31,10 +34,11 @@ def simple_metric(example, prediction, trace=None):
 # Example training and validation sets
 trainset = [
     Example(input="What is the color of the sky?", output="blue").with_inputs("input"),
-    Example(
-        input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!"
-    ).with_inputs("input"),
-] + [Example(input=f"What is the capital of {country}?", output=capital).with_inputs("input") for country, capital in capitals.items()]
+    Example(input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs("input"),
+] + [
+    Example(input=f"What is the capital of {country}?", output=capital).with_inputs("input")
+    for country, capital in capitals.items()
+]
 
 
 class ConditionalLM(LM):
@@ -73,11 +77,11 @@ def basic_request(self, prompt, num_candidates=1, **kwargs):
             # For other questions, the model will answer with the last word of the question.
             else:
                 answer = current_question.split()[-1]
-            
+
             answer = "think deeply.\nOutput: " + answer
 
-        RED, GREEN, RESET = '\033[91m', '\033[92m', '\033[0m'
-        print("=== DummyLM ===")
+        RED, GREEN, RESET = "\033[91m", "\033[92m", "\033[0m"
+        print("=== DSPDummyLM ===")
         print(prompt, end="")
         print(f"{RED}{answer}{RESET}")
         print("===")
@@ -108,20 +112,14 @@ def __call__(self, prompt, only_completed=True, return_sorted=False, **kwargs):
 
     def get_convo(self, index):
         """get the prompt + anwer from the ith message"""
-        return self.history[index]['prompt'] \
-            + " " \
-            + self.history[index]['response']['choices'][0]['text']
+        return self.history[index]["prompt"] + " " + self.history[index]["response"]["choices"][0]["text"]
 
 
 def test_bayesian_signature_optimizer_initialization():
-    optimizer = MIPRO(
-        metric=simple_metric, num_candidates=10, init_temperature=1.4, verbose=True, track_stats=True
-    )
+    optimizer = MIPRO(metric=simple_metric, num_candidates=10, init_temperature=1.4, verbose=True, track_stats=True)
     assert optimizer.metric == simple_metric, "Metric not correctly initialized"
     assert optimizer.num_candidates == 10, "Incorrect 'num_candidates' parameter initialization"
-    assert (
-        optimizer.init_temperature == 1.4
-    ), "Initial temperature not correctly initialized"
+    assert optimizer.init_temperature == 1.4, "Initial temperature not correctly initialized"
     assert optimizer.verbose is True, "Verbose flag not correctly initialized"
     assert optimizer.track_stats is True, "Track stats flag not correctly initialized"
 
@@ -165,9 +163,7 @@ def test_signature_optimizer_optimization_process():
 
 
 def test_signature_optimizer_bad_lm():
-    dspy.settings.configure(
-        lm=DummyLM([f"Optimized instruction {i}" for i in range(30)])
-    )
+    dspy.settings.configure(lm=DSPDummyLM([f"Optimized instruction {i}" for i in range(30)]))
     student = SimpleModule(signature="input -> output")
     optimizer = MIPRO(
         metric=simple_metric,
@@ -177,7 +173,7 @@ def test_signature_optimizer_bad_lm():
         track_stats=False,
     )
 
-    # Krista: when the code tries to generate bootstrapped examples, the examples are generated using DummyLM,
+    # Krista: when the code tries to generate bootstrapped examples, the examples are generated using DSPDummyLM,
     # which only outputs "Optimized instruction i" this means that none of the bootstrapped examples are successful,
     # and therefore the set of examples that we're using to generate new prompts is empty
     with pytest.raises(ValueError):
@@ -233,9 +229,9 @@ def test_optimization_and_output_verification():
         Input:
 
         ---
-        
+
         Follow the following format.
-        
+
         Input: ${input}
         Reasoning: Let's think step by step in order to ${produce the output}. We ...
         Output: ${output}
@@ -264,4 +260,4 @@ def test_optimization_and_output_verification():
         Output: Madrid"""
     )
 
-    assert lm.get_convo(-1) == expected_lm_output
\ No newline at end of file
+    assert lm.get_convo(-1) == expected_lm_output
diff --git a/tests/dsp_LM/teleprompt/test_random_search.py b/tests/dsp_LM/teleprompt/test_random_search.py
new file mode 100644
index 000000000..9d8e63d23
--- /dev/null
+++ b/tests/dsp_LM/teleprompt/test_random_search.py
@@ -0,0 +1,39 @@
+import dspy
+from dspy import Example
+from dspy.predict import Predict
+from dspy.teleprompt import BootstrapFewShotWithRandomSearch
+from dspy.utils.dummies import DSPDummyLM
+
+
+class SimpleModule(dspy.Module):
+    def __init__(self, signature):
+        super().__init__()
+        self.predictor = Predict(signature)
+
+    def forward(self, **kwargs):
+        return self.predictor(**kwargs)
+
+
+def simple_metric(example, prediction, trace=None):
+    return example.output == prediction.output
+
+
+def test_basic_workflow():
+    """Test to ensure the basic compile flow runs without errors."""
+    student = SimpleModule("input -> output")
+    teacher = SimpleModule("input -> output")
+
+    lm = DSPDummyLM(
+        [
+            "Initial thoughts",
+            "Finish[blue]",  # Expected output for both training and validation
+        ]
+    )
+    dspy.settings.configure(lm=lm)
+
+    optimizer = BootstrapFewShotWithRandomSearch(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    trainset = [
+        Example(input="What is the color of the sky?", output="blue").with_inputs("input"),
+        Example(input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs("input"),
+    ]
+    optimizer.compile(student, teacher=teacher, trainset=trainset)
diff --git a/tests/evaluate/test_evaluate.py b/tests/evaluate/test_evaluate.py
index c8034f53a..a8552c7aa 100644
--- a/tests/evaluate/test_evaluate.py
+++ b/tests/evaluate/test_evaluate.py
@@ -34,7 +34,14 @@ def test_evaluate_initialization():
 
 
 def test_evaluate_call():
-    dspy.settings.configure(lm=DummyLM({"What is 1+1?": "2", "What is 2+2?": "4"}))
+    dspy.settings.configure(
+        lm=DummyLM(
+            {
+                "What is 1+1?": {"answer": "2"},
+                "What is 2+2?": {"answer": "4"},
+            }
+        )
+    )
     devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
     program = Predict("question -> answer")
     assert program(question="What is 1+1?").answer == "2"
@@ -48,7 +55,7 @@ def test_evaluate_call():
 
 
 def test_multithread_evaluate_call():
-    dspy.settings.configure(lm=DummyLM({"What is 1+1?": "2", "What is 2+2?": "4"}))
+    dspy.settings.configure(lm=DummyLM({"What is 1+1?": {"answer": "2"}, "What is 2+2?": {"answer": "4"}}))
     devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
     program = Predict("question -> answer")
     assert program(question="What is 1+1?").answer == "2"
@@ -65,13 +72,13 @@ def test_multithread_evaluate_call():
 def test_multi_thread_evaluate_call_cancelled(monkeypatch):
     # slow LM that sleeps for 1 second before returning the answer
     class SlowLM(DummyLM):
-        def __call__(self, prompt, **kwargs):
+        def __call__(self, *args, **kwargs):
             import time
 
             time.sleep(1)
-            return super().__call__(prompt, **kwargs)
+            return super().__call__(*args, **kwargs)
 
-    dspy.settings.configure(lm=SlowLM({"What is 1+1?": "2", "What is 2+2?": "4"}))
+    dspy.settings.configure(lm=SlowLM({"What is 1+1?": {"answer": "2"}, "What is 2+2?": {"answer": "4"}}))
 
     devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
     program = Predict("question -> answer")
@@ -101,7 +108,7 @@ def sleep_then_interrupt():
 
 
 def test_evaluate_call_bad():
-    dspy.settings.configure(lm=DummyLM({"What is 1+1?": "0", "What is 2+2?": "0"}))
+    dspy.settings.configure(lm=DummyLM({"What is 1+1?": {"answer": "0"}, "What is 2+2?": {"answer": "0"}}))
     devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
     program = Predict("question -> answer")
     ev = Evaluate(
diff --git a/tests/functional/test_functional.py b/tests/functional/test_functional.py
index 9e84d8393..61ac8c15a 100644
--- a/tests/functional/test_functional.py
+++ b/tests/functional/test_functional.py
@@ -21,7 +21,7 @@ def hard_question(topic: str) -> str:
         """Think of a hard factual question about a topic."""
 
     expected = "What is the speed of light?"
-    lm = DummyLM([expected])
+    lm = DummyLM([{"hard_question": expected}])
     dspy.settings.configure(lm=lm)
 
     question = hard_question(topic="Physics")
@@ -36,7 +36,7 @@ def hard_questions(topics: List[str]) -> List[str]:
         pass
 
     expected = ["What is the speed of light?", "What is the speed of sound?"]
-    lm = DummyLM(['["What is the speed of light?", "What is the speed of sound?"]'])
+    lm = DummyLM([{"hard_questions": '["What is the speed of light?", "What is the speed of sound?"]'}])
     dspy.settings.configure(lm=lm)
 
     question = hard_questions(topics=["Physics", "Music"])
@@ -54,7 +54,7 @@ def hard_question(topic: str) -> Question:
         """Think of a hard factual question about a topic."""
 
     expected = "What is the speed of light?"
-    lm = DummyLM([f'{{"value": "{expected}"}}'])
+    lm = DummyLM([{"hard_question": f'{{"value": "{expected}"}}'}])
     dspy.settings.configure(lm=lm)
 
     question = hard_question(topic="Physics")
@@ -75,7 +75,7 @@ def answer(question: Question) -> Answer:
         pass
 
     question = Question(value="What is the speed of light?")
-    lm = DummyLM([f'{{"value": "3e8"}}'])
+    lm = DummyLM([{"answer": '{"value": "3e8"}'}])
     dspy.settings.configure(lm=lm)
 
     result = answer(question=question)
@@ -110,12 +110,10 @@ def forward(self, **kwargs):
 
     lm = DummyLM(
         [
-            "What is the speed of light?",
-            "Some bad reasoning, 3e8 m/s.",
-            "3e8",  # Bad answer 1
-            "{...}",  # Model is asked to create an example
-            "Some good reasoning...",
-            expected.model_dump_json(),  # Good answer
+            {"hard_question": "What is the speed of light?"},
+            {"reasoning": "Some bad reasoning, 3e8 m/s.", "answer": "3e8"},  # Bad answer 1
+            {"json_object": "{...}"},  # Model is asked to create an example
+            {"reasoning": "Some good reasoning, 3e8 m/s.", "answer": f"{expected.model_dump_json()}"},  # Good answer
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -145,7 +143,7 @@ class MySignature(dspy.Signature):
     expected = "What is the speed of light?"
     lm = DummyLM(
         [
-            Question(value=expected).model_dump_json(),
+            {"output": f"{Question(value=expected).model_dump_json()}"},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -229,8 +227,6 @@ def simple_metric(example, prediction, trace=None):
     bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
     compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
 
-    lm.inspect_history(n=2)
-
     # Check that the compiled student has the correct demos
     _, predict = next(compiled_student.named_sub_modules(Predict, skip_compiled=False))
     demos = predict.demos
@@ -246,28 +242,6 @@ def simple_metric(example, prediction, trace=None):
     prediction = compiled_student(input=trainset[0].input)
     assert prediction == trainset[0].output
 
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `input`, produce the fields `output`.
-
-        ---
-
-        Follow the following format.
-
-        Input: ${input}
-        Output: ${output}
-
-        ---
-
-        Input: What is the color of the sky?
-        Output: blue
-
-        ---
-
-        Input: What is the color of the sky?
-        Output: blue"""
-    )
-
 
 def test_regex():
     class TravelInformation(BaseModel):
@@ -289,11 +263,11 @@ def flight_information(email: str) -> TravelInformation:
     lm = DummyLM(
         [
             # Example with a bad origin code.
-            '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}',
+            {"flight_information": '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}'},
             # Example to help the model understand
-            "{...}",
+            {"json_object": "{...}"},
             # Fixed
-            '{"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}',
+            {"flight_information": '{"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}'},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -349,11 +323,12 @@ def flight_information(email: str) -> TravelInformation:
         [
             # Example with a bad origin code.
             (
-                "Here is your json: "
-                "{"
-                '"origin": {"code":"JFK", "lat":40.6446, "lon":-73.7797}, '
-                '"destination": {"code":"LAX", "lat":33.942791, "lon":-118.410042}, '
-                '"date": "2022-12-25"}'
+                {
+                    "flight_information": "Here is your json: {"
+                    '"origin": {"code":"JFK", "lat":40.6446, "lon":-73.7797}, '
+                    '"destination": {"code":"LAX", "lat":33.942791, "lon":-118.410042}, '
+                    '"date": "2022-12-25"}'
+                }
             ),
         ]
     )
@@ -378,9 +353,8 @@ def flight_information(email: str) -> TravelInformation:
 
     lm = DummyLM(
         [
-            "A list of bad inputs",
-            '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}',
-            '{"origin": "JFK", "destination": "LAX", "date": "bad date"}',
+            {"flight_information": '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}'},
+            {"flight_information": '{"origin": "JFK", "destination": "LAX", "date": "bad date"}'},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -402,11 +376,11 @@ def flight_information(email: str) -> TravelInformation:
     lm = DummyLM(
         [
             # First origin is wrong, then destination, then all is good
-            '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}',
-            "{...}",  # Example to help the model understand
-            '{"origin": "JFK", "destination": "LA0", "date": "2022-12-25"}',
-            "{...}",  # Example to help the model understand
-            '{"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}',
+            {"flight_information": '{"origin": "JF0", "destination": "LAX", "date": "2022-12-25"}'},
+            {"json_object": "{...}"},  # Example to help the model understand
+            {"flight_information": '{"origin": "JFK", "destination": "LA0", "date": "2022-12-25"}'},
+            {"json_object": "{...}"},  # Example to help the model understand
+            {"flight_information": '{"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}'},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -414,32 +388,6 @@ def flight_information(email: str) -> TravelInformation:
     assert flight_information(email="Some email") == TravelInformation(
         origin="JFK", destination="LAX", date=datetime.date(2022, 12, 25)
     )
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `email`, produce the fields `flight_information`.
-
-        ---
-
-        Follow the following format.
-
-        Email: ${email}
-
-        Past Error in Flight Information: An error to avoid in the future
-
-        Past Error (2) in Flight Information: An error to avoid in the future
-
-        Flight Information: ${flight_information}. Respond with a single JSON object. JSON Schema: {"properties": {"origin": {"pattern": "^[A-Z]{3}$", "title": "Origin", "type": "string"}, "destination": {"pattern": "^[A-Z]{3}$", "title": "Destination", "type": "string"}, "date": {"format": "date", "title": "Date", "type": "string"}}, "required": ["origin", "destination", "date"], "title": "TravelInformation", "type": "object"}
-
-        ---
-
-        Email: Some email
-
-        Past Error in Flight Information: String should match pattern '^[A-Z]{3}$': origin (error type: string_pattern_mismatch)
-
-        Past Error (2) in Flight Information: String should match pattern '^[A-Z]{3}$': destination (error type: string_pattern_mismatch)
-
-        Flight Information: {"origin": "JFK", "destination": "LAX", "date": "2022-12-25"}"""
-    )
 
 
 def test_field_validator():
@@ -460,37 +408,12 @@ def get_user_details() -> UserDetails:
 
     # Keep making the mistake (lower case name) until we run
     # out of retries.
-    lm = DummyLM(
-        [
-            '{"name": "lower case name", "age": 25}',
-        ]
-        * 10
-    )
+    lm = DummyLM([{"get_user_details": '{"name": "lower case name", "age": 25}'}] * 10)
     dspy.settings.configure(lm=lm)
 
     with pytest.raises(ValueError):
         get_user_details()
 
-    print(lm.get_convo(-1))
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields , produce the fields `get_user_details`.
-
-        ---
-
-        Follow the following format.
-
-        Past Error in Get User Details: An error to avoid in the future
-        Past Error (2) in Get User Details: An error to avoid in the future
-        Get User Details: ${get_user_details}. Respond with a single JSON object. JSON Schema: {"properties": {"name": {"title": "Name", "type": "string"}, "age": {"title": "Age", "type": "integer"}}, "required": ["name", "age"], "title": "UserDetails", "type": "object"}
-
-        ---
-
-        Past Error in Get User Details: Value error, Name must be in uppercase.: name (error type: value_error)
-        Past Error (2) in Get User Details: Value error, Name must be in uppercase.: name (error type: value_error)
-        Get User Details: {"name": "lower case name", "age": 25}"""
-    )
-
 
 def test_annotated_field():
     @predictor
@@ -498,7 +421,7 @@ def test(input: Annotated[str, Field(description="description")]) -> Annotated[f
         pass
 
     # First try 0, which fails, then try 0.5, which passes
-    lm = DummyLM(["0", "0.5"])
+    lm = DummyLM([{"test": "0"}, {"test": "0.5"}])
     dspy.settings.configure(lm=lm)
 
     output = test(input="input")
@@ -507,7 +430,7 @@ def test(input: Annotated[str, Field(description="description")]) -> Annotated[f
 
 
 def test_multiple_outputs():
-    lm = DummyLM([str(i) for i in range(100)])
+    lm = DummyLM([{"output": f"{i}"} for i in range(100)])
     dspy.settings.configure(lm=lm)
 
     test = TypedPredictor("input -> output")
@@ -516,7 +439,7 @@ def test_multiple_outputs():
 
 
 def test_multiple_outputs_int():
-    lm = DummyLM([str(i) for i in range(100)])
+    lm = DummyLM([{"output": f"{i}"} for i in range(100)])
     dspy.settings.configure(lm=lm)
 
     class TestSignature(dspy.Signature):
@@ -533,9 +456,9 @@ def test_multiple_outputs_int_cot():
     # Note: Multiple outputs only work when the language model "speculatively" generates all the outputs in one go.
     lm = DummyLM(
         [
-            "thoughts 0\nOutput: 0\n",
-            "thoughts 1\nOutput: 1\n",
-            "thoughts 2\nOutput: 2\n",
+            {"reasoning": "thoughts 0", "output": "0"},
+            {"reasoning": "thoughts 1", "output": "1"},
+            {"reasoning": "thoughts 2", "output": "2"},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -547,7 +470,7 @@ def test_multiple_outputs_int_cot():
 
 
 def test_parse_type_string():
-    lm = DummyLM([str(i) for i in range(100)])
+    lm = DummyLM([{"output": f"{i}"} for i in range(100)])
     dspy.settings.configure(lm=lm)
 
     test = TypedPredictor("input:int -> output:int")
@@ -557,7 +480,7 @@ def test_parse_type_string():
 
 
 def test_literal():
-    lm = DummyLM(['"2"', '"3"'])
+    lm = DummyLM([{"f": '"2"'}, {"f": '"3"'}])
     dspy.settings.configure(lm=lm)
 
     @predictor
@@ -568,7 +491,7 @@ def f() -> Literal["2", "3"]:
 
 
 def test_literal_missmatch():
-    lm = DummyLM([f'"{i}"' for i in range(5, 100)])
+    lm = DummyLM([{"f": f"{i}"} for i in range(5, 100)])
     dspy.settings.configure(lm=lm)
 
     @predictor(max_retries=1)
@@ -582,7 +505,7 @@ def f() -> Literal["2", "3"]:
 
 
 def test_literal_int():
-    lm = DummyLM(["2", "3"])
+    lm = DummyLM([{"f": "2"}, {"f": "3"}])
     dspy.settings.configure(lm=lm)
 
     @predictor
@@ -593,7 +516,7 @@ def f() -> Literal[2, 3]:
 
 
 def test_literal_int_missmatch():
-    lm = DummyLM([f"{i}" for i in range(5, 100)])
+    lm = DummyLM([{"f": f"{i}"} for i in range(5, 100)])
     dspy.settings.configure(lm=lm)
 
     @predictor(max_retries=1)
@@ -612,8 +535,8 @@ class SimpleOutput(dspy.Signature):
 
     lm = DummyLM(
         [
-            "2.1",  # Bad output
-            "0.5",  # Good output
+            {"output": "2.1"},  # Bad output
+            {"output": "0.5"},  # Good output
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -635,12 +558,12 @@ class ExampleSignature(dspy.Signature):
 
     lm = DummyLM(
         [
-            '{"fact": "The sky is blue", "varacity": true}',
-            '{"fact": "The sky is green", "varacity": false}',
-            '{"fact": "The sky is red", "varacity": true}',
-            '{"fact": "The earth is flat", "varacity": false}',
-            '{"fact": "The earth is round", "varacity": true}',
-            '{"fact": "The earth is a cube", "varacity": false}',
+            {"fact": '{"fact": "The sky is blue", "varacity": true}'},
+            {"fact": '{"fact": "The sky is green", "varacity": false}'},
+            {"fact": '{"fact": "The sky is red", "varacity": true}'},
+            {"fact": '{"fact": "The earth is flat", "varacity": false}'},
+            {"fact": '{"fact": "The earth is round", "varacity": true}'},
+            {"fact": '{"fact": "The earth is a cube", "varacity": false}'},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -676,7 +599,7 @@ class ScoredSignature(dspy.Signature):
 
     program = TypedChainOfThought(ScoredSignature)
 
-    lm = DummyLM(["Thoughts", "Output"])
+    lm = DummyLM([{"reasoning": "Thoughts", "proposed_signature": "Output"}])
     dspy.settings.configure(lm=lm)
 
     output = program(
@@ -691,25 +614,6 @@ class ScoredSignature(dspy.Signature):
 
     assert output == "Output"
 
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `attempted_signatures`, produce the fields `proposed_signature`.
-
-        ---
-
-        Follow the following format.
-
-        Attempted Signatures: ${attempted_signatures}
-        Reasoning: Let's think step by step in order to ${produce the proposed_signature}. We ...
-        Proposed Signature: ${proposed_signature}
-
-        ---
-
-        Attempted Signatures: [{"string":"string 1","score":0.5},{"string":"string 2","score":0.4},{"string":"string 3","score":0.3}]
-        Reasoning: Let's think step by step in order to Thoughts
-        Proposed Signature: Output"""
-    )
-
 
 def test_custom_reasoning_field():
     class Question(pydantic.BaseModel):
@@ -727,7 +631,7 @@ class QuestionSignature(dspy.Signature):
     program = TypedChainOfThought(QuestionSignature, reasoning=reasoning)
 
     expected = "What is the speed of light?"
-    lm = DummyLM(["Thoughts", f'{{"value": "{expected}"}}'])
+    lm = DummyLM([{"reasoning": "Thoughts", "question": f'{{"value": "{expected}"}}'}])
     dspy.settings.configure(lm=lm)
 
     output = program(topic="Physics")
@@ -735,25 +639,6 @@ class QuestionSignature(dspy.Signature):
     assert isinstance(output.question, Question)
     assert output.question.value == expected
 
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `topic`, produce the fields `question`.
-
-        ---
-
-        Follow the following format.
-
-        Topic: ${topic}
-        Custom Reasoning: Let's break this down. To generate a question about ${topic}, we should ...
-        Question: ${question}. Respond with a single JSON object. JSON Schema: {"properties": {"value": {"title": "Value", "type": "string"}}, "required": ["value"], "title": "Question", "type": "object"}
-
-        ---
-
-        Topic: Physics
-        Custom Reasoning: Let's break this down. To generate a question about Thoughts
-        Question: {"value": "What is the speed of light?"}"""
-    )
-
 
 def test_generic_signature():
     T = TypeVar("T")
@@ -766,7 +651,7 @@ class GenericSignature(dspy.Signature, Generic[T]):
     predictor = TypedPredictor(GenericSignature[int])
     assert predictor.signature.instructions == "My signature"
 
-    lm = DummyLM(["23"])
+    lm = DummyLM([{"output": "23"}])
     dspy.settings.configure(lm=lm)
 
     assert predictor().output == 23
@@ -779,7 +664,7 @@ class ValidatedSignature(dspy.Signature):
         @pydantic.field_validator("a")
         @classmethod
         def space_in_a(cls, a: str) -> str:
-            if not " " in a:
+            if " " not in a:
                 raise ValueError("a must contain a space")
             return a
 
@@ -802,11 +687,17 @@ def check_square(n):
     def next_square(n: int) -> Annotated[int, AfterValidator(check_square)]:
         """What is the next square number after n?"""
 
-    lm = DummyLM(["3", "False", "4", "True"])
+    lm = DummyLM(
+        [
+            {"next_square": "3"},
+            {"is_square": "False"},
+            {"next_square": "4"},
+            {"is_square": "True"},
+        ]
+    )
     dspy.settings.configure(lm=lm)
 
     m = next_square(n=2)
-    lm.inspect_history(n=2)
 
     assert m == 4
 
@@ -824,7 +715,7 @@ class MySignature(dspy.Signature):
         n: int = dspy.InputField()
         next_square: Annotated[int, AfterValidator(is_square)] = dspy.OutputField()
 
-    lm = DummyLM(["3", "4"])
+    lm = DummyLM([{"next_square": "3"}, {"next_square": "4"}])
     dspy.settings.configure(lm=lm)
 
     m = TypedPredictor(MySignature)(n=2).next_square
@@ -843,7 +734,7 @@ def is_square(n: int) -> int:
     def next_square(n: int) -> Annotated[int, AfterValidator(is_square)]:
         """What is the next square number after n?"""
 
-    lm = DummyLM(["3", "4"])
+    lm = DummyLM([{"next_square": "3"}, {"next_square": "4"}])
     dspy.settings.configure(lm=lm)
 
     m = next_square(n=2)
@@ -861,70 +752,34 @@ def test_demos():
         trainset=[ex.with_inputs("input") for ex in demos],
     )
 
-    lm = DummyLM(["Paris"])
+    lm = DummyLM([{"output": "Paris"}])
     dspy.settings.configure(lm=lm)
 
     assert program(input="What is the capital of France?").output == "Paris"
 
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `input`, produce the fields `output`.
-
-        ---
-
-        Follow the following format.
-
-        Input: ${input}
-        Output: ${output}
 
-        ---
-
-        Input: What is the speed of light?
-        Output: 3e8
-
-        ---
-
-        Input: What is the capital of France?
-        Output: Paris"""
-    )
-
-
-def _test_demos_missing_input():
+def test_demos_missing_input_in_demo():
     demos = [dspy.Example(input="What is the speed of light?", output="3e8")]
     program = LabeledFewShot(k=len(demos)).compile(
         student=dspy.TypedPredictor("input -> output, thoughts"),
         trainset=[ex.with_inputs("input") for ex in demos],
     )
-    dspy.settings.configure(lm=DummyLM(["My thoughts", "Paris"]))
+    lm = DummyLM([{"thoughts": "My thoughts", "output": "Paris"}])
+    dspy.settings.configure(lm=lm)
     assert program(input="What is the capital of France?").output == "Paris"
 
-    assert dspy.settings.lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `input`, produce the fields `output`.
-
-        ---
-
-        Follow the following format.
-
-        Input: ${input}
-        Thoughts: ${thoughts}
-        Output: ${output}
-
-        ---
-
-        Input: What is the speed of light?
-        Output: 3e8
-
-        ---
-
-        Input: What is the capital of France?
-        Thoughts: My thoughts
-        Output: Paris"""
-    )
-
 
 def test_conlist():
-    dspy.settings.configure(lm=DummyLM(["[]", "[1]", "[1, 2]", "[1, 2, 3]"]))
+    dspy.settings.configure(
+        lm=DummyLM(
+            [
+                {"make_numbers": "[]"},
+                {"make_numbers": "[1]"},
+                {"make_numbers": "[1, 2]"},
+                {"make_numbers": "[1, 2, 3]"},
+            ]
+        )
+    )
 
     @predictor
     def make_numbers(input: str) -> Annotated[list[int], Field(min_items=2)]:
@@ -934,7 +789,16 @@ def make_numbers(input: str) -> Annotated[list[int], Field(min_items=2)]:
 
 
 def test_conlist2():
-    dspy.settings.configure(lm=DummyLM(["[]", "[1]", "[1, 2]", "[1, 2, 3]"]))
+    dspy.settings.configure(
+        lm=DummyLM(
+            [
+                {"output": "[]"},
+                {"output": "[1]"},
+                {"output": "[1, 2]"},
+                {"output": "[1, 2, 3]"},
+            ]
+        )
+    )
 
     make_numbers = TypedPredictor("input:str -> output:Annotated[List[int], Field(min_items=2)]")
     assert make_numbers(input="What are the first two numbers?").output == [1, 2]
@@ -952,7 +816,7 @@ def check_cateogry(self):
                 raise ValueError(f"category not in {self.allowed_categories}")
             return self
 
-    lm = DummyLM(["horse", "dog"])
+    lm = DummyLM([{"category": "horse"}, {"category": "dog"}])
     dspy.settings.configure(lm=lm)
     predictor = TypedPredictor(MySignature)
 
diff --git a/tests/functional/test_signature_opt_typed.py b/tests/functional/test_signature_opt_typed.py
index 59a610e6b..6778f6b69 100644
--- a/tests/functional/test_signature_opt_typed.py
+++ b/tests/functional/test_signature_opt_typed.py
@@ -1,17 +1,13 @@
-from typing import Generic, TypeVar
+import json
 
-import pydantic
-import dspy
-from dspy.evaluate import Evaluate
-from dspy.functional import TypedPredictor
-from dspy.teleprompt.signature_opt_typed import optimize_signature, make_info
-from dspy.utils import DummyLM
+from pydantic_core import to_jsonable_python
 
+import dspy
 from dspy.evaluate import Evaluate
 from dspy.evaluate.metrics import answer_exact_match
 from dspy.functional import TypedPredictor
-import json
-from pydantic_core import to_jsonable_python
+from dspy.teleprompt.signature_opt_typed import make_info, optimize_signature
+from dspy.utils import DummyLM
 
 hotpotqa = [
     ex.with_inputs("question")
@@ -105,12 +101,14 @@ class BasicQA(dspy.Signature):
         question: str = dspy.InputField()
         answer: str = dspy.OutputField()
 
-    qa_model = DummyLM([])
+    qa_model = DummyLM([{"answer": "foo"}] * 100)
     prompt_model = DummyLM(
         [
+            {
+                "reasoning": "some thoughts",
+                "proposed_signatures": '[{"instructions": "I", "question_desc": "$q", "question_prefix": "Q:", "answer_desc": "$a", "answer_prefix": "A:"}]',
+            }
             # Seed prompts
-            "some thoughts",
-            '[{"instructions": "I", "question_desc": "$q", "question_prefix": "Q:", "answer_desc": "$a", "answer_prefix": "A:"}]',
         ]
     )
     dspy.settings.configure(lm=qa_model)
@@ -167,10 +165,8 @@ class ExpectedSignature2(dspy.Signature):
     qa_model = DummyLM([])
     prompt_model = DummyLM(
         [
-            "some thoughts",
-            json.dumps([to_jsonable_python(info1)]),
-            "some thoughts",
-            json.dumps([to_jsonable_python(info2)]),
+            {"reasoning": "some thoughts", "proposed_signatures": json.dumps([to_jsonable_python(info1)])},
+            {"reasoning": "some thoughts", "proposed_signatures": json.dumps([to_jsonable_python(info2)])},
         ]
     )
     dspy.settings.configure(lm=qa_model)
diff --git a/tests/functional/test_signature_typed.py b/tests/functional/test_signature_typed.py
index 39eb8c2ff..0554cfb3a 100644
--- a/tests/functional/test_signature_typed.py
+++ b/tests/functional/test_signature_typed.py
@@ -1,4 +1,5 @@
 from typing import Any, Optional, Union
+from dspy.adapters.chat_adapter import _format_field_value
 import pytest
 
 import pydantic
@@ -114,6 +115,9 @@ class MySignature(dspy.Signature):
     instance = build_model_instance()
     parsed_instance = parser(instance.model_dump_json())
 
+    formatted_instance = _format_field_value(instance.model_dump_json())
+    assert formatted_instance == instance.model_dump_json(), f"{formatted_instance} != {instance.model_dump_json()}"
+
     assert parsed_instance == instance, f"{instance} != {parsed_instance}"
 
 
@@ -128,13 +132,36 @@ class MySignature(dspy.Signature):
     parsed_instance = parser(instance.model_dump_json())
     assert parsed_instance == instance, f"{instance} != {parsed_instance}"
 
+    formatted_instance = _format_field_value(instance.model_dump_json())
+    assert formatted_instance == instance.model_dump_json(), f"{formatted_instance} != {instance.model_dump_json()}"
+
     # Check null case
     parsed_instance = parser("null")
     assert parsed_instance == None, "Optional[MyModel] should be None"
 
 
+def test_nested_pydantic():
+    class NestedModel(pydantic.BaseModel):
+        model: MyModel
+
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: NestedModel = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+
+    instance = NestedModel(model=build_model_instance())
+    parsed_instance = parser(instance.model_dump_json())
+
+    formatted_instance = _format_field_value(instance.model_dump_json())
+    assert formatted_instance == instance.model_dump_json(), f"{formatted_instance} != {instance.model_dump_json()}"
+
+    assert parsed_instance == instance, f"{instance} != {parsed_instance}"
+
+
 def test_dataclass():
-    from dataclasses import dataclass
+    from dataclasses import dataclass, asdict
+    import ujson
 
     @dataclass(frozen=True)
     class MyDataclass:
@@ -152,3 +179,6 @@ class MySignature(dspy.Signature):
     instance = MyDataclass("foobar", 42, 3.14, True)
     parsed_instance = parser('{"string": "foobar", "number": 42, "floating": 3.14, "boolean": true}')
     assert parsed_instance == instance, f"{instance} != {parsed_instance}"
+
+    formatted_instance = _format_field_value(ujson.dumps(asdict(instance)))
+    assert formatted_instance == ujson.dumps(asdict(instance)), f"{formatted_instance} != {ujson.dumps(asdict(instance))}"
diff --git a/tests/predict/test_chain_of_thought.py b/tests/predict/test_chain_of_thought.py
index c1d08e729..2451bef66 100644
--- a/tests/predict/test_chain_of_thought.py
+++ b/tests/predict/test_chain_of_thought.py
@@ -1,35 +1,16 @@
 import textwrap
+
 import dspy
 from dspy import ChainOfThought
 from dspy.utils import DummyLM
 
 
 def test_initialization_with_string_signature():
-    lm = DummyLM(["find the number after 1", "2"])
+    lm = DummyLM([{"reasoning": "find the number after 1", "answer": "2"}])
     dspy.settings.configure(lm=lm)
     predict = ChainOfThought("question -> answer")
     assert list(predict.extended_signature.output_fields.keys()) == [
-        "rationale",
+        "reasoning",
         "answer",
     ]
     assert predict(question="What is 1+1?").answer == "2"
-
-    print(lm.get_convo(-1))
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `question`, produce the fields `answer`.
-
-        ---
-
-        Follow the following format.
-
-        Question: ${question}
-        Reasoning: Let's think step by step in order to ${produce the answer}. We ...
-        Answer: ${answer}
-
-        ---
-
-        Question: What is 1+1?
-        Reasoning: Let's think step by step in order to find the number after 1
-        Answer: 2"""
-    )
diff --git a/tests/predict/test_chain_of_thought_with_hint.py b/tests/predict/test_chain_of_thought_with_hint.py
index c28fb375f..77c4f9ac2 100644
--- a/tests/predict/test_chain_of_thought_with_hint.py
+++ b/tests/predict/test_chain_of_thought_with_hint.py
@@ -1,10 +1,12 @@
+import textwrap
+
 import dspy
 from dspy import ChainOfThoughtWithHint
 from dspy.utils import DummyLM
 
 
 def test_cot_with_no_hint():
-    lm = DummyLM(["find the number after 1", "2"])
+    lm = DummyLM([{"rationale": "find the number after 1", "answer": "2"}])
     dspy.settings.configure(lm=lm)
     predict = ChainOfThoughtWithHint("question -> answer")
     # Check output fields have the right order
@@ -15,16 +17,9 @@ def test_cot_with_no_hint():
     ]
     assert predict(question="What is 1+1?").answer == "2"
 
-    final_convo = lm.get_convo(-1)
-    assert final_convo.endswith(
-        "Question: What is 1+1?\n"
-        "Reasoning: Let's think step by step in order to find the number after 1\n"
-        "Answer: 2"
-    )
-
 
 def test_cot_with_hint():
-    lm = DummyLM(["find the number after 1", "2"])
+    lm = DummyLM([{"rationale": "find the number after 1", "hint": "Is it helicopter?", "answer": "2"}])
     dspy.settings.configure(lm=lm)
     predict = ChainOfThoughtWithHint("question -> answer")
     assert list(predict.extended_signature2.output_fields.keys()) == [
@@ -33,11 +28,3 @@ def test_cot_with_hint():
         "answer",
     ]
     assert predict(question="What is 1+1?", hint="think small").answer == "2"
-
-    final_convo = lm.get_convo(-1)
-    assert final_convo.endswith(
-        "Question: What is 1+1?\n\n"
-        "Reasoning: Let's think step by step in order to find the number after 1\n\n"
-        "Hint: think small\n\n"
-        "Answer: 2"
-    )
diff --git a/tests/predict/test_multi_chain_comparison.py b/tests/predict/test_multi_chain_comparison.py
index 8c936a2d8..8354f64d0 100644
--- a/tests/predict/test_multi_chain_comparison.py
+++ b/tests/predict/test_multi_chain_comparison.py
@@ -30,7 +30,7 @@ class BasicQA(dspy.Signature):
 
     # Call the MultiChainComparison on the completions
     question = "What is the color of the sky?"
-    lm = DummyLM(["my rationale", "blue"])
+    lm = DummyLM([{"rationale": "my rationale", "answer": "blue"}])
     dspy.settings.configure(lm=lm)
     final_pred = compare_answers(completions, question=question)
 
diff --git a/tests/predict/test_predict.py b/tests/predict/test_predict.py
index c701b380a..8d0121eed 100644
--- a/tests/predict/test_predict.py
+++ b/tests/predict/test_predict.py
@@ -1,11 +1,13 @@
-import dspy
-from dspy import Predict, Signature, TypedPredictor
-from dspy.utils.dummies import DummyLM
 import copy
 import textwrap
+
 import pydantic
 import ujson
 
+import dspy
+from dspy import Predict, Signature, TypedPredictor
+from dspy.utils.dummies import DummyLM
+
 
 def test_initialization_with_string_signature():
     signature_string = "input1, input2 -> output"
@@ -39,20 +41,10 @@ def test_lm_after_dump_and_load_state():
 
 def test_call_method():
     predict_instance = Predict("input -> output")
-    lm = DummyLM(["test output"])
+    lm = DummyLM([{"output": "test output"}])
     dspy.settings.configure(lm=lm)
     result = predict_instance(input="test input")
     assert result.output == "test output"
-    assert lm.get_convo(-1) == (
-        "Given the fields `input`, produce the fields `output`.\n"
-        "\n---\n\n"
-        "Follow the following format.\n\n"
-        "Input: ${input}\n"
-        "Output: ${output}\n"
-        "\n---\n\n"
-        "Input: test input\n"
-        "Output: test output"
-    )
 
 
 def test_instructions_after_dump_and_load_state():
@@ -134,17 +126,38 @@ class Output(pydantic.BaseModel):
     # Demos don't need to keep the same types after saving and loading the state.
     assert new_instance.demos[0]["input"] == original_instance.demos[0].input.model_dump_json()
 
+def test_signature_fields_after_dump_and_load_state(tmp_path):
+    class CustomSignature(dspy.Signature):
+        """I am just an instruction."""
+        sentence = dspy.InputField(desc="I am an innocent input!")
+        sentiment = dspy.OutputField()
+
+    file_path = tmp_path / "tmp.json"
+    original_instance = Predict(CustomSignature)
+    original_instance.save(file_path)
+
+    class CustomSignature2(dspy.Signature):
+        """I am not a pure instruction."""
+        sentence = dspy.InputField(desc="I am a malicious input!")
+        sentiment = dspy.OutputField(desc="I am a malicious output!", prefix="I am a prefix!")
+
+    new_instance = Predict(CustomSignature2)
+    assert new_instance.signature.dump_state() != original_instance.signature.dump_state()
+    # After loading, the fields should be the same.
+    new_instance.load(file_path)
+    assert new_instance.signature.dump_state() == original_instance.signature.dump_state()
+
 
 def test_forward_method():
     program = Predict("question -> answer")
-    dspy.settings.configure(lm=DummyLM([]))
+    dspy.settings.configure(lm=DummyLM([{"answer": "No more responses"}]))
     result = program(question="What is 1+1?").answer
     assert result == "No more responses"
 
 
 def test_forward_method2():
     program = Predict("question -> answer1, answer2")
-    dspy.settings.configure(lm=DummyLM(["my first answer", "my second answer"]))
+    dspy.settings.configure(lm=DummyLM([{"answer1": "my first answer", "answer2": "my second answer"}]))
     result = program(question="What is 1+1?")
     assert result.answer1 == "my first answer"
     assert result.answer2 == "my second answer"
@@ -159,7 +172,7 @@ def test_config_management():
 
 def test_multi_output():
     program = Predict("question -> answer", n=2)
-    dspy.settings.configure(lm=DummyLM(["my first answer", "my second answer"]))
+    dspy.settings.configure(lm=DummyLM([{"answer": "my first answer"}, {"answer": "my second answer"}]))
     results = program(question="What is 1+1?")
     assert results.completions.answer[0] == "my first answer"
     assert results.completions.answer[1] == "my second answer"
@@ -170,8 +183,8 @@ def test_multi_output2():
     dspy.settings.configure(
         lm=DummyLM(
             [
-                "my 0 answer\nAnswer 2: my 2 answer",
-                "my 1 answer\nAnswer 2: my 3 answer",
+                {"answer1": "my 0 answer", "answer2": "my 2 answer"},
+                {"answer1": "my 1 answer", "answer2": "my 3 answer"},
             ],
         )
     )
@@ -202,21 +215,6 @@ class OutputOnlySignature(dspy.Signature):
 
     predictor = Predict(OutputOnlySignature)
 
-    lm = DummyLM(["short answer"])
+    lm = DummyLM([{"output": "short answer"}])
     dspy.settings.configure(lm=lm)
     assert predictor().output == "short answer"
-
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields , produce the fields `output`.
-        
-        ---
-        
-        Follow the following format.
-        
-        Output: ${output}
-        
-        ---
-        
-        Output: short answer"""
-    )
diff --git a/tests/predict/test_program_of_thought.py b/tests/predict/test_program_of_thought.py
index 2aa153a1d..007c4e664 100644
--- a/tests/predict/test_program_of_thought.py
+++ b/tests/predict/test_program_of_thought.py
@@ -1,121 +1,38 @@
-from dspy import Signature, ProgramOfThought
+import textwrap
+
 import dspy
+from dspy import ProgramOfThought, Signature
 from dspy.utils import DummyLM
-import textwrap
+
 
 class BasicQA(Signature):
     question = dspy.InputField()
     answer = dspy.OutputField(desc="often between 1 and 5 words")
 
+
 def test_pot_code_generation():
-    pot = ProgramOfThought(BasicQA)
-    lm = DummyLM([
-        "Reason_A",
-        "```python\nresult = 1+1\n```", 
-        "Reason_B",
-        "2",
-    ])
+    lm = DummyLM(
+        [
+            {"reasoning": "Reason_A", "generated_code": "```python\nresult = 1+1\n```"},
+            {"reasoning": "Reason_B", "answer": "2"},
+        ]
+    )
     dspy.settings.configure(lm=lm)
+    pot = ProgramOfThought(BasicQA)
     res = pot(question="What is 1+1?")
     assert res.answer == "2"
-    assert lm.get_convo(index=-1) == textwrap.dedent("""\
-        Given the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.
-
-        ---
-
-        Follow the following format.
-
-        Question: ${question}
-
-        Code: python code that answers the question
-
-        Code Output: output of previously-generated python code
-
-        Reasoning: Let's think step by step in order to ${produce the answer}. We ...
 
-        Answer: often between 1 and 5 words
-
-        ---
-
-        Question: What is 1+1?
-
-        Code: result = 1+1
-
-        Code Output: 2
-
-        Reasoning: Let's think step by step in order to Reason_B
-
-        Answer: 2""")
 
 def test_pot_code_generation_with_error():
-    pot = ProgramOfThought(BasicQA)
-    lm = DummyLM([
-        "Reason_A",
-        "```python\nresult = 1+0/0\n```",
-        "Reason_B", # Error: division by zero
-        "```python\nresult = 1+1\n```",
-        "Reason_C",
-        "2",
-    ])
+    lm = DummyLM(
+        [
+            {"reasoning": "Reason_A", "generated_code": "```python\nresult = 1+0/0\n```"},
+            {"reasoning": "Reason_B", "generated_code": "```python\nresult = 1+1\n```"},
+            {"reasoning": "Reason_C", "answer": "2"},
+        ]
+    )
     dspy.settings.configure(lm=lm)
+
+    pot = ProgramOfThought(BasicQA)
     res = pot(question="What is 1+1?")
     assert res.answer == "2"
-
-    # The first code example failed
-    assert lm.get_convo(index=2) == textwrap.dedent("""\
-        You are given `question`, `previous_code`, `error` due to an error in previous code.
-        Your task is to correct the error and provide the new `generated_code`.
-
-        ---
-
-        Follow the following format.
-
-        Question: ${question}
-
-        Previous Code: previously-generated python code that errored
-
-        Error: error message from previously-generated python code
-
-        Reasoning: Let's think step by step in order to ${produce the generated_code}. We ...
-
-        Code: python code that answers the question
-
-        ---
-
-        Question: What is 1+1?
-
-        Previous Code: result = 1+0/0
-
-        Error: division by zero
-
-        Reasoning: Let's think step by step in order to Reason_B""")
-
-    # The second code example succeeded
-    assert lm.get_convo(-1) == textwrap.dedent("""\
-        Given the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.
-
-        ---
-
-        Follow the following format.
-
-        Question: ${question}
-
-        Code: python code that answers the question
-
-        Code Output: output of previously-generated python code
-
-        Reasoning: Let's think step by step in order to ${produce the answer}. We ...
-
-        Answer: often between 1 and 5 words
-
-        ---
-
-        Question: What is 1+1?
-
-        Code: result = 1+1
-
-        Code Output: 2
-
-        Reasoning: Let's think step by step in order to Reason_C
-
-        Answer: 2""")
diff --git a/tests/predict/test_react.py b/tests/predict/test_react.py
index 9b187471b..2573abcb8 100644
--- a/tests/predict/test_react.py
+++ b/tests/predict/test_react.py
@@ -8,8 +8,7 @@ def test_example_no_tools():
     # Createa a simple dataset which the model will use with the Retrieve tool.
     lm = dspy.utils.DummyLM(
         [
-            "Initial thoughts",  # Thought_1
-            "Finish[blue]",  # Action_1
+            {"Thought_1": "Initial thoughts", "Action_1": "Finish[blue]"},
         ]
     )
     dspy.settings.configure(lm=lm, rm=dummy_rm())
@@ -24,28 +23,13 @@ def test_example_no_tools():
     result = program(question=question)
     assert result.answer == "blue"
 
-    # For debugging
-    print("---")
-    for row in lm.history:
-        print(row["prompt"])
-        print("Response:", row["response"]["choices"][0]["text"])
-        print("---")
-
-    assert lm.get_convo(-1).endswith(
-        "Question: What is the color of the sky?\n"
-        "Thought 1: Initial thoughts\n"
-        "Action 1: Finish[blue]"
-    )
-
 
 def test_example_search():
     # Createa a simple dataset which the model will use with the Retrieve tool.
     lm = dspy.utils.DummyLM(
         [
-            "Initial thoughts",  # Thought_1
-            "Search[the color of the sky]",  # Thought_1
-            "More thoughts",  # Thought_2
-            "Finish[blue]",  # Action_2
+            {"Thought_1": "Initial thoughts", "Action_1": "Search[the color of the sky]"},
+            {"Thought_2": "More thoughts", "Action_2": "Finish[blue]\n\n"},
         ]
     )
     rm = dummy_rm(
@@ -72,21 +56,6 @@ def test_example_search():
     result = program(question=question)
     assert result.answer == "blue"
 
-    # For debugging
-    print(lm.get_convo(-1))
-
-    assert lm.get_convo(-1).endswith(
-        "Question: What is the color of the sky?\n\n"
-        "Thought 1: Initial thoughts\n\n"
-        "Action 1: Search[the color of the sky]\n\n"
-        "Observation 1:\n"
-        "[1] «We all know the color of the sky is blue.»\n"
-        "[2] «Somethng about the sky colors»\n"
-        "[3] «This sentence is completely irellevant to answer the question.»\n\n"
-        "Thought 2: More thoughts\n\n"
-        "Action 2: Finish[blue]"
-    )
-
 
 class DummyTool1:
     name = "Tool1"
@@ -122,12 +91,9 @@ def __call__(self, *args, **kwargs):
 def test_custom_tools():
     lm = dspy.utils.DummyLM(
         [
-            "Initial thoughts",
-            "Tool1[foo]",
-            "More thoughts",
-            "Tool2[bar]",
-            "Even more thoughts",
-            "Finish[baz]",
+            {"Thought_1": "Initial thoughts", "Action_1": "Tool1[foo]"},
+            {"Thought_2": "More thoughts", "Action_2": "Tool2[bar]"},
+            {"Thought_3": "Even more thoughts", "Action_3": "Finish[baz]"},
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -143,17 +109,6 @@ def test_custom_tools():
     # each tool should be called only once
     assert tool1.num_calls == 1
     assert tool2.num_calls == 1
-    assert lm.get_convo(-1).endswith(
-        "Question: What is the color of the sky?\n\n"
-        "Thought 1: Initial thoughts\n\n"
-        "Action 1: Tool1[foo]\n\n"
-        "Observation 1: tool 1 output\n\n"
-        "Thought 2: More thoughts\n\n"
-        "Action 2: Tool2[bar]\n\n"
-        "Observation 2: tool 2 output\n\n"
-        "Thought 3: Even more thoughts\n\n"
-        "Action 3: Finish[baz]"
-    )
 
 
 def test_signature_instructions():
diff --git a/tests/predict/test_retry.py b/tests/predict/test_retry.py
index 5ceee8a62..687a18dbf 100644
--- a/tests/predict/test_retry.py
+++ b/tests/predict/test_retry.py
@@ -1,8 +1,10 @@
 import functools
+
+import pydantic
+
 import dspy
-from dspy.utils import DummyLM
 from dspy.primitives.assertions import assert_transform_module, backtrack_handler
-import pydantic
+from dspy.utils import DummyLM
 
 
 def test_retry_simple():
@@ -14,7 +16,7 @@ def test_retry_simple():
         assert f"past_{field}" in retry_module.new_signature.input_fields
     assert "feedback" in retry_module.new_signature.input_fields
 
-    lm = DummyLM(["blue"])
+    lm = DummyLM([{"answer": "blue"}])
     dspy.settings.configure(lm=lm)
     result = retry_module.forward(
         question="What color is the sky?",
@@ -23,18 +25,10 @@ def test_retry_simple():
     )
     assert result.answer == "blue"
 
-    print(lm.get_convo(-1))
-    assert lm.get_convo(-1).endswith(
-        "Question: What color is the sky?\n\n"
-        "Previous Answer: red\n\n"
-        "Instructions: Try harder\n\n"
-        "Answer: blue"
-    )
-
 
 def test_retry_forward_with_feedback():
     # First we make a mistake, then we fix it
-    lm = DummyLM(["red", "blue"])
+    lm = DummyLM([{"answer": "red"}, {"answer": "blue"}])
     dspy.settings.configure(lm=lm, trace=[])
 
     class SimpleModule(dspy.Module):
@@ -58,18 +52,10 @@ def forward(self, **kwargs):
 
     assert result.answer == "blue"
 
-    print(lm.get_convo(-1))
-    assert lm.get_convo(-1).endswith(
-        "Question: What color is the sky?\n\n"
-        "Previous Answer: red\n\n"
-        "Instructions: Please think harder\n\n"
-        "Answer: blue"
-    )
-
 
 def test_retry_forward_with_typed_predictor():
     # First we make a mistake, then we fix it
-    lm = DummyLM(['{"answer":"red"}', '{"answer":"blue"}'])
+    lm = DummyLM([{"output": '{"answer":"red"}'}, {"output": '{"answer":"blue"}'}])
     dspy.settings.configure(lm=lm, trace=[])
 
     class AnswerQuestion(dspy.Signature):
@@ -103,9 +89,3 @@ def forward(self, **kwargs):
     result = program(question="What color is the sky?")
 
     assert result.answer == "blue"
-    assert lm.get_convo(-1).endswith(
-        'Input: {"question":"What color is the sky?"}\n\n'
-        'Previous Output: {"answer":"red"}\n\n'
-        'Instructions: Please think harder\n\n'
-        'Output: {"answer":"blue"}'
-    )
diff --git a/tests/primitives/test_program.py b/tests/primitives/test_program.py
index ecbbf06ec..ea6633682 100644
--- a/tests/primitives/test_program.py
+++ b/tests/primitives/test_program.py
@@ -1,8 +1,5 @@
 import dspy
-from dspy.primitives.program import (
-    Module,
-    set_attribute_by_name,
-)  # Adjust the import based on your file structure
+from dspy.primitives.program import Module, set_attribute_by_name  # Adjust the import based on your file structure
 from dspy.utils import DummyLM
 
 
@@ -39,7 +36,14 @@ def test_predictors():
 
 def test_forward():
     program = HopModule()
-    dspy.settings.configure(lm=DummyLM({"What is 1+1?": "let me check", "let me check": "2"}))
+    dspy.settings.configure(
+        lm=DummyLM(
+            {
+                "What is 1+1?": {"query": "let me check"},
+                "let me check": {"answer": "2"},
+            }
+        )
+    )
     result = program(question="What is 1+1?").answer
     assert result == "2"
 
@@ -166,8 +170,9 @@ def __init__(self):
         self.p0 = dspy.Predict("question -> answer")
         self.p1 = self.p0
 
+
 def test_named_parameters_duplicate_references():
-   module = DuplicateModule()
-   # Only testing for whether exceptions are thrown or not
-   # As Module.named_parameters() is recursive, this is mainly for catching infinite recursion
-   module.named_parameters()
+    module = DuplicateModule()
+    # Only testing for whether exceptions are thrown or not
+    # As Module.named_parameters() is recursive, this is mainly for catching infinite recursion
+    module.named_parameters()
diff --git a/tests/retrieve/test_llama_index_rm.py b/tests/retrieve/test_llama_index_rm.py
index 35711087e..f06f96388 100644
--- a/tests/retrieve/test_llama_index_rm.py
+++ b/tests/retrieve/test_llama_index_rm.py
@@ -3,8 +3,8 @@
 import pytest
 
 import dspy
-from dsp.modules.dummy_lm import DummyLM
 from dspy.datasets import HotPotQA
+from dspy.utils.dummies import DummyLM
 
 try:
     from llama_index.core import Settings, VectorStoreIndex
diff --git a/tests/signatures/test_signature.py b/tests/signatures/test_signature.py
index c99d1bdd0..bc63db1ac 100644
--- a/tests/signatures/test_signature.py
+++ b/tests/signatures/test_signature.py
@@ -182,6 +182,9 @@ class SubSignature(Signature):
 
 
 def test_multiline_instructions():
+    lm = DummyLM([{"output": "short answer"}])
+    dspy.settings.configure(lm=lm)
+
     class MySignature(Signature):
         """First line
         Second line
@@ -190,28 +193,8 @@ class MySignature(Signature):
         output = OutputField()
 
     predictor = dspy.Predict(MySignature)
-
-    lm = DummyLM(["short answer"])
-    dspy.settings.configure(lm=lm)
     assert predictor().output == "short answer"
 
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        First line
-        Second line
-            Third line
-
-        ---
-
-        Follow the following format.
-
-        Output: ${output}
-
-        ---
-
-        Output: short answer"""
-    )
-
 
 def test_replaced_by_replace_context_manager():
     class SignatureOne(Signature):
@@ -255,3 +238,44 @@ class SignatureFour(Signature):
         assert "input4" in SignatureTwo.input_fields
     assert "input1" in SignatureOne.input_fields
     assert "input2" in SignatureTwo.input_fields
+
+
+def test_dump_and_load_state():
+    class CustomSignature(dspy.Signature):
+        """I am just an instruction."""
+
+        sentence = dspy.InputField(desc="I am an innocent input!")
+        sentiment = dspy.OutputField()
+
+    state = CustomSignature.dump_state()
+    expected = {
+        "instructions": "I am just an instruction.",
+        "fields": [
+            {
+                "prefix": "Sentence:",
+                "description": "I am an innocent input!",
+            },
+            {
+                "prefix": "Sentiment:",
+                "description": "${sentiment}",
+            },
+        ],
+    }
+    assert state == expected
+
+    class CustomSignature2(dspy.Signature):
+        """I am a malicious instruction."""
+
+        sentence = dspy.InputField(desc="I am an malicious input!")
+        sentiment = dspy.OutputField()
+
+    assert CustomSignature2.dump_state() != expected
+    # Overwrite the state with the state of CustomSignature.
+    loaded_signature = CustomSignature2.load_state(state)
+    assert loaded_signature.instructions == "I am just an instruction."
+    # After `load_state`, the state should be the same as CustomSignature.
+    assert loaded_signature.dump_state() == expected
+    # CustomSignature2 should not have been modified.
+    assert CustomSignature2.instructions == "I am a malicious instruction."
+    assert CustomSignature2.fields["sentence"].json_schema_extra["desc"] == "I am an malicious input!"
+    assert CustomSignature2.fields["sentiment"].json_schema_extra["prefix"] == "Sentiment:"
diff --git a/tests/teleprompt/test_bootstrap.py b/tests/teleprompt/test_bootstrap.py
index ba2acd556..a112947a4 100644
--- a/tests/teleprompt/test_bootstrap.py
+++ b/tests/teleprompt/test_bootstrap.py
@@ -1,10 +1,12 @@
+import textwrap
+
 import pytest
+
 import dspy
-from dspy.predict import Predict
-from dspy.utils.dummies import DummyLM
 from dspy import Example
+from dspy.predict import Predict
 from dspy.teleprompt import BootstrapFewShot
-import textwrap
+from dspy.utils.dummies import DummyLM
 
 
 # Define a simple metric function for testing
@@ -15,9 +17,7 @@ def simple_metric(example, prediction, trace=None):
 
 examples = [
     Example(input="What is the color of the sky?", output="blue").with_inputs("input"),
-    Example(
-        input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!"
-    ),
+    Example(input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!"),
 ]
 trainset = [examples[0]]
 valset = [examples[1]]
@@ -25,9 +25,7 @@ def simple_metric(example, prediction, trace=None):
 
 def test_bootstrap_initialization():
     # Initialize BootstrapFewShot with a dummy metric and minimal setup
-    bootstrap = BootstrapFewShot(
-        metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1
-    )
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
     assert bootstrap.metric == simple_metric, "Metric not correctly initialized"
 
 
@@ -50,32 +48,22 @@ def test_compile_with_predict_instances():
     dspy.settings.configure(lm=lm)
 
     # Initialize BootstrapFewShot and compile the student
-    bootstrap = BootstrapFewShot(
-        metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1
-    )
-    compiled_student = bootstrap.compile(
-        student, teacher=teacher, trainset=trainset
-    )
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
 
     assert compiled_student is not None, "Failed to compile student"
-    assert (
-        hasattr(compiled_student, "_compiled") and compiled_student._compiled
-    ), "Student compilation flag not set"
+    assert hasattr(compiled_student, "_compiled") and compiled_student._compiled, "Student compilation flag not set"
 
 
 def test_bootstrap_effectiveness():
     # This test verifies if the bootstrapping process improves the student's predictions
     student = SimpleModule("input -> output")
     teacher = SimpleModule("input -> output")
-    lm = DummyLM(["blue", "Ring-ding-ding-ding-dingeringeding!"], follow_examples=True)
+    lm = DummyLM([{"output": "blue"}, {"output": "Ring-ding-ding-ding-dingeringeding!"}], follow_examples=True)
     dspy.settings.configure(lm=lm, trace=[])
 
-    bootstrap = BootstrapFewShot(
-        metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1
-    )
-    compiled_student = bootstrap.compile(
-        student, teacher=teacher, trainset=trainset
-    )
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
 
     # Check that the compiled student has the correct demos
     assert len(compiled_student.predictor.demos) == 1
@@ -90,32 +78,6 @@ def test_bootstrap_effectiveness():
     prediction = compiled_student(input=trainset[0].input)
     assert prediction.output == trainset[0].output
 
-    # For debugging
-    print("Convo")
-    print(lm.get_convo(-1))
-
-    assert lm.get_convo(-1) == textwrap.dedent(
-        """\
-        Given the fields `input`, produce the fields `output`.
-
-        ---
-
-        Follow the following format.
-
-        Input: ${input}
-        Output: ${output}
-
-        ---
-
-        Input: What is the color of the sky?
-        Output: blue
-
-        ---
-
-        Input: What is the color of the sky?
-        Output: blue"""
-    )
-
 
 def test_error_handling_during_bootstrap():
     """
@@ -136,7 +98,7 @@ def forward(self, **kwargs):
     # Setup DummyLM to simulate an error scenario
     lm = DummyLM(
         [
-            "Initial thoughts",  # Simulate initial teacher's prediction
+            {"output": "Initial thoughts"},  # Simulate initial teacher's prediction
         ]
     )
     dspy.settings.configure(lm=lm)
@@ -161,20 +123,14 @@ def test_validation_set_usage():
 
     lm = DummyLM(
         [
-            "Initial thoughts",
-            "Finish[blue]",  # Expected output for both training and validation
+            {"output": "Initial thoughts"},
+            {"output": "Finish[blue]"},  # Expected output for both training and validation
         ]
     )
     dspy.settings.configure(lm=lm)
 
-    bootstrap = BootstrapFewShot(
-        metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1
-    )
-    compiled_student = bootstrap.compile(
-        student, teacher=teacher, trainset=trainset
-    )
+    bootstrap = BootstrapFewShot(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
+    compiled_student = bootstrap.compile(student, teacher=teacher, trainset=trainset)
 
     # Check that validation examples are part of student's demos after compilation
-    assert len(compiled_student.predictor.demos) >= len(
-        valset
-    ), "Validation set not used in compiled student demos"
+    assert len(compiled_student.predictor.demos) >= len(valset), "Validation set not used in compiled student demos"
diff --git a/tests/teleprompt/test_copro_optimizer.py b/tests/teleprompt/test_copro_optimizer.py
index c0fe712bc..c031d3c7a 100644
--- a/tests/teleprompt/test_copro_optimizer.py
+++ b/tests/teleprompt/test_copro_optimizer.py
@@ -1,20 +1,24 @@
-import textwrap
 import dspy
+from dspy import Example
 from dspy.teleprompt.signature_opt import COPRO
 from dspy.utils.dummies import DummyLM
-from dspy import Example
+
 
 # Define a simple metric function for testing
 def simple_metric(example, prediction):
     # Simplified metric for testing: true if prediction matches expected output
     return example.output == prediction.output
 
+
 # Example training and validation sets
 trainset = [
     Example(input="Question: What is the color of the sky?", output="blue").with_inputs("input"),
-    Example(input="Question: What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs("input"),
+    Example(input="Question: What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs(
+        "input"
+    ),
 ]
 
+
 def test_signature_optimizer_initialization():
     optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
     assert optimizer.metric == simple_metric, "Metric not correctly initialized"
@@ -22,6 +26,7 @@ def test_signature_optimizer_initialization():
     assert optimizer.depth == 1, "Depth not correctly initialized"
     assert optimizer.init_temperature == 1.4, "Initial temperature not correctly initialized"
 
+
 class SimpleModule(dspy.Module):
     def __init__(self, signature):
         super().__init__()
@@ -31,15 +36,27 @@ def __init__(self, signature):
     def forward(self, **kwargs):
         return self.predictor(**kwargs)
 
+
 def test_signature_optimizer_optimization_process():
     optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
-    dspy.settings.configure(lm=DummyLM(["Optimized instruction 1", "Optimized instruction 2"]))
-    
+    dspy.settings.configure(
+        lm=DummyLM(
+            [
+                {
+                    "proposed_instruction": "Optimized instruction 1",
+                    "proposed_prefix_for_output_field": "Optimized instruction 2",
+                },
+            ]
+        )
+    )
+
     student = SimpleModule("input -> output")
-    
+
     # Assuming the compile method of COPRO requires a student module, a development set, and evaluation kwargs
-    optimized_student = optimizer.compile(student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False})
-    
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
+
     # Check that the optimized student has been modified from the original
     # This check can be more specific based on how the optimization modifies the student
     assert optimized_student is not student, "Optimization did not modify the student"
@@ -47,75 +64,93 @@ def test_signature_optimizer_optimization_process():
     # Further tests can be added to verify the specifics of the optimization process,
     # such as checking the instructions of the optimized student's predictors.
 
+
 def test_signature_optimizer_statistics_tracking():
     optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
     optimizer.track_stats = True  # Enable statistics tracking
 
-    dspy.settings.configure(lm=DummyLM(["Optimized instruction"]))
+    dspy.settings.configure(
+        lm=DummyLM(
+            [
+                {
+                    "proposed_instruction": "Optimized instruction 1",
+                    "proposed_prefix_for_output_field": "Optimized instruction 2",
+                },
+            ]
+        )
+    )
     student = SimpleModule("input -> output")
-    optimized_student = optimizer.compile(student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False})
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
 
     # Verify that statistics have been tracked and attached to the optimized student
-    assert hasattr(optimized_student, 'total_calls'), "Total calls statistic not tracked"
-    assert hasattr(optimized_student, 'results_best'), "Best results statistics not tracked"
+    assert hasattr(optimized_student, "total_calls"), "Total calls statistic not tracked"
+    assert hasattr(optimized_student, "results_best"), "Best results statistics not tracked"
+
 
 # Assuming the setup_signature_optimizer fixture and simple_metric function are defined as before
 
+
 def test_optimization_and_output_verification():
-    lm = DummyLM([
-        "Optimized Prompt",
-        "Optimized Prefix",
-    ])
+    lm = DummyLM(
+        [
+            {"proposed_instruction": "Optimized Prompt", "proposed_prefix_for_output_field": "Optimized Prefix"},
+            {"reasoning": "france", "output": "Paris"},
+            {"reasoning": "france", "output": "Paris"},
+            {"reasoning": "france", "output": "Paris"},
+            {"reasoning": "france", "output": "Paris"},
+            {"reasoning": "france", "output": "Paris"},
+            {"reasoning": "france", "output": "Paris"},
+            {"reasoning": "france", "output": "Paris"},
+        ]
+    )
     dspy.settings.configure(lm=lm)
     optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
-    
+
     student = SimpleModule("input -> output")
-    
+
     # Compile the student with the optimizer
-    optimized_student = optimizer.compile(student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False})
-    
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
+
     # Simulate calling the optimized student with a new input
     test_input = "What is the capital of France?"
     prediction = optimized_student(input=test_input)
 
     print(lm.get_convo(-1))
-    
-    assert prediction.output == "No more responses"
-
-    assert lm.get_convo(-1) == textwrap.dedent("""\
-        Optimized Prompt
-
-        ---
 
-        Follow the following format.
+    assert prediction.output == "Paris"
 
-        Input: ${input}
-        Reasoning: Let's think step by step in order to ${produce the output}. We ...
-        Optimized Prefix ${output}
-
-        ---
-
-        Input: What is the capital of France?
-        Reasoning: Let's think step by step in order to No more responses
-        Optimized Prefix No more responses""")
 
 def test_statistics_tracking_during_optimization():
-    dspy.settings.configure(lm=DummyLM(["Optimized instruction for stats tracking"]))
-    
+    dspy.settings.configure(
+        lm=DummyLM(
+            [
+                {"proposed_instruction": "Optimized Prompt", "proposed_prefix_for_output_field": "Optimized Prefix"},
+            ]
+        )
+    )
+
     optimizer = COPRO(metric=simple_metric, breadth=2, depth=1, init_temperature=1.4)
     optimizer.track_stats = True  # Enable statistics tracking
-    
+
     student = SimpleModule("input -> output")
-    optimized_student = optimizer.compile(student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False})
+    optimized_student = optimizer.compile(
+        student, trainset=trainset, eval_kwargs={"num_threads": 1, "display_progress": False}
+    )
 
     # Verify that statistics have been tracked
-    assert hasattr(optimized_student, 'total_calls'), "Optimizer did not track total metric calls"
+    assert hasattr(optimized_student, "total_calls"), "Optimizer did not track total metric calls"
     assert optimized_student.total_calls > 0, "Optimizer reported no metric calls"
-    
+
     # Check if the results_best and results_latest contain valid statistics
-    assert 'results_best' in optimized_student.__dict__, "Optimizer did not track the best results"
-    assert 'results_latest' in optimized_student.__dict__, "Optimizer did not track the latest results"
+    assert "results_best" in optimized_student.__dict__, "Optimizer did not track the best results"
+    assert "results_latest" in optimized_student.__dict__, "Optimizer did not track the latest results"
     assert len(optimized_student.results_best) > 0, "Optimizer did not properly populate the best results statistics"
-    assert len(optimized_student.results_latest) > 0, "Optimizer did not properly populate the latest results statistics"
+    assert (
+        len(optimized_student.results_latest) > 0
+    ), "Optimizer did not properly populate the latest results statistics"
 
     # Additional detailed checks can be added here to verify the contents of the tracked statistics
diff --git a/tests/teleprompt/test_random_search.py b/tests/teleprompt/test_random_search.py
index e76178c80..7da4069aa 100644
--- a/tests/teleprompt/test_random_search.py
+++ b/tests/teleprompt/test_random_search.py
@@ -1,8 +1,9 @@
 import dspy
-from dspy.predict import Predict
-from dspy.utils.dummies import DummyLM
 from dspy import Example
+from dspy.predict import Predict
 from dspy.teleprompt import BootstrapFewShotWithRandomSearch
+from dspy.utils.dummies import DummyLM
+
 
 class SimpleModule(dspy.Module):
     def __init__(self, signature):
@@ -12,9 +13,11 @@ def __init__(self, signature):
     def forward(self, **kwargs):
         return self.predictor(**kwargs)
 
+
 def simple_metric(example, prediction, trace=None):
     return example.output == prediction.output
 
+
 def test_basic_workflow():
     """Test to ensure the basic compile flow runs without errors."""
     student = SimpleModule("input -> output")
@@ -34,4 +37,3 @@ def test_basic_workflow():
         Example(input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs("input"),
     ]
     optimizer.compile(student, teacher=teacher, trainset=trainset)
-