Repository: stanfordnlp/dspy
Pull request #1413 (2024-08-21 09:40:44)

fix(dspy): Allow `Union` with pydantic objects in `TypedPredictor` 
See  #1349

This uses pydantics [`TypeAdapter`](https://docs.pydantic.dev/latest/api/type_adapter/) to wrap objects into pydantic serializable instances.

This also eliminates the need to prepend single values with a value field in the JSON schema,, e.g. a field of type `list[int]` will now be expressed as `[1,2,3]` instead of `{"value": [1,2,3]}`. 

I also added tests for the most common field types to ensure that they are deserialized properly.


Patch:
diff --git a/README.md b/README.md
index c2d7bef8..f31ca47d 100644
--- a/README.md
+++ b/README.md
@@ -19,8 +19,6 @@
 
 **Documentation:** [DSPy Docs](https://dspy-docs.vercel.app/)
 
-**How is DSPy Used?:** [DSPy Use Cases](https://dspy-docs.vercel.app/docs/dspy-usecases)
-
 [![Downloads](https://static.pepy.tech/badge/dspy-ai)](https://pepy.tech/project/dspy-ai)  [![Downloads](https://static.pepy.tech/badge/dspy-ai/month)](https://pepy.tech/project/dspy-ai)
 
 ----
@@ -31,6 +29,7 @@ To make this more systematic and much more powerful, **DSPy** does two things. F
 
 **DSPy** can routinely teach powerful models like `GPT-3.5` or `GPT-4` and local models like `T5-base` or `Llama2-13b` to be much more reliable at tasks, i.e. having higher quality and/or avoiding specific failure patterns. **DSPy** optimizers will "compile" the _same_ program into _different_ instructions, few-shot prompts, and/or weight updates (finetunes) for each LM. This is a new paradigm in which LMs and their prompts fade into the background as optimizable pieces of a larger system that can learn from data. **tldr;** less prompting, higher scores, and a more systematic approach to solving hard tasks with LMs.
 
+
 ### Table of Contents
 
 If you need help thinking about your task, we recently created a [Discord server](https://discord.gg/VzS6RHHK6F) for the community.
@@ -42,6 +41,8 @@ If you need help thinking about your task, we recently created a [Discord server
 1. **[Pydantic Types](#5-pydantic-types)** 
 1. **[FAQ: Is DSPy right for me?](#6-faq-is-dspy-right-for-me)**
 
+
+
 ### Analogy to Neural Networks
 
 When we build neural networks, we don't write manual _for-loops_ over lists of _hand-tuned_ floats. Instead, you might use a framework like [PyTorch](https://pytorch.org/) to compose declarative layers (e.g., `Convolution` or `Dropout`) and then use optimizers (e.g., SGD or Adam) to learn the parameters of the network.
@@ -369,7 +370,7 @@ If you're a NLP/AI researcher (or a practitioner exploring new pipelines or new
 
 ####
 <details>
-  <summary><h4 style="display: inline">[6.a] DSPy vs. thin wrappers for prompts (OpenAI API, MiniChain, basic templating)</h4></summary>
+  <summary><h4 style="display: inline">[5.a] DSPy vs. thin wrappers for prompts (OpenAI API, MiniChain, basic templating)</h4></summary>
 
 In other words: _Why can't I just write my prompts directly as string templates?_ Well, for extremely simple settings, this _might_ work just fine. (If you're familiar with neural networks, this is like expressing a tiny two-layer NN as a Python for-loop. It kinda works.)
 
@@ -383,7 +384,7 @@ Oh, and you wouldn't need to maintain long, brittle, model-specific strings at t
 
 ####
 <details>
-  <summary><h4 style="display: inline">[6.b] DSPy vs. application development libraries like LangChain, LlamaIndex</h4></summary>
+  <summary><h4 style="display: inline">[5.b] DSPy vs. application development libraries like LangChain, LlamaIndex</h4></summary>
 
 
 > _Note: If you use LangChain as a thin wrapper around your own prompt strings, refer to answer [5.a] instead._
@@ -402,7 +403,7 @@ If you're familiar with neural networks:
 
 ####
 <details>
-  <summary><h4 style="display: inline">[6.c] DSPy vs. generation control libraries like Guidance, LMQL, RELM, Outlines</h4></summary>
+  <summary><h4 style="display: inline">[5.c] DSPy vs. generation control libraries like Guidance, LMQL, RELM, Outlines</h4></summary>
 
 
 Guidance, LMQL, RELM, and Outlines are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression.
diff --git a/docs/docs/dspy-usecases.md b/docs/docs/dspy-usecases.md
deleted file mode 100644
index 7f43f3e2..00000000
--- a/docs/docs/dspy-usecases.md
+++ /dev/null
@@ -1,65 +0,0 @@
-# DSPy Use Cases
-
-This list is ever expanding and highly incomplete (WIP)! We'll be adding a bunch more. If you would like to add your company to the list, please make a PR.
-
-## Company Use Cases
-
-| **Name**           | **Use Cases**               | **Link**                                                                                                                                                                                                                 |
-|--------------------|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Normal Computing** | Language specs from chip companies to intermediate languages | [https://www.normalcomputing.com/](https://www.normalcomputing.com/)                                                                                                                                   |
-| **Replit**         | Synthesize diffs using large pre-trained code LLMs with a few-shot prompt pipeline implemented with DSPy. | [https://blog.replit.com/code-repair](https://blog.replit.com/code-repair)                                                                                                                                  |
-| **Haize Labs**     | Automated red-teaming for LLMs. | [https://blog.haizelabs.com/posts/dspy/](https://blog.haizelabs.com/posts/dspy/)                                                                                                                         |
-| **Jetblue + Databricks** | Multiple Chatbot Use Cases  | [https://www.databricks.com/blog/optimizing-databricks-llm-pipelines-dspy](https://www.databricks.com/blog/optimizing-databricks-llm-pipelines-dspy)                                                    |
-| **PingCAP**        | Building a knowledge graph  | [https://www.pingcap.com/article/building-a-graphrag-from-wikipedia-page-using-dspy-openai-and-tidb-vector-database/](https://www.pingcap.com/article/building-a-graphrag-from-wikipedia-page-using-dspy-openai-and-tidb-vector-database/) |
-| **Plastic Labs**   | Different pipelines within Honcho | [https://blog.plasticlabs.ai/blog/User-State-is-State-of-the-Art](https://blog.plasticlabs.ai/blog/User-State-is-State-of-the-Art)                                                                       |
-
-TODO: Add
-
-* RadiantLogic AI Data Assistant
-* Starops & Saya
-* Procure.FYI
-* Cybersecurity (Stealth)
-* etc
-
-## Providers and Integrations
-
-| **Name**           | **Link**                                                                                                                                                  |
-|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Databricks**     | [https://www.databricks.com/blog/dspy-databricks](https://www.databricks.com/blog/dspy-databricks)                                                                                                |
-| **Zenbase**        | [https://zenbase.ai/](https://zenbase.ai/)                                                                                                                                                        |
-| **Langwatch**      | [https://langwatch.ai/blog/introducing-dspy-visualizer](https://langwatch.ai/blog/introducing-dspy-visualizer)                                                                                    |
-| **Gradient**       | [https://gradient.ai/blog/achieving-gpt-4-level-performance-at-lower-cost-using-dspy](https://gradient.ai/blog/achieving-gpt-4-level-performance-at-lower-cost-using-dspy)                         |
-| **Snowflake**      | [https://medium.com/snowflake/dspy-snowflake-140d6d947d73](https://medium.com/snowflake/dspy-snowflake-140d6d947d73)                                                                              |
-| **Langchain**      | [https://python.langchain.com/v0.2/docs/integrations/providers/dspy/](https://python.langchain.com/v0.2/docs/integrations/providers/dspy/)                                                      |
-| **Weaviate**       | [https://weaviate.io/blog/dspy-optimizers](https://weaviate.io/blog/dspy-optimizers)                                                                                                            |
-
-## Papers, Repos, and Blogs Using DSPy
-
-| **Name**                | **Usecase**             | **Link**                                                                                                                                                           |
-|-------------------------|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **SToRM**               | Wikipage generation     | [https://arxiv.org/abs/2402.14207v1](https://arxiv.org/abs/2402.14207v1)                                                                                           |
-| **IReRa**               | Extreme reranking       | [https://arxiv.org/abs/2401.12178](https://arxiv.org/abs/2401.12178)                                                                                               |
-| **Optimizing LM for Text2SQL using DSPy** | Text2SQL                | [https://github.com/jjovalle99/DSPy-Text2SQL](https://github.com/jjovalle99/DSPy-Text2SQL)                                                                          |
-| **DSPy PII Masking Demo by Eric Ness** | PII Masking             | [https://colab.research.google.com/drive/1KZR1sGTp_RLWUJPAiK1FKPKI-Qn9neUm?usp=sharing](https://colab.research.google.com/drive/1KZR1sGTp_RLWUJPAiK1FKPKI-Qn9neUm?usp=sharing) |
-| **The Unreasonable Effectiveness of Eccentric Automatic Prompts** | General Prompt Optimization | [https://arxiv.org/abs/2402.10949v2](https://arxiv.org/abs/2402.10949v2)                                                                                           |
-
-TODO: Add
-
-* Palimpzest
-* Agent efficiency optimization from Princeton
-* PATH
-* EDEN
-* etc
-
-## Blogs/Repos about DSPy
-
-| **Name**                     | **Link**                                                                                                                                                    |
-|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Why I bet on DSPy**         | [https://blog.isaacbmiller.com/posts/dspy](https://blog.isaacbmiller.com/posts/dspy)                                                                                                             |
-
-TODO: Add
-
-* Stephen Bryan's
-* Content from Weaviate
-* Stanford CS 224U material
-* etc
diff --git a/docs/docs/roadmap.md b/docs/docs/roadmap.md
deleted file mode 100644
index 681d92ed..00000000
--- a/docs/docs/roadmap.md
+++ /dev/null
@@ -1,119 +0,0 @@
-# Roadmap Sketch: DSPy 2.5+
-
-It’s been a year since DSPy evolved out of Demonstrate–Search–Predict (DSP), whose research started at Stanford NLP all the way back in February 2022. Thanks to 200 wonderful contributors, DSPy has introduced tens of thousands of people to building modular LM programs and optimizing their prompts and weights automatically. In this time, DSPy has grown to 160,000 monthly downloads and 16,000 stars on GitHub, becoming synonymous with prompt optimization in many circles and inspiring at least a half-dozen cool new libraries.
-
-This document is an initial sketch of DSPy’s public roadmap for the next few weeks and months, as we work on DSPy 2.5 and plan for DSPy 3.0. Suggestions and open-source contributors are more than welcome: just open an issue or submit a pull request regarding the roadmap.
-
-
-
-## Technical Objectives
-
-The thesis of DSPy is that for LMs to be useful, we have to shift from ad-hoc prompting to new notions of programming LMs. Instead of relying on LMs gaining much more general or more compositional capabilities, we need to enable developers to iteratively explore their problems and build modular software that invokes LMs for well-scoped tasks. We need to enable that through modules and optimizers that isolate how they decompose their problems and describe their system's objectives from how their LMs are invoked or fine-tuned to maximize their objectives. DSPy's goal has been to develop (and to build the community and shared infrastructure for the collective development of) the abstractions, programming patterns, and optimizers toward this thesis.
-
-To a first approximation, DSPy’s current user-facing language has the minimum number of appropriate abstractions that address the goals above: declarative signatures, define-by-run modules, and optimizers that can be composed quite powerfully. But there are several things we need to do better to realize our goals. The upcoming DSPy releases will have the following objectives.
-
-1. Polishing the core functionality.
-2. Developing more accurate, lower-cost optimizers.
-3. Building end-to-end tutorials from DSPy’s ML workflow to deployment.
-4. Shifting towards more interactive optimization & tracking.
-
-
-
-## Team & Organization
-
-DSPy is fairly unusual in its technical objectives, contributors, and audience. Though DSPy takes inspiration from PyTorch, a library for building and optimizing DNNs, there is one major difference: PyTorch was introduced well after DNNs were mature ML concepts, but DSPy seeks to establish and advance core LM Programs research: the framework is propelled by constant academic research from programming abstractions (like the original [Demonstrate–Search–Predict]() concepts, DSPy [Signatures](), or [LM Assertions]()) to NLP systems (like [STORM]() and [IReRa]()) to prompt optimizers (like [MIPRO]()) and RL (like [BetterTogether]()), among many other related directions.
-
-This research all composes into a concrete, practical library, thanks to dozens of industry contributors, many of whom are deploying apps in production using DSPy. Because of this, DSPy reaches not only of grad students and ML engineers, but also many non-ML engineers, from early adopter SWEs to hobbyists exploring new ways of using LMs. The following team, with help from many folks in the OSS community, is working towards the objectives in this Roadmap.
-
-**Project Lead:** Omar Khattab (Stanford & Databricks)
-
-**Project Mentors:** Chris Potts (Stanford), Matei Zaharia (UC Berkeley & Databricks), Heather Miller (CMU & Two Sigma)
-
-**Core Library:** Arnav Singhvi (Databricks & Stanford), Herumb Shandilya (Stanford), Hanna Moazam (Databricks), Sri Vardhamanan (Dashworks), Cyrus Nouroozi (Zenbase), Amir Mehr (Zenbase), Kyle Caverly (Modular), with special thanks to Keshav Santhanam (Stanford), Thomas Ahle (Normal Computing), Connor Shorten (Weaviate)
-
-**Prompt Optimization:** Krista Opsahl-Ong (Stanford), Michael Ryan (Stanford), Josh Purtell (Basis), with special thanks to Eric Zhang (Stanford)
-
-**Finetuning & RL:** Dilara Soylu (Stanford), Isaac Miller (Anyscale), Karel D'Oosterlinck (Ghent), with special thanks to Paridhi Masehswari (Stanford)
-
-**PL Abstractions:** Shangyin Tan (UC Berkeley), Manish Shetty (UC Berkeley), Peter Zhong (CMU)
-
-**Applications:** Jasper Xian (Waterloo), Saron Samuel (Stanford), Alberto Mancarella (Stanford), Faraz Khoubsirat (Waterloo), Saiful Haq (IIT-B), Ashutosh Sharma (UIUC)
-
-
-
-## 1) Polishing the core functionality.
-
-Over the next month, polishing is the main objective and likely the one to have the highest ROI on the experience of the average user. Conceptually, DSPy has an extremely small core. It’s nothing but (1) LMs, (2) Signatures & Modules, (3) Optimizers, and (4) Assertions. These concepts and their implementations evolved organically over the past couple of years. We are working now to consolidate what we’ve learned and refactor internally so that things “just work” out of the box for new users, who may not know all the tips-and-tricks just yet.
-
-More concretely:
-
-1. We want to increase the quality of zero-shot, off-the-shelf DSPy programs, i.e. those not yet compiled on custom data.
-2. Wherever possible, DSPy should delegate lower-level internal complexity (like managing LMs and structured generation) to emerging lower-level libraries. When required, we may fork smaller libraries out of DSPy to support infrastructure pieces as their own projects.
-3. DSPy should internally be more modular and we need higher compatibility between internal components. Specifically, we need more deeper and more native investment in (i) typed multi-field constraints, (ii) assertions, (iii) observability and experimental tracking, (iv) deployment of artifacts and related concerns like streaming and async, and (v) fine-tuning and serving open models.
-
-
-### On LMs
-
-As of DSPy 2.4, the library has approximately 20,000 lines of code and roughly another 10,000 lines of code for tests, examples, and documentation. Some of these are clearly necessary (e.g., DSPy optimizers) but others exist only because the LM space lacks the building blocks we need under the hood. Luckily, for LM interfaces, a very strong library now exists: LiteLLM, a library that unifies interfaces to various LM and embedding providers. We expect to reduce around 6000 LoCs of support for custom LMs and retrieval models by shifting a lot of that to LiteLLM.
-
-Objectives in this space include improved caching, saving/loading of LMs, support for streaming and async LM requests. Work here is currently led by Hanna Moazam and Sri Vardhamanan, building on a foundation by Cyrus Nouroozi, Amir Mehr, Kyle Caverly, and others.
-
-
-### On Signatures & Modules
-
-Traditionally, LMs offer text-in-text-out interfaces. Toward modular programming, DSPy introduced signatures for the first time (as DSP Templates in Jan 2023) as a way to structure the inputs and outputs of LM interactions. Standard prompts conflate interface (“what should the LM do?”) with implementation (“how do we tell it to do that?”). DSPy signatures isolate the former so we can infer and learn the latter from data — in the context of a bigger program. Today in the LM landscape, notions of "structured outputs" have evolved dramatically, thanks to constrained decoding and other improvements, and have become mainstream. What may be called "structured inputs" remains is yet to become mainstream outside of DSPy, but is as crucial.
-
-Objectives in this space include refining the abstractions and implementations first-class notion of LM Adapters in DSPy, as translators that sits between signatures and LM interfaces. While Optimizers adjust prompts through interactions with a user-supplied metric and data, Adapters are more concerned with building up interactions with LMs to account for, e.g. (i) non-plaintext LM interfaces like chat APIs, structured outputs, function calling, and multi-modal APIs, (ii) languages beyond English or other forms of higher-level specialization. This has been explored in DSPy on and off in various forms, but we have started working on more fundamental approaches to this problem that will offer tangible improvements to most use-cases. Work here is currently led by Omar Khattab.
-
-
-### On Finetuning & Serving
-
-In February 2023, DSPy introduced the notion of compiling to optimize the weights of an LM program. (To understand just how long ago that was in AI terms, this was before the Alpaca training project at Stanford had even started and a month before the first GPT-4 was released.) Since then, we have shown in October 2023 and, much more expansively, in July 2024, that the fine-tuning flavor of DSPy can deliver large gains for small LMs, especially when composed with prompt optimization.
-
-Overall, though, most DSPy users in practice explore prompt optimization and not weight optimization and most of our examples do the same. The primary reason for a lot of this is infrastructure. Fine-tuning in the DSPy flavor is more than just training a model: ultimately, we need to bootstrap training data for serveral different modules in a program, train multiple models and handle model selection, and then load and plug in those models into the program's modules. Doing this robustly at the level of abstraction DSPy offers requires a level of resource management that is not generally supported by external existing tools. Major efforts in this regard are currently led by Dilara Soylu and Isaac Miller.
-
-
-### On Optimizers & Assertions
-
-This is a naturally major direction in the course of polishing. We will share more thoughts here after making more progress on the three angles above.
-
-
-
-## 2) Developing more accurate, lower-cost optimizers.
-
-A very large fraction of the research in DSPy focuses on optimizing the prompts and the weights of LM programs. In December 2022, we introduced the algorithm and abstractions behind BootstrapFewShot (as Demonstrate in DSP) and several of its variants. In February 2023, we introduced the core version of what later became BootstrapFinetune. In August 2023, we introduced new variations of both of these. In December 2023, we introduced the first couple of instruction optimizers into DSPy, CA-OPRO and early versions of MIPRO. These were again upgraded in March 2024. Fast forward to June and July 2024, we released MIPROv2 for prompt optimization and BetterTogether for fine-tuning the weights of LM programs.
-
-We have been working towards a number of stronger optimizers. While we cannot share the internal details of research on new optimizers yet, we can outline the goals. A DSPy optimizer can be characterized via three angles:
-
-1. Quality: How much quality can it deliver from various LMs? How effective does it need the zero-shot program to be in order to work well?
-2. Cost: How many labeled (and unlabeled) inputs does it need? How many invocations of the program does it need? How expensive is the resulting optimized program at inference time?
-3. Robustness: How well can it generalize to different unseen data points or distributions? How sensitive is it to mistakes of the metric or labels?
-
-Over the next six months, our goal is to dramatically improve each angle of these _when the other two are held constant_.  Concretely, there are three directions here.
-
-- Benchmarking: A key prerequisite here is work on benchmarking. On the team, Michael Ryan and Shangyin Tan are leading these efforts. More soon.
-
-- Quality: The goal here is optimizers that extract, on average, 20% more on representative tasks than MIPROv2 and BetterTogether, under the usual conditions — like a few hundred inputs with labels and a good metric starting from a decent zero-shot program. Various efforts here are led by Dilara Soylu, Michael Ryan, Josh Purtell, Krista Opsahl-Ong, and Isaac Miller.
-
-- Efficiency: The goal here is optimizers that match the current best scores from MIPROv2 and BetterTogether but under 1-2 challenges like: (i) starting from only 10-20 inputs with labels, (ii) starting with a weak zero-shot program that scores 0%, (iii) where significant misalignment exists between train/validation and test, or (iii) where the user supplies no metric but provides a very small number of output judgments.
-
-
-
-## 3) Building end-to-end tutorials from DSPy’s ML workflow to deployment.
-
-Using DSPy well for solving a new task is just doing good machine learning with LMs, but teaching this is hard. On the one hand, it's an iterative process: you make some initial choices, which will be sub-optimal, and then you refine them incrementally. It's highly exploratory: it's often the case that no one knows yet how to best solve a problem in a DSPy-esque way. One the other hand, DSPy offers many emerging lessons from several years of building LM systems, in which the design space, the data regime, and many other factors are new both to ML experts and to the very large fraction of users that have no ML experience.
-
-Though current docs do address [a bunch of this](https://dspy-docs.vercel.app/docs/building-blocks/solving_your_task) in isolated ways, one thing we've learned is that we should separate teaching the core DSPy language (which is ultimately pretty small) from teaching the emerging ML workflow that works well in a DSPy-esque setting. As a natural extension of this, we need to place more emphasis on steps prior and after to the explicit coding in DSPy, from data collection to deployment that serves and monitors the optimized DSPy program in practice. This is just starting but efforts will be ramping up led by Omar Khattab, Isaac Miller, and Herumb Shandilya.
-
-
-## 4) Shifting towards more interactive optimization & tracking.
-
-Right now, a DSPy user has a few ways to observe and tweak the process of optimization. They can study the prompts before, during, and after optimization methods like `inspect_history`, built-in logging, and/or the metadata returned by optimizers. Similarly, they can rely on `program.save` and `program.load` to potentially adjust the optimized prompts by hand. Alternatively, they can use one of the many powerful observability integrations — like from Phoenix Arize, LangWatch, or Weights & Biases Weave — to observe _in real time_ the process of optimization (e.g., scores, stack traces, successful & failed traces, and candidate prompts). DSPy encourages iterative engineering by adjusting the program, data, or metrics across optimization runs. For example, some optimizers allow “checkpointing” — e.g., if you optimize with BootstrapFewShotWithRandomSearch for 10 iterations then increase to 15 iterations, the first 10 will be loaded from cache.
-
-While these can accomplish a lot of goals, there are two limitations that future versions of DSPy will seek to address.
-
-1. In general, DSPy’s (i) observability, (ii) experimental tracking, (iii) cost management, and (iii) deployment of programs should become first-class concerns via integration with tools like MLFlow. We will share more plans addressing this for DSPy 2.6 in the next 1-2 months.
-
-2. DSPy 3.0 will introduce new optimizers that prioritize ad-hoc, human-in-the-loop feedback. This is perhaps the only substantial paradigm shift we see as necessary in the foreseeable future in DSPy. It involves various research questions at the level of the abstractions, UI/HCI, and ML, so it is a longer-term goal that we will share more about in the next 3-4 month.
-
-
diff --git a/docs/docusaurus.config.ts b/docs/docusaurus.config.ts
index 9585262d..02557662 100644
--- a/docs/docusaurus.config.ts
+++ b/docs/docusaurus.config.ts
@@ -84,9 +84,11 @@ const config: Config = {
           position: 'left', // Position in the navbar
           label: 'Documentation', // Label for the navbar item
         },
+        // Navbar item for the API reference, linking to the intro document
         { to: '/docs/category/tutorials', label: 'Tutorials', position: 'left' },
+        // Navbar item for the API reference, linking to the intro document
         { to: '/api/intro', label: 'API References', position: 'left' },
-        { to: '/docs/dspy-usecases', label: 'DSPy Use Cases', position: 'left' },
+        // Navbar item for the API reference, linking to the intro document
         { to: '/docs/cheatsheet', label: 'DSPy Cheatsheet', position: 'right' },
         // Navbar item for the GitHub repository
         {
diff --git a/dspy/evaluate/evaluate.py b/dspy/evaluate/evaluate.py
index fc7f0891..be071629 100644
--- a/dspy/evaluate/evaluate.py
+++ b/dspy/evaluate/evaluate.py
@@ -36,7 +36,6 @@ class Evaluate:
         display_progress=False,
         display_table=False,
         max_errors=5,
-        return_all_scores=False,
         return_outputs=False,
         **_kwargs,
     ):
@@ -49,7 +48,6 @@ class Evaluate:
         self.error_count = 0
         self.error_lock = threading.Lock()
         self.cancel_jobs = threading.Event()
-        self.return_all_scores = return_all_scores
         self.return_outputs = return_outputs
 
         if "display" in _kwargs:
@@ -140,16 +138,15 @@ class Evaluate:
         num_threads=None,
         display_progress=None,
         display_table=None,
-        return_all_scores=None,
-        return_outputs=None,
+        return_all_scores=False,
+        return_outputs=False,
     ):
         metric = metric if metric is not None else self.metric
         devset = devset if devset is not None else self.devset
         num_threads = num_threads if num_threads is not None else self.num_threads
         display_progress = display_progress if display_progress is not None else self.display_progress
         display_table = display_table if display_table is not None else self.display_table
-        return_all_scores = return_all_scores if return_all_scores is not None else self.return_all_scores
-        return_outputs = return_outputs if return_outputs is not None else self.return_outputs
+        return_outputs = return_outputs if return_outputs is not False else self.return_outputs
         results = []
 
         def wrapped_program(example_idx, example):
diff --git a/dspy/functional/functional.py b/dspy/functional/functional.py
index 20035476..2b39cd13 100644
--- a/dspy/functional/functional.py
+++ b/dspy/functional/functional.py
@@ -116,9 +116,12 @@ class TypedPredictor(dspy.Module):
         """Return a string representation of the TypedPredictor object."""
         return f"TypedPredictor({self.signature})"
 
-    def _make_example(self, type_) -> str:
+    def _make_example(self, field) -> str:
         # Note: DSPy will cache this call so we only pay the first time TypedPredictor is called.
-        schema = json.dumps(type_.model_json_schema())
+        if hasattr(field, "model_json_schema"):
+            pass
+        schema = field.json_schema_extra["schema"]
+        parser = field.json_schema_extra["parser"]
         if self.wrap_json:
             schema = "```json\n" + schema + "\n```\n"
         json_object = dspy.Predict(
@@ -127,9 +130,9 @@ class TypedPredictor(dspy.Module):
                 "Make a very succinct json object that validates with the following schema",
             ),
         )(json_schema=schema).json_object
-        # We use the model_validate_json method to make sure the example is valid
+        # We use the parser to make sure the json object is valid.
         try:
-            type_.model_validate_json(_unwrap_json(json_object, type_.model_validate_json))
+            parser(_unwrap_json(json_object, parser))
         except (pydantic.ValidationError, ValueError):
             return ""  # Unable to make an example
         return json_object
@@ -225,32 +228,21 @@ class TypedPredictor(dspy.Module):
                         format=lambda x: x if isinstance(x, str) else str(x),
                         parser=type_,
                     )
-                elif False:
-                    # TODO: I don't like forcing the model to write "value" in the output.
-                    if not (inspect.isclass(type_) and issubclass(type_, pydantic.BaseModel)):
-                        type_ = pydantic.create_model("Output", value=(type_, ...), __base__=pydantic.BaseModel)
-                        to_json = lambda x, type_=type_: type_(value=x).model_dump_json()[9:-1]  # {"value":"123"}
-                        from_json = lambda x, type_=type_: type_.model_validate_json('{"value":' + x + "}").value
-                        schema = json.dumps(type_.model_json_schema()["properties"]["value"])
-                    else:
-                        to_json = lambda x: x.model_dump_json()
-                        from_json = lambda x, type_=type_: type_.model_validate_json(x)
-                        schema = json.dumps(type_.model_json_schema())
                 else:
                     # Anything else we wrap in a pydantic object
-                    if not (
+                    if (
                         inspect.isclass(type_)
                         and typing.get_origin(type_) not in (list, tuple)  # To support Python 3.9
                         and issubclass(type_, pydantic.BaseModel)
                     ):
-                        type_ = pydantic.create_model("Output", value=(type_, ...), __base__=pydantic.BaseModel)
-                        to_json = lambda x, type_=type_: type_(value=x).model_dump_json()
-                        from_json = lambda x, type_=type_: type_.model_validate_json(x).value
-                        schema = json.dumps(type_.model_json_schema())
-                    else:
                         to_json = lambda x: x.model_dump_json()
                         from_json = lambda x, type_=type_: type_.model_validate_json(x)
                         schema = json.dumps(type_.model_json_schema())
+                    else:
+                        adapter = pydantic.TypeAdapter(type_)
+                        to_json = lambda x: adapter.serializer.to_json(x)
+                        from_json = lambda x, type_=adapter: type_.validate_json(x)
+                        schema = json.dumps(adapter.json_schema())
                     if self.wrap_json:
                         to_json = lambda x, inner=to_json: "```json\n" + inner(x) + "\n```\n"
                         schema = "```json\n" + schema + "\n```"
@@ -260,6 +252,7 @@ class TypedPredictor(dspy.Module):
                         + (". Respond with a single JSON object. JSON Schema: " + schema),
                         format=lambda x, to_json=to_json: (x if isinstance(x, str) else to_json(x)),
                         parser=lambda x, from_json=from_json: from_json(_unwrap_json(x, from_json)),
+                        schema=schema,
                         type_=type_,
                     )
             else:  # If input field
@@ -321,7 +314,7 @@ class TypedPredictor(dspy.Module):
                         if (
                             try_i + 1 < self.max_retries
                             and prefix not in current_desc
-                            and (example := self._make_example(field.annotation))
+                            and (example := self._make_example(field))
                         ):
                             signature = signature.with_updated_fields(
                                 name,
@@ -405,9 +398,13 @@ def _func_to_signature(func):
     return dspy.Signature(fields, instructions)
 
 
-def _unwrap_json(output, from_json: Callable[[str], Union[pydantic.BaseModel, str]]):
+def _unwrap_json(output, from_json: Callable[[str], Union[pydantic.BaseModel, str, None]]):
     try:
-        return from_json(output).model_dump_json()
+        parsing_result = from_json(output)
+        if isinstance(parsing_result, pydantic.BaseModel):
+            return parsing_result.model_dump_json()
+        else:
+            return output
     except (ValueError, pydantic.ValidationError, AttributeError):
         output = output.strip()
         if output.startswith("```"):
@@ -416,6 +413,4 @@ def _unwrap_json(output, from_json: Callable[[str], Union[pydantic.BaseModel, st
             if not output.endswith("```"):
                 raise ValueError("Don't write anything after the final json ```") from None
             output = output[7:-3].strip()
-        if not output.startswith("{") or not output.endswith("}"):
-            raise ValueError("json output should start and end with { and }") from None
         return ujson.dumps(ujson.loads(output))  # ujson is a bit more robust than the standard json


Test patch:
diff --git a/tests/functional/test_functional.py b/tests/functional/test_functional.py
index 3e605566..9e84d839 100644
--- a/tests/functional/test_functional.py
+++ b/tests/functional/test_functional.py
@@ -36,7 +36,7 @@ def test_list_output():
         pass
 
     expected = ["What is the speed of light?", "What is the speed of sound?"]
-    lm = DummyLM(['{"value": ["What is the speed of light?", "What is the speed of sound?"]}'])
+    lm = DummyLM(['["What is the speed of light?", "What is the speed of sound?"]'])
     dspy.settings.configure(lm=lm)
 
     question = hard_questions(topics=["Physics", "Music"])
@@ -557,7 +557,7 @@ def test_parse_type_string():
 
 
 def test_literal():
-    lm = DummyLM([f'{{"value": "{i}"}}' for i in range(100)])
+    lm = DummyLM(['"2"', '"3"'])
     dspy.settings.configure(lm=lm)
 
     @predictor
@@ -567,8 +567,22 @@ def test_literal():
     assert f() == "2"
 
 
+def test_literal_missmatch():
+    lm = DummyLM([f'"{i}"' for i in range(5, 100)])
+    dspy.settings.configure(lm=lm)
+
+    @predictor(max_retries=1)
+    def f() -> Literal["2", "3"]:
+        pass
+
+    with pytest.raises(Exception) as e_info:
+        f()
+
+    assert e_info.value.args[1]["f"] == "Input should be '2' or '3':  (error type: literal_error)"
+
+
 def test_literal_int():
-    lm = DummyLM([f'{{"value": {i}}}' for i in range(100)])
+    lm = DummyLM(["2", "3"])
     dspy.settings.configure(lm=lm)
 
     @predictor
@@ -578,6 +592,20 @@ def test_literal_int():
     assert f() == 2
 
 
+def test_literal_int_missmatch():
+    lm = DummyLM([f"{i}" for i in range(5, 100)])
+    dspy.settings.configure(lm=lm)
+
+    @predictor(max_retries=1)
+    def f() -> Literal[2, 3]:
+        pass
+
+    with pytest.raises(Exception) as e_info:
+        f()
+
+    assert e_info.value.args[1]["f"] == "Input should be 2 or 3:  (error type: literal_error)"
+
+
 def test_fields_on_base_signature():
     class SimpleOutput(dspy.Signature):
         output: float = dspy.OutputField(gt=0, lt=1)
@@ -896,9 +924,7 @@ def _test_demos_missing_input():
 
 
 def test_conlist():
-    dspy.settings.configure(
-        lm=DummyLM(['{"value": []}', '{"value": [1]}', '{"value": [1, 2]}', '{"value": [1, 2, 3]}'])
-    )
+    dspy.settings.configure(lm=DummyLM(["[]", "[1]", "[1, 2]", "[1, 2, 3]"]))
 
     @predictor
     def make_numbers(input: str) -> Annotated[list[int], Field(min_items=2)]:
@@ -908,9 +934,7 @@ def test_conlist():
 
 
 def test_conlist2():
-    dspy.settings.configure(
-        lm=DummyLM(['{"value": []}', '{"value": [1]}', '{"value": [1, 2]}', '{"value": [1, 2, 3]}'])
-    )
+    dspy.settings.configure(lm=DummyLM(["[]", "[1]", "[1, 2]", "[1, 2, 3]"]))
 
     make_numbers = TypedPredictor("input:str -> output:Annotated[List[int], Field(min_items=2)]")
     assert make_numbers(input="What are the first two numbers?").output == [1, 2]
diff --git a/tests/functional/test_signature_opt_typed.py b/tests/functional/test_signature_opt_typed.py
index 85c8c8d5..59a610e6 100644
--- a/tests/functional/test_signature_opt_typed.py
+++ b/tests/functional/test_signature_opt_typed.py
@@ -10,7 +10,8 @@ from dspy.utils import DummyLM
 from dspy.evaluate import Evaluate
 from dspy.evaluate.metrics import answer_exact_match
 from dspy.functional import TypedPredictor
-
+import json
+from pydantic_core import to_jsonable_python
 
 hotpotqa = [
     ex.with_inputs("question")
@@ -109,7 +110,7 @@ def test_opt():
         [
             # Seed prompts
             "some thoughts",
-            '{"value": [{"instructions": "I", "question_desc": "$q", "question_prefix": "Q:", "answer_desc": "$a", "answer_prefix": "A:"}]}',
+            '[{"instructions": "I", "question_desc": "$q", "question_prefix": "Q:", "answer_desc": "$a", "answer_prefix": "A:"}]',
         ]
     )
     dspy.settings.configure(lm=qa_model)
@@ -163,18 +164,13 @@ def test_opt_composed():
 
     info2 = make_info(ExpectedSignature2)
 
-    T = TypeVar("T")
-
-    class OutputWrapper(pydantic.BaseModel, Generic[T]):
-        value: list[T]
-
     qa_model = DummyLM([])
     prompt_model = DummyLM(
         [
             "some thoughts",
-            OutputWrapper[type(info1)](value=[info1]).model_dump_json(),
+            json.dumps([to_jsonable_python(info1)]),
             "some thoughts",
-            OutputWrapper[type(info2)](value=[info2]).model_dump_json(),
+            json.dumps([to_jsonable_python(info2)]),
         ]
     )
     dspy.settings.configure(lm=qa_model)
diff --git a/tests/functional/test_signature_typed.py b/tests/functional/test_signature_typed.py
new file mode 100644
index 00000000..39eb8c2f
--- /dev/null
+++ b/tests/functional/test_signature_typed.py
@@ -0,0 +1,154 @@
+from typing import Any, Optional, Union
+import pytest
+
+import pydantic
+import dspy
+from dspy.functional import TypedPredictor
+from dspy.signatures.signature import signature_to_template
+
+
+def get_field_and_parser(signature: dspy.Signature) -> tuple[Any, Any]:
+    module = TypedPredictor(signature)
+    signature = module._prepare_signature()
+    assert "answer" in signature.fields, "'answer' not in signature.fields"
+    field = signature.fields.get("answer")
+    parser = field.json_schema_extra.get("parser")
+    return field, parser
+
+
+class Mysubmodel(pydantic.BaseModel):
+    sub_floating: float
+
+
+class MyModel(pydantic.BaseModel):
+    floating: float
+    string: str
+    boolean: bool
+    integer: int
+    optional: Optional[str]
+    sequence_of_strings: list[str]
+    union: Union[str, float]
+    submodel: Mysubmodel
+    optional_submodel: Optional[Mysubmodel]
+    optional_existing_submodule: Optional[Mysubmodel]
+
+
+def build_model_instance() -> MyModel:
+    return MyModel(
+        floating=3.14,
+        string="foobar",
+        boolean=True,
+        integer=42,
+        optional=None,
+        sequence_of_strings=["foo", "bar"],
+        union=3.14,
+        submodel=Mysubmodel(sub_floating=42.42),
+        optional_submodel=None,
+        optional_existing_submodule=Mysubmodel(sub_floating=42.42),
+    )
+
+
+@pytest.mark.parametrize(
+    "test_type,serialized, expected", [(str, "foo", "foo"), (int, "42", 42), (float, "42.42", 42.42)]
+)
+def test_basic_types(test_type: type, serialized: str, expected: Any):
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: test_type = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+    assert parser is test_type, "Parser is not correct for 'answer'"
+    assert parser(serialized) == expected, f"{test_type}({serialized})!= {expected}"
+
+
+def test_boolean():
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: bool = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+    assert parser("true"), f"Parsing 'true' failed"
+    assert not parser("false"), f"Parsing 'false' failed"
+
+
+@pytest.mark.parametrize(
+    "test_type,serialized, expected",
+    [(list[str], '["foo", "bar"]', ["foo", "bar"]), (tuple[int, float], "[42, 3.14]", (42, 3.14))],
+)
+def test_sequences(test_type: type, serialized: str, expected: Any):
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: test_type = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+
+    assert parser(serialized) == expected, f"Parsing {expected} failed"
+
+
+@pytest.mark.parametrize(
+    "test_type,serialized, expected",
+    [
+        (Optional[str], '"foobar"', "foobar"),
+        (Optional[str], "null", None),
+        (Union[str, float], "3.14", 3.14),
+        (Union[str, bool], "true", True),
+    ],
+)
+def test_unions(test_type: type, serialized: str, expected: Any):
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: test_type = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+
+    assert parser(serialized) == expected, f"Parsing {expected} failed"
+
+
+def test_pydantic():
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: MyModel = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+
+    instance = build_model_instance()
+    parsed_instance = parser(instance.model_dump_json())
+
+    assert parsed_instance == instance, f"{instance} != {parsed_instance}"
+
+
+def test_optional_pydantic():
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: Optional[MyModel] = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+
+    instance = build_model_instance()
+    parsed_instance = parser(instance.model_dump_json())
+    assert parsed_instance == instance, f"{instance} != {parsed_instance}"
+
+    # Check null case
+    parsed_instance = parser("null")
+    assert parsed_instance == None, "Optional[MyModel] should be None"
+
+
+def test_dataclass():
+    from dataclasses import dataclass
+
+    @dataclass(frozen=True)
+    class MyDataclass:
+        string: str
+        number: int
+        floating: float
+        boolean: bool
+
+    class MySignature(dspy.Signature):
+        question: str = dspy.InputField()
+        answer: MyDataclass = dspy.OutputField()
+
+    _, parser = get_field_and_parser(MySignature)
+
+    instance = MyDataclass("foobar", 42, 3.14, True)
+    parsed_instance = parser('{"string": "foobar", "number": 42, "floating": 3.14, "boolean": true}')
+    assert parsed_instance == instance, f"{instance} != {parsed_instance}"